A Virtual Reality Image and Sound Synchronization Optimization System Based on User Interaction

Abstract

This invention belongs to the field of virtual reality technology. It provides a user-interactive virtual reality image and sound synchronization optimization system, comprising: a visual constraint module, which obtains visual perception constraint values ​​that fuse occlusion and viewpoint offset by dividing the field of view into grids and analyzing sampling units; a spatial loss module, which calculates the radial base distance based on three-dimensional coordinates and generates a spatial loss coefficient by combining the scene damping coefficient; a collaborative judgment module, which introduces original sound effect parameters for audio-visual collaborative analysis, constructs a visual-spatial dual-axis dynamic model when conflicts are triggered, and outputs audio correction coefficients; a dual-axis reconstruction module, which adjusts volume, high-frequency ratio, and sound field diffusion according to the correction coefficients to achieve dynamic audio rendering; and a synchronization perception module, which verifies the audio-visual synchronization quality by comparing theoretical and actual perception values; ensuring audio-visual perception consistency in complex virtual environments and enhancing the user's immersive experience.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality technology, specifically a virtual reality image and sound synchronization optimization system based on user interaction. Background Technology

[0002] With the rapid iteration of virtual reality (VR) technology, multiplayer collaborative VR games have become one of the core applications of VR due to their strong interactivity and immersive experience. In these games, users enter a virtual collaborative environment through VR devices and rely on the coordinated perception of both visual and auditory channels to achieve teammate positioning, action coordination, and information interaction. The consistency of audio-visual perception directly determines the user's immersive experience and collaborative efficiency.

[0003] However, the current audio system design for VR multiplayer collaborative games still has shortcomings. Its sound field modeling logic is based solely on the spatial distance between the user and teammates, and adjusts the audio output by calculating the linear propagation loss through three-dimensional coordinates. It does not establish any linkage with the visual and spatial data generated by the VR device. Even if the user's view is obstructed and only the teammate's arm is visible, or the teammate is at the edge of the field of vision, the audio still maintains the omnidirectional sound field intensity of the unobstructed state. It does not decrease synchronously with the degree of image obstruction and the degree of field of vision shift, resulting in a serious disconnect between auditory perception and visual scene. In addition, the spatial attenuation calculation does not take into account the acoustic differences of the scene. The sound wave propagation characteristics (such as damping and reflection differences) of different environments such as indoor, outdoor open spaces, and caves are not included in the model. Attenuation is only calculated through a fixed distance threshold, which further exacerbates the audio-visual disharmony in complex scenes.

[0004] To this end, the present invention provides a virtual reality image and sound synchronization optimization system based on user interaction. Summary of the Invention

[0005] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0006] The technical solution adopted by this invention to solve its technical problem is: a virtual reality image and sound synchronization optimization system based on user interaction, comprising the following modules:

[0007] Visual constraint module: acquires the user's virtual reality field of view data, divides the virtual reality field of view data into a grid matrix; acquires the basic units in the grid matrix and performs visual constraint analysis to obtain visual perception constraint values;

[0008] Space loss module: Acquires the three-dimensional spatial coordinates of the user and teammates and calculates the radial base distance using Euclidean distance; performs spatial attenuation analysis based on the radial base distance to obtain the space loss coefficient;

[0009] Collaborative Judgment Module: Based on the visual perception constraint value and spatial loss coefficient, the original sound effect parameters are introduced to perform audio-visual collaborative analysis to obtain the expected audio perception value. Based on the expected audio perception value, it is determined whether a multi-sensory conflict signal is triggered. If it is triggered, a visual-spatial dual-axis dynamic model is constructed based on the visual perception constraint value and spatial loss coefficient.

[0010] Dual-axis reconstruction module: Based on the visual-spatial dual-axis dynamic model, the module obtains the original sound effect parameters of teammates and performs audio dynamic reconstruction analysis to obtain sound effect correction parameter values; based on the sound effect correction parameter values, the module dynamically renders and adjusts the original sound effects of teammates.

[0011] Synchronization Perception Module: Obtains the original sound effect parameters and actual sound effect parameters of the current frame after dynamic rendering adjustment, and performs perception deviation analysis to obtain the perception consistency value; based on the perception consistency value, identifies whether the current audio-visual synchronization meets the quality standard.

[0012] Furthermore, the process of performing the aforementioned visual constraint analysis is as follows:

[0013] Obtain the basic units in the grid matrix to identify target sampling units and occlusion sampling units, and count the number of target sampling units and occlusion sampling units respectively;

[0014] The region occlusion index is calculated by the ratio of the number of occluded sampling units to the number of target sampling units.

[0015] The center point of the user's VR device is set as the origin, and a two-dimensional coordinate system is constructed based on the origin; parallax analysis is performed based on the two-dimensional coordinate system to obtain the viewpoint offset.

[0016] The visual perception constraint value is obtained by multiplying the region occlusion index by the viewpoint offset.

[0017] Furthermore, the process of performing the disparity analysis is as follows:

[0018] Based on the two-dimensional coordinate system, the two-dimensional coordinates of all target sampling units are obtained, and the two-dimensional coordinates of all target sampling units are averaged to obtain the two-dimensional coordinates of the teammate's body center of gravity.

[0019] Calculate the Euclidean distance from the origin to the teammate's center of gravity using the Euclidean distance formula.

[0020] Input the half-width of the X-axis and the half-height of the Y-axis into the Pythagorean theorem formula to obtain the normalization factor.

[0021] The viewpoint offset is calculated by dividing the Euclidean distance by the normalization factor.

[0022] Furthermore, the process of obtaining the radial base distance is as follows:

[0023] A multi-source positioning fusion system based on VR devices can collect the three-dimensional spatial coordinates of users and teammates in real time.

[0024] The linear propagation distance between the user and teammates in three-dimensional space is calculated using the three-dimensional Euclidean distance formula and marked as the radial base distance.

[0025] Furthermore, the process of performing the spatial attenuation analysis is as follows:

[0026] Obtain the normal effective acoustic distance under different scenarios, and calculate the basis distance attenuation coefficient under different scenarios by the ratio of radial basis distance to normal effective acoustic distance;

[0027] Obtain the environmental damping coefficient under different scenarios; calculate the space loss coefficient by multiplying the base distance attenuation coefficient and the environmental damping coefficient.

[0028] Furthermore, the process of performing the aforementioned audio-visual co-analysis is as follows:

[0029] Obtain the original sound effect parameters of teammates; sum the reference high frequency ratio and the reference sound field diffusion, and multiply the summation result with the reference volume to obtain the reference effective perceived value;

[0030] Obtain the expected perceived value of the audio based on the original sound effect parameters;

[0031] The audio-visual coordination deviation value is obtained by calculating the difference between the baseline effective perceived value and the expected perceived value of the audio.

[0032] Furthermore, the method for obtaining the expected perceived value of the audio is as follows:

[0033] The expected volume value is calculated by multiplying the reference volume with the visual perception constraint value and the spatial loss coefficient.

[0034] The expected value of the high frequency ratio is obtained by multiplying the baseline high frequency ratio with the visual perception constraint value and the spatial loss coefficient.

[0035] The expected sound field diffusion value is obtained by multiplying the reference sound field diffusion value with the visual perception constraint value and the spatial loss coefficient.

[0036] The expected value of high frequency ratio is summed with the expected value of sound field diffusion, and the result of the summation is multiplied with the expected value of volume to obtain the expected audio perceived value.

[0037] Furthermore, the visual-spatial dual-axis dynamic model is constructed as follows:

[0038] The difference between the visual perception constraint value and the unit quantity is calculated, and the result of the difference calculation is multiplied by the spatial loss coefficient to obtain the audio correction coefficient.

[0039] Furthermore, the process of performing the aforementioned perceptual bias analysis is as follows:

[0040] Obtain the visual perception constraint value and spatial loss coefficient of the current frame, and obtain the theoretical audio correction coefficient based on the visual perception constraint value and spatial loss coefficient of the current frame;

[0041] Get the original sound effect parameters of teammates in the current frame and the actual sound effect parameters after rendering;

[0042] Perceptual consistency values ​​are obtained by performing consistency analysis based on the original sound effect parameters of teammates in the current frame and the actual sound effect parameters after rendering.

[0043] Furthermore, the process of performing the consistency analysis is as follows:

[0044] The high-frequency proportion of teammates in the current frame is summed with the reference sound field diffusion. The summation result is then multiplied with the reference volume to obtain the theoretical expected perceived value.

[0045] The actual high-frequency proportion of teammates in the current frame is summed with the actual sound field diffusion. The summation result is then multiplied with the actual output volume to obtain the actual effective perceived value.

[0046] The absolute difference between the actual effective perceived value and the theoretical expected perceived value is calculated, and the result of the absolute difference calculation is compared with the theoretical expected perceived value to obtain the perceived consistency value.

[0047] The beneficial effects of this invention are as follows:

[0048] 1. Obtain the user's virtual reality field of view data, and divide the virtual reality field of view data into a grid matrix; obtain the basic units in the grid matrix and perform visual constraint analysis to obtain the visual perception constraint value;

[0049] By dividing the data into grid matrices and analyzing visual constraints, the limitations of visual perception within the user's field of vision are quantified, and resource allocation is optimized to enhance the comfort and realism of the immersive experience. The three-dimensional spatial coordinates of the user and teammates are obtained, and Euclidean distance is calculated to obtain the radial base distance. Spatial attenuation analysis is performed based on the radial base distance to obtain the spatial loss coefficient. Through three-dimensional coordinate distance calculation and spatial attenuation analysis, the interaction distance loss between the user and teammates is obtained, optimizing team collaboration efficiency and enhancing the realism of interaction in the virtual space. Through audio-visual coordination analysis and multi-sensory conflict detection, the fusion effect of vision, space, and hearing can be dynamically optimized, improving the sensory consistency and user comfort of the immersive scene.

[0050] 2. Based on the visual-spatial dual-axis dynamic model, the original sound effect parameters of teammates are obtained, and audio dynamic reconstruction analysis is performed to obtain sound effect correction parameter values. Based on the sound effect correction parameter values, the original sound effects of teammates are dynamically rendered and adjusted. Through the visual-spatial dual-axis dynamic model and audio-visual collaborative analysis, efficient fusion of multi-sensory signals and conflict warning can be achieved, comprehensively improving the immersion, interactivity, and user comfort of virtual reality scenes. The original sound effect parameters and actual sound effect parameters of the current frame after dynamic rendering adjustment are obtained, and perceptual deviation analysis is performed to obtain perceptual consistency values. Based on the perceptual consistency values, it is identified whether the current audio-visual synchronization meets the quality standard. Through perceptual deviation analysis and the judgment of the audio-visual synchronization quality standard, the synchronization of audio and visual signals can be ensured in real time, significantly improving the immersion and multi-sensory experience consistency of virtual reality scenes, and reducing user discomfort or immersion breakage caused by audio-visual asynchrony. Attached Figure Description

[0051] The invention will now be further described with reference to the accompanying drawings.

[0052] Figure 1 This is a block diagram of a virtual reality image and sound synchronization optimization system based on user interaction, as described in an embodiment of the present invention.

[0053] Figure 2 This is a logic diagram for determining whether a multi-sensory conflict signal is triggered, as described in an embodiment of the present invention.

[0054] Figure 3 This is a flowchart illustrating the steps of a virtual reality image and sound synchronization optimization method based on user interaction, as described in an embodiment of the present invention. Detailed Implementation

[0055] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0056] Example 1: Please refer to Figure 1As shown in the embodiment of the present invention, a virtual reality image and sound synchronization optimization system based on user interaction is provided. This solution mainly addresses the problem of image and sound asynchrony in virtual reality scenes caused by visual occlusion, viewpoint shift, spatial distance, and scene differences. This problem can cause multi-sensory conflicts, resulting in a sense of disharmony due to the mismatch between audio perception and visual-spatial constraints (such as clear audio at a distance, no attenuation in complex scenes, and clear sound despite visual blur), which disrupts the user's immersive experience and perceptual consistency, and may even cause user discomfort. This solution sets up a visual constraint module, a spatial loss module, a collaborative judgment module, and a dual-axis reconstruction module. First, it quantifies the visual perception constraint value and spatial loss coefficient, then constructs a visual-spatial dual-axis dynamic model through audio-visual collaborative analysis, and finally dynamically corrects parameters such as audio volume, high-frequency ratio, and sound field diffusion based on this model to achieve real-time synchronization of image and sound. The system includes the following modules:

[0057] Visual constraint module: acquires the user's virtual reality field of view data, divides the virtual reality field of view data into a grid matrix; acquires the basic units in the grid matrix and performs visual constraint analysis to obtain visual perception constraint values;

[0058] The process of acquiring virtual reality field-of-view data and dividing the grid matrix based on the virtual reality field-of-view data is as follows:

[0059] Obtain the user's virtual reality field of view data at the current moment;

[0060] Understandably, virtual reality field-of-view data is a comprehensive data set generated by a user's VR device, including scene color, object depth, and position information;

[0061] The virtual reality field of view data is discretized into a grid matrix consisting of several sampling units;

[0062] The discretization process is as follows: the virtual reality field data is uniformly divided into grids along the horizontal and vertical directions to generate a regular grid matrix composed of equal-area sampling units; each sampling unit serves as the basic unit for spatial analysis.

[0063] The process of performing visual constraint analysis is as follows:

[0064] Based on computer vision technology, each sampling unit is analyzed in real time. Sampling units with virtual walls are identified and marked as interfering sampling units, sampling units with teammates' body parts are identified and marked as target sampling units, and sampling units with both virtual walls and teammates' body parts are identified and marked as occluded sampling units.

[0065] It should be noted that virtual walls are objects that obstruct the view in the scene, such as buildings, rocks, and virtual interfaces.

[0066] The number of target sampling units and occlusion sampling units are counted separately.

[0067] The region occlusion index is calculated by the ratio of the number of occluded sampling units to the number of target sampling units.

[0068] Set the center point of the user's VR device as the origin, and construct a two-dimensional coordinate system based on the origin;

[0069] The process of constructing a two-dimensional coordinate system is as follows:

[0070] The horizontal direction based on the origin is set as the X-axis (the positive direction is to the right), and the vertical direction based on the origin is set as the Y-axis (the positive direction is upward).

[0071] In some embodiments, if the field of view is 120 units horizontally and 90 units vertically, then the X coordinate of the right edge of the screen is +60, the left edge is -60, the Y coordinate of the top edge is +45, and the bottom edge is -45; each sampling unit is projected into a two-dimensional coordinate system to obtain two-dimensional coordinates;

[0072] Obtain the two-dimensional coordinates of all target sampling units, and average the two-dimensional coordinates of all target sampling units to obtain the two-dimensional coordinates of the teammate's body center of gravity.

[0073] Calculate the Euclidean distance from the origin to the teammate's center of gravity using the Euclidean distance formula.

[0074] Input the half-width of the X-axis and the half-height of the Y-axis into the Pythagorean theorem formula to obtain the normalization factor.

[0075] It should be noted that the half-width of the X-axis is half the length of the X-axis, and the half-height of the Y-axis is half the height of the Y-axis.

[0076] Understandably, the normalization factor serves to transform the Euclidean distance from the teammate's center of gravity in two dimensions to the origin into a dimensionless standard comparable value within a specific range (0,1).

[0077] The viewpoint offset is calculated by the ratio of the Euclidean distance to the normalization factor.

[0078] It should be noted that the view offset is a normalized value between 0 and 1, which measures the degree to which a teammate's body deviates from the center of the user's field of vision.

[0079] The visual perception constraint value is calculated by multiplying the region occlusion index and the viewpoint offset.

[0080] It should be noted that the physical meaning reflected by the visual perception constraint value lies in quantifying the comprehensive degree of obstruction encountered by the user's visual perception channel in the virtual environment. The visual perception constraint value is the product of the environmental occlusion effect and the viewpoint deviation effect: the area occlusion index reflects the blocking rate of the user's visual information by the environmental geometry, that is, the physical attenuation of the visual signal on the propagation path; the viewpoint deviation represents the decrease in perception efficiency caused by teammates deviating from the user's visual focus (center of vision). Specifically, the visual perception constraint value is essentially a comprehensive measure of the availability of visual information. The larger the value, the lower the efficiency and the greater the difficulty for the user to obtain information such as the status of teammates through visual perception, and the worse the effectiveness of the visual channel.

[0081] It should also be noted that the purpose of obtaining the visual perception constraint value is to enable the system to obtain the user's perception of teammates in real time, thereby identifying the moment when the visual perception is restricted and corresponding audio adjustment is required; secondly, the visual perception constraint value is the core input that drives the dynamic correction of audio. When the user's vision is restricted, the audio of teammates should be adjusted synchronously to ensure that the virtual reality image and sound are synchronized, thereby enhancing the user's immersive experience.

[0082] Space loss module: Acquires the three-dimensional spatial coordinates of the user and teammates and calculates the radial base distance using Euclidean distance; performs spatial attenuation analysis based on the radial base distance to obtain the space loss coefficient;

[0083] The process of obtaining the three-dimensional spatial coordinates of the user and teammates and calculating the radial base distance using Euclidean distance is as follows:

[0084] A multi-source positioning fusion system based on VR devices (such as laser positioning + visual inertial odometry) collects the three-dimensional spatial coordinates of the user and teammates in real time, calculates the straight-line propagation distance between the user and teammates using the three-dimensional Euclidean distance formula, and marks it as the radial base distance.

[0085] The process of obtaining the spatial loss coefficient based on the spatial attenuation analysis using the radial base distance is as follows:

[0086] Obtain the normal effective acoustic distance under different scenarios, and calculate the basis distance attenuation coefficient under different scenarios by the ratio of radial basis distance to normal effective acoustic distance;

[0087] For example, the normal effective sound distance is set according to the scene type of the user and teammates. The normal effective sound distance is 15m in indoor scenes, 40m in outdoor open scenes, and 10m in caves or complex obstructed scenes.

[0088] It should be noted that indoor scenes are enclosed spaces where obstacles are mostly made of hard materials, such as virtual conference rooms or enclosed rooms; outdoor open scenes are scenes without obvious enclosed boundaries where sound waves spread in all directions, such as virtual grasslands or open squares; cave or complex obstruction scenes are narrow spaces with dense obstacles, such as virtual caves or mazes.

[0089] Obtain the environmental damping coefficients for different scenarios: A1 for indoor scenarios, A2 for outdoor open scenarios, and A3 for cave or complex obstruction scenarios.

[0090] Preferably, A1 is 0.9, A2 is 0.7, and A3 is 0.5;

[0091] The space loss coefficient is obtained by multiplying the base distance attenuation coefficient and the environmental damping coefficient.

[0092] It should be noted that the physical meaning of the spatial loss coefficient lies in quantifying the degree of energy attenuation of sound wave signals under the combined effect of distance propagation loss and scene acoustic occlusion in virtual three-dimensional space. The spatial loss coefficient specifically reflects the combined effect of energy diffusion loss caused by spatial distance and energy absorption and blocking loss caused by scene environment during the propagation of sound waves from teammates to users. The larger the spatial loss coefficient value, the more significant the sound wave energy attenuation, and the weaker the teammates' sound effects perceived by the user.

[0093] It should also be noted that the purpose of obtaining the spatial loss coefficient is to capture the comprehensive attenuation effect of distance and scene characteristics on sound wave propagation. This avoids the incongruity caused by audio relying solely on a single distance parameter, such as clear sound at extremely long distances or no attenuation in complex scenes, which would damage the user's immersive experience. It also allows for subsequent adjustments to parameters such as teammate volume and frequency bands to achieve a synchronized audio-visual effect where visual obstruction and audio at long distances are reduced, while visual clarity and audio clarity at close distances are achieved. This significantly enhances the user's sense of collaborative immersion and perceptual consistency in the virtual environment.

[0094] Collaborative Judgment Module: Based on the visual perception constraint value and spatial loss coefficient, the original sound effect parameters are introduced to perform audio-visual collaborative analysis to obtain the expected audio perception value. Based on the expected audio perception value, it is determined whether a multi-sensory conflict signal is triggered. If it is triggered, a visual-spatial dual-axis dynamic model is constructed based on the visual perception constraint value and spatial loss coefficient.

[0095] The process of obtaining the expected audio perception value by introducing the original sound effect parameters based on the visual perception constraint value and the spatial loss coefficient, and then determining whether a multi-sensory conflict signal is triggered based on the expected audio perception value is as follows:

[0096] The audio acquisition module based on VR devices extracts the original sound effect parameters of teammates in real time;

[0097] For example, the teammate's original sound parameters include the reference volume, reference high frequency ratio, and reference sound field diffusion; the original sound parameters are the original sound parameters without any modification.

[0098] It should be noted that the reference volume is the original output intensity of the teammate's voice or sound effect, the reference high frequency ratio is the proportion of high frequency energy to the total energy of the entire frequency band obtained through audio spectrum analysis, and the reference sound field diffusion is the diffusion range of the sound effect based on audio spatial directivity data.

[0099] The expected perceived value of audio is obtained by performing audio prediction analysis based on the original sound effect parameters;

[0100] The process of audio prediction analysis is as follows:

[0101] The reference high frequency ratio and the reference sound field diffusion are summed and calculated, and the result of the summation is multiplied by the reference volume to obtain the reference effective perceived value.

[0102] Understandably, the baseline effective perceived value reflects the current uncorrected multi-dimensional perceived intensity of audio.

[0103] The expected volume value is calculated by multiplying the reference volume with the visual perception constraint value and the spatial loss coefficient.

[0104] The expected value of the high frequency ratio is obtained by multiplying the baseline high frequency ratio with the visual perception constraint value and the spatial loss coefficient.

[0105] The expected sound field diffusion value is obtained by multiplying the reference sound field diffusion value with the visual perception constraint value and the spatial loss coefficient.

[0106] The expected value of high frequency ratio is summed with the expected value of sound field diffusion, and the result of the summation is multiplied with the expected value of volume to obtain the expected audio perception value.

[0107] The audio-visual coordination deviation value is obtained by calculating the difference between the baseline effective perceived value and the expected perceived value of the audio.

[0108] In some embodiments, the audio-visual coordination deviation value is compared with a reference zero point;

[0109] Preferably, the reference zero point is 0;

[0110] If the audio-visual coordination deviation value is greater than the baseline zero point, it indicates that the overall perceived intensity of the actual audio is higher than the reasonable expectation under visual-spatial constraints, resulting in a situation where the image and sound are out of sync, triggering a multi-sensory conflict signal.

[0111] If the audio-visual coordination deviation value is equal to the baseline zero point, it means that the overall perceived intensity of the actual audio is completely matched with the expected intensity under visual-spatial constraints, and no adjustment is required.

[0112] If the audio-visual coordination deviation value is less than the baseline zero point, it means that the actual audio perception intensity is lower than the reasonable expectation, and the audio intensity needs to be supplemented.

[0113] The process of constructing a visual-spatial dual-axis dynamic model based on visual perception constraint values ​​and spatial loss coefficients, if triggered, is as follows:

[0114] If triggered, a visual-spatial dual-axis dynamic model is constructed based on the visual perception constraint value and the spatial loss coefficient; the visual perception constraint value and the spatial loss coefficient are input into the visual-spatial dual-axis dynamic model to obtain the audio correction coefficient;

[0115] It is understandable that the process of constructing a visual-spatial dual-axis dynamic model is as follows:

[0116] The difference between the visual perception constraint value and the unit quantity is calculated, and the result of the difference calculation is multiplied by the spatial loss coefficient to obtain the audio correction coefficient.

[0117] Preferably, the number of units is 1;

[0118] For example, the role of constructing a visual-spatial dual-axis dynamic model is to integrate the dual constraints of visual occlusion effect and spatial acoustic loss, and realize dynamic collaborative correction of audio-visual perception. The visual perception constraint value quantifies the degree to which teammates are occluded by virtual walls and the angle offset in the user's field of vision, and combines the spatial loss coefficient to quantify the energy attenuation of sound due to distance and environmental damping. The two together constitute the dual-axis input. The model converts the degree of visual restriction and sound propagation loss into audio correction coefficients to ensure that when the user's field of vision is partially occluded (such as only seeing a teammate's arm) or when the teammate is at the edge of the field of vision, the audio can be synchronously reduced to an intensity that matches the visual perception, avoiding the incongruity of clear sound but blurred vision. This greatly improves the user's perceptual consistency and immersion in VR multi-person collaboration.

[0119] The technical solution of this embodiment is as follows: Acquire the user's virtual reality field of view data, and divide the data into a grid matrix; acquire the basic units in the grid matrix and perform visual constraint analysis to obtain visual perception constraint values; through grid matrix division and visual constraint analysis, quantify the visual perception limitations in the user's field of view, optimize resource allocation to improve the comfort and realism of the immersive experience; acquire the three-dimensional spatial coordinates of the user and teammates and perform Euclidean distance calculation to obtain the radial base distance; perform spatial attenuation analysis based on the radial base distance to obtain the spatial loss coefficient; through three-dimensional coordinate distance calculation and spatial attenuation analysis, obtain the interaction distance loss between the user and teammates, optimize team collaboration efficiency, and enhance the realism of interaction in the virtual space; through audio-visual coordination analysis and multi-sensory conflict detection, dynamically optimize the fusion effect of vision, space, and hearing, and improve the sensory consistency and user comfort of the immersive scene.

[0120] Example 2: Please refer to Figure 1 As shown in the embodiment of the present invention, a virtual reality image and sound synchronization optimization system based on user interaction includes the following modules:

[0121] Dual-axis reconstruction module: Based on the visual-spatial dual-axis dynamic model, the module obtains the original sound effect parameters of teammates and performs audio dynamic reconstruction analysis to obtain sound effect correction parameter values; based on the sound effect correction parameter values, the module dynamically renders and adjusts the original sound effects of teammates.

[0122] The process of obtaining the original sound effect parameters of teammates based on the visual-spatial dual-axis dynamic model and performing audio dynamic reconstruction analysis to obtain the sound effect correction parameter values ​​is as follows:

[0123] Obtain the raw sound effect parameters of teammates' voices (baseline volume, baseline high frequency ratio, and baseline sound field diffusion).

[0124] The original sound effect parameters are input into the visual-spatial dual-axis dynamic model, and the sound effect correction parameter values ​​are calculated.

[0125] The process of obtaining the sound effect parameter correction values ​​is as follows:

[0126] The reference volume, reference high frequency ratio, and reference sound field diffusion are multiplied by the audio correction coefficient to obtain the volume correction value, high frequency ratio correction value, and sound field diffusion correction value, respectively.

[0127] The volume correction value, high frequency ratio correction value, and sound field diffusion correction value are integrated into a sound effect correction parameter value;

[0128] The process of dynamically rendering and adjusting the original sound effects of teammates based on sound effect correction parameter values ​​is as follows:

[0129] Based on the sound effect correction parameter values, the original sound effects of teammates are dynamically rendered and adjusted to ensure that the audio output matches the visual-spatial constraints in real time. The specific process is as follows:

[0130] The volume correction value is input into the VR audio rendering engine to replace the base volume; the audio rendering engine adjusts the sound wave amplitude in real time based on the volume correction value.

[0131] For example, when a teammate is obscured, if the base volume is 70dB and the audio correction factor K is 0.6, then the volume correction value is 70 × 0.6 = 42dB; the perceived intensity of the teammate's sound effect decreases from 70dB to 42dB, which is consistent with the degree of visual obstruction and spatial distance attenuation.

[0132] The high-frequency proportion correction value is input into the audio frequency band filtering module; by designing a dynamic adjustment algorithm for the high-frequency cutoff frequency, the reference high-frequency proportion is adjusted.

[0133] The high-frequency attenuation factor is obtained by calculating the ratio of the high-frequency ratio correction value to the benchmark high-frequency ratio; the high-frequency cutoff frequency is obtained by multiplying the high-frequency attenuation factor with the standard high-frequency threshold.

[0134] Preferably, the standard high-frequency threshold is 8000Hz;

[0135] If the high-frequency components of a teammate's sound effect are greater than the high-frequency cutoff frequency, then the high-frequency components of the sound effect that are greater than the high-frequency cutoff frequency will be filtered out.

[0136] The directional attenuation coefficient is obtained by calculating the difference between the unit quantity and the viewing angle offset; the adjusted sound field diffusion degree is obtained by multiplying the directional attenuation coefficient and the sound field diffusion correction value.

[0137] It should be noted that the physical meaning of the directional attenuation coefficient is to reflect the impact of the viewing angle shift on the perception of the sound field. When a teammate is at the edge of the user's field of vision, the sound wave propagation path deviates from the user's auditory focus, resulting in a weakening of the perception of sound field diffusion. By introducing the viewing angle shift, the sound field diffusion is dynamically reduced, so that the audio spatial distribution is consistent with the visual attention distribution.

[0138] Synchronization Perception Module: Acquires the original sound effect parameters and actual sound effect parameters of the current frame after dynamic rendering adjustment, and performs perception deviation analysis to obtain the perception consistency value; based on the perception consistency value, identifies whether the current audio-visual synchronization meets the quality standard;

[0139] The process of obtaining the original sound effect parameters and actual sound effect parameters of the current frame after dynamic rendering adjustment, and then performing perceptual deviation analysis to obtain the perceptual consistency value is as follows:

[0140] Obtain the visual perception constraint value and spatial loss coefficient of the current frame, and calculate the theoretical audio correction coefficient based on the visual perception constraint value and spatial loss coefficient of the current frame.

[0141] Get the original sound effect parameters of teammates in the current frame and the actual sound effect parameters after rendering;

[0142] It should be noted that the original sound effect parameters include the reference volume, reference high frequency ratio, and reference sound field diffusion, while the actual sound effect parameters include the actual output volume, actual high frequency ratio, and actual sound field diffusion.

[0143] The high-frequency proportion of teammates in the current frame is summed with the reference sound field diffusion. The summation result is then multiplied with the reference volume to obtain the theoretical expected perceived value.

[0144] The actual high-frequency proportion of teammates in the current frame is summed with the actual sound field diffusion. The summation result is then multiplied with the actual output volume to obtain the actual effective perceived value.

[0145] The absolute difference between the actual effective perceived value and the theoretical expected perceived value is calculated, and the result of the absolute difference calculation is compared with the theoretical expected perceived value to obtain the perceived consistency value.

[0146] The process of identifying whether the current audio-visual synchronization meets the quality standard based on the perceptual consistency value is as follows:

[0147] In some embodiments, the perceived consistency value is compared with a preset consistency limit;

[0148] It is understandable that the preset consistency limit is obtained by those skilled in the art by collecting multiple frames of theoretically expected perceived values ​​and actual effective perceived values ​​to calculate the deviation, and then comparing it with the user's actual sound effect experience.

[0149] If the perceived consistency value is greater than the preset consistency limit, it indicates that the audio-visual synchronization of the current frame is poor and there is a significant deviation.

[0150] If the perceived consistency value is less than or equal to the preset consistency limit, it indicates that the audio-visual synchronization of the current frame is good and meets the high-quality standard.

[0151] It should be noted that the physical meaning of the perceptual consistency value lies in reflecting the degree of matching between the audio rendering effect and the visual scene. The larger the perceptual consistency value, the more the auditory signal intensity exceeds the reasonable expectation under the visually limited scene, the more serious the audio-visual perception mismatch, and the higher the degree to which the user's immersion is affected. Conversely, the smaller the perceptual consistency value, the more it indicates that the sound attenuation dynamically follows the changes in visual perception, thereby ensuring the perceptual continuity and the authenticity of the immersive experience for users in complex virtual environments.

[0152] The technical solution of this embodiment is as follows: The original sound effect parameters of teammates are obtained based on a visual-spatial dual-axis dynamic model, and audio dynamic reconstruction analysis is performed to obtain sound effect correction parameter values; the original sound effects of teammates are dynamically rendered and adjusted based on the sound effect correction parameter values; through visual-spatial dual-axis dynamic model and audio-visual collaborative analysis, efficient fusion of multi-sensory signals and conflict warning can be achieved, comprehensively improving the immersion, interactivity, and user comfort of virtual reality scenes; the original sound effect parameters and actual sound effect parameters of the current frame after dynamic rendering adjustment are obtained, and perceptual deviation analysis is performed to obtain a perceptual consistency value; based on the perceptual consistency value, it is identified whether the current audio-visual synchronization meets the high-quality standard; through perceptual deviation analysis and the judgment of the high-quality standard of audio-visual synchronization, the synchronization of audio and visual signals can be ensured in real time, significantly improving the immersion and multi-sensory experience consistency of virtual reality scenes, and reducing user discomfort or immersion breakage caused by audio-visual asynchrony.

[0153] Example 3: Please refer to Figure 3 As shown in the embodiment of the present invention, a virtual reality image and sound synchronization optimization method based on user interaction includes the following steps:

[0154] Step 1: Obtain the user's virtual reality field of view data, and divide the virtual reality field of view data into a grid matrix; obtain the basic units in the grid matrix and perform visual constraint analysis to obtain visual perception constraint values;

[0155] Step 2: Obtain the three-dimensional spatial coordinates of the user and teammates and calculate the radial base distance using Euclidean distance; perform spatial attenuation analysis based on the radial base distance to obtain the spatial loss coefficient;

[0156] Step 3: Based on the visual perception constraint value and spatial loss coefficient, the original sound effect parameters are introduced to perform audio-visual co-analysis to obtain the expected audio perception value. Based on the expected audio perception value, it is determined whether a multi-sensory conflict signal is triggered. If it is triggered, a visual-spatial dual-axis dynamic model is constructed based on the visual perception constraint value and spatial loss coefficient.

[0157] Step 4: Obtain the original sound effect parameters of teammates based on the visual-spatial dual-axis dynamic model, and perform audio dynamic reconstruction analysis to obtain sound effect correction parameter values; dynamically render and adjust the original sound effects of teammates based on the sound effect correction parameter values;

[0158] Step 5: Obtain the original sound effect parameters and actual sound effect parameters of the current frame after dynamic rendering adjustment, and perform perceptual deviation analysis to obtain the perceptual consistency value; based on the perceptual consistency value, identify whether the current audio-visual synchronization meets the high-quality standard.

[0159] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A virtual reality image and sound synchronization optimization system based on user interaction, characterized in that: Includes the following modules: Visual constraint module: acquires the user's virtual reality field of view data, divides the virtual reality field of view data into a grid matrix; acquires the basic units in the grid matrix and performs visual constraint analysis to obtain visual perception constraint values; The process of performing the aforementioned visual constraint analysis is as follows: The region occlusion index is obtained by performing cell occlusion analysis on the basic cells in the mesh matrix. Obtain the angular offset between the user's field of view center direction and the teammate's location, and mark it as the field of view offset; The visual perception constraint value is calculated by multiplying the region occlusion index and the viewpoint offset. Space loss module: Acquires the three-dimensional spatial coordinates of the user and teammates and calculates the radial base distance using Euclidean distance; performs spatial attenuation analysis based on the radial base distance to obtain the space loss coefficient; Collaborative Judgment Module: Based on visual perception constraint values ​​and spatial loss coefficients, the module introduces original sound effect parameters to perform audio-visual collaborative analysis to obtain the expected audio perception value. Based on the expected audio perception value, it determines whether a multi-sensory conflict signal is triggered. If triggered, it constructs a visual-spatial dual-axis dynamic model based on visual perception constraint values ​​and spatial loss coefficients. Dual-axis reconstruction module: Based on the visual-spatial dual-axis dynamic model, the original sound effect parameters of teammates are obtained, and the sound effect correction parameter values ​​are obtained by performing dynamic audio reconstruction analysis. The original sound effects of teammates are dynamically rendered and adjusted based on the sound effect correction parameter values; Synchronization Perception Module: Acquires the original sound effect parameters and actual sound effect parameters of the current frame after dynamic rendering adjustment, and performs perception deviation analysis to obtain the perception consistency value; based on the perception consistency value, identifies whether the current audio-visual synchronization meets the quality standard; The process of performing the aforementioned perceptual bias analysis is as follows: Obtain the visual perception constraint value and spatial loss coefficient of the current frame, and obtain the theoretical audio correction coefficient based on the visual perception constraint value and spatial loss coefficient of the current frame; Get the original sound effect parameters of teammates in the current frame and the actual sound effect parameters after rendering; Perceptual consistency values ​​are obtained by performing consistency analysis based on the original sound effect parameters of teammates in the current frame and the actual sound effect parameters after rendering.

2. The virtual reality image and sound synchronization optimization system based on user interaction according to claim 1, characterized in that: The process of performing the unit occlusion analysis is as follows: Identify target sampling units and occlusion sampling units based on basic units in the grid matrix; The number of target sampling units and occlusion sampling units are counted separately. The region occlusion index is calculated by comparing the number of occluded sampling units with the number of target sampling units.

3. The virtual reality image and sound synchronization optimization system based on user interaction according to claim 1, characterized in that: The process of obtaining the radial base distance is as follows: A multi-source positioning fusion system based on VR devices can collect the three-dimensional spatial coordinates of users and teammates in real time. The linear propagation distance between the user and teammates in three-dimensional space is calculated using the three-dimensional Euclidean distance formula and marked as the radial base distance.

4. The virtual reality image and sound synchronization optimization system based on user interaction according to claim 1, characterized in that: The process of performing the spatial attenuation analysis is as follows: Obtain the normal effective acoustic distance under different scenarios, and calculate the basis distance attenuation coefficient under different scenarios by the ratio of radial basis distance to normal effective acoustic distance; Obtain the environmental damping coefficient under different scenarios; calculate the space loss coefficient by multiplying the base distance attenuation coefficient and the environmental damping coefficient.

5. The virtual reality image and sound synchronization optimization system based on user interaction according to claim 1, characterized in that: The process of performing the aforementioned audio-visual co-analysis is as follows: Obtain the original sound effect parameters of teammates; sum the reference high frequency ratio and the reference sound field diffusion, and multiply the summation result with the reference volume to obtain the reference effective perceived value; Obtain the expected perceived value of the audio based on the original sound effect parameters; The audio-visual coordination deviation value is obtained by calculating the difference between the baseline effective perceived value and the expected perceived value of the audio.

6. The virtual reality image and sound synchronization optimization system based on user interaction according to claim 5, characterized in that: The method for obtaining the expected perceived value of audio is as follows: The expected volume value is calculated by multiplying the reference volume with the visual perception constraint value and the spatial loss coefficient. The expected value of the high frequency ratio is obtained by multiplying the baseline high frequency ratio with the visual perception constraint value and the spatial loss coefficient. The expected sound field diffusion value is obtained by multiplying the reference sound field diffusion value with the visual perception constraint value and the spatial loss coefficient. The expected value of high frequency ratio is summed with the expected value of sound field diffusion, and the result of the summation is multiplied with the expected value of volume to obtain the expected audio perceived value.

7. The virtual reality image and sound synchronization optimization system based on user interaction according to claim 1, characterized in that: The method for constructing the aforementioned visual-spatial dual-axis dynamic model is as follows: The difference between the visual perception constraint value and the unit quantity is calculated, and the result of the difference calculation is multiplied by the spatial loss coefficient to obtain the audio correction coefficient.

8. The virtual reality image and sound synchronization optimization system based on user interaction according to claim 1, characterized in that: The process of performing the consistency analysis is as follows: The high-frequency proportion of teammates in the current frame is summed with the reference sound field diffusion. The summation result is then multiplied with the reference volume to obtain the theoretical expected perceived value. The actual high-frequency proportion of teammates in the current frame is summed with the actual sound field diffusion. The summation result is then multiplied with the actual output volume to obtain the actual effective perceived value. The absolute difference between the actual effective perceived value and the theoretical expected perceived value is calculated, and the result of the absolute difference calculation is compared with the theoretical expected perceived value to obtain the perceived consistency value.

Images