Virtual reality interaction method and audio-visual equipment

Abstract

The invention relates to the field of interaction data processing, in particular to a virtual reality interaction method and audio-visual equipment. The method comprises the following steps: acquiring an image and real-time face and head sensing data through virtual reality equipment, and constructing a virtual space and an audio space; the method comprises the following steps: acquiring photographic face data by utilizing face image detection, generating a sight gaze point, and calculating a deflection gaze point and coordinates through head sensing data; and obtaining a virtual visual area based on the deflection fixation point coordinates, positioning a virtual audio area, and performing area audio optimization. According to the method, the virtual audio region is accurately positioned, and corresponding region audio optimization is carried out, so that computing resources are saved.

Description

Technical Field

[0001] The present invention relates to the field of interactive data processing, and in particular to a virtual reality interaction method and audio-visual equipment. Background Art

[0002] Modern virtual reality interaction technology is in a rapid development stage, and technological innovation, content expansion, market demand and social acceptance have jointly promoted the progress of this field. The technical background of virtual reality interaction methods and audio-visual equipment is to create a comprehensive immersive experience, where users can enter a virtual world and interact naturally with the world. This usually involves using computer technology to generate a three-dimensional virtual environment, as well as various input and output devices to achieve multi-sensory interaction. Through audio-visual equipment, an immersive visual experience is provided, and spatial audio technology simulates a realistic sound environment, which can create a highly immersive environment and jointly build a virtual world of multi-sensory fusion. At the same time, the interactivity is enhanced, and the natural and intuitive interaction method enables more freedom to explore the virtual world and build a virtual space. However, if there is a demand for increased immersion, there is a burden on the computing resources of the virtual space, which reduces the system performance. Summary of the invention

[0003] Based on this, it is necessary for the present invention to provide a virtual reality interaction method and audio-visual equipment to solve at least one of the above technical problems.

[0004] To achieve the above object, a virtual reality interaction method comprises the following steps:

[0005] Step S1: inputting image data to a virtual reality device to obtain spatial image data; constructing a virtual spatial model for the spatial image data to obtain a virtual spatial model; constructing a virtual audio space for the virtual spatial model to obtain a virtual audio space;

[0006] Step S2: performing real-time face camera processing on the virtual reality device to obtain real-time photography stream data; performing face image detection on the real-time photography stream data to obtain photographic face image data; performing gaze point generation on the photographic face image data to obtain sight gaze point data; performing coordinate system mapping on the sight gaze point data to obtain gaze point coordinate data;

[0007] Step S3: acquiring head sensor data of the virtual reality device to obtain head sensor data; performing deflection gaze point vector calculation on the line of sight gaze point data based on the head sensor data to obtain deflection gaze point vector data; performing deflection coordinate calculation on the gaze point coordinate data based on the deflection gaze point vector data to obtain deflection gaze point coordinates;

[0008] Step S4: acquiring the virtual visual area of ​​the virtual space model based on the deflected gaze point coordinates to obtain virtual visual area data; locating the gaze audio area of ​​the virtual audio space based on the virtual visual area data to obtain a gaze virtual audio area; performing regional audio optimization on the virtual visual area data and the gaze virtual audio area to obtain regional audio optimization data.

[0009] The present invention constructs a virtual space model through the initial setting and space construction of the virtual reality device and the input of image data, and further creates a corresponding virtual audio space. In this way, a realistic virtual environment can be created for the user, and a foundation can be laid for subsequent audio optimization; the system focuses on capturing and analyzing the user's facial image and gaze point. Through real-time face photography and face image detection technology, the system can accurately obtain the user's facial data and photographic face image. Then, the system generates sight gaze point data, and determines the user's accurate gaze position through coordinate system mapping; the head sensor data is used to enhance the system's understanding of the user's gaze behavior. By combining the head sensor data and the sight gaze point data, the system can calculate the deflection gaze point vector, thereby accurately determining the user's deflection gaze point coordinates in the virtual space; the virtual space model and the user's gaze behavior are combined. By obtaining the user's deflection gaze point coordinates, the system can extract the corresponding virtual visual area from the virtual space model, and further locate the virtual audio area that the user is paying attention to. The system then optimizes the audio in these areas to provide a customized, immersive audio experience. At the same time, by optimizing the audio only in the visual area to reduce the amount of virtual space calculations, it helps to reduce the burden on computing resources and improve system performance.

[0010] Preferably, step S1 comprises the following steps:

[0011] Step S11: inputting image data into the virtual reality device to obtain spatial image data;

[0012] Step S12: marking the spatial image data with spatial point cloud data to obtain spatial image point cloud data;

[0013] Step S13: performing point cloud data registration on the spatial image point cloud data to obtain spatial point cloud registration data;

[0014] Step S14: extracting spatial features from the spatial image point cloud data to obtain spatial point cloud feature data;

[0015] Step S15: performing spatial entity segmentation on the spatial point cloud feature data to obtain spatial entity data;

[0016] Step S16: constructing a virtual space model based on the spatial entity data and the spatial point cloud registration data to obtain a virtual space model;

[0017] Step S17: constructing a virtual audio space for the virtual space model to obtain a virtual audio space.

[0018] The present invention obtains image information of the real world through the device, provides basic data for the subsequent spatial audio construction, so that the virtual audio space can correspond to the real environment and enhance the sense of immersion; converts the two-dimensional image into three-dimensional point cloud data with spatial position information, so that the sound can be placed in a more precise spatial position, such as walls, furniture, etc., to enhance the sense of positioning and reality of the sound; through registration, the point cloud data obtained from different perspectives and at different times are spliced ​​and integrated to build a more complete and accurate environmental model, providing a more accurate simulation basis for the propagation of sound in space; extracts geometric features and semantic features in the space, such as the shape, size, material, etc. of the room, and this information can be used to simulate the sound of the room. It is used to simulate the acoustic characteristics of sound such as reflection, absorption and diffraction in different spaces to make virtual audio more realistic; the space is divided into different physical areas, such as walls, floors, ceilings, etc., and different acoustic properties are given to them, so that the propagation and reflection of sound in different areas are more in line with reality and the spatial layering of sound is enhanced; a complete three-dimensional virtual space model is constructed, which includes spatial geometry information, semantic information and acoustic properties, providing a complete data foundation for the construction of virtual audio space; based on the constructed virtual space model, the acoustic simulation algorithm is used to calculate the propagation path and sound field distribution of sound in space, and finally a realistic spatial audio experience that matches the virtual environment is generated.

[0019] Preferably, step S17 comprises the following steps:

[0020] Step S171: setting the sound source of the virtual space model to obtain sound source setting data;

[0021] Step S172: performing sound source data collection on the sound source setting data to obtain a sound source data set;

[0022] Step S173: performing sound propagation path analysis on the sound source data set based on the virtual space model to obtain sound propagation path data;

[0023] Step S174: setting the propagation path medium of the virtual space model based on the sound propagation path data to obtain the propagation path medium data;

[0024] Step S175: performing sound reflection analysis on the propagation path medium data based on the sound source data set and the sound propagation path data to obtain sound reflection data;

[0025] Step S176: performing sound obstacle analysis on the propagation path medium data based on the sound source data set and the sound propagation path data to obtain sound obstacle data;

[0026] Step S177: Perform audio data spatial fitting on the virtual space model based on the sound reflection data and the sound obstacle data to obtain a virtual audio space.

[0027] The present invention places virtual sound sources, such as human voices and music, at specific locations in a virtual space model and sets their acoustic properties, such as volume and timbre, to provide basic data for subsequent acoustic simulations; integrates data from multiple sound sources to form a complete sound source data set, which facilitates subsequent unified sound propagation path analysis and audio rendering for multiple sound sources; calculates the propagation path of sound from the sound source to the listener, including direct sound, reflected sound, diffracted sound, etc., based on acoustic principles and geometric information of the virtual space model, to provide data support for simulating real sound propagation effects; identifies the medium properties of different areas on the path, such as air, walls, furniture, etc., according to the sound propagation path, and assigns them different acoustic parameters. The method uses the following methods to analyze the reflection of different media surfaces when the sound encounters them on the propagation path, such as absorption coefficient and reflection coefficient, so as to more accurately simulate the changes in sound propagation in different media; analyze the reflection of the sound when it encounters different medium surfaces on the propagation path, including reflection direction and reflection intensity, so that the virtual sound is more in line with the reflection law of sound in the real world and enhances the sense of space; analyze the occlusion and diffraction of the sound when it encounters obstacles on the propagation path, such as sound bypassing walls and furniture, so that the spatial shielding effect of the virtual sound is more realistic and credible; apply the sound reflection, obstacle and other data obtained in the previous steps to the virtual space model, and perform the final audio rendering to generate a complete virtual audio space containing sound sources, spatial information and acoustic effects, so that users can have an immersive auditory experience.

[0028] Preferably, step S2 comprises the following steps:

[0029] Step S21: performing real-time face camera processing on the virtual reality device to obtain real-time photography stream data;

[0030] Step S22: performing face image detection on the real-time photographic stream data to obtain photographic face image data;

[0031] Step S23: extracting the eye region of the photographed face image data to obtain photographed eye region data;

[0032] Step S24: performing pupil center positioning on the photographed eye area data to obtain pupil center positioning data;

[0033] Step S25: extracting the iris contour from the photographed eye area data to obtain iris contour image data;

[0034] Step S26: generating a gaze point for the pupil center positioning data and the iris contour image data to obtain line of sight gaze point data;

[0035] Step S27: mapping the gaze point data pair to a coordinate system to obtain gaze point coordinate data.

[0036] The present invention can obtain the facial image data stream when the user wears the device by performing real-time face camera processing on the virtual reality device, providing a basic data source for subsequent face detection and eye tracking, so that the system can capture and analyze the user's facial information in real time; by performing face image detection on the real-time photography stream data, the position and range of the face in the image can be accurately identified, and then the face area can be separated from the background, providing accurate input data for the subsequent eye area extraction; eye area extraction is a key step for the entire system to analyze the user's line of sight focus. By accurately extracting the eye area in the photographed face image, other interference information in the image can be effectively removed, and the accuracy of subsequent pupil positioning and iris recognition can be improved; pupil center positioning is an important step in determining the user's line of sight direction. This step identifies the position of the pupil in the eye by analyzing the eye image data, providing basic reference data for line of sight tracking and focus judgment; iris contour extraction can provide important features for accurate iris recognition and subsequent line of sight tracking. By extracting the contour shape, texture and other unique features of the iris, the user can be uniquely identified and the accuracy of the entire system can be improved. By combining the pupil center and iris contour information, the user's gaze point in three-dimensional space, that is, the direction and position the user is observing, can be inferred through geometric calculations or deep learning models. The gaze point is mapped from the image coordinate system to the coordinate system of the virtual scene, so that the virtual scene can interact and provide feedback in real time based on the user's line of sight, such as gaze point selection, line of sight interaction, etc., thereby enhancing the immersion and interactivity of the experience.

[0037] Preferably, step S26 comprises the following steps:

[0038] Step S261: extracting iris feature points from the iris contour image data to obtain iris feature point data;

[0039] Step S262: constructing an eyeball model based on the pupil center positioning data and the iris feature point data to obtain an eyeball model;

[0040] Step S263: acquiring the direction of the eyeball optical axis data of the eyeball model based on the pupil center positioning data to obtain the direction data of the eyeball optical axis;

[0041] Step S264: performing a sight line intersection point simulation on the eyeball optical axis direction data to obtain sight line intersection point data;

[0042] Step S265: generating a sight line fixation point for the sight line intersection point data to obtain the sight line fixation point data.

[0043] The present invention extracts representative feature points from the iris contour image through this step, such as iris texture, color change, etc. These feature points can be used to fit the eyeball model more accurately and improve the accuracy of line of sight estimation; eyeball model construction is an important step in simulating the structure and movement of the human eye. By combining the pupil center positioning data and the iris feature point data, an accurate three-dimensional eyeball model can be established. This model can simulate the shape, size and structure of the human eye, and provide a basic framework for analyzing the eyeball optical axis and line of sight movement; the acquisition of the direction of the eyeball optical axis data is the key to understanding the line of sight direction. The direction vector of the eyeball optical axis can be calculated through the pupil center positioning data. This vector represents the direction from the pupil center to the eyeball model, which provides an important reference for simulating the line of sight intersection and ultimately determining the line of sight fixation point; the lines of sight of the left and right eyes will intersect at one point in space, that is, the line of sight intersection point, which represents the specific position that the user is looking at in three-dimensional space. By simulating the intersection process of the two eyes' gazes, the user's gaze point can be inferred more accurately. The calculated gaze intersection point is used as the user's gaze point. This data can be used for various interactions and applications in the system, such as gaze selection, gaze point rendering, eye movement analysis, etc., to enhance the immersion and interactivity of the experience.

[0044] Preferably, step S3 comprises the following steps:

[0045] Step S31: acquiring head sensor data of the virtual reality device to obtain head sensor data;

[0046] Step S32: performing head posture data analysis on the head sensor data to obtain head posture data;

[0047] Step S33: performing head deflection feature analysis on the head sensor data based on the head posture data to obtain head deflection feature data;

[0048] Step S34: calculating the deflection gaze point vector of the gaze point data based on the head deflection feature data to obtain the deflection gaze point vector data;

[0049] Step S35: Calculate the deflected coordinates of the gaze point coordinate data based on the deflected gaze point vector data to obtain the deflected gaze point coordinates.

[0050] The head sensor data of the virtual reality device is the basis for capturing the user's head movement and posture changes. Through the sensors built into the virtual reality device, such as gyroscopes, accelerometers and magnetometers, the position, rotation and movement information of the user's head in three-dimensional space can be obtained in real time, providing necessary data support for the subsequent analysis of head posture and deflection characteristics; head posture data analysis can help the system understand the current state of the user's head. By processing and analyzing the head sensor data, the posture information such as rotation, pitch and deflection angle of the head in the virtual space can be obtained, providing a basic reference for the subsequent deflection feature analysis and line of sight tracking; head deflection feature analysis aims to identify and quantify the lateral deflection movement of the user's head. By further processing the head posture data, the deflection angle and speed of the head in the horizontal direction can be extracted, providing key input data for calculating the deflection gaze point vector and ultimately determining the deflection gaze point coordinates; according to the user's head deflection characteristics, a deflection vector is calculated to correct the original gaze point direction to make it more consistent with the user's actual gaze direction; by combining the head sensor data of the virtual reality device and the deflection gaze point vector, the new gaze point coordinates generated by the user's head deflection in the virtual space can be calculated. This coordinate data can correspond to objects in the virtual environment to achieve more accurate user interaction and a more natural human-computer interaction experience.

[0051] Preferably, step S33 includes the following steps:

[0052] Step S331: setting a reference posture for the head posture data to obtain the head reference posture data;

[0053] Step S332: Calculating the head deflection angle of the head sensor data based on the head reference posture data to obtain the head deflection angle;

[0054] Step S333: analyzing the deflection vector direction of the head deflection angle to obtain deflection vector direction data;

[0055] Step S334: calculating the deflection vector amplitude of the head deflection angle to obtain deflection vector amplitude data;

[0056] Step S335: performing feature data fitting on the deflection vector direction data and the deflection vector amplitude data to obtain head deflection feature data.

[0057] The present invention can provide a reference basis for subsequent deflection angle calculation and analysis by setting a head reference posture. By establishing a standard or initial head posture as a reference, the degree of deflection of the user's head can be more accurately calculated and quantified. This reference posture is usually defined as the posture of the user looking straight ahead, providing a neutral reference point for subsequent analysis; the head deflection angle calculation is the basis for understanding the lateral movement of the user's head. By comparing the head sensor data and the head reference posture data, the deflection angle of the user's head in the horizontal direction can be calculated. This angle represents the degree of deviation of the head from the reference posture, providing a quantitative indicator for subsequent deflection vector analysis and simulation; by analyzing the head deflection angle, the direction vector of the deflection can be determined. This direction vector represents whether the head deflection is in the left, right, up and down or other specific directions, providing important input information for simulating and simulating the user's line of sight movement; by analyzing the head deflection angle, the amplitude of the deflection vector, that is, the distance or angle of the deflection, can be calculated. This amplitude data represents the intensity of the head deflection, providing parameters for simulating and analyzing the user's field of view and head movement amplitude; feature data fitting is the process of integrating the deflection vector direction and amplitude data. By mathematically fitting the direction data and amplitude data, a more complete head deflection feature model can be established. This model combines the deflection direction and amplitude information, and can more accurately represent the user's head deflection behavior, providing reliable input data for subsequent eye tracking, virtual space interaction or other related applications.

[0058] Preferably, step S4 comprises the following steps:

[0059] Step S41: performing coordinate grid mapping on the virtual space model to obtain a virtual space coordinate grid;

[0060] Step S42: performing spatial positioning of the deflected gaze point on the virtual space coordinate grid based on the deflected gaze point coordinates to obtain spatial deflected gaze point data;

[0061] Step S43: acquiring a virtual visual area for the spatial deflection gaze point data to obtain virtual visual area data;

[0062] Step S44: positioning the gaze audio area in the virtual audio space based on the virtual visual area data to obtain the gaze virtual audio area;

[0063] Step S45: Perform regional audio optimization on the virtual visual area data and the gaze virtual audio area to obtain regional audio optimization data.

[0064] The present invention can establish a coordinate framework of a virtual space by mapping a virtual space model to a coordinate grid. By creating a three-dimensional coordinate grid system, a unique coordinate value can be assigned to each point in the virtual space. This provides a basic framework for deflection gaze point positioning, virtual visual acquisition and audio optimization; the deflection gaze point space positioning is the process of mapping the deflected gaze point coordinates to the virtual space coordinate grid. Through this step, the exact position of the user's deflection sight focus in the virtual space can be determined, providing a basic reference for subsequent virtual visual and audio area acquisition; by analyzing the deflection gaze point data, the area in the virtual space that enters the user's field of view, that is, the virtual visual area, can be determined. This area will become the part of the virtual environment currently visible to the user, providing a basic input for the positioning of the gaze audio area and regional audio optimization; by analyzing the virtual visual area data, the virtual audio area currently gazed by the user can be determined. This step ensures that the audio processing focuses on the virtual space area that the user is interested in or concerned about, and improves the relevance and accuracy of the audio experience; the regional audio optimization aims to improve the quality of the user's audio experience in the virtual space. Through the comprehensive analysis of the virtual visual area data and the gaze audio area data, the audio effect can be optimized. For example, enhancing the audio clarity in the virtual visual area, increasing the volume of the virtual audio area, adding ambient sound field effects, etc., to make the audio experience more realistic and immersive.

[0065] Preferably, step S45 includes the following steps:

[0066] Step S451: performing regional acoustic characteristic analysis on the virtual audio area being watched to obtain regional acoustic characteristic data;

[0067] Step S452: performing spatial regional sound field simulation on the virtual audio space based on the regional acoustic characteristic data to obtain spatial regional sound field data;

[0068] Step S453: acquiring a gaze visual path for the virtual visual area data to obtain a gaze visual path;

[0069] Step S454: performing audio occlusion recognition on the gaze visual path to obtain path audio occlusion data;

[0070] Step S455: performing regional audio simulation optimization on the spatial regional sound field data based on the path audio occluder data to obtain regional audio optimization data.

[0071] The present invention aims to understand and quantify the sound characteristics of the virtual audio area through regional acoustic characteristics analysis. By analyzing the acoustic reflection, absorption, diffusion and other acoustic characteristics of this area, regional acoustic characteristics data can be obtained. These data include material properties, boundary effects, acoustic environment, etc., which provide basic parameters for subsequent spatial regional sound field simulation; by applying acoustic principles and regional acoustic characteristics data, the propagation, reflection and attenuation effects of sound in a specific virtual audio area can be simulated and predicted. This step helps to establish a realistic spatial audio model and provide a basic framework for the final audio optimization; by analyzing the virtual visual area data, the visual path from the user's deflected gaze point to other points in the virtual visual area can be determined. This path represents the movement trajectory of the user's line of sight in the virtual space, providing an important reference for the subsequent audio occlusion recognition and optimization; audio occlusion recognition aims to identify obstacles in the virtual space that affect sound propagation and auditory experience. By analyzing the gaze visual path, audio occluders on the visual path can be identified, such as virtual walls, furniture or other virtual objects. This step provides the necessary data for regional audio simulation optimization to ensure that the propagation and reflection of sound are consistent with the virtual environment settings; through the comprehensive analysis of these two factors, the regional audio effect can be optimized. For example, the attenuation, reflection or diffraction of sound when encountering occluders is simulated, and the appropriate acoustic model is applied to achieve a more realistic audio experience. This optimization step ensures that the audio presentation in the virtual space is consistent with the actual physical environment, enhancing the immersion and sense of space in virtual reality or mixed reality applications.

[0072] The present invention also provides an audio-visual device, including a head-mounted display, for executing the virtual reality interaction method as described above, wherein the head-mounted display of the audio-visual device includes:

[0073] A virtual audio space construction module is used to input image data to a virtual reality device to obtain spatial image data; construct a virtual space model on the spatial image data to obtain a virtual space model; and construct a virtual audio space on the virtual space model to obtain a virtual audio space;

[0074] The gaze point coordinate generation module is used to perform real-time face camera processing on the virtual reality device to obtain real-time photography stream data; perform face image detection on the real-time photography stream data to obtain photographic face image data; perform gaze point generation on the photographic face image data to obtain sight gaze point data; perform coordinate system mapping on the sight gaze point data to obtain gaze point coordinate data;

[0075] The deflection gaze point generation module is used to acquire head sensor data of the virtual reality device to obtain head sensor data; perform deflection gaze point vector calculation on the line of sight gaze point data based on the head sensor data to obtain deflection gaze point vector data; perform deflection coordinate calculation on the gaze point coordinate data based on the deflection gaze point vector data to obtain deflection gaze point coordinates;

[0076] The regional audio optimization module is used to acquire the virtual visual area of ​​the virtual space model based on the deflected gaze point coordinates to obtain virtual visual area data; locate the gaze audio area of ​​the virtual audio space based on the virtual visual area data to obtain the gaze virtual audio area; and perform regional audio optimization on the virtual visual area data and the gaze virtual audio area to obtain regional audio optimization data.

[0077] In summary, the present invention provides a virtual reality interaction method and an audio-visual device, which are composed of a virtual audio space construction module, a gaze point coordinate generation module, a deflected gaze point generation module, and a regional audio optimization module, and can implement any virtual reality interaction method described in the present invention, and are used to combine the operations between computer programs running on various modules to implement any virtual reality interaction method. The internal structures of the system cooperate with each other, which can greatly reduce duplication of work and manpower investment, and can quickly and effectively provide a more accurate and efficient virtual reality interaction process, thereby simplifying the virtual reality interaction method and the audio-visual device. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments thereof made with reference to the following drawings:

[0079] Figure 1 A schematic diagram of the steps of a virtual reality interaction method of the present invention;

[0080] Figure 2 for Figure 1 Detailed step flow diagram of step S4;

[0081] Figure 3 for Figure 2 Detailed step flow diagram of step S45; DETAILED DESCRIPTION

[0082] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0083] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0084] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0085] To achieve this, please refer to Figures 1 to 3 The present invention provides a virtual reality interaction method, comprising the following steps:

[0086] Step S1: inputting image data to a virtual reality device to obtain spatial image data; constructing a virtual spatial model for the spatial image data to obtain a virtual spatial model; constructing a virtual audio space for the virtual spatial model to obtain a virtual audio space;

[0087] Step S2: performing real-time face camera processing on the virtual reality device to obtain real-time photography stream data; performing face image detection on the real-time photography stream data to obtain photographic face image data; performing gaze point generation on the photographic face image data to obtain sight gaze point data; performing coordinate system mapping on the sight gaze point data to obtain gaze point coordinate data;

[0088] Step S3: acquiring head sensor data of the virtual reality device to obtain head sensor data; performing deflection gaze point vector calculation on the line of sight gaze point data based on the head sensor data to obtain deflection gaze point vector data; performing deflection coordinate calculation on the gaze point coordinate data based on the deflection gaze point vector data to obtain deflection gaze point coordinates;

[0089] Step S4: acquiring the virtual visual area of ​​the virtual space model based on the deflected gaze point coordinates to obtain virtual visual area data; locating the gaze audio area of ​​the virtual audio space based on the virtual visual area data to obtain a gaze virtual audio area; performing regional audio optimization on the virtual visual area data and the gaze virtual audio area to obtain regional audio optimization data.

[0090] In the embodiment of the present invention, please refer to Figure 1 FIG. 1 is a flow chart of the steps of the virtual reality interaction method of the present invention. In this example, the virtual reality interaction method includes the following steps:

[0091] Step S1: inputting image data to a virtual reality device to obtain spatial image data; constructing a virtual spatial model for the spatial image data to obtain a virtual spatial model; constructing a virtual audio space for the virtual spatial model to obtain a virtual audio space;

[0092] In an embodiment of the present invention, a virtual reality device is used to scan the target space, and a color image and depth information are collected. Scanning parameters are set, the space is scanned, and real-time image stitching is performed to generate spatial image data. By using point cloud processing software, the two-dimensional image is converted into three-dimensional point cloud data, and noise reduction, feature extraction, depth image conversion and other operations are performed. Through feature matching and posture estimation, point cloud data from different perspectives are fused into a complete spatial point cloud model. Based on the point cloud registration data, spatial feature information is extracted, including normal vectors, curvature, geometric feature descriptions, and color features. According to the point cloud feature data, the point cloud is segmented into different spatial entities and semantic labels are added. Based on the segmented spatial entity data, a virtual space model is constructed using three-dimensional modeling software. Based on the virtual space model, virtual acoustic software is used to simulate the interaction of sound in space to construct a realistic virtual audio space.

[0093] Step S2: performing real-time face camera processing on the virtual reality device to obtain real-time photography stream data; performing face image detection on the real-time photography stream data to obtain photographic face image data; performing gaze point generation on the photographic face image data to obtain sight gaze point data; performing coordinate system mapping on the sight gaze point data to obtain gaze point coordinate data;

[0094] In the embodiment of the present invention, the user's face image is captured in real time by using the built-in camera of the virtual reality device, and decoded into digital image frames to form real-time photography stream data. For each frame of data, face detection is first performed to locate the face area, and then the eye area is extracted and the pupil center is located and the iris contour is extracted. The user's gaze point is calculated by combining the pupil center and iris information. Finally, the gaze point is mapped from the image coordinate system to the screen coordinate system or three-dimensional space coordinate system of the virtual reality device to obtain the final gaze point coordinate data.

[0095] Step S3: acquiring head sensor data of the virtual reality device to obtain head sensor data; performing deflection gaze point vector calculation on the line of sight gaze point data based on the head sensor data to obtain deflection gaze point vector data; performing deflection coordinate calculation on the gaze point coordinate data based on the deflection gaze point vector data to obtain deflection gaze point coordinates;

[0096] In an embodiment of the present invention, the user's head movement is tracked by utilizing high-precision sensors built into the virtual reality device, such as a gyroscope and an accelerometer, to obtain information such as the head rotation angle and acceleration. The data is denoised and fused by algorithms such as Kalman filtering to obtain accurate head posture data. Then, the head posture change rate is calculated as the head deflection feature, and a mapping relationship is established to convert it into a sight deflection vector. Finally, the original sight gaze point coordinates and the deflected gaze point vector are added to obtain the screen position coordinates that the user is about to gaze at.

[0097] Step S4: acquiring the virtual visual area of ​​the virtual space model based on the deflected gaze point coordinates to obtain virtual visual area data; locating the gaze audio area of ​​the virtual audio space based on the virtual visual area data to obtain a gaze virtual audio area; performing regional audio optimization on the virtual visual area data and the gaze virtual audio area to obtain regional audio optimization data.

[0098] In an embodiment of the present invention, a virtual space model is loaded into the system and divided into a series of coordinate grids with unique identifiers. Then, the coordinates of the user's deflected gaze point are mapped to the virtual space coordinate grid, the corresponding grid is found, and its coordinate identifier and other information are recorded as spatial deflected gaze point data. Based on the data, the user's virtual visual area is determined, such as a spherical area with the deflected gaze point as the center and the user's visual field as the radius. Then, the corresponding virtual audio area of ​​interest is located in the virtual audio space, such as the projection area of ​​the virtual visual area in the virtual audio space. Finally, according to the virtual visual area data and the virtual audio area of ​​interest, the virtual audio is optimized, such as improving the sound clarity and volume of the interest area, and the optimized audio parameters and other information are recorded as regional audio optimization data.

[0099] The present invention constructs a virtual space model through the initial setting and space construction of the virtual reality device and the input of image data, and further creates a corresponding virtual audio space. In this way, a realistic virtual environment can be created for the user, and a foundation can be laid for subsequent audio optimization; the system focuses on capturing and analyzing the user's facial image and gaze point. Through real-time face photography and face image detection technology, the system can accurately obtain the user's facial data and photographic face image. Then, the system generates sight gaze point data, and determines the user's accurate gaze position through coordinate system mapping; the head sensor data is used to enhance the system's understanding of the user's gaze behavior. By combining the head sensor data and the sight gaze point data, the system can calculate the deflection gaze point vector, thereby accurately determining the user's deflection gaze point coordinates in the virtual space; the virtual space model and the user's gaze behavior are combined. By obtaining the user's deflection gaze point coordinates, the system can extract the corresponding virtual visual area from the virtual space model, and further locate the virtual audio area that the user is paying attention to. The system then optimizes the audio in these areas to provide a customized, immersive audio experience. At the same time, by optimizing the audio only in the visual area to reduce the amount of virtual space calculations, it helps to reduce the burden on computing resources and improve system performance.

[0100] Preferably, step S1 comprises the following steps:

[0101] Step S11: inputting image data into the virtual reality device to obtain spatial image data;

[0102] In the embodiment of the present invention, the target space is scanned by using a virtual reality device with a depth sensor, and the device will collect color images and depth information at the same time. The specific operation is to set the scanning parameters, set the scanning resolution, scanning range, frame rate and other parameters according to the size and detail requirements of the target space; perform spatial scanning, hold the virtual reality device, and perform a comprehensive scan of the target space along the predetermined route, keep the device moving steadily to avoid image blur or distortion; real-time image stitching; the built-in algorithm of the device will stitch the collected images and depth information in real time to generate preliminary spatial image data.

[0103] Step S12: marking the spatial image data with spatial point cloud data to obtain spatial image point cloud data;

[0104] In an embodiment of the present invention, the acquired spatial image data is used in combination with point cloud processing software to convert a two-dimensional image into three-dimensional point cloud data and mark it. Specific operations include depth image conversion, using the depth value of each pixel in the depth image, combined with camera parameters, to back-project each pixel into three-dimensional space to form point cloud data; point cloud data downsampling, the original point cloud data is usually very large, and downsampling is required to reduce the amount of calculation. Methods such as uniform downsampling and random downsampling can be used; point cloud data denoising, due to factors such as sensor errors, there will be noise points in the point cloud data, which need to be removed using statistical filtering, radius filtering and other methods.

[0105] Step S13: performing point cloud data registration on the spatial image point cloud data to obtain spatial point cloud registration data;

[0106] In an embodiment of the present invention, point cloud data from different perspectives are spliced ​​into a complete spatial point cloud model. Specific operations include feature point extraction, extracting feature points, such as corner points, edge points, etc., from point cloud data from different perspectives. Common feature point extraction algorithms include ISS, SIFT, NARF, etc.; feature matching, using the geometric information of feature points, such as normal vectors, curvature, etc., to perform feature matching and find the correspondence between point cloud data from different perspectives; posture estimation, based on the matched feature point pairs, calculate the rotation and translation transformation matrices between point cloud data from different perspectives; point cloud fusion, using the calculated posture transformation matrix, convert point cloud data from different perspectives to the same coordinate system, fuse them, and obtain a complete spatial point cloud model, and finally obtain complete spatial point cloud data after registration and fusion, that is, spatial point cloud registration data.

[0107] Step S14: extracting spatial features from the spatial image point cloud data to obtain spatial point cloud feature data;

[0108] In an embodiment of the present invention, spatial feature information is extracted based on the spatial point cloud registration data to provide a data basis for subsequent entity segmentation and model construction. Specific operations include normal vector analysis, calculating the normal vector of each point, and describing the orientation information of the point cloud surface. Common normal vector estimation methods include PCA, least squares method, etc.; curvature calculation, calculating the curvature of each point, describing the degree of curvature of the point cloud surface. Curvature can be used to distinguish objects of different shapes such as planes, spheres, and cylinders; geometric feature description, using the normal vector, curvature and other information of the point cloud to construct the geometric feature descriptor of the point cloud, such as PFH, SHOT, FPFH, etc.; color feature extraction, if the point cloud data contains color information, color features can be extracted, such as color histograms, color moments, etc., and finally point cloud data containing normal vectors, curvature, geometric feature descriptors, color features and other information is obtained, that is, spatial point cloud feature data.

[0109] Step S15: performing spatial entity segmentation on the spatial point cloud feature data to obtain spatial entity data;

[0110] In an embodiment of the present invention, the point cloud is segmented into different spatial entities, such as walls, floors, furniture, etc., based on point cloud feature data using a clustering algorithm, a region growing algorithm, etc., and a semantic label is added to each entity. For example, a region growing-based segmentation algorithm is used to segment the point cloud into different plane regions based on features such as normal vectors and curvature of the point cloud. Then, based on the semantic labels and spatial positional relationships of the point cloud, the segmented plane regions are merged into different spatial entities, such as walls, floors, etc.

[0111] Step S16: constructing a virtual space model based on the spatial entity data and the spatial point cloud registration data to obtain a virtual space model;

[0112] In the embodiment of the present invention, a virtual space model is constructed by using 3D modeling software such as 3dsMax, Maya, Blender, etc. according to the segmented space entity data. Different modeling methods can be selected as needed, such as using simple geometric bodies to represent the space structure, or using more sophisticated models to restore the real scene. For example, using Blender software, a corresponding 3D model is created according to the geometric shape and size of the space entity.

[0113] Step S17: constructing a virtual audio space for the virtual space model to obtain a virtual audio space.

[0114] In an embodiment of the present invention, virtual acoustic software can be used to simulate the propagation and interaction of sound in space and construct a virtual audio space based on the constructed virtual space model. The construction of virtual audio space needs to take into account physical phenomena such as reflection, diffraction, and absorption of sound, as well as the auditory characteristics of the human ear. During specific operations, it is necessary to set parameters such as the position, direction, and material properties of the sound source, as well as information such as the size, shape, and sound absorption coefficient of the room. The virtual acoustic software will calculate the propagation path and sound field distribution of the sound in space based on these parameters to simulate a realistic auditory experience. For example, the sound absorption coefficients of objects such as walls, floors, and furniture can be set according to the geometric shape and material of the room to simulate the reflection and absorption of sound on surfaces of different materials to construct an audio space.

[0115] The present invention obtains image information of the real world through the device, provides basic data for the subsequent spatial audio construction, so that the virtual audio space can correspond to the real environment and enhance the sense of immersion; converts the two-dimensional image into three-dimensional point cloud data with spatial position information, so that the sound can be placed in a more precise spatial position, such as walls, furniture, etc., to enhance the sense of positioning and reality of the sound; through registration, the point cloud data obtained from different perspectives and at different times are spliced ​​and integrated to build a more complete and accurate environmental model, providing a more accurate simulation basis for the propagation of sound in space; extracts geometric features and semantic features in the space, such as the shape, size, material, etc. of the room, and this information can be used to simulate the sound of the room. It is used to simulate the acoustic characteristics of sound such as reflection, absorption and diffraction in different spaces to make virtual audio more realistic; the space is divided into different physical areas, such as walls, floors, ceilings, etc., and different acoustic properties are given to them, so that the propagation and reflection of sound in different areas are more in line with reality and the spatial layering of sound is enhanced; a complete three-dimensional virtual space model is constructed, which includes spatial geometry information, semantic information and acoustic properties, providing a complete data foundation for the construction of virtual audio space; based on the constructed virtual space model, the acoustic simulation algorithm is used to calculate the propagation path and sound field distribution of sound in space, and finally a realistic spatial audio experience that matches the virtual environment is generated.

[0116] Preferably, step S17 comprises the following steps:

[0117] Step S171: setting the sound source of the virtual space model to obtain sound source setting data;

[0118] In an embodiment of the present invention, according to the design requirements of the virtual audio space, the position, direction, type, volume and other parameters of the sound source are set in the virtual space model, and a suitable audio file is selected to correspond to it. For example, in a virtual conference room scene, a sound source can be set at the position of each seat to simulate the voice of the participants; in a virtual concert hall scene, a sound source can be set in the center of the stage to simulate the performance of musical instruments. The setting data of each sound source includes its three-dimensional coordinates in the virtual space, the type of sound source, such as point sound source, line sound source, surface sound source, directivity, initial volume, audio file path and other information.

[0119] Step S172: performing sound source data collection on the sound source setting data to obtain a sound source data set;

[0120] In an embodiment of the present invention, the setting data of all sound sources are integrated to form a sound source data set for subsequent sound propagation path analysis and audio rendering. The sound source data set can be stored in data structures such as lists and arrays, where each element represents a sound source and contains all the setting data for the sound source.

[0121] Step S173: performing sound propagation path analysis on the sound source data set based on the virtual space model to obtain sound propagation path data;

[0122] In the embodiment of the present invention, the propagation path of the sound from the sound source to the listener is calculated by using a ray tracing algorithm, a path tracing algorithm, and other methods based on the sound source data set and the geometric information of the virtual space model. The sound propagation path analysis needs to consider the reflection, diffraction, transmission, and other phenomena of the sound, as well as the influence of different materials on the sound propagation. For example, using a ray tracing algorithm, multiple sound rays are emitted from each sound source to simulate the propagation of the sound in the virtual space, and the propagation path, number of reflections, transmission coefficient, and other information of each sound ray are recorded.

[0123] Step S174: setting the propagation path medium of the virtual space model based on the sound propagation path data to obtain the propagation path medium data;

[0124] In the embodiment of the present invention, different media passed through on the sound propagation path are identified according to the sound propagation path data, and corresponding acoustic parameters are set for them, such as sound absorption coefficient, reflection coefficient, propagation speed, etc. For example, according to the result of sound ray tracing, the material of the wall, ground, furniture and other objects passed by each sound ray can be determined, and its acoustic parameters can be set according to the material type.

[0125] Step S175: performing sound reflection analysis on the propagation path medium data based on the sound source data set and the sound propagation path data to obtain sound reflection data;

[0126] In an embodiment of the present invention, the reflection phenomenon of sound on the propagation path is calculated based on the sound source data set, the sound propagation path data and the propagation path medium data. It is necessary to consider factors such as the reflection coefficient, reflection angle, and reflection energy attenuation of different materials. For example, based on the intersection of each sound line with different medium surfaces, the propagation direction and energy attenuation of the reflected sound line are calculated, and the reflected sound line is added to the sound propagation path data.

[0127] Step S176: performing sound obstacle analysis on the propagation path medium data based on the sound source data set and the sound propagation path data to obtain sound obstacle data;

[0128] In the embodiment of the present invention, the obstacles encountered by the sound on the propagation path are analyzed according to the sound source data set, the sound propagation path data and the propagation path medium data, and the diffraction phenomenon of the sound is calculated. It is necessary to consider the influence of factors such as the size, shape, and material of the obstacle on the sound diffraction. For example, for obstacles existing on the sound line propagation path, the diffraction angle and energy attenuation of the sound can be calculated according to its size and shape, and the diffracted sound line can be added to the sound propagation path data.

[0129] Step S177: Perform audio data spatial fitting on the virtual space model based on the sound reflection data and the sound obstacle data to obtain a virtual audio space.

[0130] In the embodiment of the present invention, the virtual space model is fitted with audio data space according to the sound reflection data and the sound obstacle data, and the calculated sound field information is integrated with the virtual space model to construct a virtual audio space. The specific operation includes adding information such as the sound source, sound propagation path, reflected sound, diffracted sound, etc. to the virtual space model, and adjusting the sound energy according to factors such as distance attenuation and air absorption, and finally generating audio data containing spatial information.

[0131] The present invention places virtual sound sources, such as human voices and music, at specific locations in a virtual space model and sets their acoustic properties, such as volume and timbre, to provide basic data for subsequent acoustic simulations; integrates data from multiple sound sources to form a complete sound source data set, which facilitates subsequent unified sound propagation path analysis and audio rendering for multiple sound sources; calculates the propagation path of sound from the sound source to the listener, including direct sound, reflected sound, diffracted sound, etc., based on acoustic principles and geometric information of the virtual space model, to provide data support for simulating real sound propagation effects; identifies the medium properties of different areas on the path, such as air, walls, furniture, etc., according to the sound propagation path, and assigns them different acoustic parameters. The method uses the following methods to analyze the reflection of different media surfaces when the sound encounters them on the propagation path, such as absorption coefficient and reflection coefficient, so as to more accurately simulate the changes in sound propagation in different media; analyze the reflection of the sound when it encounters different medium surfaces on the propagation path, including reflection direction and reflection intensity, so that the virtual sound is more in line with the reflection law of sound in the real world and enhances the sense of space; analyze the occlusion and diffraction of the sound when it encounters obstacles on the propagation path, such as sound bypassing walls and furniture, so that the spatial shielding effect of the virtual sound is more realistic and credible; apply the sound reflection, obstacle and other data obtained in the previous steps to the virtual space model, and perform the final audio rendering to generate a complete virtual audio space containing sound sources, spatial information and acoustic effects, so that users can have an immersive auditory experience.

[0132] Preferably, step S2 comprises the following steps:

[0133] Step S21: performing real-time face camera processing on the virtual reality device to obtain real-time photography stream data;

[0134] In an embodiment of the present invention, a camera integrated in a virtual reality device is used to capture a user's facial image in real time. The camera may be an infrared camera or a visible light camera, which is selected according to the application scenario and accuracy requirements. For example, by using a built-in infrared camera for eye tracking, a higher-precision eye image can be obtained. The image sequence captured by the camera is decoded into a digital image frame to form real-time photography stream data.

[0135] Step S22: performing face image detection on the real-time photographic stream data to obtain photographic face image data;

[0136] In the embodiment of the present invention, face detection is performed on each frame of real-time photography stream data to identify and locate the face area. Face detection can be performed using methods based on Haar features, HOG features, deep learning, etc. For example, the MTCNN model is used to perform face detection on the image frame, obtain the coordinate information of the face bounding box, and cut out the face area from the image frame according to the bounding box coordinates to obtain the photographic face image data.

[0137] Step S23: extracting the eye region of the photographed face image data to obtain photographed eye region data;

[0138] In an embodiment of the present invention, the eye area is further located and extracted in the detected face image. A feature-based method such as ASM, CLM, etc., or a deep learning-based method such as EyeNet, GazeNet, etc. can be used. For example, the eye key points in the face image, including the positions of the eye corners and eyelids, are detected using the pre-trained model provided by the library. According to the key point coordinates, the range of the eye area is determined, and the eye area image is cropped to obtain the photographic eye area data.

[0139] Step S24: performing pupil center positioning on the photographed eye area data to obtain pupil center positioning data;

[0140] In an embodiment of the present invention, the pupil center is located by locating the extracted eye region image. A method based on circle detection, such as Hough transform, RANSAC, etc., or a method based on grayscale features, such as Starburst method, center offset method, etc., can be used. For example, the center offset method based on grayscale projection is used to calculate the horizontal and vertical grayscale projection curves of the eye region image, find the peak point of the projection curve as the candidate position of the pupil center, and screen according to the grayscale features of the pupil region to determine the final pupil center coordinates and obtain pupil center positioning data.

[0141] Step S25: extracting the iris contour from the photographed eye area data to obtain iris contour image data;

[0142] In the embodiment of the present invention, the contour information of the iris is extracted from the eye region image. A method based on edge detection, such as the Canny operator, the Sobel operator, etc., or a method based on circle detection, such as the Hough transform, the RANSAC, etc., can be used. For example, the Hough transform method based on circle detection is used to detect circular edges in the eye region image, find the circular contour that meets the iris feature, and extract its edge pixel coordinates to obtain the iris contour image data.

[0143] Step S26: generating a gaze point for the pupil center positioning data and the iris contour image data to obtain line of sight gaze point data;

[0144] In the embodiment of the present invention, the user's gaze point is calculated by combining the pupil center positioning data and the iris contour image data. A method based on geometric relationships, such as a vector method, a trigonometric function method, etc., can be used. For example, the gaze direction vector is calculated based on the pupil center coordinates and the center coordinates of the iris contour; then, based on the gaze direction vector and the eyeball model, the intersection of the gaze and the virtual reality device screen is calculated as the user's gaze point, and the gaze point data is obtained.

[0145] Step S27: mapping the gaze point data pair to a coordinate system to obtain gaze point coordinate data.

[0146] In the embodiment of the present invention, by mapping the gaze point data from the image coordinate system to the screen coordinate system or three-dimensional space coordinate system of the virtual reality device, subsequent application processing is facilitated. For example, according to the screen resolution and field of view of the virtual reality device, the gaze point is mapped from the image coordinate system to the screen coordinate system to obtain the screen coordinates of the user's gaze point; or according to the posture information and camera parameters of the virtual reality device, the gaze point is mapped from the image coordinate system to the three-dimensional space coordinate system to obtain the three-dimensional coordinates of the user's gaze point in the virtual world, that is, the gaze point coordinate data.

[0147] The present invention can obtain the facial image data stream when the user wears the device by performing real-time face camera processing on the virtual reality device, providing a basic data source for subsequent face detection and eye tracking, so that the system can capture and analyze the user's facial information in real time; by performing face image detection on the real-time photography stream data, the position and range of the face in the image can be accurately identified, and then the face area can be separated from the background, providing accurate input data for the subsequent eye area extraction; eye area extraction is a key step for the entire system to analyze the user's line of sight focus. By accurately extracting the eye area in the photographed face image, other interference information in the image can be effectively removed, and the accuracy of subsequent pupil positioning and iris recognition can be improved; pupil center positioning is an important step in determining the user's line of sight direction. This step identifies the position of the pupil in the eye by analyzing the eye image data, providing basic reference data for line of sight tracking and focus judgment; iris contour extraction can provide important features for accurate iris recognition and subsequent line of sight tracking. By extracting the contour shape, texture and other unique features of the iris, the user can be uniquely identified and the accuracy of the entire system can be improved. By combining the pupil center and iris contour information, the user's gaze point in three-dimensional space, that is, the direction and position the user is observing, can be inferred through geometric calculations or deep learning models. The gaze point is mapped from the image coordinate system to the coordinate system of the virtual scene, so that the virtual scene can interact and provide feedback in real time based on the user's line of sight, such as gaze point selection, line of sight interaction, etc., thereby enhancing the immersion and interactivity of the experience.

[0148] Preferably, step S26 comprises the following steps:

[0149] Step S261: extracting iris feature points from the iris contour image data to obtain iris feature point data;

[0150] In the embodiment of the present invention, by using the iris contour image data, feature points on the iris are extracted. These feature points can be significant features of the iris texture, such as the edges and spots of the texture. Image processing techniques, such as edge detection, corner detection, etc., can be used, and deep learning models can also be used for feature extraction. For example, the Canny edge detection algorithm is used to extract edge information in the iris contour image, and the edge points are used as iris feature points; or a CNN-based feature point detection network is used to extract iris feature points.

[0151] Step S262: constructing an eyeball model based on the pupil center positioning data and the iris feature point data to obtain an eyeball model;

[0152] In an embodiment of the present invention, a three-dimensional model of the eyeball is constructed based on the pupil center positioning data and the iris feature point data. For example, a spherical model is used to approximate the eyeball, with the pupil center as the center of the sphere and the pupil radius as the radius of the sphere; or an ellipsoid model is used to more accurately describe the shape of the eyeball, and the parameters of the ellipsoid model are fitted using the iris feature points.

[0153] Step S263: acquiring the direction of the eyeball optical axis data of the eyeball model based on the pupil center positioning data to obtain the direction data of the eyeball optical axis;

[0154] In the embodiment of the present invention, for the spherical model, the vector from the pupil center to the sphere center can be directly calculated as the direction vector of the eyeball optical axis. For the ellipsoid model, the optical axis direction vector needs to be calculated based on the parameters of the ellipsoid model and the position of the pupil center. For example, the pupil center coordinates can be first converted to the local coordinate system of the ellipsoid model, and then the direction vector of the optical axis in the local coordinate system can be calculated based on the equation of the ellipsoid model, and finally the direction vector can be converted back to the world coordinate system.

[0155] Step S264: performing a sight line intersection point simulation on the eyeball optical axis direction data to obtain sight line intersection point data;

[0156] In the embodiment of the present invention, after obtaining the eye optical axis direction vectors of both eyes, the two optical axis direction vectors can be extended to calculate the intersection of the two straight lines as the line of sight intersection point. It should be noted that due to factors such as measurement errors, the two optical axis direction vectors do not intersect, and in this case, the least square method or other methods can be used to calculate the best intersection point.

[0157] Step S265: generating a sight line fixation point for the sight line intersection point data to obtain the sight line fixation point data.

[0158] In the embodiment of the present invention, the three-dimensional coordinates of the sight intersection point are converted to the screen coordinate system of the virtual reality device according to the screen parameters of the virtual reality device and the head posture information of the user. Then, the three-dimensional coordinates of the sight intersection point are projected onto the screen plane of the virtual reality device to obtain the two-dimensional coordinates of the sight gaze point on the screen.

[0159] The present invention extracts representative feature points from the iris contour image through this step, such as iris texture, color change, etc. These feature points can be used to fit the eyeball model more accurately and improve the accuracy of line of sight estimation; eyeball model construction is an important step in simulating the structure and movement of the human eye. By combining the pupil center positioning data and the iris feature point data, an accurate three-dimensional eyeball model can be established. This model can simulate the shape, size and structure of the human eye, and provide a basic framework for analyzing the eyeball optical axis and line of sight movement; the acquisition of the direction of the eyeball optical axis data is the key to understanding the line of sight direction. The direction vector of the eyeball optical axis can be calculated through the pupil center positioning data. This vector represents the direction from the pupil center to the eyeball model, which provides an important reference for simulating the line of sight intersection and ultimately determining the line of sight fixation point; the lines of sight of the left and right eyes will intersect at one point in space, that is, the line of sight intersection point, which represents the specific position that the user is looking at in three-dimensional space. By simulating the intersection process of the two eyes' gazes, the user's gaze point can be inferred more accurately. The calculated gaze intersection point is used as the user's gaze point. This data can be used for various interactions and applications in the system, such as gaze selection, gaze point rendering, eye movement analysis, etc., to enhance the immersion and interactivity of the experience.

[0160] Preferably, step S3 comprises the following steps:

[0161] Step S31: acquiring head sensor data of the virtual reality device to obtain head sensor data;

[0162] In the embodiment of the present invention, a high-precision sensor such as a gyroscope, an accelerometer, etc. is provided to track the head movement of the user. By accessing the API interface provided by the device, the raw data stream of the head sensor can be obtained in real time, including information such as the head rotation angle, acceleration, and timestamp.

[0163] Step S32: performing head posture data analysis on the head sensor data to obtain head posture data;

[0164] In the embodiment of the present invention, the head sensor data is denoised and fused by using algorithms such as Kalman filtering and complementary filtering, for example, the gyroscope data and the accelerometer data are fused to obtain a more accurate head posture estimation. At the same time, the head posture data can be converted into different representation forms as needed, such as converting the Euler angle into a rotation matrix.

[0165] Step S33: performing head deflection feature analysis on the head sensor data based on the head posture data to obtain head deflection feature data;

[0166] In an embodiment of the present invention, by calculating the rate of change of head posture data in a time series, such as the angular velocity and angular acceleration of head rotation, as a head deflection feature, the direction and amplitude of head rotation can also be analyzed, such as determining whether the head is turning left or right, whether the rotation angle is large or small, etc., and using this information as a head deflection feature.

[0167] Step S34: calculating the deflection gaze point vector of the gaze point data based on the head deflection feature data to obtain the deflection gaze point vector data;

[0168] In the embodiment of the present invention, the head deflection feature is converted into a sight line deflection vector by establishing a corresponding mapping relationship according to the type and size of the head deflection feature. For example, a two-dimensional vector proportional to the size and direction of the head rotation angular velocity can be calculated to represent the deflection direction and distance of the sight line on the screen plane.

[0169] Step S35: Calculate the deflected coordinates of the gaze point coordinate data based on the deflected gaze point vector data to obtain the deflected gaze point coordinates.

[0170] In the embodiment of the present invention, the original gaze point coordinates and the deflected gaze point vector are added to obtain new two-dimensional coordinates, which represent the screen position that the user is about to gaze at. It should be noted that the calculation of the deflected gaze point coordinates needs to take into account the screen parameters of the virtual reality device, such as screen resolution, field of view, etc.

[0171] The head sensor data of the virtual reality device is the basis for capturing the user's head movement and posture changes. Through the sensors built into the virtual reality device, such as gyroscopes, accelerometers and magnetometers, the position, rotation and movement information of the user's head in three-dimensional space can be obtained in real time, providing necessary data support for the subsequent analysis of head posture and deflection characteristics; head posture data analysis can help the system understand the current state of the user's head. By processing and analyzing the head sensor data, the posture information such as rotation, pitch and deflection angle of the head in the virtual space can be obtained, providing a basic reference for the subsequent deflection feature analysis and line of sight tracking; head deflection feature analysis aims to identify and quantify the lateral deflection movement of the user's head. By further processing the head posture data, the deflection angle and speed of the head in the horizontal direction can be extracted, providing key input data for calculating the deflection gaze point vector and ultimately determining the deflection gaze point coordinates; according to the user's head deflection characteristics, a deflection vector is calculated to correct the original gaze point direction to make it more consistent with the user's actual gaze direction; by combining the head sensor data of the virtual reality device and the deflection gaze point vector, the new gaze point coordinates generated by the user's head deflection in the virtual space can be calculated. This coordinate data can correspond to objects in the virtual environment to achieve more accurate user interaction and a more natural human-computer interaction experience.

[0172] Preferably, step S33 includes the following steps:

[0173] Step S331: setting a reference posture for the head posture data to obtain the head reference posture data;

[0174] In an embodiment of the present invention, a standard head posture is defined as a reference posture, such as a head upright and eyes looking straight ahead. The posture can be adjusted according to the actual application scenario. Then, the head posture data of the current user is obtained by using a sensor, including the rotation angle of the head around the X-axis, Y-axis and Z-axis. The obtained head posture data is compared with the pre-defined reference posture data, and the offset of the current head posture relative to the reference posture is calculated, and the offset is recorded as the head reference posture data.

[0175] Step S332: Calculating the head deflection angle of the head sensor data based on the head reference posture data to obtain the head deflection angle;

[0176] In the embodiment of the present invention, the head sensor data of the user is acquired in real time, including the rotation angle of the head around the X-axis, Y-axis and Z-axis. The head sensor data acquired in real time is compared with the head reference posture data to calculate the deflection angle of the head in each direction. For example, if the reference posture data is (0°, 0°, 0°), and the current head sensor data is (10°, -5°, 3°), then the head deflection angle is (10°, -5°, 3°).

[0177] Step S333: analyzing the deflection vector direction of the head deflection angle to obtain deflection vector direction data;

[0178] In an embodiment of the present invention, the calculated head deflection angle is converted into a deflection vector in a three-dimensional space. This can be achieved by converting the deflection angle into a direction vector in a spherical coordinate system. The deflection vector direction data can be represented as a unit vector, such as (0.87, -0.5, 0.26), which is used to represent the direction of the head deflection.

[0179] Step S334: calculating the deflection vector amplitude of the head deflection angle to obtain deflection vector amplitude data;

[0180] In the embodiment of the present invention, the amplitude of the head deflection is calculated according to the calculated head deflection angle. For example, the magnitude of the head deflection angle vector can be calculated using the Euclidean distance formula to obtain a scalar value representing the degree of head deflection, such as 11.18°.

[0181] Step S335: performing feature data fitting on the deflection vector direction data and the deflection vector amplitude data to obtain head deflection feature data.

[0182] In the embodiment of the present invention, the obtained deflection vector direction data and deflection vector amplitude data are integrated to form a multi-dimensional feature vector. For example, a four-dimensional vector (0.87, -0.5, 0.26, 11.18) can be used to represent the current head deflection feature data, where the first three elements represent the direction and the last element represents the amplitude.

[0183] The present invention can provide a reference basis for subsequent deflection angle calculation and analysis by setting a head reference posture. By establishing a standard or initial head posture as a reference, the degree of deflection of the user's head can be more accurately calculated and quantified. This reference posture is usually defined as the posture of the user looking straight ahead, providing a neutral reference point for subsequent analysis; the head deflection angle calculation is the basis for understanding the lateral movement of the user's head. By comparing the head sensor data and the head reference posture data, the deflection angle of the user's head in the horizontal direction can be calculated. This angle represents the degree of deviation of the head from the reference posture, providing a quantitative indicator for subsequent deflection vector analysis and simulation; by analyzing the head deflection angle, the direction vector of the deflection can be determined. This direction vector represents whether the head deflection is in the left, right, up and down or other specific directions, providing important input information for simulating and simulating the user's line of sight movement; by analyzing the head deflection angle, the amplitude of the deflection vector, that is, the distance or angle of the deflection, can be calculated. This amplitude data represents the intensity of the head deflection, providing parameters for simulating and analyzing the user's field of view and head movement amplitude; feature data fitting is the process of integrating the deflection vector direction and amplitude data. By mathematically fitting the direction data and amplitude data, a more complete head deflection feature model can be established. This model combines the deflection direction and amplitude information, and can more accurately represent the user's head deflection behavior, providing reliable input data for subsequent eye tracking, virtual space interaction or other related applications.

[0184] Preferably, step S4 comprises the following steps:

[0185] Step S41: performing coordinate grid mapping on the virtual space model to obtain a virtual space coordinate grid;

[0186] Step S42: performing spatial positioning of the deflected gaze point on the virtual space coordinate grid based on the deflected gaze point coordinates to obtain spatial deflected gaze point data;

[0187] Step S43: acquiring a virtual visual area for the spatial deflection gaze point data to obtain virtual visual area data;

[0188] Step S44: positioning the gaze audio area in the virtual audio space based on the virtual visual area data to obtain the gaze virtual audio area;

[0189] Step S45: Perform regional audio optimization on the virtual visual area data and the gaze virtual audio area to obtain regional audio optimization data.

[0190] As an embodiment of the present invention, refer to Figure 2 As shown, Figure 1 Detailed step flow diagram of step S4 in FIG. 1 , in this embodiment, step S4 includes the following steps:

[0191] Step S41: performing coordinate grid mapping on the virtual space model to obtain a virtual space coordinate grid;

[0192] In the embodiment of the present invention, the virtual space model is loaded into the system. Then, according to the preset accuracy, the virtual space model is divided into a series of regular or irregular coordinate grids. Each grid corresponds to a specific area in the virtual space and is assigned a unique coordinate identifier. The density of the grid can be adjusted according to the demand. For example, a denser grid can be used for an area that requires precise positioning.

[0193] Step S42: performing spatial positioning of the deflected gaze point on the virtual space coordinate grid based on the deflected gaze point coordinates to obtain spatial deflected gaze point data;

[0194] In the embodiment of the present invention, the coordinates of the user's deflected gaze point are obtained, and then the coordinates are mapped to the established virtual space coordinate grid to find the grid containing the coordinates. The grid is the position of the deflected gaze point in the virtual space, and the coordinate identifier of the grid and other related information, such as the distance and direction to the adjacent grid, are recorded as the spatial deflected gaze point data.

[0195] Step S43: acquiring a virtual visual area for the spatial deflection gaze point data to obtain virtual visual area data;

[0196] In an embodiment of the present invention, the user's virtual visual area is determined based on the obtained deflection gaze point spatial positioning information. The area can be simulated based on factors such as the user's field of view, deflection angle, etc. For example, a spherical area with the deflection gaze point as the center and the user's field of view as the radius can be defined as a virtual visual area. In addition, the shape and size of the virtual visual area can be adjusted according to the actual application scenario, for example, to simulate the field of view of the user when wearing a virtual reality device. Finally, all coordinate grids and related information in the virtual visual area are recorded as virtual visual area data.

[0197] Step S44: positioning the gaze audio area in the virtual audio space based on the virtual visual area data to obtain the gaze virtual audio area;

[0198] In the embodiment of the present invention, the corresponding fixation audio area is located in the virtual audio space according to the acquired virtual visual area data. For example, the projection area of ​​the virtual visual area in the virtual audio space can be defined as the fixation virtual audio area. The area includes the sound source currently heard by the user and the target area for audio optimization.

[0199] Step S45: Perform regional audio optimization on the virtual visual area data and the gaze virtual audio area to obtain regional audio optimization data.

[0200] In an embodiment of the present invention, virtual audio is optimized based on the obtained virtual visual area data and the virtual audio area being watched. For example, the sound clarity and volume in the virtual audio area being watched can be improved, the sound in other areas can be reduced, or the spatial orientation of the sound can be adjusted according to the user's deflected gaze point, thereby enhancing the user's immersion and auditory experience. Finally, the optimized audio parameters, area division and other information are recorded as regional audio optimization data.

[0201] The present invention can establish a coordinate framework of a virtual space by mapping a virtual space model to a coordinate grid. By creating a three-dimensional coordinate grid system, a unique coordinate value can be assigned to each point in the virtual space. This provides a basic framework for deflection gaze point positioning, virtual visual acquisition and audio optimization; the deflection gaze point space positioning is the process of mapping the deflected gaze point coordinates to the virtual space coordinate grid. Through this step, the exact position of the user's deflection sight focus in the virtual space can be determined, providing a basic reference for subsequent virtual visual and audio area acquisition; by analyzing the deflection gaze point data, the area in the virtual space that enters the user's field of view, that is, the virtual visual area, can be determined. This area will become the part of the virtual environment currently visible to the user, providing a basic input for the positioning of the gaze audio area and regional audio optimization; by analyzing the virtual visual area data, the virtual audio area currently gazed by the user can be determined. This step ensures that the audio processing focuses on the virtual space area that the user is interested in or concerned about, and improves the relevance and accuracy of the audio experience; the regional audio optimization aims to improve the quality of the user's audio experience in the virtual space. Through the comprehensive analysis of the virtual visual area data and the gaze audio area data, the audio effect can be optimized. For example, enhancing the audio clarity in the virtual visual area, increasing the volume of the virtual audio area, adding ambient sound field effects, etc., to make the audio experience more realistic and immersive.

[0202] Preferably, step S45 includes the following steps:

[0203] Step S451: performing regional acoustic characteristic analysis on the virtual audio area being watched to obtain regional acoustic characteristic data;

[0204] Step S452: performing spatial regional sound field simulation on the virtual audio space based on the regional acoustic characteristic data to obtain spatial regional sound field data;

[0205] Step S453: acquiring a gaze visual path for the virtual visual area data to obtain a gaze visual path;

[0206] Step S454: performing audio occlusion recognition on the gaze visual path to obtain path audio occlusion data;

[0207] Step S455: performing regional audio simulation optimization on the spatial regional sound field data based on the path audio occluder data to obtain regional audio optimization data.

[0208] As an embodiment of the present invention, refer to Figure 3 As shown, Figure 2 FIG. 4 is a schematic diagram of a detailed step flow chart of step S45 in FIG. 4 . In this embodiment, step S45 includes the following steps:

[0209] Step S451: performing regional acoustic characteristic analysis on the virtual audio area being watched to obtain regional acoustic characteristic data;

[0210] In an embodiment of the present invention, a three-dimensional model of the target virtual audio area is used. The model should contain information such as the geometric shape, size, material, etc. of the area. Then, select appropriate acoustic simulation software, for example, import the constructed three-dimensional model into the software. Set the sound source parameters according to actual conditions, such as the sound source type, sound pressure level, directivity, etc., as well as the air properties in the area, such as temperature, humidity, etc. Finally, run the acoustic simulation software to perform sound ray tracing and sound field calculation to obtain acoustic parameters at different positions in the area, such as reverberation time RT60, clarity index C50, speech clarity STI, etc. These acoustic parameters constitute the regional acoustic characteristic data, which are used for subsequent spatial regional sound field simulation.

[0211] Step S452: performing spatial regional sound field simulation on the virtual audio space based on the regional acoustic characteristic data to obtain spatial regional sound field data;

[0212] In an embodiment of the present invention, according to the regional acoustic characteristic data, combined with the overall structure and sound source distribution of the virtual audio space, the acoustic simulation software is used to simulate the sound field of the entire virtual audio space. During the simulation process, it is necessary to consider the sound propagation and attenuation between different areas, as well as the reflection, absorption and scattering effects of walls, floors, objects, etc. on sound. The acoustic effects in the real environment can be simulated by setting different acoustic material properties and geometric structures. After the simulation is completed, the sound pressure level, frequency response, sound field distribution and other data of each position in the virtual audio space can be obtained, and these data constitute the spatial regional sound field data.

[0213] Step S453: acquiring a gaze visual path for the virtual visual area data to obtain a gaze visual path;

[0214] In an embodiment of the present invention, the eye tracking function of an eye tracking device or a virtual reality head mounted display is used to record the eye trajectory data of the user in the virtual visual area. According to the eye trajectory data, the object or area that the user is looking at in the virtual scene is extracted, and adjacent gaze points are connected to form a continuous path, i.e., the gaze visual path. In order to improve the accuracy and stability of the path, a smoothing algorithm can be used to pre-process the eye trajectory data, such as a moving average filter, a Gaussian filter, etc.

[0215] Step S454: performing audio occlusion recognition on the gaze visual path to obtain path audio occlusion data;

[0216] In an embodiment of the present invention, by combining the gaze visual path with the three-dimensional model data of the virtual scene, it is determined whether there is an object blocking the user's line of sight on the path. For each object being gazed, the line between it and the user's viewpoint is calculated to determine whether the line intersects with other objects. If an intersection occurs, the object is considered to be blocked, and the information of the blocking object is recorded, including the object ID, the degree of occlusion, etc. Finally, the path audio blocker data is obtained.

[0217] Step S455: performing regional audio simulation optimization on the spatial regional sound field data based on the path audio occluder data to obtain regional audio optimization data.

[0218] In an embodiment of the present invention, the obtained spatial regional sound field data is optimized based on the obtained path audio obstruction data. For the obstructed sound source, its sound pressure level, frequency response and other parameters are adjusted according to the degree of obstruction to simulate the attenuation and change of the sound after being obstructed. For example, the transmission coefficient of the sound can be calculated according to the material and thickness of the obstructing object, so as to adjust the volume and timbre of the obstructed sound source. In this way, the sound in the virtual audio space can be made more consistent with the user's visual perception, and the regional audio optimization data is finally obtained.

[0219] The present invention aims to understand and quantify the sound characteristics of the virtual audio area through regional acoustic characteristics analysis. By analyzing the acoustic reflection, absorption, diffusion and other acoustic characteristics of this area, regional acoustic characteristics data can be obtained. These data include material properties, boundary effects, acoustic environment, etc., which provide basic parameters for subsequent spatial regional sound field simulation; by applying acoustic principles and regional acoustic characteristics data, the propagation, reflection and attenuation effects of sound in a specific virtual audio area can be simulated and predicted. This step helps to establish a realistic spatial audio model and provide a basic framework for the final audio optimization; by analyzing the virtual visual area data, the visual path from the user's deflected gaze point to other points in the virtual visual area can be determined. This path represents the movement trajectory of the user's line of sight in the virtual space, providing an important reference for the subsequent audio occlusion recognition and optimization; audio occlusion recognition aims to identify obstacles in the virtual space that affect sound propagation and auditory experience. By analyzing the gaze visual path, audio occluders on the visual path can be identified, such as virtual walls, furniture or other virtual objects. This step provides the necessary data for regional audio simulation optimization to ensure that the propagation and reflection of sound are consistent with the virtual environment settings; through the comprehensive analysis of these two factors, the regional audio effect can be optimized. For example, the attenuation, reflection or diffraction of sound when encountering occluders is simulated, and the appropriate acoustic model is applied to achieve a more realistic audio experience. This optimization step ensures that the audio presentation in the virtual space is consistent with the actual physical environment, enhancing the immersion and sense of space in virtual reality or mixed reality applications.

[0220] The present invention also provides an audio-visual device, including a head-mounted display, for executing the virtual reality interaction method as described above, wherein the head-mounted display of the audio-visual device includes:

[0221] A virtual audio space construction module is used to input image data to a virtual reality device to obtain spatial image data; construct a virtual space model on the spatial image data to obtain a virtual space model; and construct a virtual audio space on the virtual space model to obtain a virtual audio space;

[0222] The gaze point coordinate generation module is used to perform real-time face camera processing on the virtual reality device to obtain real-time photography stream data; perform face image detection on the real-time photography stream data to obtain photographic face image data; perform gaze point generation on the photographic face image data to obtain sight gaze point data; perform coordinate system mapping on the sight gaze point data to obtain gaze point coordinate data;

[0223] The deflection gaze point generation module is used to acquire head sensor data of the virtual reality device to obtain head sensor data; perform deflection gaze point vector calculation on the line of sight gaze point data based on the head sensor data to obtain deflection gaze point vector data; perform deflection coordinate calculation on the gaze point coordinate data based on the deflection gaze point vector data to obtain deflection gaze point coordinates;

[0224] The regional audio optimization module is used to acquire the virtual visual area of ​​the virtual space model based on the deflected gaze point coordinates to obtain virtual visual area data; locate the gaze audio area of ​​the virtual audio space based on the virtual visual area data to obtain the gaze virtual audio area; and perform regional audio optimization on the virtual visual area data and the gaze virtual audio area to obtain regional audio optimization data.

[0225] In summary, the present invention provides a virtual reality interaction method and an audio-visual device, which are composed of a virtual audio space construction module, a gaze point coordinate generation module, a deflected gaze point generation module, and a regional audio optimization module, and can implement any virtual reality interaction method described in the present invention, and are used to combine the operations between computer programs running on various modules to implement any virtual reality interaction method. The internal structures of the system cooperate with each other, which can greatly reduce duplication of work and manpower investment, and can quickly and effectively provide a more accurate and efficient virtual reality interaction process, thereby simplifying the virtual reality interaction method and the audio-visual device.

[0226] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A virtual reality interaction method, characterized in that: The following steps are involved: Step S1: inputting image data to a virtual reality device to obtain spatial image data; constructing a virtual spatial model for the spatial image data to obtain a virtual spatial model; constructing a virtual audio space for the virtual spatial model to obtain a virtual audio space; Step S2: performing real-time face camera processing on the virtual reality device to obtain real-time photography stream data; performing face image detection on the real-time photography stream data to obtain photographic face image data; performing gaze point generation on the photographic face image data to obtain sight gaze point data; performing coordinate system mapping on the sight gaze point data to obtain gaze point coordinate data; Step S3: acquiring head sensor data of the virtual reality device to obtain head sensor data; performing deflection gaze point vector calculation on the sight gaze point data based on the head sensor data to obtain deflection gaze point vector data; Deflection coordinates are calculated on the gaze point coordinate data based on the deflection gaze point vector data to obtain the deflection gaze point coordinates; Step S4: acquiring a virtual visual area of ​​the virtual space model based on the deflected gaze point coordinates to obtain virtual visual area data; Based on the virtual visual area data, the virtual audio space is positioned for the gaze audio area to obtain the gaze virtual audio area; the virtual visual area data and the gaze virtual audio area are regionally optimized to obtain regional audio optimization data.

2. The virtual reality interaction method according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: inputting image data into the virtual reality device to obtain spatial image data; Step S12: marking the spatial image data with spatial point cloud data to obtain spatial image point cloud data; Step S13: performing point cloud data registration on the spatial image point cloud data to obtain spatial point cloud registration data; Step S14: extracting spatial features from the spatial image point cloud data to obtain spatial point cloud feature data; Step S15: performing spatial entity segmentation on the spatial point cloud feature data to obtain spatial entity data; Step S16: constructing a virtual space model based on the spatial entity data and the spatial point cloud registration data to obtain a virtual space model; Step S17: constructing a virtual audio space for the virtual space model to obtain a virtual audio space.

3. The virtual reality interaction method according to claim 2, characterized in that: Step S17 includes the following steps: Step S171: setting the sound source of the virtual space model to obtain sound source setting data; Step S172: performing sound source data collection on the sound source setting data to obtain a sound source data set; Step S173: performing sound propagation path analysis on the sound source data set based on the virtual space model to obtain sound propagation path data; Step S174: setting the propagation path medium of the virtual space model based on the sound propagation path data to obtain the propagation path medium data; Step S175: performing sound reflection analysis on the propagation path medium data based on the sound source data set and the sound propagation path data to obtain sound reflection data; Step S176: performing sound obstacle analysis on the propagation path medium data based on the sound source data set and the sound propagation path data to obtain sound obstacle data; Step S177: Perform audio data spatial fitting on the virtual space model based on the sound reflection data and the sound obstacle data to obtain a virtual audio space.

4. The virtual reality interaction method according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: performing real-time face camera processing on the virtual reality device to obtain real-time photography stream data; Step S22: performing face image detection on the real-time photographic stream data to obtain photographic face image data; Step S23: extracting the eye region of the photographed face image data to obtain photographed eye region data; Step S24: performing pupil center positioning on the photographed eye area data to obtain pupil center positioning data; Step S25: extracting the iris contour from the photographed eye area data to obtain iris contour image data; Step S26: generating a gaze point for the pupil center positioning data and the iris contour image data to obtain line of sight gaze point data; Step S27: mapping the gaze point data pair to a coordinate system to obtain gaze point coordinate data.

5. The virtual reality interaction method according to claim 1, characterized in that: Step S26 includes the following steps: Step S261: extracting iris feature points from the iris contour image data to obtain iris feature point data; Step S262: constructing an eyeball model based on the pupil center positioning data and the iris feature point data to obtain an eyeball model; Step S263: acquiring the direction of the eyeball optical axis data for the eyeball model based on the pupil center positioning data to obtain the direction data of the eyeball optical axis; Step S264: performing a sight line intersection point simulation on the eyeball optical axis direction data to obtain sight line intersection point data; Step S265: generating a sight line fixation point for the sight line intersection point data to obtain the sight line fixation point data.

6. The virtual reality interaction method according to claim 5, characterized in that: Step S3 includes the following steps: Step S31: acquiring head sensor data of the virtual reality device to obtain head sensor data; Step S32: performing head posture data analysis on the head sensor data to obtain head posture data; Step S33: performing head deflection feature analysis on the head sensor data based on the head posture data to obtain head deflection feature data; Step S34: calculating the deflection gaze point vector of the gaze point data based on the head deflection feature data to obtain the deflection gaze point vector data; Step S35: Calculate the deflected coordinates of the gaze point coordinate data based on the deflected gaze point vector data to obtain the deflected gaze point coordinates.

7. The virtual reality interaction method according to claim 1, characterized in that: Step S33 includes the following steps: Step S331: setting a reference posture for the head posture data to obtain the head reference posture data; Step S332: Calculating the head deflection angle of the head sensor data based on the head reference posture data to obtain the head deflection angle; Step S333: analyzing the deflection vector direction of the head deflection angle to obtain deflection vector direction data; Step S334: calculating the deflection vector amplitude of the head deflection angle to obtain deflection vector amplitude data; Step S335: performing feature data fitting on the deflection vector direction data and the deflection vector amplitude data to obtain head deflection feature data.

8. The virtual reality interaction method according to claim 1, characterized in that: Step S4 includes the following steps: Step S41: performing coordinate grid mapping on the virtual space model to obtain a virtual space coordinate grid; Step S42: performing spatial positioning of the deflected gaze point on the virtual space coordinate grid based on the deflected gaze point coordinates to obtain spatial deflected gaze point data; Step S43: acquiring a virtual visual area for the spatial deflection gaze point data to obtain virtual visual area data; Step S44: positioning the gaze audio area in the virtual audio space based on the virtual visual area data to obtain the gaze virtual audio area; Step S45: Perform regional audio optimization on the virtual visual area data and the gaze virtual audio area to obtain regional audio optimization data.

9. The virtual reality interaction method according to claim 8, characterized in that: Step S45 includes the following steps: Step S451: performing regional acoustic characteristic analysis on the virtual audio area being watched to obtain regional acoustic characteristic data; Step S452: performing spatial regional sound field simulation on the virtual audio space based on the regional acoustic characteristic data to obtain spatial regional sound field data; Step S453: acquiring a gaze visual path for the virtual visual area data to obtain a gaze visual path; Step S454: performing audio occlusion recognition on the gaze visual path to obtain path audio occlusion data; Step S455: performing regional audio simulation optimization on the spatial regional sound field data based on the path audio occluder data to obtain regional audio optimization data.

10. An audio-visual device, characterized in that: A head mounted display is included for executing the virtual reality interaction method as claimed in claim 1, wherein the head mounted display of the audio-visual device comprises: A virtual audio space construction module is used to input image data to a virtual reality device to obtain spatial image data; construct a virtual space model on the spatial image data to obtain a virtual space model; and construct a virtual audio space on the virtual space model to obtain a virtual audio space; The gaze point coordinate generation module is used to perform real-time face camera processing on the virtual reality device to obtain real-time photography stream data; perform face image detection on the real-time photography stream data to obtain photographic face image data; perform gaze point generation on the photographic face image data to obtain sight gaze point data; perform coordinate system mapping on the sight gaze point data to obtain gaze point coordinate data; A deflection gaze point generation module is used to acquire head sensor data of a virtual reality device to obtain head sensor data; Deflection of the gaze point data is calculated based on the head sensor data to obtain the deflection of the gaze point vector data; Deflection coordinates are calculated on the gaze point coordinate data based on the deflection gaze point vector data to obtain the deflection gaze point coordinates; A regional audio optimization module is used to acquire a virtual visual area of ​​a virtual space model based on the deflected gaze point coordinates to obtain virtual visual area data; Based on the virtual visual area data, the virtual audio space is positioned with a gaze audio area to obtain a gaze virtual audio area; Regional audio optimization is performed on the virtual visual area data and the gaze virtual audio area to obtain regional audio optimization data.

Images