Light field image user eye movement data set construction method and system based on VR environment
By building virtual reality scenes in VR environment and collecting user eye movement data, a high-quality light field image user eye movement data set is built, which solves the problem of lack of accurate eye movement data sets in the existing technology, and promotes VR multimedia content optimization and intelligent processing of light field images.
Patent Information
- Application Number
- CN202411553587.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-02
- Publication Date
- 2025-02-07
AI Technical Summary
The prior art lacks eye movement data sets that can accurately record users' viewing light field images in VR environments, and cannot meet the needs of deeply optimized VR multimedia content.
By building virtual reality scenes in a VR environment, using VR headset devices with eye tracking functions to collect user eye movement data in real time, and combining image processing and data visualization technology to build a high-quality light field image user eye movement data set .
It realizes accurate recording of the user's viewing behavior of light field images in the VR environment, provides important data support, and lays the foundation for the intelligent processing of light field images and the optimization of VR multimedia content.
Smart Images
Figure CN119385498A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of virtual reality (VR) technology, and in particular to a method for constructing a light field image user eye movement data set based on virtual reality. Background Art
[0002] With the rapid development of virtual reality (VR) technology, immersive multimedia experience is gradually becoming an important interaction method. Especially in VR multimedia environments, users interact with the system through feedback from head and eye movements, and visual attention plays an important role in user experience. However, current VR multimedia experience designs usually rely on global content presentation and fixed interaction modes, lacking precise feedback on user visual attention data, which often leads to problems such as some content being insufficiently attractive to users and the user experience being not immersive enough.
[0003] In VR multimedia experience, eye movement data can effectively reflect the user's visual interest points and attention distribution. Therefore, by constructing an eye movement dataset of users watching multimedia content, the design of VR multimedia content can be further optimized, and the user's immersion and experience effect can be enhanced. In the prior art, although there are some VR datasets for head movement and user interaction, there is still a lack of datasets that can accurately record user eye movement information, which cannot meet the needs of deep optimization of VR multimedia content. Summary of the invention
[0004] The purpose of the present invention is to provide a method and system for constructing a light field image user eye movement dataset based on a VR environment, so as to solve the problem in the prior art of lacking an eye movement dataset that can accurately record the user's eye movement when viewing light field images in a VR environment, thereby providing data support for the intelligent processing of light field images and the optimization of VR multimedia content.
[0005] To achieve the above object, the present invention provides a method for constructing a light field image user eye movement dataset based on a VR environment, which specifically includes the following steps:
[0006] Image classification and selection: According to the characteristics of light field images, the images are classified into multiple categories, including complex background, simple background, high contrast, low brightness, etc.; representative light field images are selected from each category to construct the initial image set.
[0007] Image preprocessing: Preprocess the selected light field images, such as using noise reduction processing, to improve image quality; ensure that the preprocessed images are suitable for display in a VR environment to enhance user experience.
[0008] Building a VR experience environment: Use VR scene building platform software (such as Unity, etc.) to build a virtual reality scene and load pre-processed light field images; integrate VR plug-ins that support light field image display (such as COLIBRI VR, etc.) and integrate VR head-mounted display devices with eye tracking functions (such as HTC VIVE, etc.) to realize real-time collection of user eye movement information.
[0009] Eye movement data collection: Use eye tracking equipment to record the eye movement data of participants when viewing light field images in real time, including information such as pupil coordinates, and store it in files as needed.
[0010] Data processing and visualization: Format the collected eye movement data to remove outliers and invalid data; use data visualization tools (such as OpenCV, etc.) to generate eye movement trajectory graphs, heat maps, etc. to intuitively display the user's visual focus area.
[0011] Dataset construction: Associate the processed eye movement data with the corresponding light field images to establish a complete data mapping relationship; annotate and classify the dataset to generate a light field image user eye movement dataset that can be used for research and application.
[0012] Preferably, when classifying and selecting images, the classification of light field images may further include classification according to salient objects and scene categories, so as to ensure the diversity and comprehensiveness of the image set.
[0013] Preferably, when building a VR experience environment, the VR scene built using the tool includes a 6-DOF experience of the light field image, allowing the user to freely observe the light field image in the VR environment.
[0014] Preferably, when collecting eye movement data, the collection of eye movement data includes information such as pupil coordinates of the left and right eyes, line of sight direction, degree of eyelid opening and closing, etc., to ensure the integrity and accuracy of the data.
[0015] The present invention also provides a system for constructing a light field image user eye movement dataset based on the above method, the system comprising:
[0016] Image processing module: used to classify, select and preprocess light field images.
[0017] VR environment module: used to build VR experience scenes, load light field images, and integrate eye tracking devices.
[0018] Data collection module: used to collect the user's eye movement data in real time, including pupil coordinates and line of sight direction and other information.
[0019] Data processing module: used to clean, analyze and visualize the collected eye movement data.
[0020] Data storage module: used to store the processed eye movement data and corresponding light field images to build a complete data set.
[0021] Data management module: used to manage data sets, including data retrieval, sharing and security control.
[0022] The present invention collects eye movement data of users when viewing light field images in a VR environment, and constructs a high-quality eye movement dataset through the cooperation between multiple modules. This dataset can be used for research on intelligent compression, encoding, transmission and quality evaluation of light field images, and promotes the development of VR multimedia and light field image processing technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 This is a schematic diagram of the system architecture of a system for constructing a light field image user eye movement dataset based on a VR environment proposed by the present invention.
[0024] Figure 2 This is a flow chart of a method for constructing a light field image user eye movement dataset based on a VR environment proposed by the present invention. DETAILED DESCRIPTION
[0025] Example 1: System architecture
[0026] See also Figure 1 , the present invention proposes a system architecture diagram of a system for constructing a light field image user eye movement dataset based on a VR environment. The system mainly includes the following modules:
[0027] Image processing module: used to classify, select and preprocess light field images. First, according to the characteristics of light field images, they are classified into multiple categories such as complex background, simple background, high contrast, low brightness, etc. Representative light field images are selected from each category to construct an initial image set. Then, the selected images are preprocessed, such as through noise reduction, to improve the image quality and ensure that they are suitable for display in VR environment.
[0028] VR environment module: Using the VR scene construction platform, you can use various VR scene construction platforms, including but not limited to Unity, Unreal Engine and other software to help build VR scenes and load pre-processed light field images. Integrate plug-ins that support light field image display (such as COLIBRI VR, etc.) to help achieve the 6-degree-of-freedom display of light field images, allowing users to freely observe light field images in a VR environment. At the same time, use VR head-mounted display devices with eye tracking functions (such as HTC VIVE, etc.) to obtain users' eye movement information in real time.
[0029] Data collection module: used to collect the user's eye movement data in real time, including the pupil coordinates of the left and right eyes, the direction of sight, the degree of eyelid opening and closing, etc. The eye movement data is obtained through the eye tracker in the VR headset and saved in the specified file.
[0030] Data processing module: Format the collected eye movement data and remove outliers and invalid data. Use data visualization tools (such as OpenCV, etc.) to visualize the eye movement data, generate eye movement trajectory graphs and heat maps, and intuitively display the user's visual focus area.
[0031] Data storage module: used to store processed eye movement data and corresponding light field images, and establish a complete data mapping relationship. The data is stored in a structured form to facilitate subsequent retrieval and analysis.
[0032] Data management module: manages data sets, including data retrieval, sharing and security control. Formulates data access rights and security policies to ensure the legal use of data.
[0033] Example 2: Method flow
[0034] See also Figure 2 , a flowchart of a method for constructing a light field image user eye movement dataset based on a VR environment proposed by the present invention includes the following steps:
[0035] Step S1: Image classification and selection
[0036] According to the characteristics of light field images, the images are classified into categories including complex background, simple background, high contrast, low brightness objects, multiple objects and occlusion, etc. Then, several representative light field images are selected from each category to construct the initial image set.
[0037] Step S2: Image preprocessing
[0038] The selected light field images are preprocessed and processed using a noise reduction algorithm to improve the signal-to-noise ratio of the image and ensure that it is suitable for display in a VR environment.
[0039] Step S3: VR experience environment construction
[0040] Use VR scene building platforms (such as Unity, etc.) to build a VR experience environment and load pre-processed light field images. Integrate plug-ins that support light field image display (such as COLIBRI VR, etc.) to achieve 6-DOF display of light field images, allowing users to move and observe freely in the VR environment. At the same time, integrate VR head-mounted display devices with eye tracking functions (such as HTC VIVE, etc.) to achieve real-time collection of user eye movement data.
[0041] Step S4: Eye-tracking data collection
[0042] Participants wear VR headsets with eye tracking functions and watch multiple pre-selected light field images in a VR environment. The eye tracker in the headset collects participants' eye movement data in real time. Data collection includes pupil coordinates of the left and right eyes, line of sight direction, eyelid opening and closing degree, etc. During the data collection process, a data verification mechanism is used to ensure the accuracy and reliability of the data.
[0043] Step S5: Data processing and visualization
[0044] Format the collected eye movement data to remove outliers and invalid data. Format the collected eye movement data to remove outliers and invalid data. Read the participant's eye movement data file and convert the pupil coordinates into the image coordinate system. Then, draw the eye movement trajectory diagram, superimpose the gaze trajectory on the corresponding light field image, and generate a heat map to intuitively display the user's visual focus area.
[0045] Step S6: Dataset construction
[0046] The processed eye movement data is associated with the corresponding light field image to establish a complete data mapping relationship. The dataset is labeled and classified, including participant number, image number, eye movement data, head posture data, etc. Finally, a user eye movement dataset is constructed.
[0047] Example 3: Application of Dataset
[0048] By analyzing the constructed eye movement dataset, the following conclusions can be drawn:
[0049] Identification of visually significant areas: Based on the heat map, the areas in the light field image that are most likely to attract user attention can be identified, providing a reference for content optimization of the light field image.
[0050] Analysis of user viewing preferences: By collecting statistical eye movement data, we can understand users’ viewing patterns and preferences in different types of light field images, which helps to improve the design quality of VR multimedia content.
[0051] Light field image compression and transmission optimization: Using eye movement data, light field images can be compressed and encoded based on attention, improving transmission efficiency and reducing bandwidth requirements.
[0052] Example 4: System Expansion
[0053] The system and method of the present invention have good scalability and can further support:
[0054] Multi-scenario data collection: supports eye movement data collection of different types of VR content, such as panoramic videos, 3D models, etc.
[0055] Multimodal data fusion: Combine users’ physiological and psychological indicators, such as heart rate, brain waves, etc., to build a more comprehensive data set.
[0056] Training of deep learning models: Using the constructed dataset, deep learning-based models are trained for intelligent processing of light field images and VR content recommendation.
[0057] During the implementation process, it is necessary to ensure that the light, sound and other factors of the experimental environment are stable to avoid interference with the collection of eye movement data. Data security and privacy, strictly abide by data protection regulations to ensure the security of participants' personal information and data. Equipment calibration and maintenance, before each experiment, calibrate the VR headset to ensure the accuracy of eye movement data.
[0058] Through the above implementation methods, the present invention provides an efficient and reliable method and system for constructing a light field image user eye movement dataset based on a VR environment, which provides important data support for the optimization of VR multimedia content and the intelligent processing of light field images.
[0059] The above embodiments are preferred specific modes of the present invention, but the scope of protection is not limited thereto. Those skilled in the art may make changes and modifications in combination with the technical solutions and core concepts of the present invention. For example, when constructing a VR multimedia user experience data set, although a specific light field image display platform, eye tracking equipment, and data processing method are used, equivalent replacements can also be made by replacing other image processing technologies, tracking equipment, or data visualization tools. These substitutions do not deviate from the technical scope of the present invention, and all substitutions or modifications should be included in the scope of protection.
[0060] The preferred embodiments of the present invention are intended to illustrate the working principle and application of the system, so as to facilitate the understanding and implementation by technicians. The present invention is not limited to the details of a specific implementation, and technicians can adjust the method steps, experimental environment and data processing methods according to experimental requirements. For example, the collection and processing of user eye movement data can be achieved through other compatible virtual reality devices or image processing software. Based on these flexibilities, the core technical solution of the present invention can be applied to more scenarios and provide support for future VR multimedia technology and user behavior research. Ultimately, the scope of protection is defined by the claims and their equivalent technical scope.
Claims
1. A method for constructing a user eye movement dataset of a light field image based on a virtual reality (VR) environment, comprising: (a) Image processing: preprocess and classify light field images to generate an image set suitable for VR display; (b) VR environment construction: Create VR scenes, load light field images, and integrate eye tracking devices to support six-degree-of-freedom experience; (c) Eye movement data collection: real-time collection of eye movement data when viewing light field images, including pupil coordinates, line of sight direction, etc.; (d) Data processing and dataset construction: Reduce the noise of eye movement data and generate eye movement trajectory graphs and heat maps to construct an eye movement dataset.
2. The method according to claim 1, wherein: The image classification includes classification according to target objects and scene categories to ensure the diversity of the image set.
3. The method according to claim 1, wherein: The VR environment supports users to freely observe light field images and collect complete eye movement data.
4. A system for constructing a light field image user eye movement dataset based on a VR environment, comprising: (a) Image processing module: classify and preprocess light field images; (b) VR environment module: build VR scenes, load light field images and integrate eye tracking equipment; (c) Data collection module: collects user eye movement data in real time; (d) Data processing module: denoising the eye movement data and generating eye movement trajectory graphs and heat maps; (e) Data storage and management module: stores eye movement data, manages data retrieval, sharing and security control.
5. Application of the method of claims 1 to 4 to compress, encode, transmit and evaluate the quality of light field images based on eye movement datasets.
6. Application of the system described in claim 4 to analyze visual attention areas based on eye movement data to optimize the design and presentation of virtual reality content.