A human-machine interaction system based on virtual reality

Abstract

The application discloses a man-machine interaction system based on virtual reality 、 Interaction management module 、 Fatigue management module, the data management module is used for collecting eye data, position data of virtual articles in a virtual environment, eye movement data, user data, storing the collected data in a database after pretreatment; the interaction management module is used for judging whether the user interacts or not through the eye data, and determining the virtual article that the user interacts through calculation, analyzing the eye movement data of the user, and interacting through the interaction behavior corresponding to the eye movement data, the fatigue management module is used for calculating the fatigue index of the user through the user data, judging whether the user is in a fatigue state or not, collecting fatigue data, deriving a fatigue prediction model through historical fatigue data, predicting a time period in which the user is prone to fatigue, and warning the user, the application has the characteristics of improving user experience and predicting fatigue.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to a human-computer interaction system based on virtual reality. Background Technology

[0002] Virtual reality (VR) technology is a technology that uses computer technology to generate a virtual three-dimensional environment, which users enter and interact with by wearing special devices. The goal of VR technology is to create immersive experiences for users, making them feel as if they are in a virtual world and interacting with virtual objects and environments.

[0003] In recent years, with the rapid development of virtual reality technology, human-computer interaction technologies in virtual reality have become increasingly diverse. Existing virtual reality technologies mainly rely on gesture recognition, eye tracking, and voice recognition for human-computer interaction. Eye tracking technology captures the user's gaze direction to determine the user's focus in the virtual environment, thus enabling human-computer interaction. However, existing eye tracking can only simply lock onto virtual objects and obtain corresponding object information, without enabling other interactive behaviors, which affects the user experience. Furthermore, users are prone to fatigue after prolonged interaction in a virtual environment, affecting their physical health. Therefore, it is essential to design a virtual reality-based human-computer interaction system that improves user experience and predicts fatigue. Summary of the Invention

[0004] The purpose of this invention is to provide a human-computer interaction system based on virtual reality to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, the present invention provides the following technical solution: a human-computer interaction system based on virtual reality, the system specifically comprising:

[0006] The data management module is used to manage the data generated in the human-computer interaction system;

[0007] The interaction management module is used to manage the interaction behavior between users and virtual items in the virtual space;

[0008] The fatigue management module is used to manage users when they are fatigued.

[0009] According to the above technical solution, the data management module includes:

[0010] The data acquisition unit is used to collect and store eye data, location data of virtual objects in the virtual environment, eye movement data, and user data;

[0011] The data processing unit is used to preprocess the collected eye data, the location data of virtual objects in the virtual environment, eye movement data, and user data.

[0012] According to the above technical solution, the interactive management module includes:

[0013] The interaction judgment unit is used to determine whether to perform human-computer interaction by comparing the time difference between eye movements;

[0014] The interaction determination unit is used to determine the virtual object that the user interacts with by calculating the rotation angle of the eyeball and the angle between the virtual object and the corresponding camera.

[0015] The interaction execution unit is used to perform human-computer interaction by analyzing eye-tracking data and using the corresponding interactive behaviors.

[0016] According to the above technical solution, the fatigue management module includes:

[0017] The fatigue assessment unit is used to determine the user's physical condition by accessing user data.

[0018] The fatigue prediction unit is used to predict the time periods when a user is likely to be in a state of fatigue based on historical fatigue data.

[0019] The alert unit is used to issue a warning when the user is fatigued.

[0020] According to the above technical solution, the data acquisition unit includes:

[0021] Virtual reality headsets are equipped with built-in cameras to capture eye movements, collect eye and eye movement data, and collect data on virtual objects within virtual spaces.

[0022] Sensors are used to collect the user's heart rate data.

[0023] According to the above technical solution, the specific operation steps of the operation method of the virtual reality-based human-computer interaction system are as follows:

[0024] Step S1: Establish a database, collect user eye data, virtual object location data, eye movement data, and user data in real time, preprocess the data, and store the preprocessed data in the corresponding database;

[0025] Step S2: Determine whether the user interacts. When the user interacts with the computer, the virtual object is determined by calculating the rotation angle of the eyeball and the angle between the virtual object and the corresponding camera. The user then interacts with the virtual object by analyzing the user's eye movement data.

[0026] Step S3: Retrieve user data from the user database, calculate the user's fatigue index, issue a warning to the user, collect timestamps of the user's fatigue state, predict the time periods when the user is likely to be fatigued, and remind the user.

[0027] According to the above technical solution, the establishment of an eye database and the real-time collection of eye data include eye pupil images, eye pupil coordinates, initial coordinates of the eye pupil, and distance between the eye and the display. The eye images are captured in real time by deploying an infrared camera, a coordinate system is established for the eye images, and the coordinates corresponding to the eye pupils are obtained. The timestamps corresponding to the eye pupil coordinate data are obtained, and the eye pupil coordinate data and the corresponding timestamp data are stored in the eye database.

[0028] Establish a virtual environment database, obtain the location coordinates of virtual objects in the virtual environment in real time, store the coordinate data of virtual objects in the virtual environment database, establish an eye-tracking database, and store the eye-tracking data set by the system and the corresponding interactive behaviors in the eye-tracking database.

[0029] A user database is established to collect user data in real time. The user data is preprocessed and then stored in the user database. The user data includes heart rate data, blink frequency, runtime, and pupil contraction data.

[0030] According to the above technical solution, step S2 further includes the following steps:

[0031] Step S21: Retrieve the coordinate data (x) of the pupil from the eye database within the system-defined period X. t ,y t ), initial coordinates of the pupil (x0, y0), distance L between the eyeball and the monitor, calculate the rotation angle of the eyeball. Where t = 1, 2, ..., m, the rotation angle of the eyeball is stored in the database, and the difference Δθ between the rotation angles of the eyeballs between adjacent pupil images within the period X is calculated. y =|θ t+1 -θ t | where y = 1, 2, ..., n, the threshold Δθ0 in the database is used to calculate the difference Δθ between all eyeball rotation angles within the period X. y Marking is performed, and the number of marks is counted using a counter. When Δθ y When ≥Δθ0, mark it as the first eye-tracking result, increment the value of counter one, and obtain the number M of consecutively marked first eye-tracking results in counter one. When Δθ yWhen Δθ < 0, it is marked as the second eye movement result, the value of counter two is incremented, the number N of consecutive markings of the second eye movement result in counter two is obtained, the thresholds N0 and M0 set in the database are called. When M ≥ M0, it is determined that the user is quickly browsing virtual items. When N < M0 or M < M0, it is determined that the user is performing normal eye activities. When N ≥ N0, it is determined that the user is performing an interaction behavior with the virtual item;

[0032] Step S22: When the system determines that the user is performing an interaction behavior, the corresponding eyeball rotation angle θ in the database is called t , and the corresponding direction is V = x t + y t + L, and the corresponding direction feature D = (θ t , V) of the eyeball is obtained. The coordinate data (x i , y i , z i ) of the virtual item in the virtual environment database is called, where i = 1, 2,..., x is the number of virtual items in the database, and the included angle between the virtual item and the position of the camera in the virtual environment is calculated where (x C , y C , z C ) are the coordinates of the camera in the virtual space, and the direction corresponding to the virtual item is represented as V i = (x i - x C ) + (y i - y C ) + (z i - z C ), and the direction feature corresponding to the virtual item is obtained as D i = (θ i , V i ), and the direction feature corresponding to the virtual item is stored in the database;

[0033] Step S23: Call the direction feature D corresponding to the virtual item in the database i , and calculate the difference between the direction feature of the eyeball and the direction feature of the virtual item Select the virtual item with the smallest difference between the direction feature of the eyeball and the direction feature of the virtual item as the virtual item for interaction with the user, and display the simple information of the corresponding item on the virtual item.

[0034] According to the above technical solution, the step S23 further includes:

[0035] After the user selects a virtual item for interaction through eye selection, the system generates a set interaction interface including different eye movement data and corresponding interaction behaviors. The system obtains the timestamp of the interaction, and calls the pictures of the eyeballs and the corresponding pupil coordinates (x t ,y t ) in the eye database for a period of time after the corresponding timestamp, and the initial coordinates (x0, y0) of the eyeball pupil. The motion characteristics K of the eyeball are extracted, including the rotation direction of the eyeball, the number of blinks, and the number of eye closures after selecting the virtual item for interaction;

[0036] The rotation direction of the eyeball is obtained by comparing the magnitudes of the pupil coordinates of the eyeball. When x t >x0, it is determined that the eyeball rotates to the right. When x t <x0, it is determined that the eyeball rotates to the left. When y t >y0, it is determined that the eyeball rotates upward. When y t <y0, it is determined that the eyeball rotates downward, and the obtained rotation direction of the eyeball is recorded into the motion characteristics K of the eyeball. The eye movement data K x in the eye movement database and the corresponding interaction behaviors are called. Here, x is the type of interaction behavior in the database. By comparing the rotation direction of the eyeball in the motion characteristics of the eyeball with the rotation direction of the eyeball in the eye movement database, the interaction behavior corresponding to the eye movement data with the same rotation direction of the eyeball is selected to interact with the virtual item;

[0037] The number of blinks is obtained by screening the pictures of the eyeballs without pupils, obtaining the corresponding timestamps, and calculating the timestamp difference ΔT between the pictures without pupils and the next picture with pupils. The threshold ΔT0 set in the database is called. When ΔT < ΔT0, it is determined that the user blinks, and the number of blinks of the user is counted. The obtained number of blinks is recorded into the motion characteristics K of the eyeball. The eye movement data K x in the eye movement database and the corresponding interaction behaviors are called. By comparing the number of blinks in the motion characteristics of the eyeball with the number of blinks in the eye movement database, the interaction behavior corresponding to the eye movement data with the same number of blinks is selected to interact with the virtual item;

[0038] The number of eye closures is obtained by screening the pictures of the eyeballs without pupils, obtaining the corresponding timestamps, and calculating the timestamp difference ΔT between the pictures without pupils and the next picture with pupils. The threshold ΔT0 set in the database is called. When ΔT ≥ ΔT0, it is determined that the user closes the eyes, and the number of eye closures is recorded. The obtained number of eye closures is recorded into the motion characteristics K of the eyeball. The eye movement data K x in the eye movement database and the corresponding interaction behaviors are called. By comparing the number of eye closures in the motion characteristics of the eyeball with the number of eye closures in the eye movement database, the interaction behavior corresponding to the eye movement data with the same number of eye closures is selected to interact with the virtual item.

[0039] According to the above technical solution, step S3 further includes the following steps:

[0040] Step S31: Retrieve user database data on central rate α, blink frequency β, runtime γ, and pupil constriction. Calculate the user's fatigue index Where, ω α ω β ω γ , The system calls the threshold ε0 set in the database for the weight parameters set in the database. When ε < ε0, the user's physical condition is judged to be normal. When ε ≥ ε0, the user's physical condition is judged to be fatigued. The system prompts the user that the user is fatigued and collects the timestamps corresponding to the fatigue state and stores the corresponding timestamps in the database.

[0041] Step S32: Call the timestamp data of historical fatigue states in the database, retrieve the corresponding model algorithm from the database, use the timestamp data of historical fatigue states as sample data to train the model, obtain the fatigue time prediction model, predict the time period when the user is likely to enter a fatigue state, and when the user approaches the predicted fatigue time period, the system will remind the user a period of time in advance to avoid excessive fatigue.

[0042] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention collects eye data, location data of virtual objects in a virtual environment, eye movement data, and user data. After preprocessing the collected data, it stores it in a database. It determines whether the user is interacting based on eye data, and identifies the virtual object the user is interacting with by calculating the similarity between the eye rotation angle and the angle between the virtual object and the camera. It analyzes the user's eye movement data, uses the corresponding interactive behavior based on the eye movement data to perform interaction, and calculates the user fatigue index by calling user data to determine whether the user is in a fatigued state. It also collects fatigue data, derives a fatigue prediction model based on historical fatigue data, predicts the time periods when the user is prone to fatigue, and issues warnings to the user. This invention has the characteristics of improving user experience and predicting fatigue. Attached Figure Description

[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0044] Figure 1 This is a schematic diagram of the module composition of a virtual reality-based human-computer interaction system provided in Embodiment 1 of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Example 1: Example 1 of the present invention provides a human-computer interaction system based on virtual reality. Figure 1 This is a schematic diagram of the module composition of a virtual reality-based human-computer interaction system provided in Embodiment 1 of the present invention, as shown below. Figure 1 As shown, the system includes:

[0047] The data management module is used to manage the data generated in the human-computer interaction system;

[0048] The interaction management module is used to manage the interaction behavior between users and virtual items in the virtual space;

[0049] The fatigue management module is used to manage users when they are fatigued.

[0050] In some embodiments of the present invention, the following are included:

[0051] The data acquisition unit is used to collect and store eye data, location data of virtual objects in the virtual environment, eye movement data, and user data;

[0052] The data processing unit is used to preprocess the collected eye data, the location data of virtual objects in the virtual environment, eye movement data, and user data.

[0053] In some embodiments of the present invention, the following are included:

[0054] The interaction judgment unit is used to determine whether to perform human-computer interaction by comparing the time difference between eye movements;

[0055] The interaction determination unit is used to determine the virtual object that the user interacts with by calculating the rotation angle of the eyeball and the angle between the virtual object and the corresponding camera.

[0056] The interaction execution unit is used to perform human-computer interaction by analyzing eye-tracking data and using the corresponding interactive behaviors.

[0057] In some embodiments of the present invention, the following are included:

[0058] The fatigue assessment unit is used to determine the user's physical condition by accessing user data.

[0059] The fatigue prediction unit is used to predict the time periods when a user is likely to be in a state of fatigue based on historical fatigue data.

[0060] The alert unit is used to issue a warning when the user is fatigued.

[0061] In some embodiments of the present invention, the following are included:

[0062] Virtual reality headsets are equipped with built-in cameras to capture eye movements, collect eye and eye movement data, and collect data on virtual objects within virtual spaces.

[0063] Sensors are used to collect the user's heart rate data.

[0064] Example 2: This example can be applied to human-computer interaction scenarios. The method can be executed by a virtual reality-based human-computer interaction system provided in Example 1. The method specifically includes the following steps:

[0065] Step S1: Establish a database, collect user eye data, virtual object location data, eye movement data, and user data in real time, preprocess the data, and store the preprocessed data in the corresponding database;

[0066] Step S2: Determine whether the user interacts. When the user interacts with the computer, the virtual object is determined by calculating the rotation angle of the eyeball and the angle between the virtual object and the corresponding camera. The user then interacts with the virtual object by analyzing the user's eye movement data.

[0067] Step S3: Retrieve user data from the user database, calculate the user's fatigue index, issue a warning to the user, collect timestamps of the user's fatigue state, predict the time periods when the user is likely to be fatigued, and remind the user.

[0068] In this embodiment of the invention, an eye database is established to collect eye data in real time. The eye data includes eyeball and pupil images, eyeball and pupil coordinates, initial coordinates of the eyeball and pupil, and distance between the eyeball and the display. Eyeball images are captured in real time by deploying an infrared camera, a coordinate system is established for the eyeball images, and the coordinates corresponding to the eyeball and pupil are obtained. The timestamp corresponding to the eyeball and pupil coordinate data is obtained, and the eyeball and pupil coordinate data and the corresponding timestamp data are stored in the eye database.

[0069] Establish a virtual environment database to obtain the location coordinates of virtual objects in the virtual environment in real time, and store the coordinate data of virtual objects in the virtual environment database; establish an eye-tracking database to store the eye-tracking data set by the system and the corresponding interactive behaviors in the eye-tracking database;

[0070] A user database is established to collect user data in real time. After preprocessing the user data, it is stored in the user database. The user data includes heart rate data, blink frequency, running duration, and pupil constriction data.

[0071] In an embodiment of the present invention, step S2 further includes the following steps:

[0072] Step S21: Call the coordinate data (x t , y t ) of the eyeball pupil, the initial coordinates (x0, y0) of the eyeball pupil, and the distance L between the eyeball and the display in the eye database within the system-set period X, and calculate the rotation angle of the eyeball where t = 1, 2,..., m, store the rotation angle of the eyeball in the database, and calculate the difference Δθ in the rotation angle of the eyeball between adjacent pupil pictures within the period X y = |θ t+1 - θ t |, where y = 1, 2,..., n, call the threshold Δθ0 in the database, mark all the differences Δθ in the rotation angle of the eyeball within the period X y , and count the number of marked ones through a counter. When Δθ y ≥Δθ0, mark it as the first eye movement result, increase the value of counter one by one, and obtain the number M of consecutive marked first eye movement results in counter one. When Δθ y <Δθ0, mark it as the second eye movement result, increase the value of counter two by one, and obtain the number N of consecutive marked second eye movement results in counter two. Call the thresholds N0 and M0 set in the database. When M≥M0, it is determined that the user is quickly browsing virtual items. When N < N0 or M < M0, it is determined that the user is performing normal eye activities. When N≥N0, it is determined that the user is interacting with virtual items;

[0073] Step S22: When the system determines that the user is performing an interaction behavior, call the corresponding rotation angle θ of the eyeball in the database t , and the corresponding direction is V = x t + y t + L, to obtain the direction feature D = (θ t , V) of the eyeball. Call the coordinate data (x i , y i , z i ) of the virtual item in the virtual environment database, where i = 1, 2,..., x is the number of virtual items in the database, and calculate the included angle between the virtual item and the position of the camera in the virtual environment where (x C , y C , z C ) is the coordinate of the camera in the virtual space, and the direction corresponding to the virtual item is represented as Vi = (x i - x C ) + (y i - y C ) + (z i - z C ), obtaining the direction feature corresponding to the virtual item as D i = (θ i , V i ), and storing the direction feature corresponding to the virtual item into the database;

[0074] Step S23: Call the direction feature D corresponding to the virtual item in the database i , and calculate the difference between the direction feature of the eyeball and the direction feature of the virtual item Select the virtual item with the smallest difference between the direction feature of the eyeball and the direction feature of the virtual item as the virtual item for interacting with the user, and display the simple information of the corresponding item on the virtual item.

[0075] In the embodiment of the present invention, the step S23 further includes:

[0076] When the user selects the virtual item to be interacted with through the eyes, the system generates a set interaction interface including different eye movement data and corresponding interaction behaviors. The system obtains the timestamp of the interaction, calls the pictures of the eyeball, the corresponding pupil coordinates (x t , y t ) and the initial coordinates (x0, y0) of the eyeball pupil in the eye database for a period of time after the corresponding timestamp, and extracts the movement feature K of the eyeball including the rotation direction, the number of blinks, and the number of eye closures of the eyeball after selecting the virtual item to be interacted with;

[0077] The rotation direction of the eyeball is obtained by comparing the sizes of the pupil coordinates of the eyeball. When x t > x0, it is determined that the eyeball rotates to the right. When x t < x0, it is determined that the eyeball rotates to the left. When y t > y0, it is determined that the eyeball rotates upward. When y t < y0, it is determined that the eyeball rotates downward, and the obtained rotation direction of the eyeball is recorded into the movement feature K of the eyeball. Call the eye movement data K x in the eye movement database and the corresponding interaction behaviors. Among them, x is the type of interaction behavior in the database. By comparing the rotation direction of the eyeball in the movement feature of the eyeball with the rotation direction of the eyeball in the eye movement database, select the interaction behavior corresponding to the eye movement data with the same rotation direction of the eyeball to interact with the virtual item;

[0078] The blink count is determined by filtering images of the eyeball that do not show a pupil, obtaining the corresponding timestamps, calculating the timestamp difference ΔT between the image without a pupil and the next image with a pupil, and calling a threshold ΔT0 set in the database. When ΔT < ΔT0, the user is considered to have blinked, and the number of blinks is counted. The obtained blink count is recorded in the eye movement feature K, and the eye movement data K in the eye movement database is called. x The corresponding interactive behavior is to compare the number of blinks in the eye movement features with the number of blinks in the eye movement database, and select the interactive behavior corresponding to the eye movement data with the same number of blinks to interact with the virtual object;

[0079] The number of eye closures is determined by filtering images of the eyeball that do not show pupils, obtaining their corresponding timestamps, calculating the timestamp difference ΔT between the image without pupils and the next image showing pupils, and calling a threshold ΔT0 set in the database. When ΔT ≥ ΔT0, the user is considered to have closed their eyes, and the number of times they closed their eyes is recorded. This number of eye closures is then entered into the eye movement feature K, and eye movement data K from the eye movement database is retrieved. x Corresponding interactive behaviors are generated by comparing the number of times the eyes close in the eye movement characteristics with the number of times the eyes close in the eye movement database. Interactive behaviors corresponding to eye movement data with the same number of times the eyes close are selected to interact with virtual objects. When the user confirms the interactive behavior, the corresponding interactive behavior is performed on the virtual object by analyzing the user's eye movement behavior. For example, blinking twice corresponds to enlarging the virtual object, rotating the eye to the left corresponds to rotating the virtual object to the left, and closing the eyes corresponds to ending the interactive behavior, which improves the user's experience of virtual reality human-computer interaction.

[0080] In this embodiment of the invention, step S3 further includes the following steps:

[0081] Step S31: Retrieve user database data on central rate α, blink frequency β, runtime γ, and pupil constriction. Calculate the user's fatigue index Where, ω α ω β ω γ , The system calls the threshold ε0 set in the database for the weight parameters set in the database. When ε < ε0, the user's physical condition is judged to be normal. When ε ≥ ε0, the user's physical condition is judged to be fatigued. The system prompts the user that the user is fatigued and collects the timestamps corresponding to the fatigue state and stores the corresponding timestamps in the database.

[0082] Step S32: Call the timestamp data of historical fatigue states in the database, retrieve the corresponding model algorithm from the database, use the timestamp data of historical fatigue states as sample data to train the model, obtain the fatigue time prediction model, predict the time period when the user is likely to enter a fatigue state, and when the user approaches the predicted fatigue time period, the system will remind the user a period of time in advance to avoid excessive fatigue.

[0083] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0084] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A human-computer interaction system based on virtual reality, characterized in that: The system specifically includes: The data management module is used to manage the data generated in the human-computer interaction system; The interaction management module is used to manage the interaction behavior between users and virtual items in the virtual space; The fatigue management module is used to manage users when they are fatigued. The method for operating a virtual reality-based human-computer interaction system includes: Step S1: Establish a database, collect user eye data, virtual object location data, eye movement data, and user data in real time, preprocess the data, and store the preprocessed data in the corresponding database; Step S2: Determine whether the user interacts. When the user interacts with the computer, the virtual object is determined by calculating the rotation angle of the eyeball and the angle between the virtual object and the corresponding camera. The user then interacts with the virtual object by analyzing the user's eye movement data. Step S3: Retrieve user data from the user database, calculate the user's fatigue index, issue a warning to the user, collect timestamps of when the user is in a fatigued state, predict the time periods when the user is likely to be in a fatigued state, and remind the user accordingly; An eye database is established to collect eye data in real time. The eye data includes images of the pupil, coordinates of the pupil, initial coordinates of the pupil, and distance between the eye and the display. Infrared cameras are deployed to capture eye images in real time. A coordinate system is established for the eye images, and the coordinates of the pupil are obtained. The timestamps corresponding to the pupil coordinate data are obtained, and the pupil coordinate data and the corresponding timestamp data are stored in the eye database. Establish a virtual environment database, obtain the location coordinates of virtual objects in the virtual environment in real time, store the coordinate data of virtual objects in the virtual environment database, establish an eye-tracking database, and store the eye-tracking data set by the system and the corresponding interactive behaviors in the eye-tracking database. A user database is established to collect user data in real time. The user data is preprocessed and then stored in the user database. The user data includes heart rate data, blink frequency, runtime, and pupil constriction data. Step S2 further includes the following steps: Step S21: Call the system-defined cycle Coordinate data of pupil in the inner eye database Initial coordinates of the pupil Distance between the eyeball and the monitor Calculate the rotation angle of the eyeball ,in, The rotation angle of the eyeball is stored in a database, and the cycle is calculated. Difference in eye rotation angle between adjacent pupil images ,in, Call the threshold from the database , for the period The difference in rotation angle of all eyeballs Mark the items and count the number of marks using a counter. At that time, mark it as the first eye-tracking result, increment the value of counter one, and obtain the number of consecutively marked first eye-tracking results in counter one. ,when When the result is recorded as the second eye-tracking result, the value of counter two is incremented, and the number of consecutively recorded second eye-tracking results in counter two is obtained. Call the threshold set in the database , ,when When determining that a user is quickly browsing virtual items, or When the system determines that the user is performing normal eye movements, At that time, determine the user's interaction behavior with virtual items; Step S22: When the system determines that the user is performing an interactive action, it retrieves the corresponding eye rotation angle from the database. The corresponding direction is To obtain the directional features corresponding to the eyeball Retrieve coordinate data of virtual items from the virtual environment database. ,in, Given the number of virtual items in the database, calculate the angle between the virtual items and the camera positions in the virtual environment. ,in The coordinates of the camera in the virtual space are represented as follows: the direction corresponding to the virtual object is represented as... The directional features corresponding to the virtual items are obtained as follows: The directional features corresponding to virtual items are stored in the database; Step S23: Call the directional features corresponding to the virtual items in the database. Calculate the difference between the directional features of the eyeball and the directional features of virtual objects. The virtual item with the smallest difference between the direction feature of the eyeball and the direction feature of the virtual item is selected as the virtual item to be interacted with by the user, and simple information about the corresponding item is displayed on the virtual item.

2. The human-computer interaction system based on virtual reality according to claim 1, characterized in that: The data management module includes: The data acquisition unit is used to collect and store eye data, location data of virtual objects in the virtual environment, eye movement data, and user data; The data processing unit is used to preprocess the collected eye data, the location data of virtual objects in the virtual environment, eye movement data, and user data.

3. The human-computer interaction system based on virtual reality according to claim 2, characterized in that: The interactive management module includes: The interaction judgment unit is used to determine whether to perform human-computer interaction by comparing the time difference between eye movements; The interaction determination unit is used to determine the virtual object that the user interacts with by calculating the rotation angle of the eyeball and the angle between the virtual object and the corresponding camera. The interaction execution unit is used to perform human-computer interaction by analyzing eye-tracking data and using the corresponding interactive behaviors.

4. A human-computer interaction system based on virtual reality according to claim 3, characterized in that: The fatigue management module includes: The fatigue assessment unit is used to determine the user's physical condition by accessing user data. The fatigue prediction unit is used to predict the time periods when a user is likely to be in a state of fatigue based on historical fatigue data. The alert unit is used to issue a warning when the user is fatigued.

5. A human-computer interaction system based on virtual reality according to claim 4, characterized in that: The data acquisition unit includes: Virtual reality headsets are equipped with built-in cameras to capture eye movements, collect eye and eye movement data, and collect data on virtual objects within virtual spaces. Sensors are used to collect the user's heart rate data.

6. A human-computer interaction system based on virtual reality according to claim 5, characterized in that: Step S23 further includes: When a user selects a virtual object to interact with using their eyes, the system generates a pre-defined interactive interface that includes different eye movement data and corresponding interactive behaviors. The system obtains the timestamp of the interaction and retrieves images of the eyeballs and corresponding pupil coordinates from the eye database for a period of time after the corresponding timestamp. Initial coordinates of the pupil Extracting eye movement features This includes the direction of eye rotation, the number of blinks, and the number of times the eyes close after selecting a virtual object that needs to be interacted with; The direction of eyeball rotation is obtained by comparing the sizes between the coordinates of the eyeball and the pupil. When judging the rightward rotation of the eyeball, when To determine if the eyeball is rotating to the left, when... When judging the upward rotation of the eyeball, when At that time, the downward rotation of the eyeball is determined, and the obtained direction of eyeball rotation is recorded into the eye movement feature. In the middle, eye-tracking data is called from the eye-tracking database. With the corresponding interactive behaviors, among which, To identify the types of interactive behaviors in the database, the eye rotation direction in the eye movement features is compared with the eye rotation direction in the eye movement database. Interactive behaviors corresponding to eye movement data with the same eye rotation direction are selected to interact with virtual objects. The blink count was determined by filtering eye images that did not show pupils, obtaining their corresponding timestamps, and calculating the timestamp difference between the image without pupils and the next image showing pupils. Call the threshold set in the database ,when At the same time, the system detects when the user blinks, counts the number of blinks, and records the blink count into the eye movement characteristics. In the middle, eye-tracking data is called from the eye-tracking database. The corresponding interactive behavior is to compare the number of blinks in the eye movement features with the number of blinks in the eye movement database, and select the interactive behavior corresponding to the eye movement data with the same number of blinks to interact with the virtual object; The number of times the eyes are closed is determined by filtering images of the eyeball that do not show pupils, obtaining the corresponding timestamps, and calculating the timestamp difference between the image without pupils and the next image that shows pupils. Call the threshold set in the database ,when The system determines when a user closes their eyes and records the number of times they close their eyes, then inputs the recorded number of eye closures into the eye movement characteristics database. In the middle, eye-tracking data is called from the eye-tracking database. The system interacts with virtual objects by comparing the number of times the eyes close in the eye movement features with the number of times the eyes close in the eye movement database, and selecting the interaction behavior corresponding to the eye movement data with the same number of times the eyes close.

7. A human-computer interaction system based on virtual reality according to claim 6, characterized in that: Step S3 further includes the following steps: Step S31: Retrieve central rate data from the user database blinking frequency runtime Pupil constriction data Calculate the user's fatigue index ,in, , , , The weight parameters set in the database are used to call the thresholds set in the database. ,when When the user's physical condition is determined to be normal, When the system determines that the user is in a state of fatigue, it will notify the user that the user is in a state of fatigue and collect the timestamps corresponding to the fatigue state and store the corresponding timestamps in the database. Step S32: Call the timestamp data of historical fatigue states in the database, retrieve the corresponding model algorithm from the database, use the timestamp data of historical fatigue states as sample data to train the model, obtain the fatigue time prediction model, predict the time period when the user is likely to enter a fatigue state, and when the user approaches the predicted fatigue time period, the system will remind the user a period of time in advance to avoid excessive fatigue.

Images