Virtual reality picture lag detection method and device and computer equipment

Abstract

The invention relates to a virtual reality picture lag detection method and device, computer equipment, a storage medium and a computer program product. The method can be applied to the field of artificial intelligence, for example, the method is applied to a scene in which an intelligent VR device detects a lagging state of a displayed virtual reality picture; the method comprises the following steps: acquiring eye movement data of a target object in a process of watching a displayed virtual reality picture; extracting fixation point change features and pupil change features of the target object from the eye movement data; determining an interaction state between the target object and each displayed virtual reality picture according to the fixation point change feature and the pupil change feature; and determining a lagging state of the virtual reality picture according to the interaction state. By adopting the method, the accuracy of lagging detection on the virtual reality picture can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, computer equipment, storage medium and computer program product for detecting freezes in virtual reality images. Background Art

[0002] Virtual Reality (VR) uses computer technology to simulate the real world to create a virtual world, allowing users to immerse themselves in the real world. Virtual reality technology is widely used in games, online office, audio and video entertainment and other fields.

[0003] In the related technology, whether the VR displayed picture is smooth is determined by detecting the system performance during VR operation. For example, whether the VR displayed picture is smooth is determined by the network delay. The factors that affect the smoothness of the VR displayed picture are relatively complex. This method is prone to judging that the VR displayed picture is not stuck based on the network delay, while in fact the VR displayed picture is stuck, and the accuracy is low. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for detecting freezes in virtual reality images in response to the above-mentioned technical problems, which can improve the accuracy of freeze detection in virtual reality images.

[0005] In a first aspect, the present application provides a method for detecting freezes in a virtual reality screen. The method comprises:

[0006] Obtain eye movement data of a target object when viewing a displayed virtual reality screen; extract gaze point change characteristics and pupil change characteristics of the target object from the eye movement data; determine the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and pupil change characteristics; determine the freeze state of the virtual reality screen according to the interaction state.

[0007] In a second aspect, the present application also provides a device for detecting freezes in a virtual reality screen. The device comprises:

[0008] An eye movement data acquisition module is used to acquire the eye movement data of the target object when viewing the displayed virtual reality picture;

[0009] An eye movement feature extraction module is used to extract the gaze point change features and pupil change features of the target object from the eye movement data;

[0010] An interaction state determination module, used to determine the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and pupil change characteristics;

[0011] The freeze state determination module is used to determine the freeze state of the virtual reality screen according to the interaction state.

[0012] In some embodiments, the freeze detection device for virtual reality screen can be applied to virtual reality games, and the freeze detection device for virtual reality screen also includes: a virtual reality screen display module, which is used to send a first data acquisition request to a server based on the progress information of the virtual reality game; receive game screen data returned by the server in response to the first data acquisition request; and display the virtual reality screen of the virtual reality game based on the game screen data.

[0013] In some embodiments, the freeze detection device for a virtual reality screen also includes: a first receiving unit, used to receive reference eye movement features and screen categories sent by a server; the reference eye movement features are eye movement features predicted based on a virtual reality screen without freezes; accordingly, an eye movement feature extraction module, used to extract the gaze point change features and pupil change features of the target object from the eye movement data when the screen category includes a screen element change category and a brightness change category; accordingly, an interaction state determination module, used to determine the interaction state between the target object and each displayed virtual reality screen based on the reference eye movement features, the gaze point change features and the pupil change features.

[0014] In some embodiments, the reference eye movement feature includes a first reference eye movement feature and a second reference eye movement feature, the first reference eye movement feature is used to characterize the change of the gaze point, and the second reference eye movement feature is used to characterize the change of the pupil; the interaction state determination module is further used to determine a first similarity between the first reference eye movement feature and the gaze point change feature; determine a second similarity between the second reference eye movement feature and the pupil change feature; and determine the interaction state between the target object and each displayed virtual reality screen based on the first similarity and the second similarity.

[0015] In some embodiments, the freeze state includes that the virtual reality screen is in a freeze state; the interaction state determination module includes: a first interaction state determination unit, used to perform linear processing on the first similarity and the second similarity to obtain a target similarity; when the target similarity meets the abnormal interaction condition, it is determined that the target object and the displayed virtual reality screens are in an abnormal interaction state; accordingly, the freeze state determination module is used to determine that the virtual reality screen is in a freeze state according to the abnormal interaction state.

[0016] In some embodiments, the freeze state includes that the virtual reality screen is not in a freeze state; the interaction state determination module includes a second interaction state determination unit, which is used to perform linear processing on the first similarity and the second similarity to obtain a target similarity; when the target similarity does not meet the abnormal interaction condition, it is determined that the target object and the displayed virtual reality screens are in a normal interaction state; accordingly, the freeze state determination module is used to determine that the virtual reality screen is not in a freeze state according to the normal interaction state.

[0017] In some embodiments, the freeze detection device for a virtual reality screen further includes: a reference eye movement feature acquisition module, configured to send a second data acquisition request to a server; the second data acquisition request is configured to instruct the server to predict a reference eye movement feature corresponding to a virtual reality screen without freeze based on a screen category of the virtual reality screen; and receive the screen category and the reference eye movement feature sent by the server;

[0018] Accordingly, the eye movement feature extraction module is used to determine the gaze point change feature of the target object according to the eye movement data when the picture category includes the picture element change category; the reference eye movement feature includes a first reference eye movement feature for characterizing the gaze point change;

[0019] Correspondingly, the interaction state determination module is used to determine the interaction state between the target object and each displayed virtual reality screen according to the first reference eye movement feature and the determined gaze point change feature.

[0020] In some embodiments, predicting a reference eye movement feature corresponding to a virtual reality picture without any freeze based on the picture category of the virtual reality picture includes: obtaining the picture category corresponding to the virtual reality picture; when the picture category is a picture element change category, extracting eye movement features of the virtual reality picture without any freeze through a first eye movement model to obtain a first reference eye movement feature.

[0021] In some embodiments, the eye movement feature extraction module is further used to determine pupil change features of the target object according to the eye movement data when the picture category includes a brightness change category; the reference eye movement feature includes a second reference eye movement feature for characterizing pupil changes;

[0022] Correspondingly, the interaction state determination module is used to determine the interaction state between the target object and each displayed virtual reality screen according to the second reference eye movement feature and the determined pupil change feature.

[0023] In some embodiments, predicting reference eye movement features corresponding to a virtual reality picture without any freeze based on the picture category of the virtual reality picture includes: obtaining the picture category corresponding to the virtual reality picture; when the picture category is a brightness change category, extracting eye movement features of the virtual reality picture without any freeze through a second eye movement model to obtain a second reference eye movement feature for characterizing pupil changes.

[0024] In some embodiments, the freeze detection device for a virtual reality screen also includes: an identity information determination module, which is used to periodically obtain an iris image of the target object; extract iris features of the iris image, and classify and process the iris features to obtain the identity information of the target object; accordingly, an eye movement data acquisition module 1501, which is used to obtain the eye movement data of the target object when the identity information meets the identity determination conditions.

[0025] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0026] Obtain eye movement data of a target object when viewing a displayed virtual reality screen; extract gaze point change characteristics and pupil change characteristics of the target object from the eye movement data; determine the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and pupil change characteristics; determine the freeze state of the virtual reality screen according to the interaction state.

[0027] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0028] Obtain eye movement data of a target object when viewing a displayed virtual reality screen; extract gaze point change characteristics and pupil change characteristics of the target object from the eye movement data; determine the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and pupil change characteristics; determine the freeze state of the virtual reality screen according to the interaction state.

[0029] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0030] Obtain eye movement data of a target object when viewing a displayed virtual reality screen; extract gaze point change characteristics and pupil change characteristics of the target object from the eye movement data; determine the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and pupil change characteristics; determine the freeze state of the virtual reality screen according to the interaction state.

[0031] The above-mentioned virtual reality screen freeze detection method, device, computer equipment, storage medium and computer program product obtain the eye movement data of the target object when viewing the displayed virtual reality screen, extract the gaze point change characteristics and pupil change characteristics from the eye movement data, determine the interaction state between the target object and the displayed virtual reality screen according to the gaze point change characteristics and pupil change characteristics, and determine the freeze state of the virtual reality screen according to the interaction state; based on the real eye movement behavior of the target object viewing the displayed virtual reality screen, determine whether the virtual reality screen has freezes, so that the freeze state of the virtual reality screen is related to the real eye movement behavior of the target object viewing the displayed virtual reality screen, and then the existence of freezes in the virtual reality screen is freezes that can be actually observed by the target object, thereby improving the accuracy of freeze detection of the virtual reality screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A diagram showing an application environment of a method for detecting freezes in a virtual reality screen in an embodiment;

[0033] Figure 2 1 is a flow chart of a method for detecting freezes in a virtual reality screen in one embodiment;

[0034] Figure 3 A schematic diagram of a process of determining identity information based on an iris image in one embodiment;

[0035] Figure 4 A schematic diagram of a process of interacting between a virtual reality device and a server to display a virtual reality screen of a virtual reality game in an embodiment;

[0036] Figure 5 A schematic diagram of a flow chart of determining a freeze state of a virtual reality screen when the screen category includes a screen element change category and a brightness change category in an embodiment;

[0037] Figure 6 A flowchart of determining a stuck state according to whether target similarity satisfies an abnormal interaction condition in an embodiment;

[0038] Figure 7 A schematic diagram of a flow chart of determining a freeze state of a virtual reality screen when the screen category includes a screen element change category in an embodiment;

[0039] Figure 8 is a schematic diagram of a picture category classification model and a first eye movement model in one embodiment;

[0040] Fig. 9 is a schematic diagram of extracting a first reference eye movement feature in an embodiment;

[0041] Fig.10 A schematic diagram of a flow chart of determining a freeze state of a virtual reality picture when the picture category includes a brightness change category in an embodiment;

[0042] Fig.11 is a schematic diagram of extracting a second reference eye movement feature in one embodiment;

[0043] Fig.12 A schematic diagram of the structure of a VR device in one embodiment;

[0044] Fig.13 1 is a schematic diagram of a method for detecting freezes in a virtual reality image applied to a virtual reality video in an embodiment;

[0045] Fig.14 A schematic diagram of a flow chart of a method for detecting freezes in a virtual reality screen in another embodiment;

[0046] Fig.15 It is a structural block diagram of a freeze detection device for a virtual reality screen in one embodiment;

[0047] Fig.16 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0050] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0051] The method for detecting freeze in a virtual reality screen provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The freeze detection method of the virtual reality screen can be applied to the terminal 102, or it can be applied to the system composed of the terminal 102 and the server 104.

[0052] Taking the application of the freeze detection method of the virtual reality screen to the terminal 102 as an example; the terminal 102 obtains the eye movement data of the target object when watching the displayed virtual reality screen; the terminal 102 extracts the gaze point change characteristics and pupil change characteristics of the target object from the eye movement data; the terminal 102 determines the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and pupil change characteristics; the terminal 102 determines the freeze state of the virtual reality screen according to the interaction state.

[0053] Among them, the terminal 102 can be a virtual reality device, such as virtual reality glasses and virtual reality helmets; it can also be a portable wearable device, and the portable wearable device can be a smart head-mounted device, etc.

[0054] The server 104 may be an independent physical server or a service node in a blockchain system. The service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP).

[0055] In addition, server 104 can also be a server cluster composed of multiple physical servers, and can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0056] The terminal 102 and the server 104 may be connected via Bluetooth, USB (Universal Serial Bus) or a network or other communication connection methods, which are not limited in this application.

[0057] In some embodiments, Figure 2 As shown, a method for detecting freeze in a virtual reality screen is provided. Figure 1 , or by Figure 1 The server and terminal in the method are executed in cooperation with each other. Figure 1 The following steps are used as an example to illustrate the terminal execution in:

[0058] Step 202: Acquire eye movement data of the target object when viewing the displayed virtual reality image.

[0059] The target object is an object viewing the virtual reality screen. In practical applications, the target object may be a user. The virtual reality screen is played on a terminal, and the terminal is a virtual reality device for example; the virtual reality screen is played through the virtual reality device, for example, the user wears the virtual reality device, the virtual reality device plays the virtual reality screen, and the user can watch the virtual reality screen.

[0060] In some embodiments, there are multiple virtual reality screens, and the virtual reality screen can be understood as a screen segment composed of multiple virtual reality screens that are continuous at the time of playback. In actual applications, the virtual reality device can obtain screen data of the first virtual reality screen segment from the server, load the first virtual reality screen segment based on the screen data, and obtain screen data of the second virtual reality screen segment, so that when the first virtual reality screen segment is played, the second virtual reality screen segment is played next.

[0061] Among them, eye movement data is the data generated by detecting the target object's eye movement and pupil changes when the target object is watching the displayed virtual reality screen.

[0062] In some embodiments, the eye movement data includes but is not limited to a gaze sequence and a pupil diameter sequence; the gaze sequence can be used to analyze the focus direction of the target object's eyes when viewing the virtual reality screen. The pupil diameter sequence can be used to analyze the changes in pupil diameter of the target object's eyes when viewing the virtual reality screen.

[0063] A gaze sequence includes multiple gaze points and the gaze durations corresponding to the multiple gaze points; a gaze point is where the line of sight falls, which can indicate the position where the eyes are looking, and the gaze point can be represented by pixel coordinates in a virtual reality screen; the duration is the length of time the line of sight stays at the gaze point; multiple gaze points are arranged in chronological order, for example, in a gaze sequence, the gaze point at time t is before the gaze point at time t+1.

[0064] The pupil diameter sequence includes multiple pupil diameters, wherein the multiple pupil diameters are arranged in chronological order. For example, in the pupil diameter sequence, the pupil diameter at the tth moment is before the pupil diameter at the t+1th moment.

[0065] In some embodiments, taking a virtual reality device as an example, the virtual reality device is configured with an eye movement data acquisition unit, the eye movement data acquisition unit may be an eye tracker, and the virtual reality device may acquire eye movement data through the eye tracker.

[0066] In some embodiments, when the target object views a virtual reality screen through a virtual reality device, the virtual reality device obtains the target object's eye movement trajectory and pupil diameter sequence through an eye tracker, determines the gaze point and the gaze duration corresponding to the gaze point through the eye movement trajectory, and then determines the gaze sequence based on the gaze point and the corresponding gaze duration.

[0067] Step 204: extracting the target object's gaze point change characteristics and pupil change characteristics from the eye movement data.

[0068] Among them, the gaze point change feature can be used to reflect the change of the sight point, and the gaze point change feature can be represented in the form of a vector; the pupil change feature can be used to reflect the change of the pupil diameter, and the pupil diameter feature can also be represented in the form of a vector.

[0069] In some embodiments, the virtual reality device can input eye movement data into a first feature extraction model, output gaze point change features through the first feature extraction model, input eye movement data into a second feature extraction model, and output pupil change features through the second feature extraction model.

[0070] Specifically, the eye movement data includes a gaze sequence and a pupil diameter sequence. The virtual reality device can input the gaze sequence into a first feature extraction model to obtain a gaze point change feature; and input the pupil diameter sequence into a second feature extraction model to obtain a pupil change feature.

[0071] In some embodiments, the acquired eye movement data may contain noise. The virtual reality device can denoise the eye movement data to improve the quality of the denoised eye movement data, input the denoised eye movement data into a first feature extraction model to obtain a gaze point change feature, and input the denoised eye movement data into a second feature extraction model to obtain a pupil change feature.

[0072] Step 206: Determine the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and the pupil change characteristics.

[0073] Among them, the interaction state can be a normal interaction state or an abnormal interaction state. The normal interaction state can indicate that the target object is immersed in interacting with the virtual reality screen, and the abnormal interaction state can indicate that the target object is not immersed in interacting with the virtual reality screen. When the displayed virtual reality screen is stuck, it may cause the target object to be unable to immerse itself in interacting with the virtual reality screen.

[0074] In some embodiments, when the target object is immersed in the state of interacting with the virtual reality screen, the change of the gaze point is regular, for example, when the virtual character in the virtual reality screen is controlled to move, the gaze point follows the movement of the character. When the target object is not immersed in the state of interacting with the virtual reality screen, the change of the gaze point is disordered, for example, when the virtual character in the virtual reality screen is controlled to move, the virtual reality screen freezes, the target object will observe other positions of the virtual reality screen, and the gaze point will not always stay on the virtual character.

[0075] For example, the virtual character is moved from the left side to the right side of the virtual reality screen. In a normal interaction state, the position of the gaze point changes along the movement trajectory of the virtual character. In an abnormal interaction state, the position change of the gaze point may show irregular jumps.

[0076] In some embodiments, when the target object is immersed in the state of interacting with the virtual reality screen, the pupil changes in a regular manner. For example, when the target object focuses on the virtual reality screen, the pupil will expand and shrink within a certain range, that is, the pupil diameter changes within a certain range; when the target object is not immersed in the state of interacting with the virtual reality screen, the line of sight is scattered, the pupil diameter will increase, and exceed the normal range of change.

[0077] In some embodiments, the virtual reality device can input gaze point change features and pupil change features into an interactive focus prediction model, and output a focus prediction result used to characterize whether the target object is focused on the virtual reality screen through the interactive focus prediction model; when the focus prediction result is a first focus result, it means that the target object is focused on the virtual reality screen, and the interaction state is a normal interaction state; when the focus prediction result is a second focus result, it means that the target object is not focused on the virtual reality screen, and the interaction state is an abnormal interaction state.

[0078] In some embodiments, the virtual reality device obtains a reference eye movement feature. The reference eye movement feature may be an eye movement feature corresponding to a virtual reality screen without lag. The reference eye movement feature may include a first reference eye movement feature for characterizing changes in gaze points and a second reference eye movement feature for characterizing changes in pupil diameter.

[0079] The virtual reality device can compare and obtain a first gap between a first reference eye movement feature and a gaze point change feature, and compare and obtain a second gap between a second reference eye movement feature and a pupil change feature, and determine a target gap based on the first gap and the second gap, where the target gap is used to reflect the gap between eye movement data of viewing a virtual reality screen and eye movement data of viewing a virtual reality screen without lag; when the target gap falls within a preset range, it is determined that the interaction state between the target object and the displayed virtual reality screen is a normal interaction state, and when the target gap does not fall within the preset range, it is determined that the interaction state between the target object and the displayed virtual reality screen is an abnormal interaction state.

[0080] Step 208: Determine the freeze state of the virtual reality screen according to the interaction state.

[0081] The freeze state of the virtual reality screen includes that the virtual reality screen is in a freeze state, or that the virtual reality screen is not in a freeze state.

[0082] In some embodiments, when the interaction state is a normal interaction state, the virtual reality device determines the freeze state of the virtual reality screen as follows: the virtual reality screen is not in a freeze state; when the interaction state is an abnormal interaction state, the virtual reality device determines the freeze state of the virtual reality screen as follows: the virtual reality screen is in a freeze state.

[0083] In some embodiments, the freeze state of the virtual reality screen is: when the virtual reality screen is in a freeze state, it may be caused by poor network quality. When it is determined that the virtual reality screen is in a freeze state, the virtual reality device can reduce the picture quality of the virtual reality screen. Further, the picture quality of the virtual reality screen can be reduced by reducing the picture resolution to reduce the amount of data required for network transmission, thereby improving the freeze state of the virtual reality screen.

[0084] In the above-mentioned method for detecting freeze in a virtual reality screen, eye movement data of a target object in the process of viewing a displayed virtual reality screen is obtained, gaze point change features and pupil change features are extracted from the eye movement data, and the interaction state between the target object and the displayed virtual reality screen is determined based on the gaze point change features and the pupil change features, and the freeze state of the virtual reality screen is determined based on the interaction state; based on the real eye movement behavior of the target object viewing the displayed virtual reality screen, it is determined whether the virtual reality screen has freeze, so that the freeze state of the virtual reality screen is related to the real eye movement behavior of the target object viewing the displayed virtual reality screen, and thus the freeze in the virtual reality screen is a freeze that can be actually observed by the target object, thereby improving the accuracy of freeze detection for the virtual reality screen.

[0085] In some embodiments, the method for detecting freezes in a virtual reality screen also includes: regularly acquiring an iris image of a target object; extracting iris features of the iris image, and classifying and processing the iris features to obtain identity information of the target object; acquiring eye movement data of the target object while viewing the displayed virtual reality screen, including: when the identity information meets an identity determination condition, acquiring eye movement data of the target object while viewing the displayed virtual reality screen.

[0086] The iris image is obtained by photographing the eyes of the target object, and the iris image can be photographed by a camera. In practical applications, the virtual reality device is equipped with a camera for obtaining the iris image. The virtual reality device can also be equipped with an iris meter to obtain the iris image.

[0087] In some embodiments, the virtual reality device extracts features from the iris image through a feature extraction module to obtain iris features, and classifies the iris features through a classification model to obtain identity information of the target object.

[0088] In some embodiments, the virtual reality device can perform image enhancement processing on the iris image to improve the image quality of the enhanced iris image; in actual applications, the virtual reality device can perform image enhancement processing on the iris image through adaptive contrast enhancement, adaptive image equalization, etc.

[0089] Virtual reality equipment can segment the enhanced iris image and remove noise and non-iris parts through segmentation. In practical applications, the enhanced iris image can be dilated, eroded and other operations can be performed through the segmentation method of morphological transformation to obtain a binary image corresponding to the enhanced iris image.

[0090] The virtual reality device extracts features from the binary image through a feature extraction network to obtain iris features, and classifies the iris features through a classification network to obtain identity information of the target object. Specifically, the classification network classifies the iris features to obtain probability values ​​of the target object belonging to each preset identity information, and uses the preset identity information corresponding to the maximum probability value among the probability values ​​as the identity information of the target object; wherein the preset identity information may be the identity information of each object that has viewed the virtual reality screen using the virtual reality device.

[0091] For example, Figure 3 As shown, the virtual reality device periodically obtains the iris image of the target object, processes the iris image to obtain a binary image, inputs the binary image into a feature extraction network to obtain iris features, and inputs the iris features into a classification network to obtain the identity information of the target object.

[0092] In some embodiments, since the virtual reality device periodically acquires iris images, the virtual reality device can determine whether the identity information meets the identity determination condition based on the currently determined identity information and the last determined identity information. The identity determination condition can be that the currently determined identity information is the same as the last determined identity information.

[0093] When the currently determined identity information is the same as the last determined identity information, it is determined that the identity information meets the identity determination condition; it should be noted that the identity information determined each time is the same as the last determined identity information, that is, it is always the same object during the process of viewing the displayed virtual reality screen. When the identity information meets the identity determination condition, the freeze detection can be continued to obtain the eye movement data of the target object during the process of viewing the displayed virtual reality screen.

[0094] When the currently determined identity information is different from the last determined identity information, it is determined that the identity information does not meet the identity determination condition; when the identity information is different, it means that the target object may have been changed during the process of viewing the displayed virtual reality screen. When the identity information does not meet the identity determination condition, the freeze detection can be stopped.

[0095] In some embodiments, after periodically acquiring an iris image of the target object and extracting iris features of the iris image, a method for detecting freezes in a virtual reality screen may include: determining, based on the iris features and reference iris features, whether the identity information of the target object meets an identity determination condition; and when the identity information of the target object meets the identity determination condition, acquiring eye movement data of the target object while viewing the displayed virtual reality screen.

[0096] Among them, the reference identity information can be determined when the target object views the screen for the first time through the virtual reality device. For example, when the target object plays a virtual reality game through the virtual reality device and logs into the virtual reality game through the target account, the virtual reality device obtains the initial iris image of the target object, and obtains the reference iris features based on the initial iris image recognition. The virtual reality device binds the reference iris features to the target account; when the target object logs into the virtual reality game again through the target account, the virtual reality device periodically obtains the iris image of the target object and extracts the iris features, and obtains the reference iris features corresponding to the target account.

[0097] According to the iris feature and the reference iris feature, it is determined that the identity information of the target object meets the identity determination condition, which can be that the similarity between the iris feature and the reference iris feature is greater than a preset threshold. When the similarity between the iris feature and the reference iris feature is greater than the preset threshold, it means that the target object is consistent with the object corresponding to the reference iris feature. The preset threshold can be set according to actual needs, and the preset threshold can be 80%. The specific value of the preset threshold is not limited in the embodiment of the present application.

[0098] In the above embodiment, identity information is realized through the iris image of the target object, and when it is determined that the identity information meets the identity determination conditions, the eye movement data of the target object is obtained, thereby ensuring the consistency of the identity of the target object during the stutter detection process, so that the stutter detection can be performed on the same object, and the eye movement data used for the stutter detection is of the same object, thereby improving the accuracy of the stutter detection.

[0099] In some embodiments, the method for detecting a freeze in a virtual reality screen is applied to a virtual reality game. Before obtaining the eye movement data of a target object while viewing a displayed virtual reality screen, the method for detecting a freeze in a virtual reality screen further includes: sending a first data acquisition request to a server based on progress information of the virtual reality game; receiving game screen data returned by the server in response to the first data acquisition request; and displaying the virtual reality screen of the virtual reality game based on the game screen data.

[0100] Among them, the progress information includes the current time of the virtual reality game, and may also include the currently triggered operation instructions, the operation instructions include but are not limited to: launch instructions, equipment pickup instructions and movement instructions, the movement instructions may be left movement instructions, right movement instructions, jump instructions, etc.

[0101] The first data acquisition request includes progress information; the game screen data is data used to display the virtual reality screen.

[0102] In some embodiments, Figure 4As shown, the virtual reality device periodically obtains the current time and operation instructions of the virtual reality game, generates a first data acquisition request according to the current time and operation instructions, sends the first data acquisition request to the server, the server receives the first data acquisition request, obtains the game screen data corresponding to the screen that the virtual reality game is about to display according to the current time and operation instructions in the first data acquisition request, the server sends the game screen data to the virtual reality device, and the virtual reality device displays the virtual reality screen of the virtual reality game according to the game screen data.

[0103] When the hardware performance and network environment of the virtual reality device are good, the virtual reality device can display a virtual reality picture without any lag based on the game screen data. When the hardware performance or network environment is poor, the virtual reality device may display a virtual reality picture without any lag based on the game screen data.

[0104] In some embodiments, the freeze detection method for virtual reality screens can also be applied to scenarios such as virtual reality videos and virtual reality concerts. Taking the freeze detection method for virtual reality screens applied to virtual reality videos as an example, in a scenario where teaching is achieved through virtual reality videos, the progress information may be the moment of the currently displayed teaching screen. The virtual reality device generates a first data acquisition request based on the moment of the currently displayed teaching screen, and sends the first data acquisition request to the server. The server acquires video screen data based on the first data acquisition request, and sends the video screen data to the virtual reality device. The virtual reality device displays the virtual reality screen of the virtual reality video based on the video screen data.

[0105] In the above embodiment, the method for detecting freeze of a virtual reality screen can be applied to a virtual reality game, where game screen data is obtained based on the longitude information of the virtual reality game, and the virtual reality screen is displayed based on the game screen data, so as to subsequently obtain eye movement data of the target object watching the virtual reality screen displayed in the virtual reality game, and perform freeze detection on the virtual reality game.

[0106] In some embodiments, the method for detecting freezes in a virtual reality screen also includes: receiving reference eye movement features and screen categories sent by a server; the reference eye movement features are eye movement features predicted based on a virtual reality screen without freezes; extracting gaze point change features and pupil change features of the target object from the eye movement data, including: when the screen category includes a screen element change category and a brightness change category, extracting gaze point change features and pupil change features of the target object from the eye movement data; determining the interaction state between the target object and each displayed virtual reality screen based on the gaze point change features and pupil change features, including: determining the interaction state between the target object and each displayed virtual reality screen based on the reference eye movement features, the gaze point change features and the pupil change features.

[0107] The picture category is a change category of the virtual reality picture, which is used to reflect the prominent changes of the virtual reality picture; the picture category can be a picture element change category or a brightness change category.

[0108] The screen element change category is used to reflect the changes in the position movement of key elements in the virtual reality screen. The key element can be the element that is most concerned by the target object in the virtual reality screen. For example, in a virtual reality game, the key element can be the virtual character controlled by the target object. In a virtual reality concert, the key element can be the performer in the virtual reality screen.

[0109] The brightness change category is used to reflect changes in brightness in a virtual reality screen. For example, in a virtual reality game, a virtual character moves from a darker environment to a brighter environment, or a combustion or explosion occurs in a virtual reality game, causing the brightness of the virtual reality screen to change.

[0110] The reference eye movement features are the eye movement features predicted by the server for a virtual reality picture without any lag. The virtual reality picture without any lag can be determined by the server based on the picture data; for example, in a virtual reality game, the picture data is the game picture data, when applied to a virtual reality video, the picture data is the video picture data, and when applied to a virtual reality concert, the picture data is the concert picture data.

[0111] For example, Figure 5 As shown, the method for detecting freeze in a virtual reality screen includes:

[0112] Step 501, obtaining the eye movement data of the target object when watching the displayed virtual reality picture; Step 502, receiving the reference eye movement features and picture categories sent by the server; the reference eye movement features are eye movement features predicted based on the virtual reality picture without freeze; Step 503, when the picture category includes the picture element change category and the brightness change category, extracting the gaze point change features and pupil change features of the target object from the eye movement data; Step 504, determining the interaction state between the target object and each displayed virtual reality picture based on the reference eye movement features, the gaze point change features and the pupil change features; Step 505, determining the freeze state of the virtual reality picture according to the interaction state.

[0113] In some embodiments, when the virtual reality device needs to display a virtual reality screen, it sends a data acquisition request to the server. The server obtains screen data according to the data acquisition request. The server obtains a virtual reality screen without any lag based on the screen data, identifies the virtual reality screen without any lag, and obtains the screen category of the virtual reality screen without any lag.

[0114] The picture category includes a picture element change category and a brightness change category, which means that the virtual reality picture includes changes in the position movement of key elements and changes in brightness; when the picture category includes a picture element change category and a brightness change category, the server inputs the virtual reality picture without any lag into the eye movement model, and obtains the reference eye movement features through the eye movement model prediction; the server sends the reference eye movement features and the picture category to the virtual reality device.

[0115] When the image categories acquired by the virtual reality device include the image element change categories and the brightness change categories, the virtual reality device can input the eye movement data into the first feature extraction model, output the gaze point change features through the first feature extraction model, input the eye movement data into the second feature extraction model, and output the pupil change features through the second feature extraction model. The interaction state between the target object and each displayed virtual reality image is determined based on the reference eye movement features, the gaze point change features, and the pupil change features.

[0116] In some embodiments, the eye movement model is obtained by training the initial eye movement model; the initial eye movement model includes an initial image feature extraction network and an initial eye movement feature prediction network, the parameters of the initial image feature extraction network are updated through the first data set to obtain the image feature extraction network, the initial eye movement feature prediction network is connected after the image feature extraction network, the parameters of the image feature extraction network are fixed, the parameters of the initial eye movement feature prediction network are updated according to the second data set to obtain the eye movement feature prediction network, and the eye movement model is obtained according to the image feature extraction network and the eye movement feature prediction network.

[0117] The first data set includes a training picture without any freeze, and an image feature label and an eye movement feature label corresponding to the training picture. The training picture is input into the initial image feature extraction network to obtain a training feature image. The parameters of the initial image feature extraction network are updated according to the training feature image and the image feature label until the initial image feature extraction network converges, and the converged initial image feature extraction network is used as the image feature extraction network. It should be noted that the training picture includes multiple training picture frames, and the training feature image includes training feature images corresponding to each of the multiple training picture frames, and the image feature label also includes image feature labels corresponding to each of the multiple training picture frames.

[0118] The initial eye movement feature prediction network is connected after the image feature extraction network; the training picture is input into the image feature extraction network, the training eye movement features are output through the initial eye movement feature prediction network, the parameters of the image feature extraction network are fixed, the parameters of the initial eye movement feature prediction network are updated through the training eye movement features and the eye movement feature labels until the initial eye movement feature prediction network converges, and the eye movement model is obtained according to the image feature extraction network and the converged initial eye movement feature prediction network.

[0119] After the image feature extraction network is trained, the parameters of the image feature extraction network are fixed and the parameters of the initial eye movement feature prediction network are adjusted, thereby improving the robustness and accuracy of the trained eye movement model, so that the eye movement model can be applied to the reference eye movement feature prediction of virtual reality images with complex elements.

[0120] In the above embodiment, reference eye movement features are predicted based on a virtual reality picture without any freeze, and when the picture category includes a picture element category and a brightness change category, gaze point change features and pupil change features are extracted from the eye movement data, and the interaction state is determined based on the reference eye movement features, gaze point change features and pupil change features, so that the interaction state is determined based on the reference eye movement features corresponding to the virtual picture without any freeze, and the eye movement features corresponding to the virtual reality picture displayed by the target object, thereby improving the accuracy of the interaction state and thereby improving the accuracy of freeze detection.

[0121] In some embodiments, the reference eye movement feature includes a first reference eye movement feature and a second reference eye movement feature, the first reference eye movement feature is used to characterize the change of the gaze point, and the second reference eye movement feature is used to characterize the change of the pupil; based on the reference eye movement feature, the gaze point change feature and the pupil change feature, determining the interaction state between the target object and each displayed virtual reality screen, including: determining a first similarity between the first reference eye movement feature and the gaze point change feature; determining a second similarity between the second reference eye movement feature and the pupil change feature; determining the interaction state between the target object and each displayed virtual reality screen based on the first similarity and the second similarity.

[0122] The first reference eye movement feature is obtained by predicting the change of the gaze point based on a virtual reality picture without any freeze, and the second reference eye movement feature is obtained by predicting the change of the pupil based on a virtual reality picture without any freeze.

[0123] Among them, the first similarity is used to characterize the similarity between the first reference eye movement feature and the gaze point change feature, and the second similarity is used to characterize the similarity between the second reference eye movement feature and the pupil change feature; since the first reference eye movement feature and the second reference eye movement feature are determined based on a virtual reality picture without any lag, the higher the similarity between the first reference eye movement feature and the gaze point change feature, and the higher the similarity between the second reference eye movement feature and the pupil change feature, the closer the viewed virtual reality picture is to the virtual reality picture without any lag.

[0124] In some embodiments, the server inputs a virtual reality picture without any lag into an eye movement model, predicts a first reference eye movement feature and a second reference eye movement feature through the eye movement model, and sends the first reference feature and the second reference feature to the virtual reality device.

[0125] The virtual reality device can calculate the cosine distance between the first reference eye movement feature and the gaze point change feature to obtain a first similarity between the first reference eye movement feature and the gaze point change feature, and can calculate the cosine distance between the second reference eye movement feature and the pupil change feature to obtain a second similarity between the second reference eye movement feature and the pupil change feature.

[0126] In some embodiments, the virtual reality device determines the interaction state between the target object and each displayed virtual reality screen based on the first similarity and the second similarity, and when the first similarity or the second similarity meets the abnormal interaction condition, the interaction state is determined to be an abnormal interaction state, when the first similarity or the second similarity meets the abnormal interaction condition, when the first similarity or the second similarity does not meet the abnormal interaction condition, the interaction state is determined to be a normal interaction state. In this embodiment, the abnormal interaction condition may be that the first similarity or the second similarity is less than a preset similarity, and the preset similarity can be obtained according to actual conditions. The embodiment of the present application does not limit the specific value of the preset similarity.

[0127] In some embodiments, the virtual reality device determines the interaction state between the target object and each displayed virtual reality screen based on the first similarity and the second similarity, and the target similarity can be determined based on the first similarity and the second similarity. When the target similarity meets the abnormal interaction condition, the interaction state is determined to be an abnormal interaction state, and when the target similarity does not meet the abnormal interaction condition, the interaction state is determined to be a normal interaction state. In this embodiment, the abnormal interaction condition can be that the target similarity is less than the preset similarity. The preset similarity can be obtained according to the actual situation. The embodiment of the present application does not limit the specific value of the preset similarity.

[0128] In the above embodiment, when the picture category includes a picture element change category and a brightness change category, the first similarity between the first reference eye movement feature and the gaze point change feature, and the second similarity between the second reference eye movement feature and the pupil change feature are combined to determine the interaction state between the target object and each displayed virtual reality picture, thereby improving the accuracy of the interaction state and further improving the accuracy of freeze detection.

[0129] In some embodiments, the freeze state includes a virtual reality screen being in a freeze state; determining the interaction state between the target object and each displayed virtual reality screen based on the first similarity and the second similarity, including: linearly processing the first similarity and the second similarity to obtain the target similarity; when the target similarity meets the abnormal interaction condition, determining that the target object and each displayed virtual reality screen are in an abnormal interaction state; determining the freeze state of the virtual reality screen based on the interaction state, including: determining that the virtual reality screen is in a freeze state based on the abnormal interaction state.

[0130] The abnormal interaction condition may be that the target similarity is less than a preset similarity, and the preset similarity may be set according to actual needs, for example, the preset similarity may be 80%.

[0131] In some embodiments, the virtual reality device performs linear processing on the first similarity and the second similarity to obtain a target similarity. The target similarity may be obtained by calculating an average of the first similarity and the second similarity; or the target similarity may be obtained by weighted summing the first similarity and the second similarity based on a first weight for changes in gaze point and a second weight for changes in pupil. The first weight for changes in gaze point and the second weight for changes in pupil may be set according to actual needs. The embodiments of the present application do not limit the specific values ​​of the first weight and the second weight.

[0132] When the target similarity is less than the preset similarity, it is determined that the interaction state between the target object and the displayed virtual reality screens is an abnormal interaction state, and then it is determined that the virtual reality screen is in a stuck state; it should be noted that when the target similarity is less than the preset similarity, it means that the eye movement data of the target object in the process of watching the displayed virtual reality screen is significantly different from the eye movement data of watching the virtual reality screen without any stuck, and then it can be determined that the virtual reality screen is in a stuck state.

[0133] In the above embodiment, the first similarity and the second similarity are linearly processed to obtain the target similarity. When the target similarity meets the abnormal interaction condition, the interaction state is determined to be an abnormal interaction state, that is, the first similarity between the first reference eye movement feature and the gaze point change feature, and the second similarity between the second reference eye movement feature and the pupil change feature are combined to determine the interaction state, thereby improving the accuracy of the interaction state and thereby improving the accuracy of the freeze detection.

[0134] In some embodiments, the freeze state includes that the virtual reality screen is not in a freeze state; determining the interaction state between the target object and each displayed virtual reality screen based on the first similarity and the second similarity, including: linearly processing the first similarity and the second similarity to obtain the target similarity; when the target similarity does not meet the abnormal interaction condition, determining that the target object and each displayed virtual reality screen are in a normal interaction state; determining the freeze state of the virtual reality screen based on the interaction state, including: determining that the virtual reality screen is not in a freeze state based on the normal interaction state.

[0135] The target similarity does not satisfy the abnormal interaction condition, which may be that the target similarity is greater than or equal to a preset similarity.

[0136] In some embodiments, the virtual reality device performs linear processing on the first similarity and the second similarity to obtain a target similarity. The target similarity may be obtained by calculating an average of the first similarity and the second similarity; or the target similarity may be obtained by weighted summing the first similarity and the second similarity based on a first weight for changes in gaze point and a second weight for changes in pupil. The first weight for changes in gaze point and the second weight for changes in pupil may be set according to actual needs. The embodiments of the present application do not limit the specific values ​​of the first weight and the second weight.

[0137] When the target similarity is greater than or equal to the preset similarity, it is determined that the interaction state between the target object and the displayed virtual reality screens is a normal interaction state, and furthermore, it is determined that the virtual reality screen is not in a stuck state; it should be noted that when the target similarity is greater than or equal to the preset similarity, it means that the eye movement data of the target object during viewing the displayed virtual reality screen is less different from the eye movement data of the target object during viewing the virtual reality screen without any stuck state, and furthermore, it can be determined that the virtual reality screen is not in a stuck state.

[0138] For example, Figure 6 As shown, the method for detecting freeze in a virtual reality screen includes:

[0139] Step 601: Acquire eye movement data of a target object when viewing a displayed virtual reality image;

[0140] Step 602: receiving reference eye movement features and picture categories sent by the server; the reference eye movement features are eye movement features predicted based on a virtual reality picture without freezes;

[0141] Step 603: When the picture category includes the picture element change category and the brightness change category, extracting the gaze point change feature and pupil change feature of the target object from the eye movement data;

[0142] Step 604: determining a first similarity between the first reference eye movement feature and the gaze point change feature; determining a second similarity between the second reference eye movement feature and the pupil change feature; performing linear processing on the first similarity and the second similarity to obtain a target similarity;

[0143] Step 605A: when the target similarity satisfies the abnormal interaction condition, it is determined that the target object and each displayed virtual reality screen are in an abnormal interaction state; and according to the abnormal interaction state, it is determined that the virtual reality screen is in a stuck state;

[0144] Step 605B: when the target similarity does not meet the abnormal interaction condition, it is determined that the target object and each displayed virtual reality screen are in a normal interaction state; based on the normal interaction state, it is determined that the virtual reality screen is not in a stuck state.

[0145] In the above embodiment, the first similarity and the second similarity are linearly processed to obtain the target similarity. When the target similarity does not meet the abnormal interaction condition, the interaction state is determined to be a normal interaction state, that is, the first similarity between the first reference eye movement feature and the gaze point change feature, and the second similarity between the second reference eye movement feature and the pupil change feature are combined to determine the interaction state, thereby improving the accuracy of the interaction state and thereby improving the accuracy of the freeze detection.

[0146] In some embodiments, before obtaining the eye movement data of the target object when viewing the displayed virtual reality picture, the method for detecting freezes in the virtual reality picture further includes: sending a second data acquisition request to the server; the second data acquisition request is used to instruct the server to predict reference eye movement features corresponding to virtual reality pictures without freezes based on the picture category of the virtual reality picture; after obtaining the eye movement data of the target object when viewing the displayed virtual reality picture, the method for detecting freezes in the virtual reality picture further includes: receiving the picture category and reference eye movement features sent by the server; when the picture category includes a picture element change category, determining the gaze point change features of the target object according to the eye movement data; the reference eye movement features include a first reference eye movement feature for characterizing the gaze point change; and determining the interaction state between the target object and each displayed virtual reality picture based on the first reference eye movement feature and the determined gaze point change feature.

[0147] The second data acquisition request is used to instruct the server to identify the picture category and extract reference eye movement features according to the picture category.

[0148] In some embodiments, a freeze detection method for a virtual reality screen is applied to a virtual reality video as an example for explanation; after the virtual reality device displays the virtual reality screen, a second data acquisition request can be sent to the server, the server receives the second data acquisition request, determines a virtual reality screen without freezes corresponding to the screen data according to the second data acquisition request, and the server classifies the virtual reality screen without freezes to obtain a screen category; when the screen category is a screen element change category, the server extracts a first reference eye movement feature for characterizing the change of the gaze point for the virtual reality screen without freezes, and sends the screen element change category and the first reference eye movement feature to the virtual reality device.

[0149] The virtual reality device receives the picture element change category and the first reference eye movement feature sent by the server, and the virtual reality device obtains the eye movement data of the target object when viewing the displayed virtual reality picture. When the received picture category is the picture element change category, the virtual reality device extracts the gaze point change feature of the target object based on the eye movement data.

[0150] The virtual reality device determines the similarity between the first reference eye movement feature and the extracted gaze point change feature. When the similarity meets the abnormal interaction condition, that is, when the similarity is less than a preset similarity, it is determined that the interaction state between the target object and each displayed virtual reality screen is an abnormal interaction state. When the similarity is greater than or equal to the preset similarity, it is determined that the interaction state between the target object and each displayed virtual reality screen is a normal interaction state.

[0151] For example, Figure 7 As shown, the method for detecting freeze in a virtual reality screen includes:

[0152] Step 701: Acquire eye movement data of a target object when viewing a displayed virtual reality image;

[0153] Step 702: receiving a first reference eye movement feature and picture category sent by a server;

[0154] Step 703: when the picture category includes a picture element change category, extracting the gaze point change feature of the target object from the eye movement data;

[0155] Step 704: determining the interaction state between the target object and each displayed virtual reality screen according to the first reference eye movement feature and the determined gaze point change feature;

[0156] Step 705: Determine the freeze state of the virtual reality screen according to the interaction state.

[0157] In the above embodiment, when the picture category to which the virtual reality picture belongs is the picture element change category, it means that the picture elements in the virtual reality picture mainly change in position. When the picture elements change in position, the eye movement data mainly causes the gaze point to change. Therefore, the interaction state can be determined based on the first reference eye movement feature and the gaze point change feature, without the need to extract the second reference eye movement feature and the pupil change feature, thereby reducing the amount of data required to determine the interaction state, improving the efficiency of determining the interaction state, and thereby improving the efficiency of freeze detection.

[0158] In some embodiments, predicting a reference eye movement feature corresponding to a virtual reality picture without any freeze based on the picture category of the virtual reality picture includes: obtaining the picture category corresponding to the virtual reality picture; when the picture category is a picture element change category, extracting eye movement features of the virtual reality picture without any freeze through a first eye movement model to obtain a first reference eye movement feature.

[0159] The first eye movement model is used to predict reference eye movement features that characterize changes in gaze points. The first eye movement model includes an image feature extraction network and a first eye movement feature prediction network.

[0160] In some embodiments, the server inputs the virtual reality picture without any lag into a picture category classification model to obtain a picture category corresponding to the virtual reality picture; the picture category classification model includes an image feature extraction network and a picture classification network, the server inputs the virtual reality picture without any lag into the image feature extraction network to obtain a feature image of the virtual reality picture without any lag, and inputs the feature image into the picture classification network to obtain the picture category.

[0161] When the picture category is a picture element change category, the virtual reality picture is input into the image feature extraction network of the first eye movement model, and a feature image corresponding to the virtual reality picture is output through the image feature extraction network. The feature image is input into the first eye movement feature prediction network, and the first reference eye movement feature is output through the first eye movement feature prediction network.

[0162] It should be noted that, since there are multiple virtual reality screens, the feature images include feature images corresponding to each of the multiple virtual reality screens. The feature images corresponding to each of the multiple virtual reality screens are input into a screen classification network to obtain a screen category; the feature images corresponding to each of the multiple virtual reality screens are input into a first eye movement feature prediction network, and the first reference eye movement feature is output through the first eye movement feature prediction network.

[0163] For example, Figure 8 As shown, the picture category classification model 801 includes an image feature extraction network and a picture classification network, and the first eye movement model 802 includes an image feature extraction network and a first eye movement feature prediction network; Fig. 9 As shown, a virtual reality picture without lag is input into an image feature extraction network to obtain a feature image, and the feature image is input into a picture classification network to obtain a picture category; when the picture category is a picture element change category, the feature image is input into a first eye movement feature prediction network to obtain a first reference eye movement feature.

[0164] In some embodiments, the training process of the picture category classification model can be as follows: obtaining a third data set, the third data set including: a training picture without any freeze, and a picture category label corresponding to the training picture; obtaining a trained image feature extraction network, connecting an initial picture classification network after the image feature extraction network, and fixing the parameters of the image feature extraction network; inputting the training picture without any freeze into the image feature extraction network to obtain a training feature image, inputting the training feature image into the initial picture classification network to obtain a training picture category, adjusting the parameters of the initial picture classification network according to the training picture category and the picture category label corresponding to the training picture until the initial picture classification network converges, using the converged initial picture classification network as the picture classification network, and obtaining a picture category classification model according to the image feature extraction network and the picture classification network.

[0165] In some embodiments, the training process of the first eye movement model may be to obtain a fourth data set, the fourth data set including: a training picture without any freeze, and a gaze point change feature label corresponding to the training picture; obtain a trained image feature extraction network, connect the first initial eye movement network after the image feature extraction network, and fix the parameters of the image feature extraction network; input the training picture without any freeze into the image feature extraction network to obtain a training feature image, input the training feature image into the first initial eye movement network to obtain a first training eye movement feature, adjust the parameters of the first initial eye movement network based on the first training eye movement feature and the gaze point change feature label corresponding to the training picture until the first initial eye movement network converges, use the converged first initial eye movement network as the first eye movement network, and obtain the first eye movement model according to the image feature extraction network and the first eye movement network.

[0166] It should be noted that there are no training pictures with freezes in the fourth data set, and all of them are training pictures with picture element changes. Since the training pictures include multiple training picture frames, the training feature images include training feature images corresponding to each of the multiple training picture frames.

[0167] In the above embodiment, the server identifies the picture category of the virtual reality picture. When the picture category of the virtual reality picture is a picture element change category, it means that the main changes in the positions of the picture elements in the virtual reality picture occur. Then, the server extracts the first reference eye movement feature, so as to subsequently determine the interaction state according to the first reference eye movement feature and the gaze point change feature, thereby reducing the amount of data required to determine the interaction state and improving the efficiency of determining the interaction state.

[0168] In some embodiments, before obtaining the eye movement data of the target object when viewing the displayed virtual reality picture, the method for detecting freezes in the virtual reality picture further includes: sending a second data acquisition request to the server; the second data acquisition request is used to instruct the server to predict reference eye movement features corresponding to when the virtual reality picture is not freezed based on the picture category of the virtual reality picture; after obtaining the eye movement data of the target object when viewing the displayed virtual reality picture, the method for detecting freezes in the virtual reality picture further includes: receiving the picture category and reference eye movement features sent by the server; when the picture category includes a brightness change category, determining pupil change features of the target object according to the eye movement data; the reference eye movement features include second reference eye movement features for characterizing pupil changes; and determining the interaction state between the target object and each displayed virtual reality picture based on the second reference eye movement features and the determined pupil change features.

[0169] In some embodiments, a freeze detection method for a virtual reality screen is applied to a virtual reality concert as an example for explanation; the server receives a second data acquisition request, determines a virtual reality screen without freezes corresponding to the screen data based on the second data acquisition request, and classifies the virtual reality screen without freezes to obtain a screen category; when the screen category is a brightness change category, the server extracts a second reference eye movement feature for characterizing pupil changes for the virtual reality screen without freezes, and sends the brightness change category and the second reference eye movement feature to the virtual reality device.

[0170] The virtual reality device receives the brightness change category and the second reference eye movement feature sent by the server, and obtains the eye movement data of the target object when viewing the displayed virtual reality picture. When the received picture category is the brightness change category, the virtual reality device extracts the pupil change feature of the target object based on the eye movement data.

[0171] The virtual reality device can determine the similarity between the second reference eye movement feature and the extracted pupil change feature. When the similarity meets the abnormal interaction condition, that is, when the similarity is less than the preset similarity, it is determined that the interaction state between the target object and the displayed virtual reality screens is an abnormal interaction state. When the similarity is greater than or equal to the preset similarity, it is determined that the interaction state between the target object and the displayed virtual reality screens is a normal interaction state.

[0172] For example, Fig.10 As shown, the method for detecting freeze in a virtual reality screen includes:

[0173] Step 1001: Acquire eye movement data of a target object when viewing a displayed virtual reality image;

[0174] Step 1002: receiving a second reference eye movement feature and picture category sent by a server;

[0175] Step 1003: When the picture category includes a brightness change category, extract pupil change features of the target object from the eye movement data;

[0176] Step 1004: determining the interaction state between the target object and each displayed virtual reality screen according to the second reference eye movement feature and the determined pupil change feature;

[0177] Step 1005: Determine the freeze state of the virtual reality screen according to the interaction state.

[0178] In the above embodiment, when the picture category to which the virtual reality picture belongs is the brightness change category, it means that the main change in the brightness in the virtual reality picture is the position change. When the brightness changes, the eye movement data mainly causes the pupil diameter to change. Then, the interaction state can be determined based on the second reference eye movement feature and the pupil change feature, without extracting the first reference eye movement feature and the gaze point change feature, thereby reducing the amount of data required to determine the interaction state, improving the efficiency of determining the interaction state, and thereby improving the efficiency of freeze detection.

[0179] In some embodiments, predicting reference eye movement features corresponding to a virtual reality picture without any freeze based on the picture category of the virtual reality picture includes: obtaining the picture category corresponding to the virtual reality picture; when the picture category is a brightness change category, extracting eye movement features of the virtual reality picture without any freeze through a second eye movement model to obtain a second reference eye movement feature for characterizing pupil changes.

[0180] The second eye movement model is used to predict reference eye movement features that characterize pupil changes. The second eye movement model includes an image feature extraction network and a second eye movement feature prediction network.

[0181] In some embodiments, the server inputs a virtual reality picture without lag into a picture category classification model to obtain a picture category corresponding to the virtual reality picture. When the picture category is a brightness change category, the virtual reality picture is input into an image feature extraction network of a second eye movement model, a feature image corresponding to the virtual reality picture is output through the image feature extraction network, the feature image is input into a second eye movement feature prediction network, and a second reference eye movement feature is output through the second eye movement feature prediction network.

[0182] It should be noted that, since there are multiple virtual reality screens, the feature image includes feature images corresponding to each of the multiple virtual reality screens. The feature images corresponding to each of the multiple virtual reality screens are input into the second eye movement feature prediction network, and the second reference eye movement feature is output through the second eye movement feature prediction network.

[0183] For example, Fig.11 As shown, the picture category classification model includes an image feature extraction network and a picture classification network, and the second eye movement model includes an image feature extraction network and a second eye movement feature prediction network; a virtual reality picture without any lag is input into the image feature extraction network to obtain a feature image, and the feature image is input into the picture classification network to obtain a picture category; when the picture category is a brightness change category, the feature image is input into the second eye movement feature prediction network to obtain a second reference eye movement feature.

[0184] In some embodiments, the training process of the second eye movement model may be to obtain a fifth data set, the fifth data set including: a training picture without any freeze, and a pupil change feature label corresponding to the training picture; obtain a trained image feature extraction network, connect a second initial eye movement network after the image feature extraction network, and fix the parameters of the image feature extraction network; input the training picture without any freeze into the image feature extraction network to obtain a training feature image, input the training feature image into the second initial eye movement network to obtain a second training eye movement feature, adjust the parameters of the second initial eye movement network based on the second training eye movement feature and the pupil change feature label corresponding to the training picture until the second initial eye movement network converges, use the converged second initial eye movement network as the second eye movement network, and obtain the second eye movement model according to the image feature extraction network and the second eye movement network.

[0185] It should be noted that there are no training pictures with freezes in the fifth data set, and all of them are training pictures with brightness changes. Since the training pictures include multiple training picture frames, the training feature images include training feature images corresponding to the multiple training picture frames.

[0186] In the above embodiment, the server identifies the picture category of the virtual reality picture. When the picture category of the virtual reality picture is a brightness change category, it means that the main change in the virtual reality picture is the brightness. Then the server extracts the second reference eye movement feature to facilitate the subsequent determination of the interaction state based on the second reference eye movement feature and the brightness change feature, thereby reducing the amount of data required to determine the interaction state and improving the efficiency of determining the interaction state.

[0187] In some embodiments, the method for detecting freeze in a virtual reality screen is applied to a VR device running a VR game to detect freeze in the VR game; Fig.12 As shown, the VR device includes: an eye movement data detection module, an iris recognition module and a freeze detection module.

[0188] Among them, the eye movement data detection module includes: an eye tracker and an eye movement feature extraction unit; the eye tracker is used to obtain eye movement data, and the eye movement feature extraction unit is used to process the eye movement data to obtain gaze point change characteristics and pupil change characteristics; the iris recognition module includes: an iris meter, an iris image processing unit and an identity recognition unit; the iris meter is used to obtain iris images, the iris image processing unit is used to extract iris features of iris images, and the identity recognition unit is used to determine identity information based on iris features; the freeze detection module includes: an interaction state determination unit and a freeze state determination unit, the interaction state determination unit is used to determine the interaction state according to the gaze point change characteristics and the pupil change characteristics, and the freeze state determination unit is used to determine the freeze state according to the interaction state.

[0189] In some embodiments, Fig.13As shown, the freeze detection method of the virtual reality picture can be applied to the virtual reality video; the virtual reality device obtains the progress information of the virtual reality video, generates a first data acquisition request according to the progress information, and sends the first data acquisition request to the server; the server receives the first data acquisition request, obtains the picture data according to the progress information, and sends the picture data to the virtual reality device; the virtual reality device displays the virtual reality picture according to the picture data; and obtains the eye movement data of the target object in the process of watching the displayed virtual reality picture;

[0190] The virtual reality device sends a second data acquisition request to the server, and the server acquires a picture category corresponding to the virtual reality picture based on the picture data; when the picture category is a picture element change category, eye movement features are extracted from the virtual reality picture without lag using a first eye movement model to obtain a first reference eye movement feature; the server sends the first reference eye movement feature and the picture element change category to the virtual reality device;

[0191] The virtual reality device receives a picture element change category and a first reference eye movement feature, and determines a gaze point change feature of a target object according to the eye movement data; determines an interaction state between the target object and each displayed virtual reality picture according to the first reference eye movement feature and the determined gaze point change feature; determines that the virtual reality picture is not in a stuck state when the interaction state is a normal interaction state; and determines that the virtual reality picture is in a stuck state when the interaction state is a normal interaction state.

[0192] In some embodiments, Fig.14 As shown, the method for detecting freeze in a virtual reality screen includes:

[0193] Step 1401, sending a first data acquisition request to a server according to progress information of a virtual reality game; receiving game screen data returned by the server in response to the first data acquisition request; and displaying a virtual reality screen of the virtual reality game according to the game screen data;

[0194] Step 1402, periodically acquiring an iris image of the target object; extracting iris features from the iris image, and classifying and processing the iris features to obtain identity information of the target object; when the identity information meets the identity determination conditions, acquiring eye movement data of the target object in the process of viewing the displayed virtual reality screen;

[0195] Step 1403A, sending a second data acquisition request to the server; the second data acquisition request is used to instruct the server to acquire a picture category corresponding to the virtual reality picture, and when the picture category includes a picture element change category and a brightness change category, predicting a reference eye movement feature corresponding to the virtual reality picture without freeze, the reference eye movement feature including a first reference eye movement feature and a second reference eye movement feature;

[0196] Step 1404A, receiving the reference eye movement features and picture categories sent by the server, and when the picture categories include picture element change categories and brightness change categories, extracting the gaze point change features and pupil change features of the target object from the eye movement data;

[0197] Step 1405A, determining a first similarity between the first reference eye movement feature and the gaze point change feature; determining a second similarity between the second reference eye movement feature and the pupil change feature; performing linear processing on the first similarity and the second similarity to obtain a target similarity;

[0198] Step 1406A1: when the target similarity satisfies the abnormal interaction condition, determining that the target object and each displayed virtual reality screen are in an abnormal interaction state; and determining that the virtual reality screen is in a stuck state according to the abnormal interaction state;

[0199] Step 1406A2: when the target similarity does not meet the abnormal interaction condition, determine that the target object and each displayed virtual reality screen are in a normal interaction state; and determine that the virtual reality screen is not in a stuck state according to the normal interaction state;

[0200] Step 1403B, sending a second data acquisition request to the server; the second data acquisition request is used to instruct the server to obtain a picture category corresponding to the virtual reality picture, and when the picture category includes a picture element change category, extracting eye movement features of the virtual reality picture without freeze by using the first eye movement model to obtain a first reference eye movement feature;

[0201] Step 1404B, receiving the reference eye movement features and the picture category sent by the server, and when the picture category includes the picture element change category, determining the gaze point change feature of the target object according to the eye movement data; the reference eye movement features include a first reference eye movement feature for characterizing the gaze point change; determining the interaction state between the target object and each displayed virtual reality picture according to the first reference eye movement feature and the determined gaze point change feature, and determining the freeze state of the virtual reality picture according to the interaction state;

[0202] Step 1403C, sending a second data acquisition request to the server; the second data acquisition request is used to instruct the server to obtain a picture category corresponding to the virtual reality picture, and when the picture category includes a brightness change category, extracting eye movement features of the virtual reality picture without freezes by using a second eye movement model to obtain a second reference eye movement feature for characterizing pupil changes;

[0203] Step 1404C, receiving the reference eye movement features and picture categories sent by the server, when the picture category includes a brightness change category, determining the pupil change features of the target object according to the eye movement data; the reference eye movement features include a second reference eye movement feature for characterizing the pupil change; based on the second reference eye movement features and the determined pupil change features, determining the interaction state between the target object and each displayed virtual reality picture, and determining the freeze state of the virtual reality picture according to the interaction state.

[0204] In the above-mentioned method for detecting freeze in a virtual reality screen, eye movement data of a target object in the process of viewing a displayed virtual reality screen is obtained, gaze point change features and pupil change features are extracted from the eye movement data, and the interaction state between the target object and the displayed virtual reality screen is determined based on the gaze point change features and the pupil change features, and the freeze state of the virtual reality screen is determined based on the interaction state; based on the real eye movement behavior of the target object viewing the displayed virtual reality screen, it is determined whether the virtual reality screen has freeze, so that the freeze state of the virtual reality screen is related to the real eye movement behavior of the target object viewing the displayed virtual reality screen, and thus the freeze in the virtual reality screen is a freeze that can be actually observed by the target object, thereby improving the accuracy of freeze detection for the virtual reality screen.

[0205] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0206] Based on the same inventive concept, the embodiment of the present application also provides a virtual reality screen freeze detection device for implementing the virtual reality screen freeze detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more virtual reality screen freeze detection devices provided below can refer to the limitations of the virtual reality screen freeze detection method above, and will not be repeated here.

[0207] In one embodiment, Fig.15As shown, a freeze detection device for a virtual reality screen is provided, comprising: an eye movement data acquisition module 1501, an eye movement feature extraction module 1502, an interaction state determination module 1503 and a freeze state determination module 1504, wherein:

[0208] The eye movement data acquisition module 1501 is used to acquire the eye movement data of the target object when viewing the displayed virtual reality picture;

[0209] An eye movement feature extraction module 1502 is used to extract the gaze point change features and pupil change features of the target object from the eye movement data;

[0210] An interaction state determination module 1503, used to determine the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and the pupil change characteristics;

[0211] The freeze state determination module 1504 is used to determine the freeze state of the virtual reality screen according to the interaction state.

[0212] In some embodiments, the freeze detection device for virtual reality screen can be applied to virtual reality games, and the freeze detection device for virtual reality screen also includes: a virtual reality screen display module, which is used to send a first data acquisition request to a server based on the progress information of the virtual reality game; receive game screen data returned by the server in response to the first data acquisition request; and display the virtual reality screen of the virtual reality game based on the game screen data.

[0213] In some embodiments, the freeze detection device for a virtual reality screen also includes: a first receiving unit, used to receive reference eye movement features and screen categories sent by a server; the reference eye movement features are eye movement features predicted based on a virtual reality screen without freezes; accordingly, an eye movement feature extraction module 1502, used to extract the gaze point change features and pupil change features of the target object from the eye movement data when the screen category includes a screen element change category and a brightness change category; accordingly, an interaction state determination module 1503, used to determine the interaction state between the target object and each displayed virtual reality screen based on the reference eye movement features, the gaze point change features and the pupil change features.

[0214] In some embodiments, the reference eye movement feature includes a first reference eye movement feature and a second reference eye movement feature, the first reference eye movement feature is used to characterize the change of the gaze point, and the second reference eye movement feature is used to characterize the change of the pupil; the interaction state determination module 1503 is also used to determine a first similarity between the first reference eye movement feature and the gaze point change feature; determine a second similarity between the second reference eye movement feature and the pupil change feature; and determine the interaction state between the target object and each displayed virtual reality screen based on the first similarity and the second similarity.

[0215] In some embodiments, the freeze state includes the virtual reality screen being in a freeze state; the interaction state determination module 1503 includes a first interaction state determination unit, which is used to perform linear processing on the first similarity and the second similarity to obtain a target similarity; when the target similarity meets the abnormal interaction condition, it is determined that the target object and the displayed virtual reality screens are in an abnormal interaction state; accordingly, the freeze state determination module 1504 is used to determine that the virtual reality screen is in a freeze state according to the abnormal interaction state.

[0216] In some embodiments, the freeze state includes that the virtual reality screen is not in a freeze state; the interaction state determination module 1503 includes a second interaction state determination unit, which is used to perform linear processing on the first similarity and the second similarity to obtain a target similarity; when the target similarity does not meet the abnormal interaction condition, it is determined that the target object and the displayed virtual reality screens are in a normal interaction state; accordingly, the freeze state determination module 1504 is used to determine that the virtual reality screen is not in a freeze state according to the normal interaction state.

[0217] In some embodiments, the freeze detection device for a virtual reality screen further includes: a reference eye movement feature acquisition module, configured to send a second data acquisition request to a server; the second data acquisition request is configured to instruct the server to predict a reference eye movement feature corresponding to a virtual reality screen without freeze based on a screen category of the virtual reality screen; and receive the screen category and the reference eye movement feature sent by the server;

[0218] Accordingly, the eye movement feature extraction module 1502 is used to determine the gaze point change feature of the target object according to the eye movement data when the picture category includes the picture element change category; the reference eye movement feature includes a first reference eye movement feature for characterizing the gaze point change;

[0219] Correspondingly, the interaction state determination module 1503 is used to determine the interaction state between the target object and each displayed virtual reality screen according to the first reference eye movement feature and the determined gaze point change feature.

[0220] In some embodiments, predicting a reference eye movement feature corresponding to a virtual reality picture without any freeze based on the picture category of the virtual reality picture includes: obtaining the picture category corresponding to the virtual reality picture; when the picture category is a picture element change category, extracting eye movement features of the virtual reality picture without any freeze through a first eye movement model to obtain a first reference eye movement feature.

[0221] In some embodiments, the eye movement feature extraction module 1502 is further used to determine pupil change features of the target object according to the eye movement data when the picture category includes a brightness change category; the reference eye movement feature includes a second reference eye movement feature for characterizing pupil changes;

[0222] Correspondingly, the interaction state determination module 1503 is used to determine the interaction state between the target object and each displayed virtual reality screen according to the second reference eye movement feature and the determined pupil change feature.

[0223] In some embodiments, predicting reference eye movement features corresponding to a virtual reality picture without any freeze based on the picture category of the virtual reality picture includes: obtaining the picture category corresponding to the virtual reality picture; when the picture category is a brightness change category, extracting eye movement features of the virtual reality picture without any freeze through a second eye movement model to obtain a second reference eye movement feature for characterizing pupil changes.

[0224] In some embodiments, the freeze detection device for a virtual reality screen also includes: an identity information determination module, which is used to periodically obtain an iris image of the target object; extract iris features of the iris image, and classify and process the iris features to obtain the identity information of the target object; accordingly, an eye movement data acquisition module 1501, which is used to obtain the eye movement data of the target object when the identity information meets the identity determination conditions.

[0225] Each module in the above-mentioned virtual reality screen freeze detection device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0226] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.16As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless method can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for detecting a freeze in a virtual reality screen is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0227] Those skilled in the art will understand that Fig.16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0228] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0229] Obtain eye movement data of a target object when viewing a displayed virtual reality screen; extract gaze point change characteristics and pupil change characteristics of the target object from the eye movement data; determine the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and pupil change characteristics; determine the freeze state of the virtual reality screen according to the interaction state.

[0230] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0231] Obtain eye movement data of a target object when viewing a displayed virtual reality screen; extract gaze point change characteristics and pupil change characteristics of the target object from the eye movement data; determine the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and pupil change characteristics; determine the freeze state of the virtual reality screen according to the interaction state.

[0232] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0233] Obtain eye movement data of a target object when viewing a displayed virtual reality screen; extract gaze point change characteristics and pupil change characteristics of the target object from the eye movement data; determine the interaction state between the target object and each displayed virtual reality screen according to the gaze point change characteristics and pupil change characteristics; determine the freeze state of the virtual reality screen according to the interaction state.

[0234] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0235] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0236] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0237] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting freeze in a virtual reality screen, characterized in that: The method comprises: Acquire eye movement data of the target object when viewing the displayed virtual reality image; Extracting the gaze point change characteristics and pupil change characteristics of the target object from the eye movement data; Determining an interaction state between the target object and each of the displayed virtual reality images according to the gaze point change characteristics and the pupil change characteristics; A freeze state of the virtual reality screen is determined according to the interaction state.

2. The method according to claim 1, characterized in that Applied to virtual reality games; before obtaining the eye movement data of the target object in the process of viewing the displayed virtual reality picture, the method further includes: Sending a first data acquisition request to a server according to the progress information of the virtual reality game; Receiving game screen data returned by the server in response to the first data acquisition request; The virtual reality screen of the virtual reality game is displayed according to the game screen data.

3. The method according to claim 1, characterized in that: The method further comprises: receiving a reference eye movement feature and a picture category sent by a server; the reference eye movement feature is an eye movement feature predicted based on the virtual reality picture without lag; The step of extracting the gaze point change characteristics and pupil change characteristics of the target object from the eye movement data includes: When the picture category includes a picture element change category and a brightness change category, extracting the gaze point change feature and the pupil change feature of the target object from the eye movement data; The determining, according to the gaze point change feature and the pupil change feature, the interaction state between the target object and each of the displayed virtual reality images comprises: Based on the reference eye movement features, the gaze point change features and the pupil change features, an interaction state between the target object and each of the displayed virtual reality images is determined.

4. The method according to claim 3, characterized in that: The reference eye movement feature includes a first reference eye movement feature and a second reference eye movement feature, wherein the first reference eye movement feature is used to characterize a change in a gaze point, and the second reference eye movement feature is used to characterize a change in a pupil; The determining, based on the reference eye movement feature, the gaze point change feature, and the pupil change feature, the interaction state between the target object and each of the displayed virtual reality images comprises: Determining a first similarity between the first reference eye movement feature and the gaze point change feature; determining a second similarity between the second reference eye movement feature and the pupil change feature; An interaction state between the target object and each of the displayed virtual reality images is determined according to the first similarity and the second similarity.

5. The method according to claim 4, characterized in that The freeze state includes the virtual reality screen being in a freeze state; and determining the interaction state between the target object and each of the displayed virtual reality screens according to the first similarity and the second similarity includes: Performing linear processing on the first similarity and the second similarity to obtain a target similarity; When the target similarity satisfies an abnormal interaction condition, determining that the target object and each of the displayed virtual reality images are in an abnormal interaction state; The determining the freeze state of the virtual reality screen according to the interaction state includes: According to the abnormal interaction state, it is determined that the virtual reality screen is in a stuck state.

6. The method according to claim 4, characterized in that The freeze state includes that the virtual reality screen is not in a freeze state; and determining the interaction state between the target object and each of the displayed virtual reality screens according to the first similarity and the second similarity includes: Performing linear processing on the first similarity and the second similarity to obtain a target similarity; When the target similarity does not meet the abnormal interaction condition, determining that the target object and each of the displayed virtual reality images are in a normal interaction state; The determining the freeze state of the virtual reality screen according to the interaction state includes: According to the normal interaction state, it is determined that the virtual reality screen is not in a stuck state.

7. The method according to claim 1, characterized in that Before acquiring the eye movement data of the target object in the process of viewing the displayed virtual reality picture, the method further includes: Sending a second data acquisition request to the server; the second data acquisition request is used to instruct the server to predict reference eye movement features corresponding to the virtual reality picture without freeze based on the picture category of the virtual reality picture; After acquiring the eye movement data of the target object in the process of viewing the displayed virtual reality picture, the method further includes: Receiving the picture category and the reference eye movement feature sent by a server; When the picture category includes a picture element change category, determining a gaze point change feature of the target object according to the eye movement data; the reference eye movement feature includes a first reference eye movement feature for characterizing a gaze point change; The interaction state between the target object and each of the displayed virtual reality images is determined based on the first reference eye movement feature and the determined gaze point change feature.

8. The method according to claim 7, characterized in that The reference eye movement features corresponding to the virtual reality picture without freeze are predicted according to the picture category of the virtual reality picture, including: Obtaining a picture category corresponding to the virtual reality picture; When the picture category is the picture element change category, eye movement features are extracted from the virtual reality picture without any freeze using a first eye movement model to obtain the first reference eye movement features.

9. The method according to claim 1, characterized in that: Before acquiring the eye movement data of the target object in the process of viewing the displayed virtual reality picture, the method further includes: Sending a second data acquisition request to the server; the second data acquisition request is used to instruct the server to predict the reference eye movement features corresponding to the virtual reality picture when the virtual reality picture is not stuck according to the picture category of the virtual reality picture; After acquiring the eye movement data of the target object in the process of viewing the displayed virtual reality picture, the method further includes: Receiving the picture category and the reference eye movement feature sent by a server; When the picture category includes a brightness change category, determining pupil change characteristics of the target object according to the eye movement data; the reference eye movement characteristics include a second reference eye movement characteristic for characterizing pupil changes; The interaction state between the target object and each of the displayed virtual reality images is determined based on the second reference eye movement feature and the determined pupil change feature.

10. The method according to claim 9, characterized in that The reference eye movement features corresponding to the virtual reality picture without freeze are predicted according to the picture category of the virtual reality picture, including: Obtaining a picture category corresponding to the virtual reality picture; When the picture category is a brightness change category, eye movement features are extracted from the virtual reality picture without any freeze using a second eye movement model to obtain a second reference eye movement feature for characterizing pupil changes.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: regularly acquiring an iris image of the target object; Extracting iris features from the iris image and classifying the iris features to obtain identity information of the target object; The step of obtaining the eye movement data of the target object when viewing the displayed virtual reality picture includes: When the identity information meets the identity determination condition, the eye movement data of the target object in the process of viewing the displayed virtual reality picture is obtained.

12. A device for detecting freeze in a virtual reality screen, characterized in that: The device comprises: An eye movement data acquisition module is used to acquire the eye movement data of the target object when viewing the displayed virtual reality picture; An eye movement feature extraction module, used to extract the gaze point change features and pupil change features of the target object from the eye movement data; An interaction state determination module, used to determine the interaction state between the target object and each of the displayed virtual reality images according to the gaze point change characteristics and the pupil change characteristics; A freeze state determination module is used to determine the freeze state of the virtual reality screen according to the interaction state.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images