Multimedia resource display method and device based on eye control interaction and related equipment

Abstract

This application provides a multimedia resource display method, apparatus, and related equipment based on eye-tracking interaction. It offers an interaction scheme for displaying multimedia resources that accurately determines the user's selection intent through gaze behavior. By analyzing eye-tracking data generated during the user's viewing of media resources, the application determines the user's gaze behavior, identifies the selected interactive element based on the gaze behavior, and further displays media resources pre-associated with the interactive element. The interaction method proposed in this application does not require the user to execute preset fixed action commands, highly conforming to natural visual observation habits. Furthermore, because the user's gaze behavior accurately reflects their interests, it ensures that the selected content is what the user is most interested in. The entire interaction process is intuitive, smooth, and seamless, enhancing the user's immersion and acceptance.

Description

Technical Field

[0001] This application relates to the field of information interaction technology, and in particular to a multimedia resource display method, device, computer equipment, and computer program product based on eye-tracking interaction. Background Technology

[0002] Museums, art galleries, science and technology museums, and other cultural venues have long regarded exhibitions as an important means of popularizing knowledge among the public. In recent years, digital exhibitions have rapidly developed as an emerging exhibition format. They replace or supplement traditional physical and textual displays with multimedia resources such as audio, video, animation, and interactive programs, and achieve dynamic presentation of content through terminals such as screens and projectors. With its advantages of strong interactivity, diverse formats, and easy content updates, digital exhibitions have become an indispensable and important form in the contemporary exhibition system. However, the current mainstream digital exhibition interaction methods all have certain drawbacks in public exhibition scenarios. For example, touch screen interaction requires visitors to be close to the device to operate it and has high requirements for the installation location and structure of the device; remote control interaction increases the risk of loss and damage of hardware devices and requires additional distribution and recycling management; motion sensing or gesture interaction usually requires viewers to make gestures of a specific amplitude, which is easily restricted by space in public places and may interfere with the viewing experience of others.

[0003] In contrast, eye tracking, as a natural and non-contact interaction method, can capture the user's gaze position in real time and has great potential. However, existing eye-tracking technologies usually require users to perform explicit actions to express their intentions, which can disrupt the viewing process in public exhibition scenarios and affect the smoothness and immersion of the viewing experience.

[0004] Therefore, there is an urgent need for an eye-tracking interaction solution that can operate naturally in public exhibition settings and accurately determine the user's intentions. Summary of the Invention

[0005] This application provides a multimedia resource display method, device, and related equipment based on eye-tracking interaction, and provides an interaction scheme for displaying multimedia resources that accurately judges the user's selection intent through gaze behavior.

[0006] To achieve the above objectives, the main technical solutions adopted in this application include: In a first aspect, embodiments of this application provide a method for displaying multimedia resources based on eye-tracking interaction, the method comprising: The display includes a first media resource comprising at least one interactive element, the association between the interactive element and a second media resource being pre-configured; During the display of the first media resource, eye movement data of the user while viewing the first media resource is collected, and the eye movement data is analyzed to obtain the user's gaze behavior; Based on the gaze behavior, it is determined that the user has selected a target interactive element among the interactive elements by gazing, and a second media resource associated with the target interactive element is displayed.

[0007] Optionally, determining, based on the gaze behavior, that the user has selected a target interactive element among the interactive elements through gaze includes: Determine that the target position of the user's gaze behavior falls within the range of the currently gazed interactive element within a preset sliding time window, and generate an interactive icon on the currently gazed interactive element. The first gaze duration is obtained based on the user's gaze behavior toward the interactive icon; When the first gaze duration is greater than or equal to a preset first gaze duration threshold, the interactive element selected by the user during the current gaze is determined to be the target interactive element.

[0008] Optionally, determining, based on the gaze behavior, that the user has selected a target interactive element among the interactive elements through gaze includes: Determine that the target position of the user's gaze behavior within a preset sliding time window falls within the range of the currently gazed interactive element; determine that the interactive element currently gazed upon by the user is the target interactive element; or, Based on the user's gaze behavior toward the currently gazed interactive icon, a second gaze duration is obtained, and the currently gazed interactive icon is pre-set on the interactive element; when the second gaze duration is greater than or equal to a preset second gaze duration threshold, it is determined that the user has selected the currently gazed interactive element as the target interactive element.

[0009] Optionally, the interactive icon is in its initial state before being viewed by the user; The method further includes: As the first gaze duration or the second gaze duration gradually increases, a transition animation is played, which is used to show the visual transition of the interactive icon from the initial state to the final state. When the first gaze duration or the second gaze duration is greater than or equal to the first duration threshold or the second duration threshold, the transition animation finishes playing, and an interactive icon of the final state is generated on the currently gazed interactive element.

[0010] Optionally, the step of analyzing the eye-tracking data to obtain the user's gaze behavior includes: The eye-tracking data is sampled within a preset sliding time window to obtain a set of sampling points; Calculate the dispersion of the set of sampling points, the dispersion characterizing the stability of the user's visual axis during the sliding window; When the dispersion is less than a preset dispersion threshold at the target location, the user's gaze behavior toward the target location within the sliding time window is determined.

[0011] Optionally, after displaying the second media resource associated with the target interactive element, the method further includes: Based on the gaze behavior, it is determined that the user selected a new target interactive element from the interactive elements by gazing. Stop displaying the current second media resource and switch to displaying the second media resource associated with the new target interactive element.

[0012] Optionally, after displaying the second media resource associated with the target interactive element, the method further includes: The user's physiological behavior data is detected, and the physiological behavior data includes at least one of eye movements, head posture, and body position; Based on the physiological behavior data, the display parameters of the second media resource associated with the target interactive element are dynamically adjusted. The display parameters include at least one of the following: playback rate, volume, screen scaling ratio, sound field spatial position, and screen rendering perspective.

[0013] Secondly, embodiments of this application provide a multimedia resource display device based on eye-tracking interaction, the device comprising: The first display unit is used to display a first media resource including at least one interactive element, wherein the association between the interactive element and the second media resource is pre-configured; An eye-tracking analysis unit is used to collect eye-tracking data of a user while viewing the first media resource during the display of the first media resource, and to analyze the eye-tracking data to obtain the user's gaze behavior; An interactive display unit is used to determine, based on the gaze behavior, that the user has selected a target interactive element among the interactive elements by gazing, and to display a second media resource associated with the target interactive element.

[0014] Thirdly, embodiments of this application provide a computer device, including: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes these computer instructions to perform the multimedia resource display method based on eye-tracking interaction as described in the first aspect.

[0015] Fourthly, embodiments of this application provide a computer program product, characterized in that it includes computer instructions, which are used to cause a computer to execute the multimedia resource display method based on eye-tracking interaction described in the first aspect.

[0016] The multimedia resource display method based on eye-tracking interaction proposed in this application determines the user's gaze behavior by analyzing eye-tracking data generated during the viewing of media resources. Based on the gaze behavior, the selected interactive element is determined, and media resources pre-associated with the interactive element are further displayed. The interaction method proposed in this application does not require the user to execute preset fixed action commands, highly conforming to natural visual observation habits. Furthermore, since the user's gaze behavior accurately reflects their interests, it ensures that the selected content is what the user is most interested in. The entire interaction process is intuitive, smooth, and seamless, enhancing the user's immersion and acceptance. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the steps of a multimedia resource display method based on eye-tracking interaction, as provided in an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of a virtual museum display system implemented based on the eye-controlled interaction-based multimedia resource display method provided in this embodiment of the present application.

[0020] Figure 3 This is a schematic diagram illustrating three eye-controlled interaction methods provided in this application embodiment.

[0021] Figure 4 This is a structural diagram of a multimedia resource display device based on eye-tracking interaction provided in an embodiment of this application.

[0022] Figure 5 This is a schematic diagram of the structure of a computer device example provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Exhibitions are an important means of information dissemination. Cultural venues such as museums, art galleries, and science and technology museums can popularize knowledge through exhibitions for visitors; scenic spots, planning museums, and corporate showrooms can also use exhibitions to convey the core values ​​of nature and culture, urban development, or industrial innovation to the public.

[0025] With the development of technology, the form of exhibitions is constantly evolving. Among them, digital exhibitions use multimedia resources such as audio, video, animation, and interactive programs to replace or supplement traditional physical and textual displays, and achieve dynamic presentation of content through terminals such as screens and projectors. Digital exhibitions, with their advantages of strong interactivity, diverse formats, and easy content updates, have been widely used in recent years and have become an indispensable and important form in the contemporary exhibition system.

[0026] However, current mainstream digital exhibition interaction methods all have certain drawbacks in public exhibition settings. For example, touchscreen interaction requires visitors to be close to the device to operate it, which limits the size and installation location of the touchscreen device to ensure ease of use. Furthermore, when the exhibition equipment, such as a projector or screen, is large, users may need to repeatedly move between their operating position and the optimal viewing position to obtain the best angle, affecting the viewing experience. Using remote controls instead of touchscreens carries a higher risk of loss and damage due to their small size, and requires additional manpower for distribution and collection. Motion-sensing or gesture-based interaction methods typically require viewers to make specific gestures, which are easily limited by space and may interfere with the viewing experience of others when there are large crowds at public exhibitions.

[0027] Compared to the aforementioned interaction methods, eye tracking, as a non-contact and natural interaction method, has advantages such as ease of operation, low requirements for hardware installation location, and low risk of damage, and thus holds great potential. However, existing eye-tracking technologies typically require users to perform explicit actions to express their intentions, such as blinking. Some actions even require users to supplement with non-eye behaviors, such as head movements, for explicit confirmation. This interaction method requires deliberate action from the user, which can disrupt the viewing process in public exhibition settings, affecting the smoothness and immersion of the viewing experience. Some overly deliberate confirmation actions can also easily make viewers feel embarrassed and unnatural in public. On the other hand, attempting to directly analyze the user's intentions without requiring them to perform fixed actions can easily make the user feel a lack of control, and the current method has a relatively high rate of misinterpretation of user intentions.

[0028] According to an embodiment of this application, a multimedia resource display method based on eye-tracking interaction is provided to solve the above-mentioned problems. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] This embodiment provides a multimedia resource display method based on eye-tracking interaction, such as... Figure 1 As shown, it includes the following steps: Step 110: Display a first media resource including at least one interactive element, the association between the interactive element and the second media resource being pre-configured.

[0030] In this embodiment, the first media resource can be displayed through various multimedia terminals with display capabilities, such as screens, projectors, and virtual reality (VR) devices. The first media resource is in the form of content that humans can visually perceive, so as to facilitate subsequent eye tracking, such as still images, dynamic videos, 3D models, or virtual scenes.

[0031] Interactive elements, as core components for audience interaction in digital displays, can be categorized into explicit and implicit forms. Explicit elements are directly presented on the primary media resource in a visual manner, such as highlighted areas or functional icons added to the resource surface. Implicit elements, on the other hand, are not directly visible. A typical example is the Area of ​​Interest (AOI) pre-defined in the primary media resource. Although this area is not marked during the display, it can trigger a response when the audience interacts, achieving "hidden" interaction. These two forms together expand the interactive dimension of the exhibition, making the content not only visually presented but also tangible and perceptible in terms of behavior.

[0032] The full functionality of interactive elements relies on pre-associated secondary media resources. Unlike the primary media resources, which are the main focus of the exhibition, secondary media resources only require direct reception and understanding by human senses, thus allowing for a wider variety of flexible options. Examples include audio, still images, and moving videos. Each interactive element has at least one corresponding secondary media resource. The primary and secondary media resources together constitute all the content of the exhibition, and their flexible selection expands the depth and dimension of information presentation. The relationships between interactive elements and secondary media resources can be stored and managed through mapping tables, database records, or other methods.

[0033] Step 120: During the display of the first media resource, collect eye-tracking data of the user while watching the first media resource, and analyze the eye-tracking data to obtain the user's gaze behavior.

[0034] In this embodiment, during the continuous display of the first media resource, the user's eye movement data is acquired. Specifically, the eye movement data can include signals such as the user's gaze point position, gaze duration, and pupil changes. The system can be configured to continuously acquire the user's eye movement data during the display period, or to acquire the user's eye movement data periodically according to a pre-set cycle.

[0035] The device for collecting eye-tracking data can be a terminal that displays primary media resources, such as a VR device with eye-tracking capabilities, or a pre-set dedicated eye-tracking data collection device, such as an eye tracker placed near a large display screen.

[0036] After acquiring eye-tracking data, the collected eye-tracking data is analyzed and processed. The analysis process aims to extract gaze behavior information with interactive intent from the raw eye-tracking data. Specifically, it can be done by pre-dividing a range on the first media resource and determining whether the user's gaze position falls within a certain range in order to determine the user's gaze behavior.

[0037] Step 130: Based on the gaze behavior, determine that the user has selected the target interactive element through gaze and display the second media resource associated with the target interactive element.

[0038] In this embodiment, the user's selection intent is determined based on their gaze behavior to determine the user's choice of interactive elements. The user's gaze behavior may involve gazing at any position of the first media resource. When the gaze behavior is detected to meet the triggering conditions for a certain interactive element, it is determined that the interactive element has been selected by the user and becomes the target interactive element. The number of target interactive elements can be set according to actual needs based on display conditions such as the size and shape of the display area. For example, it can be set that the user can only select one target interactive element at a time to display a corresponding set of second media resources, or it can be set that more than one target interactive element can be selected to display multiple second media resources simultaneously.

[0039] The triggering conditions for interactive elements can be, for example, detecting that the duration of continuous gaze on the interactive element exceeds a threshold, or detecting that the number of times the user gazes on the same interactive element within a preset time range exceeds a threshold, or detecting that the continuous gaze trajectory formed by the user's gaze behavior passes through the interactive element more than a threshold, or that the shape of the gaze trajectory near the interactive element meets preset requirements, etc. This embodiment does not impose specific limitations. Different triggering conditions can be set for different interactive elements in the first media resource, or the same triggering conditions can be set uniformly.

[0040] The display method of the second media resource is selected according to its type. For example, if it is an audio type second media resource, the display method can be to play the audio. If it is visual content such as images or videos, the display method can be to pop up a display window and overlay it on the display interface of the first media resource, or switch to full screen and combine it with the display interface of the second media resource.

[0041] In one embodiment, interactive elements that have already been displayed as associated second media resources can be marked by highlighting or outlining. Furthermore, the interactive element can be set to prevent it from being activated again as a target interactive element to reduce accidental triggering.

[0042] The interaction method proposed in this application does not require users to execute preset, fixed action commands, and highly aligns with natural visual observation habits. Because the user's gaze behavior accurately reflects their interests, it ensures that the selected content is what the user is most interested in. The entire interaction process is intuitive, smooth, and seamless, enhancing user immersion and acceptance. The second embodiment of this application further specifies the multimedia resource display method based on eye-tracking interaction in the first embodiment in a more detailed and specific way. Some or all of the technical features in the second embodiment can be combined with or replaced by the first embodiment, either individually or in combination, to obtain more feasible multimedia resource display methods based on eye-tracking interaction.

[0043] The multimedia resource display method based on eye-tracking interaction in the second embodiment of this application will be described in detail below: Optionally, determining whether a user has selected a target interactive element through gazing based on the gazing behavior includes: determining that the target position of the user's gazing behavior falls within the range of the currently gazing interactive element within a preset sliding time window, and generating an interactive icon on the currently gazing interactive element; obtaining a first gazing duration based on the user's gazing behavior on the interactive icon; and determining that the user has selected the currently gazing interactive element as the target interactive element when the first gazing duration is greater than or equal to a preset first gazing duration threshold.

[0044] In this embodiment, the specific method by which the user selects the target interactive element is further defined. Specifically, the process of selecting the target interactive element adopts a two-stage confirmation mechanism.

[0045] First, during the user's viewing of the primary media resource, it is determined whether the user's gaze falls within the range of a certain interactive element within a preset sliding time window, thus confirming whether the user is currently looking at that interactive element. The value of the sliding time window can be set according to actual needs. In one embodiment, the lower limit of the sliding time window is 100ms, and the upper limit is 400ms. When it is determined that the target position is within the range of a certain interactive element, it is considered that the user has shown initial interest in that interactive element, and an interactive icon is generated on that interactive element.

[0046] The newly generated interactive icons serve as visual cues, overlaid on the corresponding positions of the interactive elements that initially interest the user, guiding the user to the next stage of confirmation. The display position of the interactive icons can be set according to actual needs, such as being placed above, below, left, or right of the interactive element, or a combination of two of the aforementioned directions, or at the center, etc. This embodiment does not impose specific limitations.

[0047] Continue collecting the user's eye-tracking data, focusing primarily on their gaze behavior towards newly generated interactive icons. When the system detects that the user's gaze has shifted to and lingered on the interactive icon, it begins recording the first gaze duration, i.e., the duration for which the user continuously gazes at the interactive icon. When the first gaze duration is greater than or equal to a preset first gaze duration threshold, it is determined that the user has explicitly selected the interactive element, and the currently gazed interactive element is identified as the target interactive element.

[0048] The two-stage confirmation method proposed in this embodiment understands the user's intent in the first stage by analyzing the relationship between the target position of the user's gaze behavior and the range of interactive elements within the sliding time window; in the second stage, it confirms the user's intent by explicitly displaying the interactive icon and analyzing the user's gaze duration on the interactive icon, thereby reducing the probability of false triggering while enhancing the controllability and accuracy of the interaction.

[0049] The generated interactive icon can be set to not disappear after generation to indicate that the interactive icon has been interacted with, and to indicate that the associated content has been viewed by highlighting it after the associated second media resource has been displayed; or, it can be set to disappear when the second media resource is displayed; or, it can be set to disappear after the associated second media resource has been displayed to avoid obscuring the media resource being displayed.

[0050] In one embodiment, the second media resource corresponding to the interactive element is an audio explanation, and the interactive icon is a semi-transparent 3D speaker icon, which prompts the user to continue looking at the audio to trigger the explanation.

[0051] Optionally, based on the gaze behavior, determining that the user has selected a target interactive element through gaze includes: determining that the target position of the user's gaze behavior falls within the range of the currently gazed interactive element within a preset sliding time window; determining that the currently gazed interactive element selected by the user is the target interactive element; or, based on the user's gaze behavior on the currently gazed interactive icon, obtaining a second gaze duration, wherein the currently gazed interactive icon is preset on the interactive element; when the second gaze duration is greater than or equal to a preset second gaze duration threshold, determining that the currently gazed interactive element selected by the user is the target interactive element.

[0052] This embodiment also proposes two other single-stage confirmation methods for selecting interactive elements. Specifically, the first interactive method proposed in this embodiment is: directly determining whether the target position of the user's gaze behavior within a preset sliding time window falls within the range of a certain interactive element (regardless of whether it has an interactive icon). When it is determined that the target position falls within the range of a certain interactive element, it is determined that the user has selected that interactive element.

[0053] In one embodiment, the first interaction method can be applied to situations where the interactive element has a large coverage area. For example, when displaying a painting, a mountain in the painting can be set as an interactive element, and it can be determined whether the user's gaze is continuously focused on the area of ​​the mountain interactive element within the sliding time window.

[0054] The second interaction method proposed in this embodiment is as follows: An interactive icon is pre-set on the interactive element; that is, the icon is an inherent component of the interactive element and is already present when the first media resource is displayed. In this interaction method, looking at other locations on the interactive element will not trigger a timer; only when looking at the pre-set interactive icon will the timer record the duration of continuous looking, which is recorded as the second looking duration. When the second looking duration is greater than or equal to a preset second looking duration threshold (which may be the same as or different from the aforementioned first duration threshold), it is determined that the user has a clear interaction intention, and the currently looked-at interactive element is identified as the target interactive element.

[0055] In one embodiment, the lower limit of the second gaze duration threshold is 150ms and the upper limit is 1000ms, and the specific value can be 600ms.

[0056] This embodiment proposes two other simpler and more direct single-stage interaction methods to trigger the display of the second media resource, which are suitable for application scenarios with high requirements for interaction efficiency or operation simplicity.

[0057] Optionally, the interactive icon is in its initial state before being viewed by the user; the method further includes: playing a transition animation as the first or second gaze duration gradually increases, the transition animation being used to show the visual transition of the interactive icon from the initial state to the final state; when the first or second gaze duration is greater than or equal to a first or second duration threshold, the transition animation finishes playing, and the final state interactive icon is generated on the currently viewed interactive element.

[0058] This embodiment further defines the technical feature of interactive icons. Specifically, in the method described in the foregoing embodiment, interactive icons can be applied to the following two interaction methods: when it is determined that the target position of the user's gaze behavior is within the range of an interactive element, an interactive icon is generated on the interactive element; or, an interactive icon is pre-set on the interactive element.

[0059] In this embodiment, the interactive icon is initially in its initial state when applying the two interaction methods described above. The initial state can be semi-transparent, outlined, low-saturation, or a reduced core pattern, etc., to indicate to the user that an interactive icon exists at that location and that the icon has not yet been selected.

[0060] After the user notices the prompt, they gaze at the interactive icon. As the first or second gaze duration gradually increases, a pre-set transition animation plays. The purpose of the transition animation is to demonstrate the visual transition of the interactive icon from its initial state to its final state. For example, the icon may gradually fill up, brighten, enlarge, change color, or complete some kind of progress change as the gaze time accumulates, and its animation progress is synchronized with or mapped to the accumulation of the second gaze duration.

[0061] When the duration of the first or second gaze is greater than or equal to a preset threshold for the second gaze duration, it indicates that the user has confirmed the selection of the interactive element corresponding to the icon. At this point, the transition animation finishes playing, and the interactive icon is fully presented in its final state. The final state can be visually represented by opacity, highlighting, or complete filling, to intuitively indicate that the user's intent has been recognized. At this point, the currently gazed interactive element is identified as the target interactive element, triggering the display of subsequent secondary media resources.

[0062] This embodiment dynamically binds the cumulative gaze duration with the visual state changes of interactive icons, providing users with clear and real-time operational feedback and enhancing the perceptibility and controllability of the interaction.

[0063] Optionally, analyzing eye-tracking data to obtain the user's gaze behavior includes: sampling eye-tracking data within a preset sliding time window to obtain a set of sampling points; calculating the dispersion of the set of sampling points, where dispersion characterizes the stability of the user's visual axis during the sliding window; and determining the user's gaze behavior at the target location within the sliding time window when the dispersion at the target location is less than a preset dispersion threshold.

[0064] This embodiment further defines the method for analyzing eye-tracking data.

[0065] Specifically, eye-tracking data analysis can be achieved using the I-DT (Identification by Dispersion-Threshold) fixed-point detection algorithm. Eye-tracking data acquired during the display of the first media resource is divided into sliding time windows. Within each sliding time window, sampling is performed to obtain a series of sampling points arranged in chronological order, forming a sampling point set. Each sampling point represents the position of the user's gaze on the first media resource at a given moment.

[0066] Dispersion characterization reflects the spatial clustering of sampling points within a sliding time window, and thus reflects the stability of the user's gaze axis. The smaller the dispersion, the more focused and stable the user's gaze is, and the more likely it is to constitute an effective fixation; conversely, it may be a saccade or unintentional browsing.

[0067] In one embodiment, the dispersion can be calculated using a distance dispersion algorithm, which calculates the distance between each eye-tracking sampling point and all other sampling points, and selects the maximum distance from them.

[0068] When the calculation results show that the user's gaze dispersion is less than the preset dispersion threshold within the sliding time window, it is determined that the user has taken that area as the target area, and the user's gaze behavior toward the target area is determined.

[0069] In one embodiment, the dispersion threshold has a lower limit of 0.7° and an upper limit of 1.6°; specifically, 1° can be used as the dispersion threshold. Based on the user's location, the user's eye position, the media resource display terminal position, and the sampling point display position, geometric calculations using trigonometric functions are performed to determine the actual distance below the dispersion threshold as the basis for judgment.

[0070] Gaze behavior can serve as a basis for subsequently determining whether a user has selected a certain interactive element, thereby triggering the display of a second media resource associated with that interactive element. This embodiment proposes to reflect the stability of the user's gaze through dispersion and to identify gaze behavior, thereby improving the accuracy of gaze behavior identification, helping to distinguish between intentional gaze and accidental glances, and thus enhancing the reliability of eye-controlled interaction.

[0071] Optionally, after displaying the second media resource associated with the target interactive element, the method further includes: determining, based on the gaze behavior, that the user has selected a new target interactive element through gaze; stopping the display of the current second media resource and switching to display the second media resource associated with the new target interactive element.

[0072] This embodiment proposes that during the display of the first media resource, the user's eye-tracking data will be continuously or periodically collected and analyzed to obtain gaze behavior. Based on the above gaze behavior, it can be determined whether the user's interest shifts during the display of the second media resource, so as to determine whether a new second media resource needs to be displayed.

[0073] Specifically, a "new target interactive element" refers to another interactive element that is different from the previously selected interactive element. The determination process still follows the general mechanism in the aforementioned embodiments, that is, by analyzing the user's gaze behavior towards a certain interactive element, it is determined whether it meets the selection criteria.

[0074] Once the user has selected the new target interactive element, the system immediately performs two operations: (1) Stop displaying the currently playing or displayed secondary media resource; (2) Switch the display of the second media resource associated with the new target interactive element.

[0075] For example, if an explanation of an interactive element is being played, and the user's attention is focused on a new interactive element, the current explanation should be stopped and the explanation of the new interactive element should be played instead.

[0076] The aforementioned "stop" and "switch" operations constitute a continuous media resource update process, ensuring that the second media resource corresponding to the user's most recently selected interactive element is always displayed with the highest priority. This mechanism allows users to freely browse different interactive elements within the first media resource and dynamically and instantly obtain multimedia information matching their current focus, improving the smoothness of the interaction.

[0077] Optionally, after displaying the second media resource associated with the target interactive element, the method further includes: detecting the user's physiological behavior data, which includes at least one of eye movements, head posture, and body position; and dynamically adjusting the display parameters of the second media resource associated with the target interactive element based on the physiological behavior data, which includes at least one of playback rate, volume, screen scaling ratio, sound field spatial position, and screen rendering perspective.

[0078] This embodiment proposes that after starting to display the second media resource associated with the target interactive element, further operations such as sensing and responding to the user's state are performed.

[0079] Specifically, data reflecting a user's current physical state or interaction intent is considered as the user's physiological behavior data. This physiological behavior data includes at least one of the following: eye movements, head posture, and body position. Eye movements may include blinking frequency, pupil changes, or eyelid closure degree; head posture may include pitch angle, yaw angle, or roll angle; and body position may include the spatial coordinates or orientation of the user's torso relative to the display device. This data can be acquired through corresponding sensors integrated into the terminal device (such as inertial measurement units, depth cameras, eye trackers, etc.) or obtained through image recognition.

[0080] Based on the detected physiological behavior data, the display parameters used to control the presentation mode of the second media resource associated with the target interactive element are dynamically adjusted, including at least one of the following: playback rate, volume, screen scaling ratio, sound field spatial position, and screen rendering perspective.

[0081] In one embodiment, for example, when a user's head is detected slowly approaching a virtual exhibit, the screen zoom ratio can be automatically increased or the rendering view can be adjusted to provide a more detailed viewing angle. In another embodiment, when it is detected that the user has not blinked for a long time or has kept their head in a stable orientation, the volume can be increased appropriately or the current playback rate can be maintained to enhance the information delivery. In another embodiment, when a lateral movement of the user's body position is detected, the spatial position of the sound field can be adjusted accordingly to keep the audio source consistent with the virtual exhibit in three-dimensional space.

[0082] Through the above mechanism, the system can optimize the presentation of secondary media resources in real time based on the user's natural physiological behavior without requiring any additional explicit user operation, making multimedia presentations more adaptable and immersive.

[0083] The third embodiment of this application provides a method for displaying traditional Chinese paintings based on the eye-tracking interaction-based multimedia resource display method described in the foregoing embodiments.

[0084] Specifically, this embodiment is based on a VR device (VR HMD) with integrated eye-tracking functionality. In this embodiment, the eye-tracking sampling rate is greater than or equal to 60Hz, and the recognition accuracy is better than 0.5°.

[0085] In this embodiment, the first media resource is a virtual museum constructed based on an exhibition of traditional Chinese paintings. The virtual museum scene is built using high-precision modeling and rendering technology, including exhibition space, exhibit wall displays, ambient lighting, interactive controls, and virtual exhibits. The virtual exhibits can be high-definition digital versions of traditional Song Dynasty landscape paintings. Visual elements in the paintings are pre-defined as interactive elements (also known as Areas of Interest or AOIs) based on expert opinions. For example, distant mountains, rivers, thatched cottages, figures, and calligraphic inscriptions in the paintings are all set as different interactive elements.

[0086] In this embodiment, the second media resource is an audio file that explains the artwork and is associated with the aforementioned interactive elements, and is pre-generated based on information such as the artwork's historical background, technical features, and cultural symbolism.

[0087] like Figure 2 As shown, when the method is executed, the user wears a VR device to enter a virtual museum, and the user's eye movement data is collected and processed in real time.

[0088] Based on the foregoing embodiments, this embodiment can use three eye-tracking interaction methods, referred to in this example as implicit mode, explicit mode, and a combination of implicit and explicit modes. In this embodiment, the interaction method can be selected by the user. Figure 3 The diagram shown illustrates three interaction methods.

[0089] In an implicit mode, eye-tracking data is analyzed in real time. A sliding time window is used to calculate the dispersion of continuous eye-tracking sampling points, with each 100ms segment of eye-tracking sampling points serving as a time window. If a user's eye-tracking sampling point set for a particular area of ​​interest (AOI) meets a preset dispersion threshold, such as 1°, it is considered an expression of interest in that area, and the associated audio explanation for that AOI is played. For example, if a user naturally gazes at a distant mountain area in a painting for more than 100ms, the system automatically triggers its associated audio explanation: "The mountain range here is depicted using the 'high distance' technique, reflecting the transcendent pursuit of the literati." In the explicit mode, the emphasis is on the user's active choice. A semi-transparent 3D speaker icon is generated as a static button in the center of the AOI. As the user looks at the speaker icon, it gradually fills with an opaque color to create visual feedback. If the user continues to look at it for a preset dwell time threshold of 600ms, the associated audio narration is triggered. For example, if the user looks at the speaker icon displayed in the pine tree area for more than 600ms, its associated audio narration will be triggered: "Pine trees symbolize resilience and longevity, and are a common image in Song Dynasty landscape paintings." This method allows users to actively determine the order and depth of information acquisition, avoiding information interference caused by automatic triggering, making it suitable for teaching and guided tour scenarios.

[0090] In the combination of implicit and explicit methods, the system employs a two-stage triggering mechanism. When the user's gaze dispersion on a certain area of ​​interest (AOI) is less than 1° within a 100ms sliding time window, the system generates a semi-transparent 3D speaker icon as a static button in that area. The audio narration does not play immediately; the user needs to look at the speaker icon again and remain for 600ms to confirm their intention to play. Only after confirmation does the system trigger the audio narration associated with the speaker icon. For example, if the speaker icon appears after the user looks at a water ripple area, the system will only trigger its associated audio narration if the user looks at the speaker icon again for 600ms: "The water ripples in the painting are depicted using a net-like texture technique, showing the flow and resilience of the river." This mechanism effectively avoids accidental triggering while maintaining the user's sense of free exploration during the natural viewing process.

[0091] The fourth embodiment of this application also proposes a multimedia resource display device based on eye-tracking interaction, such as... Figure 4 As shown, the device includes: The first display unit 410 is used to display a first media resource including at least one interactive element, and the association between the interactive element and the second media resource is pre-configured. The eye-tracking analysis unit 420 is used to collect eye-tracking data of the user while viewing the first media resource during the display of the first media resource, and to analyze the eye-tracking data to obtain the user's gaze behavior; The interactive display unit 430 is used to determine, based on the gaze behavior, that the user has selected a target interactive element by gazing, and to display a second media resource associated with the target interactive element.

[0092] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0093] In this embodiment, the multimedia resource display device based on eye-tracking interaction is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0094] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application, such as... Figure 5As shown, the computer device includes one or more processors 510, memory 520, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 5 Take the 510 processor as an example.

[0095] Processor 510 may be a central processing unit, a network processor, or a combination thereof. Processor 510 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0096] The memory 520 stores instructions executable by at least one processor 510 to cause the at least one processor 510 to perform the method shown in the above embodiments.

[0097] The memory 520 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 520 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 520 may optionally include memory remotely located relative to the processor 510, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0098] The memory 520 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 520 may also include a combination of the above types of memory.

[0099] The computer device also includes a communication interface 530 for communicating with other devices or communication networks.

[0100] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0101] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method of any embodiment of this application.

[0102] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

[0103] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0104] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0105] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0106] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0107] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0108] It is understood that in the specific implementation of this application, data such as user information, location information, and navigation data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0109] The methods, apparatus, computer devices, computer-readable storage media, and computer program products described in the above embodiments can be implemented by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0110] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, computer devices, computer-readable storage media, and computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-readable program code.

[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, computer devices, computer-readable storage media, and computer programs according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0116] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, computer equipment, computer-readable storage media, and computer programs are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0117] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0118] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for displaying multimedia resources based on eye control interaction, characterized in that, The method includes: The display includes a first media resource comprising at least one interactive element, the association between the interactive element and a second media resource being pre-configured; During the display of the first media resource, eye movement data of the user while viewing the first media resource is collected, and the eye movement data is analyzed to obtain the user's gaze behavior; Based on the gaze behavior, it is determined that the user has selected a target interactive element among the interactive elements by gazing, and a second media resource associated with the target interactive element is displayed.

2. The method of claim 1, wherein, The step of determining, based on the gaze behavior, that the user has selected a target interactive element from the interactive elements through gaze includes: Determine that the target position of the user's gaze behavior falls within the range of the currently gazed interactive element within a preset sliding time window, and generate an interactive icon on the currently gazed interactive element. The first gaze duration is obtained based on the user's gaze behavior toward the interactive icon; When the first gaze duration is greater than or equal to a preset first gaze duration threshold, the interactive element selected by the user during the current gaze is determined to be the target interactive element.

3. The method of claim 1, wherein, The step of determining, based on the gaze behavior, that the user has selected a target interactive element from the interactive elements through gaze includes: Determine that the target position of the user's gaze behavior within a preset sliding time window falls within the range of the currently gazed interactive element; determine that the interactive element currently gazed upon by the user is the target interactive element; or, Based on the user's gaze behavior toward the currently gazed interactive icon, a second gaze duration is obtained, and the currently gazed interactive icon is pre-set on the interactive element; when the second gaze duration is greater than or equal to a preset second gaze duration threshold, it is determined that the user has selected the currently gazed interactive element as the target interactive element.

4. The method according to claim 2 or 3, characterized in that, The interactive icon is in its initial state before being viewed by the user; The method further includes: As the first gaze duration or the second gaze duration gradually increases, a transition animation is played, which is used to show the visual transition of the interactive icon from the initial state to the final state. When the first gaze duration or the second gaze duration is greater than or equal to the first duration threshold or the second duration threshold, the transition animation finishes playing, and an interactive icon of the final state is generated on the currently gazed interactive element.

5. The method of claim 1, wherein, The analysis of the eye-tracking data to obtain the user's gaze behavior includes: The eye-tracking data is sampled within a preset sliding time window to obtain a set of sampling points; Calculate the dispersion of the set of sampling points, the dispersion characterizing the stability of the user's visual axis during the sliding window; When the dispersion is less than a preset dispersion threshold at the target location, the user's gaze behavior toward the target location within the sliding time window is determined.

6. The method of claim 1, wherein, After displaying the second media resource associated with the target interactive element, the method further includes: Based on the gaze behavior, it is determined that the user selected a new target interactive element from the interactive elements by gazing. Stop displaying the current second media resource and switch to displaying the second media resource associated with the new target interactive element.

7. The method according to claim 1, characterized in that, After displaying the second media resource associated with the target interactive element, the method further includes: The user's physiological behavior data is detected, and the physiological behavior data includes at least one of eye movements, head posture, and body position; Based on the physiological behavior data, the display parameters of the second media resource associated with the target interactive element are dynamically adjusted. The display parameters include at least one of the following: playback rate, volume, screen scaling ratio, sound field spatial position, and screen rendering perspective.

8. A multimedia resource display device based on eye-tracking interaction, characterized in that, The device includes: The first display unit is used to display a first media resource including at least one interactive element, wherein the association between the interactive element and the second media resource is pre-configured; An eye-tracking analysis unit is used to collect eye-tracking data of a user while viewing the first media resource during the display of the first media resource, and to analyze the eye-tracking data to obtain the user's gaze behavior; An interactive display unit is used to determine, based on the gaze behavior, that the user has selected a target interactive element among the interactive elements by gazing, and to display a second media resource associated with the target interactive element.

9. A computer device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the multimedia resource display method based on eye-tracking interaction as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes computer instructions for causing a computer to execute the multimedia resource display method based on eye-tracking interaction as described in any one of claims 1 to 7.

Images