System for visualizing gaze points on dynamic multi-window displays using eye-trackers

Abstract

Apparatuses, methods, and systems can be configured to track user visual interaction with one or more areas of interest of a graphical image displayed on a display of a user device. Embodiments can include an apparatus having a processor communicatively connected to a non-transitory computer-readable medium. The computer-readable medium can have code stored thereon such that the telecommunication apparatus is configured to track user visual interaction with one or more areas of interest of a graphical image displayed on a display screen.

Description

SYSTEM FOR VISUALIZING GAZE POINTS ON DYNAMIC MULTI-WINDOWDISPLAYS USING EYE-TRACKERSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application is related to and claims the benefit of priority of U.S. Provisional Application 63 / 667,487, filed on July 3, 2024, the entire contents of which are incorporated by reference.STATEMENT REGARDING FEDERALLY FUNDED RESEARCH

[0002] This invention was made with government support under Grant No. DE-SC0021757 awarded by the Department of Energy. The Government has certain rights in the invention.FIELD

[0003] Embodiments relate to apparatuses, methods, and systems configured to track user visual interaction with one or more areas of interest (AO1, e.g., a button, display element, a word) displayed on a dynamic display (e g., a touch screen display, monitor, television, etc.). In particular, embodiments relate to a computer-implemented method for tracking user visual interaction with one or more areas of interest of a graphical image displayed on a display of a user device, including pre-defining one or more areas of interest of the graphical image, storing at least one screenshot of each pre-defined area, performing at least one of a pattern detection (e.g., pattern matching, pattern-matching evaluation, etc.) or a hyper-text markup language (HTML) visual element tracking (e.g., HTML interface evaluation), logging attention data for the area of interest, and storing the logged attention data to generate at least one report identifying a duration of time that the user visually interacted with the pre-defined area of interest.BACKGROUND

[0004] Eye tracking is a method used to investigate eye movements, gaze behavior, and pupil dilation that is becoming more popular in research and industry. Many different research fields are currently benefiting from eye tracking (e.g., perception, attention, memory, reading, psychopathology, ophthalmology, neuroscience, human-computer interaction, animal research, human factors, consumer behavior, optometry, etc.). Besides research, eye trackers are implemented in many commercially available technologies, such as laptops and augmented reality headsets in different industries such as healthcare, medicine, marketing, and advertising.

[0005] Eye trackers provide information on the coordinates of where the user is looking at (eye gazes and fixation), the logs of rapid motion of the eye from one fixation to another (saccades), pupil dilation, pupil size, etc. This information can help researchers and scientists to understand user behavior. As eye tracking technologies are becoming a ubiquitous technology in more commercial devices, there is an increasing demand for more effective tools for data collection and analysis.SUMMARY

[0006] Currently, there are two kinds of eye-tracking applications: proprietary applications that come with the eye tracker, such as Tobbi Pro Lab and GazePoint Analysis, or applications by third-party companies that work with a variety of eye trackers such as iMotions. These tools offer different user interfaces, each with their own strengths and weaknesses. It is the responsibility of the user to assess which software is compatible with their eye tracker and most appropriate for their particular application.

[0007] Traditionally, eye-tracking applications were designed to work with static environments, or environments where elements are fixed on the screen and areas of interest always have the same coordinates during an eye-tracking session. Conversely, dynamic environments are environments in which the elements on the screen do not stay in their place and / or change size, appear / di sappear over time and / or in response to user inputs. With the ever-growing use of dynamic environments, these applications have been trying to adapt their software to dynamic environments. Their approach, however, is not the most user-friendly, practical, and real-time option. Current approaches require the experimenter to determine frame by frame how the area of interest (AOI) is moving, changing size, and when it is not on the screen. In the current state of eye-tracking software, the typical way to define dynamic AOIs (AOIs that move) is to redefine them in every single frame of a recorded video of visual interaction. For instance, for a 2-minute video with 60 frames per second, the location of the AOIs in 7200 frames needs to be determined manually. Some companies have tried interpolation, meaning there is no need to determine the location of the AOIs in all frames but only in key frames and their locations between these key frames are inferred. However, an analyst needs to review the video frame by frame and determine the location of the AOIs for changes in size and to define where and if the locations still exist manually. Therefore, the quality of its process depends primarily on human moderation.

[0008] We have developed methods and systems to automate the process of tracking a user’s gaze, or visual interaction, with areas of interest (AOIs) displayed in a dynamic display environment during a session of use. The methods and systems allow a monitor (e.g., an experimenter / researcher) to pre-define one or more areas of interest on the display, such as by selecting a window within the display as an area of interest, and / or taking a screenshot of an areaof interest. The location of the pre-defined area(s) of interest on the display, if present, is then determined by a pattern detection on the display and / or an HTML visual element tracking, which allows for determining locations of AOIs after the display has dynamically changed or as the display is dynamically changing (e.g., the display has been / is being scrolled, re-sized, etc.).

[0009] The systems and methods then use eye-tracking data (e.g., gaze coordinates and time) to determine which AOIs, if any, the user visually interacted with (e.g., if the gaze coordinates are located in an AOI), at what time the user visually interacted with the AOI, and for how long, also called dwell time or fixation time, a user visually interacted with the AOI. The location of each AOI within the dynamically changing display and user visual interaction with the AOI(s) can be tracked and / or monitored by an experimenter (e.g., researcher, monitor, etc.) on one or more alternate displays (e.g., one or more remote displays) in real-time, and outputs containing eyetracking data, user input data (e.g., mouse clicks, keyboard strokes, etc.), and AOI data can be generated for later analysis of the user’s visual interaction with the display during a use session.

[0010] In an exemplary embodiment, a computer-implemented method for tracking a user’s visual interaction with one or more areas of interest in a graphical image displayed on a user device’s display is provided. The method can include pre-defining, via one or more user inputs, one or more areas of interest of the graphical image; storing at least one screenshot of each predefined area of interest in a memory and / or associating each pre-defined area of interest with hypertext markup language (HTML) code stored in a file that is run to generate the graphical image displayed on display; performing, via a processor, at least one of a pattern detection or an HTML visual element tracking; and storing logged attention data in a non-transitory computer- readable medium to generate at least one report identifying each pre-defined area of interest that the user visually interacted with and / or a duration of time that the user visually interacted withthe pre-defined area of interest. The pattern detection can include comparing one or more of the at least one screenshot stored in the memory to the graphical image displayed on the display of the user device to identify a pattern match of the at least one pre-defined area of interest of the screenshot with a portion of the graphical image displayed on the display; and in response to identifying the pattern match, logging attention data for the area of interest of the pattern match. The HTML visual element tracking can include, in response to determining that the area of interest is displayed in the graphical image displayed on the display via evaluation of the HTML code being run to generate the graphical image displayed on the display, logging attention data for the at least one area of interest associated with the HTML code being run to generate the graphical image displayed on the display.

[0011] In some embodiments, the pattern detection can be performed while the graphical image displayed on the display is modified.

[0012] In some embodiments, the graphical image displayed on the display can be modified multiple times during a session of use by the user device and the pattern detection can be performed for the entirety of the session.

[0013] In some embodiments, the HMTL visual element tracking can be performed while the graphical image displayed on the display is modified.

[0014] In some embodiments, the graphical image displayed on the display can be modified multiple times during a session of use by the user device and the HMTL visual element tracking can be performed for the entirety of the session.

[0015] In some embodiments, the attention data can include a list of each area of interest that the user visually interacted with during the session of use, a timestamp for each time that the uservisually interacted with each area of interest, and a dwell time for each visual interaction that the user had with each area of interest.

[0016] In some embodiments, the attention data can be based on gaze location data. Gaze location data can include one or more locations within the graphical image displayed on the display that the user visually interacted with during the session of use and a timestamp for each time that the user visually interacted with each of the one or more locations within the graphical image displayed on the display. In some embodiments, the gaze location data can be based on sensor data generated by one or more sensors configured to track a user’s visual interaction with the graphical image displayed on the display.

[0017] In another exemplary embodiment, a telecommunication apparatus for tracking user visual interaction with one or more areas of interest of a graphical image displayed on a display is provided. The apparatus can include a processor communicatively connected to a non- transitory computer-readable medium. The computer-readable medium can have code stored thereon that can configure the apparatus to pre-define, via one or more user inputs, one or more areas of interest of the graphical image; store at least one screenshot of each pre-defined area of interest in a memory and / or associate each pre-defined area of interest with hypertext markup language (HTML) code stored in a file that is run to generate the graphical image displayed on display; perform, via a processor, at least one of a pattern detection or an HTML visual element tracking; and store logged attention data in a non-transitory computer-readable medium to generate at least one report identifying each pre-defined area of interest that the user visually interacted with and / or a duration of time that the user visually interacted with the pre-defined area of interest. The pattern detection can include comparing one or more of the at least one screenshot stored in the memory to the graphical image displayed on the display of the userdevice to identify a pattern match of the at least one pre-defined area of interest of the screenshot with a portion of the graphical image displayed on the display; and in response to identifying the pattern match, logging attention data for the area of interest of the pattern match. The HTML visual element tracking can include, in response to determining that the area of interest is displayed in the graphical image displayed on the display via evaluation of the HTML code being run to generate the graphical image displayed on the display, logging attention data for the at least one area of interest associated with the HTML code being run to generate the graphical image displayed on the display.

[0018] In some embodiments, the pattern detection can be performed while the graphical image displayed on the display is modified.

[0019] In some embodiments, the graphical image displayed on the display can be modified multiple times during a session of use by the user device and the pattern detection can be performed for the entirety of the session.

[0020] In some embodiments, the HMTL visual element tracking can be performed while the graphical image displayed on the display is modified.

[0021] In some embodiments, the graphical image displayed on the display can be modified multiple times during a session of use by the user device and the HMTL visual element tracking can be performed for the entirety of the session.

[0022] In some embodiments, the attention data can include a list of each area of interest that the user visually interacted with during the session of use, a timestamp for each time that the user visually interacted with each area of interest, and a dwell time for each visual interaction that the user had with each area of interest.

[0023] In some embodiments, the attention data can be based on gaze location data. Gaze location data can include one or more locations within the graphical image displayed on the display that the user visually interacted with during the session of use and a timestamp for each time that the user visually interacted with each of the one or more locations within the graphical image displayed on the display. In some embodiments, the gaze location data can be based on sensor data generated by one or more sensors configured to track a user’s visual interaction with the graphical image displayed on the display.

[0024] In some embodiments, the telecommunication apparatus can be a user device.

[0025] In some embodiments, the telecommunication apparatus can be a computer device that hosts a service that is communicably connectable to at least one user device for generation of the graphical image to be displayed on a display of the at least one user device.

[0026] In some embodiments, associating each pre-defined area of interest with HTML code stored in a file that is run to generate the graphical image displayed on the display can include selecting at least one window of the graphical image as an area of interest.

[0027] In another exemplary embodiment, a non-transitory computer-readable medium having code stored thereon, the code defining a method performed by a computer device when a processor of the computer device runs the code, is provided. The method performed by the computer device can include pre-defining, via one or more user inputs, one or more areas of interest of the graphical image; storing at least one screenshot of each pre-defined area of interest in a memory and / or associating each pre-defined area of interest with hypertext markup language (HTML) code stored in a file that is run to generate the graphical image displayed on display; performing, via a processor, at least one of a pattern detection or an HTML visual element tracking; and storing logged attention data in a non-transitory computer-readable medium togenerate at least one report identifying each pre-defined area of interest that the user visually interacted with and / or a duration of time that the user visually interacted with the pre-defined area of interest. The pattern detection can include comparing one or more of the at least one screenshot stored in the memory to the graphical image displayed on the display of the user device to identify a pattern match of the at least one pre-defined area of interest of the screenshot with a portion of the graphical image displayed on the display; and in response to identifying the pattern match, logging attention data for the area of interest of the pattern match. The HTML visual element tracking can include, in response to determining that the area of interest is displayed in the graphical image displayed on the display via evaluation of the HTML code being run to generate the graphical image displayed on the display, logging attention data for the at least one area of interest associated with the HTML code being run to generate the graphical image displayed on the display.

[0028] Other details, objectives, and advantages of a telecommunication apparatus, telecommunication system, process for tracking user visual interaction with one or more areas of interest of a graphical image displayed on a display, and methods of making and using the same will become apparent as the following description of certain exemplary embodiments thereof proceeds.BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other objects, aspects, features, advantages, and possible exemplary applications of the present innovation will be more apparent from the following more particular description thereof, presented in conjunction with the following drawings. Like reference numbers used in the drawings may identify like components.

[0030] FIG. 1 is a block diagram of a first exemplary embodiment of a telecommunication apparatus 1 for tracking user visual interaction with one or more areas of interest of a graphical image displayed on a display. An embodiment of the telecommunication apparatus 1 can include an exemplary embodiment of a visual interaction monitoring system 5 and / or an exemplary embodiment of a computer device 3.

[0031] FIG. 2 is a block diagram of a second exemplary embodiment of a telecommunication apparatus 1 for tracking user visual interaction with one or more areas of interest of a graphical image displayed on a display. An embodiment of the telecommunication apparatus 1 can include an exemplary embodiment of a user device 13 and / or monitor device 15, which can be used in embodiments of the telecommunication apparatus 1 and visual interaction monitoring system 5.

[0032] FIG. 3 is a block diagram of a third exemplary embodiment of a telecommunication apparatus 1 for tracking user visual interaction with one or more areas of interest of a graphical image displayed on a display. An embodiment of the telecommunication apparatus 1 can include an exemplary embodiment of a user device 13, a monitor device 15, and a server device 17, which can be used in embodiments of the telecommunication apparatus 1 and the visual interaction monitoring system 5.

[0033] FIG. 4 is a schematic block diagram of an exemplary embodiment of a display 7 configured to display a graphical image 9 having one or more pre-defined areas of interest 11.

[0034] FIG. 5 is a schematic block diagram of the display of FIG. 5 depicting the graphical image 9 displayed on the display 7 after a user has altered (e g., scrolled) the graphical image 9 displayed on the display 7.

[0035] FIG. 6 is a flow chart illustrating an exemplary embodiment of a method for tracking user visual interaction with one or more areas of interest of a graphical image displayed on a display.DETAILED DESCRIPTION

[0036] The following description is of exemplary embodiments that are presently contemplated for carrying out the present invention. This description is not to be taken in a limiting sense but is made merely for the purpose of describing the general principles and features of the present invention. The scope of the present invention is not limited by this description.

[0037] Referring to FIGS. 1-4, a telecommunication apparatus 1 for tracking user visual interaction with one or more areas of interest 11 (AOI(s)) of a graphical image 9 displayed on display 7 can include at least one computer device 3 including a processor 3a (Pre.) communicatively connected to a non-transitory computer-readable memory 3c (Mem.) and at least one transceiver 3b (TRCVR). The graphical image can include text content, graphs, artwork, interactive objects such as buttons, or other visual indicia. In some embodiments, the graphical image can be a graphical user interface (GUI) having visual content displayed, for example.

[0038] The transceiver 3b can include a local area network transceiver, a wide area network transceiver, a Wi-Fi transceiver, a Bluetooth transceiver, a cellular network transceiver, and / or other type of transceiver or combination of transceivers. The type of communicative connection(s) facilitated by at least one transceiver 3b can facilitate the collection and storage of user visual interaction data (e.g., gaze location data, attention data, etc.) and the processing and / or providing of that data to a user and / or other computer device 3.

[0039] The non-transitory memory 3c can include flash memory, a hard drive, a solid-state drive, or other non-transitory memory. In some embodiments, the non-transitory memory 3c can include a type of non-transitory computer-readable medium that is accessible to the processor 3a, allowing the processor to run code stored in the memory 3c, for example.

[0040] The processor 3a can include a central processing unit, microcontroller, core processor, array of processors, or other type of hardware processor. The memory 3c can be communicatively connected to the processor 3a so that the processor 3a can run at least one application (App) stored in the memory 3c. The memory 3c can also have at least one data store (DS), which can include code, gaze location data, attention data, pre-defined AOIs, files, databases, libraries, and / or other types of stored data that may be used when the processor 3a runs the code of one or more applications (App). For instance, the processor 3a can process code of an application (App) to run the application to perform a pre-defined method in accordance with programming code for the application that defines the method to be performed.

[0041] In an exemplary embodiment, the telecommunication apparatus 1 can be configured to track a user’s visual interaction with one or more AOIs 11 of a graphical image 9 displayed on a display 7 by analyzing gaze location data, e.g., by the processor 3 a, to determine if a user visually interacted with any of the one or more AOIs 11. In some embodiments, the graphical image 9 can be generated and displayed on the display 7 by the processor 3a, such as by code stored in the memory 3c and run by the processor 3a. In some embodiments, the graphical image 9 can be generated by a different computer device 3 (e.g., a smartphone, a computer, a server, array of servers, cloud server, etc.) that is communicatively connected with the telecommunication apparatus 1 and the processor 3a can receive data from the different computer device 3 that causes the processor 3a to display the graphical image 9 on the display 7. The display 7 can be any suitable type of display for displaying graphical images (e.g., screen, touch screen, monitor, television, laptop computer, tablet, etc.) and the graphical image 9 can be any type of graphical image 9 that a user may view and / or interact with, such as webpages,images, operating system interfaces, user interfaces, a GUI, or any other suitable graphical image that can be displayed on the display 7.

[0042] The telecommunication apparatus 1 can include or be communicatively connected to one or more sensors S configured to track a user’s eye movement during a use session in which the user visually interacts with the graphical image 9 displayed on the display 7. The sensor(s) S can be any suitable sensor for tracking eye movement, such as an eye-tracking camera, a mobile eye tracker, a head-stabilized eye-tracker, etc., and can be configured to transmit sensor data to the processor 3a. The sensor(s) S can also include other sensors that may collect data related to a user's eye movement as the user may look at the graphical image 9 shown on the display 7.

[0043] The processor 3a can be configured to run an application and / or process code (e.g., Apps and / or code stored in memory 3c) configured to generate gaze location data that can include coordinates (e.g., x, y coordinates) of each location of the graphical image 9 / display 7 with which the user visually engaged, and / or timestamp data (e.g., the time(s) that the user visually interacted with each location / coordinates) based on the sensor data. For example, the processor 3a can be configured to host and / or process code for a vision-tracking software application, such as Tobbi Pro Lab, Gazepoint Analysis, or any other suitable eye-tracking applications / means, that can use the sensor data from the one or more sensors S to generate the gaze location data. In some embodiments, the gaze location data can be stored, e.g., in the memory 3c, and can be retrieved by the processor 3a for analysis and / or processing of the gaze location data.

[0044] The telecommunication apparatus 1 can include or be communicatively connected with at least one input device ID (e.g., a keyboard, a pointer device, a button, a touch screen display, etc.) to receive input from one or more users and can also include or be communicatively connected to at least one output device OD (e.g., a speaker, a display, a touch screen display, aprinter, etc.) to provide output to the user(s). The processor 3a can be configured to receive user input(s) via the at least one input device (ID). For example, the processor 3 a can receive one or more user inputs to pre-define and / or interact with one or more AOIs 11 of one or more graphical images 9 displayed on one or more displays 7.

[0045] The telecommunication apparatus 1 can be configured to receive, e.g., via the communicative connection CC between the input device(s) ID and the processor 3a, one or more user inputs to pre-define each of the one or more AOIs 11 of the graphical image 9 displayed on the display 7. In some embodiments, the one or more user inputs to pre-define one or more AOIs 11 can be a screenshot selection, (e.g., via a mouse, touchscreen, etc.) of each AOI 11 located in the graphical image 9. In other words, a user (e.g., a user whose visual interaction is being, or going to be, tracked) and / or a monitor (e.g., a monitor / researcher who is observing, or going to observe, the user’s visual interaction) can use at least one input device ID to pre-define one or more AOIs 11 located in the graphical image 9 to take a screenshot of each AOI 11. In some embodiments, the one or more user inputs to pre-define one or more AOIs 11 can be a selection (e.g., via a mouse, touchscreen, etc.) of an application window, e.g., an operating system application window, that is located in the graphical image 9 displayed on the display 7.

[0046] The processor 3a can be configured to store, in the memory 3c, one or more screenshots for each AOI 11 pre-defined via a screenshot selection user input and / or to associate each AOI pre-defined via a window selection user input with hyper-text markup language (HTML) code stored in a file that is run to generate the graphical image 9 displayed on the display 7. In some embodiments, one or more AOIs 11 can be pre-defined prior to the beginning of a use session. In some embodiments, one or more AOIs 11 can be pre-defined during a use session.

[0047] In an exemplary embodiment, the telecommunication apparatus 1 can be configured to identify the existence of and / or location of one or more AOIs 11 in the graphical image 9 displayed on the display 7 by performing a pattern detection (e.g., pattern matching, patternmatching evaluation, etc.) and / or an HTML visual element tracking (e.g., HTML interface evaluation). For example, the processor 3a can run code (e.g., code stored in the memory 3c) that causes the processor 3a to perform the pattern detection and / or HTML visual element tracking.

[0048] The pattern detection (e.g., pattern matching, pattern-matching evaluation, etc.) can identify the existence and / or location of one or more pre-defined AOIs 11 in the graphical image 9 by parsing the graphical image 9 (e.g., taking a screenshot of the entire graphical image 9 displayed on the display 7) and comparing the parsed graphical image 9 to the stored one or more screenshots for each pre-defined AOI 11 to determine if one or more pattern matches exist between the parsed graphical image 9 and the stored one or more screenshots for each predefined AOI 11. In some embodiments, the processor 3a can run a pattern-matching algorithm, such as a scale invariant feature transform (SIFT) algorithm, to determine key points, and generate an associated descriptor for each key point, of the parsed graphical image 9 and each of the one or more screenshots for each pre-defined AOI 11, and then compare the descriptors for each key point of the parsed graphical image 9 and the one or more screenshots of each predefined AOI 11 to determine the existence and / or location of one or more pattern matches in the graphical image 9 displayed on the display 7. In some embodiments, the pattern detection can use one or more pattern detection / image matching models (e.g., Al models, neural networks, etc.), such as GRIM, to determine if one or more pattern matches exist between the parsed graphical image 9 and the stored one or more screenshots for each pre-defined AOI 11.

[0049] In other words, the pattern detection can determine the existence and / or location of one or more pre-defined AOIs 11 in the graphical image 9 displayed on the display 7 by determining one or more pattern matches between the descriptors of each key point of the graphical image 9 and the descriptors of each key point of each pre-defined AOI 11. The pattern detection (e.g., pattern matching, pattern-matching evaluation, etc.) can be performed continuously, e.g., via the processor 3a, for at least a portion of the use session, to identify the existence and / or location of one or more pre-defined AOIs 11 in the graphical image 9. For example, in some embodiments, the pattern detection can be performed at a pre-determined and / or user-specified time interval, such as every 50 milliseconds (ms), every 100 ms, every second, every 10 seconds, etc. It should be understood that the above time intervals are merely exemplary and that any suitable time interval can be used.

[0050] For example, the pattern detection (e.g., pattern matching, pattern-matching evaluation, etc.) can use one or more searching and / or comparison algorithms, such as approximate nearest neighbor, to compare the descriptors of the graphical image 9 and the descriptors of each predefined AOI 11 to find pattern matches between the graphical image 9 displayed on the display 7 and one or more pre-defined AOIs 11. In an exemplary embodiment, the pattern detection can use Fast Library for Approximate Nearest Neighbor (FLANN) to generate an index (e g., in the memory 3c) for storing and searching / comparing the descriptors to determine pattern matches between the one or more pre-defined AOIs 11 and the parsed graphical image 9 displayed on the display 7. For example, pattern matches between the parsed graphical image 9 and the one or more pre-defined AOIs 11 can be determined based on a Euclidean distance (e.g., in the index) between a descriptor of the parsed graphical image 9 and one or more descriptors of the one or more pre-defined AOIs 11.

[0051] Each key point is characterized by its location, scale, and orientation, relative to the graphical image 9 displayed on the display 7, and each descriptor provides a detailed and invariant representation of its associated key point, such that pattern matches between the graphical image 9 and one or more pre-defined AOIs 11 can be determined in dynamic display conditions, such as a change in location, orientation, size / scale, etc., of the one or more AOIs 11. For example, as shown in FIGS. 4 and 5, as a user interacts (e.g., scrolls, zooms in, zooms out, etc.) with the graphical image 9 displayed on the display 7, the location and / or size of one or more AOIs 11 and / or the total number of AOIs 11 in the graphical image 9 displayed on the display 7 can dynamically change. However, because the descriptors associated with each key point of each AOI 11 are invariant to location, orientation, size / scale, etc., the pattern detection (e.g., pattern matching, pattern-matching evaluation, etc.) can determine a pattern match between a pre-defined AOI 11 and the parsed graphical image 9 displayed on the display 7 regardless of dynamic changes that may have occurred to the AOI 11 in the graphical image 9 displayed on the display 7. Because the location of each key point of the graphical image 9 displayed on the display 7 is known, the location of each pre-defined AOI 11 in the graphical image 9 can be determined based on the known location of each graphical image 9 key point for which a pattern match is determined.

[0052] In some embodiments, the one or more searching and / or comparison algorithms can use pre-defined search parameters and / or index parameters to identify / detect pattern matches. The index can be built from the descriptors of one or more of the images (e.g., the parsed graphical image 9), which can allow pattern detection algorithm to perform fast and efficient matching against the descriptors extracted from one or more comparison images (e.g., the at least one screenshot). The choice of index type and its parameters can significantly affect the speed andaccuracy of the matching process. For example, an index generated by a feature extraction algorithm (e.g., SURF) can create a structured search space that can enable faster and more accurate matching of features.

[0053] In some embodiments, the efficiency and / or accuracy of the pattern detection (e.g., pattern matching, pattern-matching evaluation, etc.) can be increased. For example, the pattern detection can include reducing the resolution and / or altering the color scheme (e.g., converting to black and white) of the parsed graphical image 9 (e.g., a screenshot of the graphical image 9) displayed on the display 7 to reduce the computational bandwidth required to perform the comparison of the parsed graphical image 9 and the one or more pre-defined AOI 11 screenshots.

[0054] In some embodiments, the pattern detection can include one or more filtering techniques, such as a Lowe’s ratio test, to increase the accuracy of determined pattern matches. For example, if the pattern detection (e.g., pattern matching, pattern-matching evaluation, etc.) determines that multiple descriptors of the one or more pre-defined AOIs 11 match with a descriptor of the parsed graphical image 9, the pattern detection can use the Lowe’s ratio test to analyze the Euclidean distance (e.g., in the index) between the AOI 11 descriptor that is the closest pattern match to the parsed graphical image 9 descriptor and the AOI 11 descriptor that is the next closest pattern match to the parsed graphical image 9 descriptor, and determine the accuracy of a pattern match based on the difference between the Euclidean distance of the descriptor that is the closest match and the Euclidean distance of the descriptor that is the next closest match. For example, as the difference between the Euclidean distance of the closest matching AOI 11 descriptor and the parsed graphical image 9 descriptor increases relative to the second closest matching AOI 11 descriptor and the parsed graphical image 9 descriptor, the accuracy of the closest-matching pattern match increases.

[0055] In some embodiments, the pattern detection (e.g., pattern matching, pattern-matching evaluation, etc.) can include logging and / or generating (e.g., via the processor 3a) attention data for each pre-defined AOI 11 determined to be pattern matched with the graphical image 9 displayed on the display 7. In an exemplary embodiment, the attention data can include a list of each pre-defined AOI 11 that was identified / determined as being located in the graphical image 9 displayed on the display 7 during a use session and / or that was interacted with by the user (e.g., via visual interaction and / or interaction via one or more user inputs received via at least one input device ID) during the use session, timestamp data for the time that the user interacted with each pre-defined AOI 11, and / or dwell (e.g., fixation) data for the total amount of time that the user spent interacting (e g., visually interacting) with each pre-defined AOI 11.

[0056] For example, logging the attention data can include comparing (e.g., by the processor 3 a) the gaze location data (e.g., gaze location data generated, stored, and / or or received by the processor 3a) and / or user input data (e.g., the location in the graphical image 9 that corresponds to one or more user inputs, such as a mouse click, touchscreen input, keyboard input, etc., that selects, clicks, or otherwise interacts with an area or window in the graphical image 9) with the location of each pre-defined AOI 11 identified as a pattern match with the graphical image 9 displayed on the display 7 by the pattern detection to determine if, at what time, and for what duration of time, the user interacted with each pre-defined AOI 11 in the graphical image 9 displayed on the display 7. In some embodiments, the logged attention data can be stored in the memory 3c (e.g., via the processor 3a) to generate (e.g., via the processor 3a) at least one output (e.g., a report, data file, visual display, user interface, etc.) identifying each pre-defined AOI 11 that the user interacted with (e.g., visual interaction and / or user input interaction), the time thatthe user interacted with each pre-defined AOI 1 1, and / or the dwell / fixation time (e.g., duration of tim) that the user interacted with each pre-defined AOI 11.

[0057] In some embodiments, the graphical image 9 displayed on the display 7 can be generated in an HTML environment (e.g., the graphical image 9 can be a website or other HTML-based graphical image 9 displayed on the display 7). In such embodiments, the HTML visual element tracking (e.g., HTML interface evaluation) can identify the existence and / or location of an AOI 11 in the graphical image 9 by evaluating, or analyzing, the HTML code being run to generate the graphical image 9 displayed on the display 7. For example, when one or more AOIs 11 are pre-defined via one or more user inputs that select one or more HTML -generated application windows, the unique HTML code associated with each pre-defined AOI 11 (e.g., the unique HTML code used to generate each application window selected as a pre-defined AOI 11) can be stored, for example, in the memory 3c, and can be associated, in the memory 3c, with its associated pre-defined AOI 11.

[0058] The HTML visual element tracking (e.g., HTML interface evaluation) can be performed continuously, e.g., via the processor 3a, for at least a portion of the use session, to evaluate the HTML code being run to generate the graphical image 9 displayed on the display 7 to determine if the unique HTML code associated with one or more pre-defined AOIs 11 is present in the HTML code being run to generate the graphical image 9 displayed on the display screen 7. The location in the graphical image 9 displayed on the display 7 of each pre-defined AOI 11 found to be present in the HTML code run to generate the graphical image 9 can be determined from the HTML code. In other words, the HTML code being run to generate the graphical image 9 displayed on the display 7 contains the location in the graphical image 9 for each window and / or element that is generated in the graphical image 9 by the HTML code.

[0059] The HTML visual element tracking (e.g., HTML interface evaluation) can determine the existence and / or location of each of the one or more pre-defined AOIs 11 in dynamic display conditions, such as a change in location, orientation, size / scale, etc., of the one or more AOIs 11. For example, as shown in FIGS. 4 and 5, as a user interacts (e.g., scrolls, zooms in, zooms out, etc.) with the graphical image 9 displayed on the display 7, the location and / or size of one or more AOIs 11 and / or the total number of AOIs 11 in the graphical image 9 displayed on the display 7 can dynamically change. However, because the HTML visual element tracking determines the existence and / or location of one or more AOIs 11 in the graphical image 9 displayed on the display 7 based on a determination that the unique HTML code associated with the one or more pre-defined AOIs 11 is present in the HTML code run to generate the graphical image 9, the HTML visual element tracking can determine a pattern match between a predefined AOI 11 and the parsed graphical image 9 displayed on the display 7 regardless of dynamic changes that may have occurred to the AOI 11 in the graphical image 9 displayed on the display 7. Because the location of each key point of the graphical image 9 displayed on the display 7 is known, the location of each pre-defined AOI 11 in the graphical image 9 can be determined based on the known location of each graphical image 9 key point for which a pattern match is determined.

[0060] In some embodiments, the HTML visual element tracking (e.g., HTML interface evaluation) can include logging and / or generating (e.g., via the processor 3a) attention data for each pre-defined AOI 1 1 determined to be located within the graphical image 9 displayed on the display 7. In an exemplary embodiment, the attention data can include a list of each pre-defined AOI 11 that was identified / determined as being located in the graphical image 9 displayed on the display 7 during a use session and / or that was interacted with by the user (e.g., via visualinteraction and / or interaction via one or more user inputs received via at least one input device ID) during the use session, timestamp data for the time that the user interacted with each predefined AOI 11, and / or dwell / fixation data for the total amount of time that the user spent interacting (e.g., visually interacting) with each pre-defined AOI 11.

[0061] For example, logging the attention data can include comparing (e.g., by the processor 3a) the gaze location data (e.g., gaze location data generated, stored, and / or or received by the processor 3a) and / or user input data (e.g., the location in the graphical image 9 that corresponds to one or more user inputs, such as a mouse click, touchscreen input, keyboard input, etc., that selects, clicks, or otherwise interacts with an area or window in the graphical image 9) with the location of each pre-defined AOI 11 identified in the HTML coded run to generate the graphical image 9 displayed on the display 7 by the HTML visual element tracking to determine if, at what time, and for what duration of time, the user interacted with each pre-defined AOI 11 in the graphical image 9 displayed on the display 7. In some embodiments, the logged attention data can be stored in the memory 3c (e.g., via the processor 3a) to generate (e.g., via the processor 3a) at least one output (e.g., a report, data file, visual display, user interface, etc.) identifying each pre-defined AOI 11 that the user interacted with (e.g., visual interaction and / or user input interaction), the time that the user interacted with each pre-defined AOI 11, and / or the dwell / fixation time (e.g., duration of time) that the user interacted with each pre-defined AOI 11.

[0062] In some embodiments, the telecommunication apparatus 1 can be a stand-alone computer device 3 configured to track user visual interaction with one or more pre-defined AOIs 11 of a graphical image 9 displayed on a display 7. For example, the telecommunication apparatus 1 can be a computer device 3 (e.g., a smartphone, tablet, laptop computer, desktop computer, workstation, etc.) configured to track a user’s visual interaction in a local monitoringenvironment. In other words, the telecommunication apparatus 1 can be a computer device 3 that includes the processor 3a, the at least one transceiver 3b, the memory 3 c, the one or more sensors S, the at least one input device ID, and the at least one output device OD (e.g., display 7).

[0063] In some embodiments, the telecommunication apparatus 1 can include a visual interaction monitoring system 5 that can include one or more computer devices 3, one or more sensors S, at least one input device ID, and at least one output device OD. For example, in some embodiments, the apparatus can include a visual interaction monitoring system 5 that includes at least one computer device 3 that is a user device 13 (e.g., a smartphone, tablet, laptop computer, desktop computer, workstation, etc.) that is communicatively connected (e.g., via a local area network connection, internet connection, etc.) to a computer device 3 that is a host / monitor device 15 (e.g., a smartphone, tablet, laptop computer, desktop computer, workstation, server, etc.). The user device 13 and / or the monitor device 15 can each include or be communicatively connected to a processor 3a, at least one transceiver 3b, a non-transitory computer-readable memory 3c, at least one sensor S, at least one input device ID, and / or at least one output device OD. Each of the user device 13 and the monitor device 15 can be configured to receive, e.g., via the processor 3 a, one or more user inputs to pre-define one or more AOIs 11, and to store the one or more pre-defined AOIs 11 (e.g., one or more screenshots of and / or HTML code associated with each pre-defined AOI 11) in the memory 3c.

[0064] For example, in some embodiments, the monitor device 15 can receive one or more user inputs from a monitor to pre-define each AOI 1 1 for which the monitor wishes to track a user’s interaction with. The processor 3a of the monitor device 15 can be configured (e.g., can run code stored in the memory 3c) to generate a graphical user interface (GUI) to be displayed on a display 7 of the monitor device 15 that can display the graphical image 9 displayed on a display7 of the user device 13 such that the monitor device 15 can pre-define one or more AOIs 11 in the graphical image 9 displayed on the display 7 of the user device 13. In other words, the monitor device 15 can be configured to mirror the graphical image 9 displayed on the display 7 of the user device 13 in a GUI displayed on the display 7 of the monitor device 15 that allows the monitor using the monitor device 15 to interact with and / or observe the graphical image 9 displayed on the display 7 of the user device 13.

[0065] In such embodiments, the GUI can provide functionality that allows the monitor to use the monitor device 15 to quickly and easily observe the user’s interaction with one or more predefined AOIs 11 in a graphical image 9 displayed on the display 7 of the user device 13. For example, the monitor can select, via the GUI, color-code each pre-defined AOI 11 and / or user gaze point locations. In other words, the monitor can use at least one input device ID (e.g., a mouse, a keyboard, a touch screen, etc.) of the monitor device 15 to pre-define one or more AOIs 11 (e.g., via screenshot selection and / or application window selection) and to select a color to associate with each pre-defined AOI 11 that creates a color border around each pre-defined AOI 11 that is determined to be in the graphical image 9 displayed on the display 7 of the user device 13 (e.g., determined via pattern detection and / or HTML visual element tracking). In some embodiments, the color border for each pre-defined AOI 11 can be displayed only on the monitor device 15. That is, color-coding each pre-defined AOI 11 can allow the monitor to quickly identify one or more pre-defined AOIs 11 that are in the graphical image 9 displayed on the user display 7 of the user device 13.

[0066] Similarly, the monitor can color-code, or select (e.g., via the GUI) a color to be associated with the gaze point locations (e.g., gaze location data) of the user’s gaze, such that the location(s) of the user’s gaze relative to the graphical image 9 displayed on the display 7 of theuser device 13 will be displayed in the associated color in the mirrored graphical image 9 displayed on the display 7 of the monitor device 15. In such embodiments, the user device 13 and the monitor device 15 can be configured to exchange (e.g., send and / or receive) data (e.g., gaze location data, pre-defined AOI 11 data, sensor data, graphical image data, HTML code, attention data, and / or any other suitable data) with (e.g., to and / or from) each other via the communicative connection between the user device 13 and the monitor device 15, such that the monitor can observe the user’s interaction with the one or more pre-defined AOIs 11 in the graphical image 9 displayed on the display 7 of the user device 13 in real-time or near-real-time via the monitor device 15 generating data received from the user device 13 related to the user's interactions with the graphical image 9 displayed to the user via the display 7 of the user's device.

[0067] In some embodiments, the telecommunication apparatus 1 can include a visual interaction monitoring system 5 that includes a computer device 3 having a processor 3 a connected to a non- transitory computer-readable medium (e.g., memory 3c) that is a server device 17 (e.g., a remote cloud-based server, an array of servers, etc.) that is communicatively connected (e.g., via a wireless connection) to at least one other computer device 3, e.g., user device 13 and / or monitor device 15, at least one sensor S, the at least one input device ID, and / or the at least one output device OD. In such embodiments, the server device 17 can be configured to exchange (e.g., transmit and / or receive) data (e.g., visual interaction data, pre-defined AOI 11 data, sensor data, graphical image data, HTML code, and / or any other suitable data) with (e.g., to and / or from) each of the at least one other computer devices 3, at least one sensor S, at least one input device ID, and / or at least one output device OD that are communicatively connected to the server device17.

[0068] It should be understood that above-described telecommunication apparatus 1 embodiment configurations are merely exemplary, and the telecommunication apparatus 1 can include any suitable number and / or configuration of computer devices 3, systems 5, sensors S, input devices ID, and / or output devices OD for tracking a user’s visual interaction with one or more AOIs 11 in a graphical image 9 displayed on a display 7.

[0069] Embodiments of the telecommunication apparatus 1 and / or the visual interaction monitoring system 5 can be provided and configured to implement a method for tracking user visual interaction with one or more AOIs 11 of a graphical image 9 displayed on a display 7 of a user device (e.g., computer device 3 / 13). Such a method can include a first step SI of predefining, via one or more user inputs (e.g., user inputs received via the at least one input device ID) one or more areas of interest (AOIs) 11 of the graphical image 9. In some embodiments, the one or more AOIs 11 can be pre-defined prior to a user beginning a use session. In some embodiments, the one or more AOIs 11 can be pre-defined during a user’s use session (e.g., predefined during the use session by a monitor observing the use session).

[0070] The method can include a second step S2 of storing at least one screenshot of each predefined AOI 11 in a memory (e.g., memory 3c) and / or associating each pre-defined AOI 11 with hypertext markup language (HTML) code stored in a file that is run to generate the graphical image 9 displayed on the display 7.

[0071] The method can include a third step S3 of performing, via a processor (e.g., processor 3a) at least one of a pattern detection (e.g., pattern matching, pattern-matching evaluation, etc.) or a HTML visual element tracking (e.g., HTML interface evaluation). The pattern detection can include comparing one or more of the at least one screenshot stored in the memory to the graphical image 9 displayed on the display 7 of the user device (e.g., a computer device 3) toidentify a pattern match of the pre-defined AOI 1 1 of the at least one screenshot with a portion of the graphical image 9 displayed on the display 7, and, in response to identifying the pattern match, logging attention data for the AOI 11 of the pattern match.

[0072] The HTML visual element tracking (e.g., HTML interface evaluation) can include, in response to determining that at least one of the one or more pre-defined AOIs 11 is displayed in the graphical image 9 displayed on the display 7 via evaluation of the HTML code being run to generate the graphical image 9 displayed on the display 7, logging attention data for the at least one pre-defined AOI 11 associated with the HTML code being run to generate the graphical image 9 displayed on the display 7.

[0073] In some embodiments, the pattern detection (e.g., pattern matching, pattern-matching evaluation, etc.) and / or the HTML visual element tracking (e.g., HTML interface evaluation) can be performed while the graphical image 9 displayed on the display 7 is modified (e.g., via one or more user inputs), for example, the display can be scrolled, zoomed in, zoomed out, etc. In some embodiments, the graphical image 9 displayed on the display 7 is modified multiple times during a session of use by the user and the pattern detection and / or the HTML visual element tracking are performed for an entirety of the session of use. In some embodiments, the graphical image 9 displayed on the display 7 may be modified by the system itself (e.g., via processor 3a). For example, the graphical image 9 displayed on the display 7 may be modified (e.g., scrolled, zoomed, changed, etc.) based on the execution of code (e.g., by the processor 3a) during the use session to generate the graphical image 9. In other words, the graphical image 9 displayed on the display 7 can be a dynamic graphical image that changes in response to user input and / or changes based on code executed to generate the graphical image 9 that is predetermined based on the type of use session being conducted.

[0074] In an HTML-based environment, a JavaScript code can be used (e.g., run by the processor 3a) to read the user's web browser (e.g., Chrome or Edge). This script can help a researcher (e.g., experimenter, monitor, etc.) to select / pre-define their AOIs within a GUI, and the HTML code can then identify Areas of Interest (AOIs) within the HTML content and continuously track their positions on the screen. In some embodiments, positional data (e.g., coordinates of one or more AOIs 11) can be updated, recorded (e.g., stored), and / or transmitted at a pre-determined and / or user-defined time interval, for example, every 50 ms, every 100 ms, every second, every 10 seconds, etc. The positional data can then be mapped to the graphical image 9 displayed on the display 7 (e.g., via processor 3a) to determine whether the user visually interacted with one or more of the AOIs 11 on the graphical image 9 displayed on the display 7. If user visual interaction with one or more AOIs 11 is determined, visual interaction data (e g., gaze location data, attention data, etc.) can be logged (e.g., by the processor 3a), such as stored in one or more data files, e.g., a CSV file. In some embodiments, the visual interaction data logged can include AOI-specific visual interaction data, that is, visual interaction data for visual interactions only with one or more AOIs 11. In some embodiments, the logged visual interaction data (e.g., gaze location data, attention data, etc.) can include visual interaction data for all user visual interactions with the display 7, that is, visual interaction data for user visual interactions with one or more AOIs 11 and / or user visual interaction data for any / all areas of the graphical image 9 displayed on the display 7. This visual interaction data can be used later for further analysis if needed. For example, such analysis can include analyzing which AOIs 1 1 have not been interacted with, identifying unique and / or common visual interaction scan paths through multiple AOIs 11, identifying dwell / fixation time distributions for AOIs 11, etc.

[0075] In some embodiments, the processor 3a can be configured to record the user's screen activity and save it as an MP4 video. Such an embodiment can provide robust AOI tracking even when the user scrolls, zooms in or out, or changes the screen resolution, maintaining accuracy under dynamic viewing conditions. Logged visual interaction data can be further processed to generate insightful reports, including visual representations like heatmaps and gaze plots, which can help to summarize user attention patterns and interaction behaviors.

[0076] The method can include a fourth step S4 of storing the logged attention data in a non- transitory computer-readable medium (e.g., memory 3a) to generate at least one output (e.g., a report, data file, visual display, user interface, etc.) identifying each pre-defined AOI 11 that the user interacted with (e.g., visually interacted with) and / or a duration of time that the user interacted with each pre-defined AOI 11.

[0077] In some embodiments, the attention data can include a list of each AOI 11 that a user interacted with (e.g., visually interacted with) during a session of use, a timestamp for each time that the user interacted with each AOI 11, and a dwell / fixation time for each visual interaction that the user had with each AOI 11. In some embodiments, the attention data can be based on gaze location data, and the gaze location data can include one or more locations within the graphical image 9 displayed on the display 7 that the user visually interacted with during the session of use and a timestamp for each time that the user visually interacted with each of the one or more locations within the graphical image 9 displayed on the display 7. In some embodiments, the gaze location data can be based on sensor data generated by one or more sensors configured to track a user’s visual interaction with the graphical image 9 displayed on the display 7.

[0078] In some embodiments, visual interaction data (e.g., attention data, gaze location data, etc.) can be used for behavioral analysis, to understand user tendencies and pieces of informationusers use before making a decision, etc., by storing (e.g., in the memory 3c) the display recording and the visual interaction data. For example, if a researcher / experimenter / monitor decides after a use session has concluded that they want to also evaluate user visual interaction with one or more AOIs 11 that were not evaluated during the use session, the stored data can be evaluated via pattern detection after a use session has concluded.

[0079] The visual interaction data can include any data suitable for determining a user’s visual interaction with the graphical image 9 displayed on the display 7 and / or one or more AOIs 11 of the graphical image 9. For example, in some embodiments, the visual interaction data can include the total time a user visually interacted with each AOI 11, patterns of visual interaction transitions between AOIs 11, AOIs 11 that were not visually interacted with, etc. In some embodiments, visual interaction data for one or more user use sessions can be compared (e.g., via the processor 3a) to visual interaction data for one or more different user use sessions to identify user visual interaction patterns, commonalities, etc.

[0080] It should be understood that some components, features, and / or configurations may be described in connection with only one particular embodiment, but these same components, features, and / or configurations can be applied or used with many other embodiments and should be considered applicable to the other embodiments, unless stated otherwise or unless such a component, feature, and / or configuration is technically impossible to use with the other embodiment. Thus, the components, features, and / or configurations of the various embodiments can be combined together in any manner and such combinations are expressly contemplated and disclosed by this statement.

[0081] It will be apparent to those skilled in the art that numerous modifications and variations of the described examples and embodiments are possible considering the above teachings of thedisclosure. The disclosed examples and embodiments are presented for purpose of illustration only. Other alternate embodiments may include some or all of the features disclosed herein. Therefore, it is the intent to cover all such modifications and alternate embodiments as may come within the true scope of this invention, which is to be given the full breadth thereof.

[0082] It should be understood that modifications to the embodiments disclosed herein can be made to meet a particular set of design criteria. Therefore, while certain exemplary embodiments of the devices, systems, apparatuses, and methods of using and making the same disclosed herein have been discussed and illustrated, it is to be distinctly understood that the invention is not limited thereto but may otherwise be variously embodied and practiced within the scope of the following claims.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for tracking user visual interaction with one or more areas of interest of a graphical image displayed on a display of a user device, the method comprising: pre-defining, via one or more user inputs, one or more areas of interest of the graphical image; storing at least one screenshot of each pre-defined area of interest in a memory and / or associating each pre-defined area of interest with hypertext markup language (HTML) code stored in a file that is run to generate the graphical image displayed on the display; performing, via a processor, at least one of a pattern detection or a HTML visual element tracking, wherein the pattern detection comprises: comparing one or more of the at least one screenshot stored in the memory to the graphical image displayed on the display of the user device to identify a pattern match of the pre-defined area of interest of the at least one screenshot with a portion of the graphical image displayed on the display; and in response to identifying the pattern match, logging attention data for the area of interest of the pattern match; wherein the HTML visual element tracking comprises: in response to determining that at least one of the one or more pre-defined areas of interest is displayed in the graphical image displayed on the display via evaluation of the HTML code being run to generate the graphical image displayed on the display, logging attention data for the at least one pre-defined area of interest associated with the HTML code being run to generate the graphical image displayed on the display; andstoring the logged attention data in a non-transitory computer-readable medium to generate at least one report identifying each pre-defined area of interest that the user visually interacted with and / or a duration of time that the user visually interacted with each pre-defined area of interest.

2. The method of claim 1, wherein the pattern detection is performed while the graphical image displayed on the display is modified.

3. The method of claim 2, wherein the graphical image displayed on the display is modified multiple times during a session of use by the user device and the pattern detection is performed for the entirety of the session.

4. The method of claim 1, wherein the HTML visual element tracking is performed while the graphical image displayed on the display is modified.

5. The method of claim 4, wherein the graphical image displayed on the display is modified multiple times during a session of use by the user device and the HTML visual element tracking is performed for the entirety of the session.

6. The method of claim 1 , wherein the attention data comprises a list of each area of interest that the user visually interacted with during the session of use, a timestamp for each time that the user visually interacted with each area of interest, and a dwell time for each visual interaction that the user had with each area of interest.

7. The method of claim 6, wherein the attention data is based on gaze location data, wherein the gaze location data comprises one or more locations within the graphical image displayed on the display that the user visually interacted with during the session of use and a timestamp for each time that the user visually interacted with each of the one or more locations within the graphical image displayed on the display.

8. The method of claim 7, wherein the gaze location data is based on sensor data generated by one or more sensors configured to track a user’s visual interaction with the graphical image displayed on the display.

9. A telecommunication apparatus for tracking user visual interaction with one or more areas of interest of a graphical image displayed on a display, the apparatus comprising: a processor communicatively connected to a non-transitory computer-readable medium; the non-transitory computer-readable medium having code stored thereon such that the telecommunication apparatus is configured to: pre-define, via one or more user inputs, one or more areas of interest of the graphical image; store at least one screenshot of each pre-defined area of interest in a memory and / or associate each pre-defined area of interest with hypertext markup language (HTML) code stored in a file that is run to generate the graphical image displayed on the display; perform, via the processor, at least one of a pattern detection or a HTML visual element tracking,wherein the pattern detection comprises: comparing one or more of the at least one screenshot stored in the memory to the graphical image displayed on the display of the user device to identify a pattern match of the pre-defined area of interest of the at least one screenshot with a portion of the graphical image displayed on the display; and in response to identifying the pattern match, logging attention data for the area of interest of the pattern match; wherein the HTML visual element tracking comprises: in response to determining that at least one of the one or more pre-defined areas of interest is displayed in the graphical image displayed on the display via evaluation of the HTML code being run to generate the graphical image displayed on the display, logging attention data for the at least one pre-defined area of interest associated with the HTML code being run to generate the graphical image displayed on the display; and store the logged attention data in the non-transitory computer-readable medium to generate at least one report identifying each pre-defined area of interest that the user visually interacted with and / or a duration of time that the user visually interacted with the pre-defined area of interest.

10. The apparatus of claim 9, wherein the pattern detection is performed while the graphical image displayed on the display is modified.11 . The apparatus of claim 10, wherein the graphical image displayed on the display is modified multiple times during a session of use by the user and the pattern detection is performed for the entirety of the session of use.

12. The apparatus of claim 9, wherein the HTML visual element tracking is performed while the graphical image displayed on the display is modified.

13. The apparatus of claim 12, wherein the graphical image displayed on the display is modified multiple times during a session of use by the user and the HTML visual element tracking is performed for the entirety of the session.

14. The apparatus of claim 9, wherein the attention data comprises a list of each area of interest that the user visually interacted with during the session of use, a timestamp for each time that the user visually interacted with each area of interest, and a dwell time for each visual interaction that the user had with each area of interest.

15. The apparatus of claim 14, wherein the attention data is based on gaze location data, wherein the gaze location data comprises one or more locations within the graphical image displayed on the display that the user visually interacted with during the session of use and a timestamp for each time that the user visually interacted with each of the one or more locations within the graphical image displayed on the display.

16. The apparatus of claim 15, wherein the gaze location data is based on sensor data generated by one or more sensors configured to track a user’s visual interaction with the graphical image displayed on the display.

17. The apparatus of claim 9, wherein the telecommunication apparatus is a user device.

18. The apparatus of claim 9, wherein the telecommunication apparatus is a computer device that hosts a service that is communicably connectable to at least one user device for generation of the graphical image to be displayed on a display of the at least one user device.

19. The apparatus of claim 9, wherein associating each pre-defined area of interest with HTML code stored in a file that is run to generate the graphical image displayed on the display comprises selecting at least one window of the graphical image as an area of interest.

20. A non-transitory computer-readable medium having code stored thereon, the code defining a method performed by a computer device when a processor of the computer device runs the code, the method comprising: pre-defining, via one or more user inputs, one or more areas of interest of a graphical image displayed on a display; storing at least one screenshot of each pre-defined area of interest in a memory and / or associating each pre-defined area of interest with hypertext markup language (HTML) code stored in a file that is run to generate the graphical image displayed on the display; performing, via the processor, at least one of a pattern detection or a HTML visual element tracking,wherein the pattern detection comprises: comparing one or more of the at least one screenshot stored in the memory to the graphical image displayed on the display to identify a pattern match of the at least one predefined area of interest of the screenshot with a portion of the graphical image displayed on the display; and in response to identifying the pattern match, logging attention data for the area of interest of the pattern match; wherein the HTML visual element tracking comprises: in response to determining that the area of interest is displayed in the graphical image displayed on the display via evaluation of the HTML code being run to generate the graphical image displayed on the display, logging attention data for the at least one area of interest associated with the HTML code being run to generate the graphical image displayed on the display; and storing the logged attention data to generate at least one report identifying each predefined area of interest that the user visually interacted with and / or a duration of time that a user visually interacted with the pre-defined area of interest.

Images