Eye movement mapping method, device, edge computing device and computer-readable storage medium

Abstract

Embodiments of the present application provide an eye movement point mapping method, apparatus, edge computing device, and computer-readable storage medium. These methods acquire eye movement point video data and a target image, process the eye movement point video data, and obtain bitmap data corresponding to the target frame. Image registration analysis is then performed on the target image and the bitmap data. Based on the image registration analysis results, the eye movement point data included in the bitmap data is mapped to the target image. Embodiments of the present application achieve eye movement point mapping from eye movement video data to the target image without manual labeling, thereby improving eye movement point mapping efficiency.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and specifically relates to an eye movement point mapping method, apparatus, edge computing device, and computer-readable storage medium. Background Art

[0002] Eye tracking mapping technology has been widely used in autonomous driving, virtual reality (VR), augmented reality (AR), game development, security monitoring, and other fields. Existing eye tracking mapping is achieved through manual annotation, which is inefficient. Summary of the Invention

[0003] Embodiments of the present application provide an eye movement point mapping method, apparatus, edge computing device, and computer-readable storage medium to improve the efficiency of eye movement mapping.

[0004] In a first aspect, an embodiment of the present application provides an eye movement point mapping method, comprising:

[0005] Obtaining eye movement point video data and a target image, wherein the target image is a target image uploaded by the user through the user interface of the application end;

[0006] Processing the eye-movement point video data to obtain bitmap data corresponding to a target frame, wherein the target frame is an initial frame containing eye-movement point data in the eye-movement point video data;

[0007] Performing feature extraction on the target image to obtain target image features corresponding to the target image;

[0008] Obtaining bitmap data of a sub-image region including eye movement point data in the bitmap data corresponding to the target frame, wherein the sub-image region is within a preset number of pixels around the eye movement point coordinates;

[0009] Performing image registration analysis based on the target image features and the bitmap data of the sub-image area;

[0010] According to the image registration analysis result, the eye movement point data included in the bitmap data is mapped to the target image.

[0011] Optionally, the processing of the eye-movement point video data to obtain bitmap data corresponding to the target frame includes:

[0012] Performing frame processing on the eye movement point video data to obtain a plurality of initial frames;

[0013] determining an initial frame containing eye movement point data as a target frame;

[0014] Performing bitmap processing on the target subframe to obtain bitmap data corresponding to the target subframe.

[0015] Optionally, the processing of the eye-movement point video data to obtain bitmap data corresponding to the target frame includes:

[0016] Performing frame processing on the eye movement point video data to obtain a plurality of initial frames;

[0017] Performing bitmap processing on the initial frame to obtain bitmap data corresponding to the initial frame;

[0018] The bitmap data corresponding to the initial frame containing the eye movement point data is determined as the bitmap data corresponding to the target frame.

[0019] Optionally, also include:

[0020] The target image features are stored.

[0021] Optionally, before mapping the eye movement point data included in the bitmap data to the target image based on the image registration analysis result, the method further includes:

[0022] Based on the dynamic configuration parameters, denoising processing is performed on the bitmap data corresponding to the target frame, wherein the denoising processing includes: signal noise reduction processing, lost data compensation processing and / or blink data processing.

[0023] Optionally, before mapping the eye movement point data included in the bitmap data to the target image based on the image registration analysis result, the method further includes:

[0024] Converting the target frame's bitmap data into a multidimensional array;

[0025] The performing image registration analysis on the target image and the bitmap data includes:

[0026] Image registration analysis is performed based on the multidimensional array corresponding to the target frame and the target image.

[0027] Optionally, obtaining eye-movement point video data and target image includes:

[0028] The first process receives the eye-tracking point video data uploaded by the user through the application end, and sends the eye-tracking point video data to the second process;

[0029] The second process receives the target image uploaded by the user through the application end, and sends the eye movement point video data and the target image to the third process.

[0030] Optionally, the performing frame processing on the eye-movement point video data to obtain a plurality of initial frames includes:

[0031] The third process performs frame processing on the eye movement point data to obtain multiple initial frames, and performs a hash operation based on relevant information of each initial frame to generate identification information corresponding to the initial frame, wherein the relevant information includes: a timestamp corresponding to the initial frame, an order of the initial frames in the eye movement point video data, and / or information of a device that generates the eye movement point video data;

[0032] The step of determining the initial frame containing the eye movement point data as the target frame includes:

[0033] The third process determines whether the initial frame contains the eye movement point based on whether the initial frame contains the eye movement point coordinates, and determines the initial frame containing the eye movement point as the target frame;

[0034] The performing bitmap processing on the target frame to obtain bitmap data corresponding to the target frame includes:

[0035] The third process performs bitmap processing on the target frame to obtain bitmap data corresponding to the target frame, and sends the bitmap data and identification information corresponding to the target frame to the fourth process.

[0036] Optionally, the performing denoising processing on the bitmap data corresponding to the target frame based on the dynamic configuration parameters includes:

[0037] The fourth process performs denoising on the bitmap data corresponding to the target frame based on the dynamic configuration parameters, writes the denoised bitmap data into a shared memory in a preset format, and sends the memory address and data length of the bitmap data to the fifth process using a preset communication protocol.

[0038] The preset format includes: a file header, an information header, a color table and pixel data.

[0039] Optionally, converting the bitmap data of the target frame into a multidimensional array includes:

[0040] The fifth process reads the bitmap data from the shared memory according to the memory address and data length sent by the fourth process; converts the bitmap data into a multidimensional array, and sends the multidimensional array to the sixth process;

[0041] The performing image registration analysis based on the multidimensional array corresponding to the target frame and the target image, and mapping the eye movement point data included in the bitmap data to the target image according to the image registration analysis result, includes:

[0042] The sixth process performs image registration analysis based on the multidimensional array corresponding to the target frame and the target image, and maps the eye movement point data included in the bitmap data to the target image according to the image registration analysis result. The eye movement point mapping result includes: eye movement hotspot map and / or gaze sequence.

[0043] Optionally, the steps executed by the first to fourth processes are implemented in a first computer programming language program; and the steps executed by the fifth to sixth processes are implemented in a second computer programming language program.

[0044] Optionally, also include:

[0045] The initial frames that do not contain eye movement data are discarded.

[0046] In a second aspect, an embodiment of the present application provides an eye movement point mapping device, comprising:

[0047] An acquisition module is used to acquire eye movement point video data and a target image, wherein the target image is a target image uploaded by the user through the user interface of the application end;

[0048] a processing module, configured to process the eye-movement point video data to obtain bitmap data corresponding to a target frame, wherein the target frame is an initial frame containing eye-movement point data in the eye-movement point video data;

[0049] An analysis module is configured to extract features from the target image to obtain target image features corresponding to the target image; obtain bitmap data of a sub-image area including eye movement point data in the bitmap data corresponding to the target frame, wherein the sub-image area is within a preset number of pixels around the eye movement point coordinates; perform image registration analysis based on the target image features and the bitmap data of the sub-image area; and map the eye movement point data included in the bitmap data to the target image according to the image registration analysis results.

[0050] Optionally, the processing module is specifically used to perform frame processing on the eye-movement point video data to obtain multiple initial frames; determine the initial frame containing the eye-movement point data as the target frame; and perform bitmap processing on the target frame to obtain bitmap data corresponding to the target frame.

[0051] Optionally, the processing module is specifically used to perform frame processing on the eye-movement point video data to obtain multiple initial frames; perform bitmap processing on the initial frames to obtain bitmap data corresponding to the initial frames; and determine that the bitmap data corresponding to the initial frames containing eye-movement point data is the bitmap data corresponding to the target frame.

[0052] Optionally, the analysis module is further configured to store the target image features.

[0053] Optionally, the processing module is further configured to perform denoising processing on the bitmap data corresponding to the target frame based on dynamic configuration parameters, wherein the denoising processing includes: signal noise reduction processing, lost data compensation processing and / or blink data processing.

[0054] Optionally, the processing module is further configured to convert the bitmap data of the target frame into a multidimensional array; and the analysis module is specifically configured to perform image registration analysis based on the multidimensional array corresponding to the target frame and the target image.

[0055] Optionally, the acquisition module is specifically used to receive eye-movement point video data uploaded by the user through the application end through the first process, and send the eye-movement point video data to the second process; the second process receives the target image uploaded by the user through the application end, and sends the eye-movement point video data and the target image to the third process.

[0056] Optionally, the processing module is specifically used to perform frame processing on the eye movement point data through a third process to obtain multiple initial frames, and perform hash operation based on relevant information of each initial frame to generate identification information corresponding to the initial frame, wherein the relevant information includes: the timestamp corresponding to the initial frame, the order of the initial frames in the eye movement point video data and / or the device information for generating the eye movement point video data; the third process determines whether the initial frame contains the eye movement point based on whether the initial frame contains the eye movement point coordinates, and determines that the initial frame containing the eye movement point is the target frame; the third process performs bitmap processing on the target frame to obtain bitmap data corresponding to the target frame, and sends the bitmap data and identification information corresponding to the target frame to the fourth process.

[0057] Optionally, the processing module is specifically used to perform denoising on the bitmap data corresponding to the target frame based on the dynamic configuration parameters through a fourth process, and write the denoised bitmap data into a shared memory in a preset format, and send the memory address and data length storing the bitmap data to a fifth process via a preset communication protocol; wherein the preset format includes: a file header, an information header, a color table and pixel data.

[0058] Optionally, the processing module is specifically used to read the bitmap data from the shared memory through the fifth process according to the memory address and data length sent by the fourth process; and convert the bitmap data into a multidimensional array, and send the multidimensional array to the sixth process; the analysis module is specifically used to perform image registration analysis based on the multidimensional array corresponding to the target frame and the target image through the sixth process, and according to the image registration analysis result, map the eye movement point data included in the bitmap data to the target image, and the eye movement point mapping result includes: eye movement hotspot map and / or gaze sequence.

[0059] Optionally, the steps executed by the first to fourth processes are implemented in a first computer programming language program; and the steps executed by the fifth to sixth processes are implemented in a second computer programming language program.

[0060] Optionally, the processing module is further configured to discard initial frames that do not contain eye movement point data.

[0061] In a third aspect, an embodiment of the present application provides an edge computing device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the eye movement mapping method as described in any one of the first aspects are implemented.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the eye movement mapping method as described in any one of the first aspects are implemented.

[0063] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed by a processor, implements the steps of the eye movement mapping method as described in any one of the first aspects.

[0064] The eye movement point mapping method, apparatus, edge computing device, and computer-readable storage medium provided in the embodiments of the present application obtain eye movement point video data and a target image, process the eye movement point video data, and obtain bitmap data corresponding to the target frame. The target image and the bitmap data are then subjected to image registration analysis. Based on the image registration analysis results, the eye movement point data included in the bitmap data is mapped to the target image. This achieves eye movement point mapping from the eye movement video data to the target image without manual labeling, thereby improving eye movement point mapping efficiency. During the eye movement mapping process, the eye movement point video data is processed to obtain bitmap data corresponding to the target frame. The bitmap data is used to uniformly represent the image data of the target frame, thereby improving data consistency. Since the bitmap data has direct memory access characteristics, data transmission and processing time can be reduced, further improving the efficiency of eye movement point mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A flowchart of an eye movement point mapping method provided in an embodiment of the present application;

[0066] Figure 2A A schematic flow chart of another eye movement point mapping method provided in an embodiment of the present application;

[0067] Figure 2B A schematic flow chart of another eye movement point mapping method provided in an embodiment of the present application;

[0068] Figure 3 A flowchart of another eye movement point mapping method provided in an embodiment of the present application;

[0069] Figure 4 A flowchart of another eye movement point mapping method provided in an embodiment of the present application;

[0070] Figure 5 A flowchart of another eye movement point mapping method provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of the structure of an eye movement mapping system provided in an embodiment of the present application;

[0072] Figure 7 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0073] Figure 8 A structural diagram of an eye movement point mapping device is provided for an embodiment of the present application;

[0074] Figure 9 A schematic diagram of the structure of an edge computing device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0075] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0076] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0077] The term "indication" in this application can be either a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, the operation to be performed, or the requested result, etc. in the instruction sent; an indirect indication can be understood as the receiver determining the corresponding information based on the instruction sent by the sender, or making a judgment and determining the operation to be performed or the requested result, etc. based on the judgment result.

[0078] The embodiments of the present application can monitor the position and movement of the eyes in real time through image acquisition devices such as infrared cameras, eye trackers, security surveillance cameras, medical imaging equipment, vehicle-mounted cameras, virtual reality equipment, eye tracking equipment, smart phones or tablet computers, and synchronize the captured eye gaze points, eye movement trajectories and other eye movement data (called eye movement points) to the visual content viewed by the user in real time or afterwards, such as video frames or images. By analyzing the correspondence between the target sub-frames and the visual content, the visual elements that the user focuses on within a specific time period can be determined, and an eye gaze heat map or line of sight tracking path can be generated, thereby achieving eye movement point mapping. In the eye movement point mapping process, the embodiments of the present application perform bitmap processing on the target video frame in the eye movement point video data, and perform eye movement point mapping analysis based on the bitmap data. Since the bitmap data has the direct memory access feature, the time for data transmission and processing can be reduced, thereby improving the efficiency of eye movement point mapping.

[0079] Furthermore, the embodiment of the present application also performs denoising on the bitmap data corresponding to the target frame based on the dynamic configuration parameters, and performs eye movement point mapping analysis based on the denoised bitmap data, thereby improving the quality and reliability of the eye movement data and the accuracy of the mapping results.

[0080] Furthermore, the inter-process communication in the embodiment of the present application is based on a preset communication protocol, which improves the communication efficiency and further improves the efficiency of eye movement mapping.

[0081] Furthermore, the embodiment of the present application uses different programming languages ​​for processing at different processing stages of eye movement mapping, which fully utilizes the advantages of each programming language and further improves the efficiency of eye movement mapping.

[0082] The technical solution of this application is described below with reference to several specific embodiments.

[0083] Figure 1 A flow chart of an eye movement mapping method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method of this embodiment is as follows:

[0084] S100: Acquire eye movement point video data and target image.

[0085] Among them, eye movement video data refers to video data that captures and records information such as the movement of the viewer's eyes and changes in the gaze point, the trajectory of gaze movement, the duration of gaze, blinking frequency, and changes in pupil size when the viewer watches a video or image through eye tracking technology.

[0086] Eye tracking video data can be captured through eye trackers, eye tracking devices, cameras, smartphones or tablets, VR devices, in-vehicle cameras, security cameras, and medical imaging devices. For example, when a user is viewing an advertisement or webpage, a camera can capture the user's eye tracking video data. Another example is that in a video conference, a camera captures the video footage of the participants. Another example is that in an automated driving assistance system, an in-vehicle camera monitors the driver's eye tracking video data. In a security monitoring system, a camera captures the eye tracking video footage. Another example is that in remote medical diagnosis, medical imaging equipment records the patient's eye tracking video data.

[0087] Users can upload eye movement video data through the application. Specifically, they can select and upload pre-recorded eye movement video files on the application interface of the application to prepare for subsequent data processing and eye movement mapping analysis.

[0088] The target image is a key reference in eye movement mapping analysis. It is closely related to the eye movement video data. It can be a static screenshot of the video frame watched by the user, a specific image, or any visual content that the user wants to analyze. It is used to map the captured eye movement data (such as the gaze point and the line of sight movement path) to a specific visual element.

[0089] The user can upload the target image through the user interface of the application, for example, by selecting the "Upload Image" or "Select Image" option in the application interface and then selecting a corresponding image file from the local file system.

[0090] S300: Process the eye-movement point video data to obtain bitmap data corresponding to the target frame.

[0091] The target frame is a frame in the eye-movement point video data.

[0092] A possible implementation of this step is as follows Figure 2A As shown, including: S301A-S305A:

[0093] S301A: performing frame processing on the eye-movement point video data to obtain a plurality of initial frames.

[0094] Framing eye tracking video data involves breaking down a continuous video stream into individual still images. The video can be segmented into a series of initial frames based on its frame rate (the number of frames per second). Each frame represents a single moment in the video and contains all the visual information at that moment, including image content and eye tracking data.

[0095] S302A: Determine whether the initial frame contains eye-movement point data. If so, execute S303A; if not, execute S304A.

[0096] S303A: Determine the initial frame containing the eye movement point data as the target frame.

[0097] In this step, to ensure that each initial frame can be uniquely identified and tracked, a hash operation is performed based on the relevant information of each initial frame to generate identification information corresponding to the initial frame. This identification information can uniquely identify the corresponding initial frame, wherein the relevant information includes: the timestamp corresponding to the initial frame (i.e., the specific time point at which the frame appears in the video), the order of the initial frame in the eye movement point video data (i.e., the frame number, indicating the position of the frame in the video stream), and / or information about the device that generated the eye movement point video data (e.g., device model, serial number, etc.). This identification information can uniquely identify the corresponding frame, thereby ensuring that the frame can be accurately identified and tracked during the eye movement mapping process, thereby improving the accuracy and reliability of data analysis.

[0098] The multiple initial frames obtained through the frame processing are analyzed. For example, eye tracking technology can be used to analyze each frame of the image and detect and identify the eye movement point data contained therein. This eye movement point data typically includes key data such as the eye's gaze position (eye movement point coordinates), pupil size, and blink frequency. Once the system identifies the presence of an eye movement point, the corresponding initial frame is marked as the target frame containing the eye movement point data.

[0099] S304A: Discard the initial frame that does not contain eye movement point data.

[0100] That is, the initial sub-frames that do not contain eye movement points will not participate in the subsequent data processing. By only performing subsequent processing on the target sub-frames containing eye movement point data, the effectiveness of the processing is improved and the processing efficiency is further improved.

[0101] S305A: Perform bitmap processing on the target frame to obtain bitmap data corresponding to the target frame.

[0102] Bitmap processing of target frames involves converting video frames into a bitmap format. A bitmap is a digital image file that contains the original image information, ensuring the accuracy and integrity of the image content. Bitmap data is stored in a pre-set format, which includes a file header, information header, color table, and pixel data.

[0103] Another possible implementation of this step is as follows Figure 2B Shown, including:

[0104] S301B: performing frame processing on the eye-movement point video data to obtain a plurality of initial frames.

[0105] This step is described in detail in S301B and will not be repeated here.

[0106] S302B: Perform bitmap processing on the initial frame to obtain bitmap data corresponding to the initial frame.

[0107] The bitmap processing of the initial frame is a process of converting the video frame into a bitmap format. The bitmap data is stored in a preset format, wherein the preset format includes: a file header, an information header, a color table and pixel data.

[0108] S303B: Acquire bitmap data corresponding to the initial frame containing eye movement point data as bitmap data corresponding to the target frame.

[0109] That is, the bitmap data containing the eye movement point data is determined as the bitmap data corresponding to the target frame. Only the bitmap data of the target frame containing the eye movement point data is subsequently processed, thereby improving the effectiveness of the processing and further improving the processing efficiency.

[0110] In this embodiment, the original pixel data of the image is retained through bitmap data, providing accurate visual content information for eye movement point mapping. In addition, bitmap is a widely supported image format and is easily compatible with other image processing software and algorithms, thereby improving compatibility. Bitmap data can be directly mapped to memory for fast access and processing without the need to copy data, thereby improving processing efficiency.

[0111] S500: performing image registration analysis on the target image and the bitmap data, and mapping the eye movement point data included in the bitmap data to the target image according to the image registration analysis result.

[0112] One possible implementation involves extracting features from the target image to obtain target image features corresponding to the target image, storing the target image features, and performing image registration analysis based on the target image features and the bitmap data. By storing the target image features, when importing additional eye-tracking video data for eye-tracking mapping with the target image, the stored target image features can be directly used, eliminating the need to repeatedly extract features from the target image, further improving processing efficiency.

[0113] One possible implementation of image registration analysis based on the target image feature and the bitmap data is as follows: determining whether the content of the bitmap data in the target image feature is greater than or equal to a preset range; if the content of the bitmap data in the target image feature is greater than or equal to the preset range, determining that the target image feature and the bitmap data are successfully registered, wherein the preset range is part or all, and the proportion of the part in the bitmap data is greater than a preset proportion; for example, if the proportion of the target image feature in the bitmap data is greater than 1 / 3, then it is considered that the target image feature and the bitmap data are successfully registered. If the content of the bitmap data in the target image is less than the preset range, then it is determined that the target image feature and the bitmap data have failed to be registered.

[0114] Another possible implementation of image registration analysis based on the target image features and the bitmap data is to obtain bitmap data of a sub-image region containing eye movement point data within the bitmap data corresponding to the target frame. For example, the sub-image region is a preset number of pixels surrounding the eye movement point coordinates, and image registration analysis is performed based on the target image features and the bitmap data of the sub-image region. Since registration is performed using the sub-image region containing eye movement point data and the target image features, the amount of data involved in registration is reduced, thereby further improving processing efficiency.

[0115] The eye movement data is mapped onto the target image. Based on the mapping analysis results, an eye movement mapping result for the target image is generated. After mapping, the eye movement mapping result presented on the target image may include an eye movement gaze heat map, gaze movement path, and / or gaze duration. This enables automatic mapping of eye movement data from the eye movement video image to the target image and visual presentation on the target image, improving eye movement mapping efficiency and intuitively demonstrating the user's visual attention.

[0116] The eye movement mapping results can be presented in the form of graphics or images on the target image, such as displaying areas where the user's gaze time is longer with markers of different colors or sizes, or drawing the trajectory of the gaze movement.

[0117] This embodiment acquires eye-tracking video data and a target image, processes the eye-tracking video data, and obtains bitmap data corresponding to the target frame. Image registration analysis is then performed on the target image and the bitmap data. Based on the image registration analysis results, the eye-tracking data included in the bitmap data is mapped to the target image. This eliminates the need for manual labeling and achieves eye-tracking mapping from the eye-tracking video data to the target image, improving eye-tracking mapping efficiency. During the eye-tracking mapping process, bitmap data corresponding to the target frame is obtained by processing the eye-tracking video data. The bitmap data is then used to uniformly represent the image data of the target frame, improving data consistency. The direct memory access feature of bitmap data reduces data transmission and processing time, further improving eye-tracking mapping efficiency.

[0118] Figure 3 A flowchart of another eye movement point mapping method provided in an embodiment of the present application is shown below. Figure 3 is Figure 1 On the basis of the embodiment shown, further, before S500, the method further includes:

[0119] S401: performing denoising processing on the bitmap data corresponding to the target frame based on dynamic configuration parameters.

[0120] The denoising process includes: signal noise reduction process, lost data compensation process and / or blink data process.

[0121] Users can set various dynamic parameters to optimize the accuracy and quality of eye tracking mapping based on their needs and preferences. Based on user-defined or predefined dynamic parameters, noise and outliers are identified in the bitmap data. These parameters may include the strength of signal noise reduction, missing data compensation strategies, and blink event processing methods. Appropriate denoising algorithms, such as filters or statistical methods, are applied to reduce or eliminate random fluctuations and noise interference in the data. For example, signal noise reduction processing may use filters to smooth the data, such as sliding mean filters or sliding median filters, to remove noise from the video. Users can set the window size (e.g., 3, 5, 7, 9) to smooth the data and reduce the impact of outliers. Missing data compensation processing may use interpolation techniques to estimate and fill in missing data points. Users can set a maximum interval length. Missing data with intervals smaller than this length will be compensated using linear interpolation to handle blinks or missed eye movements. Blink data processing may identify blink events and temporarily ignore or correct the associated data to reduce their impact on the analysis results. This denoising process improves the signal-to-noise ratio of the bitmap data, ensuring the accuracy and reliability of the eye movement mapping analysis, and providing a clearer and more precise data foundation for subsequent eye movement data analysis and result presentation. This embodiment also allows the user to specify whether to analyze data based on the left or right eye.

[0122] In this embodiment, eye movement mapping analysis is performed based on the bitmap data corresponding to the target frame and the target image. Before obtaining the eye movement point mapping result of the target image, the bitmap data corresponding to the target frame is denoised based on dynamic configuration parameters to improve the quality and accuracy of the eye movement data, thereby improving the accuracy and reliability of the eye movement point mapping analysis.

[0123] Figure 4 A flowchart of another eye movement mapping method provided in an embodiment of the present application is shown below. Figure 4 is Figure 3 On the basis of the embodiment shown, further, before S500, the method further includes:

[0124] S402: Convert the bitmap data of the target frame into a multi-dimensional array.

[0125] Converting the target frame's bitmap data into a multidimensional array converts the structured information in the bitmap data into a format suitable for further mathematical and scientific computation. Multidimensional arrays are a data structure used in Python for efficient numerical computation. Multidimensional arrays are stored as contiguous blocks in memory, making data access and processing faster. The multidimensional array format is supported by many Python scientific computing libraries, such as SciPy, Pandas, and Matplotlib, making data exchange and processing more convenient. It provides powerful multidimensional array objects, derived objects (such as mask arrays and matrices), and a set of convenient functions for fast array manipulation. By converting the bitmap data into a multidimensional array, you can leverage the rich mathematical functions and operations provided by the Numpy library for efficient data processing and analysis, such as array slicing, mathematical operations, and statistical analysis. This improves the efficiency of eye tracking mapping. Because multidimensional arrays meet the data format requirements of most machine learning algorithms, users can seamlessly switch between algorithm models without re-coding or debugging.

[0126] Accordingly, a possible implementation of S500 is shown in S500′:

[0127] S500 ′: performing image registration analysis based on the multidimensional array corresponding to the target frame and the target image, and mapping the eye movement point data included in the bitmap data to the target image according to the image registration analysis result.

[0128] Optionally, the bitmap data for each target frame is converted into a multidimensional array. These arrays are widely used in scientific computing for their efficient numerical computation capabilities. An image analysis algorithm can be retrieved from at least one preset algorithm in the algorithm library for image registration analysis. Using multidimensional arrays, target frames and eye movement data, including gaze points, gaze movement paths, and gaze duration, can be accurately processed and analyzed. The data in these multidimensional arrays is then associated with the target image, and an algorithm is used to determine the user's visual focus and attention distribution while watching the video. After the analysis is complete, an eye movement mapping result is generated for the target image. These results intuitively demonstrate the user's visual attention patterns during viewing, such as gaze hotspots and gaze movement trajectories. This process not only improves the accuracy and efficiency of eye movement data analysis, but also makes the final eye movement mapping results more intuitive and easy to understand.

[0129] In this embodiment, before performing eye movement mapping analysis, the bitmap data of the target frame is converted into a multidimensional array, and then image registration analysis is performed based on the multidimensional array corresponding to the target frame and the target image. According to the image registration analysis results, the eye movement point data included in the bitmap data is mapped to the target image. The rich mathematical functions and operations provided by the multidimensional array (such as the Numpy library) are used to perform efficient data processing and analysis, such as array slicing, mathematical operations, statistical analysis, etc., thereby improving the efficiency of eye movement mapping. Since the multidimensional array meets the data format requirements of most machine learning or algorithms, that is, the multidimensional data includes the common features of the data requirements of the machine learning / artificial intelligence algorithm in the algorithm library, even if the actual algorithm selected in the back-end algorithm library changes, these image processing processes on the front end do not need to be changed, so that users can switch algorithm models without feeling, without the need for secondary coding and debugging.

[0130] Figure 5 A flow chart of another eye movement point mapping method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the method of this embodiment is as follows:

[0131] S51: The first process receives the eye-tracking point video data uploaded by the user through the application end, and sends the eye-tracking point video data to the second process.

[0132] In the first process, the system receives eye-tracking video data uploaded by the user through a graphical user interface. This data is selected and uploaded by the user on the application side. Once uploaded, the first process is responsible for forwarding this raw eye-tracking video data to the second process.

[0133] S52: The second process receives the target image uploaded by the user through the application end, and sends the eye movement point video data and the target image to the third process.

[0134] The second process receives the target image uploaded by the user, which serves as the reference for the user's eye tracking analysis. It also receives the eye tracking video data from the first process. After receiving these two types of data, the second process integrates them and sends them to the third process.

[0135] S53: The third process performs frame processing on the eye-movement point data to obtain multiple initial frames, and performs hash operation based on the relevant information of each initial frame to generate identification information corresponding to the initial frame, wherein the relevant information includes: the timestamp corresponding to the initial frame, the order of the initial frames in the eye-movement point video data and / or the device information that generates the eye-movement point video data.

[0136] The third process performs frame processing on the eye tracking video data. This process breaks down the continuous video stream into independent frames, generating multiple initial frames. A hash operation is then performed using information related to each initial frame, such as the timestamp (corresponding to the initial frame), the order of the frames in the video (the order of the initial frames in the eye tracking video data), and information about the device that generated the video data (generating the identification information corresponding to the initial frame). This operation generates identification information uniquely corresponding to each initial frame, providing uniqueness for subsequent data processing and tracking.

[0137] S54: The third process determines whether the initial frame contains the eye movement point based on whether the initial frame contains the eye movement point coordinates, and determines the initial frame containing the eye movement point as the target frame.

[0138] In the third stage, the system needs to determine which initial frames actually contain gaze point data. This is achieved by checking each initial frame for the presence of gaze point coordinates. Only those initial frames containing gaze point coordinates are identified as target frames. These frames serve as key input for the gaze point mapping analysis and enter the next bitmap processing stage.

[0139] S55: The third process performs bitmap processing on the target frame to obtain bitmap data corresponding to the target frame, and sends the bitmap data and identification information corresponding to the target frame to the fourth process.

[0140] The third process performs bitmap processing on the frames identified as target sub-frames. This process converts the frames into a bitmap format, preserving the image's pixel-level information. After processing, the bitmap data and corresponding identification information are sent to the fourth process. This bitmap data transmission provides visual information for subsequent denoising and eye tracking analysis.

[0141] S56: The fourth process denoises the bitmap data corresponding to the target frame based on the dynamic configuration parameters, writes the denoised bitmap data into the shared memory in a preset format, and sends the memory address and data length of the bitmap data stored to the fifth process via a preset communication protocol; wherein the preset format includes: a file header, an information header, a color table, and pixel data.

[0142] The fourth process denoises the bitmap data corresponding to the target frame. This step, based on user-configured dynamic parameters, aims to improve data quality and reduce noise interference. The denoised data is written to shared memory in a pre-set format, including a file header, information header, color table, and pixel data. The fourth process then sends the memory address and data length of the bitmap data to the fifth process via a pre-set communication protocol.

[0143] S57: The fifth process reads the bitmap data from the shared memory according to the memory address and data length sent by the fourth process, converts the bitmap data into a multidimensional array, and sends the multidimensional array to the sixth process.

[0144] The fifth process reads the bitmap data from shared memory based on the memory address and data length information provided by the fourth process, without copying the data. After reading, the bitmap data is converted into a multidimensional array, a data structure suitable for efficient numerical calculations and processing. Once converted, the multidimensional array is sent to the sixth process for eye-mapping analysis.

[0145] S58: The sixth process performs image registration analysis based on the multidimensional array corresponding to the target frame and the target image, and maps the eye movement point data included in the bitmap data to the target image according to the image registration analysis result.

[0146] In the sixth process, image registration analysis is performed based on the multidimensional array received from the fifth process and the target image uploaded by the user. Specifically, bitmap data for a sub-image region containing eye-tracking point data within the bitmap data corresponding to the target frame is obtained. For example, the sub-image region is a preset number of pixels surrounding the eye-tracking point coordinates. Image registration analysis is performed based on the target image features and the bitmap data of the sub-image region. Because registration is performed using the sub-image region containing eye-tracking point data and the target image features, the amount of data involved in registration is reduced, thereby further improving processing efficiency.

[0147] This analysis process identifies eye movement hotspots and gaze sequences, providing an intuitive view for understanding the user's visual attention allocation.

[0148] S59: The sixth process sends the eye movement point mapping result and identification information to the second process.

[0149] After completing the eye movement mapping analysis, the sixth process sends the eye movement point mapping results and corresponding identification information obtained from the analysis back to the second process.

[0150] S60: The second process presents the eye movement point mapping result on the target image.

[0151] In the second step, the gaze mapping results are presented on the target image. This step allows users to visually correlate the gaze data with the target image, providing a deeper understanding of the user's visual attention and behavior patterns. This visualization is the final output of the entire process, providing users with an easy-to-understand analysis view.

[0152] This embodiment implements cross-process communication and ensures frame traceability through the unique identification information of the initial frame generated by hashing, thereby improving the accuracy of matching eye movement data with video frames. Denoising and missing data compensation ensure the integrity of eye movement data and reduce data loss caused by factors such as blinking, equipment errors, or environmental interference. Dynamic configuration parameters allow users to adjust denoising and compensation strategies according to different research purposes and needs, providing methodological flexibility. The efficiency of eye movement mapping analysis is achieved through fast data transmission through shared memory and preset communication protocols, as well as efficient numerical calculations of multi-dimensional arrays. The intuitive visualization results are presented in the form of eye movement heat maps and gaze sequences, allowing users to easily understand the eye movement data.

[0153] In the above embodiment, the steps executed by the first to fourth processes are implemented in a first computer programming language program, and the steps executed by the fifth to sixth processes are implemented in a second computer programming language program. The first computer programming language may be, for example, the C# programming language, and the second computer programming language may be, for example, the Python programming language. This enables cross-language and cross-process communication, fully leveraging the advantages of different languages. For example, the C# programming language facilitates low-level development and interfacing with various peripheral interfaces and communications, while the Python programming language has advantages in machine learning, computer vision, artificial intelligence, and image analysis and processing, thereby further improving the efficiency of eye movement mapping.

[0154] Figure 6 This is a structural diagram of an eye movement mapping system provided in an embodiment of the present application, such as Figure 6 As shown, the system of the embodiment of the present application includes: a data production module 601, an application module 602, a support module 603 and a deployment terminal 604, wherein the data generation module 601 is used to generate eye movement point video data and target images, and the application module 602 includes: an import video data module, a dynamic parameter configuration module and an eye movement mapping processing module, wherein the import video data module is used to perform Figure 5 The first process in the dynamic parameter configuration module is used to perform the steps Figure 5 The fourth process in the step of executing the eye movement mapping processing module is used to perform Figure 5 The steps executed by the sixth process in , where each process specifically executes the corresponding steps, see Figure 5 The support module 603 includes: a data transmission module, an algorithm module and a data consistency module, wherein the data transmission module is used to execute Figure 5 The algorithm module is used to execute the steps from the first process to the sixth process in Figure 5 The sixth process in the data consistency module is used to perform the steps Figure 5For details on how each process executes the corresponding steps, see Figure 5 The deployment end 604 includes: an edge end and a cloud end, which can be deployed on the edge end using edge computing or on the cloud end using cloud computing.

[0155] Figure 7 A schematic diagram of an application scenario provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, Figure 7 It shows the specific implementation of eye movement mapping technology in practical applications. In this application scenario, the left side shows a video of the experimenter in the simulated driving process. The video captures the experimenter's eye movement points through the eye tracking hardware device. These eye movement point data are recorded and synchronized with the video screen to provide a basis for analyzing the experimenter's visual attention in the simulated driving task. On the right is the target image uploaded by the user for eye movement mapping. As the video on the left plays, the system processes the video screen and its corresponding eye movement point coordinates frame by frame. These data are based on Figure 5 The system processes the data according to the process described in the previous section. When the system identifies an eye movement point in the video frame and is able to find the corresponding mapping location in the target image on the right, it will mark the corresponding location in the target image with a preset color (for example, a red dot). This mapping method intuitively shows where the experimenter's gaze falls during simulated driving, allowing researchers to clearly observe and analyze the distribution of their visual attention. Through this intuitive mapping result, researchers can conduct in-depth analysis and research on visual attention during driving.

[0156] Figure 8 A structural diagram of an eye movement point mapping device is provided for an embodiment of the present application. Figure 8 As shown, the device of this embodiment includes: an acquisition module 801, a processing module 802 and an analysis module 803, the acquisition module 801 is used to acquire eye-movement point video data and a target image; the processing module 802 is used to process the eye-movement point video data to obtain bitmap data corresponding to a target frame, wherein the target frame is a frame in the eye-movement point video data; the analysis module 803 is used to perform image registration analysis on the target image and the bitmap data; according to the image registration analysis result, the eye-movement point data included in the bitmap data is mapped to the target image.

[0157] Optionally, the processing module 802 is specifically used to perform frame processing on the eye-movement point video data to obtain multiple initial frames; determine the initial frame containing the eye-movement point data as the target frame; and perform bitmap processing on the target frame to obtain bitmap data corresponding to the target frame.

[0158] Optionally, the processing module 802 is specifically used to perform frame processing on the eye-movement point video data to obtain multiple initial frames; perform bitmap processing on the initial frames to obtain bitmap data corresponding to the initial frames; and obtain the bitmap data corresponding to the initial frames containing eye-movement point data as the bitmap data corresponding to the target frames.

[0159] Optionally, the analysis module 803 is specifically configured to extract features from the target image to obtain target image features corresponding to the target image; store the target image features; and perform image registration analysis based on the target image features and the bitmap data.

[0160] Optionally, the analysis module 803 is specifically used to determine that the target image feature and the bitmap data are successfully aligned if the content of the bitmap data included in the target image feature is greater than or equal to a preset range, wherein the preset range is part or all, and the proportion of the part in the bitmap data is greater than a preset proportion; and / or, if the content of the bitmap data included in the target image is less than the preset range, determine that the target image feature and the bitmap data are failed to be aligned.

[0161] Optionally, the analysis module 803 is specifically configured to obtain bitmap data of a sub-image region including eye movement point data in the bitmap data corresponding to the target frame; and perform image registration analysis based on the target image features and the bitmap data of the sub-image region.

[0162] Optionally, the processing module 802 is further configured to perform denoising processing on the bitmap data corresponding to the target frame based on dynamic configuration parameters, wherein the denoising processing includes: signal noise reduction processing, lost data compensation processing and / or blink data processing.

[0163] Optionally, the processing module 802 is further configured to convert the bitmap data of the target frame into a multidimensional array; and the analysis module 803 is specifically configured to perform image registration analysis based on the multidimensional array corresponding to the target frame and the target image.

[0164] Optionally, the acquisition module 801 is specifically used to receive eye-movement point video data uploaded by the user through the application end through the first process, and send the eye-movement point video data to the second process; the second process receives the target image uploaded by the user through the application end, and sends the eye-movement point video data and the target image to the third process.

[0165] Optionally, the processing module 802 is specifically used to perform frame processing on the eye movement point data through a third process to obtain multiple initial frames, and perform hash operation based on relevant information of each initial frame to generate identification information corresponding to the initial frame, wherein the relevant information includes: the timestamp corresponding to the initial frame, the order of the initial frames in the eye movement point video data and / or the device information for generating the eye movement point video data; the third process determines whether the initial frame contains the eye movement point based on whether the initial frame contains the eye movement point coordinates, and determines that the initial frame containing the eye movement point is the target frame; the third process performs bitmap processing on the target frame to obtain bitmap data corresponding to the target frame, and sends the bitmap data and identification information corresponding to the target frame to the fourth process.

[0166] Optionally, the processing module 802 is specifically used to perform denoising on the bitmap data corresponding to the target frame based on the dynamic configuration parameters through the fourth process, and write the denoised bitmap data into the shared memory in a preset format, and send the memory address and data length storing the bitmap data to the fifth process via a preset communication protocol; wherein the preset format includes: file header, information header, color table and pixel data.

[0167] Optionally, the processing module 802 is specifically used to read the bitmap data from the shared memory through the fifth process according to the memory address and data length sent by the fourth process; and convert the bitmap data into a multidimensional array, and send the multidimensional array to the sixth process; the analysis module 803 is specifically used to perform image registration analysis based on the multidimensional array corresponding to the target frame and the target image through the sixth process, and according to the image registration analysis result, map the eye movement point data included in the bitmap data to the target image, and the eye movement point mapping result includes: eye movement hotspot map and / or gaze sequence.

[0168] Optionally, the steps executed by the first to fourth processes are implemented in a first computer programming language program; and the steps executed by the fifth to sixth processes are implemented in a second computer programming language program.

[0169] Optionally, the processing module 802 is further configured to discard initial frames that do not contain eye movement point data.

[0170] The device of this embodiment corresponds to the technical solution that can be used to execute the above-mentioned method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0171] The embodiment of the present application also provides an edge computing device, including a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the following is achieved: Figure 1-Figure 5 The steps of the eye movement point mapping method.

[0172] The embodiment of the present application further provides a computer-readable storage medium, wherein a program or instruction is stored on the computer-readable storage medium, and when the program or instruction is executed by a processor, the following is achieved: Figure 1-Figure 5 The steps of the eye movement point mapping method.

[0173] The present application also provides a computer program product, which, when executed by a processor, implements the following Figure 1-Figure 5 The steps of the eye movement point mapping method.

[0174] The following specific reference Figure 9 , which shows a schematic diagram of the structure of an edge computing device 700 suitable for implementing the embodiments of the present application. The edge computing device 700 in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), eye trackers, AR / VR / MR smart glasses, wearable devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. The edge computing device shown is only an example and should not bring any limitations to the functions and occupancy range of the embodiments of the present application.

[0175] like Figure 9 As shown, the edge computing device 700 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 710, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 720 or the program loaded from the storage device 780 to the random access memory (RAM) 730. Various programs and data required for the operation of the edge computing device are also stored in the RAM 730. The processing device 710, the ROM 720, and the RAM 730 are connected to each other via a bus 740. An input / output (I / O) interface 750 is also connected to the bus 740.

[0176] Typically, the following devices can be connected to the I / O interface 750: input devices 760 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 770 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 780 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 790. The communication devices 790 can allow the edge computing device 700 to communicate with other devices wirelessly or by wire to exchange data. Although the edge computing device 700 is shown with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0177] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 790, or installed from the storage device 780, or installed from the ROM 720. When the computer program is executed by the processing device 710, the above-mentioned functions defined in the eye movement mapping method of the embodiment of the present application are performed.

[0178] It should be noted that the computer-readable medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0179] In some embodiments, the user end and the management end can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0180] The above-mentioned computer-readable medium may be included in the above-mentioned edge computing device; or it may exist independently without being assembled into the edge computing device.

[0181] The computer-readable medium carries one or more programs. When the one or more programs are executed by the edge computing device, the edge computing device:

[0182] Obtain eye movement video data and target images;

[0183] Processing the eye-movement point video data to obtain bitmap data corresponding to a target frame, wherein the target frame is a frame in the eye-movement point video data;

[0184] performing image registration analysis on the target image and the bitmap data;

[0185] According to the image registration analysis result, the eye movement point data included in the bitmap data is mapped to the target image.

[0186] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0187] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0188] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0189] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0190] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0191] According to one or more embodiments of the present application, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute any eye movement mapping method as described in the embodiments of the present application.

[0192] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

[0193] In addition, although adopting specific order to describe each operation, this should not be interpreted as requiring these operations to be executed in the specific order shown or in sequential order.Under certain environment, multitasking and parallel processing may be advantageous.Similarly, although comprising some specific implementation details in the above discussion, these should not be interpreted as limiting the scope of the application.Some features described in the context of separate embodiment can also be implemented in a single embodiment in combination.On the contrary, the various features described in the context of a single embodiment also can be implemented in multiple embodiments individually or in the mode of any suitable subcombination.

[0194] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. An eye movement point mapping method, characterized in that: include: Obtaining eye movement point video data and a target image, wherein the target image is a target image uploaded by the user through the user interface of the application end; Processing the eye-movement point video data to obtain bitmap data corresponding to a target frame, wherein the target frame is an initial frame containing eye-movement point data in the eye-movement point video data; Performing feature extraction on the target image to obtain target image features corresponding to the target image; Obtaining bitmap data of a sub-image region including eye movement point data in the bitmap data corresponding to the target frame, wherein the sub-image region is within a preset number of pixels around the eye movement point coordinates; Performing image registration analysis based on the target image features and the bitmap data of the sub-image area; According to the image registration analysis result, the eye movement point data included in the bitmap data is mapped to the target image.

2. The method according to claim 1, characterized in that The processing of the eye-movement point video data to obtain bitmap data corresponding to the target frame includes: Performing frame processing on the eye movement point video data to obtain a plurality of initial frames; determining an initial frame containing eye movement point data as a target frame; Performing bitmap processing on the target subframe to obtain bitmap data corresponding to the target subframe.

3. The method according to claim 1, characterized in that The processing of the eye-movement point video data to obtain bitmap data corresponding to the target frame includes: Performing frame processing on the eye movement point video data to obtain a plurality of initial frames; Performing bitmap processing on the initial frame to obtain bitmap data corresponding to the initial frame; The bitmap data corresponding to the initial frame containing the eye movement point data is determined as the bitmap data corresponding to the target frame.

4. The method according to claim 1, wherein Also includes: The target image features are stored.

5. The method according to claim 2, characterized in that Before mapping the eye movement point data included in the bitmap data to the target image according to the image registration analysis result, the method further includes: Based on the dynamic configuration parameters, denoising processing is performed on the bitmap data corresponding to the target frame, wherein the denoising processing includes: signal noise reduction processing, lost data compensation processing and / or blink data processing.

6. The method according to claim 5, characterized in that Before mapping the eye movement point data included in the bitmap data to the target image according to the image registration analysis result, the method further includes: Converting the target frame's bitmap data into a multidimensional array; The performing image registration analysis on the target image and the bitmap data includes: Image registration analysis is performed based on the multidimensional array corresponding to the target frame and the target image.

7. The method according to claim 6, characterized in that The acquiring of eye movement point video data and target image includes: The first process receives the eye-tracking point video data uploaded by the user through the application end, and sends the eye-tracking point video data to the second process; The second process receives the target image uploaded by the user through the application end, and sends the eye movement point video data and the target image to the third process.

8. The method according to claim 7, characterized in that The step of performing frame processing on the eye-movement point video data to obtain a plurality of initial frames includes: The third process performs frame processing on the eye movement point data to obtain multiple initial frames, and performs a hash operation based on relevant information of each initial frame to generate identification information corresponding to the initial frame, wherein the relevant information includes: a timestamp corresponding to the initial frame, an order of the initial frames in the eye movement point video data, and / or information of a device that generates the eye movement point video data; The step of determining the initial frame containing the eye movement point data as the target frame includes: The third process determines whether the initial frame contains the eye movement point based on whether the initial frame contains the eye movement point coordinates, and determines the initial frame containing the eye movement point as the target frame; The performing bitmap processing on the target frame to obtain bitmap data corresponding to the target frame includes: The third process performs bitmap processing on the target frame to obtain bitmap data corresponding to the target frame, and sends the bitmap data and identification information corresponding to the target frame to the fourth process.

9. The method according to claim 8, characterized in that The denoising process is performed on the bitmap data corresponding to the target frame based on the dynamic configuration parameters, including: The fourth process performs denoising on the bitmap data corresponding to the target frame based on the dynamic configuration parameters, writes the denoised bitmap data into a shared memory in a preset format, and sends the memory address and data length of the bitmap data to the fifth process using a preset communication protocol. The preset format includes: a file header, an information header, a color table and pixel data.

10. The method according to claim 9, characterized in that The step of converting the target framed bitmap data into a multidimensional array includes: The fifth process reads the bitmap data from the shared memory according to the memory address and data length sent by the fourth process; converts the bitmap data into a multidimensional array, and sends the multidimensional array to the sixth process; The performing image registration analysis based on the multidimensional array corresponding to the target frame and the target image, and mapping the eye movement point data included in the bitmap data to the target image according to the image registration analysis result, includes: The sixth process performs image registration analysis based on the multidimensional array corresponding to the target frame and the target image, and maps the eye movement point data included in the bitmap data to the target image according to the image registration analysis result. The eye movement point mapping result includes: eye movement hotspot map and / or gaze sequence.

11. The method according to claim 10, characterized in that The steps executed by the first to fourth processes are implemented in a first computer programming language program; the steps executed by the fifth to sixth processes are implemented in a second computer programming language program.

12. The method according to claim 11, characterized in that Also includes: The initial frames that do not contain eye movement data are discarded.

13. An eye movement point mapping device, characterized in that: include: An acquisition module is used to acquire eye movement point video data and a target image, wherein the target image is a target image uploaded by the user through the user interface of the application end; a processing module, configured to process the eye-movement point video data to obtain bitmap data corresponding to a target frame, wherein the target frame is an initial frame containing eye-movement point data in the eye-movement point video data; An analysis module is configured to extract features from the target image to obtain target image features corresponding to the target image; obtain bitmap data of a sub-image area including eye movement point data in the bitmap data corresponding to the target frame, wherein the sub-image area is within a preset number of pixels around the eye movement point coordinates; perform image registration analysis based on the target image features and the bitmap data of the sub-image area; and map the eye movement point data included in the bitmap data to the target image according to the image registration analysis results.

14. An edge computing device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the eye movement mapping method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the eye movement mapping method according to any one of claims 1 to 12 are implemented.

Images