Multi-modal eye movement data mapping method and device, edge computing equipment and medium

Abstract

The invention discloses a multi-modal eye movement data mapping method and device, edge computing equipment and a medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring an infrared eye movement image collected by an infrared camera and an RGB target image collected by a visible light camera; the infrared eye movement image comprises an infrared image and eye movement data; determining a first infrared image block including eye movement data in the infrared eye movement image; in response to a condition that a matching point in the first infrared image block meets a preset condition, calculating a coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the first infrared image block; the matching points represent points with similar characteristics in the infrared eye movement image and the RGB target image; and mapping the eye movement data to the RGB target image according to the coordinate mapping relation. According to the embodiment of the invention, the mapping error between different modal images can be reduced, and the accuracy of eye movement data mapping is improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a multimodal eye movement data mapping method, device, edge computing device and medium. Background Art

[0002] In the field of multimodal image processing, the fusion technology of infrared images and visible light images has attracted much attention due to its unique advantages in obtaining the thermal characteristics and color and texture details of objects. In eye tracking technology, how to map eye movement points from infrared images to visible light images so that data based on different modalities can be effectively interacted and applied has become a hot topic of research.

[0003] In related technologies, multimodal image fusion technology mainly focuses on image fusion at the feature level or pixel level, but there is still a lack of an effective method to accurately map eye movement data from one modality to another, so as to achieve effective interaction based on data from different modalities. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a multimodal eye movement data mapping method, device, edge computing device and medium to reduce the mapping error between different modal images and improve the accuracy of eye movement data mapping.

[0005] In a first aspect, the present application provides a multimodal eye movement data mapping method, comprising:

[0006] Acquire an infrared eye movement image collected by an infrared camera and an RGB target image collected by a visible light camera; the infrared eye movement image includes an infrared image and eye movement data;

[0007] Determining a first infrared image block including eye movement data in the infrared eye movement image;

[0008] In response to a matching point in the first infrared image block satisfying a preset condition, calculating a coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the first infrared image block; the matching point represents a point having similar features in the infrared eye movement image and the RGB target image;

[0009] The eye movement data is mapped to the RGB target image according to the coordinate mapping relationship.

[0010] According to the multimodal eye movement data mapping method of the present application, an infrared eye movement image collected by an infrared camera and an RGB target image collected by a visible light camera are obtained; the infrared eye movement image includes an infrared image and eye movement data; a first infrared image block including eye movement data in the infrared eye movement image is determined; in response to the matching points in the first infrared image block satisfying a preset condition, a coordinate mapping relationship between the infrared eye movement image and the RGB target image is calculated according to the first infrared image block; the matching points represent points with similar features in the infrared eye movement image and the RGB target image; and the eye movement data is mapped to the RGB target image according to the coordinate mapping relationship. The embodiment of the present application obtains infrared eye movement images and RGB target images of different modalities, and calculates the coordinate mapping relationship between the infrared eye movement image and the RGB target image when the infrared image block containing eye movement data meets the preset conditions, so that the calculation process pays more attention to the area in the image related to the eye movement data, rather than the entire infrared eye movement image, thereby reducing the offset between images of different modalities, thereby reducing the mapping error between images of different modalities, and improving the accuracy of eye movement data mapping.

[0011] According to one embodiment of the present application, before acquiring a first infrared image block including eye movement data in an infrared eye movement image, the method includes:

[0012] Inputting the infrared eye movement image and the RGB target image into a preset multimodal model, so that the multimodal model can extract feature points of the infrared eye movement image and the RGB target image, and matching them according to the extracted feature points; or preprocessing the infrared eye movement image and the RGB target image, wherein the preprocessing includes removing feature points within a preset range of the boundary of the infrared eye movement image and removing feature points within a preset range of the boundary of the RGB target image, and inputting the preprocessed infrared eye movement image and the RGB target image into a preset multimodal model, so that the multimodal model can extract feature points of the preprocessed infrared eye movement image and the RGB target image, and matching them according to the extracted feature points;

[0013] The successfully matched feature points are determined as matching points.

[0014] In this embodiment, since the multimodal model is capable of processing and analyzing data from different modalities, cross-modal shared features of infrared eye movement images and RGB target images can be extracted through the multimodal model, thereby providing a more accurate correspondence when matching feature points and improving the accuracy of cross-modal feature point matching.

[0015] In this embodiment, by removing feature points within a preset range of the image boundary, the negative impact of noise or incompleteness that may exist in the image boundary area on the matching accuracy can be reduced, so that more focus can be placed on more stable and information-rich areas in the image, thereby improving the matching accuracy.

[0016] According to one embodiment of the present application, the multimodal model includes a self-attention unit, a cross-attention unit and a classifier; the input dimension of the cross-attention unit includes a grayscale channel dimension; the self-attention unit and the cross-attention unit are used to extract feature points of the infrared eye movement image and the RGB target image, and calculate the confidence that the feature points of the infrared eye movement image and the feature points of the RGB target image are matching points, and the classifier is used to output the feature point as a matching point when the confidence that the feature point is a matching point is greater than a preset threshold.

[0017] In this embodiment, by setting the input dimension of the cross-attention unit to include the grayscale channel dimension, the model pays more attention to the temperature distribution characteristics of the image rather than the color characteristics during the feature extraction process, thereby improving the accuracy of feature extraction of infrared eye movement images; the confidence of the feature points as matching points is calculated through the self-attention mechanism. When the confidence exceeds the preset threshold, the classifier can control the model to stop reasoning. These feature points are identified by the classifier and output as matching points, which not only improves the accuracy of matching point identification, but also avoids unnecessary calculations and reduces the consumption of computing resources.

[0018] According to one embodiment of the present application, the infrared eye movement image is a video frame in a target video stream, and the method further includes:

[0019] In response to a failure in matching between the infrared eye-movement image and the RGB target image, a video frame is reselected from the target video stream to replace the infrared eye-movement image and the RGB target image for feature point matching.

[0020] In this embodiment, when feature point matching fails to successfully establish an accurate correspondence between the infrared eye movement image and the RGB target image, a new video frame can be flexibly selected from a continuous video stream as the infrared eye movement image, thereby reducing the risk of interruption of the entire eye tracking process due to a single matching failure and improving the continuity of the system.

[0021] According to one embodiment of the present application, reselecting a video frame from the target video stream to replace the infrared eye movement image and the RGB target image for feature point matching includes:

[0022] A video frame adjacent to the video frame corresponding to the infrared eye movement image in the target video stream or other video frames in the target video stream are determined as the reselected video frames.

[0023] In this embodiment, when the feature point matching between the infrared eye movement image and the RGB target image fails, a video frame adjacent to the current infrared eye movement image is selected from the target video stream as a replacement to re-perform feature point matching, so that the reselected video frame maintains temporal continuity, or other video frames are selected as replacements, thereby improving the stability and accuracy of the eye tracking process.

[0024] According to one embodiment of the present application, the method further includes:

[0025] Storing feature points of the RGB target image extracted by the multimodal model;

[0026] Reselecting a video frame from the target video stream to replace the infrared eye movement image and the RGB target image for feature point matching includes:

[0027] The replaced infrared eye movement image is input into a preset multimodal model so that the multimodal model can extract feature points of the infrared eye movement image and match the feature points of the infrared eye movement image with the feature points of the stored RGB target image.

[0028] In this embodiment, the features of the RGB target image are extracted and stored through a multimodal model. When other video frame images are matched with the RGB target image, the feature points of the RGB target image can be called for feature matching. There is no need to repeatedly extract the features of the RGB target image, thereby improving processing efficiency.

[0029] According to an embodiment of the present application, determining a first infrared image block including eye movement data in the infrared eye movement image includes:

[0030] Determining a position of the eye movement data in the infrared eye movement image;

[0031] The first infrared image block is determined according to the position of the eye movement data in the infrared eye movement image.

[0032] In this embodiment, by accurately identifying the position of the eye movement data and determining the first infrared image block according to the position of the eye movement data, irrelevant areas in the processed image can be reduced so that the infrared image block contains more information related to the eye movement data.

[0033] According to an embodiment of the present application, determining the first infrared image block according to the position of the eye movement data in the infrared eye movement image includes:

[0034] An area within a first preset range centered at the position of the eye movement data in the infrared eye movement image is determined as the first infrared image block.

[0035] In this embodiment, by determining the area within the first preset range centered on the position of the eye movement data as the first infrared image block, the calculation process can focus on the area near the eye movement data, thereby reducing invalid calculations in irrelevant areas and improving calculation efficiency and accuracy.

[0036] According to an embodiment of the present application, the preset conditions include: the number of matching points in the infrared image block is greater than or equal to a preset number, and / or the multiple matching points in the infrared image block are not collinear; and / or,

[0037] The method further includes: in response to the first infrared image block not satisfying a preset condition, acquiring a second infrared image block including the eye movement data in the infrared eye movement image; and / or,

[0038] The method further includes: in response to the second infrared image block satisfying a preset condition, calculating a coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the second infrared image block.

[0039] In this embodiment, a preset condition is set to require that the number of matching points is greater than or equal to a preset number, and these matching points are not collinear, so that there are enough matching points to calculate the coordinate mapping relationship. The condition that multiple matching points are not collinear enables the matching points to provide sufficient geometric information to solve the coordinate mapping relationship between the infrared eye movement image and the RGB target image, thereby further improving the accuracy of multimodal eye movement data mapping.

[0040] In this embodiment, by obtaining a second infrared image block including eye movement data in the infrared eye movement image when the first infrared image block does not meet the preset conditions, even if a sufficient number or a suitable distribution of matching points cannot be found within the range of the first infrared image block, the possibility of finding valid matching points can be increased by adjusting the search area, thereby enhancing the flexibility and success rate of the multimodal eye movement data mapping process.

[0041] According to an embodiment of the present application, the step of obtaining a second infrared image block including the eye movement data in the infrared eye movement image includes:

[0042] Acquire the second infrared image block according to the position of the first infrared image block in the infrared eye movement image; or,

[0043] Determine the position of the eye movement data in the infrared eye movement image; determine an area within a second preset range centered on the position of the eye movement data in the infrared eye movement image as the second infrared image block; wherein the second preset range is larger than the first preset range.

[0044] In this embodiment, by acquiring the second infrared image block according to the position of the first infrared image block in the infrared eye movement image, a new search range can be re-determined with the first infrared image block as a reference, thereby improving the efficiency of the multimodal eye movement data mapping calculation process.

[0045] In this embodiment, by expanding the first infrared image block into the second infrared image block, even if a sufficient number or a suitable distribution of matching points cannot be found within the initial search range, the possibility of finding valid matching points can be increased by expanding the search area, thereby enhancing the flexibility and success rate of the multimodal eye movement data mapping process.

[0046] According to an embodiment of the present application, calculating the coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the first infrared image blocks includes:

[0047] Acquire a first position coordinate of a matching point in the first infrared image block in the infrared eye movement image and a second position coordinate of the matching point in the RGB target image;

[0048] A homography matrix corresponding to the matching point is calculated according to the first position coordinates and the second position coordinates; wherein the homography matrix represents the coordinate mapping relationship.

[0049] In this embodiment, by calculating the corresponding position coordinates of the matching points in the infrared eye movement image and the RGB target image, a mathematical model can be established to describe the spatial relationship between these matching points, and based on this, a homography matrix is ​​determined to represent the coordinate mapping relationship. Representing the coordinate mapping relationship by the homography matrix can adapt to complex image transformation situations, thereby improving the accuracy and robustness of multimodal eye movement data mapping.

[0050] According to one embodiment of the present application, the method further includes:

[0051] Determining the sum of confidences of matching points in the first infrared image block; calculating a matching score based on the sum of confidences, the number of matching points in the first infrared image block, and the area of ​​the first infrared image block; wherein the matching score represents the accuracy of mapping the eye movement data to the RGB target image; and / or,

[0052] The eye movement data includes at least one of an eye gaze point, a gaze direction, and a sight track.

[0053] In this embodiment, the matching score is calculated by comprehensively considering the sum of the confidences of the first target matching points, the number of matching points, and the area of ​​the first infrared image block, which effectively quantifies the accuracy of the position of the eye movement data in the RGB target image, so that the matching score can provide a comprehensive indicator for evaluating the credibility and accuracy of the matching results.

[0054] In a second aspect, the present application provides a multimodal eye movement data mapping device, comprising:

[0055] An acquisition module, used to acquire an infrared eye movement image acquired by an infrared camera and an RGB target image acquired by a visible light camera; the infrared eye movement image includes an infrared image and eye movement data;

[0056] A determination module, configured to determine a first infrared image block including eye movement data in the infrared eye movement image;

[0057] a calculation module, configured to calculate a coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the first infrared image block in response to a matching point in the first infrared image block satisfying a preset condition; the matching point represents a point having similar features in the infrared eye movement image and the RGB target image;

[0058] A mapping module is used to map the eye movement data to the RGB target image according to the coordinate mapping relationship.

[0059] According to the multimodal eye movement data mapping device of the present application, an infrared eye movement image collected by an infrared camera and an RGB target image collected by a visible light camera are obtained; the infrared eye movement image includes an infrared image and eye movement data; a first infrared image block including eye movement data in the infrared eye movement image is determined; in response to the matching points in the first infrared image block satisfying a preset condition, a coordinate mapping relationship between the infrared eye movement image and the RGB target image is calculated according to the first infrared image block; the matching points represent points with similar features in the infrared eye movement image and the RGB target image; and the eye movement data is mapped to the RGB target image according to the coordinate mapping relationship. The embodiment of the present application obtains infrared eye movement images and RGB target images of different modalities, and calculates the coordinate mapping relationship between the infrared eye movement image and the RGB target image when the infrared image block containing eye movement data meets the preset conditions, so that the calculation process pays more attention to the area in the image related to the eye movement data, rather than the entire infrared eye movement image, thereby reducing the offset between images of different modalities, thereby reducing the mapping error between images of different modalities, and improving the accuracy of eye movement data mapping.

[0060] In a third aspect, the present application provides an edge computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal eye movement data mapping method as described in the first aspect above is implemented.

[0061] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the multimodal eye movement data mapping method as described in the first aspect above is implemented.

[0062] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the multimodal eye movement data mapping method as described in the first aspect above.

[0063] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the multimodal eye movement data mapping method as described in the first aspect above.

[0064] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0065] According to the multimodal eye movement data mapping method of the present application, an infrared eye movement image collected by an infrared camera and an RGB target image collected by a visible light camera are obtained; the infrared eye movement image includes an infrared image and eye movement data; a first infrared image block including eye movement data in the infrared eye movement image is determined; in response to the matching points in the first infrared image block satisfying a preset condition, a coordinate mapping relationship between the infrared eye movement image and the RGB target image is calculated according to the first infrared image block; the matching points represent points with similar features in the infrared eye movement image and the RGB target image; and the eye movement data is mapped to the RGB target image according to the coordinate mapping relationship. The embodiment of the present application obtains infrared eye movement images and RGB target images of different modalities, and calculates the coordinate mapping relationship between the infrared eye movement image and the RGB target image when the infrared image block containing eye movement data meets the preset conditions, so that the calculation process pays more attention to the area in the image related to the eye movement data, rather than the entire infrared eye movement image, thereby reducing the offset between images of different modalities, thereby reducing the mapping error between images of different modalities, and improving the accuracy of eye movement data mapping.

[0066] Furthermore, in some embodiments, since the multimodal model can process and analyze data from different modalities, the cross-modal shared features of the infrared eye movement image and the RGB target image can be extracted through the multimodal model, so that a more accurate correspondence can be provided when matching feature points, thereby improving the accuracy of cross-modal feature point matching. By removing feature points within a preset range of the image boundary, the negative impact of noise or incompleteness that may exist in the image boundary area on the matching accuracy can be reduced, so that more focus can be placed on more stable and information-rich areas in the image, thereby improving the accuracy of matching.

[0067] Furthermore, in some embodiments, by setting the input dimension of the cross-attention unit to include the grayscale channel dimension, the model pays more attention to the temperature distribution features of the image rather than the color features during the feature extraction process, thereby improving the accuracy of feature extraction of infrared eye movement images; the confidence of the feature points as matching points is calculated through the self-attention mechanism. When the confidence exceeds a preset threshold, the classifier can control the model to stop reasoning. These feature points are identified by the classifier and output as matching points, which not only improves the accuracy of matching point identification, but also avoids unnecessary calculations and reduces the consumption of computing resources.

[0068] Furthermore, in some embodiments, when feature point matching fails to successfully establish an accurate correspondence between the infrared eye movement image and the RGB target image, a new video frame can be flexibly selected from a continuous video stream as the infrared eye movement image, thereby reducing the risk of interruption of the entire eye tracking process due to a single matching failure and improving the continuity of the system.

[0069] Furthermore, in some embodiments, when the feature point matching between the infrared eye movement image and the RGB target image fails, a video frame adjacent to the current infrared eye movement image is selected from the target video stream as a replacement to re-perform feature point matching, so that the reselected video frame maintains temporal continuity, or other video frames are selected as replacements, thereby improving the stability and accuracy of the eye tracking process.

[0070] Furthermore, in some embodiments, the features of the RGB target image are extracted and stored through a multimodal model. When other video frame images are matched with the RGB target image, the feature points of the RGB target image can be called for feature matching. There is no need to repeatedly extract the features of the RGB target image, thereby improving processing efficiency.

[0071] Furthermore, in some embodiments, by accurately identifying the location of the eye movement data and determining the first infrared image block based on the location of the eye movement data, irrelevant areas in the processed image can be reduced so that the infrared image block contains more information related to the eye movement data.

[0072] Furthermore, in some embodiments, by determining the area within a first preset range centered on the position of the eye movement data as the first infrared image block, the calculation process can focus on the area near the eye movement data, thereby reducing invalid calculations in irrelevant areas and improving calculation efficiency and accuracy.

[0073] Furthermore, in some embodiments, by setting a preset condition requiring that the number of matching points is greater than or equal to a preset number, these matching points are not collinear, so that there are enough matching points to calculate the coordinate mapping relationship, and the condition that multiple matching points are not collinear enables the matching points to provide sufficient geometric information to solve the coordinate mapping relationship between the infrared eye movement image and the RGB target image, further improving the accuracy of multimodal eye movement data mapping. By obtaining a second infrared image block including eye movement data in the infrared eye movement image when the first infrared image block does not meet the preset condition, even if a sufficient number or a suitable distribution of matching points cannot be found within the range of the first infrared image block, the possibility of finding valid matching points can be increased by adjusting the search area, thereby enhancing the flexibility and success rate of the multimodal eye movement data mapping process.

[0074] Furthermore, in some embodiments, by obtaining the second infrared image block according to the position of the first infrared image block in the infrared eye movement image, the new search range can be re-determined with the first infrared image block as a reference, thereby improving the efficiency of the multimodal eye movement data mapping calculation process. By expanding the first infrared image block into the second infrared image block, even if a sufficient number of matching points or a suitable distribution cannot be found within the initial search range, the possibility of finding valid matching points can be increased by expanding the search area, thereby enhancing the flexibility and success rate of the multimodal eye movement data mapping process.

[0075] Furthermore, in some embodiments, by calculating the corresponding position coordinates of the matching points in the infrared eye movement image and the RGB target image, a mathematical model can be established to describe the spatial relationship between these matching points, and based on this, a homography matrix is ​​determined to represent the coordinate mapping relationship. Representing the coordinate mapping relationship by the homography matrix can adapt to complex image transformation situations, thereby improving the accuracy and robustness of multimodal eye movement data mapping.

[0076] Furthermore, in some embodiments, the matching score is calculated by comprehensively considering the sum of the confidence levels of the first target matching points, the number of matching points, and the area of ​​the first infrared image block, thereby effectively quantifying the accuracy of the position of the eye movement data in the RGB target image, so that the matching score can provide a comprehensive indicator for evaluating the credibility and accuracy of the matching results.

[0077] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 It is a flowchart of a multimodal eye movement data mapping method provided in an embodiment of the present application;

[0080] Figure 2 is a schematic diagram of a feature point matching process provided in an embodiment of the present application;

[0081] Figure 3 is a schematic diagram of a multimodal eye movement data mapping process provided by an embodiment of the present application;

[0082] Figure 4 is a structural schematic diagram of a multimodal eye movement data mapping device provided in an embodiment of the present application;

[0083] Figure 5 It is a structural diagram of the edge computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0085] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0086] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0087] In the following, in combination with the accompanying drawings, the multimodal eye movement data mapping method, apparatus, edge computing device and medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0088] The multimodal eye movement data mapping method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0089] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0090] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0091] The multimodal eye movement data mapping method provided in the embodiment of the present application may be executed by an edge computing device or a functional module or functional entity in the edge computing device that can implement the multimodal eye movement data mapping method. The edge computing device mentioned in the embodiment of the present application may be a portable device (such as a mobile phone, a tablet computer), a wearable device (such as a smart watch, a smart bracelet), an eye tracker, an AR (Augmented Reality) / VR (Virtual Reality) / XR (Extended Reality) / MR (Mixed Reality) device, an edge server, an edge gateway, a vehicle-mounted device, a cloud computing device, etc. The multimodal eye movement data mapping method provided in the embodiment of the present application is described below using the edge computing device as an example of the execution subject.

[0092] like Figure 1 As shown, the multimodal eye movement data mapping method includes: step 110, step 120, step 130 and step 140.

[0093] Step 110: Acquire an infrared eye movement image collected by an infrared camera and an RGB target image collected by a visible light camera; the infrared eye movement image includes eye movement data.

[0094] In eye tracking technology, the traditional RGB image acquisition method can provide good visual effects in a well-lit environment. However, in special conditions such as low light, at night, when the user wears sunglasses or there is an obstruction, the quality of the collected RGB images will be significantly reduced, and it may even be impossible to obtain effective user eye movement information from the collected RGB images.

[0095] Since infrared light has strong penetrating power, it can penetrate the lenses of sunglasses or other obstructions. In addition, due to the characteristics of infrared radiation, infrared images (IR) mainly rely on the thermal radiation of the object itself for imaging. Therefore, infrared images collected by infrared cameras can obtain the user's eye movement information in various environments such as low light, at night, wearing sunglasses, and under obstructions. Therefore, compared with RGB images, collecting infrared images can improve the scene universality, completeness, and accuracy of collected data.

[0096] However, due to the monochrome characteristics of infrared images, they are not as colorful as RGB images, which conforms to the visual habits of the human eye. Therefore, it is necessary to map the infrared images and eye movement data collected by the infrared camera to RGB images for easy visualization.

[0097] In the embodiment of the present application, the infrared eye movement image may include an infrared image and eye movement data collected by an infrared camera of an eye tracking device. For example, the eye tracking device may be a head-mounted device, and the infrared camera of the head-mounted device collects an infrared image of the field of view of the user wearing the head-mounted device, and the infrared image includes things within the field of view of the user wearing the head-mounted device. The eye movement data may be data related to the user's eye movement captured by an eye tracker of the head-mounted device, for example, the eye movement data may be data such as eye gaze point, gaze direction, line of sight trajectory, gaze duration, etc. The eye tracker may be a wearable eye tracker, a screen-type eye tracker, a head-mounted eye tracker, etc., and the eye tracker may capture the real-time state of the eye, such as pupil, corneal reflection, etc. to obtain eye movement data. It should be noted that the eye tracking device may also be other types of devices, such as a telemetry eye tracker, and the embodiment of the present application does not limit the specific type of the eye tracking device.

[0098] In one scenario, the head-mounted device can be equipped with an infrared camera and an eye tracker. The infrared camera is responsible for collecting infrared images of the user's field of view, while the eye tracker synchronously collects eye movement data such as the user's gaze point, eye movement trajectory, and gaze duration within the field of view.

[0099] In an embodiment of the present application, the RGB target image may be an RGB image captured by a visible light camera, and the shooting angle of the visible light camera is different from that of the infrared camera. For example, in one scenario, a user is in a room, and an infrared image may be captured by an infrared camera on a head-mounted device worn by the user. The capturing angle of the infrared camera may change as the user's neck turns. A visible light camera may also be fixed in the room, and the visible light camera may change the shooting angle or may be fixed. By mapping the eye movement data in the infrared eye movement image captured by the infrared camera to the RGB target image captured by the visible light camera, it is possible to analyze the user's gaze behavior.

[0100] Step 120: Determine the first infrared image block including eye movement data in the infrared eye movement image.

[0101] In the embodiment of the present application, the first infrared image block is a local area of ​​the infrared eye movement image, and the eye movement data is included in the local area.

[0102] In some embodiments, the position of the eye movement data in the infrared eye movement image can be pre-acquired, such as pre-annotated manually, or calculated by an algorithm. For example, the position and movement of the user's eyes can be captured by an eye tracker, and the reflection generated by the light irradiating into the eyes can be used to extract the reflection characteristics of the pupil and cornea, and the direction of the line of sight can be calculated by analyzing the pupil position and the corneal reflection point. Using geometric principles, combined with the eye structure and light path, the specific position of the eye movement data in the infrared image can be inferred.

[0103] Step 130, in response to the matching points in the first infrared image block satisfying a preset condition, calculating the coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the first infrared image block; the matching points represent points with similar features in the infrared eye movement image and the RGB target image.

[0104] In the embodiment of the present application, the matching points represent points with similar features in the infrared eye movement image and the RGB target image. These points with similar features can be some unique and identifiable feature points in the image, such as bright spots, spots, corner points or other significant visual features. These feature points are equivalent to visual landmarks in the image and are parts of the image that are easily identified and distinguished. In tasks such as image registration, stereoscopic vision, image stitching or object recognition, matching points are used to establish correspondence between different images. It should be noted that feature points and matching points can be a pixel point or an area including multiple pixels.

[0105] In some embodiments, feature point matching can be used to identify which feature points in the infrared eye movement image and the RGB target image are matching points. Feature point matching is a process of extracting feature points from two or more images and identifying whether these feature points are matching points. Specifically, feature extraction algorithms, such as scale-invariant feature transformation, accelerated robust features, etc., can be used to extract feature points in the infrared eye movement image and the RGB target image, respectively, and descriptors can be generated for the feature points. The descriptors can be quantitative representations of the features of the neighborhood around the feature points, which can be calculated by the pixel values ​​of the neighborhood around the feature points. The descriptors are used to describe the characteristics of the feature points so that the feature points can be identified and matched in different images. Then, points matching the feature points in the infrared eye movement image are searched in the RGB target image. For example, the descriptors of the feature points in the infrared eye movement image and the RGB target image can be compared, and the similarity of the feature points can be evaluated by distance metrics, such as Euclidean distance and Hamming distance, and the feature points with similarity greater than a preset threshold are determined as matching points.

[0106] In the embodiment of the present application, matching points are obtained by matching feature points, and the spatial relationship between the infrared eye movement image and the RGB target image can be established, providing an accurate reference benchmark for subsequent infrared image segmentation, and enabling the eye movement data to be effectively located and mapped to the corresponding position of the RGB target image. In addition, the determination of matching points also helps to optimize the allocation of computing resources, because it is possible to focus on these matching points without redundantly processing the entire image, thereby improving the efficiency and accuracy of eye tracking.

[0107] In an embodiment of the present application, when the first infrared image block meets preset conditions, such as the number and position distribution of matching points in the first infrared image block meet certain conditions, the coordinate mapping relationship between the infrared eye movement infrared image block and the RGB target image can be calculated through image processing and computer vision algorithms.

[0108] Step 140: Map the eye movement data to the RGB target image according to the coordinate mapping relationship.

[0109] In an embodiment of the present application, after determining the position of the eye movement data in the infrared eye movement image, an operation can be performed based on the coordinate mapping relationship and the position of the eye movement data in the infrared eye movement image to obtain the position of the eye movement data in the RGB target image.

[0110] According to the multimodal eye movement data mapping method of the present application, an infrared eye movement image collected by an infrared camera and an RGB target image collected by a visible light camera are obtained; the infrared eye movement image includes an infrared image and eye movement data; a first infrared image block including eye movement data in the infrared eye movement image is determined; in response to the matching points in the first infrared image block satisfying a preset condition, a coordinate mapping relationship between the infrared eye movement image and the RGB target image is calculated according to the first infrared image block; the matching points represent points with similar features in the infrared eye movement image and the RGB target image; and the eye movement data is mapped to the RGB target image according to the coordinate mapping relationship. The embodiment of the present application obtains infrared eye movement images and RGB target images of different modalities, and calculates the coordinate mapping relationship between the infrared eye movement image and the RGB target image when the infrared image block containing the eye movement data satisfies the preset conditions, so that the calculation process pays more attention to the area in the image related to the eye movement data, rather than the entire infrared eye movement image, thereby reducing the offset between images of different modalities, thereby reducing the mapping error between images of different modalities, and improving the accuracy of eye movement data mapping.

[0111] In some embodiments, before acquiring a first infrared image block including eye movement data in an infrared eye movement image, the method includes:

[0112] Inputting the infrared eye movement image and the RGB target image into a preset multimodal model so that the multimodal model can extract feature points of the infrared eye movement image and the RGB target image, and matching is performed based on the extracted feature points; or preprocessing the infrared eye movement image and the RGB target image, the preprocessing includes removing feature points within a preset range of the infrared eye movement image boundary, and removing feature points within a preset range of the RGB target image boundary, and inputting the preprocessed infrared eye movement image and the RGB target image into a preset multimodal model so that the multimodal model can extract feature points of the preprocessed infrared eye movement image and the RGB target image, and matching is performed based on the extracted feature points;

[0113] The successfully matched feature points are determined as matching points.

[0114] In this embodiment, the multimodal model is a machine learning model that can process and integrate images from different modalities. The multimodal model can understand and associate the intrinsic connections between images of different modalities, thereby improving the accuracy of feature point matching.

[0115] In this embodiment, a multimodal model can be pre-trained so that the multimodal model can recognize and process feature points in images of different modalities. For example, the neural network in the multimodal model can analyze the input infrared eye movement image and RGB target image through its multi-layer structure, extract key visual features, including texture, temperature, shape, edge, etc., and then calculate the similarity of the feature points. If the similarity is greater than a certain value, it means that the match is successful, and the successfully matched feature points are output as matching points. In this embodiment, the multimodal model used can be a LightGLUE model, an MMT (Multimodal Transformer) model, a ViLT (Vision-and-Language Transformer) model, etc., and the embodiments of the present application are not limited to this.

[0116] In this embodiment, during the feature point matching process of the multimodal model, feature extraction is performed on the infrared eye movement image and the RGB target image respectively, and the extracted feature points are mapped to a common multimodal space, so that images of different modalities can be compared and matched in a unified space.

[0117] In some embodiments, the preprocessed infrared eye movement image and RGB target image can be input into a preset neural network model so that the neural network model can extract feature points of the preprocessed infrared eye movement image and RGB target image, and match them according to the extracted feature points. By removing feature points within a preset range of the image boundary, the negative impact of noise or incompleteness that may exist in the image boundary area on the matching accuracy can be reduced, so that more focus can be placed on more stable and information-rich areas in the image, thereby improving the accuracy of matching.

[0118] In this embodiment, since the multimodal model is capable of processing and analyzing data from different modalities, cross-modal shared features of infrared eye movement images and RGB target images can be extracted through the multimodal model, thereby providing a more accurate correspondence when matching feature points and improving the accuracy of cross-modal feature point matching.

[0119] In some embodiments, the multimodal model includes a self-attention unit, a cross-attention unit and a classifier; the input dimension of the cross-attention unit includes a grayscale channel dimension; the self-attention unit and the cross-attention unit are used to extract feature points of the infrared eye movement image and the RGB target image, and calculate the confidence that the feature points of the infrared eye movement image and the feature points of the RGB target image are matching points, and the classifier is used to output the feature point as a matching point when the confidence that the feature point is a matching point is greater than a preset threshold.

[0120] In this embodiment, the feature point matching process of the multimodal model is as follows: Figure 2As shown, a multimodal model may include multiple layers, each layer includes a self-attention unit and a cross-attention unit, and these layers jointly process the input image pair, which can be used to extract feature points of infrared eye tracking images and RGB target images, and calculate the matching degree between feature points of different images and the obviousness of feature points. The matching degree and the obviousness of feature points are represented by confidence scores. The information of feature points extracted by different layers will be different because different layers may focus on different feature dimensions. Among them, confidence can be calculated by different methods, such as feature point-based descriptors or through additional confidence branch networks.

[0121] The multimodal model also includes a classifier, which can be a binary classifier that determines whether a feature point meets the matching condition based on the confidence of the feature point. For example, when the confidence calculated between the feature point of the infrared eye movement image and the feature point of the RGB target image is greater than a preset threshold, the two feature points are determined to be matching points, and the matching model is controlled to stop calculation and reasoning, and the output is the matching point. Figure 2 As shown in the figure, the leftmost side is the input, which is two images. The first row represents infrared eye tracking images, and the second row represents RGB target images. The dots in the images represent feature points. Through the layers and classifiers of the matching model, the output matching points are the dots in the two images after the classifier. The points containing "×" in the circles in the images are feature points that have not been matched. Among them, the output layer of the multimodal model can use a lightweight output head to convert the output of the classifier into the final task output, such as the location of the matching point, the confidence of the matching point, etc. Due to the reduction of parameters and computation, the lightweight output head can provide fast response during reasoning and is suitable for real-time applications.

[0122] In this embodiment, the input dimension of the cross attention unit includes the grayscale channel dimension. The grayscale channel refers to a separate brightness channel in the image, which represents the brightness value of each pixel in the image. In a grayscale image, each pixel has only one value, which represents the color depth of the pixel, that is, the brightness.

[0123] Taking the LightGLUE model adopted by the multimodal model as an example, according to the characteristics of infrared images, the input dimension of the cross-attention unit of the LightGLUE model can be adjusted from the original (w, h, 3) to (w, h, 1). Among them, w in (w, h, 3) and (w, h, 1) represents the image width, h represents the image height, 3 represents the number of channels, that is, the three color channels of the RGB image, and 1 represents the grayscale channel. Since the image can be represented by features after passing through the multimodal model, and the infrared image contains more temperature information, therefore, adjusting the input dimension from 3 channels to 1 channel can make the model pay more attention to the image's temperature distribution characteristics rather than color features during the feature extraction process, thereby improving the accuracy of feature extraction of infrared eye movement images.

[0124] In this embodiment, by setting the input dimension of the cross-attention unit to include the grayscale channel dimension, the model pays more attention to the temperature distribution characteristics of the image rather than the color characteristics during the feature extraction process, thereby improving the accuracy of feature extraction of infrared eye movement images; the confidence of the feature points as matching points is calculated through the self-attention mechanism. When the confidence exceeds the preset threshold, the classifier can control the model to stop reasoning. These feature points are identified by the classifier and output as matching points, which not only improves the accuracy of matching point identification, but also avoids unnecessary calculations and reduces the consumption of computing resources.

[0125] In some embodiments, the infrared eye movement image is a video frame in a target video stream, and the method further includes:

[0126] In response to the failure of matching the infrared eye movement image and the RGB target image, a video frame is reselected from the target video stream to replace the infrared eye movement image and the RGB target image for feature point matching.

[0127] In this embodiment, the target video stream is a continuous sequence of video frames used to analyze the user's visual behavior in eye tracking. The target video stream can be captured by an infrared camera, and the user's line of sight movement can be accurately tracked by analyzing the target video stream. The infrared eye image can be a video frame in the target video stream. When the infrared eye image and the RGB target image fail to match, it means that the feature points in the infrared eye image are not similar enough to the feature points in the RGB target image. Other video frames in the target video stream can be selected to replace the infrared eye image as a new infrared eye image to match the RGB target image.

[0128] In this embodiment, when feature point matching fails to successfully establish an accurate correspondence between the infrared eye movement image and the RGB target image, a new video frame can be flexibly selected from a continuous video stream as the infrared eye movement image, thereby reducing the risk of interruption of the entire eye tracking process due to a single matching failure and improving the continuity of the system.

[0129] In some embodiments, reselecting a video frame from the target video stream to replace the infrared eye movement image and the RGB target image for feature point matching includes:

[0130] A video frame adjacent to the video frame corresponding to the infrared eye movement image in the target video stream or other video frames in the target video stream are determined as the reselected video frames.

[0131] In this embodiment, the video frame adjacent to the video frame corresponding to the infrared eye movement image or other video frames in the target video stream can be determined as the reselected video frame as the new infrared eye movement image to continue matching with the RGB target image. In this embodiment, when the feature point matching between the infrared eye movement image and the RGB target image fails, a video frame adjacent to the current infrared eye movement image is selected from the target video stream as a replacement to re-match the feature points, so that the reselected video frame also maintains the continuity in time, or other video frames are selected as replacements, thereby improving the stability and accuracy of the eye tracking process.

[0132] In some embodiments, the method further comprises:

[0133] Store the feature points of the RGB target image extracted by the multimodal model;

[0134] Reselect video frames from the target video stream to replace the infrared eye movement image and the RGB target image for feature point matching, including:

[0135] The replaced infrared eye movement image is input into a preset multimodal model so that the multimodal model can extract feature points of the infrared eye movement image and match the feature points of the infrared eye movement image with feature points of the stored RGB target image.

[0136] In this embodiment, since the infrared eye movement image and the RGB target image may fail to match, if a new video frame is selected as the infrared eye movement image to match the RGB target image after each matching failure, and the feature points of the RGB target image are extracted, computing resources will be wasted. Therefore, the feature points of the RGB target image can be stored, so that after the matching fails, the feature points of a new infrared eye movement image can be extracted, and the feature points of the stored RGB target image can be reused without repeatedly extracting the feature points of the RGB target image.

[0137] In this embodiment, the features of the RGB target image are extracted and stored through a multimodal model. When other video frame images are matched with the RGB target image, the feature points of the RGB target image can be called for feature matching. There is no need to repeatedly extract the features of the RGB target image, thereby improving processing efficiency.

[0138] In some embodiments, determining a first infrared image block including eye movement data in the infrared eye movement image includes:

[0139] Determine the location of the eye movement data in the infrared eye movement image;

[0140] A first infrared image block is determined according to a position of the eye movement data in the infrared eye movement image.

[0141] In this embodiment, the position of the eye movement data in the infrared eye movement image can be pre-acquired, such as pre-annotated manually, or calculated by an algorithm. For example, the position and movement of the user's eyes can be captured by an eye tracker, and the reflection generated by the light irradiating into the eyes can be used to extract the reflection characteristics of the pupil and cornea, and the direction of the line of sight can be calculated by analyzing the pupil position and the corneal reflection point. Using geometric principles, combined with the eye structure and light path, the specific position of the eye movement data in the infrared image can be inferred.

[0142] After determining the position of the eye movement data in the infrared eye movement image, the first infrared image block can be determined according to the position of the eye movement data. For example, the first infrared image block is an area within a preset range of the infrared eye movement image that contains the position of the eye movement data, and the first infrared image block can be a circular area, a square area, etc.

[0143] In this embodiment, by accurately identifying the position of the eye movement data and determining the first infrared image block according to the position of the eye movement data, irrelevant areas in the processed image can be reduced so that the infrared image block contains more information related to the eye movement data.

[0144] In some embodiments, determining the first infrared image block according to the position of the eye movement data in the infrared eye movement image includes:

[0145] An area within a first preset range centered at the position of the eye movement data in the infrared eye movement image is determined as a first infrared image block.

[0146] In this embodiment, the first preset range is an area centered on the position of the eye movement data in the infrared eye movement image. The first preset range can be a circular area, a square area, etc. The first preset range is smaller than the infrared eye movement image. For example, if the resolution of the infrared eye movement image is W*H, the first preset range can be an area centered on the eye movement data with a size of 0.1W and 0.1H.

[0147] In this embodiment, by determining the area within the first preset range centered on the position of the eye movement data as the first infrared image block, the calculation process can focus on the area near the eye movement data, thereby reducing invalid calculations in irrelevant areas and improving calculation efficiency and accuracy.

[0148] In some embodiments, the preset conditions include: the number of matching points in the infrared image block is greater than or equal to a preset number, and / or the multiple matching points in the infrared image block are not collinear.

[0149] In this embodiment, a preset condition is set to require that the number of matching points is greater than or equal to a preset number, and at least three of these matching points are not collinear, so that there are enough matching points to calculate the coordinate mapping relationship. The condition that multiple matching points are not collinear enables the matching points to provide sufficient geometric information to solve the coordinate mapping relationship between the infrared eye movement image and the RGB target image, thereby further improving the accuracy of eye movement data mapping.

[0150] In some embodiments, the method further comprises:

[0151] In response to the first infrared image block not satisfying a preset condition, acquiring a second infrared image block including eye movement data in the infrared eye movement image;

[0152] In response to the second infrared image block satisfying a preset condition, a coordinate mapping relationship between the infrared eye movement image and the RGB target image is calculated according to the second infrared image block.

[0153] In this embodiment, if the first infrared image block does not meet the preset conditions, a second infrared image block can be re-determined, the second infrared image block also includes eye movement data, and the second infrared image block and the first infrared image block are different local areas in the infrared eye movement image. The second infrared image block and the first infrared image block can be infrared image blocks of the same shape, or infrared image blocks of different shapes; the second infrared image block can be an infrared image block obtained by translating the first infrared image block, or an infrared image block obtained by enlarging the first infrared image block.

[0154] In this embodiment, if the second infrared image block meets the preset condition, the coordinate mapping relationship between the infrared eye movement image and the RGB target image can be calculated according to the second infrared image block.

[0155] In this embodiment, by obtaining a second infrared image block including eye movement data in the infrared eye movement image when the first infrared image block does not meet the preset conditions, even if a sufficient number or a suitable distribution of matching points cannot be found within the range of the first infrared image block, the possibility of finding valid matching points can be increased by adjusting the search area, thereby enhancing the flexibility and success rate of the multimodal eye movement data mapping process.

[0156] In some embodiments, obtaining a second infrared image block including eye movement data in the infrared eye movement image includes:

[0157] The second infrared image block is acquired according to the position of the first infrared image block in the infrared eye movement image.

[0158] In this embodiment, the second infrared image block may be an infrared image block obtained by translating the first infrared image block, or an infrared image block obtained by enlarging the first infrared image block. The second infrared image block may be determined by the position of the first infrared image block.

[0159] In this embodiment, by acquiring the second infrared image block according to the position of the first infrared image block in the infrared eye movement image, a new search range can be re-determined with the first infrared image block as a reference, thereby improving the efficiency of the multimodal eye movement data mapping calculation process.

[0160] In some embodiments, obtaining a second infrared image block including eye movement data in the infrared eye movement image includes:

[0161] Determine the position of the eye movement data in the infrared eye movement image; determine an area within a second preset range centered on the position of the eye movement data in the infrared eye movement image as a second infrared image block; wherein the second preset range is larger than the first preset range.

[0162] In this embodiment, the first preset range can be expanded to a second preset range to obtain a second infrared image block, so as to expand the search range to find enough matching points, thereby improving the accuracy of calculating the coordinate mapping relationship. For example, the first preset range can be expanded to 1.1 times, 1.2 times, 1.5 times, 2 times, or other multiples of the original to obtain the second preset range.

[0163] In this embodiment, by expanding the first infrared image block into the second infrared image block, even if a sufficient number or a suitable distribution of matching points cannot be found within the initial search range, the possibility of finding valid matching points can be increased by expanding the search area, thereby enhancing the flexibility and success rate of the multimodal eye movement data mapping process.

[0164] In some embodiments, calculating the coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the first infrared image blocks includes:

[0165] Acquire first position coordinates of a matching point in the first infrared image block in the infrared eye movement image and second position coordinates of the matching point in the RGB target image;

[0166] The homography matrix corresponding to the matching point is calculated according to the first position coordinates and the second position coordinates; wherein the homography matrix represents the coordinate mapping relationship.

[0167] The homography matrix describes the projection relationship between different planes. This projection relationship can be used to map points on one plane to corresponding points on another plane, and is usually used in scenes such as image matching, stereoscopic vision, and image stitching. In this embodiment, the coordinate mapping relationship can be represented by a homography matrix, which is used to transform the points in the infrared eye movement image into the coordinate system of the RGB target image so that the two images can be aligned. For example, P1 is a point in the infrared eye movement image, and P2 is a point in the RGB target image, then the coordinate mapping relationship between P1 and P2 can be expressed as: P2=M·P1. Where M represents the homography matrix.

[0168] In this embodiment, the first position coordinates of the matching point in the first infrared image block in the infrared eye movement image and the second position coordinates of the matching point in the RGB target image can be obtained. For example, there are four matching points in the first infrared image block, namely matching point 1, matching point 2, matching point 3 and matching point 4. The coordinates of matching point 1 in the infrared eye movement image are The coordinates in the RGB target image are The coordinates of matching point 2 in the infrared eye movement image are The coordinates in the RGB target image are The coordinates of matching point 3 in the infrared eye movement image are The coordinates in the RGB target image are The coordinates of matching point 4 in the infrared eye movement image are The coordinates in the RGB target image are Then use the position coordinates of these matching points in different images to construct a linear equation system to calculate the homography matrix M:

[0169]

[0170] Among them, (m 11 ,m 12 ,Λ,m 43 ) are the elements of the homography matrix M, n = 1, 2, 3, 4.

[0171] Then, the least square method or other numerical methods, such as direct linear transformation, are used to solve the above equations to obtain the value of the element M of the homography matrix M. It can be seen that if the number of matching points is greater than 4 and these matching points are not collinear, a unique solution to the homography matrix H can be obtained. Therefore, the preset condition can be determined based on the number of matching points and / or the distribution of matching points. For example, the preset condition can be set to be greater than or equal to 4, and these matching points are not collinear. Of course, the above calculation process is based on the example of 4 matching points. When the number of matching points is other values, the above calculation process can be adaptively modified according to the number of matching points.

[0172] like Figure 3 As shown, the five-pointed star in the infrared eye movement image represents the position of the eye movement data. After the coordinate mapping relationship is calculated, the position of the eye movement data can be calculated with the coordinate mapping relationship to obtain the position of the eye movement data in the RGB target image.

[0173] In this embodiment, by calculating the corresponding position coordinates of the matching points in the infrared eye movement image and the RGB target image, a mathematical model can be established to describe the spatial relationship between these matching points, and based on this, a homography matrix is ​​determined to represent the coordinate mapping relationship. Representing the coordinate mapping relationship by the homography matrix can adapt to complex image transformation situations, thereby improving the accuracy and robustness of multimodal eye movement data mapping.

[0174] In some embodiments, the method further comprises:

[0175] Determine the sum of confidences of matching points in the first infrared image block; calculate a matching score based on the sum of confidences, the number of matching points in the first infrared image block, and the area of ​​the first infrared image block; wherein the matching score represents the accuracy of mapping the eye movement data to the RGB target image.

[0176] In this embodiment, the sum of the confidences reflects the reliability of the matching points. The higher the confidence, the more accurate the identification of the matching points, thereby providing a solid foundation for coordinate mapping. Secondly, the number of matching points directly affects the stability and accuracy of the coordinate mapping. More matching points can provide richer spatial constraints, which helps to improve the accuracy of matching. Furthermore, the area of ​​the preset range is considered to take into account the distribution density of the matching points, so that in search areas of different sizes, the matching score can reflect the actual matching quality. Combining these factors, the matching score can provide a comprehensive indicator to evaluate the credibility and accuracy of the matching results.

[0177] In some embodiments, the formula

[0178] Score=∑Score Matching_Points *Number Matching_Points *(1 / Area Subregion )

[0179] Calculate the matching score;

[0180] Among them, Score represents the matching score, Score Matching_Points Represents the sum of the confidences of the matching points in the first infrared image block. The confidence indicates the obviousness and matching degree of the feature points. The more obvious the feature and the higher the matching degree, the higher the confidence. Matching_Points Indicates the number of matching points in the first infrared image block, Area SubregionRepresents the area of ​​the first preset range. It should be noted that, if the coordinate mapping relationship between the infrared eye movement image and the RGB target image is calculated based on the second infrared image block, the matching score is correspondingly calculated based on the sum of the confidences of the matching points in the second infrared image block and the number of matching points in the second infrared image block and the area of ​​the second infrared image block.

[0181] In this embodiment, the matching score is calculated by comprehensively considering the sum of the confidences of the first target matching points, the number of matching points, and the area of ​​the first infrared image block, which effectively quantifies the accuracy of the position of the eye movement data in the RGB target image, so that the matching score can provide a comprehensive indicator for evaluating the credibility and accuracy of the matching results.

[0182] The multimodal eye movement data mapping method provided in the embodiment of the present application can be executed by a multimodal eye movement data mapping device. In the embodiment of the present application, the multimodal eye movement data mapping method is executed by a multimodal eye movement data mapping device as an example to illustrate the multimodal eye movement data mapping method provided in the embodiment of the present application.

[0183] The embodiment of the present application also provides a multimodal eye movement data mapping device.

[0184] like Figure 4 As shown, the multimodal eye movement data mapping device includes:

[0185] An acquisition module 410 is used to acquire an infrared eye movement image acquired by an infrared camera and an RGB target image acquired by a visible light camera; the infrared eye movement image includes eye movement data;

[0186] A determination module 420, configured to determine a first infrared image block including eye movement data in the infrared eye movement image;

[0187] A calculation module 430, configured to calculate a coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the first infrared image block in response to the matching points in the first infrared image block satisfying a preset condition;

[0188] The mapping module 440 is used to map the eye movement data to the RGB target image according to the coordinate mapping relationship.

[0189] According to the multimodal eye movement data mapping device of the present application, an infrared eye movement image collected by an infrared camera and an RGB target image collected by a visible light camera are obtained; the infrared eye movement image includes eye movement data; a first infrared image block including eye movement data in the infrared eye movement image is determined; in response to the matching points in the first infrared image block satisfying a preset condition, a coordinate mapping relationship between the infrared eye movement image and the RGB target image is calculated according to the first infrared image block; and the eye movement data is mapped to the RGB target image according to the coordinate mapping relationship. The embodiment of the present application obtains infrared eye movement images and RGB target images of different modalities, and calculates the coordinate mapping relationship between the infrared eye movement image and the RGB target image when the infrared image block containing the eye movement data satisfies the preset conditions, so that the calculation process pays more attention to the area in the image related to the eye movement data, rather than the entire infrared eye movement image, thereby reducing the offset between images of different modalities, thereby reducing the mapping error between images of different modalities, and improving the accuracy of eye movement data mapping.

[0190] In some embodiments, the acquisition module 410 is further configured to:

[0191] Inputting the infrared eye movement image and the RGB target image into a preset multimodal model so that the multimodal model can extract feature points of the infrared eye movement image and the RGB target image, and matching is performed based on the extracted feature points; or preprocessing the infrared eye movement image and the RGB target image, the preprocessing includes removing feature points within a preset range of the infrared eye movement image boundary, and removing feature points within a preset range of the RGB target image boundary, and inputting the preprocessed infrared eye movement image and the RGB target image into a preset multimodal model so that the multimodal model can extract feature points of the preprocessed infrared eye movement image and the RGB target image, and matching is performed based on the extracted feature points;

[0192] The successfully matched feature points are determined as matching points.

[0193] In some embodiments, the acquisition module 410 is further configured to:

[0194] In response to the failure of matching the infrared eye movement image and the RGB target image, a video frame is reselected from the target video stream to replace the infrared eye movement image and the RGB target image for feature point matching.

[0195] In some embodiments, the acquisition module 410 is further configured to:

[0196] A video frame adjacent to the video frame corresponding to the infrared eye movement image in the target video stream or other video frames in the target video stream are determined as the reselected video frames.

[0197] In some embodiments, the acquisition module 410 is further configured to:

[0198] Store the feature points of the RGB target image extracted by the multimodal model;

[0199] Reselect video frames from the target video stream to replace the infrared eye movement image and the RGB target image for feature point matching, including:

[0200] The replaced infrared eye movement image is input into a preset multimodal model so that the multimodal model can extract feature points of the infrared eye movement image and match the feature points of the infrared eye movement image with feature points of the stored RGB target image.

[0201] In some embodiments, the determination module 420 is further configured to:

[0202] Determine the location of the eye movement data in the infrared eye movement image;

[0203] A first infrared image block is determined according to a position of the eye movement data in the infrared eye movement image.

[0204] In some embodiments, the determination module 420 is further configured to:

[0205] An area within a first preset range centered at the position of the eye movement data in the infrared eye movement image is determined as a first infrared image block.

[0206] In some embodiments, the determination module 420 is further configured to:

[0207] Acquire the second infrared image block according to the position of the first infrared image block in the infrared eye movement image; or,

[0208] Determine the position of the eye movement data in the infrared eye movement image; determine an area within a second preset range centered on the position of the eye movement data in the infrared eye movement image as a second infrared image block; wherein the second preset range is larger than the first preset range.

[0209] In some embodiments, the calculation module 430 is further configured to:

[0210] Acquire first position coordinates of a matching point in the first infrared image block in the infrared eye movement image and second position coordinates of the matching point in the RGB target image;

[0211] The homography matrix corresponding to the matching point is calculated according to the first position coordinates and the second position coordinates; wherein the homography matrix represents the coordinate mapping relationship.

[0212] In some embodiments, the mapping module 440 is further configured to:

[0213] Determine the sum of confidences of matching points in the first infrared image block; calculate a matching score based on the sum of confidences, the number of matching points in the first infrared image block, and the area of ​​the first infrared image block; wherein the matching score represents the accuracy of mapping the eye movement data to the RGB target image.

[0214] The multimodal eye movement data mapping device in the embodiment of the present application can be an edge computing device, or a component in the edge computing device, such as an integrated circuit or a chip. The edge computing device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the edge computing device can be a mobile phone, a tablet computer, a laptop computer, an eye tracker, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personaldigital assistant, PDA), etc. It can also be a server, a network attached storage (Network AttachedStorage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited. In an optional example, a multimodal eye movement data mapping device is deployed as an edge computing device or component on the edge deployment solution of the terminal side, which can migrate at least part of the computing tasks to the terminal side device, give full play to the advantages of edge computing to improve data processing efficiency, reduce latency and network resource occupancy and dependence, and is also beneficial to data security and privacy protection.

[0215] The multimodal eye movement data mapping device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0216] In some embodiments, Figure 5 As shown, an embodiment of the present application also provides an edge computing device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, each process of the above-mentioned multimodal eye movement data mapping method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0217] It should be noted that the edge computing devices in the embodiments of the present application include the above-mentioned mobile edge computing devices and non-mobile edge computing devices.

[0218] The embodiment of the present application also provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned multimodal eye movement data mapping method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0219] The processor is the processor in the edge computing device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0220] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned multimodal eye movement data mapping method when executed by a processor.

[0221] The processor is the processor in the edge computing device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0222] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned multimodal eye movement data mapping method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0223] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0224] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0225] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0226] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0227] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0228] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A multimodal eye movement data mapping method, characterized in that: include: Acquire an infrared eye movement image collected by an infrared camera and an RGB target image collected by a visible light camera; The infrared eye movement image includes an infrared image and eye movement data; Determine a first infrared image block including eye movement data in the infrared eye movement image; In response to a matching point in the first infrared image block satisfying a preset condition, calculating a coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the first infrared image block; the matching point represents a point having similar features in the infrared eye movement image and the RGB target image; The eye movement data is mapped to the RGB target image according to the coordinate mapping relationship.

2. The method according to claim 1, characterized in that Before determining the first infrared image block including the eye movement data in the infrared eye movement image, the method includes: Inputting the infrared eye movement image and the RGB target image into a preset multimodal model, so that the multimodal model can extract feature points of the infrared eye movement image and the RGB target image, and matching them according to the extracted feature points; or preprocessing the infrared eye movement image and the RGB target image, wherein the preprocessing includes removing feature points within a preset range of the boundary of the infrared eye movement image and removing feature points within a preset range of the boundary of the RGB target image, and inputting the preprocessed infrared eye movement image and the RGB target image into a preset multimodal model, so that the multimodal model can extract feature points of the preprocessed infrared eye movement image and the RGB target image, and matching them according to the extracted feature points; The successfully matched feature points are determined as matching points.

3. The method according to claim 2, characterized in that The multimodal model includes a self-attention unit, a cross-attention unit and a classifier; the input dimension of the cross-attention unit includes a grayscale channel dimension; the self-attention unit and the cross-attention unit are used to extract feature points of the infrared eye movement image and the RGB target image, and calculate the confidence that the feature points of the infrared eye movement image and the feature points of the RGB target image are matching points, and the classifier is used to output the feature point as a matching point when the confidence that the feature point is a matching point is greater than a preset threshold.

4. The method according to claim 2, characterized in that The infrared eye-movement image is a video frame in a target video stream, and the method further includes: In response to a failure in matching between the infrared eye-movement image and the RGB target image, a video frame is reselected from the target video stream to replace the infrared eye-movement image and the RGB target image for feature point matching.

5. The method according to claim 4, characterized in that Reselecting a video frame from the target video stream to replace the infrared eye movement image and the RGB target image for feature point matching includes: A video frame adjacent to the video frame corresponding to the infrared eye movement image in the target video stream or other video frames of the target video stream are determined as the reselected video frames.

6. The method according to claim 4, characterized in that The method further comprises: Storing feature points of the RGB target image extracted by the multimodal model; Reselecting a video frame from the target video stream to replace the infrared eye movement image and the RGB target image for feature point matching includes: The replaced infrared eye movement image is input into a preset multimodal model so that the multimodal model can extract feature points of the infrared eye movement image and match the feature points of the infrared eye movement image with the feature points of the stored RGB target image.

7. The method according to claim 1, characterized in that The determining of the first infrared image block including eye movement data in the infrared eye movement image comprises: Determining a position of the eye movement data in the infrared eye movement image; The first infrared image block is determined according to the position of the eye movement data in the infrared eye movement image.

8. The method according to claim 7, characterized in that The determining the first infrared image block according to the position of the eye movement data in the infrared eye movement image comprises: An area within a first preset range centered at the position of the eye movement data in the infrared eye movement image is determined as the first infrared image block.

9. The method according to claim 1, characterized in that: The preset conditions include: the number of matching points in the infrared image block is greater than or equal to a preset number, and / or the multiple matching points in the infrared image block are not collinear; and / or, The method further includes: in response to the first infrared image block not satisfying a preset condition, acquiring a second infrared image block including the eye movement data in the infrared eye movement image; and / or, The method further includes: in response to the second infrared image block satisfying a preset condition, calculating a coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the second infrared image block.

10. The method according to claim 9, characterized in that The step of obtaining a second infrared image block including the eye movement data in the infrared eye movement image comprises: Acquire the second infrared image block according to the position of the first infrared image block in the infrared eye movement image; or, Determine the position of the eye movement data in the infrared eye movement image; determine an area within a second preset range centered on the position of the eye movement data in the infrared eye movement image as the second infrared image block; wherein the second preset range is larger than the first preset range.

11. The method according to claim 1, characterized in that: The calculating the coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the first infrared image blocks includes: Acquire a first position coordinate of a matching point in the first infrared image block in the infrared eye movement image and a second position coordinate of the matching point in the RGB target image; A homography matrix corresponding to the matching point is calculated according to the first position coordinates and the second position coordinates; wherein the homography matrix represents the coordinate mapping relationship.

12. The method according to claim 1, characterized in that The method further comprises: Determining the sum of confidences of matching points in the first infrared image block; calculating a matching score based on the sum of confidences, the number of matching points in the first infrared image block, and the area of ​​the first infrared image block; wherein the matching score represents the accuracy of mapping the eye movement data to the RGB target image; and / or, The eye movement data includes at least one of an eye gaze point, a gaze direction, and a sight track.

13. A multimodal eye movement data mapping device, characterized in that: include: An acquisition module is used to acquire an infrared eye movement image acquired by an infrared camera and an RGB target image acquired by a visible light camera; The infrared eye movement image includes an infrared image and eye movement data; A determination module, configured to determine a first infrared image block including eye movement data in the infrared eye movement image; a calculation module, configured to calculate a coordinate mapping relationship between the infrared eye movement image and the RGB target image according to the first infrared image block in response to a matching point in the first infrared image block satisfying a preset condition; The matching points represent points having similar features in the infrared eye movement image and the RGB target image; A mapping module is used to map the eye movement data to the RGB target image according to the coordinate mapping relationship.

14. An edge computing device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the multimodal eye movement data mapping method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the multimodal eye movement data mapping method according to any one of claims 1 to 12 is implemented.

Images