Eye tracking method, apparatus, device, medium and product

Abstract

The present disclosure relates to the technical field of machine vision, and in particular, to an eye movement tracking method, device, equipment, medium and product. The method comprises: acquiring a human eye image of a target person and mouse operation information on a terminal; extracting eye movement information in the human eye image; correcting the eye movement information through a mapping model to obtain corrected eye movement information; wherein the mapping model is pre-constructed based on the association between corresponding eye movement information samples and mouse operation information samples in different regions of the terminal screen; in the case that the similarity between the corrected eye movement information and the mouse operation information meets the requirements, determining a region of interest based on the mouse operation information, and statistically analyzing the region of interest. The present disclosure can eliminate the system error caused by different directions of the camera, ensure the accurate alignment between the eye movement information and the mouse operation information, and improve the accuracy of the eye movement tracking process.

Description

Technical Field

[0001] This disclosure relates to the technical field of machine vision, and more specifically, to an eye-tracking method, apparatus, device, medium, and product. Background Technology

[0002] In industrial production, weld defect assessment is a crucial step in ensuring product structural strength and safety. Eye-tracking technology can capture human visual gaze behavior in real time, which can be used to analyze the focus of assessors during weld defect assessment.

[0003] In related technologies, eye-tracking acquisition devices that are usually located in fixed positions are often misaligned due to their variable installation positions and varying shooting angles. This results in a misalignment between the eye-tracking coordinates and the screen pixel coordinates, making it impossible to achieve a precise match between the eye-tracking coordinates and the mouse drawing coordinates, thus leading to insufficient accuracy in eye-tracking. Summary of the Invention

[0004] This disclosure is made in view of the above-mentioned problems. This disclosure provides an eye-tracking method, apparatus, device, medium, and product.

[0005] In a first aspect, embodiments of this disclosure provide an eye-tracking method, comprising:

[0006] Acquire images of the target person's eyes and information about their mouse actions on the terminal.

[0007] Extract eye movement information from human eye images.

[0008] The eye-tracking information is corrected by a mapping model to obtain the corrected eye-tracking information. The mapping model is pre-constructed in different areas of the terminal screen based on the correlation between corresponding eye-tracking information samples and mouse operation information samples.

[0009] If the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements, the region of interest is determined based on the mouse operation information, and statistical analysis is performed on the region of interest.

[0010] In one embodiment of the first aspect, if the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements, a region of interest is determined based on the mouse operation information, including: Calculate the pixel distance between the corrected eye-tracking information and the mouse operation information.

[0011] Within a preset time window, if the pixel distance is less than a distance threshold and the duration of eye-tracking gaze is greater than a duration threshold, the mouse drawing trajectory corresponding to the mouse operation information is captured.

[0012] Regions of interest are generated based on mouse-drawn paths.

[0013] In one embodiment of the first aspect, generating a region of interest based on a mouse-drawn trajectory includes: When the mouse trajectory is drawn using multiple points, polygon fitting is performed on the mouse trajectory to generate the region of interest.

[0014] When the mouse cursor is drawn as a bounding box, a rectangular region is extracted from the mouse cursor and then the region of interest is generated.

[0015] In one embodiment of the first aspect, statistical analysis of the region of interest includes: Extract visual feature parameters of the region of interest; these visual feature parameters include geometric features, texture features, and grayscale features.

[0016] Visual feature parameters are input into the analysis model to output analysis results; the analysis results include the defect type, defect level and defect cause of weld defects.

[0017] In one embodiment of the first aspect, the mapping model is constructed in the following manner: A sample set is collected from different areas of the terminal screen; the sample set includes eye-tracking information samples and mouse operation information samples.

[0018] An initial model was constructed based on the correlation between eye-tracking information samples and mouse operation information samples.

[0019] The model parameters of the initial model are iteratively optimized based on the sample set, and the optimized model is obtained.

[0020] In one embodiment of the first aspect, after generating the region of interest based on the mouse drawing trajectory, the method further includes: The distribution function of fixation density within the region of interest is determined based on the corrected eye-tracking information.

[0021] The region with the highest fixation density in the distribution function is identified as the core region. Statistical analysis of the region of interest is then performed based on the visual feature parameters of the region of interest and the visual feature parameters of the core region.

[0022] In one embodiment of the first aspect, the error between the corrected eye-tracking information and the mouse operation information is less than a preset percentage of the terminal screen pixels.

[0023] In a second aspect, embodiments of this disclosure provide an eye-tracking device, comprising: The data acquisition unit is used to acquire images of the target person's eyes and mouse operation information on the terminal; The information extraction unit is used to extract eye movement information from human eye images; The information correction unit is used to correct eye movement information through a mapping model to obtain corrected eye movement information; wherein, the mapping model is pre-constructed in different areas of the terminal screen based on the correlation between corresponding eye movement information samples and mouse operation information samples.

[0024] The region analysis unit is used to determine the region of interest based on the mouse operation information, provided that the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements, and to perform statistical analysis on the region of interest.

[0025] In a third aspect, embodiments of this disclosure provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps in the above-described eye-tracking method are performed.

[0026] In a fourth aspect, embodiments of this disclosure provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the eye-tracking method described above.

[0027] In a fifth aspect, embodiments of this disclosure provide a computer program product stored in a storage medium, the program product being executed by at least one processor to implement the steps in the eye-tracking method described above.

[0028] As will be described in detail below, eye-tracking methods, apparatuses, devices, media, and products according to embodiments of the present disclosure. In embodiments of the present disclosure, the eye-tracking system first acquires an image of a target person's eyes and mouse operation information on a terminal, then extracts eye movement information from the eye image, and corrects the eye movement information using a mapping model to obtain corrected eye movement information. This mapping model is pre-constructed in different areas of the terminal screen based on the correlation between corresponding eye movement information samples and mouse operation information samples. Finally, if the similarity between the corrected eye movement information and the mouse operation information meets the requirements, a region of interest is determined based on the mouse operation information, and statistical analysis is performed on the region of interest. Through the above processing method, the constructed mapping model can eliminate systematic errors caused by different camera orientations based on the correlation between eye movement information samples and mouse operation information samples, thereby effectively improving the flexibility of the eye tracking process. By inputting the real-time extracted eye movement information into the mapping model, the model outputs corrected eye movement information, which ensures accurate alignment between eye movement information and mouse operation information. This enables the analysis of the region of interest to be triggered when the eye movement information and mouse operation information are highly aligned, thereby improving the accuracy of the eye tracking process.

[0029] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0030] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0031] Figure 1 This is a flowchart of an eye-tracking method according to an embodiment of the present disclosure.

[0032] Figure 2 This is an overall flowchart of an eye-tracking process according to an embodiment of the present disclosure.

[0033] Figure 3 This is a schematic diagram of an eye-tracking device according to an embodiment of the present disclosure.

[0034] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present disclosure.

[0035] Figure 5 This is a schematic diagram of a computer program product according to an embodiment of the present disclosure. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0037] Research has found that in industrial production, weld defect identification is a key step in ensuring product structural strength and safety. Currently, the mainstream identification method still relies on professional appraisers to visually delineate the Region of Interest (ROI) of weld defects on terminal equipment and combine this with experience to determine the defect type and level.

[0038] Meanwhile, with the development of artificial intelligence technology, AI models for weld defect detection based on machine vision have become a research hotspot, and the training of AI models requires a large number of labeled ROI feature samples.

[0039] Eye-tracking technology can capture human visual gaze behavior in real time, which can be used to analyze the focus of inspectors during the weld defect judgment process. If eye-tracking is combined with manual delineation of ROI, automatic acquisition and feature statistics of ROI can be achieved.

[0040] In related technologies, eye-tracking acquisition devices that are usually located in fixed positions are often misaligned due to their variable installation positions and varying shooting angles. This results in a misalignment between the eye-tracking coordinates and the screen pixel coordinates, making it impossible to achieve a precise match between the eye-tracking coordinates and the mouse drawing coordinates, thus leading to insufficient accuracy in eye-tracking.

[0041] To facilitate understanding of this embodiment, a detailed description of the eye-tracking method disclosed in this disclosure is provided first. This method is applied to defect assessment scenarios for pressure vessel welds, and can also be applied to defect assessment scenarios for various industrial welds such as steel structures, ships, and pipelines. See [link to relevant documentation]. Figure 1 The diagram shows a flowchart of an eye-tracking method provided in an embodiment of this disclosure. The method includes steps S101 to S104, wherein: Step S101: Obtain the target person's eye image and mouse operation information on the terminal.

[0042] In this embodiment of the disclosure, the eye-tracking system can acquire images of the target person's eyes through the terminal camera, acquire images of weld defects displayed on the terminal, and simultaneously acquire mouse operation coordinate information when the target person outlines the area of ​​interest of the weld defect.

[0043] The target person can be an appraiser, and the terminal camera can be set to a built-in camera on a laptop or an infrared camera, which does not require a fixed position. It can also be adjusted as needed.

[0044] The eye-tracking system mentioned in this disclosure includes an image acquisition module, a coordinate acquisition module, a data synchronization module, and an eye-tracking feature extraction module.

[0045] The image acquisition module uses a laptop's built-in 1080P camera to capture images of the human eye and weld defects displayed on the terminal at a frame rate of 30fps.

[0046] The coordinate acquisition module can collect mouse operation information of the target person when drawing ROI based on the mouse driver on the computer, and the acquisition frequency is synchronized with the image acquisition, for example, 30fps.

[0047] The data synchronization module adds a unified timestamp to the collected human eye images, weld defect images, and mouse operation information. The timestamp accuracy can be 1ms, thereby ensuring that the time synchronization error of the three types of data does not exceed 10ms. The synchronized data is then stored in a local database for subsequent processing.

[0048] Step S102: Extract eye movement information from the human eye image.

[0049] Furthermore, eye movement features can be extracted from human eye images to obtain eye movement information, which includes, but is not limited to, the eye gaze coordinates of the target person.

[0050] Specifically, the human eye image can be preprocessed by the eye movement feature extraction module to convert the color human eye image into a grayscale image, and then Gaussian filtering can be used to denoise the converted grayscale image to eliminate the influence of ambient light and camera noise.

[0051] Then, the pupil-corneal reflection method and edge detection algorithm are used to process the denoised image to detect the pupil contour and corneal reflection spot, and the pupil center coordinates and spot center coordinates are obtained by ellipse fitting.

[0052] Finally, the offset between the pupil center coordinates and the light spot center coordinates is calculated. Combined with the pixel mapping relationship of the laptop screen, the pixel coordinates of eye movement fixation on the weld defect image are used as eye movement fixation coordinates, that is, eye movement information in the human eye image.

[0053] Step S103: Correct the eye movement information using a mapping model to obtain corrected eye movement information; wherein, the mapping model is pre-constructed in different areas of the terminal screen based on the correlation between corresponding eye movement information samples and mouse operation information samples.

[0054] In this embodiment of the disclosure, a specified number of eye-tracking information samples and mouse operation information samples can be collected by randomly clicking the mouse in the entire area of ​​the terminal display screen. A mapping model is constructed based on the correlation between the eye-tracking information samples and the mouse operation information samples. The eye-tracking information is then mapped and automatically corrected using the mapping model to obtain corrected eye-tracking information that is independent of the camera position.

[0055] Step S104: If the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements, determine the region of interest based on the mouse operation information, and perform statistical analysis on the region of interest.

[0056] In this embodiment of the disclosure, the similarity between the corrected eye movement information and the mouse operation information meets the requirement that the eye movement gaze coordinates and the mouse operation coordinates of the mouse drawing ROI are highly matched, and the area corresponding to the mouse operation information that meets the similarity requirement is determined as the target ROI of the weld defect, so as to perform multi-dimensional statistical analysis on the target ROI.

[0057] In the above embodiments, the constructed mapping model can eliminate systematic errors caused by different camera orientations based on the correlation between eye movement information samples and mouse operation information samples, thereby effectively improving the flexibility of the eye tracking process. By inputting the real-time extracted eye movement information into the mapping model, the model outputs corrected eye movement information, which ensures accurate alignment between eye movement information and mouse operation information. This enables the analysis of the region of interest to be triggered when the eye movement information and mouse operation information are highly aligned, thereby improving the accuracy of the eye tracking process.

[0058] In an optional implementation, if the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements, the above steps, specifically including the following steps, involve determining the region of interest based on the mouse operation information: Calculate the pixel distance between the corrected eye-tracking information and the mouse operation information; Within a preset time window, if the pixel distance is less than a distance threshold and the duration of eye-tracking gaze is greater than a duration threshold, the mouse drawing trajectory corresponding to the mouse operation information is captured. The region of interest is generated based on the mouse trajectory.

[0059] Here, the similarity requirement can be understood as follows: within a preset time window, the pixel distance between the eye-tracking gaze coordinates (i.e., eye-tracking information) and the mouse operation coordinates (i.e., mouse operation information) is less than a preset threshold, and the duration of the eye-tracking gaze is greater than the duration threshold.

[0060] The preset time window can be set to 50-200ms, the preset threshold is 5-10 screen pixels, and the duration threshold is 100-300ms. The above values ​​can be flexibly adjusted according to different actual application scenarios.

[0061] In this embodiment, the pixel distance between the corrected eye movement information and the mouse operation information is first calculated. Within a preset time window, if the pixel distance is less than a distance threshold and the duration of eye movement fixation is greater than a duration threshold, a high match between the eye movement information and the mouse operation information is determined so as to automatically capture the ROI outline drawn by the mouse, that is, the mouse drawing trajectory corresponding to the mouse operation information, and thereby generate a region of interest based on the mouse drawing trajectory.

[0062] In the above implementation, the stability of eye tracking is determined by pixel distance and the duration of eye gaze, ensuring that the position of the eye and the position of the mouse point are at the same height. Based on the consistency of position height, the effectiveness of the generated region of interest is ensured, so as to support the accuracy of the entire eye tracking.

[0063] In an optional implementation, the above steps generate the region of interest based on the mouse drawing trajectory, specifically including the following steps: When the mouse drawing trajectory is a multi-point drawing, polygon fitting is performed on the mouse drawing trajectory, and the region of interest is generated after fitting. When the mouse outline is a bounding box, a rectangular region is extracted from the mouse outline, and the region of interest is generated after extraction.

[0064] In this embodiment of the disclosure, the shape of the defect area under different drawing methods can be restored according to the drawing habits of the target person. For multi-point drawing, the outline of the defect area can be accurately restored by polygon fitting, removing hand shaking noise and generating a smooth closed area, i.e. region of interest. For box selection drawing, a rectangular area can be extracted directly from the mouse drawing trajectory by rectangle extraction to generate the corresponding region of interest, thereby ensuring the geometric accuracy of the region of interest.

[0065] In an optional implementation, the above steps involve statistical analysis of the region of interest, specifically including the following steps: Extract visual feature parameters of the region of interest; wherein, the visual feature parameters include geometric features, texture features, and grayscale features; The visual feature parameters are input into the analysis model to output analysis results; wherein, the analysis results include the defect type, defect level and defect cause of the weld defect.

[0066] Here, geometric features include the area, perimeter, aspect ratio, roundness, and concavity / convexity of the defect contour of the ROI; texture features include the gray-level co-occurrence matrix features, local binary pattern features, and texture direction entropy of the ROI; gray-level features include the average gray level, gray-level variance, gray-level extreme values, and gray-level histogram features of the ROI.

[0067] Specifically, the ROI has an area of ​​256 pixels², a perimeter of 68 pixels, an aspect ratio of 3.2, a circularity of 0.35, and a concavity / convexity of 0.82; the gray-level co-occurrence matrix has a contrast of 125.6, a correlation of 0.78, and an entropy of 4.2; the mean of the local binary pattern features is 0.32, and the variance is 0.15; the average gray level is 128, the gray level variance is 36.5, the maximum gray level is 201, and the minimum gray level is 56. A 16-dimensional feature vector from the gray-level histogram is also extracted.

[0068] In this regard, all extracted visual feature parameters can be standardized and stored in the corresponding weld defect feature database.

[0069] Furthermore, visual feature parameters are input into the analysis model so that the analysis model can output the weld defect type determination result, defect level assessment result, and defect cause analysis result.

[0070] For example, the defect type is a transverse crack, the defect level is II, and the defect is caused by cold cracking due to excessive welding stress.

[0071] The construction process of the analysis model is as follows: First, the visual feature parameters of the historical region of interest are used as input features, and the corresponding defect type, defect level and defect cause are used as labels to construct a training dataset. Then, the initial analysis model is trained using supervised learning to obtain the trained analysis model.

[0072] After that, an online learning mechanism can be used to supplement the training dataset in real time with the newly acquired visual feature parameters and the corresponding new analysis results, so as to achieve iterative optimization of the analysis model.

[0073] The analysis model can be built on the YOLOv8 architecture and trained using mini-batch gradient descent with a batch size of 32 and 100 training rounds. It also features an online learning mechanism that allows newly collected samples to be input into the model in real time for fine-tuning, enabling continuous iterative optimization of the analysis model.

[0074] For the mapping model mentioned above, the coordinate matching error can be used as a reward function to iteratively optimize the model parameters, reducing the updated coordinate matching error from 5 pixels to 3 pixels. For the analysis model, the newly collected 1000 samples can be added to the training dataset to retrain the model, improving the detection accuracy of the updated model for weld crack defects in pressure vessels from 92.5% to 96.8%, thus achieving adaptive optimization of the model.

[0075] In the above embodiments, by integrating geometric features, texture features and grayscale features into a multidimensional analysis, geometric features quantify morphology, texture features quantify texture, and grayscale features quantify brightness, thereby ensuring the comprehensiveness of statistical analysis of the region of interest.

[0076] In an optional implementation, the above steps construct the mapping model in the following manner, specifically including the following steps: A sample set is collected from different areas of the terminal screen; wherein, the sample set includes the eye-tracking information sample and the mouse operation information sample; An initial model is constructed based on the correlation between the eye-tracking information samples and the mouse operation information samples; The model parameters of the initial model are iteratively optimized based on the sample set, and the optimized model is the mapping model.

[0077] In this embodiment of the disclosure, multiple sets of eye-tracking information samples and mouse operation information samples can be selected covering the entire area of ​​the terminal display screen. For example, the target person can be asked to randomly click the mouse within the entire area of ​​the screen, and 100 sets of mouse operation coordinates and corresponding initial eye-tracking gaze coordinates can be collected, covering the upper left, upper right, lower left, lower right and center areas of the screen to form a sample set.

[0078] This disclosure can process the correlation between eye-tracking information samples and mouse operation information samples based on a perspective transformation algorithm to establish an initial model. The parameters of the perspective transformation include rotation angle, translation amount, and scaling factor.

[0079] Based on this, the least squares method can be used to fit the sample set and iteratively optimize the parameters of the initial model until the average pixel error between the corrected eye-tracking gaze coordinates and the mouse operation coordinates of the model output is less than 5 pixels. The optimized model is then obtained.

[0080] Specifically, the error between the corrected eye-tracking information and mouse operation information can be set to be less than a preset percentage of the terminal screen pixels, which can be adjusted to 2%.

[0081] In the above embodiments, distortion patterns in different regions can be captured by full-screen sampling. By constructing an initial model and iteratively optimizing the initial model, the eye-tracking process of this disclosure can be adapted to any camera position, ultimately obtaining a high-precision mapping model that is independent of the camera position.

[0082] In an optional implementation, after generating the region of interest based on the mouse drawing trajectory, the method further includes the following steps: Based on the corrected eye movement information, determine the distribution function of fixation density within the region of interest; The region with the highest gaze density in the distribution function is identified as the core region, and statistical analysis is performed on the region of interest based on the visual feature parameters of the region of interest and the visual feature parameters of the core region.

[0083] In this embodiment, discrete corrected eye movement information is transformed into a continuous density distribution function to eliminate single-point noise. The region with the highest gaze density is determined as the core region based on the extreme value of the distribution function, thereby achieving accurate positioning of the core region. Finally, the overall visual feature parameters of the ROI and the local visual feature parameters of the core region are comprehensively fused and analyzed to further ensure the comprehensiveness of the region of interest generation process and assist in supporting the accuracy of the eye tracking process.

[0084] In addition, this disclosure also supports determining the core area based on the target person's gaze duration, gaze frequency, and re-gaze behavior.

[0085] Specifically, the region of interest can be first gridded, and the gaze duration, number of gazes, and number of re-views of the target person in each grid within a specified time can be counted. Region filtering conditions can be set by setting thresholds or weighted summation. Based on the region filtering conditions, grids that meet the conditions are determined, and adjacent grids that meet the region filtering conditions are merged to obtain the core region, thus supporting the flexibility of the core region selection process.

[0086] The following is combined Figure 2 The eye-tracking process described above is as follows: S201: Obtain the target person's eye image and mouse operation information on the terminal.

[0087] S202: Extract eye movement information from human eye images.

[0088] S203: Correct eye movement information by using a mapping model to obtain corrected eye movement information.

[0089] Here, the mapping model is pre-constructed based on the correlation between corresponding eye-tracking information samples and mouse operation information samples in different areas of the terminal screen.

[0090] S204: Calculate the pixel distance between the corrected eye-tracking information and the mouse operation information.

[0091] S205: Within a preset time window, if the pixel distance is less than a distance threshold and the duration of eye-tracking gaze is greater than a duration threshold, capture the mouse drawing trajectory corresponding to the mouse operation information.

[0092] S206: Generate region of interest based on mouse drawing trajectory.

[0093] S207: Perform statistical analysis on the region of interest.

[0094] As can be seen from the above description, the technical solution disclosed herein has the following advantages: (1) Achieve automatic eye tracking correction independent of camera position: No professional fixed eye tracking acquisition equipment is required. It is compatible with any position camera on the laptop. Through perspective transformation and parameter iteration optimization, a mapping model is constructed to eliminate the coordinate offset caused by camera position and shooting angle, so as to achieve accurate matching between eye gaze coordinates and mouse operation coordinates, and make the matching error not exceed 2% of the screen pixels.

[0095] (2) Improve the efficiency and standardization of weld defect ROI acquisition: By using the eye-mouse coordinate height matching judgment condition, the target ROI of weld defect is automatically acquired, replacing the traditional manual annotation, which greatly improves the ROI acquisition efficiency; at the same time, the ROI extraction standard is unified, avoiding the subjectivity and differences of manual annotation, and providing high-quality samples for subsequent feature statistics and model training.

[0096] (3) Construct a closed-loop data process from manual judgment to intelligent detection: deeply integrate ROI feature extraction with large model analysis and AI model training to realize the automatic input of feature parameters to the large model and the real-time supplementation of AI model training samples. At the same time, utilize online learning and self-learning mechanisms to realize the continuous iterative optimization of the intelligent model and complete the closed-loop data process from manual visual judgment to intelligent machine detection.

[0097] (4) Achieve multi-dimensional statistical analysis of weld defect target area: Combine eye-tracking statistical information and ROI visual feature parameters to conduct statistical analysis of weld defect target area from two dimensions: human attention behavior and actual defect characteristics. Establish a correlation model between human judgment and defect characteristics to provide scientific data analysis support for the formulation of industrial weld defect judgment standards and the optimization of detection methods.

[0098] Reference Figure 3 The diagram shown is a schematic representation of an eye-tracking device according to an embodiment of this disclosure. The device includes: a data acquisition unit 30, an information extraction unit 31, an information correction unit 32, and a region analysis unit 33; wherein: The data acquisition unit is used to acquire images of the target person's eyes and mouse operation information on the terminal; An information extraction unit is used to extract eye movement information from the human eye image; An information correction unit is used to correct the eye movement information through a mapping model to obtain corrected eye movement information; wherein, the mapping model is pre-constructed in different areas of the terminal screen based on the correlation between corresponding eye movement information samples and mouse operation information samples; The region analysis unit is used to determine the region of interest based on the mouse operation information, and to perform statistical analysis on the region of interest, provided that the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements.

[0099] In one possible implementation, the device is also used for: Calculate the pixel distance between the corrected eye-tracking information and the mouse operation information; Within a preset time window, if the pixel distance is less than a distance threshold and the duration of eye-tracking gaze is greater than a duration threshold, the mouse drawing trajectory corresponding to the mouse operation information is captured. The region of interest is generated based on the mouse trajectory.

[0100] In one possible implementation, the device is also used for: When the mouse drawing trajectory is a multi-point drawing, polygon fitting is performed on the mouse drawing trajectory, and the region of interest is generated after fitting. When the mouse outline is a bounding box, a rectangular region is extracted from the mouse outline, and the region of interest is generated after extraction.

[0101] In one possible implementation, the device is also used for: Extract visual feature parameters of the region of interest; wherein, the visual feature parameters include geometric features, texture features, and grayscale features; The visual feature parameters are input into the analysis model to output analysis results; wherein, the analysis results include the defect type, defect level and defect cause of the weld defect.

[0102] In one possible implementation, the device is also used for: A sample set is collected from different areas of the terminal screen; wherein, the sample set includes the eye-tracking information sample and the mouse operation information sample; An initial model is constructed based on the correlation between the eye-tracking information samples and the mouse operation information samples; The model parameters of the initial model are iteratively optimized based on the sample set, and the optimized model is the mapping model.

[0103] In one possible implementation, the device is also used for: Based on the corrected eye movement information, determine the distribution function of fixation density within the region of interest; The region with the highest gaze density in the distribution function is identified as the core region, and statistical analysis is performed on the region of interest based on the visual feature parameters of the region of interest and the visual feature parameters of the core region.

[0104] In one possible implementation, the device is also used for: The error between the corrected eye-tracking information and the mouse operation information is less than a preset percentage of the terminal screen pixels.

[0105] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0106] Corresponding to Figure 1In addition to the eye-tracking method described in this disclosure, this embodiment also provides an electronic device 40, such as... Figure 4 The diagram shown is a schematic representation of an electronic device 40 provided in an embodiment of this disclosure, including: The system includes a processor 41, a memory 42, and a bus 43. The memory 42 stores execution instructions and includes main memory 421 and external memory 422. The main memory 421, also called internal memory, temporarily stores the computational data in the processor 41, as well as data exchanged with external memory such as a hard disk. The processor 41 exchanges data with the external memory 422 through the main memory 421. When the electronic device 40 is running, the processor 41 communicates with the memory 42 through the bus 43, causing the processor 41 to execute the following instructions: Acquire images of the target person's eyes and mouse operation information on the terminal; Extract eye movement information from the human eye image; The eye movement information is corrected by a mapping model to obtain corrected eye movement information; wherein, the mapping model is pre-constructed in different areas of the terminal screen based on the correlation between corresponding eye movement information samples and mouse operation information samples; If the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements, a region of interest is determined based on the mouse operation information, and statistical analysis is performed on the region of interest.

[0107] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the eye-tracking method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0108] This disclosure also provides a computer program product 50, such as... Figure 5 The diagram shown is a schematic diagram of the structure of a computer program product 50 provided in an embodiment of this disclosure. The computer program product 50 carries a computer program 51. The program included in the computer program 51 can be used to execute the steps of the eye-tracking method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0109] The above description, with reference to the accompanying drawings, outlines an eye-tracking method, apparatus, device, medium, and product according to embodiments of the present disclosure. In these embodiments, the eye-tracking system first acquires an image of the target person's eyes and mouse operation information on a terminal. Then, it extracts eye movement information from the eye image and corrects the eye movement information using a mapping model to obtain corrected eye movement information. This mapping model is pre-constructed in different areas of the terminal screen based on the correlation between corresponding eye movement information samples and mouse operation information samples. Finally, if the similarity between the corrected eye movement information and the mouse operation information meets the requirements, a region of interest is determined based on the mouse operation information, and statistical analysis is performed on the region of interest. Through the above processing method, the constructed mapping model can eliminate systematic errors caused by different camera orientations based on the correlation between eye movement information samples and mouse operation information samples, thereby effectively improving the flexibility of the eye tracking process. By inputting the real-time extracted eye movement information into the mapping model, the model outputs corrected eye movement information, which ensures accurate alignment between eye movement information and mouse operation information. This enables the analysis of the region of interest to be triggered when the eye movement information and mouse operation information are highly aligned, thereby improving the accuracy of the eye tracking process.

[0110] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0111] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0112] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0113] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0114] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0115] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0116] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An eye-tracking method, characterized in that, include: Acquire images of the target person's eyes and mouse operation information on the terminal; Extract eye movement information from the human eye image; The eye movement information is corrected by a mapping model to obtain corrected eye movement information; wherein, the mapping model is pre-constructed in different areas of the terminal screen based on the correlation between corresponding eye movement information samples and mouse operation information samples; If the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements, the region of interest is determined based on the mouse operation information, and statistical analysis is performed on the region of interest. Wherein, determining the region of interest based on the mouse operation information, when the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements, includes: Calculate the pixel distance between the corrected eye-tracking information and the mouse operation information; Within a preset time window, if the pixel distance is less than a distance threshold and the duration of eye-tracking gaze is greater than a duration threshold, the mouse drawing trajectory corresponding to the mouse operation information is captured. The region of interest is generated based on the mouse drawing trajectory; The statistical analysis of the region of interest includes: Extract visual feature parameters of the region of interest; wherein, the visual feature parameters include geometric features, texture features, and grayscale features; The visual feature parameters are input into the analysis model to output analysis results; wherein, the analysis results include the defect type, defect level and defect cause of the weld defect.

2. The method according to claim 1, characterized in that, The step of generating the region of interest based on the mouse drawing trajectory includes: When the mouse drawing trajectory is a multi-point drawing, polygon fitting is performed on the mouse drawing trajectory, and the region of interest is generated after fitting. When the mouse outline is a bounding box, a rectangular region is extracted from the mouse outline, and the region of interest is generated after extraction.

3. The method according to claim 1, characterized in that, The mapping model is constructed in the following ways: A sample set is collected from different areas of the terminal screen; wherein, the sample set includes the eye-tracking information sample and the mouse operation information sample; An initial model is constructed based on the correlation between the eye-tracking information samples and the mouse operation information samples; The model parameters of the initial model are iteratively optimized based on the sample set, and the optimized model is the mapping model.

4. The method according to claim 1, characterized in that, After generating the region of interest based on the mouse drawing trajectory, the method further includes: Based on the corrected eye movement information, determine the distribution function of fixation density within the region of interest; The region with the highest gaze density in the distribution function is identified as the core region, and statistical analysis is performed on the region of interest based on the visual feature parameters of the region of interest and the visual feature parameters of the core region.

5. The method according to claim 1, characterized in that, The error between the corrected eye-tracking information and the mouse operation information is less than a preset percentage of the terminal screen pixels.

6. An eye-tracking device, characterized in that, include: The data acquisition unit is used to acquire images of the target person's eyes and mouse operation information on the terminal; An information extraction unit is used to extract eye movement information from the human eye image; An information correction unit is used to correct the eye movement information through a mapping model to obtain corrected eye movement information; wherein, the mapping model is pre-constructed in different areas of the terminal screen based on the correlation between corresponding eye movement information samples and mouse operation information samples; The region analysis unit is used to determine the region of interest based on the mouse operation information when the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements, and to perform statistical analysis on the region of interest. Wherein, determining the region of interest based on the mouse operation information, when the similarity between the corrected eye-tracking information and the mouse operation information meets the requirements, includes: Calculate the pixel distance between the corrected eye-tracking information and the mouse operation information; Within a preset time window, if the pixel distance is less than a distance threshold and the duration of eye-tracking gaze is greater than a duration threshold, the mouse drawing trajectory corresponding to the mouse operation information is captured. The region of interest is generated based on the mouse drawing trajectory; The statistical analysis of the region of interest includes: Extract visual feature parameters of the region of interest; wherein, the visual feature parameters include geometric features, texture features, and grayscale features; The visual feature parameters are input into the analysis model to output analysis results; wherein, the analysis results include the defect type, defect level and defect cause of the weld defect.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the eye-tracking method as described in any one of claims 1 to 5.

8. A computer program product, characterized in that, The computer program product is stored in a storage medium and is executed by at least one processor to implement the eye-tracking method as described in any one of claims 1 to 5.

Images