Continuous image annotation quality assessment method and related equipment based on eye movement data

Abstract

The present invention relates to the field of image annotation technology, and specifically discloses a continuous image annotation quality assessment method based on eye movement data and related equipment, wherein the method comprises the steps of: obtaining an eye movement data set corresponding to a target image set, the target image set being an image set for which an annotation task has been completed, the eye movement data set comprising multiple groups of eye movement data information corresponding to the annotators generated during the annotation process of different annotated images in the target image set; obtaining gaze entropy information of the corresponding annotated images based on the transfer of each group of eye movement data information in a pre-divided region of interest; evaluating the annotation quality of each annotated image based on the size of the gaze entropy information, so as to screen and obtain a first abnormal image; the method of the present application uses gaze entropy information as a measure of uncertainty to assess the image annotation quality based on the complexity of the annotator's attention distribution, thereby improving the accuracy and reliability of the annotation quality assessment.

Description

Technical Field

[0001] The present application relates to the field of image annotation technology, and in particular to a continuous image annotation quality assessment method based on eye movement data and related equipment. Background Art

[0002] The traditional continuous image annotation process relies on a one-time batch annotation process followed by a subsequent review. In this process, annotators must complete numerous tasks in a concentrated manner, followed by a comprehensive review by reviewers. If the annotations are unsatisfactory, they must be returned for revision, resulting in a cumbersome and inefficient process.

[0003] While existing technologies attempt to evaluate the annotation process using eye movement data, their application has significant shortcomings. Existing technologies primarily rely on simple factors such as the range and velocity of eye movement data for simple evaluations. These technologies focus solely on the eye movement data of a single image, ignoring the correlation between eye movement data and more comprehensive feature analysis across images. This results in a single, one-sided assessment, leading to insufficient accuracy.

[0004] There is currently no effective technical solution to the above problems. Summary of the Invention

[0005] The purpose of this application is to provide a continuous image annotation quality assessment method and related equipment based on eye movement data to improve the accuracy and reliability of annotation quality assessment.

[0006] In a first aspect, the present application provides a method for evaluating the quality of continuous image annotation based on eye movement data, the method comprising the following steps:

[0007] S1. Acquire an eye movement dataset corresponding to a target image set, wherein the target image set is an image set for which the labeling task has been completed, and the eye movement dataset includes multiple sets of eye movement data information corresponding to the labelers' labeling of different labeled images in the target image set;

[0008] S2. Obtaining gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest;

[0009] S3. Evaluate the annotation quality of each annotated image according to the size of the gaze entropy information to screen and obtain a first abnormal image.

[0010] The continuous image annotation quality assessment method based on eye movement data of the present application evaluates the image annotation quality by introducing gaze entropy information determined based on the transfer of eye movement data information in a pre-divided region of interest, so as to evaluate the image annotation quality based on the complexity of the annotator's attention distribution, thereby improving the accuracy and reliability of the annotation quality assessment.

[0011] In the method for evaluating the quality of continuous image annotation based on eye movement data, the step of obtaining gaze entropy information of each annotated image includes:

[0012] S21, obtaining a transfer probability between different regions of interest based on the transfer of the eye movement data information in the pre-divided regions of interest;

[0013] S22, iteratively calculating and obtaining the stationary probability of each region of interest according to the transition probability;

[0014] S23, calculating and obtaining a transfer entropy according to the stationary probability and the transfer probability;

[0015] S24, obtaining a stationary entropy according to the stationary probability calculation;

[0016] S25. Obtain the gaze entropy information by combining the transfer entropy and the stationary entropy.

[0017] In this example, the transfer probability obtained in step S21 quantifies the annotator's attention transfer pattern, transfer frequency, and transfer habits between different areas. The stable probability obtained in step S22 reflects the annotator's long-term attention level to each area of ​​interest, which helps to understand the annotator's overall attention distribution. The transfer entropy obtained in step S23 is used to measure the uncertainty and complexity of eye movement transfer. The stable entropy obtained in step S24 reflects the uniformity of the annotator's overall attention distribution. The gaze entropy information obtained in step S25 is a comprehensive indicator determined by the combined transfer entropy and stable entropy, which can comprehensively reflect the eye movement characteristics during the annotation process and provide an important basis for evaluating the annotation quality.

[0018] The method for evaluating the quality of continuous image annotation based on eye movement data, wherein step S21 includes the following steps:

[0019] The transfer probability between different regions of interest is obtained according to the transfer conditions of the eye movement data information in different transfer intervals in the pre-divided regions of interest.

[0020] The method of this application not only focuses on the eye movement transfer between adjacent time points, but also considers the transfer pattern on a longer time scale, which can more comprehensively capture the eye movement characteristics during the labeling process, thereby providing a more accurate transfer probability estimation.

[0021] The continuous image annotation quality assessment method based on eye movement data, wherein the annotated images in the target image set are continuous frame images, and all the annotated images in the target image set are annotated based on the same annotation rules;

[0022] The method further comprises the steps of:

[0023] S4, calculating the gaze synchronization of each adjacent annotated image based on the position matching degree of the eye movement data information between the consecutively annotated adjacent annotated images with respect to the time sequence;

[0024] S5. Evaluate the annotation quality of each annotated image according to the degree of gaze synchronization to screen and obtain a second abnormal image.

[0025] In the method for evaluating the quality of continuous image annotation based on eye movement data, step S4 includes the following steps:

[0026] S41, acquiring annotated eye movement data according to the eye movement data information, wherein the annotated eye movement data includes a plurality of gaze coordinates at different time points;

[0027] S42, extracting multiple gaze coordinates of the annotated eye movement data corresponding to each annotated image based on a preset temporal matching relationship;

[0028] S43: Calculate the gaze synchronization between adjacent annotated images based on the temporal matching relationship and the gaze coordinates.

[0029] In the method for evaluating the quality of continuous image annotation based on eye movement data, wherein the preset temporal matching relationship includes equally divided time nodes, step S42 includes the following steps:

[0030] S421, dividing the annotation time of the annotated eye movement data of each annotated image into equal parts based on the number of equally divided time nodes to form discrete time points;

[0031] S422: Extracting a plurality of gaze coordinates from the corresponding annotated eye movement data based on the discrete time points.

[0032] In the method for evaluating the quality of continuous image annotation based on eye movement data, step S5 includes the following steps:

[0033] S51, calculating the average gaze synchronization degree of each annotated image according to the gaze synchronization degree, and obtaining the average gaze synchronization degree according to all the average gaze synchronization degrees;

[0034] S52, calculating and obtaining a synchronization standard deviation based on the gaze synchronization mean and the average gaze synchronization;

[0035] S53: Evaluate the annotation quality of each annotated image according to the difference between the gaze synchronization mean and the synchronization standard deviation, so as to screen and obtain the second abnormal image.

[0036] In a second aspect, the present application further provides a device for evaluating the quality of continuous image annotation based on eye movement data, the device comprising:

[0037] an acquisition module, configured to acquire an eye movement dataset corresponding to a target image set, wherein the target image set is an image set for which the annotation task has been completed, and the eye movement dataset includes multiple sets of eye movement data information corresponding to the annotators during the annotation process of different annotated images in the target image set;

[0038] A first calculation module is used to obtain gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest;

[0039] The first evaluation module is configured to evaluate the annotation quality of each annotated image according to the size of the gaze entropy information, so as to screen and obtain a first abnormal image.

[0040] The continuous image annotation quality assessment device based on eye movement data of the present application assesses the image annotation quality by introducing gaze entropy information determined based on the transfer of eye movement data information in a pre-divided region of interest, so as to assess the image annotation quality based on the complexity of the annotator's attention distribution, thereby improving the accuracy and reliability of the annotation quality assessment.

[0041] In a third aspect, the present application further provides an electronic device comprising a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps of the method provided in the first aspect are executed.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, runs the steps of the method provided in the first aspect above.

[0043] From the above, it can be seen that the present application provides a method and related equipment for evaluating the quality of continuous image annotation based on eye movement data, wherein the method of the present application's continuous image annotation quality evaluation device based on eye movement data evaluates the image annotation quality by introducing gaze entropy information determined based on the transfer of eye movement data information in a pre-divided region of interest. The gaze entropy information can reflect the long-term attention distribution of the annotator and can be used as a basis for a more comprehensive and in-depth evaluation, so that the device of the present application can use gaze entropy information as a measure of uncertainty to evaluate the image annotation quality based on the complexity of the annotator's attention distribution, thereby improving the accuracy and reliability of the annotation quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a method for evaluating the quality of continuous image annotation based on eye movement data provided in some embodiments of the present application.

[0045] Figure 2Flowchart of a method for evaluating the quality of continuous image annotation based on eye movement data provided in some other embodiments of the present application.

[0046] Figure 3 A schematic diagram of the structure of a continuous image annotation quality assessment device based on eye movement data provided in some embodiments of the present application.

[0047] Figure 4 This is a schematic structural diagram of a continuous image annotation quality assessment device based on eye movement data provided in some other embodiments of the present application.

[0048] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0049] Reference numerals: 201, acquisition module; 202, first calculation module; 203, first evaluation module; 204, second calculation module; 205, second evaluation module; 301, processor; 302, memory; 303, communication bus. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0051] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0052] Traditional methods for assessing the quality of continuous image annotation rely primarily on a one-time batch annotation process followed by subsequent review. This approach suffers from inefficiencies and inaccurate assessments. While existing technologies attempt to evaluate the annotation process using eye movement data, significant shortcomings remain. Existing technologies primarily rely on simple factors derived from eye movement data and focus solely on the eye movement data of a single image, neglecting inter-image correlations and more comprehensive feature analysis. This results in simplistic and one-sided assessments that fail to accurately reflect the true state of annotation quality.

[0053] Specifically, in a large-scale image annotation project, annotators need to process thousands of images continuously. The annotation quality of each image directly affects the subsequent data analysis and the training effect of the machine learning model. Since traditional methods cannot evaluate the annotation quality in real time, a large number of low-quality annotations may accumulate. For example, in an autonomous driving scene recognition project, annotators need to continuously annotate key elements such as roads, vehicles, and pedestrians. If the annotation quality assessment is inaccurate, it may cause a large amount of noisy data to appear during model training, affecting the recognition accuracy. Among them, although eye movement data is introduced into the evaluation process, it only considers simple movement range and rate, and cannot capture the complexity of the annotator's attention distribution and the attention change pattern during the continuous annotation process.

[0054] Therefore, if the problem of insufficient accuracy in quality assessment of continuous image annotation cannot be effectively solved, serious technical consequences will occur. First, low-quality annotation data will directly affect the training effect of the machine learning model, resulting in a decline in model performance and failure to achieve the expected recognition or classification accuracy. Secondly, inaccurate assessments will increase the workload of subsequent manual review, significantly increasing project costs and time consumption. In addition, the inability to detect and correct annotation problems in a timely manner may cause annotation errors to accumulate and spread in large amounts of data, ultimately affecting the availability of the entire dataset. In some key application areas, such as medical image analysis or security monitoring systems, this problem may lead to serious decision-making errors and bring immeasurable risks. Therefore, improving the accuracy of continuous image annotation quality assessment has become a technical problem that needs to be solved urgently. It is necessary to develop a more comprehensive and accurate evaluation method to ensure the quality and reliability of the annotation data.

[0055] First, please refer to Figure 1 and Figure 2 Some embodiments of the present application provide a method for evaluating the quality of continuous image annotation based on eye movement data, the method comprising the following steps:

[0056] S1. Obtain an eye movement dataset corresponding to a target image set, where the target image set is an image set for which the annotation task has been completed. The eye movement dataset includes multiple sets of eye movement data information corresponding to the annotators' annotation of different annotated images in the target image set;

[0057] S2. Obtain gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest;

[0058] S3. Evaluate the annotation quality of each annotated image according to the size of the gaze entropy information to screen and obtain a first abnormal image.

[0059] Specifically, the target image set refers to the image set that has completed the labeling task.

[0060] An eye movement dataset refers to a data set that records the eye movements of annotators during the annotation process. Specifically, it can be implemented using a data structure containing information such as timestamps and gaze point coordinates. It can be collected and generated by eye tracking devices such as eye trackers. Each set of eye movement data information in the eye movement dataset corresponds to the eye movement data generated during the annotation process of an annotated image in the target image set.

[0061] The area of ​​interest (AOI) refers to a specific area pre-divided in the area where the annotated image is located, and can be defined by geometric shapes such as rectangles, circles or irregular polygons.

[0062] More specifically, gaze entropy information refers to an information entropy indicator used to measure the distribution of annotator's attention, which can be implemented using an entropy calculation formula based on information theory.

[0063] More specifically, the core innovation of the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application lies in the introduction of a gaze entropy information analysis method based on eye movement data to evaluate the quality of continuous image annotation. This processing method not only takes into account the eye movement data of a single image, but also captures deeper information in the annotation process through the calculation of gaze entropy information. Gaze entropy information can effectively measure the distribution balance of the annotator's attention in different areas of interest and the frequency of switching attention areas, thereby providing a more comprehensive and accurate annotation quality assessment indicator.

[0064] The working principle of the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application can be described in detail as follows:

[0065] First, step S1 obtains the eye movement dataset corresponding to the target image set. This step uses a specialized eye tracking device to record the annotator's eye movement data during the annotation process. Eye movement data typically includes information such as timestamp, gaze point coordinates, and gaze duration. This data is divided into multiple groups, each of which corresponds to the annotation process of a single annotated image.

[0066] Next, step S2 obtains gaze entropy information based on the transfer of eye movement data information in pre-divided regions of interest. In specific operations, the area where each annotated image is located is divided into multiple regions of interest based on a predetermined division rule. These regions of interest can be defined according to the importance or complexity of the image content. During the image annotation process, the annotator's gaze point will shift and switch between different regions of interest. By analyzing the transfer of gaze points between these regions in the eye movement data, the transfer probability between different regions of interest can be calculated to form a probability matrix. The gaze entropy information can be obtained by combining the transfer probability of this matrix based on the information entropy calculation formula.

[0067] Finally, step S3 can evaluate the annotation quality of each annotated image according to the size of the gaze entropy information by setting a reasonable threshold, and identify and screen the annotated images whose gaze entropy information does not meet the corresponding threshold (too large or too small) as the first abnormal images. These first abnormal images represent that there may be problems with their annotation quality and need further review or re-annotation.

[0068] More specifically, in an embodiment of the present application, gaze entropy information can simultaneously reflect the distribution balance and switching frequency of the annotator's attention. A higher gaze entropy value usually indicates that the annotator has paid sufficient attention to different parts of the image, while a lower value may mean that the attention is too concentrated or dispersed. Compared with traditional simple eye movement indicators, this evaluation method can more comprehensively reflect the complexity of the annotation process, thereby improving the accuracy of the evaluation.

[0069] The continuous image annotation quality assessment method based on eye movement data in an embodiment of the present application assesses the image annotation quality by introducing gaze entropy information determined based on the transfer of eye movement data information in a pre-divided region of interest. The gaze entropy information can reflect the long-term attention distribution of the annotator and can be used as a basis for a more comprehensive and in-depth assessment. This allows the method of the present application to use gaze entropy information as a measure of uncertainty to assess the image annotation quality based on the complexity of the annotator's attention distribution, thereby improving the accuracy and reliability of the annotation quality assessment.

[0070] In some preferred embodiments, the step of acquiring gaze entropy information of each annotated image includes:

[0071] S21, obtaining the transition probability between different regions of interest based on the transition of the eye movement data information in the pre-divided regions of interest;

[0072] S22, iteratively calculating the stationary probability of each region of interest according to the transition probability;

[0073] S23, calculating and obtaining the transfer entropy according to the stationary probability and the transfer probability;

[0074] S24. Obtaining stationary entropy based on stationary probability calculation;

[0075] S25. Obtain gaze entropy information by integrating transfer entropy and stationary entropy.

[0076] Specifically, steps S21 to S25 are a process of calculating and obtaining corresponding gaze entropy information for one annotated image, so step S2 is actually a process of executing steps S21 to S25 for each annotated image in the target image set to obtain corresponding multiple gaze entropy information.

[0077] More specifically, the transfer probability obtained in step S21 quantifies the annotator's attention transfer pattern, transfer frequency, and transfer habits between different areas. The stable probability obtained in step S22 reflects the annotator's long-term attention level to each area of ​​interest, which helps to understand the annotator's overall attention distribution. The transfer entropy obtained in step S23 is used to measure the uncertainty and complexity of eye movement transfer. The stable entropy obtained in step S24 reflects the uniformity of the annotator's overall attention distribution. The gaze entropy information obtained in step S25 is a comprehensive indicator determined by the combined transfer entropy and stable entropy. It can comprehensively reflect the eye movement characteristics during the annotation process and provide an important basis for evaluating the annotation quality.

[0078] More specifically, the regions of interest can be divided according to image content, annotation task requirements or expert experience; in an embodiment of the present application, the regions of interest are preferably regions of interest set based on the key content of the annotation, that is, they are divided by object categories or regions of interest are set by pre-planned areas, thereby constructing multiple regions of interest corresponding to different object categories.

[0079] More specifically, the transition probability can be calculated by counting the number of transitions between different regions of interest in the eye movement data and normalizing it to obtain a transition probability matrix. For example, if the number of transitions from region A to region B is 10 and the total number of transitions is 100, then the transition probability from A to B is 0.1.

[0080] More specifically, in the embodiment of the present application, the transition probability is calculated based on the following formula:

[0081] (1)

[0082] More specifically, where p ij is the transfer probability of the gaze point generated by the eye movement data in the set of eye movement data information from the i-th region of interest to the j-th region of interest, x ij The number of times the gaze point generated by the eye movement data in the set of eye movement data information is transferred from the i-th region of interest to the j-th region of interest, where n is the total number of regions of interest.

[0083] It should be noted that the gaze point is the point in the eye movement data that represents the annotator's gaze state. It can be defined as a relatively stable gaze position point when the eyeball is within a certain threshold range of deviation (usually 2°), within a certain minimum time (usually 100-200ms), and the eyeball movement speed is lower than a certain threshold (usually 15°-100° per second).

[0084] More specifically, the calculation of the stationary probability adopts an iterative method. Initially, the stationary probability of each region can be assumed to be equal, and then it is continuously updated according to the transition probability matrix until convergence. This process can be implemented using the stationary distribution calculation method of the Markov chain; in the embodiment of the present application, the stationary probability is preferably calculated based on the following formula:

[0085] (2)

[0086] in, represents the stationary probability of the jth region of interest at the yth iteration, represents the stationary probability of the i-th region of interest at the y+1th iteration, p ji The transition probability of the gaze point generated by the eye movement data in the set of eye movement data information shifting from the jth region of interest to the ith region of interest, and the initial stationary probability is defined as Step S22 is based on formula (2) and after multiple iterations, a stable probability that tends to be stable can be obtained, thereby obtaining the stable probability corresponding to different regions of interest, and the stable probability of the determined i-th region of interest is recorded as π i , wherein the iterative process can be performed based on a preset number of iterations or by judging whether the stationary probability converges.

[0087] More specifically, the stable stationary probability obtained in step S22 can clearly reflect the stable distribution of the annotator's attention on each region of interest during long-term annotation.

[0088] More specifically, the transfer entropy in step S23 is calculated using the Shannon entropy formula, that is, it is calculated based on the following formula:

[0089] (3)

[0090] Among them, S is the set of all regions of interest, H t is the transfer entropy; higher H t This indicates that the annotator frequently switches the focus area during the annotation process; a lower H t This means that the annotator's eye movement pattern is relatively stable.

[0091] More specifically, the stationary entropy in step S24 is calculated based on the following formula:

[0092] (4)

[0093] Among them, H s is the stationary entropy; higher H s This indicates that the annotator's attention is evenly distributed across different regions of interest; a lower H s This means that the annotator's attention is overly focused on certain areas of interest.

[0094] More specifically, step S25 can obtain gaze entropy information by combining transfer entropy and stationary entropy in a superposition or weighted superposition manner. In the embodiment of the present application, it is preferably performed based on the following formula:

[0095] Q=ω1H t +ω2H s (5)

[0096] Among them, Q is the gaze entropy information, ω1 and ω2 are configuration weights, which can be set according to the research purpose and the importance of each indicator. For example, if more emphasis is placed on annotation stability, ω1 can be appropriately increased. If more emphasis is placed on annotation comprehensiveness, ω2 can be appropriately increased.

[0097] In the embodiment of the present application, ω1 and ω2 satisfy ω1+ω2=1, and more preferably ω1=ω2=0.5.

[0098] The continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application analyzes eye movement data from multiple angles, extracts more comprehensive and in-depth transfer entropy and stationary entropy to constitute gaze entropy information, and the gaze entropy information comprehensively reflects the annotator's attention allocation and transfer pattern on this image. Compared with relying solely on simple eye movement indicators, the solution of the present application not only considers the dynamic characteristics of eye movements, but also considers the long-term attention distribution, thereby providing a more comprehensive and in-depth evaluation basis, which can more comprehensively reflect the cognitive complexity and attention allocation in the annotation process, and help to more accurately identify potential annotation problems, thereby improving the accuracy and reliability of annotation quality assessment.

[0099] In some preferred embodiments, step S21 includes the following steps:

[0100] S211 , obtaining transition probabilities between different regions of interest based on the transition conditions of the eye movement data information in different transition intervals in the pre-divided regions of interest.

[0101] Specifically, in this embodiment, the method of the present application not only focuses on the eye movement transfer between adjacent time points, but also considers the transfer pattern on a longer time scale, which can more comprehensively capture the eye movement characteristics in the labeling process, thereby providing a more accurate transfer probability estimation.

[0102] More specifically, in practical applications, a variety of methods can be used to implement transfer analysis at different transfer intervals. For example, multiple time windows can be set, such as 1 second, 2 seconds, 5 seconds, etc., and the eye movement transfer within these time windows can be counted separately. Another method is to use a sliding window technique to capture transfer patterns at different time scales by continuously moving a fixed-size time window. In addition, a weighted average method can be considered to integrate transfer conditions at different transfer intervals. For example, a higher weight can be given to transfers with shorter time intervals, while a relatively lower weight can be given to transfers with longer time intervals. This can preserve long-term information while placing greater emphasis on short-term eye movement patterns.

[0103] More specifically, in step S211, in order to highlight the short-term eye movement change trend, the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application introduces the frame interval weight based on the following formula:

[0104] (6)

[0105] Among them, m is the maximum transfer interval frame number, w k is the frame interval weight under k transfer interval frames.

[0106] After completing the configuration of the frame interval weight, step S211 is optimized for (1) to obtain the following calculation formula for the translation probability combined with the frame interval weight:

[0107] (7)

[0108] in, The number of times the gaze point generated by the eye movement data in the set of eye movement data information is transferred from the i-th region of interest to the j-th region of interest within the k-th transfer interval frame number.

[0109] In some other embodiments, step S2 can directly use the transfer entropy calculated based on formula (3) or the stationary entropy calculated based on formula (4) as gaze entropy information, that is, step S3 can directly use the transfer entropy or stationary entropy to evaluate the annotation quality of each annotated image at the expense of part of the evaluation accuracy to screen out the first abnormal image, so as to further improve the evaluation efficiency. If it is known in advance that the eye movement pattern of the annotator is relatively stable, the annotation quality of the annotated image annotated by the annotator can be directly evaluated based on the stationary entropy.

[0110] In some preferred embodiments, the annotated images in the target image set are continuous frame images, and all the annotated images in the target image set are annotated based on the same annotation rule;

[0111] The method further comprises the steps of:

[0112] S4, calculating the gaze synchronization of each adjacent annotated image based on the position matching degree of the eye movement data information between the consecutively annotated adjacent annotated images with respect to the time sequence;

[0113] S5. Evaluate the annotation quality of each annotated image according to the degree of gaze synchronization to screen and obtain a second abnormal image.

[0114] Specifically, the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application introduces a new assessment indicator, gaze synchronization, to quantify the degree of synchronization of the annotator's visual attention between consecutive images, so as to solve the problem of insufficient assessment accuracy caused by a single assessment method in the continuous image annotation quality assessment.

[0115] More specifically, before executing step S4 and step S5, the continuous image annotation quality assessment method based on eye movement data of the embodiment of the present application first needs to ensure that all annotated images are based on the same annotation rules (for example, by constraining the annotator through an algorithm to start the annotation operation from the upper left corner of the annotated object), so as to provide a consistent basis for the assessment; since the annotated images are continuous frame images, the corresponding objects to be annotated have a continuous displacement relationship between adjacent annotated images (two images that appear as continuous frames in time sequence), so the annotation boxes in the qualified adjacent annotated images should be adjacent in position and have a certain matching relationship. At the same time, due to the constraints on the annotation rules, the annotators should respectively place the corresponding boxes in the adjacent annotated images. The change trajectory of the generated gaze points should also be adjacent in position and have a certain distance matching relationship; therefore, step S4 introduces gaze synchronization based on time series matching of eye movement data as another quality evaluation indicator. This indicator not only takes into account the annotation quality of a single image, but also focuses on the correlation between adjacent images in the continuous annotation process, and can effectively identify images with abnormal annotation quality; the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application introduces the step of gaze synchronization to perform a secondary evaluation of the annotated image, and combines the previous evaluation processing of gaze entropy information to realize multi-dimensional evaluation, significantly improve the accuracy and comprehensiveness of the evaluation, and can more effectively identify abnormal situations in the annotation process.

[0116] In some preferred embodiments, step S4 includes the following steps:

[0117] S41. Acquire labeled eye movement data according to the eye movement data information, where the labeled eye movement data includes a plurality of gaze coordinates at different time points;

[0118] S42, extracting multiple gaze coordinates of the annotated eye movement data corresponding to each annotated image based on a preset temporal matching relationship;

[0119] S43: Calculate the gaze synchronization between adjacent annotated images based on the temporal matching relationship and the gaze coordinates.

[0120] Specifically, first, step S41 obtains annotated eye movement data containing gaze coordinates of multiple time points to remove invalid eye movement data generated by actual non-annotation behavior, so that the gaze change trajectory of the remaining annotated eye movement data theoretically conforms to the same annotation rules, thereby providing a basis for analysis; then, step S42 ensures the comparability of eye movement data between different annotated images through a preset timing matching relationship, and extracts the gaze coordinates corresponding to each annotated image based on this relationship to ensure data consistency; finally, step S43 uses the timing matching relationship and the extracted gaze coordinates to calculate the gaze synchronization between adjacent annotated images, thereby evaluating the continuity and consistency of the annotation quality.

[0121] More specifically, to ensure comparability of eye movement data across different annotated images, step S42 introduces a preset temporal matching relationship. This temporal matching relationship can be established based on a variety of strategies, such as equal-interval sampling, keyframe matching, or dynamic time warping. Selecting an appropriate temporal matching relationship is crucial for subsequent analysis, as it directly affects the accuracy of gaze coordinate extraction and gaze synchronization calculation.

[0122] More specifically, the process of extracting multiple gaze coordinates in step S42 may adopt an interpolation algorithm or a nearest neighbor selection method to ensure that the extracted gaze coordinates can accurately reflect the eye movement trajectory during the labeling process.

[0123] More specifically, the method for calculating gaze synchronization in step S43 may include calculation methods such as Euclidean distance, cosine similarity or dynamic time warping distance. Selecting an appropriate calculation method can more accurately reflect the continuity and consistency of the annotation process.

[0124] More specifically, the above steps achieve the precise extraction and analysis of eye movement data during the continuous image annotation process. Combined with the preset timing matching relationship, the comparability of eye movement data between different annotated images is ensured, solving the problem of traditional methods that only focus on the eye movement data of a single image while ignoring the relationship between images. The gaze synchronization finally calculated can comprehensively evaluate the continuity and consistency of the annotation quality, providing a more reliable quality assessment method.

[0125] In some preferred embodiments, the preset timing matching relationship includes equally dividing the number of time nodes, and step S42 includes the following steps:

[0126] S421, dividing the annotation time of the annotated eye movement data of each annotated image into equal parts based on the number of equally divided time nodes to form discrete time points;

[0127] S422: Extract multiple gaze coordinates from the corresponding annotated eye movement data based on discrete time points.

[0128] Specifically, steps S421 and S422 solve the problem of how to extract comparable eye movement data during the continuous image annotation process by dividing the continuous annotation time into discrete time points and extracting gaze coordinates at these time points, ensuring that the eye movement data between different annotated images can be compared on the same time scale, and improving the accuracy and reliability of subsequent gaze synchronization calculations.

[0129] More specifically, in step S421, the annotation time of the annotated eye movement data of each annotated image is divided equally based on the number of equally divided time nodes, which can discretize the continuous annotation time and extract the gaze coordinates in the eye movement data at a unified time scale, so that the eye movement data between different annotated images can be compared at the same time node, laying the foundation for the subsequent gaze synchronization calculation.

[0130] In some preferred embodiments, step S43 includes the following steps:

[0131] S431 : Pairing gaze coordinates between adjacent annotated images based on a matching relationship at discrete time points, and calculating gaze synchronization between adjacent annotated images according to a distance relationship between the paired gaze coordinates.

[0132] Specifically, the above steps solve the problem of how to compare eye movement data from different images during the continuous annotation process by establishing matching relationships between discrete time points. Pairing gaze coordinates based on these matching relationships ensures the comparability of gaze data between different images. Then, by calculating the synchronization of the distance relationship between these paired gaze coordinates, the consistency of the continuous annotation process is quantitatively assessed. This processing method not only considers the eye movement characteristics of a single image, but also focuses on the temporal relationship during the continuous annotation process, thereby more accurately identifying anomalies in the annotation process and improving the accuracy and comprehensiveness of the annotation quality assessment.

[0133] More specifically, gaze synchronization can be calculated using a variety of metrics. The simplest approach is to calculate the Euclidean distance between paired gaze coordinates. More complex methods may include factors such as gaze duration and the similarity of gaze trajectories. For example, the dynamic time warping (DTW) algorithm can be used to calculate the similarity of two gaze sequences, which is better able to handle changes in time scale.

[0134] More specifically, compared with the prior art, the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application has the following advantages: First, it not only focuses on the eye movement characteristics of a single image, but also considers the temporal relationship during the continuous annotation process, providing a more comprehensive quality assessment. Second, by introducing a matching mechanism for discrete time points, it solves the problem of comparing eye movement data between images with different annotation durations. Finally, this method can more sensitively detect abnormal changes in the annotation process, such as sudden shifts in attention or inconsistent application of annotation rules, thereby improving the accuracy and efficiency of annotation quality control.

[0135] More specifically, in the embodiment of the present application, the gaze synchronization between adjacent annotated images is calculated based on the following formula:

[0136] (8)

[0137] in, is the gaze synchronization degree of the gaze coordinates corresponding to the t-th discrete time point in the adjacent z-th annotated image and the z+1-th annotated image, is the t-th discrete time point in the adjacent z-th annotated image and the z+1-th annotated image, x z and y z are the horizontal and vertical coordinates of the gaze coordinates at the t-th discrete time point in the z-th annotated image, respectively, z+1 and y z+1 are the horizontal and vertical coordinates of the gaze coordinates at the t-th discrete time point in the z+1-th annotated image, respectively.

[0138] In some preferred embodiments, step S5 includes the following steps:

[0139] S51, calculating the average gaze synchronization degree of each annotated image according to the gaze synchronization degree, and obtaining the average gaze synchronization degree according to all the average gaze synchronization degrees;

[0140] S52, calculating and obtaining a synchronization standard deviation based on the gaze synchronization mean and the average gaze synchronization;

[0141] S53 . Evaluate the annotation quality of each annotated image according to the difference between the gaze synchronization mean and the synchronization standard deviation, so as to screen and obtain a second abnormal image.

[0142] Specifically, step S51 calculates the average value of all gaze synchronization degrees and establishes an overall reference standard. Step S52 uses the synchronization standard deviation to quantify the degree of discreteness of gaze synchronization. Step S53 comprehensively considers the difference between the gaze synchronization mean and the synchronization standard deviation to evaluate the annotation quality, thereby screening out the second abnormal image. Compared with the traditional method that relies only on a single indicator, this multidimensional evaluation method can better adapt to the characteristics of different data sets by introducing the statistical concepts of mean and standard deviation, thereby improving the versatility and robustness of the evaluation method, significantly improving the accuracy and reliability of anomaly detection, and reducing the possibility of misjudgment and missed judgment.

[0143] More specifically, in the embodiment of the present application, the average gaze synchronization degree can be calculated by first calculating the average gaze synchronization degree between adjacent images, and then calculating the average gaze synchronization degree based on all the average gaze synchronization degrees, wherein the average gaze synchronization degree is calculated based on the following formula:

[0144] (9)

[0145] Among them, d z is the mean gaze synchronization between the adjacent zth annotated image and the z+1th annotated image, is the total number of discrete time points.

[0146] More specifically, the average gaze synchronization is calculated based on the following formula:

[0147] (10)

[0148] Among them, d mean is the average gaze synchronization, and np is the total number of annotated images in the target image set.

[0149] More specifically, the synchronization standard deviation is calculated based on the following formula:

[0150] (11)

[0151] Among them, d sd is the standard deviation of synchronization.

[0152] In some preferred embodiments, step S53 includes the steps of:

[0153] S531. Evaluate the annotation quality of each annotated image by comparing the mean value of the gaze synchronization with the deviation range determined based on the synchronization standard deviation. When the mean value of the gaze synchronization exceeds the deviation range, define the subsequent annotated image in the corresponding adjacent annotated images as a second abnormal image.

[0154] Specifically, the mean value of gaze synchronization reflects the continuity of the adjacent image annotation process, which helps to identify sudden changes in annotation quality. The introduction of synchronization standard deviation and deviation range provides an objective and dynamic judgment standard for anomaly detection. The continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application compares the size relationship between the mean value of gaze synchronization and the deviation range determined based on the synchronization standard deviation. When the mean value of gaze synchronization exceeds the deviation range, the subsequently annotated image in the corresponding adjacent annotated images is defined as the second abnormal image.

[0155] More specifically, in the embodiment of the present application, the deviation range is preferably set in combination with the average gaze synchronization and the synchronization standard deviation, such as [d mean - 2d sd , d mean + 2d sd ], so when d z <d mean - 2d sd or d z >d mean + 2d sd , it means that the annotator’s gaze position when annotating the z+1th annotated image significantly deviates from the gaze position of the zth annotated image, resulting in a high probability of an anomaly in the annotation of the z+1th annotated image, and the z+1th annotated image is labeled as the second abnormal image.

[0156] In some other preferred embodiments, step S53 includes the steps of:

[0157] S531′, comparing the mean value of gaze synchronization with the deviation range determined based on the synchronization standard deviation to evaluate the annotation quality of each annotated image; when two consecutive gaze synchronizations exceed the deviation range, defining the middle annotated image among the corresponding three consecutive adjacent annotated images as the second abnormal image.

[0158] Specifically, similar to step S531, the introduction of the synchronization standard deviation and deviation range provides an objective and dynamic judgment standard for anomaly detection; however, a sudden change in the mean value of gaze synchronization may also be purely due to a sudden and drastic change in the position of the annotated object in the annotated image. Therefore, the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application further integrates the two consecutive gaze synchronizations corresponding to three consecutive adjacent annotated images to perform a screening and evaluation of the second abnormal image.

[0159] More specifically, in the embodiment of the present application, the deviation range is also preferably set in combination with the average gaze synchronization and the synchronization standard deviation, such as [d mean - 2d sd , d mean + 2dsd ], so when d i <d mean - 2d sd or d i >d mean + 2d sd and d i+1 <d mean - 2d sd or d i+1 >d mean + 2d sd , the i+1 labeled images are labeled as the second abnormal images to further improve the evaluation accuracy.

[0160] Second, please refer to Figure 3 and Figure 4 Some embodiments of the present application further provide a device for evaluating the quality of continuous image annotation based on eye movement data, the device comprising:

[0161] An acquisition module 201 is configured to acquire an eye movement dataset corresponding to a target image set, where the target image set is an image set for which the annotation task has been completed. The eye movement dataset includes multiple sets of eye movement data information corresponding to the annotators' annotation of different annotated images in the target image set.

[0162] A first calculation module 202 is configured to obtain gaze entropy information of the corresponding annotated image based on the transfer of each set of eye movement data information in the pre-divided region of interest;

[0163] The first evaluation module 203 is configured to evaluate the annotation quality of each annotated image according to the size of the gaze entropy information, so as to screen and obtain the first abnormal image.

[0164] The continuous image annotation quality assessment device based on eye movement data in an embodiment of the present application assesses the image annotation quality by introducing gaze entropy information determined based on the transfer of eye movement data information in a pre-divided region of interest. The gaze entropy information can reflect the long-term attention distribution of the annotator and can be used as a basis for a more comprehensive and in-depth assessment. This enables the device of the present application to use gaze entropy information as a measure of uncertainty to assess the image annotation quality based on the complexity of the annotator's attention distribution, thereby improving the accuracy and reliability of the annotation quality assessment.

[0165] In some preferred embodiments, the device further comprises:

[0166] A second calculation module 204 is configured to calculate the gaze synchronization of each adjacent annotated image based on the position matching degree of the eye movement data information between the consecutively annotated adjacent annotated images with respect to the time sequence;

[0167] The second evaluation module 205 is configured to evaluate the annotation quality of each annotated image according to the degree of gaze synchronization, so as to screen and obtain a second abnormal image.

[0168] In some preferred embodiments, the continuous image annotation quality assessment device based on eye movement data of an embodiment of the present application is used to execute the continuous image annotation quality assessment method based on eye movement data provided in the first aspect above.

[0169] Thirdly, please refer to Figure 5 Some embodiments of the present application also provide a structural diagram of an electronic device. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not marked). The memory 302 stores computer-readable instructions executable by the processor 301. When the electronic device is running, the processor 301 executes the computer-readable instructions to execute the method in any optional implementation of the above embodiments.

[0170] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method of any optional implementation of the above embodiment is executed. The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0171] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0172] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0174] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0175] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A continuous image annotation quality assessment method based on eye movement data, characterized in that: The method comprises the following steps: S1. Acquire an eye movement dataset corresponding to a target image set, wherein the target image set is an image set for which the labeling task has been completed, and the eye movement dataset includes multiple sets of eye movement data information corresponding to the labelers' labeling of different labeled images in the target image set; S2. Obtaining gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest; S3, evaluating the annotation quality of each annotated image according to the size of the gaze entropy information to screen and obtain a first abnormal image; The steps for obtaining gaze entropy information of each annotated image include: S21, obtaining a transfer probability between different regions of interest based on the transfer of the eye movement data information in the pre-divided regions of interest; S22, iteratively calculating and obtaining the stationary probability of each region of interest according to the transition probability; S23, calculating and obtaining a transfer entropy according to the stationary probability and the transfer probability; S24, obtaining a stationary entropy according to the stationary probability calculation; S25, combining the transfer entropy and the stationary entropy to obtain the gaze entropy information; Step S21 includes the following steps: Obtaining the transfer probability between different regions of interest based on the transfer conditions of the eye movement data information in different transfer intervals in the pre-divided regions of interest; The annotated images in the target image set are continuous frame images, and all the annotated images in the target image set are annotated based on the same annotation rule; The method further comprises the steps of: S4, calculating the gaze synchronization of each adjacent annotated image based on the position matching degree of the eye movement data information between the consecutively annotated adjacent annotated images with respect to the time sequence; S5, evaluating the annotation quality of each annotated image according to the size of the gaze synchronization to screen and obtain a second abnormal image; Step S4 includes the following steps: S41, acquiring annotated eye movement data according to the eye movement data information, wherein the annotated eye movement data includes a plurality of gaze coordinates at different time points; S42, extracting multiple gaze coordinates of the annotated eye movement data corresponding to each annotated image based on a preset temporal matching relationship; S43: Calculate the gaze synchronization between adjacent annotated images based on the temporal matching relationship and the gaze coordinates.

2. The continuous image annotation quality assessment method based on eye movement data according to claim 1, characterized in that: The preset timing matching relationship includes equally divided time nodes, and step S42 includes the following steps: S421, dividing the annotation time of the annotated eye movement data of each annotated image into equal parts based on the number of equally divided time nodes to form discrete time points; S422: Extracting a plurality of gaze coordinates from the corresponding annotated eye movement data based on the discrete time points.

3. The continuous image annotation quality assessment method based on eye movement data according to claim 1, characterized in that: Step S5 includes the following steps: S51, calculating the average gaze synchronization degree of each annotated image according to the gaze synchronization degree, and obtaining the average gaze synchronization degree according to all the average gaze synchronization degrees; S52, calculating and obtaining a synchronization standard deviation based on the gaze synchronization mean and the average gaze synchronization; S53: Evaluate the annotation quality of each annotated image according to the difference between the gaze synchronization mean and the synchronization standard deviation, so as to screen and obtain the second abnormal image.

4. A continuous image annotation quality assessment device based on eye movement data, characterized in that: The device comprises: an acquisition module, configured to acquire an eye movement dataset corresponding to a target image set, wherein the target image set is an image set for which the annotation task has been completed, and the eye movement dataset includes multiple sets of eye movement data information corresponding to the annotators during the annotation process of different annotated images in the target image set; A first calculation module is used to obtain gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest; A first evaluation module is used to evaluate the annotation quality of each annotated image according to the size of the gaze entropy information to screen and obtain a first abnormal image; The steps for obtaining gaze entropy information of each annotated image include: S21, obtaining a transfer probability between different regions of interest based on the transfer of the eye movement data information in the pre-divided regions of interest; S22, iteratively calculating and obtaining the stationary probability of each region of interest according to the transition probability; S23, calculating and obtaining a transfer entropy according to the stationary probability and the transfer probability; S24, obtaining a stationary entropy according to the stationary probability calculation; S25, combining the transfer entropy and the stationary entropy to obtain the gaze entropy information; Step S21 includes the following steps: Obtaining the transfer probability between different regions of interest based on the transfer conditions of the eye movement data information in different transfer intervals in the pre-divided regions of interest; The annotated images in the target image set are continuous frame images, and all the annotated images in the target image set are annotated based on the same annotation rule; The device further comprises: A second calculation module is used to calculate the gaze synchronization of each adjacent annotated image based on the position matching degree of the eye movement data information between the consecutively annotated adjacent annotated images with respect to the time sequence; A second evaluation module is used to evaluate the annotation quality of each annotated image according to the degree of gaze synchronization, so as to screen and obtain a second abnormal image; The process of calculating the gaze synchronization of each adjacent annotated image based on the position matching degree of the eye movement data information between the consecutively annotated adjacent annotated images in terms of time sequence includes the following steps: S41, acquiring annotated eye movement data according to the eye movement data information, wherein the annotated eye movement data includes a plurality of gaze coordinates at different time points; S42, extracting multiple gaze coordinates of the annotated eye movement data corresponding to each annotated image based on a preset temporal matching relationship; S43: Calculate the gaze synchronization between adjacent annotated images based on the temporal matching relationship and the gaze coordinates.

5. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 3 are executed.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are executed.

Images