Continuous image annotation quality evaluation method based on eye movement data and related equipment

Abstract

The invention relates to the technical field of image annotation, and particularly discloses a continuous image annotation quality evaluation method based on eye movement data and related equipment, and the method comprises the steps: obtaining an eye movement data set corresponding to a target image set which is an image set with an annotation task completed, the eye movement data set comprises a plurality of groups of eye movement data information generated in the labeling process of different labeling images of the target image set corresponding to labeling personnel; gaze entropy information of the corresponding annotated image is obtained according to the transfer condition of each group of eye movement data information in a pre-divided region of interest; evaluating the annotation quality of each annotation image according to the size of the staring entropy information so as to screen and obtain a first abnormal image; according to the method, the gaze entropy information is used as a measure for measuring uncertainty, and the image annotation quality is evaluated based on the complexity of attention distribution of annotators, so that the accuracy and reliability of annotation quality evaluation are improved.

Description

Technical Field

[0001] The present application relates to the technical field of image annotation, and in particular to a continuous image annotation quality assessment method based on eye movement data and related equipment. Background Art

[0002] The traditional continuous image annotation process relies on a one-time batch annotation and subsequent review model. In this process, the annotator needs to complete a large number of tasks in a concentrated manner, and then the auditor will conduct a comprehensive review. If the annotation is not qualified, it will be returned for correction. This process is cumbersome and inefficient.

[0003] At present, although existing technologies attempt to evaluate the annotation process through eye movement data, their application has obvious shortcomings. Existing technologies mainly perform simple evaluation based on simple factors such as the movement range and speed of eye movement data, and only focus on the eye movement data of a single image, ignoring the association of eye movement data between images and more comprehensive feature analysis. There are shortcomings such as single and one-sided evaluation, which leads to insufficient evaluation accuracy.

[0004] There is currently no effective technical solution to the above problems. Summary of the invention

[0005] The purpose of this application is to provide a continuous image annotation quality assessment method and related equipment based on eye movement data to improve the accuracy and reliability of annotation quality assessment.

[0006] In a first aspect, the present application provides a method for evaluating the quality of continuous image annotation based on eye movement data, the method comprising the following steps: S1. Acquire an eye movement data set corresponding to a target image set, wherein the target image set is an image set for which a labeling task has been completed, and the eye movement data set includes multiple groups of eye movement data information corresponding to the labelers generated during the labeling process of different labeled images in the target image set; S2, acquiring gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest; S3. Evaluate the annotation quality of each annotated image according to the size of the gaze entropy information to screen and obtain a first abnormal image.

[0007] The continuous image annotation quality assessment method based on eye movement data of the present application evaluates the image annotation quality by introducing gaze entropy information determined based on the transfer of eye movement data information in a pre-divided region of interest, so as to evaluate the image annotation quality based on the complexity of the annotator's attention distribution, thereby improving the accuracy and reliability of the annotation quality assessment.

[0008] In the method for evaluating the quality of continuous image annotation based on eye movement data, the step of acquiring gaze entropy information of each annotated image comprises: S21, obtaining the transfer probability between different regions of interest according to the transfer status of the eye movement data information in the pre-divided regions of interest; S22, iteratively calculating and obtaining the stationary probability of each region of interest according to the transition probability; S23, calculating and obtaining a transfer entropy according to the stationary probability and the transfer probability; S24, obtaining a stationary entropy according to the stationary probability calculation; S25. Obtain the gaze entropy information by combining the transfer entropy and the stationary entropy.

[0009] In this example, the transfer probability obtained in step S21 quantifies the annotator's attention transfer pattern, transfer frequency and transfer habits between different areas. The stable probability obtained in step S22 reflects the annotator's attention level to each area of ​​interest in the long term, which helps to understand the annotator's overall attention distribution. The transfer entropy obtained in step S23 is used to measure the uncertainty and complexity of eye movement transfer. The stable entropy obtained in step S24 reflects the uniformity of the annotator's overall attention distribution. The gaze entropy information obtained in step S25 is a comprehensive indicator determined by the comprehensive transfer entropy and stable entropy, which can comprehensively reflect the eye movement characteristics in the annotation process and provide an important basis for evaluating the annotation quality.

[0010] The method for evaluating the quality of continuous image annotation based on eye movement data, wherein step S21 comprises the following steps: The transfer probability between different regions of interest is obtained according to the transfer conditions of the eye movement data information in different transfer intervals in the pre-divided regions of interest.

[0011] The method of the present application not only focuses on the eye movement transfer between adjacent time points, but also considers the transfer pattern on a longer time scale, which can more comprehensively capture the eye movement characteristics during the labeling process, thereby providing a more accurate transfer probability estimation.

[0012] The continuous image annotation quality assessment method based on eye movement data, wherein the annotated images in the target image set are continuous frame images, and all the annotated images in the target image set are annotated based on the same annotation rule; The method further comprises the steps of: S4, calculating the gaze synchronization of each adjacent annotated image based on the position matching degree of the eye movement data information between the consecutively annotated adjacent annotated images with respect to the time sequence; S5. Evaluate the annotation quality of each annotated image according to the gaze synchronization degree to screen and obtain a second abnormal image.

[0013] The method for evaluating the quality of continuous image annotation based on eye movement data, wherein step S4 comprises the following steps: S41, acquiring annotated eye movement data according to the eye movement data information, wherein the annotated eye movement data includes a plurality of gaze coordinates at different time points; S42, extracting multiple gaze coordinates of the annotated eye movement data corresponding to each annotated image based on a preset time sequence matching relationship; S43: Calculate the gaze synchronization between each adjacent annotated image based on the temporal matching relationship and the gaze coordinates.

[0014] In the method for evaluating the quality of continuous image annotation based on eye movement data, the preset time sequence matching relationship includes equally divided time nodes, and step S42 includes the following steps: S421, dividing the labeling time of the labeled eye movement data of each labeled image equally based on the number of equally divided time nodes to form discrete time points; S422: Extract multiple gaze coordinates in the corresponding annotated eye movement data based on the discrete time points.

[0015] The method for evaluating the quality of continuous image annotation based on eye movement data, wherein step S5 comprises the following steps: S51, calculating the average gaze synchronization of each annotated image according to the gaze synchronization, and obtaining the average gaze synchronization according to all the average gaze synchronizations; S52, calculating and obtaining a synchronization standard deviation according to the gaze synchronization mean and the average gaze synchronization; S53: Evaluate the annotation quality of each annotated image according to the difference between the gaze synchronization mean and the synchronization standard deviation, so as to screen and obtain the second abnormal image.

[0016] In a second aspect, the present application further provides a continuous image annotation quality assessment device based on eye movement data, the device comprising: An acquisition module is used to acquire an eye movement data set corresponding to a target image set, wherein the target image set is an image set for which a labeling task has been completed, and the eye movement data set includes multiple groups of eye movement data information corresponding to the labelers generated during the labeling process of different labeled images in the target image set; A first calculation module is used to obtain gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest; The first evaluation module is used to evaluate the annotation quality of each annotated image according to the size of the gaze entropy information, so as to screen and obtain the first abnormal image.

[0017] The continuous image annotation quality assessment device based on eye movement data of the present application assesses the image annotation quality by introducing gaze entropy information determined based on the transfer of eye movement data information in a pre-divided region of interest, so as to assess the image annotation quality based on the complexity of the annotator's attention distribution, thereby improving the accuracy and reliability of the annotation quality assessment.

[0018] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect are executed.

[0019] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps in the method provided in the first aspect are executed.

[0020] From the above, it can be seen that the present application provides a method for evaluating the quality of continuous image annotation based on eye movement data and related equipment, wherein the method of the present application's continuous image annotation quality evaluation device based on eye movement data evaluates the image annotation quality by introducing gaze entropy information determined based on the transfer of eye movement data information in a pre-divided region of interest. The gaze entropy information can reflect the long-term attention distribution of the annotator and can be used as a basis for a more comprehensive and in-depth evaluation, so that the device of the present application can use the gaze entropy information as a measure of uncertainty to evaluate the image annotation quality based on the complexity of the annotator's attention distribution, thereby improving the accuracy and reliability of the annotation quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a method for evaluating the quality of continuous image annotation based on eye movement data provided in some embodiments of the present application.

[0022] Figure 2 A flowchart of a method for evaluating the quality of continuous image annotation based on eye movement data provided in some other embodiments of the present application.

[0023] Figure 3 A schematic diagram of the structure of a continuous image annotation quality assessment device based on eye movement data provided in some embodiments of the present application.

[0024] Figure 4 A schematic diagram of the structure of a continuous image annotation quality assessment device based on eye movement data provided in some other embodiments of the present application.

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0026] Figure numerals: 201, acquisition module; 202, first calculation module; 203, first evaluation module; 204, second calculation module; 205, second evaluation module; 301, processor; 302, memory; 303, communication bus. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0028] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0029] In the quality assessment of continuous image annotation, traditional methods mainly rely on a one-time batch annotation and subsequent review model. This method has problems such as low efficiency and inaccurate evaluation. Although existing technologies have attempted to evaluate the annotation process through eye movement data, there are still obvious shortcomings. Existing technologies mainly evaluate based on simple factors of eye movement data, and only focus on the eye movement data of a single image, ignoring the association between eye movement data between images and more comprehensive feature analysis. This leads to a single and one-sided evaluation result that cannot accurately reflect the true situation of annotation quality.

[0030] Specifically, in a large-scale image annotation project, annotators need to process thousands of images continuously. The annotation quality of each image directly affects the subsequent data analysis and the training effect of the machine learning model. Since traditional methods cannot evaluate the annotation quality in real time, a large number of low-quality annotations may accumulate. For example, in an autonomous driving scene recognition project, annotators need to continuously annotate key elements such as roads, vehicles, and pedestrians. If the annotation quality assessment is inaccurate, a large amount of noisy data may appear during model training, affecting the recognition accuracy. Among them, although eye movement data is introduced into the evaluation process, it only considers simple movement range and rate, and cannot capture the complexity of the annotator's attention distribution and the attention change pattern during the continuous annotation process.

[0031] Therefore, if the problem of insufficient accuracy of continuous image annotation quality assessment cannot be effectively solved, serious technical consequences will occur. First, low-quality annotation data will directly affect the training effect of the machine learning model, resulting in a decline in model performance and failure to achieve the expected recognition or classification accuracy. Secondly, inaccurate assessments will increase the workload of later manual review, greatly increasing project costs and time consumption. In addition, the inability to detect and correct annotation problems in a timely manner may cause annotation errors to accumulate and spread in large amounts of data, ultimately affecting the availability of the entire data set. In some key application areas, such as medical image analysis or security monitoring systems, this problem may lead to serious decision-making errors and bring immeasurable risks. Therefore, improving the accuracy of continuous image annotation quality assessment has become a technical problem that needs to be solved urgently, and a more comprehensive and accurate evaluation method needs to be developed to ensure the quality and reliability of the annotation data.

[0032] First, please refer to Figure 1 and Figure 2 Some embodiments of the present application provide a method for evaluating the quality of continuous image annotation based on eye movement data, the method comprising the following steps: S1. Obtain an eye movement data set corresponding to a target image set, where the target image set is an image set for which the annotation task has been completed, and the eye movement data set includes multiple groups of eye movement data information corresponding to the annotators in the annotation process of different annotated images in the target image set; S2, obtaining gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest; S3. Evaluate the annotation quality of each annotated image according to the size of the gaze entropy information to screen and obtain the first abnormal image.

[0033] Specifically, the target image set refers to the image set that has completed the labeling task.

[0034] An eye movement dataset refers to a data set that records the eye movements of annotators during the annotation process. Specifically, it can be implemented using a data structure containing information such as timestamps and gaze point coordinates. It can be collected and generated through eye tracking devices such as eye trackers. Each set of eye movement data information in the eye movement dataset corresponds to the eye movement data generated during the annotation process of an annotated image in the target image set.

[0035] The area of ​​interest (AOI area) refers to a specific area pre-divided in the area where the annotated image is located, and can be defined in geometric shapes such as rectangles, circles or irregular polygons.

[0036] More specifically, gaze entropy information refers to an information entropy indicator used to measure the distribution of annotator's attention, which can be implemented using an entropy calculation formula based on information theory.

[0037] More specifically, the core innovation of the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application is the introduction of a gaze entropy information analysis method based on eye movement data to evaluate the continuous image annotation quality. This processing method not only takes into account the eye movement data of a single image, but also captures deeper information in the annotation process through the calculation of gaze entropy information. The gaze entropy information can effectively measure the distribution balance of the annotator's attention in different areas of interest and the frequency of switching attention areas, thereby providing a more comprehensive and accurate annotation quality assessment indicator.

[0038] The working principle of the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application can be described in detail as follows: First, step S1 obtains the eye movement data set corresponding to the target image set. This step uses a special eye tracking device to record the eye movement data of the annotator during the annotation process. The eye movement data usually includes information such as timestamp, gaze point coordinates, and gaze duration. These data are divided into multiple groups, and each group of eye movement data information corresponds to the annotation process of an annotated image.

[0039] Next, step S2 obtains gaze entropy information based on the transfer of eye movement data information in pre-divided regions of interest. In specific operations, the area where each annotated image is located is divided into multiple regions of interest based on a predetermined division rule. These regions of interest can be defined according to the importance or complexity of the image content. During the image annotation process, the annotator's gaze point will be transferred and switched between different regions of interest. The transfer probability between different regions of interest can be calculated by analyzing the transfer of the gaze point in the eye movement data between these regions, forming a probability matrix. The gaze entropy information can be obtained by calculating the transfer probability of this matrix based on the information entropy calculation formula.

[0040] Finally, step S3 can evaluate the annotation quality of each annotated image according to the size of the gaze entropy information by setting a reasonable threshold, and identify and screen the annotated images whose gaze entropy information does not meet the corresponding threshold (too large or too small) as the first abnormal images. These first abnormal images represent that there may be problems with their annotation quality and need further review or re-annotation.

[0041] More specifically, in the embodiments of the present application, gaze entropy information can simultaneously reflect the distribution balance and switching frequency of the annotator's attention. A higher gaze entropy value usually indicates that the annotator has paid sufficient attention to different parts of the image, while a lower value may mean that the attention is too concentrated or scattered. Compared with traditional simple eye movement indicators, this evaluation method can more comprehensively reflect the complexity of the annotation process, thereby improving the accuracy of the evaluation.

[0042] The continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application assesses the image annotation quality by introducing gaze entropy information determined based on the transfer of eye movement data information in a pre-divided region of interest. The gaze entropy information can reflect the long-term attention distribution of the annotator and can be used as a basis for a more comprehensive and in-depth assessment. The method of the present application can use gaze entropy information as a measure of uncertainty to assess the image annotation quality based on the complexity of the annotator's attention distribution, thereby improving the accuracy and reliability of the annotation quality assessment.

[0043] In some preferred embodiments, the step of acquiring gaze entropy information of each annotated image includes: S21, obtaining the transfer probability between different regions of interest according to the transfer situation of the eye movement data information in the pre-divided regions of interest; S22, iteratively calculating and obtaining the stationary probability of each region of interest according to the transition probability; S23, calculating and obtaining the transfer entropy according to the stationary probability and the transfer probability; S24, obtaining the stationary entropy according to the stationary probability calculation; S25. Comprehensive transfer entropy and stationary entropy to obtain gaze entropy information.

[0044] Specifically, step S21 to step S25 are the process of calculating and obtaining corresponding gaze entropy information for one annotated image, so step S2 is actually the process of executing steps S21 to S25 for each annotated image in the target image set to obtain corresponding multiple gaze entropy information.

[0045] More specifically, the transfer probability obtained in step S21 quantifies the annotator's attention transfer pattern, transfer frequency and transfer habits between different areas. The stable probability obtained in step S22 reflects the annotator's attention level to each area of ​​interest in the long term, which helps to understand the annotator's overall attention distribution. The transfer entropy obtained in step S23 is used to measure the uncertainty and complexity of eye movement transfer. The stable entropy obtained in step S24 reflects the uniformity of the annotator's overall attention distribution. The gaze entropy information obtained in step S25 is a comprehensive indicator determined by the comprehensive transfer entropy and stable entropy, which can comprehensively reflect the eye movement characteristics in the annotation process and provide an important basis for evaluating the annotation quality.

[0046] More specifically, the region of interest can be divided according to image content, annotation task requirements or expert experience; in an embodiment of the present application, the region of interest is preferably a region of interest set based on the annotated key content, that is, the region of interest is divided by object category or set by pre-planned area, thereby constructing multiple regions of interest corresponding to different object categories.

[0047] More specifically, the transition probability can be obtained by counting the number of transitions between different regions of interest in the eye movement data and normalizing them to obtain a transition probability matrix. For example, if the number of transitions from region A to region B is 10 times and the total number of transitions is 100 times, then the transition probability from A to B is 0.1.

[0048] More specifically, in the embodiment of the present application, the transition probability is calculated based on the following formula: (1) More specifically, where p ij is the transfer probability of the gaze point generated by the eye movement data in the group of eye movement data information from the i-th region of interest to the j-th region of interest, x ij The number of times the gaze point generated by the eye movement data in the group of eye movement data information is transferred from the i-th region of interest to the j-th region of interest, and n is the total number of regions of interest.

[0049] It should be noted that the gaze point is the gaze point in the eye movement data that represents the annotator's gaze state. It can be defined as a relatively stable gaze position point when the eye is within a certain threshold range of deviation (usually 2°), within a certain minimum time (usually 100-200ms), and the eye movement speed is lower than a certain threshold (usually 15°-100° per second).

[0050] More specifically, the calculation of the stationary probability adopts an iterative method. Initially, it can be assumed that the stationary probabilities of each region are equal, and then it is continuously updated according to the transition probability matrix until convergence. This process can be implemented using the stationary distribution calculation method of the Markov chain; in the embodiment of the present application, the stationary probability is preferably calculated based on the following formula: (2) in, represents the stationary probability of the jth region of interest at the yth iteration, represents the stationary probability of the i-th region of interest at the y+1th iteration, p ji is the transition probability of the gaze point generated by the eye movement data in the group of eye movement data information moving from the jth region of interest to the ith region of interest. The initial stable probability is defined as Step S22 is based on formula (2) and after multiple iterations, a stable probability that tends to be stable can be obtained, thereby obtaining stable probabilities corresponding to different regions of interest, and the stable probability of the determined i-th region of interest is recorded as π i , wherein the iteration process can be performed based on a preset number of iterations or by determining whether the stationary probability converges.

[0051] More specifically, the stable stationary probability obtained in step S22 can clearly reflect the stable distribution of the annotator's attention in each region of interest during long-term annotation.

[0052] More specifically, the transfer entropy in step S23 is calculated using the Shannon entropy formula, that is, calculated based on the following formula: (3) Among them, S is the set of all regions of interest, H t is the transfer entropy; higher H t This indicates that the annotator frequently switches the focus area during the annotation process; a lower H t This means that the annotator's eye movement pattern is relatively stable.

[0053] More specifically, the stationary entropy in step S24 is calculated based on the following formula: (4) Among them, H s is the stationary entropy; higher H s This indicates that the attention of the annotators is evenly distributed across different regions of interest; the lower the H s This means that the annotator's attention is overly focused on certain areas of interest.

[0054] More specifically, step S25 can obtain gaze entropy information by combining transfer entropy and stationary entropy in a superposition or weighted superposition manner. In the embodiment of the present application, it is preferably performed based on the following formula: Q = ω 1 H t +ω 2 H s (5) Among them, Q is the gaze entropy information, ω 1 and ω 2 To configure the weight, you can set it according to the research purpose and the importance of each indicator. If you pay more attention to the stability of the annotation, you can appropriately increase ω 1 , and if we emphasize the comprehensiveness of the annotation, we can appropriately increase the 2 .

[0055] In the embodiment of the present application, ω 1 and ω 2 Satisfy 1 +ω 2 =1, and more preferably ω 1 =ω 2 =0.5.

[0056] The continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application analyzes eye movement data from multiple angles, extracts more comprehensive and in-depth transfer entropy and stationary entropy to constitute gaze entropy information, and the gaze entropy information comprehensively reflects the annotator's attention allocation and transfer pattern on this image. Compared with relying solely on simple eye movement indicators, the scheme of the present application not only considers the dynamic characteristics of eye movements, but also considers the long-term attention distribution, thereby providing a more comprehensive and in-depth evaluation basis, which can more comprehensively reflect the cognitive complexity and attention allocation in the annotation process, and help to more accurately identify potential annotation problems, thereby improving the accuracy and reliability of annotation quality assessment.

[0057] In some preferred embodiments, step S21 includes the following steps: S211 . Obtain the transition probability between different regions of interest according to the transition conditions of the eye movement data information in different transition intervals in the pre-divided regions of interest.

[0058] Specifically, in this embodiment, the method of the present application not only focuses on the eye movement transfer between adjacent time points, but also considers the transfer pattern on a longer time scale, which can more comprehensively capture the eye movement characteristics in the labeling process, thereby providing a more accurate transfer probability estimation.

[0059] More specifically, in practical applications, a variety of methods can be used to implement transfer analysis at different transfer intervals. For example, multiple time windows can be set, such as 1 second, 2 seconds, 5 seconds, etc., to count the eye movement transfers in these time windows. Another method is to use sliding window technology to capture transfer patterns on different time scales by continuously moving a fixed-size time window; in addition, you can also consider using a weighted average method to synthesize transfer conditions at different transfer intervals. For example, you can give higher weights to transfers with shorter time intervals, and give relatively lower weights to transfers with longer time intervals. In this way, while retaining long-time scale information, you can emphasize eye movement patterns in short periods of time.

[0060] More specifically, in step S211, in order to highlight the short-term eye movement change trend, the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application introduces the frame interval weight based on the following formula: (6) Among them, m is the maximum transfer interval frame number, w k is the frame interval weight under k transfer interval frames.

[0061] After completing the configuration of the frame interval weight, step S211 is optimized for (1) to obtain the following calculation formula for the translation probability combined with the frame interval weight: (7) in, The number of times a gaze point generated by the eye movement data in the group of eye movement data information is transferred from the i-th region of interest to the j-th region of interest within the k-th transfer interval frame number.

[0062] In some other embodiments, step S2 can directly use the transfer entropy calculated based on formula (3) or the stationary entropy calculated based on formula (4) as gaze entropy information, that is, step S3 can directly use the transfer entropy or stationary entropy to evaluate the annotation quality of each annotated image at the expense of part of the evaluation accuracy to screen the first abnormal image to further improve the evaluation efficiency. If it is known in advance that the eye movement pattern of the annotator is relatively stable, the annotation quality of the annotated image annotated by the annotator can be directly evaluated based on the stationary entropy.

[0063] In some preferred embodiments, the annotated images in the target image set are continuous frame images, and all the annotated images in the target image set are annotated based on the same annotation rule; The method further comprises the steps of: S4, calculating the gaze synchronization of each adjacent annotated image based on the position matching degree of the eye movement data information between the consecutively annotated adjacent annotated images with respect to the time sequence; S5. Evaluate the annotation quality of each annotated image according to the degree of gaze synchronization to screen and obtain a second abnormal image.

[0064] Specifically, the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application introduces a new assessment indicator, namely gaze synchronization, to quantify the degree of synchronization of the annotator's visual attention between continuous images, so as to solve the problem of insufficient assessment accuracy caused by a single assessment method in the continuous image annotation quality assessment.

[0065] More specifically, before executing step S4 and step S5, the continuous image annotation quality assessment method based on eye movement data of the embodiment of the present application first needs to ensure that all annotated images are based on the same annotation rules (for example, by constraining the annotator through an algorithm to start the annotation operation from the upper left corner of the annotated object), so as to provide a consistent basis for the assessment; since the annotated images are continuous frame images, there is a continuous displacement relationship between the corresponding objects to be annotated in adjacent annotated images (two images that appear as continuous frames in time sequence), so the annotation boxes in the adjacent annotated images that are qualified should be adjacent in position and have a certain matching relationship. At the same time, due to the constraints on the annotation rules, the annotators should respectively place the annotation boxes in the adjacent annotated images. The change trajectory of the generated gaze points should also be adjacent in position and have a certain distance matching relationship; therefore, step S4 introduces gaze synchronization based on time series matching of eye movement data as another quality evaluation indicator. This indicator not only takes into account the annotation quality of a single image, but also pays attention to the correlation between adjacent images in the continuous annotation process, and can effectively identify images with abnormal annotation quality; the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application introduces the step of gaze synchronization to perform a secondary evaluation of the annotated image, and combines the previous evaluation processing of gaze entropy information to realize multi-dimensional evaluation, which significantly improves the accuracy and comprehensiveness of the evaluation, and can more effectively identify abnormal situations in the annotation process.

[0066] In some preferred embodiments, step S4 comprises the following steps: S41, acquiring annotated eye movement data according to the eye movement data information, where the annotated eye movement data includes a plurality of gaze coordinates at different time points; S42, extracting multiple gaze coordinates of the annotated eye movement data corresponding to each annotated image based on a preset time sequence matching relationship; S43: Calculate the gaze synchronization between each adjacent annotated image based on the temporal matching relationship and the gaze coordinates.

[0067] Specifically, first, step S41 obtains annotated eye movement data including gaze coordinates of multiple time points to remove invalid eye movement data generated by actual non-annotation behaviors, so that the gaze change trajectory of the remaining annotated eye movement data theoretically conforms to the same annotation rules, thereby providing a basis for analysis; then, step S42 ensures the comparability of eye movement data between different annotated images through a preset timing matching relationship, and extracts the gaze coordinates corresponding to each annotated image based on this relationship to ensure data consistency; finally, step S43 uses the timing matching relationship and the extracted gaze coordinates to calculate the gaze synchronization between adjacent annotated images, thereby evaluating the continuity and consistency of the annotation quality.

[0068] More specifically, to ensure the comparability of eye movement data between different annotated images, step S42 introduces a preset timing matching relationship, which can be formulated based on a variety of strategies, such as equal time interval sampling, key frame matching, or dynamic time warping. Selecting a suitable timing matching relationship is crucial for subsequent analysis because it directly affects the accuracy of gaze coordinate extraction and gaze synchronization calculation.

[0069] More specifically, the process of extracting multiple gaze coordinates in step S42 may use an interpolation algorithm or a nearest neighbor selection method to ensure that the extracted gaze coordinates can accurately reflect the eye movement trajectory during the labeling process.

[0070] More specifically, the method for calculating gaze synchronization in step S43 may include calculation methods such as Euclidean distance, cosine similarity or dynamic time warping distance. Selecting a suitable calculation method can more accurately reflect the continuity and consistency of the annotation process.

[0071] More specifically, the above steps achieve the precise extraction and analysis of eye movement data during the continuous image annotation process. Combined with the preset timing matching relationship, the comparability of eye movement data between different annotated images is ensured, solving the problem of traditional methods that only focus on the eye movement data of a single image while ignoring the relationship between images. The gaze synchronization finally calculated can comprehensively evaluate the continuity and consistency of the annotation quality, providing a more reliable quality assessment method.

[0072] In some preferred implementations, the preset timing matching relationship includes equally dividing the number of time nodes, and step S42 includes the following steps: S421, dividing the labeling time of the labeled eye movement data of each labeled image equally based on the number of equally divided time nodes to form discrete time points; S422: extracting a plurality of gaze coordinates in the corresponding annotated eye movement data based on discrete time points.

[0073] Specifically, step S421 and step S422 solve the problem of how to extract comparable eye movement data in the continuous image annotation process by dividing the continuous annotation time into discrete time points and extracting gaze coordinates at these time points, ensuring that the eye movement data between different annotated images can be compared on the same time scale, thereby improving the accuracy and reliability of subsequent gaze synchronization calculation.

[0074] More specifically, in step S421, the annotation time of the annotated eye movement data of each annotated image is equally divided based on the number of equally divided time nodes, which can discretize the continuous annotation time and extract the gaze coordinates in the eye movement data with a unified time scale, so that the eye movement data between different annotated images can be compared at the same time nodes, laying the foundation for the subsequent gaze synchronization calculation.

[0075] In some preferred embodiments, step S43 includes the following steps: S431 , pairing gaze coordinates between adjacent annotated images based on a matching relationship at discrete time points, and calculating gaze synchronization between adjacent annotated images according to a distance relationship between the paired gaze coordinates.

[0076] Specifically, the above steps solve the problem of how to compare the eye movement data of different images in the continuous annotation process by establishing a matching relationship between discrete time points. The practice of pairing gaze coordinates based on the matching relationship between discrete time points ensures the comparability of gaze data between different images. Then, by calculating the synchronization degree of the distance relationship between these paired gaze coordinates, the consistency in the continuous annotation process is quantitatively evaluated. This processing method not only considers the eye movement characteristics of a single image, but also pays attention to the temporal relationship in the continuous annotation process, so that abnormal situations in the annotation process can be more accurately identified, and the accuracy and comprehensiveness of the annotation quality assessment are improved.

[0077] More specifically, gaze synchronization can be calculated using a variety of metrics. The simplest approach is to calculate the Euclidean distance between paired gaze coordinates. More complex methods may include considering factors such as gaze duration and similarity of gaze trajectories. For example, the dynamic time warping (DTW) algorithm can be used to calculate the similarity of two gaze sequences, which can better handle changes in time scale.

[0078] More specifically, compared with the prior art, the continuous image annotation quality assessment method based on eye movement data of the embodiment of the present application has the following advantages: First, it not only focuses on the eye movement characteristics of a single image, but also considers the temporal relationship in the continuous annotation process, providing a more comprehensive quality assessment. Secondly, by introducing a matching mechanism for discrete time points, the problem of comparing eye movement data between images of different annotation durations is solved. Finally, this method can more sensitively detect abnormal changes in the annotation process, such as sudden shifts in attention or inconsistent application of annotation rules, thereby improving the accuracy and efficiency of annotation quality control.

[0079] More specifically, in the embodiment of the present application, the gaze synchronization between adjacent annotated images is calculated based on the following formula: (8) in, is the gaze synchronization degree of the gaze coordinates corresponding to the t-th discrete time point in the adjacent z-th annotated image and the z+1-th annotated image, is the t-th discrete time point in the adjacent z-th annotated image and z+1-th annotated image, x z and zare the horizontal and vertical coordinates of the gaze coordinates at the t-th discrete time point in the z-th annotated image, respectively, z+1 and z+1 are the abscissa and ordinate of the gaze coordinates at the tth discrete time point in the z+1th annotated image, respectively.

[0080] In some preferred embodiments, step S5 comprises the following steps: S51, calculating the average gaze synchronization of each annotated image according to the gaze synchronization, and obtaining the average gaze synchronization according to all the average gaze synchronizations; S52, calculating and obtaining the synchronization standard deviation according to the gaze synchronization mean and the average gaze synchronization; S53: Evaluate the annotation quality of each annotated image according to the difference between the gaze synchronization mean and the synchronization standard deviation, so as to screen and obtain a second abnormal image.

[0081] Specifically, step S51 calculates the average value of all gaze synchronization and establishes an overall reference standard. Step S52 uses the synchronization standard deviation to quantify the discrete degree of gaze synchronization. Step S53 comprehensively considers the difference between the gaze synchronization mean and the synchronization standard deviation to evaluate the annotation quality, thereby screening out the second abnormal image. Compared with the traditional method that only relies on a single indicator, this multi-dimensional evaluation method can better adapt to the characteristics of different data sets by introducing the statistical concepts of mean and standard deviation, thereby improving the versatility and robustness of the evaluation method, significantly improving the accuracy and reliability of anomaly detection, and reducing the possibility of misjudgment and missed judgment.

[0082] More specifically, in the embodiment of the present application, the average gaze synchronization degree can be calculated by first calculating the average gaze synchronization degree between adjacent images, and then calculating the average gaze synchronization degree based on all the average gaze synchronization degrees, wherein the average gaze synchronization degree is calculated based on the following formula: (9) Among them, d z is the mean gaze synchronization between the adjacent z-th annotated image and the z+1-th annotated image, is the total number of discrete time points.

[0083] More specifically, the average gaze synchronization is calculated based on the following formula: (10) Among them, d mean is the average gaze synchronization, and np is the total number of annotated images in the target image set.

[0084] More specifically, the synchronization standard deviation is calculated based on the following formula: (11) Among them, d sd is the synchronization standard deviation.

[0085] In some preferred embodiments, step S53 comprises the steps of: S531. Compare the mean value of gaze synchronization with the deviation range determined based on the synchronization standard deviation to evaluate the annotation quality of each annotated image. When the mean value of gaze synchronization exceeds the deviation range, define the subsequent annotated image in the corresponding adjacent annotated images as the second abnormal image.

[0086] Specifically, the mean gaze synchronization reflects the continuity of the adjacent image annotation process, which helps to identify sudden changes in the annotation quality. The introduction of the synchronization standard deviation and the deviation range provides an objective and dynamic judgment criterion for anomaly detection. The continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application compares the size relationship between the mean gaze synchronization and the deviation range determined based on the synchronization standard deviation. When the mean gaze synchronization exceeds the deviation range, the subsequently annotated image in the corresponding adjacent annotated images is defined as the second abnormal image.

[0087] More specifically, in the embodiment of the present application, the deviation range is preferably set in combination with the average gaze synchronization and the synchronization standard deviation, such as [d mean - 2d sd , d mean + 2d sd ], so when d z <d mean - 2d sd or z >d mean + 2d sd , it means that the annotator’s gaze position when annotating the z+1th annotated image significantly deviates from the gaze position of the zth annotated image, resulting in a high probability of an annotated abnormality in the annotation of the z+1th annotated image, and the z+1th annotated image is annotated as the second abnormal image.

[0088] In some other preferred embodiments, step S53 includes the steps of: S531′, compare the relationship between the mean value of gaze synchronization and the deviation range determined based on the synchronization standard deviation to evaluate the annotation quality of each annotated image, and when two consecutive gaze synchronizations exceed the deviation range, define the middle annotated image among the corresponding three consecutive adjacent annotated images as the second abnormal image.

[0089] Specifically, similar to step S531, the introduction of the synchronization standard deviation and the deviation range provides an objective and dynamic judgment criterion for anomaly detection; however, a sudden change in the mean value of gaze synchronization may also be purely caused by a sudden and drastic change in the position of the annotated object in the annotated image. Therefore, the continuous image annotation quality assessment method based on eye movement data in the embodiment of the present application further integrates two consecutive gaze synchronizations corresponding to three consecutive adjacent annotated images to perform screening and evaluation of the second abnormal image.

[0090] More specifically, in the embodiment of the present application, the deviation range is also preferably set in combination with the average gaze synchronization and the synchronization standard deviation, such as [d mean - 2d sd , d mean + 2d sd ], so when d i <d mean - 2d sd or i >d mean + 2d sd And d i+1 <d mean - 2d sd or i+1 >d mean + 2d sd When the i+1 labeled image is marked as the second abnormal image, the evaluation accuracy is further improved.

[0091] Second, please refer to Figure 3 and Figure 4 Some embodiments of the present application further provide a continuous image annotation quality assessment device based on eye movement data, the device comprising: An acquisition module 201 is used to acquire an eye movement data set corresponding to a target image set, where the target image set is an image set for which the annotation task has been completed, and the eye movement data set includes multiple groups of eye movement data information corresponding to the annotators generated during the annotation process of different annotated images in the target image set; A first calculation module 202 is used to obtain gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest; The first evaluation module 203 is used to evaluate the annotation quality of each annotated image according to the size of the gaze entropy information, so as to screen and obtain the first abnormal image.

[0092] The continuous image annotation quality assessment device based on eye movement data of the embodiment of the present application assesses the image annotation quality by introducing gaze entropy information determined based on the transfer of eye movement data information in a pre-divided region of interest. The gaze entropy information can reflect the long-term attention distribution of the annotator and can be used as a basis for a more comprehensive and in-depth assessment. The device of the present application can assess the image annotation quality based on the complexity of the annotator's attention distribution based on the gaze entropy information as a measure of uncertainty, thereby improving the accuracy and reliability of the annotation quality assessment.

[0093] In some preferred embodiments, the device further comprises: A second calculation module 204 is used to calculate the gaze synchronization of each adjacent annotated image based on the position matching degree of the eye movement data information between the consecutively annotated adjacent annotated images with respect to the time sequence; The second evaluation module 205 is used to evaluate the annotation quality of each annotated image according to the degree of gaze synchronization, so as to screen and obtain the second abnormal image.

[0094] In some preferred implementations, the continuous image annotation quality assessment device based on eye movement data of an embodiment of the present application is used to execute the continuous image annotation quality assessment method based on eye movement data provided in the first aspect above.

[0095] Third, please refer to Figure 5 Some embodiments of the present application also provide a structural diagram of an electronic device. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not marked). The memory 302 stores computer-readable instructions executable by the processor 301. When the electronic device is running, the processor 301 executes the computer-readable instructions to execute the method in any optional implementation of the above embodiments.

[0096] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method in any optional implementation of the above embodiment is executed. The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0097] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0098] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0100] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0101] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A continuous image annotation quality assessment method based on eye movement data, characterized in that: The method comprises the following steps: S1. Acquire an eye movement data set corresponding to a target image set, wherein the target image set is an image set for which a labeling task has been completed, and the eye movement data set includes multiple groups of eye movement data information corresponding to the labelers generated during the labeling process of different labeled images in the target image set; S2, acquiring gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest; S3. Evaluate the annotation quality of each annotated image according to the size of the gaze entropy information to screen and obtain a first abnormal image.

2. The continuous image annotation quality assessment method based on eye movement data according to claim 1, characterized in that: The steps for obtaining gaze entropy information of each annotated image include: S21, obtaining the transfer probability between different regions of interest according to the transfer situation of the eye movement data information in the pre-divided regions of interest; S22, iteratively calculating and obtaining the stationary probability of each region of interest according to the transition probability; S23, calculating and obtaining a transfer entropy according to the stationary probability and the transfer probability; S24, obtaining a stationary entropy according to the stationary probability calculation; S25. Obtain the gaze entropy information by combining the transfer entropy and the stationary entropy.

3. The continuous image annotation quality assessment method based on eye movement data according to claim 2, characterized in that: Step S21 includes the following steps: The transfer probability between different regions of interest is obtained according to the transfer conditions of the eye movement data information in different transfer intervals in the pre-divided regions of interest.

4. The continuous image annotation quality assessment method based on eye movement data according to claim 1, characterized in that: The annotated images in the target image set are continuous frame images, and all the annotated images in the target image set are annotated based on the same annotation rule; The method further comprises the steps of: S4, calculating the gaze synchronization of each adjacent annotated image based on the position matching degree of the eye movement data information between the consecutively annotated adjacent annotated images with respect to the time sequence; S5. Evaluate the annotation quality of each annotated image according to the gaze synchronization degree to screen and obtain a second abnormal image.

5. The continuous image annotation quality assessment method based on eye movement data according to claim 4, characterized in that: Step S4 includes the following steps: S41, acquiring annotated eye movement data according to the eye movement data information, wherein the annotated eye movement data includes a plurality of gaze coordinates at different time points; S42, extracting multiple gaze coordinates of the annotated eye movement data corresponding to each annotated image based on a preset time sequence matching relationship; S43: Calculate the gaze synchronization between each adjacent annotated image based on the temporal matching relationship and the gaze coordinates.

6. The continuous image annotation quality assessment method based on eye movement data according to claim 5, characterized in that: The preset timing matching relationship includes equally dividing the number of time nodes, and step S42 includes the following steps: S421, dividing the labeling time of the labeled eye movement data of each labeled image equally based on the number of equally divided time nodes to form discrete time points; S422: Extract multiple gaze coordinates in the corresponding annotated eye movement data based on the discrete time points.

7. The continuous image annotation quality assessment method based on eye movement data according to claim 4, characterized in that: Step S5 includes the following steps: S51, calculating the average gaze synchronization of each annotated image according to the gaze synchronization, and obtaining the average gaze synchronization according to all the average gaze synchronizations; S52, calculating and obtaining a synchronization standard deviation according to the gaze synchronization mean and the average gaze synchronization; S53: Evaluate the annotation quality of each annotated image according to the difference between the gaze synchronization mean and the synchronization standard deviation, so as to screen and obtain the second abnormal image.

8. A continuous image annotation quality assessment device based on eye movement data, characterized in that: The device comprises: An acquisition module is used to acquire an eye movement data set corresponding to a target image set, wherein the target image set is an image set for which a labeling task has been completed, and the eye movement data set includes multiple groups of eye movement data information corresponding to the labelers generated during the labeling process of different labeled images in the target image set; A first calculation module is used to obtain gaze entropy information of the corresponding annotated image according to the transfer of each group of eye movement data information in the pre-divided region of interest; The first evaluation module is used to evaluate the annotation quality of each annotated image according to the size of the gaze entropy information, so as to screen and obtain the first abnormal image.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 7 are executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are executed.

Images