Image annotation workload statistics methods and related equipment

By assigning a stable global index and label number combination to image annotation tasks, a unique work unit identifier is formed, effective work is recorded, and batch speed checks and administrator acceptance are performed. This solves the problem of duplicate counting in image annotation workload statistics and ensures the accuracy and consistency of work time statistics.

CN123117927APending Publication Date: 2026-11-17SHENZHEN YUYI ROBOT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611500032.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-24
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing methods for calculating image annotation workload cannot effectively distinguish between annotation work done on the same image at different labeling stages, resulting in repeated viewing and submission operations being counted repeatedly in the workload, leading to distorted statistical results.

Method used

By assigning a stable global index to each image and combining it with a label number to form a unique work unit identifier, effective work is recorded and speed checks and administrator acceptance are performed within batches, ensuring that effective work hours are only counted after quality verification.

Benefits of technology

This technology enables independent counting of annotation work for the same image in multi-label annotation scenarios, eliminating duplicate counting across stages and ensuring consistency between work time statistics and annotation quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117927A_ABST
    Figure CN123117927A_ABST
Patent Text Reader

Abstract

This application relates to the field of data processing technology and provides a method and related equipment for statistical analysis of image annotation workload. The method assigns a stable global index to each image in an image sample set and a label number to each label, resulting in a sample sequence and a label sequence. The label number and the stable global index are combined to obtain a unique work unit identifier. Effective labor is assessed for annotation operations; work units that pass the assessment are recorded as work events and aggregated into the current batch. Once the batch sealing conditions are met, the batch is closed, resulting in a batch awaiting inspection. A speed check is performed on the batch awaiting inspection to obtain a batch awaiting acceptance. For batches that pass acceptance, the effective working hours of the annotators are calculated based on the configured working hours. This application constructs a unique work unit identifier by combining the label number and the stable global index, decoupling effective workload and effective working hours into independent statistical levels, ensuring the accuracy of workload statistics in multi-label scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and related equipment for calculating the workload of image annotation. Background Technology

[0002] With the rapid development of artificial intelligence technology, image annotation, as a core part of machine learning data preparation, places increasingly higher demands on the accurate statistics of annotation workload. Existing methods for calculating image annotation workload typically use the number of images or the number of keystrokes as the statistical metric. When an annotation task involves multiple labeling stages, the annotation work for the same image at different labeling stages cannot be independently identified and distinguished, leading to repeated viewing, repeated submissions, and other operations being counted repeatedly in the workload, resulting in distorted workload statistics.

[0003] In existing technologies, some solutions attempt to filter duplicate operations by recording timestamps or operation logs. However, due to the lack of a unique work unit identification mechanism based on the label stage, it is impossible to fundamentally distinguish between effective annotation work and duplicate operations for the same image at different label stages. Furthermore, effective workload is directly linked to effective working hours, and annotation work that has not undergone quality acceptance is also converted into working hours, resulting in unreliable statistical results. Summary of the Invention

[0004] This invention provides a method and related equipment for calculating image annotation workload, in order to solve the technical problem in existing image annotation workload calculation methods that the lack of a unique work unit identification mechanism based on the label stage leads to repeated operations being counted repeatedly in the workload under multiple label stages.

[0005] In a first aspect, embodiments of this application provide a method for calculating image annotation workload, including: Obtain the image sample set and the corresponding label configuration, assign a stable global index to each image in the image sample set, assign a label number to each label, and obtain a sample sequence and label sequence with a stable global index; The tag number in the sample sequence and tag sequence is combined with the stable global index to obtain a unique working unit identifier; Based on the unique work unit identifier, the annotation operation of the personnel in the work view is judged as valid labor. The work units that pass the judgment are recorded as work events and aggregated into the current batch. After the batch sealing conditions are met, the batch is closed and the batch to be inspected is obtained. Based on the time records of work events within the batch to be inspected, the average activity time of a unit workload within the batch is checked to obtain the batch to be accepted. Based on the sampling acceptance results of the batches to be accepted by the administrator, the batches that have passed the acceptance are converted according to the configured working hours to obtain the effective working hours of the marked personnel.

[0006] Secondly, embodiments of this application provide an image annotation workload statistics device, comprising: The sequence construction module is used to obtain the image sample set and the corresponding label configuration, assign a stable global index to each image in the image sample set, assign a label number to each label, and obtain a sample sequence and a label sequence with a stable global index. The unit identification module is used to combine the tag number in the sample sequence and the tag sequence with the stable global index to obtain a unique working unit identifier; The labor judgment module is used to make valid labor judgments on the annotation operations of the personnel in the work view based on the unique work unit identifier, record the work units that pass the judgment as work events and aggregate them into the current batch, close the batch after the batch sealing conditions are met, and obtain the batch to be inspected. The speed check module is used to check the average activity time of a unit workload within the batch based on the time records of work events within the batch to be inspected, and to obtain the batch to be accepted. The working hours conversion module is used to convert the batches that have passed the acceptance inspection according to the configured working hours based on the sampling acceptance results of the batches to be accepted by the administrator, so as to obtain the effective working hours of the marked personnel.

[0007] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described image annotation workload statistics method.

[0008] Fourthly, embodiments of this application provide a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described image annotation workload statistics method.

[0009] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, enables the implementation of the above-described image annotation workload statistics method.

[0010] In one solution provided by the aforementioned image annotation workload statistics method, device, equipment, medium, and program products, a combination of label number and stable global index is used as the unique identifier of the work unit. This ensures that the annotation work for the same image at different labeling stages is counted independently and separately, thus eliminating the problem of duplicate counting across stages in multi-label annotation scenarios from the identification level. Based on this, the solution designs effective workload and effective working hours as two independent statistical levels. Effective workload is recorded immediately when the work unit passes the effective workload determination, while effective working hours are only converted after the batch passes speed checks and administrator acceptance. This decoupling ensures that annotation work without quality verification cannot be directly converted into working hours, guaranteeing consistency between working hour statistics and annotation quality. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the structure of an image annotation workload statistics system according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an image annotation workload statistics method according to an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of the implementation process of step S20; Figure 4 yes Figure 2 A schematic diagram of the implementation process of step S30; Figure 5 This is a schematic diagram of an image annotation workload statistics device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0016] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0017] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0018] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0019] To address the problems mentioned above in the background art, this application provides a method, apparatus, device, medium, and program product for calculating image annotation workload. The image annotation workload calculation method provided by this invention can be applied to applications such as... Figure 1 The image annotation workload statistics system shown includes a client and a server.

[0020] In one embodiment, such as Figure 2 As shown, a method for calculating the workload of image annotation is provided, and this method is applied to... Figure 1 Taking the image annotation workload statistics system as an example, the following steps are included: S10: Obtain the image sample set and the corresponding label configuration, assign a stable global index to each image in the image sample set, assign a label number to each label, and obtain a sample sequence and label sequence with a stable global index; S20: Combine the tag number in the sample sequence and tag sequence with the stable global index to obtain a unique working unit identifier; S30: Based on the unique work unit identifier, perform effective labor judgment on the annotation operation of the personnel in the work view, record the work units that pass the judgment as work events and aggregate them into the current batch, close the batch after the batch sealing conditions are met, and obtain the batch to be inspected; S40: Based on the time records of work events within the batch to be inspected, perform a speed check on the average activity time per unit workload within the batch to obtain the batch to be accepted. S50: Based on the administrator's sampling acceptance results of the batch to be accepted, the batch that has passed the acceptance is converted according to the configured working hours to obtain the effective working hours of the marked personnel.

[0021] In this embodiment, the image sample set can be a pre-prepared dataset of images to be labeled, such as several medical images, street view images, or industrial inspection images. During initialization, the system assigns a unique and stable global index to each image according to the fixed storage order of the image sample set. This index remains unchanged throughout the entire labeling task lifecycle and does not change with subsequent labeling operations, modifications to the sample state, or batch changes. For example, if the image sample set contains 1000 images, they are sequentially assigned indices from 0 to 999 to obtain the sample sequence.

[0022] It should be noted that the label configuration in this embodiment refers to a predefined label system for annotation tasks. The system assigns a unique label number to each label according to the label configuration order of the annotation task, and each label number corresponds to an independent annotation stage. For example, if an annotation task includes two labels: "whether it contains a target object" and "target object category," then label numbers label_0 and label_1 are assigned respectively, resulting in a label sequence.

[0023] In this embodiment, the system performs a one-to-one correspondence between the stable global index in the sample sequence and the label number in the label sequence to form a working unit identifier. Specifically, for the combination of stable global index i and label number label_j, its working unit identifier can be represented as (label_j, i).

[0024] It should be noted that the same stable global index and different label numbers are combined to form independent working unit identifiers. For example, an image with index 5 corresponds to working unit identifier (label_0, 5) in the label_0 stage and (label_1, 5) in the label_1 stage. The two are independent of each other and represent a complete judgment of the image in different annotation stages.

[0025] Based on this, the system initializes an independent counting cursor (counted_cursor) for each labeling stage corresponding to each label number. The counting cursor records the maximum stable global index that has been completed for that labeling stage. For example, the initial counting cursor for the label_0 stage is -1, indicating that no work units have been counted in that stage. Through the above combination and cursor initialization, the system performs uniqueness verification of the work unit identifier, ensuring that the combination of the same label number and the same stable global index uniquely corresponds to a work unit at any given time, thus obtaining a uniquely verified work unit identifier.

[0026] In this embodiment, annotators perform annotation operations on image samples in the work view. The system determines the effective workload of each annotation operation based on a unique work unit identifier. Specifically, the system records the image display time of the current work unit and calculates the image display duration when the annotator completes the judgment of the current annotation stage. This duration is compared with a preset minimum viewing time to obtain the viewing time determination result. For example, if the preset minimum viewing time is 0.45 seconds, and the image display duration is less than 0.45 seconds, then this operation is not counted as effective workload.

[0027] Provided the viewing time requirement is met, the system further compares the stable global index of the current work unit with the counting cursor of the annotation stage to which the current work unit belongs. Only when the stable global index crosses the counting cursor for the first time and the annotator completes the judgment of the current annotation stage, the system records the work unit as a work event and advances the counting cursor. For example, if the counting cursor of the label_0 stage is currently 5, and the annotator completes the judgment of the image with index 6 in the label_0 stage, the system records the work unit as a work event and advances the counting cursor of the label_0 stage to 6, obtaining the work event and the updated counting cursor. If the annotator returns to view the image with index 3 and submits again, since 3 is not greater than the current cursor 6, the system does not repeat the counting.

[0028] The scenarios for completing the current label stage judgment include: determining that the current label is matched and writing the current label; determining that the current label is not matched and proceeding to the next image to complete the judgment; marking samples that cannot be judged as unknown; and marking invalid samples as delete.

[0029] Based on this, the system continuously aggregates work events into the current batch. When the number of work events in a batch reaches the batch sealing condition, the batch is closed, resulting in a batch awaiting inspection. For example, if the batch sealing condition is configured to have 100 work events in a batch, then every 100 accumulated work events will form a batch awaiting inspection.

[0030] In this embodiment, after the batch is closed, the system performs a speed check based on the time records of work events within the batch. Specifically, the system classifies the time intervals between adjacent work events within the batch to be checked: time intervals not exceeding a first threshold are counted as activity time, time intervals exceeding a second threshold are counted as idle time, and records between the first and second thresholds are considered indefinite intervals, thus obtaining the batch activity time statistics. For example, if the first threshold is configured as 10 seconds and the second threshold is configured as 30 seconds, if the time interval between two adjacent work events is 8 seconds, it is counted as activity time; if the interval is 45 seconds, it is counted as idle time; and if the interval is 20 seconds, it is recorded as an indefinite interval.

[0031] After obtaining the batch activity time statistics, the system calculates the ratio of the total activity time within the batch to the number of work events within the batch, and compares this ratio with the preset minimum activity time per work unit. For example, if the preset minimum activity time per work unit is 0.48 seconds, and the total activity time within the batch is 40 seconds with 100 work events, the ratio is 0.4 seconds, which is lower than 0.48 seconds. The system determines that the batch's speed is abnormal and transfers it to the rework process, and the effective workload of this batch is not converted into working hours. If the ratio is not lower than 0.48 seconds, the system transfers the batch to the pending acceptance status, resulting in a batch awaiting acceptance.

[0032] In this embodiment, the batches to be accepted are randomly sampled and accepted by the administrator. The administrator checks the labeling results of each sample, marks the samples with labeling errors as incorrect, and finally submits a conclusion of acceptance or failure.

[0033] It should be noted that the administrator's sampling inspection, error marking, and acceptance judgment operations are not included in the effective workload of the annotator, nor do they advance the annotator's counting cursor. The above operations are only retained as audit evidence for quality traceability.

[0034] For batches that pass acceptance, the system calculates the effective working hours for the annotation personnel based on the configured working hours. For example, each passing batch is converted to 0.5 hours. If three batches pass acceptance on a given day, the annotation personnel will have 1.5 hours of effective working hours for that day. For batches that fail acceptance, the system transfers them to the rework process. The annotation personnel then perform the entire batch rework in the work view according to the original batch work unit sequence. After the rework is completed, the batch is resubmitted for administrator acceptance, and the annotation personnel's effective working hours are obtained.

[0035] In one embodiment, obtaining the image sample set and corresponding label configuration, assigning a stable global index to each image in the image sample set, and assigning a label number to each label to obtain a sample sequence and label sequence with a stable global index includes: According to the fixed storage order of the image sample set, each image in the image sample set is assigned a unique and stable global index that does not change with subsequent annotation operations, thus obtaining the sample sequence; According to the label configuration order of the labeling task, each label is assigned a unique label number to obtain a label sequence, where each label number corresponds to an independent labeling stage.

[0036] In this embodiment, the image sample set can be a pre-prepared dataset of images to be labeled, such as several medical images, street view images, or industrial inspection images. During the initialization phase of the labeling task, the system reads the image sample set and assigns a unique, stable global index to each image according to the fixed order of the image files in the storage directory. Taking a sample set containing 1000 images as an example, the system assigns indices 0 to 999 sequentially, where index 0 corresponds to the first image in the storage order, and index 999 corresponds to the last image, thus obtaining the sample sequence.

[0037] It's important to note that the "stability" of the stable global index is reflected in the following two aspects: First, once assigned, the index is written to persistent storage and does not change with subsequent annotation operations. For example, if an annotator performs operations such as hit detection, miss detection, marking as unknown, or delete on an image, it will not cause any changes to the image's index. Second, even if some samples are removed from the candidate queue due to being marked as invalid during subsequent annotation processes, the indexes of the remaining samples will not be rearranged. The correspondence between each image and its stable global index remains unique and fixed throughout the entire annotation task lifecycle. This stability is the foundation for subsequent work unit identifier uniqueness verification and counting cursor deduplication mechanisms. If the index changes, the same image may correspond to different work unit identifiers at the same annotation stage, leading to duplicate counting or omissions.

[0038] In constructing the label sequence, label configuration refers to the predefined label system for the annotation task, describing which dimensions need to be judged sequentially for the image in this annotation task. The system assigns a unique label number to each label according to the definition order of the labels in the label configuration file, with each label number corresponding to an independent annotation stage. Taking an annotation task containing two labels—"whether it contains a target object" and "target object category"—as an example, the system assigns label numbers label_0 and label_1 respectively according to the configuration order. Label_0 corresponds to the first annotation stage, requiring annotators to determine whether each image contains a target object; label_1 corresponds to the second annotation stage, requiring annotators to further determine the specific category of the target object for images that have passed the first stage. The two annotation stages are independent of each other, each maintaining its own independent candidate sample range and counting state, resulting in a label sequence.

[0039] It is understood that the number and meaning of labels for different annotation tasks can be flexibly configured according to actual needs. The two labels in this embodiment are only illustrative examples and do not constitute a limitation on this solution.

[0040] like Figure 3 As shown, step S20 specifically includes the following steps: S21: Combine the stable global index in the sample sequence with the label number in the label sequence in a one-to-one correspondence. The same stable global index and different label numbers are combined to form independent working unit identifiers, thus obtaining a set of working unit identifiers. S22: Based on the set of working unit identifiers, initialize an independent counting cursor for each labeling stage corresponding to each label number to obtain the initial counting cursor for each labeling stage. The counting cursor records the maximum stable global index that the labeling stage has completed counting. S23: Based on the set of working unit identifiers and the initial counting cursors of each labeling stage, perform a uniqueness check on the combination of the same label number and the same stable global index to obtain a unique working unit identifier.

[0041] In this embodiment, the system performs a one-to-one correspondence between each stable global index in the sample sequence and each label number in the label sequence to form a working unit identifier. Specifically, for a combination with a stable global index of i and a label number of label_j, its working unit identifier is represented as (label_j, i). Taking a sample sequence containing 1000 images and a label sequence containing two label numbers, label_0 and label_1, as an example, the system forms a total of 2000 working unit identifiers. The set of working unit identifiers corresponding to label_0 is {(label_0, 0), (label_0, 1), …, (label_0, 999)}, and the set of working unit identifiers corresponding to label_1 is {(label_1, 0), (label_1, 1), …, (label_1, 999)}, thus obtaining the working unit identifier set.

[0042] It should be noted that the working unit identifiers formed by the same stable global index and different label number combinations are independent of each other. Taking the image with index 5 as an example, its working unit identifier is (label_0, 5) in the label_0 stage and (label_1, 5) in the label_1 stage. The two represent a complete judgment of the image in different annotation stages. The judgment of the image completed by the annotator in the label_0 stage does not affect the counting status of the same image in the label_1 stage. The effective workload of each stage is accumulated independently.

[0043] In this embodiment, the system initializes an independent counting cursor (counted_cursor) for each labeling stage based on the set of work unit identifiers. Taking label_0 and label_1 as examples, the system initializes counting cursors for both stages with an initial value of -1, indicating that no work units have been counted in each stage, thus obtaining the initial counting cursor for each labeling stage. The counting cursor monotonically increases with the writing of valid work events during subsequent labeling processes, and its current value is always equal to the maximum stable global index that the labeling stage has completed counting. For example, the current value of the counting cursor for the label_0 stage is 10, indicating that this stage has completed the counting of valid work for images with indices 0 to 10.

[0044] In this embodiment, the system performs a uniqueness check on combinations of identical label numbers and identical stable global indices within the work unit identifier set. Specifically, for any work unit identifier (label_j, i), the system checks whether this combination uniquely corresponds to a work unit throughout the entire annotation task lifecycle; that is, no two different work events simultaneously have the same label number and stable global index combination. If the uniqueness check passes, the system confirms the work unit identifier is valid and obtains a unique work unit identifier. During subsequent annotation processes, if annotators repeatedly submit the same image for the same annotation stage, the system determines that the work unit has been counted based on the counting cursor and no longer generates new work events, thus fundamentally avoiding the problem of repeated viewing and submission being repeatedly counted as valid workload.

[0045] In one embodiment, such as Figure 4 As shown, step S30 specifically includes the following steps: S31: Compare the image display duration of the current work unit in the work view with the preset minimum viewing time to obtain the viewing time determination result; S32: Based on the viewing time determination result, compare the stable global index of the current work unit with the counting cursor of the annotation stage to which the current work unit belongs. Only when the stable global index crosses the counting cursor for the first time and the annotator completes the current annotation stage determination, record the work unit as a work event and advance the counting cursor to obtain the work event and the updated counting cursor. S33: Based on the work events and the updated counting cursor, continuously aggregate the work events into the current batch. When the number of work events in the batch reaches the batch sealing condition, close the batch and obtain the batch to be inspected.

[0046] In this embodiment, when an annotator performs annotation operations on image samples in the working view, the system starts timing the moment the image is displayed in the working view and continuously records the image display duration. In step S31, when the annotator triggers the current annotation stage judgment operation, the system reads the current image display duration and compares it with the preset minimum viewing time to obtain the viewing time judgment result. If the image display duration is less than the preset minimum viewing time, the viewing time judgment result is not met, the system intercepts this operation, and does not enter the subsequent effective work judgment process; if the image display duration is not less than the preset minimum viewing time, the viewing time judgment result is met, and the system continues to execute step S32. When the annotator exits the working view midway, if the current work unit has not yet formed a work event, the system discards this timing record; after re-entering the working view, the system restarts timing, and the annotator needs to meet the viewing time requirement again and complete the judgment.

[0047] In step S32, assuming the time determination result is satisfied, the system reads the stable global index of the current work unit and the current value of the counting cursor for the annotation stage to which the current work unit belongs, and compares the two. When the stable global index is greater than the current value of the counting cursor, and the annotator has completed the determination of the current annotation stage, the system constructs a work event object, writing fields such as the current work unit identifier, completion timestamp, operation type, label status before and after the operation, and cursor status into the work event. Simultaneously, the counting cursor for the current annotation stage is advanced to the current stable global index, resulting in the work event and the updated counting cursor. When the stable global index is not greater than the current value of the counting cursor, the system determines that the work unit has completed counting, does not construct a work event, does not advance the cursor, and does not count it as valid workload. The counting cursor remains monotonically increasing throughout the entire annotation task lifecycle and does not regress due to undo operations, lookbacks, or administrator revisions.

[0048] The specific operation types for completing the current annotation stage judgment include: determining if the current label is matched and writing the current label; determining if the current label is not matched and moving to the next image; marking the current sample as unknown; and marking the current sample as delete. All four operations are considered as completing a valid judgment for the current annotation stage. Key presses unrelated to the current annotation stage, submissions that do not meet the minimum viewing time, and actions that only view but do not trigger a judgment do not meet the valid work judgment conditions, and the system does not construct a work event.

[0049] In step S33, the system writes each work event that passes the valid labor determination into the work event list of the current batch and maintains the count of work events within the batch in real time. After each work event is written, the system checks whether the number of work events within the batch has reached the batch sealing condition. If the sealing condition is reached, the system records the batch closing timestamp, switches the batch status from in progress to pending inspection, saves the continuous index range of each labeling stage within the batch and the cross-label formation order, and simultaneously starts a new batch to continue receiving subsequent work events, thus obtaining the batch pending inspection.

[0050] In one embodiment, after recording the work unit as a work event and advancing the counting cursor, and before continuously aggregating the work events into the current batch, the method further includes: The work events are persisted in the following order: first appended to the pending event log and written to disk, then the workload snapshot is updated, to obtain the pending event log and the workload snapshot. Based on the pending event log and workload snapshot, the workload snapshot is loaded first when the program restarts, and then the work events in the pending event log that have not yet been merged into the snapshot are replayed to obtain the set of events to be replayed. Based on the set of events to be replayed, the event identifier and counting cursor of each event to be replayed are double-checked. Only when the event identifier does not exist in the event record of the day and the stable global index crosses the counting cursor for the first time is it included in the effective workload, so as to obtain the counting cursor and effective workload consistent with those before the anomaly.

[0051] In this embodiment, after constructing the work event object and advancing the counting cursor, the system immediately performs storage operations according to a fixed persistence order. First, the system serializes the work events and writes them to the pending event log `pending_work_events.jsonl` and the daily event log `work_events.jsonl` via append-only writing. Immediately after writing, it calls the disk refresh interface to ensure that the event data is written to disk at the physical storage level. After disk write confirmation, the system then performs a workload snapshot update operation, persisting key fields such as the counting cursor, batch status, and effective workload for each annotation stage as temporary files, replacing the original snapshot file entirely, thus obtaining the pending event log and workload snapshot. This "write event log to disk first, then update snapshot" order ensures that in the event of an abnormal exit at any time, completed work events can be recovered through a replay mechanism after restarting, without being lost due to incomplete snapshot updates.

[0052] When the program restarts, the system first reads the most recent complete workload snapshot file from the local directory and restores the count cursors, batch status, and effective workload for each annotation stage based on the snapshot content. After the snapshot is loaded, the system scans the pending event log, reads the work events one by one in the order they were written, compares them with the set of event identifiers that have been merged in the event log for the day, and filters out the work events that have not yet been merged into the snapshot, forming the set of events to be replayed.

[0053] After obtaining the set of events to be replayed, the system performs a double check on each event in the set according to the order in which the events were written. The first check checks whether the event identifier already exists in the event record for the day. If it does, the event is considered to have been merged, and the system skips the event and continues processing the next one. After the first check passes, the system performs a second check, checking whether the stable global index corresponding to the event is greater than the current value of the counting cursor in the current annotation stage. If it is greater, it is considered the first time the cursor has been crossed, and the event is counted as valid workload, and the counting cursor of the corresponding annotation stage is advanced to the stable global index. If it is not greater, it is considered a historical work unit that has already been counted, and the event is skipped and not counted again. After all the events to be replayed have been processed, the system obtains the same counting cursor and valid workload as before the anomaly, the recovery process ends, and the program continues to run normally.

[0054] In one embodiment, the step of performing a speed check on the average activity time per unit workload within the batch based on the time records of work events within the batch to be inspected, to obtain the batch to be accepted, includes: The time intervals between adjacent work events within the batch to be inspected are classified. Time intervals not exceeding a first threshold are counted as activity time, time intervals exceeding a second threshold are counted as idle time, and records between the first and second thresholds are considered uncertain intervals. This yields the batch activity time statistics, where the first threshold is less than the second threshold. Based on the batch activity time statistics, the ratio of the total activity time in the batch to the number of work events in the batch is compared with the preset minimum activity time per work unit. Batches with a ratio lower than the preset minimum activity time per work unit are transferred to the rework process and their effective workload is not converted into working hours. Batches with a ratio not lower than the preset minimum activity time per work unit are transferred to the pending acceptance state, thus obtaining the pending acceptance batch.

[0055] In this embodiment, after the batch is closed, the system immediately initiates a speed check process based on the time records of the work events within the batch. The system extracts the completion timestamps of adjacent work events sequentially according to the writing order of the work events within the batch, and calculates the time interval between them. For each time interval, the system performs three-class processing based on a first threshold and a second threshold: if the time interval does not exceed the first threshold, it is determined that the annotator is in a continuous operation state, and the interval duration is added to the total batch activity time; if the time interval exceeds the second threshold, it is determined that the annotator is in an idle or interrupted state, and the interval is included in the idle time, not in the total activity time; if the time interval is between the first and second thresholds, the system cannot clearly determine whether it belongs to activity or idle, and the interval is recorded separately as an uncertain interval, neither included in the total activity time nor the idle time, and is retained for subsequent analysis. After traversing all adjacent work event pairs within the batch, the system summarizes the batch activity time statistics, which includes the total activity time, the total idle time, and the total uncertain interval.

[0056] After obtaining the batch activity time statistics, the system divides the total activity time within the batch by the number of work events within the batch to obtain the average activity time per unit of work, i.e., the average activity time corresponding to each work unit within the batch. The system compares this ratio with the preset minimum activity time per work unit. If the ratio is lower than the preset minimum activity time per work unit, the system determines that the batch has an abnormal speed, switches the batch status to failed, and initiates a rework process. The effective workload recorded in this batch is retained as a process record but is not included in the effective work time calculation. If the ratio is not lower than the preset minimum activity time per work unit, the system determines that the batch speed is normal, switches the batch status to pending acceptance, and obtains the batch awaiting acceptance.

[0057] In one embodiment, based on the batch activity time statistics, the ratio of the total activity time to the number of work events within the batch is compared with a preset minimum activity time per work unit. Batches with a ratio lower than the preset minimum activity time per work unit are transferred to a rework process, and their effective workload is not converted into working hours. Batches with a ratio not lower than the preset minimum activity time per work unit are transferred to a pending acceptance state. The batches to be accepted include: Obtain the average activity time per unit workload for historically accepted batches to obtain a historical speed sample sequence; Based on the historical velocity sample sequence, the mean and standard deviation of all velocity values ​​in the sample sequence are calculated to obtain the velocity baseline and fluctuation range; Based on the speed baseline and fluctuation range, the dynamic speed threshold is obtained by subtracting a preset multiple of the standard deviation from the speed baseline. Based on the dynamic speed threshold, the ratio of the total activity time in a batch to the number of work events in the batch is compared with the dynamic speed threshold. Batches with a ratio lower than the dynamic speed threshold are transferred to the rework process, and batches with a ratio not lower than the dynamic speed threshold are transferred to the pending acceptance state, thus obtaining the batches to be accepted.

[0058] In this embodiment, before performing the speed check, the system first reads the average activity time per unit workload of historically accepted batches from persistent storage. The speed values ​​of each historical batch are then arranged in chronological order of acceptance time, forming a historical speed sample sequence. This sequence dynamically expands with each acceptance event. If the initial number of historical samples is insufficient, the system reverts to using a preset fixed threshold for the speed check. Once the number of historical samples accumulates to the configured minimum sample size, it switches to a dynamic threshold calculation mode.

[0059] After obtaining the historical speed sample sequence, the system performs statistical calculations on all speed values ​​in the sequence, calculates the arithmetic mean of all speed values ​​as the speed baseline μ, and simultaneously calculates the standard deviation σ of all speed values ​​relative to the mean, thus obtaining the speed baseline and fluctuation range. The speed baseline μ reflects the typical annotation speed level of the annotator in historically accepted batches, and the standard deviation σ reflects the normal fluctuation range of this speed level.

[0060] After obtaining the velocity baseline and fluctuation range, the system calculates the dynamic velocity threshold by subtracting the product of a preset multiple k and the standard deviation σ from the velocity baseline μ, i.e., dynamic velocity threshold = μ - k × σ, where k is a configurable parameter. The dynamic velocity threshold is automatically adjusted as historical sample sequences are updated. When the overall historical velocity level of the annotators increases, the dynamic velocity threshold is adjusted upward accordingly; when the historical velocity fluctuation range expands, the dynamic velocity threshold is adjusted downward accordingly, thus adaptively reflecting the actual velocity distribution characteristics of the annotators.

[0061] After obtaining the dynamic speed threshold, the system divides the total activity time of the current batch under inspection by the number of work events within the batch to obtain the average activity time per unit workload of the current batch. This value is then compared with the dynamic speed threshold. If the average activity time per unit workload of the current batch is lower than the dynamic speed threshold, the system determines that the batch speed is abnormal and transfers the batch into the rework process, and its effective workload is not converted into working hours; if it is not lower than the dynamic speed threshold, the system transfers the batch to the pending acceptance state, resulting in a batch awaiting acceptance.

[0062] After the batch to be accepted is sampled and accepted by the administrator, the system adds the average activity time per unit workload of the batch to the historical speed sample sequence, and recalculates the mean and standard deviation based on the updated complete sequence to obtain the updated historical speed sample sequence and speed baseline.

[0063] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0064] In one embodiment, an image annotation workload statistics device is provided, which corresponds one-to-one with the image annotation workload statistics method described in the above embodiments. For example... Figure 5 As shown, the image annotation workload statistics device includes: The sequence construction module 501 is used to obtain the image sample set and the corresponding label configuration, assign a stable global index to each image in the image sample set, assign a label number to each label, and obtain a sample sequence and a label sequence with a stable global index. The unit identification module 502 is used to combine the tag number in the sample sequence and the tag sequence with the stable global index to obtain a unique working unit identifier; The labor judgment module 5033 is used to make valid labor judgments on the annotation operations of the personnel in the work view based on the unique work unit identifier, record the work units that pass the judgment as work events and aggregate them into the current batch, close the batch after the batch sealing conditions are met, and obtain the batch to be inspected. The speed check module 504 is used to check the average activity time of a unit workload in the batch based on the time record of the work events in the batch to be inspected, so as to obtain the batch to be accepted. The working hours conversion module 505 is used to convert the batch that has passed the acceptance according to the configured working hours based on the sampling acceptance results of the batch to be accepted by the administrator, so as to obtain the effective working hours of the marked personnel.

[0065] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0066] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0067] This application also provides a computer device, such as... Figure 6 As shown, the computer device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, it implements the steps in any of the above method embodiments, or when the processor executes the computer program, it implements the functions of each module / unit in the above device embodiments.

[0068] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0069] Those skilled in the art will understand that Figure 6 The computer device described is merely an example and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0070] The aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0071] The memory can be an internal storage unit of the computer device, such as a hard drive or RAM. The memory can also be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the computer device.

[0072] This application also provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0073] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0074] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0075] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0076] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0077] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for calculating the workload of image annotation, characterized in that, The method includes: Obtain the image sample set and the corresponding label configuration, assign a stable global index to each image in the image sample set, assign a label number to each label, and obtain a sample sequence and label sequence with a stable global index; The tag number in the sample sequence and tag sequence is combined with the stable global index to obtain a unique working unit identifier; Based on the unique work unit identifier, the annotation operation of the personnel in the work view is judged as valid labor. The work units that pass the judgment are recorded as work events and aggregated into the current batch. After the batch sealing conditions are met, the batch is closed and the batch to be inspected is obtained. Based on the time records of work events within the batch to be inspected, the average activity time of a unit workload within the batch is checked to obtain the batch to be accepted. Based on the sampling acceptance results of the batches to be accepted by the administrator, the batches that have passed the acceptance are converted according to the configured working hours to obtain the effective working hours of the marked personnel.

2. The image annotation workload statistics method according to claim 1, characterized in that, The process of obtaining the image sample set and corresponding label configuration, assigning a stable global index to each image in the image sample set, and assigning a label number to each label, to obtain a sample sequence and label sequence with a stable global index includes: According to the fixed storage order of the image sample set, each image in the image sample set is assigned a unique and stable global index that does not change with subsequent annotation operations, thus obtaining the sample sequence; According to the label configuration order of the labeling task, each label is assigned a unique label number to obtain a label sequence, where each label number corresponds to an independent labeling stage.

3. The image annotation workload statistics method according to claim 1, characterized in that, The step of combining the label numbers in the sample sequence and label sequence with the stable global index to obtain a unique working unit identifier includes: The stable global index in the sample sequence is matched one-to-one with the label number in the label sequence. The same stable global index is combined with different label numbers to form independent working unit identifiers, thus obtaining a set of working unit identifiers. Based on the set of work unit identifiers, an independent counting cursor is initialized for each labeling stage corresponding to each label number, so as to obtain the initial counting cursor for each labeling stage. The counting cursor records the maximum stable global index that the labeling stage has completed counting. Based on the set of work unit identifiers and the initial counting cursors of each labeling stage, a uniqueness check is performed on combinations of the same label number and the same stable global index to obtain a unique work unit identifier.

4. The image annotation workload statistics method according to claim 2, characterized in that, The process involves determining the validity of annotation operations performed by personnel in the work view based on the unique work unit identifier. Work units that pass the determination are recorded as work events and aggregated into the current batch. Once the batch sealing conditions are met, the batch is closed, resulting in the following batches to be inspected: The image display duration of the current work unit in the work view is compared with the preset minimum viewing time to obtain the viewing time determination result; Based on the viewing time determination result, the stable global index of the current work unit is compared with the counting cursor of the annotation stage to which the current work unit belongs. Only when the stable global index crosses the counting cursor for the first time and the annotator completes the current annotation stage determination, the work unit is recorded as a work event and the counting cursor is advanced to obtain the work event and the updated counting cursor. Based on the work events and the updated counting cursor, the work events are continuously aggregated into the current batch. When the number of work events in the batch reaches the batch sealing condition, the batch is closed, and the batch to be inspected is obtained.

5. The image annotation workload statistics method according to claim 4, characterized in that, After recording the work unit as a work event and advancing the counting cursor, and before continuously aggregating work events into the current batch, the process includes: The work events are persisted in the following order: first appended to the pending event log and written to disk, then the workload snapshot is updated, to obtain the pending event log and the workload snapshot. Based on the pending event log and workload snapshot, the workload snapshot is loaded first when the program restarts, and then the work events in the pending event log that have not yet been merged into the snapshot are replayed to obtain the set of events to be replayed. Based on the set of events to be replayed, the event identifier and counting cursor of each event to be replayed are double-checked. Only when the event identifier does not exist in the event record of the day and the stable global index crosses the counting cursor for the first time is it included in the effective workload, so as to obtain the counting cursor and effective workload consistent with those before the anomaly.

6. The image annotation workload statistics method according to claim 1, characterized in that, The step involves performing a speed check on the average activity time per unit workload within the batch based on the time records of work events within the batch to be inspected, resulting in batches to be accepted, including: The time intervals between adjacent work events within the batch to be inspected are classified. Time intervals not exceeding a first threshold are counted as activity time, time intervals exceeding a second threshold are counted as idle time, and records between the first and second thresholds are considered uncertain intervals. This yields the batch activity time statistics, where the first threshold is less than the second threshold. Based on the batch activity time statistics, the ratio of the total activity time in the batch to the number of work events in the batch is compared with the preset minimum activity time per work unit. Batches with a ratio lower than the preset minimum activity time per work unit are transferred to the rework process and their effective workload is not converted into working hours. Batches with a ratio not lower than the preset minimum activity time per work unit are transferred to the pending acceptance state, thus obtaining the pending acceptance batch.

7. An image annotation workload statistics device, characterized in that, include: The sequence construction module is used to obtain the image sample set and the corresponding label configuration, assign a stable global index to each image in the image sample set, assign a label number to each label, and obtain a sample sequence and a label sequence with a stable global index. The unit identification module is used to combine the tag number in the sample sequence and the tag sequence with the stable global index to obtain a unique working unit identifier; The labor judgment module is used to make valid labor judgments on the annotation operations of the personnel in the work view based on the unique work unit identifier, record the work units that pass the judgment as work events and aggregate them into the current batch, close the batch after the batch sealing conditions are met, and obtain the batch to be inspected. The speed check module is used to check the average activity time of a unit workload within the batch based on the time records of work events within the batch to be inspected, and to obtain the batch to be accepted. The working hours conversion module is used to convert the batches that have passed the acceptance inspection according to the configured working hours based on the sampling acceptance results of the batches to be accepted by the administrator, so as to obtain the effective working hours of the marked personnel.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the image annotation workload statistics method as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the image annotation workload statistics method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, enables the implementation of the steps of the image annotation workload statistics method as described in any one of claims 1 to 6.