SYSTEM AND METHOD FOR AUDITING IMAGE INSPECTION QUALITY - Patent application
By calculating the minimum number of tests and confidence intervals for image inspection, the method addresses the limitations of existing quality assurance methods, ensuring accurate dataset quality and improving AI model performance.
Patent Information
- Application Number
- JP2023180339
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2023-03-30
- Filing Date
- 2023-10-19
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-10-19
AI Technical Summary
Existing methods for auditing image inspection quality in annotated datasets for AI training and evaluation are limited by the reliance on error rates, which do not provide a clear indication of the actual quality of the dataset, leading to potential misconceptions about dataset quality and negative impacts on AI model performance.
A method and apparatus that calculate the minimum number of tests required to achieve a specified confidence interval and target error rate, ensuring accurate dataset inspection by determining the number of frames to inspect and providing indications of insufficient quality or error rates exceeding the target, thereby improving the accuracy of AI model training and evaluation.
Ensures accurate dataset quality assurance by confirming the inspection quality to the extent specified by the end user, guaranteeing more reliable training and evaluation of AI models.
Smart Images

Figure 0007577814000004 
Figure 0007577814000005 
Figure 0007577814000006
Abstract
Description
[Technical field]
[0001] Systems and methods consistent with exemplary embodiments of the present disclosure relate to providing a method for auditing image inspection quality for annotated image datasets. [Background technology]
[0002] Annotated image datasets are used for various artificial intelligence (AI) processes, e.g., as training datasets for training computer vision AI models, evaluation datasets for evaluating the performance of AI models, etc. In the related art, annotated image datasets provided by image annotators (e.g., third-party annotation vendors) undergo an annotation quality assurance process before being released to end users or customers (e.g., to train or evaluate their AI models). The annotation quality assurance process identifies discrepancies between the customer or end user's expectations and the vendor / annotator's understanding, and identifies annotation errors due to poor or sloppy annotations, human error, etc.
[0003] In the related art, a common metric / indicator of the overall quality for annotated image datasets is the error rate of the sampled population. For this purpose, randomly selected sample images (at different times) are selected and, if possible, inspected.
[0004] For example, if there are 100 frames to be examined (each taken at a different time point), 10 frames are randomly selected as a sample population. If 2 / 10 of the sample frames are found to have errors, then the overall quality of the annotated image dataset is judged to have an overall error rate of 20%.
[0005] However, the degree of statistical data obtained by related art methods that utilize only error rates may be limited. For example, the actual quality of the population may not be easily indicated or demonstrated to the data user apart from the mere error rate. In particular, it may not be clear what the amount of annotations to be inspected should be to obtain a reliable measure of quality. Furthermore, such methods may not be clear on what objects should be inspected, resulting in rare objects being excluded from inspection. Thus, such problems may lead to misinterpretation of the actual quality of the dataset during the dataset audit process, resulting in cases where the dataset falls below the pre-specified quality required by the end user / customer. This inevitably causes a negative impact on the performance of the AI model and a loss of trust from the end user / customer.
[0006] Therefore, there is a need for methods that can ensure more accurate data sets, thereby improving the training, evaluation and performance of AI models. Summary of the Invention
[0007] According to one or more exemplary embodiments, an apparatus and method are provided for auditing image inspection quality of an annotated image dataset. In particular, the apparatus and method according to the exemplary embodiments calculates the minimum number of inspections required to meet a target error rate, along with confidence intervals and interval widths, so that the inspection quality can be confirmed to a degree specified by the end user / customer. Thus, the accuracy of the dataset can be ensured, which leads to more accurate training / evaluation / performance of the AI model.
[0008] According to an embodiment, a method is provided for auditing inspection of image annotation quality in an annotated image dataset, the method includes obtaining the annotated image dataset, determining a minimum number of inspections of the annotated image dataset based on a predetermined confidence interval, a predetermined target error rate, and a predetermined interval width, selecting a number of frames of the annotated image dataset for inspection based on the minimum number of inspections, and outputting the selected number of frames of the annotated image dataset for inspection.
[0009] According to an embodiment, outputting the selected plurality of frames for inspection includes comparing the current number of inspections to a minimum number of inspections, outputting an indication of insufficient inspection based on determining that the current number of inspections is less than the minimum number of inspections, and continuing to output the selected plurality of frames until the number of inspected frames is equal to or greater than the minimum number of inspections.
[0010] According to an embodiment, the method further includes determining a number of errors based on the inspection and calculating a sample error rate based on the number of errors. The method further includes determining whether the sample error rate exceeds a target error rate based on the sample error rate, and outputting an indication of poor quality based on a determination that the sample error rate exceeds the target error rate.
[0011] According to an embodiment, outputting an indication of poor quality further includes providing a message to an operator that the quality needs to be improved.
[0012] According to an embodiment, the predetermined confidence interval is one of 99%, 95% or 90%. The predetermined confidence interval is based on one of the F-distribution or the T-distribution.
[0013] According to an embodiment, an apparatus for auditing inspection of image annotation quality in an annotated image dataset includes at least one memory storing computer-readable instructions and at least one processor, wherein the at least one processor is configured to execute the computer-readable instructions to obtain the annotated image dataset, determine a minimum number of inspections of the annotated image dataset based on a predetermined confidence interval, a predetermined target error rate, and a predetermined interval width, select a number of frames of the annotated image dataset for inspection based on the minimum number of inspections, and output the selected number of frames of the annotated image dataset for inspection.
[0014] The at least one processor is further configured to execute computer readable instructions to output the selected plurality of frames for inspection by comparing the current number of inspections to the minimum number of inspections and, based on determining that the current number of inspections is less than the minimum number of inspections, outputting an indication of insufficient inspection and continuing to output the selected plurality of frames until the number of inspected frames is greater than or equal to the minimum number of inspections.
[0015] The at least one processor is further configured to execute computer readable instructions to determine a number of errors based on the inspection and to calculate an error rate for the sample based on the number of errors.
[0016] The at least one processor is further configured to execute computer readable instructions to determine, based on the error rate of the samples, whether the error rate of the samples exceeds a target error rate, and to output an indication of poor quality based on a determination that the error rate of the samples exceeds the target error rate.
[0017] The at least one processor is further configured to execute computer readable instructions to output an indication of poor quality by providing a message to an operator that quality needs to be improved.
[0018] Additional aspects will be set forth in part in the description that follows, and in part will be obvious from the description, or may be realized by practice of illustrative embodiments of the present disclosure. [Brief description of the drawings]
[0019] [Figure 1] FIG. 1 is a diagram of example components of an apparatus according to an example embodiment. [Diagram 2] FIG. 2 is a flowchart illustrating an annotated image inspection audit process in accordance with one or more exemplary embodiments. [Diagram 3] FIG. 3 is a flowchart illustrating an annotated imaging examination result evaluation process in accordance with one or more exemplary embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] Features, aspects, and advantages of certain exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements and in which:
[0021] The following detailed description of the exemplary embodiments refers to the accompanying drawings. The present disclosure provides illustration and description, but is not intended to be exhaustive or to limit the exemplary embodiment(s) to the precise form disclosed. Modifications and variations are possible in light of the disclosure or may be acquired from practice of the exemplary embodiment(s). Moreover, one or more features or components of one exemplary embodiment may be incorporated in or combined with another exemplary embodiment (or one or more features of another exemplary embodiment). In addition, it is understood that in the flowcharts and descriptions of operations provided herein, one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and the order of one or more operations may be changed.
[0022] It will be apparent that the exemplary embodiments of the systems and / or methods and / or non-transitory computer-readable storage media described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement the systems and / or methods does not limit one or more exemplary embodiments. Thus, the operation and behavior of the systems and / or methods and / or non-transitory computer-readable storage media are described herein without reference to any specific software code. It is understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.
[0023] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible example embodiments. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible example embodiments includes each dependent claim in combination with every other claim in the claim set.
[0024] No element, act, or instruction used herein should be construed as critical or essential unless expressly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." When only one item is intended, the term "one" or similar language is used. Also, as used herein, the terms "having," "including," or similar terms are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated to the contrary. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood as including only A, only B, or both A and B.
[0025] 1 is a diagram of example components of an image annotation auditor 100. As shown in FIG. 1, the image annotation auditor 100 includes a bus 110, a processor 120, a memory 130, a storage component 140, an input component 150, an output component 160, and a communication interface 170.
[0026] The bus 110 includes components that enable communication between components of the image annotation auditor 100. The processor 120 is implemented in hardware, firmware, or a combination of hardware and software. The processor 120 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In one or more exemplary embodiments, the processor 120 includes one or more processors that are programmable to perform functions. The memory 130 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions used by the processor 120.
[0027] The storage component 140 stores information and / or software related to the operation and use of the image annotation audit apparatus 100. For example, the storage component 140 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium along with a corresponding drive. The input component 150 includes components that allow the image annotation audit apparatus 100 to receive information, for example, via a user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and / or a microphone). Additionally or alternatively, the input component 150 includes sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). The output component 160 includes components that provide output information from the image annotation audit apparatus 100 (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs)).
[0028] The communication interface 170 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the image annotation auditing apparatus 100 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 170 enables the image annotation auditing apparatus 100 to receive information from and / or provide information to another device. For example, the communication interface 170 includes, but is not limited to, an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, or equivalents thereof.
[0029] The image annotation auditor 100 performs one or more exemplary processes described herein. According to one or more exemplary embodiments, the image annotation auditor 100 performs these processes in response to the processor 120 executing software instructions stored by a non-transitory computer-readable medium, such as the memory 130 and / or the storage component 140. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space across multiple physical storage devices.
[0030] The software instructions may be read into memory 130 and / or storage component 140 from another computer-readable medium or from another device via communication interface 170. The software instructions stored in memory 130 and / or storage component 140, when executed, cause processor 120 to perform one or more operations described herein.
[0031] Additionally or alternatively, hardwired circuitry can be used in place of or in combination with software instructions to perform one or more of the operations described herein, and thus, one or more exemplary embodiments described herein are not limited to any specific combination of hardware circuitry and software.
[0032] The number and arrangement of components shown in Figure 1 are provided as an example. In practice, image annotation auditor 100 may include additional components, fewer components, different components, or components arranged differently than those shown in Figure 1. Additionally or alternatively, a set of components (e.g., one or more components) of image annotation auditor 100 may perform one or more functions described as being performed by another set of components of image annotation auditor 100.
[0033] FIG. 2 is a diagram illustrating an annotated image inspection quality audit process 200 in accordance with one or more exemplary embodiments.
[0034] 2, in operation S210, an annotated image dataset is obtained. It should be noted that according to some embodiments, the annotated image dataset may already be provided. The annotated image dataset is obtained using any suitable method according to an embodiment.
[0035] In act S220, a minimum number of inspections of the annotated image dataset acquired in act S210 is determined. According to an embodiment, the minimum number of inspections is determined based on a confidence interval, a target error rate, and an interval width.
[0036] According to embodiments, the confidence interval is calculated / predetermined before the test. In particular, the confidence interval should be understood as an interval within which the value of the parameter in the dataset falls with a specified probability. According to some embodiments, the confidence interval is 95%, but it should be understood that other values (99%, 90%, etc.) may be used according to embodiments.
[0037] It should be understood that the above confidence intervals can be based on a standard T-distribution or an F-distribution. For example, one skilled in the art can appreciate that a standard T-distribution is more useful in situations where there is a larger sample size, and an F-distribution is more useful in situations where there is a smaller sample size or where the expected error rate is unknown.
[0038] The desired standard error rate may be a value requested by an end user (e.g., a customer) or may be set by default. This is typically 5% in instances where high accuracy is not required, and 1% in instances where high accuracy is required, although it should be understood that these are merely indicative values and any target error rate may be specified.
[0039] According to an embodiment, the (confidence) interval width is calculated / predetermined before the test. In particular, the interval width is defined as the size between the upper and lower limits of a given class in the distribution. This is typically given as a percentage value of uncertainty (e.g. ±2.5%, ±5%, etc.), but it should be understood that other formats for expressing the interval width may be given.
[0040] Based on the confidence interval, the target error rate and the interval width, the minimum number of tests is easily determined. In particular, according to one embodiment, calculations based on the T-distribution can be used.
[0041] An example of the calculations used to obtain confidence intervals in the case of the T-distribution is given below based on equation (1).
[0042]
number
[0043] where φ is the target error rate, p is the error rate for a given population (i.e., the expected chance that an error is present in the data), n is the total number of samples, α is the significance level (i.e., a value that indicates whether the error is statistically significant or not, typically 5%), and g(n-1,1-α) is the binomial probability distribution function from n-1 to 1-α. An example of the calculation used to obtain the confidence interval width β in the T-distribution is given by equation (2) below.
[0044]
number
[0045] An example of the calculation used to obtain the total number of samples n' in the T-distribution is given by equation (3) below.
[0046]
number
[0047] Nevertheless, it should be appreciated that any suitable method for calculating the confidence intervals, confidence interval widths, and target numbers of samples may be used in accordance with the present embodiment.
[0048] Based on the above embodiment, those skilled in the art can also understand that the confidence interval and interval width are determined based on the parameters of the significance level α and the error rate p of a given population, and vice versa. In any case, the above embodiment illustrates how the target number / minimum number of samples to be tested (e.g., n') is determined based on the confidence interval width and confidence interval based on the T-distribution. It should be understood that similar methods and calculations can be used when using the F-distribution or other types of frequency distributions.
[0049] 2, in operation S230, a number of frames of the dataset are selected for examination based on the minimum number of examinations determined in operation S220. According to one embodiment, a population is segmented from the original dataset based on a target / minimum number of samples to be examined.
[0050] In operation S240, the frames selected in operation S230 are output for inspection. According to an embodiment, inspection proceeds to inspect each selected frame for errors based on a target / minimum number of samples to be inspected. According to one embodiment, this is done on a frame-by-frame basis. The inspection tracks the current number of frames actually inspected against the target / minimum number of samples. If it is determined that the number of samples of the inspected frame is less than the required target / minimum number of samples, the process outputs an indication of insufficient inspection (e.g., a message to an operator / inspector). The inspection then continues (e.g., frames continue to be output) until the number of inspected frames is equal to or greater than the minimum number of inspections.
[0051] 3 illustrates an example flowchart illustrating an annotated imaging inspection result evaluation process 300 according to one or more example embodiments. The annotated imaging inspection result evaluation process 300 may be performed after the steps illustrated in FIG. 2 or may be performed separately from the steps illustrated in FIG.
[0052] In act S310, the number of errors is determined, which may be provided based on summing the number of errors from the examination in act S240 of Figure 2, or through some other process according to some embodiments.
[0053] Based on the number of errors, an error rate for the samples is determined in operation 320. For example, this is obtained by comparing the number of errors to the number of samples examined.
[0054] In operation S330, it is determined whether the sample error rate exceeds a target error rate (e.g., the target error rate used in operation S220 with respect to FIG. 2 above). If the sample error rate is higher than the target error rate, this indicates that the quality of the data set is below the end user / customer's desired target error rate.
[0055] In operation S340, based on the determination of operation S330, a message is output to a user. For example, the message may indicate that the data set is of low quality. According to some embodiments, the message further includes an indication that some corrective action should be taken by the operator / inspector. For example, this is done by at least sending a message to the operator / inspector that the image annotation should be redone and re-audited at a later date, although it should be understood that any suitable corrective action may be taken nonetheless. It should be understood that the indication does not necessarily have to be in the form of a message, but may be some simple visual indicator (e.g., color).
[0056] Based on the above embodiment, the test quality can be verified to the degree specified by the end user / customer by calculating the minimum number of tests required to meet the target error rate, along with the confidence interval and interval width. Thus, the accuracy of the data set can be guaranteed, which leads to more accurate training / evaluation / performance of the AI model.
[0057] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the example embodiment or embodiments to the precise forms disclosed. Modifications and variations are possible in light of the disclosure or may be acquired from practice of the example embodiment or embodiments.
[0058] One or more exemplary embodiments relate to systems, methods and / or computer-readable media at any possible level of technical detail integration. Furthermore, one or more of the above-described components may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). A computer-readable medium includes a non-transitory computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform operations.
[0059] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge-in-groove structures having instructions recorded thereon, and any suitable combination thereof. A computer readable storage medium, as used herein, should not be construed as being a transitory signal itself, such as an electric signal transmitted over a wire, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electric signal transmitted over a wire.
[0060] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium into each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.
[0061] The computer readable program code / instructions for performing the operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages such as Smalltalk, C++, and procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or a connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In one or more exemplary embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuit to perform an aspect or operation.
[0062] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, generate means for performing the function / acts specified in the block or blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored in a computer readable storage medium capable of directing a computer, programmable data processing apparatus, and / or other apparatus to function in a particular manner, such that the computer readable storage medium having stored thereon includes an article of manufacture including instructions implementing an aspect of the function / acts specified in the block or blocks of the flowcharts and / or block diagrams.
[0063] The computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable apparatus, or other device to execute a series of operational steps to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the function / act specified in the block or blocks of the flowcharts and / or block diagrams.
[0064] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible example implementations of the systems, methods, and computer-readable media according to one or more exemplary embodiments. In this regard, each block in the flowcharts or block diagrams represents a microservice, a module, a segment, or a portion of instructions, which includes one or more executable instructions for implementing the specified logical function. The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks than those depicted in the figures. In one or more alternative exemplary embodiments, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed concurrently or substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or acts, or executes a combination of special-purpose hardware and computer instructions.
[0065] It will be apparent that the systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement the systems and / or methods does not limit one or more of the exemplary embodiments. Thus, although the operation and behavior of the systems and / or methods are described herein without reference to specific software code, it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.
Claims
1. 1. A method for auditing a check of image annotation quality in an annotated image dataset, the method comprising: Obtaining an annotated image dataset; determining a minimum number of inspections of the annotated image dataset based on a predetermined confidence interval, a predetermined target error rate, and a predetermined interval width; selecting a number of frames of the annotated image dataset for inspection based on the minimum number of inspections; and outputting the selected plurality of frames of the annotated image dataset for inspection; and Including, The method, wherein the confidence interval is an interval within which the value of a parameter in a dataset falls with a specified probability, and the interval width is the size between the upper and lower limits of a given class in a distribution.
2. Outputting the selected plurality of frames for inspection includes: comparing the current number of tests to said minimum number of tests; based on determining that the current number of tests is less than the minimum number of tests; outputting an indication of a missing test; continuing to output the selected plurality of frames until a number of frames examined is equal to or greater than the minimum number of examinations; The method of claim 1 , comprising:
3. determining a number of errors based on said examination; calculating an error rate for the sample based on the number of errors; The method of claim 1 or 2, further comprising:
4. determining, based on the error rate of the sample, whether the error rate of the sample exceeds the target error rate; outputting an indication of poor quality based on a determination that the error rate of the sample exceeds the target error rate; The method of claim 3 further comprising:
5. outputting the indication of quality defect The method of claim 4 , further comprising providing a message to the operator that quality needs to be improved.
6. The method of claim 1 or 2, wherein the predetermined confidence interval is one of 99%, 95% or 90%.
7. The method of claim 1 or 2, wherein the predetermined confidence interval is based on one of an F-distribution or a T-distribution.
8. 1. An apparatus for auditing a check of image annotation quality in an annotated image dataset, comprising: at least one memory storing computer readable instructions; At least one processor Equipped with The at least one processor executes the computer readable instructions to: Obtain an annotated image dataset, determining a minimum number of inspections of the annotated image dataset based on a predetermined confidence interval, a predetermined target error rate, and a predetermined interval width; selecting a number of frames of the annotated image dataset for inspection based on the minimum number of inspections; outputting the selected frames of the annotated image dataset for inspection; The confidence interval is an interval within which a value of a parameter in a data set falls with a specified probability, and the interval width is the size between the upper and lower limits of a given class in a distribution.
9. The at least one processor further executes the computer readable instructions to: comparing the current number of tests to said minimum number of tests; based on determining that the current number of tests is less than the minimum number of tests; Outputs a display of insufficient inspection, continuing to output the selected plurality of frames until the number of frames examined is equal to or greater than the minimum number of examinations.
9. The apparatus of claim 8, configured to output the selected plurality of frames for inspection by:
10. The at least one processor further executes the computer readable instructions to: determining a number of errors based on said examination; 10. Apparatus according to claim 8 or 9, configured to calculate a sample error rate based on the number of errors.
11. The at least one processor further executes the computer readable instructions to: determining whether the error rate of the sample exceeds the target error rate based on the error rate of the sample; The apparatus of claim 10 , configured to output an indication of poor quality based on a determination that an error rate for the samples exceeds the target error rate.
12. The at least one processor further executes the computer readable instructions to: The apparatus of claim 11 , configured to output the indication of poor quality by providing a message to an operator that quality needs to be improved.
13. 10. Apparatus according to claim 8 or 9, wherein the predetermined confidence interval is one of 99%, 95% or 90%.
14. 10. The apparatus of claim 8 or 9, wherein the predetermined confidence interval is based on one of an F-distribution or a T-distribution.
15. A non-transitory computer-readable storage medium having instructions executable by at least one processor, comprising: The instructions cause the processor to: Obtaining an annotated image dataset; determining a minimum number of inspections of the annotated image dataset based on a predetermined confidence interval, a predetermined target error rate, and a predetermined interval width; selecting a number of frames of the annotated image dataset for inspection based on the minimum number of inspections; and outputting the selected plurality of frames of the annotated image dataset for inspection; and carrying out a method comprising: The confidence interval is an interval within which a value of a parameter in a dataset falls with a specified probability, and the interval width is the size between an upper and lower limit of a given class in a distribution.
16. Outputting the selected plurality of frames for inspection includes: comparing the current number of tests to said minimum number of tests; based on determining that the current number of tests is less than the minimum number of tests; outputting an indication of a missing test; continuing to output the selected plurality of frames until a number of frames examined is equal to or greater than the minimum number of examinations; 20. The non-transitory computer readable storage medium of claim 15, comprising:
17. The method comprises: determining a number of errors based on said examination; calculating an error rate for the sample based on the number of errors; 17. The non-transitory computer-readable storage medium of claim 15 or 16, further comprising:
18. The method comprises: determining, based on the error rate of the sample, whether the error rate of the sample exceeds the target error rate; outputting an indication of poor quality based on a determination that the error rate of the sample exceeds the target error rate; 20. The non-transitory computer-readable storage medium of claim 17, further comprising:
19. outputting the indication of quality defect 20. The non-transitory computer-readable medium of claim 18, further comprising providing a message to an operator that quality needs to be improved.
20. 17. The non-transitory computer readable storage medium of claim 15 or 16, wherein the predetermined confidence interval is one of 99%, 95% or 90%.
Citation Information
Patent Citations
Method and device for controlling multi-person cooperation image annotation quality
CN110991486A
Image inspection support apparatus and method
JP2019215698A
Method for sample analysis, electronic device, storage medium, and program product
JP2023042582A
