Image determination method and device, electronic equipment and computer program product

Abstract

The invention provides an image determination method and device, electronic equipment and a computer program product, and relates to the technical field of machine vision. The method comprises the steps of obtaining a to-be-labeled image set, wherein the to-be-labeled image set comprises a plurality of target labeled images; receiving an image annotation result of the annotation personnel for the target annotation image, wherein the image annotation result is obtained by carrying out image annotation by the annotation personnel based on an image annotation standard generated after the annotation training operation; obtaining a reference annotation image, and performing cross verification on the image annotation result based on the reference annotation image to obtain an annotation verification result; and based on the annotation verification result, determining a target quality image from the plurality of target annotation images. According to the method and the device, through a multi-round joint evaluation and cross validation mechanism, the annotation consistency is remarkably improved, the problem that the quality of the annotated high-quality image is unstable can be effectively solved, and meanwhile, the construction efficiency of the high-quality image set can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of machine vision technology, and in particular to an image determination method, an image determination device, an electronic device, and a computer program product. Background Technology

[0002] In computer vision and deep learning, building high-quality image datasets is a fundamental step in model training. With the rapid development of artificial intelligence, higher demands are being placed on the accuracy and efficiency of image annotation. However, current image annotation processes still face numerous challenges. First, manual annotation is easily influenced by subjective factors; different annotators may have significantly different standards for judging image quality, making it difficult to guarantee data consistency. Second, traditional annotation methods are inefficient and cannot meet the needs of building large-scale datasets. Furthermore, existing automated annotation tools perform poorly when handling complex scenes, failing to accurately identify defects such as compression artifacts, blurring, or noise in images. These problems severely restrict the efficiency of building high-quality datasets, thereby affecting the model performance of downstream tasks. Summary of the Invention

[0003] This disclosure provides an image determination method, an image determination apparatus, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problems of low efficiency in constructing high-quality image sets and unstable quality of annotated high-quality images in related technologies. The technical solution of this disclosure is as follows:

[0004] According to a first aspect of the present disclosure, an image determination method is provided, comprising: acquiring a set of images to be labeled, the set of images to be labeled including a plurality of target labeled images; receiving image labeling results of the target labeled images by annotators, the image labeling results being obtained by the annotators based on image labeling standards generated after annotation training operations; acquiring a reference labeled image; performing cross-validation on the image labeling results based on the reference labeled image to obtain a labeling validation result; and determining a target quality image from the plurality of target labeled images based on the labeling validation result.

[0005] In one exemplary embodiment of this disclosure, the annotation training operation is performed through the following steps: obtaining an initial image set, the initial image set including multiple initial images; performing image preprocessing on the initial images to obtain test annotation images; determining the test annotation results of the annotators for the test annotation images; and performing joint training on the test annotation results until the test annotation results meet the image annotation standard, the image annotation standard being that the annotation indicators of the test annotation results meet the indicator threshold.

[0006] In one exemplary embodiment of this disclosure, determining the test annotation result of the annotator for the test annotation image includes: receiving the test annotation result of the annotator for the test annotation image; determining the test annotation sequence number corresponding to the test annotation result and the personnel identifier corresponding to the annotator; constructing a test annotation mapping table based on the personnel identifier, the test annotation sequence number and the annotation date; and storing the test annotation result based on the test annotation mapping table.

[0007] In one exemplary embodiment of this disclosure, receiving the test annotation results of the annotator for the test annotation image includes: determining a pre-configured image annotation tool, the image annotation tool including an annotation status display area, an image display area, and an operation prompt area; receiving the test annotation operation of the annotator for the test annotation image based on the image annotation tool; and determining the test annotation result based on the test annotation operation.

[0008] In one exemplary embodiment of this disclosure, the step of cross-validating the image annotation results based on the reference annotation image to obtain annotation verification results includes: constructing a verification image set based on a pre-configured reference image proportion, the reference annotation image, and the target annotation image; determining the image quality score corresponding to the verification image in the verification image set; and performing cross-validation based on the image quality score and the image annotation results to obtain the annotation verification results.

[0009] In one exemplary embodiment of this disclosure, determining the image quality score corresponding to the verification image in the verification image set includes: determining the quality evaluation index value corresponding to the verification image, wherein the quality evaluation index value includes the artifact detection result and local sharpness of the verification image; and determining the image quality score based on the artifact detection result and the local sharpness.

[0010] In one exemplary embodiment of this disclosure, determining a target quality image from the plurality of target labeled images based on the annotation verification result includes: selecting a target quality image from the plurality of target labeled images when the annotation verification result is greater than or equal to a preset threshold, wherein the target quality image is an image whose image annotation result is a specified annotation identifier; constructing a target image set based on the plurality of target quality images, wherein the target image set is used for model training; and restarting the annotation training operation when the annotation verification result is less than the preset threshold.

[0011] According to a second aspect of the present disclosure, an image determination apparatus is provided, comprising: an image set acquisition module, configured to acquire an image set to be labeled, the image set to be labeled including a plurality of target labeled images; an annotation result receiving module, configured to receive image annotation results of the target labeled images by annotators, the image annotation results being obtained by the annotators through image annotation standards generated after annotation training operations; an annotation verification module, configured to acquire a reference labeled image, and perform cross-validation on the image annotation results based on the reference labeled image to obtain an annotation verification result; and a target image determination module, configured to determine a target quality image from the plurality of target labeled images based on the annotation verification result.

[0012] In one exemplary embodiment of this disclosure, the image determination device includes an annotation training module, configured to: acquire an initial image set, the initial image set including multiple initial images; perform image preprocessing on the initial images to obtain test annotation images; determine the test annotation results of the annotators for the test annotation images; and perform joint training on the test annotation results until the test annotation results meet the image annotation standard, the image annotation standard being that the annotation indicators of the test annotation results meet the indicator threshold.

[0013] In one exemplary embodiment of this disclosure, the annotation training module includes a test annotation unit, configured to: receive test annotation results from the annotator for the test annotation image; determine the test annotation sequence number corresponding to the test annotation result and the personnel identifier corresponding to the annotator; construct a test annotation mapping table based on the personnel identifier, the test annotation sequence number and the annotation date; and store the test annotation results based on the test annotation mapping table.

[0014] In one exemplary embodiment of this disclosure, the test annotation unit includes an annotation result receiving subunit, configured to: determine a pre-configured image annotation tool, the image annotation tool including an annotation status display area, an image display area, and an operation prompt area; receive test annotation operations performed by the annotator on the test annotation image based on the image annotation tool; and determine the test annotation result based on the test annotation operations.

[0015] In one exemplary embodiment of this disclosure, the annotation verification module includes an annotation verification unit, configured to: construct a verification image set based on a pre-configured reference image proportion, the reference annotation image, and the target annotation image; determine the image quality score corresponding to the verification image in the verification image set; and perform cross-validation based on the image quality score and the image annotation result to obtain the annotation verification result.

[0016] In one exemplary embodiment of this disclosure, the annotation verification unit includes a quality score determination subunit, configured to: determine the quality evaluation index value corresponding to the verification image, wherein the quality evaluation index value includes the artifact detection result and local sharpness of the verification image; and determine the image quality score based on the artifact detection result and the local sharpness.

[0017] In one exemplary embodiment of this disclosure, the target image determination module includes a target image determination unit, configured to: select a target quality image from the plurality of target labeled images when the annotation verification result is greater than or equal to a preset threshold, wherein the target quality image is an image whose image annotation result is a specified annotation identifier; construct a target image set based on the plurality of target quality images, wherein the target image set is used for model training; and restart the annotation training operation when the annotation verification result is less than the preset threshold.

[0018] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the image determination method described in any of the preceding claims.

[0019] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image determination method described in any of the preceding claims.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image determination method described in any of the preceding claims.

[0021] The technical solutions provided by the embodiments of this disclosure offer at least the following beneficial effects: On the one hand, by performing image annotation operations based on image annotation standards generated from annotation training operations, the annotation results of the target annotated images have a unified standard, which can effectively avoid annotation deviations caused by subjective differences among annotators and significantly improve annotation quality. On the other hand, a scheme for determining the target quality image based on the image annotation results is provided, which can effectively improve annotation efficiency and fully meet the training needs of various computer vision models.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0024] Figure 1 This is a flowchart illustrating an image determination method according to an exemplary embodiment.

[0025] Figure 2 This is a flowchart illustrating, according to an exemplary embodiment, the determination of a target quality image through joint training operations and sub-packet annotation operations.

[0026] Figure 3 This is a flowchart illustrating quality management of a target-annotated image according to an exemplary embodiment.

[0027] Figure 4 This is a block diagram illustrating an image determining apparatus according to an exemplary embodiment.

[0028] Figure 5 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown schematically. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0030] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0031] Current mainstream image annotation techniques mainly include three approaches: manual annotation, semi-automated annotation tools, and fully automated evaluation. While manual annotation offers high flexibility, it has significant limitations. Different annotators have subjective differences in their understanding of "high quality," making it difficult to guarantee data consistency. Furthermore, manual annotation is inefficient and cannot support the construction of large-scale datasets.

[0032] While semi-automated annotation tools combine human review and automated pre-annotation, they still have significant drawbacks. These tools are highly dependent on the quality of the pre-trained model; insufficient training data or significant differences in scenarios can lead to serious mislabeling issues.

[0033] While fully automated evaluation schemes appear efficient, they suffer from fundamental limitations. Traditional image quality assessment algorithms, such as those based on Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM), cannot accurately identify visual defects in complex scenes, and the evaluation results differ significantly from human visual perception. These technical bottlenecks severely restrict the efficiency of constructing high-quality datasets.

[0034] Based on this, according to embodiments of the present disclosure, an image determination method, an image determination apparatus, an electronic device, a computer-readable storage medium, and a computer program product are proposed.

[0035] Figure 1 This is a flowchart illustrating an image determination method according to an exemplary embodiment, such as... Figure 1 As shown, the image determination method can be used in computer devices. The computer devices described in this disclosure may include mobile terminal devices such as mobile phones, tablets, laptops, PDAs, and personal digital assistants (PDAs), as well as fixed terminal devices such as desktop computers. This exemplary embodiment illustrates the application of the method to a computer device; it is understood that the method can also be applied to a server, and further to a system including both computer devices and a server, and implemented through the interaction between the computer device and the server. Specifically, it includes the following steps.

[0036] Step S110: Obtain the image set to be labeled, which includes multiple target labeled images;

[0037] Step S120: Receive the image annotation results of the annotator for the target image. The image annotation results are obtained by the annotator based on the image annotation standards generated after the annotation training operation.

[0038] Step S130: Obtain the reference labeled image, and perform cross-validation on the image annotation results based on the reference labeled image to obtain the annotation validation results;

[0039] Step S140: Based on the annotation verification results, determine the target quality image from multiple target annotation images.

[0040] According to the image determination method in this example embodiment, on the one hand, image annotation is performed based on the image annotation standards generated by the annotation training operation, so that the annotation results of the target annotated image have a unified standard, which can effectively avoid the annotation deviation problem caused by the subjective differences of the annotators and significantly improve the annotation quality. On the other hand, an implementation scheme for determining the target quality image based on the image annotation results is provided, which can effectively improve the annotation efficiency and fully meet the training needs of various computer vision models.

[0041] The image determination method in this example embodiment will be further described below.

[0042] In one exemplary embodiment of this disclosure, for step S110, an image set to be labeled is obtained, which includes multiple target labeled images. The image set to be labeled can be an image set composed of multiple target labeled images to be labeled. The target labeled images can be images to be quality labeled.

[0043] After obtaining the set of images to be labeled, the annotators will label the target images according to image annotation standards. For example, images whose image quality index threshold is greater than a specified threshold will be labeled as target quality images, such as high-quality images; other images whose index threshold does not meet the preset conditions will be labeled as low-quality images. The image annotation standards used by the annotators are determined based on pre-performed annotation training operations.

[0044] In one exemplary embodiment of this disclosure, the annotation training operation is performed through the following steps: obtaining an initial image set, which includes multiple initial images; performing image preprocessing on the initial images to obtain test annotation images; determining the test annotation results of the annotators for the test annotation images; and performing joint training on the test annotation results until the test annotation results meet the image annotation standards, where the image annotation standards are the annotation indicators of the test annotation results meeting the indicator threshold.

[0045] The initial image set can be a collection of images constructed from multiple initial images. The initial images can be the original images used for annotation training operations. Image preprocessing can involve resizing or downsampling the initial images. The test annotation images can be the images used in the annotation training phase. These test annotation images are shown to the annotators, who then perform test annotation operations based on them. The test annotation results can be the annotation labels determined by the annotators based on the image quality of the test annotation images; for example, the test annotation results can include low-quality images (0) and high-quality images (1). The image annotation standards can be the standards that the annotators use when performing image annotation operations.

[0046] refer to Figure 2 , Figure 2 This is a flowchart illustrating, according to an exemplary embodiment, the determination of target quality images through joint training operations and sub-package annotation operations. The image annotation scheme disclosed herein includes two stages: a joint evaluation stage and a sub-package stage. The joint evaluation stage can be the annotation training stage, while the sub-package stage can be the stage where annotators perform image annotation operations according to image annotation standards. The annotation training stage is a crucial step in aligning annotation operation standards. Through multiple batches of annotation training, all annotators are required to jointly review the same batch of image sets to ensure uniformity in image annotation standards. The sub-package stage employs an independent annotation mode to improve efficiency. The sub-package stage focuses on efficiently producing annotated images, using a distributed annotation mode where each annotator independently processes and allocates data.

[0047] After obtaining the initial image set, image preprocessing is performed on the initial images within the set. For example, in the image preprocessing stage, images with a shorter side exceeding 2048 pixels are downsampled using a multi-phase image interpolation algorithm (such as Lanczos interpolation). This ensures high-quality image preservation while maintaining processing efficiency, thus guaranteeing the accuracy of subsequent annotations. The preprocessed images are then used as test annotation images, which will have uniform image quality metrics.

[0048] Then, the annotators will perform image annotation operations on the test images. Annotators can label images without artifacts or with clear regions of interest as high-quality images. Based on the above annotation criteria, the annotators will annotate each test image one by one to determine the test annotation results. After obtaining the test annotation results, joint training will be conducted on the test annotation results until all annotators reach a unified standard for judging image quality, such as when the annotation consistency reaches a specified threshold.

[0049] Annotation consistency can be defined as the number of times an image is labeled as the same result by different annotators reaching a specified threshold, such as a specified threshold of 90%, where the number of times the same image is labeled as a high-quality image (1) reaches more than 90% of the total number of annotations; and the image quality index value of the test annotated image labeled as a high-quality image (1) usually needs to meet the index threshold. Through the joint training phase, annotators can reach a consensus on image evaluation annotations, which significantly reduces the quality fluctuation range compared to manual annotation methods and effectively solves the data deviation problem caused by subjective differences.

[0050] In one exemplary embodiment of this disclosure, determining the test annotation results of the annotator for the test annotation image includes: receiving the test annotation results of the annotator for the test annotation image; determining the test annotation sequence number corresponding to the test annotation result and the personnel identifier corresponding to the annotator; constructing a test annotation mapping table based on the personnel identifier, the test annotation sequence number and the annotation date; and storing the test annotation results based on the test annotation mapping table.

[0051] The test annotation sequence number can be the identifier number corresponding to an image that has been tested and annotated. The personnel identifier can be a unique number corresponding to the annotator; for example, the personnel identifier can be represented by the annotator's number, or it can be represented by both the annotator's number and name. The annotation date can be the specific date on which the annotator performed the test annotation operation. The test annotation mapping table can be a mapping table used to store relevant information about the test annotation results completed by the annotators.

[0052] After the annotators complete the test annotation operation and obtain the test annotation results, they will store the results. The test annotation sequence number of the test-annotated image will be obtained. This sequence number can be the image ID of the test-annotated image, or it can be generated from other information, such as a combination of the image ID and the number of times it was annotated. Additionally, the personnel identification and annotation date corresponding to the test-annotated image will be obtained, and the test annotation results will be stored according to the above information.

[0053] This disclosure establishes strict storage specifications for the test annotation result storage process. For example, it establishes a hierarchical directory based on the annotation date and the annotator's number, and can also maintain a test annotation mapping table based on the test annotation sequence number, annotator's name, and annotation date. By establishing a comprehensive storage structure, it ensures complete traceability of the subsequent annotation process for each image.

[0054] In one exemplary embodiment of this disclosure, step S120, receiving test annotation results from an annotator for a test annotation image, includes: determining a pre-configured image annotation tool, the image annotation tool including an annotation status display area, an image display area, and an operation prompt area; receiving test annotation operations from an annotator for the test annotation image based on the image annotation tool; and determining test annotation results based on the test annotation operations.

[0055] The image annotation tool can be an intelligent tool for receiving test annotation operations from annotators. The annotation status display area can be a display area for showing the image annotation progress. The image display area can be an area for displaying test-annotated images. The operation prompt area can be a display area for displaying operation prompts from annotators or preset shortcut keys.

[0056] To further improve the efficiency of image annotation, this disclosure provides an image annotation tool. For example, the image annotation tool is developed based on the PyQt5 framework, and the environment can be configured by the command "pip install PyQt5". The tool's startup command is simple and intuitive, and the interface design emphasizes practicality and efficiency. The display page of the image annotation tool may include, but is not limited to, an annotation status display area, an image display area, and an operation prompt area. The annotation status display area can show the current annotation processing progress in real time through a status bar, such as Process: 1 / 200, indicating that there are 200 images to be annotated, and one image has been annotated so far. The image display area is used to display the image currently being annotated. The image annotation tool requires the image to be displayed at 100% original size to avoid misjudgments caused by scaling. The image display area can also display basic information about the image, including image size, image name, and image type. The operation prompt area is used to display operation prompts to the annotator, such as the use of shortcut keys.

[0057] To improve operational efficiency, the image annotation tool features intelligent shortcut keys. For example, key 1 marks low-quality images, key 2 marks high-quality images, and the Z key allows for quick undoing and correction. This human-computer interaction design significantly improves annotation efficiency. The intelligent tool enables a single user to process up to 1000 images per day, which is 5-8 times more efficient than traditional manual annotation, fully meeting the training needs of various computer vision models. Furthermore, the image annotation tool can seamlessly integrate with various data acquisition systems, providing a technological foundation for building an end-to-end data production pipeline.

[0058] In one exemplary embodiment of this disclosure, cross-validation of image annotation results based on reference annotation images is performed to obtain annotation verification results. This includes: constructing a verification image set based on a pre-configured reference image ratio, reference annotation images, and target annotation images; determining the image quality score corresponding to the verification images in the verification image set; and performing cross-validation based on the image quality score and the image annotation results to obtain annotation verification results.

[0059] The reference image proportion can be the percentage of reference-annotated images in the total number of images. Reference-annotated images can be labeled images used for reference during cross-validation. The validation image set can be an image set constructed by combining reference-annotated images and target-annotated images according to a certain image ratio. The image quality score can be an evaluation score determined based on the image's own quality indicators. The annotation validation result can be the result obtained after cross-validating the image annotation results.

[0060] After the test annotation operation is completed, this disclosure also provides a quality management scheme to monitor the annotation quality of the image annotation results obtained after the test annotation operation. (Reference) Figure 3 , Figure 3 This is a flowchart illustrating quality management of a target labeled image according to an exemplary embodiment. Quality monitoring can employ a sampling mechanism, based on cross-validation processing of a reference labeled image and the target labeled image. The reference labeled image can be an image whose image classification label has been determined, such as a reference labeled image already having a corresponding image quality identifier (0 or 1), and this image quality identifier is a verified and accurate identifier.

[0061] For the image annotation results of the target labeled images, cross-validation can be performed based on reference labeled images. This can be done using a pre-built validation image set. For example, the proportion of reference labeled images in the total number of images in the entire validation image set can be pre-configured, such as a 1:10 ratio. 100 reference labeled images are then fixedly selected from every 1000 images for accuracy verification, ensuring consistently stable annotation quality. Post-processing management and control uses the F1 score as the core metric. The F1 score is the harmonic mean of precision and recall, commonly used to evaluate the performance of binary classification models.

[0062] During the cross-validation process, the annotators need to annotate all the validation images in the validation image set. The cross-validation mechanism is then used to verify the annotation results (100 images out of every 1000 images are annotated together) to obtain the annotation verification results, so as to achieve continuous monitoring of the quality of the image annotation results.

[0063] In one exemplary embodiment of this disclosure, determining the image quality score corresponding to the verification image in the verification image set includes: determining the quality evaluation index value corresponding to the verification image, wherein the quality evaluation index value includes the artifact detection result and local sharpness of the verification image; and determining the image quality score based on the artifact detection result and local sharpness.

[0064] The quality evaluation index value can be a specific score determined according to the specific evaluation criteria of the image quality evaluation index. The artifact detection result can be the result obtained after artifact detection processing of the verification image. Artifact detection aims to automatically identify and locate artifact regions in the image. Local sharpness can be an index value obtained by evaluating the sharpness of the region of interest in the verification image. The image quality score can be a comprehensive image score determined by combining the artifact detection result and the local sharpness.

[0065] In the cross-validation process, this disclosure uses both image quality score and image annotation results for cross-validation. The evaluation of image quality score focuses on image defects rather than subjective aesthetics. Specific core evaluation criteria may include, but are not limited to, the following key indicators, which can be selected based on subsequent image processing tasks. The key indicators selected in this scheme include: first, typical artifact detection, covering compression damage (such as ringing effects, block artifacts), blurring (ground, vegetation, etc.), background noise, oversharpening, and abnormal color transitions, etc. These detections are performed on the validation image to obtain artifact detection results; second, subject sharpness evaluation, focusing on checking the focus quality of key objects. Local regions mainly refer to the regions of interest in the image, determining the sharpness of the regions of interest in the validation image. Then, based on the artifact detection results and local sharpness, the corresponding image quality score of the validation image is jointly determined.

[0066] To facilitate standard implementation, this disclosure provides 200 labeled example images, stored in corresponding file directories, clearly distinguished by the _0 (low-quality images) and _1 (high-quality images) suffixes, providing an intuitive reference for annotators. Cross-validation using objective image evaluation metrics can further improve the accuracy of image annotation results.

[0067] It will be readily understood by those skilled in the art that, in some other exemplary embodiments of this disclosure, other image evaluation metrics can be selected to calculate image quality scores based on subsequent image processing tasks. These image processing tasks may include, but are not limited to, low-light enhancement, super-resolution reconstruction, and image inpainting. This disclosure supports flexible expansion based on image processing tasks, and the evaluation criteria can be adjusted according to different task requirements, making the overall solution more adaptable.

[0068] In one exemplary embodiment of this disclosure, determining a target quality image from multiple target labeled images based on the annotation verification result includes: when the annotation verification result is greater than or equal to a preset threshold, selecting a target quality image from multiple target labeled images, wherein the target quality image is an image whose image annotation result is a specified annotation identifier; constructing a target image set based on multiple target quality images, wherein the target image set is used for model training; and restarting the annotation training operation when the annotation verification result is less than the preset threshold.

[0069] After obtaining the annotation verification results, the F1 score of the annotation verification results can be compared with a preset threshold, such as 80%, 85%, or 90%. When the F1 score of the annotation verification results is greater than or equal to the preset threshold, the image classification result is considered to meet the pre-configured accuracy index. At this time, the target annotation image can be obtained, and a target quality image, such as a high-quality image, can be selected from the target annotation images and used in subsequent task processing, such as model training.

[0070] However, when the F1 score of the annotation verification result is less than a preset threshold, the image classification result is considered not to meet the pre-configured accuracy metric. In this case, the training annotation operation can be restarted for the batch of images that do not meet the accuracy metric until the annotation consistency obtained after the training annotation operation meets the preset threshold. After the training annotation operation meets the preset threshold, the annotators will then perform image annotation operations to determine the target quality images, providing a data foundation for subsequent image processing tasks.

[0071] In summary, the image determination method disclosed herein involves: acquiring a set of images to be labeled, including multiple target labeled images; receiving image annotation results from annotators for the target labeled images, the image annotation results being obtained by the annotators based on image annotation standards generated after annotation training operations; acquiring reference labeled images; cross-validating the image annotation results based on the reference labeled images to obtain annotation verification results; and determining the target quality image from the multiple target labeled images based on the annotation verification results. On one hand, by performing image annotation operations based on image annotation standards generated after annotation training operations, the annotation results of the target labeled images have a unified standard, which can effectively avoid annotation deviations caused by subjective differences of annotators and significantly improve annotation quality. On the other hand, it provides an implementation scheme for determining the target quality image based on image annotation results, which can effectively improve annotation efficiency and fully meet the training needs of various computer vision models. Furthermore, by using intelligent annotation tools, the manual intervention is further reduced, improving annotation efficiency. Finally, by constructing a robust storage structure, the annotation process of each image can be ensured to be traceable, and combined with an accuracy evaluation system, continuous monitoring and optimization of annotation quality can be achieved, providing reliable data support for subsequent image processing tasks.

[0072] Figure 4 This is a block diagram illustrating an image determining apparatus according to an exemplary embodiment. (Refer to...) Figure 4 The image determination device 400 includes: an image set acquisition module 410, an annotation result receiving module 420, an annotation verification module 430, and a target image determination module 440.

[0073] Specifically, the image set acquisition module 410 is used to acquire an image set to be labeled, which includes multiple target labeled images; the labeling result receiving module 420 is used to receive the image labeling results of the labelers for the target labeled images, which are obtained by the labelers based on the image labeling standards generated after the labeling training operation; the labeling verification module 430 is used to acquire a reference labeled image and perform cross-validation on the image labeling results based on the reference labeled image to obtain the labeling verification result; and the target image determination module 440 is used to determine the target quality image from the multiple target labeled images based on the labeling verification result.

[0074] In one exemplary embodiment of this disclosure, the image determination device 400 includes a labeling training module, configured to: acquire an initial image set, the initial image set including multiple initial images; perform image preprocessing on the initial images to obtain test labeled images; determine the test labeling results of the labelers for the test labeled images; and perform joint training on the test labeling results until the test labeling results meet the image labeling standards, the image labeling standards being that the labeling indicators of the test labeling results meet the indicator threshold.

[0075] In one exemplary embodiment of this disclosure, the annotation training module includes a test annotation unit, configured to: receive test annotation results from annotators for test annotation images; determine the test annotation sequence number corresponding to the test annotation result and the personnel identifier corresponding to the annotator; construct a test annotation mapping table based on the personnel identifier, the test annotation sequence number and the annotation date; and store the test annotation results based on the test annotation mapping table.

[0076] In one exemplary embodiment of this disclosure, the test annotation unit includes an annotation result receiving subunit, configured to: determine a pre-configured image annotation tool, the image annotation tool including an annotation status display area, an image display area and an operation prompt area; receive test annotation operations performed by annotators on the test annotation image based on the image annotation tool; and determine the test annotation result based on the test annotation operations.

[0077] In one exemplary embodiment of this disclosure, the annotation verification module 430 includes an annotation verification unit, which is used to: construct a verification image set based on a pre-configured reference image ratio, a reference annotation image, and a target annotation image; determine the image quality score corresponding to the verification image in the verification image set; and perform cross-validation based on the image quality score and the image annotation result to obtain the annotation verification result.

[0078] In one exemplary embodiment of this disclosure, the annotation verification unit includes a quality score determination subunit, configured to: determine the quality evaluation index value corresponding to the verification image, wherein the quality evaluation index value includes the artifact detection result and local sharpness of the verification image; and determine the image quality score based on the artifact detection result and local sharpness.

[0079] In one exemplary embodiment of this disclosure, the target image determination module 440 includes a target image determination unit, configured to: select a target quality image from multiple target labeled images when the annotation verification result is greater than or equal to a preset threshold, wherein the target quality image is an image whose image annotation result is a specified annotation identifier; construct a target image set based on the multiple target quality images, wherein the target image set is used for model training; and restart the annotation training operation when the annotation verification result is less than the preset threshold.

[0080] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0081] The following reference Figure 5 To describe an electronic device 500 according to such an embodiment of the present disclosure. Figure 5 The electronic device 500 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0082] like Figure 5 As shown, the electronic device 500 is manifested in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including storage unit 520 and processing unit 510), and a display unit 540.

[0083] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this disclosure.

[0084] Storage unit 520 may include readable media in the form of volatile storage units, such as random access memory (RAM) 521 and / or cache memory 522, and may further include read-only memory (ROM) 523.

[0085] Storage unit 520 may also include a program / utility 524 having a set (at least one) program module 525, such program module 525 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0086] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0087] Electronic device 500 can also communicate with one or more external devices 570 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. As shown, network adapter 560 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0088] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of the device to complete the image determination method described above. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0089] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the image determination method described in any one of the preceding embodiments.

[0090] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0091] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image determination method, characterized in that, include: Obtain a set of images to be labeled, which includes multiple target labeled images; The system receives image annotation results from annotators for the target image, the image annotation results being obtained by the annotators based on image annotation standards generated after annotation training operations; Obtain a reference labeled image, and perform cross-validation on the image labeling results based on the reference labeled image to obtain the labeling validation results; Based on the annotation verification results, a target quality image is determined from the plurality of target annotated images.

2. The method according to claim 1, characterized in that, The annotation training operation is performed through the following steps: Obtain an initial image set, which includes multiple initial images; The initial image is preprocessed to obtain the test annotation image; Determine the test annotation results of the annotator for the test annotation image; Joint training is performed on the test annotation results until the test annotation results meet the image annotation standard, where the image annotation standard is that the annotation indicators of the test annotation results meet the indicator threshold.

3. The method according to claim 2, characterized in that, Determining the test annotation results of the annotator for the test annotation image includes: Receive the test annotation results from the annotator for the test annotation image; Determine the test labeling sequence number corresponding to the test labeling result, and the personnel identifier corresponding to the labeling personnel; Construct a test label mapping table based on the personnel identifier, the test label sequence number, and the label date; The test annotation results are stored based on the test annotation mapping table.

4. The method according to claim 3, characterized in that, Receiving the test annotation results from the annotator for the test annotation image includes: A pre-configured image annotation tool is determined, the image annotation tool including an annotation status display area, an image display area, and an operation prompt area; Based on the image annotation tool, the system receives test annotation operations from the annotator for the test annotation image; The test annotation result is determined based on the test annotation operation.

5. The method according to claim 1, characterized in that, The process of cross-validating the image annotation results based on the reference annotated image to obtain annotation validation results includes: A verification image set is constructed based on the pre-configured reference image proportions, the reference labeled image, and the target labeled image; Determine the image quality score corresponding to the verification image in the verification image set; The annotation verification result is obtained by cross-validating the image quality score with the image annotation result.

6. The method according to claim 5, characterized in that, Determining the image quality score corresponding to the verification image in the verification image set includes: Determine the quality evaluation index value corresponding to the verification image, wherein the quality evaluation index value includes the artifact detection result and local sharpness of the verification image; The image quality score is determined based on the artifact detection results and the local sharpness.

7. The method according to claim 1, characterized in that, The step of determining the target quality image from the plurality of target labeled images based on the annotation verification results includes: When the annotation verification result is greater than or equal to a preset threshold, a target quality image is selected from the plurality of target annotation images, wherein the target quality image is an image whose image annotation result is a specified annotation identifier; Based on multiple target quality images, a target image set is constructed, and the target image set is used for model training. When the annotation verification result is less than a preset threshold, the annotation training operation is restarted.

8. An image determining device, characterized in that, include: An image set acquisition module is used to acquire an image set to be labeled, wherein the image set to be labeled includes multiple target labeled images; The annotation result receiving module is used to receive the image annotation results of the annotator for the target annotated image. The image annotation results are obtained by the annotator based on the image annotation standards generated after the annotation training operation. The annotation verification module is used to acquire a reference annotation image, perform cross-validation on the image annotation results based on the reference annotation image, and obtain the annotation verification results. The target image determination module is used to determine the target quality image from the plurality of target labeled images based on the annotation verification results.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the image determination method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image determination method according to any one of claims 1 to 7.

Images