A crowd-sourcing based image annotation method, system and medium

Abstract

The embodiment of the application provides a picture labeling method and system based on crowdsourcing and a medium. The method comprises the following steps: obtaining picture features to be filled and classifying and labeling a circle annotation rate according to picture adaptation to a professional crowdsourcing group, performing picture feature screening and identification to obtain a true picture feature quantity and a feature labeling rate, if the true picture feature quantity and the feature labeling rate pass threshold comparison, screening effective personnel meeting an average requirement according to an effective labeling rate of each crowdsourcing personnel, extracting a historical circle labeling rate, processing to obtain a professional circle annotation labeling effective correction coefficient, performing threshold comparison to judge total circle annotation effectiveness, if the total circle annotation effectiveness is effective, repairing the picture and verifying the effectiveness, so as to screen effective personnel according to a circle labeling effect of the professional crowdsourcing personnel, judge circle labeling effectiveness of the effective personnel, repair the picture and verify the repair effect, verify crowdsourcing labeling effectiveness and verify the picture repair effect through the picture labeling effect of the crowdsourcing personnel, and realize the crowdsourcing labeling and repair technology of the picture.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of picture annotation, in particular to a picture annotation method and system based on crowdsourcing and a medium. BACKGROUND

[0002] Picture annotation is a common annotation type, which mainly identifies and annotates an unprocessed picture so as to further correct picture information features to obtain accurate picture information. However, a large number of professional pictures cannot be identified and annotated by computers or limited personnel, which easily causes workload backlog or large deviation. Therefore, a user group annotation method of adopting professional crowdsourcing personnel to annotate pictures is an effective way to meet the annotation of a large number of picture information. However, due to great differences in quality, profession, content and background information of different pictures, a traditional system identification crowdsourcing distribution method cannot verify the reliability of crowdsourcing annotation, and cannot judge the repair effect of the picture after crowdsourcing annotation. Therefore, a common situation of large error of crowdsourcing picture annotation and repair effect occurs. How to verify and obtain effective crowdsourcing picture annotation quality and realize high-quality picture annotation repair is a current technical gap.

[0003] In view of the above problems, an effective technical solution is currently needed. SUMMARY

[0004] The purpose of the embodiment of the application is to provide a picture annotation method, system and medium based on crowdsourcing, which can screen effective crowdsourcing personnel according to the circle annotation effect verification result of professional crowdsourcing personnel, judge the circle annotation effectiveness of the effective crowdsourcing personnel, judge the picture repair and verify the repair effect, realize the annotation effectiveness verification of the crowdsourcing personnel through the picture annotation effect of the crowdsourcing personnel, and realize the crowdsourcing annotation and repair means of the picture.

[0005] The embodiment of the application also provides a picture annotation method based on crowdsourcing, which comprises the following steps:

[0006] A picture to be annotated is obtained, information pickup identification of the picture is performed to obtain picture integrity information and picture category information, a picture quality level is obtained by performing integrity quality level division on the picture through a preset crowdsourcing platform, and picture attribute classification is performed to obtain picture attribute category information;

[0007] Picture pairing processing is performed according to the picture attribute category information to obtain an adaptive professional crowdsourcing group and a corresponding professional annotation difficulty coefficient, a plurality of picture features to be filled and a corresponding plurality of picture feature vacancy rates are obtained by performing identification processing on the picture through the professional crowdsourcing group, and a picture vacancy rate average index is obtained by processing;

[0008] The picture feature circle annotation result is obtained by respectively performing picture feature circle annotation on the pictures by the crowd sourcing personnel of the professional crowd sourcing group, including corresponding circle label information obtained by classification circle annotation, and each circle annotation rate of corresponding circle label frequency is obtained, and key correct picture features, high abnormal picture features and key error picture features are respectively screened by threshold comparison according to each circle annotation rate;

[0009] The real feature quantity and corresponding feature annotation rate data are obtained by respectively performing identification on each key correct picture feature and each key error picture feature, and threshold comparison is performed on each corresponding annotation threshold value by comparing with a preset feature annotation rate threshold value, if the result meets the preset requirement, the real abnormal feature quantity and corresponding abnormal feature annotation rate data are obtained by identifying each high abnormal picture feature, and the third annotation threshold comparison result is obtained by threshold comparison with a preset abnormal feature annotation rate threshold value;

[0010] If the third annotation threshold comparison result meets the preset threshold comparison requirement, each classification feature effective annotation rate data and each classification feature average effective annotation rate data of the picture feature circle annotation result of each crowd sourcing personnel of the professional crowd sourcing group are obtained, and effective crowd sourcing annotation personnel meeting each classification feature average effective annotation rate data are screened, and corresponding historical effective circle annotation rate data are extracted;

[0011] The professional circle annotation effective correction coefficient is obtained by processing the historical effective circle annotation rate data of each effective crowd sourcing annotation personnel in combination with the picture quality grade, professional annotation difficulty coefficient and picture vacancy rate average index, and the total circle annotation effectiveness of each effective crowd sourcing annotation personnel is judged by threshold comparison with a preset professional picture feature annotation effect detection threshold value;

[0012] If the total circle annotation effectiveness is effective, the pictures are filled and corrected by the effective crowd sourcing annotation personnel corresponding to the plurality of to-be-filled picture features and the plurality of key error picture features to obtain the repaired pictures, the picture repair effectiveness result is obtained by comparing the repaired pictures with the standardized processed pictures, and the records of each effective crowd sourcing annotation personnel are updated.

[0013] Optionally, in the picture annotation method based on crowd sourcing provided in the embodiments of the present application, the pictures to be annotated are obtained, and the picture integrity information and the picture category information are obtained by performing information picking and identification on the pictures, the picture quality grade is obtained by performing integrity quality grade division on the pictures through a preset crowd sourcing platform, and the picture attribute category information is obtained by performing attribute classification on the pictures, including:

[0014] Obtaining pictures to be annotated;

[0015] According to a preset picture recognition model, information of the picture is picked up and recognized to obtain picture completeness information and picture category information;

[0016] According to the picture completeness information, the picture is classified according to a preset picture completeness classification level through a preset crowdsourcing platform to obtain a picture quality level;

[0017] According to the picture category information, the picture is classified according to a preset attribute classification method to obtain picture attribute category information;

[0018] The picture attribute category information includes picture element content information, picture object field information and picture scene information.

[0019] Optionally, in the picture labeling method based on crowdsourcing, the picture pairing processing according to the picture attribute category information is performed to obtain an adaptive professional crowdsourcing group and a corresponding professional labeling difficulty coefficient, and the picture is processed by the professional crowdsourcing group to obtain a plurality of to-be-filled picture features and a corresponding plurality of picture feature vacancy rates, and an average picture vacancy rate index is obtained by processing, including:

[0020] According to the picture element content information, the picture object field information and the picture scene information, the picture information is processed by the preset crowdsourcing platform to obtain an adaptive professional crowdsourcing group and a corresponding professional labeling difficulty coefficient corresponding to the picture;

[0021] The picture is processed by the professional crowdsourcing group to obtain a plurality of to-be-filled picture features and a corresponding plurality of picture feature vacancy rates of the picture supplemented by a plurality of crowdsourcing personnel of the professional crowdsourcing group;

[0022] According to the picture feature vacancy rate, an average picture vacancy rate index is obtained.

[0023] Optionally, in the picture labeling method based on crowdsourcing, the picture feature classification circle annotation labeling of the picture is performed by the plurality of crowdsourcing personnel of the professional crowdsourcing group to obtain picture feature circle annotation labeling results, including corresponding circle information obtained by classification circle annotation labeling, and obtaining each circle annotation rate of a corresponding circle frequency, and screening key correct picture features, high abnormal picture features and key error picture features according to each circle annotation rate through threshold comparison, including:

[0024] The picture feature classification circle annotation labeling of the picture is performed by the plurality of crowdsourcing personnel of the professional crowdsourcing group to obtain picture feature circle annotation labeling results;

[0025] The image feature annotation results include the correct image feature annotation information obtained by the first annotation, the abnormal image feature annotation information obtained by the second annotation, and the incorrect image feature annotation information obtained by the third annotation.

[0026] Based on the frequency of each circle annotation in the first, second, and third circle annotations of each image feature in the image, the first circle annotation rate corresponding to each correct image feature, the second circle annotation rate corresponding to each abnormal image feature, and the third circle annotation rate corresponding to each incorrect image feature are obtained.

[0027] Mark the correct image features that meet the preset first looping threshold requirements among the correct image features corresponding to the first looping rate as key correct image features;

[0028] The abnormal image features that meet the preset second loop injection threshold requirements among the abnormal image features corresponding to the second loop injection rate are marked as high abnormal image features;

[0029] Among the error image features corresponding to the third round of injection rate, those that meet the preset third round of injection threshold are marked as key error image features.

[0030] Optionally, in the crowdsourcing-based image annotation method described in this application embodiment, the step of verifying and identifying each of the key correct image features and each of the key incorrect image features to obtain the true number of features and the corresponding feature annotation rate data, and comparing them with a preset feature annotation rate threshold to obtain the corresponding annotation threshold comparison result, and if the results all meet the preset requirements, then identifying each of the high-abnormal image features to obtain the true number of abnormal features and the corresponding abnormal feature annotation rate data, and comparing them with a preset abnormal feature annotation rate threshold to obtain a third annotation threshold comparison result, includes:

[0031] The number of true and correct features and the corresponding correct feature labeling rate data are obtained by verifying and recognizing each of the key correct image features using a preset feature verification and recognition model.

[0032] The number of real error features and the corresponding error feature labeling rate data are obtained by verifying and identifying the features of each key error image through a preset feature verification and identification model.

[0033] The correct feature annotation rate data is compared with the preset correct feature annotation rate threshold to obtain the first annotation threshold comparison result.

[0034] The error feature labeling rate data is compared with the preset error feature labeling rate threshold to obtain the second labeling threshold comparison result.

[0035] If the comparison results of the first and second labeling thresholds both meet the preset threshold comparison requirements, then the features of each of the high-abnormal images are identified by the preset image recognition model to obtain the number of real abnormal features and the corresponding abnormal feature labeling rate data.

[0036] The abnormal feature annotation rate data is compared with the preset abnormal feature annotation rate threshold to obtain the third annotation threshold comparison result.

[0037] Optionally, in the crowdsourcing-based image annotation method described in this application embodiment, if the third annotation threshold comparison result meets the preset threshold comparison requirement, then the effective annotation rate data of each category feature and the average effective annotation rate data of each category feature of the image feature annotation results of each crowdsourcing person in the professional crowdsourcing group are obtained, and the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature are selected, and the corresponding historical effective annotation rate data is extracted, including:

[0038] If the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then the correct feature effective annotation rate data, the incorrect feature effective annotation rate data, and the abnormal feature effective annotation rate data of the image feature annotation results of each crowdsourcing member in the professional crowdsourcing group are obtained.

[0039] Obtain the average effective annotation rate data for correct features, the average effective annotation rate data for incorrect features, and the average effective annotation rate data for abnormal features from all crowdsourcing personnel;

[0040] Valid crowdsourced labelers are selected from all crowdsourced personnel who meet the criteria for average effective labeling rate of correct features, average effective labeling rate of incorrect features, and average effective labeling rate of abnormal features.

[0041] Extract the corresponding historical valid annotation rate data for each valid crowdsourced annotator.

[0042] Optionally, in the crowdsourcing-based image annotation method described in this application embodiment, the step of processing the historical effective annotation rate data of each effective crowdsourcing annotator in conjunction with the image quality level, professional annotation difficulty coefficient, and average image gap rate index to obtain a professional annotation effectiveness correction coefficient, and then comparing it with a preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of each effective crowdsourcing annotator, includes:

[0043] Based on the historical effective annotation rate data of each effective crowdsourced annotator, combined with the image quality level, professional annotation difficulty coefficient and average image gap rate index, the effective correction coefficient for professional annotation is obtained.

[0044] The effectiveness of each crowdsourced annotation is determined by comparing the professional annotation correction coefficient with the preset professional image feature annotation effect detection threshold, and the overall annotation effectiveness of each effective crowdsourced annotator is determined based on the threshold comparison result.

[0045] Optionally, in the crowdsourcing-based image annotation method described in this application embodiment, if the overall annotation validity is valid, the step of extracting the corresponding multiple unfilled image features and multiple key error image features of the valid crowdsourcing annotators to fill and correct the image to obtain a repaired image, comparing the repaired image with the standardized image to obtain the image repair effect result, and updating the records of each valid crowdsourcing annotator, includes:

[0046] If the overall annotation validity is valid, then extract the corresponding features of the multiple images to be filled and the features of the multiple key error images of the valid crowdsourced annotators;

[0047] The image is filled and corrected based on the features of the multiple images to be filled and the features of the multiple key error images to obtain the repaired image;

[0048] The image repair results are obtained by comparing the repaired image with the standardized image.

[0049] The image repair performance records of each valid crowdsourcing labeler are updated based on the image repair performance results.

[0050] Secondly, embodiments of this application provide a crowdsourcing-based image annotation system. The system includes a memory and a processor. The memory includes a program for a crowdsourcing-based image annotation method. When the program for the crowdsourcing-based image annotation method is executed by the processor, it performs the following steps:

[0051] The system acquires images to be labeled, performs information picking and recognition on the images to obtain image completeness information and image category information, classifies the images into completeness quality levels through a preset crowdsourcing platform to obtain image quality levels, and classifies the images into attributes to obtain image attribute category information.

[0052] Based on the image attribute category information, image matching processing is performed to obtain suitable professional crowdsourcing groups and corresponding professional labeling difficulty coefficients. The professional crowdsourcing groups are used to identify and process the images to obtain multiple image features to be filled and corresponding multiple image feature vacancy rates. The average index of image vacancy rate is then obtained.

[0053] The image features are classified and annotated by multiple crowdsourcing personnel in the professional crowdsourcing group to obtain the image feature annotation results, including the corresponding annotation information of each annotation, the annotation rate of each annotation frequency, and the key correct image features, high abnormal image features and key error image features are selected by threshold comparison based on each annotation rate.

[0054] The key correct image features and key incorrect image features are respectively verified and identified to obtain the number of real features and the corresponding feature annotation rate data. The data are then compared with the preset feature annotation rate threshold to obtain the corresponding annotation threshold comparison result. If the results meet the preset requirements, the high abnormal image features are identified to obtain the number of real abnormal features and the corresponding abnormal feature annotation rate data. The data are then compared with the preset abnormal feature annotation rate threshold to obtain the third annotation threshold comparison result.

[0055] If the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then obtain the effective annotation rate data of each category feature and the average effective annotation rate data of each category feature of each crowdsourcing personnel in the professional crowdsourcing group, and filter out the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature, and extract the corresponding historical effective annotation rate data.

[0056] Based on the historical effective annotation rate data of each effective crowdsourced annotator, combined with the image quality level, professional annotation difficulty coefficient and average image gap rate index, the professional annotation effective correction coefficient is obtained. Then, it is compared with the preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of each effective crowdsourced annotator.

[0057] If the overall annotation validity is valid, the corresponding features of the multiple images to be filled and the features of the multiple key error images of the valid crowdsourced annotators are extracted to fill and correct the images to obtain the repaired images. The repaired images are compared with the standardized images to obtain the image repair effect results, and the records of each valid crowdsourced annotator are updated.

[0058] Thirdly, embodiments of this application also provide a computer-readable storage medium, which includes a crowdsourcing-based image annotation method program. When the crowdsourcing-based image annotation method program is executed by a processor, it implements the steps of the crowdsourcing-based image annotation method as described in any of the preceding claims.

[0059] As can be seen from the above, the image annotation method, system, and medium based on crowdsourcing provided in this application obtain the image quality level and attribute category of the image to be annotated, and obtain multiple image features to be filled and the average index of the missing rate for image recognition by a professional crowdsourcing group. The image features are then categorized and annotated to obtain each annotation information and annotation rate, and image features are filtered. The filtered image features are then distinguished to obtain the number of true / correct / incorrect / abnormal image features and feature annotation rate data. The results are compared with thresholds for each feature to proceed to the next step. If all thresholds are met, the effective annotation rate data for each category of features of each crowdsourcing worker is obtained, and effective crowdsourcing annotators who meet the average effective annotation rate data for each category of features are selected and extracted. Based on historical effective annotation rate data, and combined with image quality level, professional annotation difficulty coefficient, and average image gap rate index, a professional annotation effectiveness correction coefficient is obtained. Then, a threshold comparison is used to determine the overall annotation effectiveness. If effective, the image is filled and corrected, and the repair effect is verified and the record is updated. Thus, based on the annotation effect verification results of professional crowdsourcing personnel, effective crowdsourcing personnel are selected, and the annotation effectiveness of effective crowdsourcing personnel is judged. If the judgment is passed, the image is repaired, and the repair effect is verified. This achieves the verification of crowdsourcing personnel's annotation effectiveness and image repair effect through the image annotation effect of crowdsourcing personnel, realizing a crowdsourcing annotation and repair method for images.

[0060] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 A flowchart illustrating the crowdsourcing-based image annotation method provided in this application embodiment;

[0063] Figure 2 A flowchart illustrating the process of obtaining image quality level and image attribute category information using a crowdsourcing-based image annotation method provided in this application embodiment;

[0064] Figure 3 A flowchart illustrating the process of obtaining the professional annotation difficulty coefficient and the average image gap rate index for the crowdsourcing-based image annotation method provided in this application embodiment;

[0065] Figure 4 The flowchart illustrates how the crowdsourcing-based image annotation method provided in this application obtains and filters key correct image features, highly abnormal image features, and key error image features.

[0066] Figure 5 This is a flowchart illustrating the comparison results of various annotation thresholds obtained by the crowdsourcing-based image annotation method provided in this application embodiment. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0068] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0069] Please refer to Figure 1 , Figure 1 This is a flowchart of a crowdsourcing-based image annotation method according to some embodiments of this application. This crowdsourcing-based image annotation method is used in terminal devices, such as computers and mobile phones. The crowdsourcing-based image annotation method includes the following steps:

[0070] S11. Obtain the image to be labeled, and perform information picking and recognition on the image to obtain image completeness information and image category information. Use a preset crowdsourcing platform to classify the image completeness quality level to obtain the image quality level, and classify the image attributes to obtain image attribute category information.

[0071] S12. Based on the image attribute category information, perform image matching processing to obtain a suitable professional crowdsourcing group and a corresponding professional labeling difficulty coefficient. Through the professional crowdsourcing group, perform image recognition processing to obtain multiple image features to be filled and corresponding multiple image feature vacancy rates, and process to obtain the average index of image vacancy rates.

[0072] S13. The image features are classified and annotated by multiple crowdsourcing personnel in the professional crowdsourcing group to obtain the image feature annotation results, including the corresponding annotation information of each annotation obtained by classification and annotation, and the annotation rate of each annotation frequency. Based on the annotation rate, key correct image features, high abnormal image features and key error image features are selected by threshold comparison.

[0073] S14. The key correct image features and the key incorrect image features are respectively verified and identified to obtain the number of real features and the corresponding feature annotation rate data. The data are then compared with the preset feature annotation rate threshold to obtain the corresponding annotation threshold comparison result. If the results meet the preset requirements, the high abnormal image features are identified to obtain the number of real abnormal features and the corresponding abnormal feature annotation rate data. The data are then compared with the preset abnormal feature annotation rate threshold to obtain the third annotation threshold comparison result.

[0074] S15. If the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then obtain the effective annotation rate data of each category feature and the average effective annotation rate data of each category feature of each crowdsourcing personnel in the professional crowdsourcing group, and filter out the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature, and extract the corresponding historical effective annotation rate data.

[0075] S16. Based on the historical effective annotation rate data of each effective crowdsourced annotator, combined with the image quality level, professional annotation difficulty coefficient and average index of image gap rate, the effective correction coefficient of professional annotation is obtained. Then, the coefficient is compared with the preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of each effective crowdsourced annotator.

[0076] S17. If the overall annotation validity is valid, extract the corresponding features of the multiple images to be filled and the features of the multiple key error images of the valid crowdsourcing annotators to fill and correct the images to obtain the repaired images. Compare the repaired images with the standardized images to obtain the image repair results and update the records of each valid crowdsourcing annotator.

[0077] To verify the effectiveness of crowdsourced image annotation and its repair capabilities, and to improve the reliability and effectiveness of post-annotation repair, this technology first examines the effective annotation rate of correct / incorrect / abnormal image features after initial screening by professional crowdsourcers. This verifies the accuracy and effectiveness of batch-level crowdsourcer image annotation. If a crowdsourcer passes the annotation effectiveness review, high-performing crowdsourcers exceeding the average annotation level are selected. Further overall annotation effectiveness is then verified based on the professional difficulty of the target image, feature gaps, and image quality. If the verification passes, the image is repaired according to the standard features of the high-performing crowdsourcers, and the effectiveness of the repaired image is verified. This approach improves the efficiency of image annotation and repair effectiveness by verifying and screening high-precision image annotation crowdsourcers. The system obtains the image quality level and attribute category of the image to be labeled, and obtains multiple image features and average missing rate index from the appropriate professional crowdsourcing group for image recognition. It then classifies and annotates the image features to obtain each annotation information and annotation rate, and filters the image features. The system also detects the selected image features to obtain the number of true / correct / incorrect / abnormal image features and feature annotation rate data. The results are compared with thresholds to proceed to the next step. If all thresholds are met, the system obtains the effective annotation rate data for each category of features from each crowdsourcing person, filters out valid crowdsourcing annotators who meet the average effective annotation rate data for each category, and extracts the corresponding historical effective annotation rate data. This data is then combined with the image quality level, professional annotation difficulty coefficient, and average missing rate index to obtain the effective correction coefficient for professional annotation. Finally, the system compares the results with thresholds to determine the overall annotation effectiveness. If effective, the system fills and corrects the image, verifies the repair effect, and updates the records.

[0078] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the process of obtaining image quality level and image attribute category information using a crowdsourcing-based image annotation method according to some embodiments of this application. According to embodiments of the present invention, the steps of obtaining the image to be annotated, performing information picking and recognition on the image to obtain image completeness information and image category information, classifying the image for completeness quality level using a preset crowdsourcing platform to obtain image quality level, and classifying the image for attributes to obtain image attribute category information are as follows:

[0079] S21. Obtain the image to be labeled;

[0080] S22. Based on the preset image recognition model, perform information picking and recognition on the image to obtain image completeness information and image category information;

[0081] S23. Using a preset crowdsourcing platform, the image is classified into quality levels according to the preset image integrity classification level based on the image integrity information, and the image quality level is obtained.

[0082] S24. Based on the image category information, classify the image according to a preset attribute classification method to obtain image attribute category information;

[0083] S25. The image attribute category information includes image element content information, image object domain information, and image scene information.

[0084] The process begins by acquiring images to be labeled. Using a pre-defined image recognition model, information is extracted and identified to obtain image completeness and category information. Then, a pre-defined third-party crowdsourcing platform categorizes the images according to their completeness, resulting in image quality levels. These quality levels indirectly determine the effectiveness and impact of image labeling. Simultaneously, images are categorized according to a pre-defined attribute classification method to obtain image attribute category information. This categorizes the image's content, scene, domain, and background information to facilitate professional crowdsourcing allocation. Image attribute category information includes the image's element content, the domain of the object being displayed, and the scene information.

[0085] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the process of obtaining the professional annotation difficulty coefficient and the average index of image vacancy rate in a crowdsourcing-based image annotation method according to some embodiments of this application. According to embodiments of the present invention, the step of obtaining a suitable professional crowdsourcing group and a corresponding professional annotation difficulty coefficient by performing image matching processing based on the image attribute category information, obtaining multiple image features to be filled and corresponding multiple image feature vacancy rates through the professional crowdsourcing group's image recognition processing, and then processing to obtain the average index of image vacancy rate, specifically involves:

[0086] S31. Based on the image element content information, image object domain information and image scene information, perform professional matching processing of image information through the preset crowdsourcing platform to obtain a professional crowdsourcing group that matches the image and a corresponding professional labeling difficulty coefficient.

[0087] S32. The image is processed by the professional crowdsourcing group to obtain multiple image features to be filled and the corresponding missing rates of multiple image features.

[0088] S33. Obtain the average index of image gap rate based on the image feature gap rate.

[0089] To more efficiently and accurately identify image annotation features for effective image recognition and repair, image information needs to be professionally matched using a pre-set crowdsourcing platform based on image element content information, image object domain information, and image scene information. This process yields a professional crowdsourcing group that matches the image and a corresponding professional annotation difficulty coefficient. Specifically, the appropriate professional crowdsourcing group is selected based on the image information. Multiple individuals within the professional crowdsourcing group then process the image to obtain the image features to be filled and the image feature gap rate obtained from the processing by multiple individuals. Finally, the image feature gap rate of all individuals is processed to obtain the average image gap rate index. The formula for calculating the average image gap rate index is as follows:

[0090] ;

[0091] in, This is the average index of image gap rate. The image feature gap rate for a single person is given by r, where r is the number of people. These are preset feature coefficients (obtained by querying the preset crowdsourcing platform database).

[0092] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the process of obtaining and filtering key correct image features, highly abnormal image features, and key error image features according to some embodiments of this application, based on a crowdsourcing-based image annotation method. According to an embodiment of the present invention, the step of obtaining image feature annotation results by having multiple crowdsourcing personnel in the professional crowdsourcing group perform image feature classification and annotation on the images, including the corresponding annotation information obtained from the classification and annotation, obtaining the annotation rate of each annotation frequency, and filtering key correct image features, highly abnormal image features, and key error image features respectively based on the annotation rate through threshold comparison, specifically:

[0093] S41. The image features are classified and annotated by multiple crowdsourcing personnel in the professional crowdsourcing group to obtain the image feature annotation results;

[0094] S42. The image feature annotation results include the correct image feature annotation information obtained by the first annotation, the abnormal image feature annotation information obtained by the second annotation, and the incorrect image feature annotation information obtained by the third annotation.

[0095] S43. Based on the image features in the image, perform first-circle annotation, second-circle annotation, and third-circle annotation frequency respectively to obtain the first-circle annotation rate corresponding to each correct image feature, the second-circle annotation rate corresponding to each abnormal image feature, and the third-circle annotation rate corresponding to each incorrect image feature.

[0096] S44. Mark the correct image features that meet the preset first circle-injection threshold requirements among the correct image features corresponding to the first circle-injection rate as key correct image features.

[0097] S45. Mark the abnormal image features that meet the preset second looping threshold requirements among the abnormal image features corresponding to the second looping rate as high abnormal image features;

[0098] S46. Mark the error image features that meet the preset third-round injection threshold requirements among the error image features corresponding to the third-round injection rate as key error image features.

[0099] Because image annotation involves classifying correct, incorrect, and unconfirmed ambiguous abnormal features, the effectiveness of professional crowdsourcing in image annotation requires classifying, circling, and labeling these three types of image features. Circling involves identifying and highlighting target features, while labeling involves providing on-demand annotations to the feature information. The annotation results, obtained from multiple crowdsourcing personnel within a professional crowdsourcing group, show the following: first, annotations for correct image features; second, annotations for abnormal image features; and third, annotations for incorrect image features. Furthermore, each annotated image... The frequency of each feature being circled by all professional crowdsourcing personnel is used to obtain the first circle annotation rate for each correct image feature, the second circle annotation rate for each abnormal image feature, and the third circle annotation rate for each incorrect image feature. Then, the image features with high-frequency circle annotation are selected and marked as the filtered image features. That is, by comparing the preset thresholds, the image features that meet the corresponding circle annotation threshold requirements for each correct / abnormal / incorrect image feature corresponding to the first / second / third circle annotation rate are identified and marked as key correct image features / highly abnormal image features / key incorrect image features, respectively. In this way, the various types of image features to be circled and annotated are filtered at high frequency to improve the accuracy of image annotation.

[0100] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating the process of obtaining various annotation threshold comparison results for a crowdsourcing-based image annotation method in some embodiments of this application. According to an embodiment of the present invention, the method involves verifying and identifying the key correct image features and the key incorrect image features to obtain the true number of features and corresponding feature annotation rate data, and then comparing these data with a preset feature annotation rate threshold to obtain corresponding annotation threshold comparison results. If all results meet the preset requirements, the method identifies the high-abnormal image features to obtain the true number of abnormal features and corresponding abnormal feature annotation rate data, and then compares this data with a preset abnormal feature annotation rate threshold to obtain a third annotation threshold comparison result. Specifically:

[0101] S51. The key correct image features are verified and identified by a preset feature verification and recognition model to obtain the number of true correct features and the corresponding correct feature labeling rate data.

[0102] S52. The key error image features are identified and verified using a preset feature verification and recognition model to obtain the number of real error features and the corresponding error feature labeling rate data.

[0103] S53. Compare the correct feature annotation rate data with the preset correct feature annotation rate threshold to obtain the first annotation threshold comparison result;

[0104] S54. Compare the error feature labeling rate data with the preset error feature labeling rate threshold to obtain the second labeling threshold comparison result;

[0105] S55. If the comparison results of the first and second annotation thresholds both meet the preset threshold comparison requirements, then the high-abnormality image features are identified by the preset image recognition model to obtain the number of real abnormal features and the corresponding abnormal feature annotation rate data.

[0106] S56. Compare the abnormal feature annotation rate data with the preset abnormal feature annotation rate threshold to obtain the third annotation threshold comparison result.

[0107] The selection of key correct image features, high-abnormality image features, and key incorrect image features requires further verification of their authenticity. This involves using a pre-defined feature detection and recognition model to verify the authenticity of each key correct image feature and key incorrect image feature, and then calculating the ratio of the number of valid and genuine labeled features to the number of labeled features in the corresponding category. The ratio of valid labels can be used to verify the actual quality of the image labeling effect by the crowdsourcing personnel. The effect of image labeling is initially judged by comparing the correct / incorrect feature labeling rate data with the corresponding pre-defined correct / incorrect feature labeling rate thresholds. If both conditions are met, the abnormal feature labeling rate data is compared with the threshold to obtain the third labeling threshold comparison result.

[0108] According to an embodiment of the present invention, if the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then the effective annotation rate data of each category feature and the average effective annotation rate data of each category feature of the image feature annotation results of each crowdsourcing member in the professional crowdsourcing group are obtained, and the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature are selected, and the corresponding historical effective annotation rate data are extracted, specifically as follows:

[0109] If the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then the correct feature effective annotation rate data, the incorrect feature effective annotation rate data, and the abnormal feature effective annotation rate data of the image feature annotation results of each crowdsourcing member in the professional crowdsourcing group are obtained.

[0110] Obtain the average effective annotation rate data for correct features, the average effective annotation rate data for incorrect features, and the average effective annotation rate data for abnormal features from all crowdsourcing personnel;

[0111] Valid crowdsourced labelers are selected from all crowdsourced personnel who meet the criteria for average effective labeling rate of correct features, average effective labeling rate of incorrect features, and average effective labeling rate of abnormal features.

[0112] Extract the corresponding historical valid annotation rate data for each valid crowdsourced annotator.

[0113] If the comparison result of the third annotation threshold also meets the preset requirements, it indicates that the image annotation effect of the professional crowdsourcing personnel meets the reliability requirements. Then, the high-precision annotators with more accurate annotation results among the professional crowdsourcing personnel are further selected as the target personnel for the next step of image repair. By obtaining the effective annotation rate data of each category feature of each crowdsourcing personnel, i.e. annotation effect data, and selecting the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature, i.e., selecting the high-precision annotators whose annotation results exceed the average level, and extracting the historical effective annotation rate data corresponding to the selected personnel, i.e., the historical annotation performance data of the selected annotators.

[0114] According to an embodiment of the present invention, the step of processing the historical effective annotation rate data of each effective crowdsourced annotator in conjunction with the image quality level, professional annotation difficulty coefficient, and average image gap rate index to obtain a professional annotation effectiveness correction coefficient, and then comparing it with a preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of each effective crowdsourced annotator, specifically involves:

[0115] Based on the historical effective annotation rate data of each effective crowdsourced annotator, combined with the image quality level, professional annotation difficulty coefficient and average image gap rate index, the effective correction coefficient for professional annotation is obtained.

[0116] The effectiveness of each crowdsourced annotation is determined by comparing the professional annotation correction coefficient with the preset professional image feature annotation effect detection threshold, and the overall annotation effectiveness of each effective crowdsourced annotator is determined based on the threshold comparison result.

[0117] To further verify the predictive effectiveness of the selected effective crowdsourced annotators in image repair, a professional annotation effectiveness correction coefficient was calculated based on the historical effective annotation rate data of the personnel, combined with image quality level, professional annotation difficulty coefficient, and average image gap rate index. This coefficient is calculated by comprehensively correcting the historical annotation performance data of effective crowdsourced annotators, considering image quality, annotation difficulty, and gap status. The correction coefficient for the predicted annotation result is then compared with a preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of the effective crowdsourced annotators. The formula for calculating the professional annotation effectiveness correction coefficient is as follows:

[0118] ;

[0119] in, For professional annotation, an effective correction factor is provided. This represents the historical effective annotation rate data for the i-th effective crowdsourced annotator, where n is the number of effective crowdsourced annotators. The difficulty level is marked with a professional label. Image quality level, This is the average index of image gap rate. , , These are preset feature coefficients (obtained by querying the preset crowdsourcing platform database).

[0120] According to an embodiment of the present invention, if the overall annotation validity is valid, the corresponding multiple unfilled image features and multiple key error image features of the valid crowdsourcing annotators are extracted to fill and correct the image to obtain a repaired image. The repaired image is then compared with the standardized image to obtain the image repair effect result, and the records of each valid crowdsourcing annotator are updated. Specifically:

[0121] If the overall annotation validity is valid, then extract the corresponding features of the multiple images to be filled and the features of the multiple key error images of the valid crowdsourced annotators;

[0122] The image is filled and corrected based on the features of the multiple images to be filled and the features of the multiple key error images to obtain the repaired image;

[0123] The image repair results are obtained by comparing the repaired image with the standardized image.

[0124] The image repair performance records of each valid crowdsourcing labeler are updated based on the image repair performance results.

[0125] Specifically, when the overall effectiveness of the crowdsourced annotation personnel passes the test, it indicates that the image annotation and repair ability of the effective crowdsourced annotation personnel has passed the evaluation. Then, the corresponding features of multiple images to be filled and multiple key error images of the effective crowdsourced annotation personnel are extracted, and the images are filled and corrected to obtain the repaired images. The repair effect of the repaired images is then tested, and the personnel's image repair effect record is updated according to the repair effect results.

[0126] This invention also discloses a crowdsourcing-based image annotation system, including a memory and a processor. The memory includes a crowdsourcing-based image annotation method program. When the processor executes the crowdsourcing-based image annotation method program to correct abnormal vital signs data, it performs the following steps:

[0127] The system acquires images to be labeled, performs information picking and recognition on the images to obtain image completeness information and image category information, classifies the images into completeness quality levels through a preset crowdsourcing platform to obtain image quality levels, and classifies the images into attributes to obtain image attribute category information.

[0128] Based on the image attribute category information, image matching processing is performed to obtain suitable professional crowdsourcing groups and corresponding professional labeling difficulty coefficients. The professional crowdsourcing groups are used to identify and process the images to obtain multiple image features to be filled and corresponding multiple image feature vacancy rates. The average index of image vacancy rate is then obtained.

[0129] The image features are classified and annotated by multiple crowdsourcing personnel in the professional crowdsourcing group to obtain the image feature annotation results, including the corresponding annotation information of each annotation, the annotation rate of each annotation frequency, and the key correct image features, high abnormal image features and key error image features are selected by threshold comparison based on each annotation rate.

[0130] The key correct image features and key incorrect image features are respectively verified and identified to obtain the number of real features and the corresponding feature annotation rate data. The data are then compared with the preset feature annotation rate threshold to obtain the corresponding annotation threshold comparison result. If the results meet the preset requirements, the high abnormal image features are identified to obtain the number of real abnormal features and the corresponding abnormal feature annotation rate data. The data are then compared with the preset abnormal feature annotation rate threshold to obtain the third annotation threshold comparison result.

[0131] If the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then obtain the effective annotation rate data of each category feature and the average effective annotation rate data of each category feature of each crowdsourcing personnel in the professional crowdsourcing group, and filter out the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature, and extract the corresponding historical effective annotation rate data.

[0132] Based on the historical effective annotation rate data of each effective crowdsourced annotator, combined with the image quality level, professional annotation difficulty coefficient and average image gap rate index, the professional annotation effective correction coefficient is obtained. Then, it is compared with the preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of each effective crowdsourced annotator.

[0133] If the overall annotation validity is valid, the corresponding features of the multiple images to be filled and the features of the multiple key error images of the valid crowdsourced annotators are extracted to fill and correct the images to obtain the repaired images. The repaired images are compared with the standardized images to obtain the image repair effect results, and the records of each valid crowdsourced annotator are updated.

[0134] To verify the effectiveness of crowdsourced image annotation and its repair capabilities, and to improve the reliability and effectiveness of post-annotation repair, this technology first examines the effective annotation rate of correct / incorrect / abnormal image features after initial screening by professional crowdsourcers. This verifies the accuracy and effectiveness of batch-level crowdsourcer image annotation. If a crowdsourcer passes the annotation effectiveness review, high-performing crowdsourcers exceeding the average annotation level are selected. Further overall annotation effectiveness is then verified based on the professional difficulty of the target image, feature gaps, and image quality. If the verification passes, the image is repaired according to the standard features of the high-performing crowdsourcers, and the effectiveness of the repaired image is verified. This approach improves the efficiency of image annotation and repair effectiveness by verifying and screening high-precision image annotation crowdsourcers. The system obtains the image quality level and attribute category of the image to be labeled, and obtains multiple image features and average missing rate index from the appropriate professional crowdsourcing group for image recognition. It then classifies and annotates the image features to obtain each annotation information and annotation rate, and filters the image features. The system also detects the selected image features to obtain the number of true / correct / incorrect / abnormal image features and feature annotation rate data. The results are compared with thresholds to proceed to the next step. If all thresholds are met, the system obtains the effective annotation rate data for each category of features from each crowdsourcing person, filters out valid crowdsourcing annotators who meet the average effective annotation rate data for each category, and extracts the corresponding historical effective annotation rate data. This data is then combined with the image quality level, professional annotation difficulty coefficient, and average missing rate index to obtain the effective correction coefficient for professional annotation. Finally, the system compares the results with thresholds to determine the overall annotation effectiveness. If effective, the system fills and corrects the image, verifies the repair effect, and updates the records.

[0135] According to an embodiment of the present invention, the steps of obtaining the image to be labeled, performing information picking and recognition on the image to obtain image completeness information and image category information, classifying the image into completeness quality levels through a preset crowdsourcing platform to obtain image quality levels, and classifying the image into attributes to obtain image attribute category information are as follows:

[0136] Get the image to be labeled;

[0137] The image is processed by a pre-defined image recognition model to obtain information about the image's completeness and category.

[0138] The image quality level is obtained by classifying the image according to the image integrity information through a preset crowdsourcing platform;

[0139] Based on the image category information, the image is classified according to a preset attribute classification method to obtain image attribute category information;

[0140] The image attribute category information includes image element content information, image object domain information, and image scene information.

[0141] The process begins by acquiring images to be labeled. Using a pre-defined image recognition model, information is extracted and identified to obtain image completeness and category information. Then, a pre-defined third-party crowdsourcing platform categorizes the images according to their completeness, resulting in image quality levels. These quality levels indirectly determine the effectiveness and impact of image labeling. Simultaneously, images are categorized according to a pre-defined attribute classification method to obtain image attribute category information. This categorizes the image's content, scene, domain, and background information to facilitate professional crowdsourcing allocation. Image attribute category information includes the image's element content, the domain of the object being displayed, and the scene information.

[0142] According to an embodiment of the present invention, the step of obtaining a suitable professional crowdsourcing group and a corresponding professional labeling difficulty coefficient by performing image matching processing based on the image attribute category information, obtaining multiple image features to be filled and corresponding multiple image feature gap rates by performing image recognition processing through the professional crowdsourcing group, and processing to obtain the average index of image gap rates, specifically involves:

[0143] Based on the image element content information, image object domain information, and image scene information, the image information is professionally matched through the preset crowdsourcing platform to obtain a professional crowdsourcing group that matches the image and a corresponding professional labeling difficulty coefficient.

[0144] The professional crowdsourcing group performs recognition processing on the image to obtain multiple image features to be filled in by multiple crowdsourcing personnel in the professional crowdsourcing group, as well as the corresponding multiple image feature gap rates.

[0145] The average index of image gap rate is obtained by processing the image feature gap rate.

[0146] To more efficiently and accurately identify image annotation features for effective image recognition and repair, image information needs to be professionally matched using a pre-set crowdsourcing platform based on image element content information, image object domain information, and image scene information. This process yields a professional crowdsourcing group that matches the image and a corresponding professional annotation difficulty coefficient. Specifically, the appropriate professional crowdsourcing group is selected based on the image information. Multiple individuals within the professional crowdsourcing group then process the image to obtain the image features to be filled and the image feature gap rate obtained from the processing by multiple individuals. Finally, the image feature gap rate of all individuals is processed to obtain the average image gap rate index. The formula for calculating the average image gap rate index is as follows:

[0147] ;

[0148] in, This is the average index of image gap rate. The image feature gap rate for a single person is given by r, where r is the number of people. These are preset feature coefficients (obtained by querying the preset crowdsourcing platform database).

[0149] According to an embodiment of the present invention, the step of obtaining image feature annotation results by having multiple crowdsourcing personnel in the professional crowdsourcing group classify and annotate the images, including the corresponding annotation information obtained from the classification and annotation, obtaining the annotation rate of each annotation frequency, and filtering key correct image features, high-abnormal image features, and key erroneous image features based on each annotation rate through threshold comparison, specifically:

[0150] The image feature annotation results are obtained by having multiple crowdsourcing personnel in the professional crowdsourcing group perform image feature classification and annotation on the images respectively;

[0151] The image feature annotation results include the correct image feature annotation information obtained by the first annotation, the abnormal image feature annotation information obtained by the second annotation, and the incorrect image feature annotation information obtained by the third annotation.

[0152] Based on the frequency of each circle annotation in the first, second, and third circle annotations of each image feature in the image, the first circle annotation rate corresponding to each correct image feature, the second circle annotation rate corresponding to each abnormal image feature, and the third circle annotation rate corresponding to each incorrect image feature are obtained.

[0153] Mark the correct image features that meet the preset first looping threshold requirements among the correct image features corresponding to the first looping rate as key correct image features;

[0154] The abnormal image features that meet the preset second loop injection threshold requirements among the abnormal image features corresponding to the second loop injection rate are marked as high abnormal image features;

[0155] Among the error image features corresponding to the third round of injection rate, those that meet the preset third round of injection threshold are marked as key error image features.

[0156] Because image annotation involves classifying correct, incorrect, and unconfirmed ambiguous abnormal features, the effectiveness of professional crowdsourcing in image annotation requires classifying, circling, and labeling these three types of image features. Circling involves identifying and highlighting target features, while labeling involves providing on-demand annotations to the feature information. The annotation results, obtained from multiple crowdsourcing personnel within a professional crowdsourcing group, show the following: first, annotations for correct image features; second, annotations for abnormal image features; and third, annotations for incorrect image features. Furthermore, each annotated image... The frequency of each feature being circled by all professional crowdsourcing personnel is used to obtain the first circle annotation rate for each correct image feature, the second circle annotation rate for each abnormal image feature, and the third circle annotation rate for each incorrect image feature. Then, the image features with high-frequency circle annotation are selected and marked as the filtered image features. That is, by comparing the preset thresholds, the image features that meet the corresponding circle annotation threshold requirements for each correct / abnormal / incorrect image feature corresponding to the first / second / third circle annotation rate are identified and marked as key correct image features / highly abnormal image features / key incorrect image features, respectively. In this way, the various types of image features to be circled and annotated are filtered at high frequency to improve the accuracy of image annotation.

[0157] According to an embodiment of the present invention, the method of verifying and identifying the key correct image features and the key incorrect image features to obtain the number of true features and the corresponding feature annotation rate data, and comparing them with the preset feature annotation rate threshold to obtain the corresponding annotation threshold comparison result, and if the results all meet the preset requirements, then identifying the high-abnormal image features to obtain the number of true abnormal features and the corresponding abnormal feature annotation rate data, and comparing them with the preset abnormal feature annotation rate threshold to obtain the third annotation threshold comparison result, specifically:

[0158] The number of true and correct features and the corresponding correct feature labeling rate data are obtained by verifying and recognizing each of the key correct image features using a preset feature verification and recognition model.

[0159] The number of real error features and the corresponding error feature labeling rate data are obtained by verifying and identifying the features of each key error image through a preset feature verification and identification model.

[0160] The correct feature annotation rate data is compared with the preset correct feature annotation rate threshold to obtain the first annotation threshold comparison result.

[0161] The error feature labeling rate data is compared with the preset error feature labeling rate threshold to obtain the second labeling threshold comparison result.

[0162] If the comparison results of the first and second labeling thresholds both meet the preset threshold comparison requirements, then the features of each of the high-abnormal images are identified by the preset image recognition model to obtain the number of real abnormal features and the corresponding abnormal feature labeling rate data.

[0163] The abnormal feature annotation rate data is compared with the preset abnormal feature annotation rate threshold to obtain the third annotation threshold comparison result.

[0164] The selection of key correct image features, high-abnormality image features, and key incorrect image features requires further verification of their authenticity. This involves using a pre-defined feature detection and recognition model to verify the authenticity of each key correct image feature and key incorrect image feature, and then calculating the ratio of the number of valid and genuine labeled features to the number of labeled features in the corresponding category. The ratio of valid labels can be used to verify the actual quality of the image labeling effect by the crowdsourcing personnel. The effect of image labeling is initially judged by comparing the correct / incorrect feature labeling rate data with the corresponding pre-defined correct / incorrect feature labeling rate thresholds. If both conditions are met, the abnormal feature labeling rate data is compared with the threshold to obtain the third labeling threshold comparison result.

[0165] According to an embodiment of the present invention, if the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then the effective annotation rate data of each category feature and the average effective annotation rate data of each category feature of the image feature annotation results of each crowdsourcing member in the professional crowdsourcing group are obtained, and the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature are selected, and the corresponding historical effective annotation rate data are extracted, specifically as follows:

[0166] If the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then the correct feature effective annotation rate data, the incorrect feature effective annotation rate data, and the abnormal feature effective annotation rate data of the image feature annotation results of each crowdsourcing member in the professional crowdsourcing group are obtained.

[0167] Obtain the average effective annotation rate data for correct features, the average effective annotation rate data for incorrect features, and the average effective annotation rate data for abnormal features from all crowdsourcing personnel;

[0168] Valid crowdsourced labelers are selected from all crowdsourced personnel who meet the criteria for average effective labeling rate of correct features, average effective labeling rate of incorrect features, and average effective labeling rate of abnormal features.

[0169] Extract the corresponding historical valid annotation rate data for each valid crowdsourced annotator.

[0170] If the comparison result of the third annotation threshold also meets the preset requirements, it indicates that the image annotation effect of the professional crowdsourcing personnel meets the reliability requirements. Then, the high-precision annotators with more accurate annotation results among the professional crowdsourcing personnel are further selected as the target personnel for the next step of image repair. By obtaining the effective annotation rate data of each category feature of each crowdsourcing personnel, i.e. annotation effect data, and selecting the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature, i.e., selecting the high-precision annotators whose annotation results exceed the average level, and extracting the historical effective annotation rate data corresponding to the selected personnel, i.e., the historical annotation performance data of the selected annotators.

[0171] According to an embodiment of the present invention, the step of processing the historical effective annotation rate data of each effective crowdsourced annotator in conjunction with the image quality level, professional annotation difficulty coefficient, and average image gap rate index to obtain a professional annotation effectiveness correction coefficient, and then comparing it with a preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of each effective crowdsourced annotator, specifically involves:

[0172] Based on the historical effective annotation rate data of each effective crowdsourced annotator, combined with the image quality level, professional annotation difficulty coefficient and average image gap rate index, the effective correction coefficient for professional annotation is obtained.

[0173] The effectiveness of each crowdsourced annotation is determined by comparing the professional annotation correction coefficient with the preset professional image feature annotation effect detection threshold, and the overall annotation effectiveness of each effective crowdsourced annotator is determined based on the threshold comparison result.

[0174] To further verify the predictive effectiveness of the selected effective crowdsourced annotators in image repair, a professional annotation effectiveness correction coefficient was calculated based on the historical effective annotation rate data of the personnel, combined with image quality level, professional annotation difficulty coefficient, and average image gap rate index. This coefficient is calculated by comprehensively correcting the historical annotation performance data of effective crowdsourced annotators, considering image quality, annotation difficulty, and gap status. The correction coefficient for the predicted annotation result is then compared with a preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of the effective crowdsourced annotators. The formula for calculating the professional annotation effectiveness correction coefficient is as follows:

[0175] ;

[0176] in, For professional annotation, an effective correction factor is provided. This represents the historical effective annotation rate data for the i-th effective crowdsourced annotator, where n is the number of effective crowdsourced annotators. The difficulty level is marked with a professional label. Image quality level, This is the average index of image gap rate. , , These are preset feature coefficients (obtained by querying the preset crowdsourcing platform database).

[0177] According to an embodiment of the present invention, if the overall annotation validity is valid, the corresponding multiple unfilled image features and multiple key error image features of the valid crowdsourcing annotators are extracted to fill and correct the image to obtain a repaired image. The repaired image is then compared with the standardized image to obtain the image repair effect result, and the records of each valid crowdsourcing annotator are updated. Specifically:

[0178] If the overall annotation validity is valid, then extract the corresponding features of the multiple images to be filled and the features of the multiple key error images of the valid crowdsourced annotators;

[0179] The image is filled and corrected based on the features of the multiple images to be filled and the features of the multiple key error images to obtain the repaired image;

[0180] The image repair results are obtained by comparing the repaired image with the standardized image.

[0181] The image repair performance records of each valid crowdsourcing labeler are updated based on the image repair performance results.

[0182] Specifically, when the overall effectiveness of the crowdsourced annotation personnel passes the test, it indicates that the image annotation and repair ability of the effective crowdsourced annotation personnel has passed the evaluation. Then, the corresponding features of multiple images to be filled and multiple key error images of the effective crowdsourced annotation personnel are extracted, and the images are filled and corrected to obtain the repaired images. The repair effect of the repaired images is then tested, and the personnel's image repair effect record is updated according to the repair effect results.

[0183] A third aspect of the present invention provides a readable storage medium including a crowdsourcing-based image annotation method program, wherein when the crowdsourcing-based image annotation method program is executed by a processor, it implements the steps of the crowdsourcing-based image annotation method as described in any of the preceding claims.

[0184] This invention discloses a crowdsourcing-based image annotation method, system, and medium. It obtains the image quality level and attribute category of the image to be annotated, and acquires multiple image features and an average missing rate index for image recognition by a suitable professional crowdsourcing group. The system then classifies and annotates these image features to obtain annotation information and annotation rates, and performs feature filtering. It further distinguishes between true / correct / incorrect / abnormal image features and their annotation rates, and compares the results against thresholds for each feature. If all thresholds are met, it acquires the effective annotation rate data for each category of features from each crowdsourcing worker, filters out effective crowdsourcing annotators who meet the average effective annotation rate data for each category, and extracts their corresponding historical data. Effective annotation rate data is combined with image quality level, professional annotation difficulty coefficient, and average image gap rate index to obtain a professional annotation effectiveness correction coefficient. Then, a threshold comparison is used to determine the overall annotation effectiveness. If effective, the image is filled and corrected, and the repair effect is verified and the record is updated. Based on the annotation effect verification results of professional crowdsourcing personnel, effective crowdsourcing personnel are selected, and the annotation effectiveness of these personnel is judged. If the judgment is passed, the image is repaired, and the repair effect is verified. This achieves the goal of verifying the annotation effectiveness of crowdsourcing personnel and the image repair effect through their annotation performance, realizing a crowdsourcing annotation and repair method for images.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0186] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0188] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A crowdsourcing-based image annotation method, characterized in that, Includes the following steps: The system acquires images to be labeled, performs information picking and recognition on the images to obtain image completeness information and image category information, classifies the images into completeness quality levels through a preset crowdsourcing platform to obtain image quality levels, and classifies the images into attributes to obtain image attribute category information. Based on the image attribute category information, image matching processing is performed to obtain suitable professional crowdsourcing groups and corresponding professional labeling difficulty coefficients. The professional crowdsourcing groups are used to identify and process the images to obtain multiple image features to be filled and corresponding multiple image feature vacancy rates. The average index of image vacancy rate is then obtained. The image features are classified and annotated by multiple crowdsourcing personnel in the professional crowdsourcing group to obtain the image feature annotation results, including the corresponding annotation information of each annotation, the annotation rate of each annotation frequency, and the key correct image features, high abnormal image features and key error image features are selected by threshold comparison based on each annotation rate. The key correct image features and key incorrect image features are respectively verified and identified to obtain the number of real features and the corresponding feature annotation rate data. The data are then compared with the preset feature annotation rate threshold to obtain the corresponding annotation threshold comparison result. If the results meet the preset requirements, the high abnormal image features are identified to obtain the number of real abnormal features and the corresponding abnormal feature annotation rate data. The data are then compared with the preset abnormal feature annotation rate threshold to obtain the third annotation threshold comparison result. If the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then obtain the effective annotation rate data of each category feature and the average effective annotation rate data of each category feature of each crowdsourcing personnel in the professional crowdsourcing group, and filter out the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature, and extract the corresponding historical effective annotation rate data. Based on the historical effective annotation rate data of each effective crowdsourced annotator, combined with the image quality level, professional annotation difficulty coefficient and average image gap rate index, the professional annotation effective correction coefficient is obtained. Then, it is compared with the preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of each effective crowdsourced annotator. If the overall annotation validity is valid, the corresponding features of the multiple images to be filled and the features of the multiple key error images of the valid crowdsourced annotators are extracted to fill and correct the images to obtain the repaired images. The repaired images are compared with the standardized images to obtain the image repair effect results, and the records of each valid crowdsourced annotator are updated.

2. The image annotation method based on crowdsourcing according to claim 1, characterized in that, The process involves acquiring images to be labeled, performing information extraction and recognition to obtain image completeness and category information, classifying images into completeness quality levels using a pre-set crowdsourcing platform to obtain image quality levels, and classifying images into attribute categories to obtain image attribute category information, including: Get the image to be labeled; The image is processed by a pre-defined image recognition model to obtain information about the image's completeness and category. The image quality level is obtained by classifying the image according to the image integrity information through a preset crowdsourcing platform; Based on the image category information, the image is classified according to a preset attribute classification method to obtain image attribute category information; The image attribute category information includes image element content information, image object domain information, and image scene information.

3. The image annotation method based on crowdsourcing according to claim 2, characterized in that, The process of matching images based on the image attribute category information to obtain suitable professional crowdsourcing groups and corresponding professional labeling difficulty coefficients, using these professional crowdsourcing groups to identify and process images to obtain multiple unfilled image features and corresponding feature gap rates, and then processing these to obtain an average image gap rate index, including: Based on the image element content information, image object domain information, and image scene information, the image information is professionally matched through the preset crowdsourcing platform to obtain a professional crowdsourcing group that matches the image and a corresponding professional labeling difficulty coefficient. The professional crowdsourcing group performs recognition processing on the image to obtain multiple image features to be filled in by multiple crowdsourcing personnel in the professional crowdsourcing group, as well as the corresponding multiple image feature gap rates. The average index of image gap rate is obtained by processing the image feature gap rate.

4. The image annotation method based on crowdsourcing according to claim 3, characterized in that, The process involves multiple crowdsourcing personnel in the professional crowdsourcing group performing image feature classification and annotation on the images to obtain image feature annotation results. This includes obtaining the corresponding annotation information for each annotation, acquiring the annotation rate for each annotation frequency, and filtering key correct image features, highly abnormal image features, and key erroneous image features based on threshold comparisons using each annotation rate. The image feature annotation results are obtained by having multiple crowdsourcing personnel in the professional crowdsourcing group perform image feature classification and annotation on the images respectively; The image feature annotation results include the correct image feature annotation information obtained by the first annotation, the abnormal image feature annotation information obtained by the second annotation, and the incorrect image feature annotation information obtained by the third annotation. Based on the frequency of each circle annotation in the first, second, and third circle annotations of each image feature in the image, the first circle annotation rate corresponding to each correct image feature, the second circle annotation rate corresponding to each abnormal image feature, and the third circle annotation rate corresponding to each incorrect image feature are obtained. Mark the correct image features that meet the preset first looping threshold requirements among the correct image features corresponding to the first looping rate as key correct image features; The abnormal image features that meet the preset second loop injection threshold requirements among the abnormal image features corresponding to the second loop injection rate are marked as high abnormal image features; Among the error image features corresponding to the third round of injection rate, those that meet the preset third round of injection threshold are marked as key error image features.

5. The crowdsourcing-based image annotation method according to claim 4, characterized in that, The process involves verifying and identifying the key correct image features and the key incorrect image features to obtain the number of true features and the corresponding feature annotation rate data. These are then compared with a preset feature annotation rate threshold to obtain a corresponding annotation threshold comparison result. If all results meet the preset requirements, the process involves identifying the high-abnormal image features to obtain the number of true abnormal features and the corresponding abnormal feature annotation rate data. This data is then compared with a preset abnormal feature annotation rate threshold to obtain a third annotation threshold comparison result, including: The number of true and correct features and the corresponding correct feature labeling rate data are obtained by verifying and recognizing each of the key correct image features using a preset feature verification and recognition model. The number of real error features and the corresponding error feature labeling rate data are obtained by verifying and identifying the features of each key error image through a preset feature verification and identification model. The correct feature annotation rate data is compared with the preset correct feature annotation rate threshold to obtain the first annotation threshold comparison result. The error feature labeling rate data is compared with the preset error feature labeling rate threshold to obtain the second labeling threshold comparison result. If the comparison results of the first and second labeling thresholds both meet the preset threshold comparison requirements, then the features of each of the high-abnormal images are identified by the preset image recognition model to obtain the number of real abnormal features and the corresponding abnormal feature labeling rate data. The abnormal feature annotation rate data is compared with the preset abnormal feature annotation rate threshold to obtain the third annotation threshold comparison result.

6. The image annotation method based on crowdsourcing according to claim 5, characterized in that, If the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then the effective annotation rate data of each category feature and the average effective annotation rate data of each category feature of each crowdsourcing member in the professional crowdsourcing group are obtained, and the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature are selected, and the corresponding historical effective annotation rate data are extracted, including: If the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then the correct feature effective annotation rate data, the incorrect feature effective annotation rate data, and the abnormal feature effective annotation rate data of the image feature annotation results of each crowdsourcing member in the professional crowdsourcing group are obtained. Obtain the average effective annotation rate data for correct features, the average effective annotation rate data for incorrect features, and the average effective annotation rate data for abnormal features from all crowdsourcing personnel; Valid crowdsourced labelers are selected from all crowdsourced personnel who meet the criteria for average effective labeling rate of correct features, average effective labeling rate of incorrect features, and average effective labeling rate of abnormal features. Extract the corresponding historical valid annotation rate data for each valid crowdsourced annotator.

7. The crowdsourcing-based image annotation method according to claim 6, characterized in that, The process involves processing the historical effective annotation rate data of each effective crowdsourced annotator with the image quality level, professional annotation difficulty coefficient, and average image gap rate index to obtain a professional annotation effectiveness correction coefficient. This coefficient is then compared with a preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of each effective crowdsourced annotator. This includes: Based on the historical effective annotation rate data of each effective crowdsourced annotator, combined with the image quality level, professional annotation difficulty coefficient and average image gap rate index, the effective correction coefficient for professional annotation is obtained. The effectiveness of each crowdsourced annotation is determined by comparing the professional annotation correction coefficient with the preset professional image feature annotation effect detection threshold, and the overall annotation effectiveness of each effective crowdsourced annotator is determined based on the threshold comparison result.

8. The image annotation method based on crowdsourcing according to claim 7, characterized in that, If the overall annotation validity is valid, then the corresponding features of the multiple images to be filled and the features of the multiple key error images of the valid crowdsourced annotators are extracted to fill and correct the images to obtain the repaired images. The repaired images are compared with the standardized images to obtain the image repair effect results, and the records of each valid crowdsourced annotator are updated, including: If the overall annotation validity is valid, then extract the corresponding features of the multiple images to be filled and the features of the multiple key error images of the valid crowdsourced annotators; The image is filled and corrected based on the features of the multiple images to be filled and the features of the multiple key error images to obtain the repaired image; The image repair results are obtained by comparing the repaired image with the standardized image. The image repair performance records of each valid crowdsourcing labeler are updated based on the image repair performance results.

9. A crowdsourcing-based image annotation system, characterized in that, The system includes a memory and a processor. The memory contains a program for a crowdsourcing-based image annotation method. When the program for the crowdsourcing-based image annotation method is executed by the processor, it performs the following steps: The system acquires images to be labeled, performs information picking and recognition on the images to obtain image completeness information and image category information, classifies the images into completeness quality levels through a preset crowdsourcing platform to obtain image quality levels, and classifies the images into attributes to obtain image attribute category information. Based on the image attribute category information, image matching processing is performed to obtain suitable professional crowdsourcing groups and corresponding professional labeling difficulty coefficients. The professional crowdsourcing groups are used to identify and process the images to obtain multiple image features to be filled and corresponding multiple image feature vacancy rates. The average index of image vacancy rate is then obtained. The image features are classified and annotated by multiple crowdsourcing personnel in the professional crowdsourcing group to obtain the image feature annotation results, including the corresponding annotation information of each annotation, the annotation rate of each annotation frequency, and the key correct image features, high abnormal image features and key error image features are selected by threshold comparison based on each annotation rate. The key correct image features and key incorrect image features are respectively verified and identified to obtain the number of real features and the corresponding feature annotation rate data. The data are then compared with the preset feature annotation rate threshold to obtain the corresponding annotation threshold comparison result. If the results meet the preset requirements, the high abnormal image features are identified to obtain the number of real abnormal features and the corresponding abnormal feature annotation rate data. The data are then compared with the preset abnormal feature annotation rate threshold to obtain the third annotation threshold comparison result. If the comparison result of the third annotation threshold meets the preset threshold comparison requirements, then obtain the effective annotation rate data of each category feature and the average effective annotation rate data of each category feature of each crowdsourcing personnel in the professional crowdsourcing group, and filter out the effective crowdsourcing annotators who meet the average effective annotation rate data of each category feature, and extract the corresponding historical effective annotation rate data. Based on the historical effective annotation rate data of each effective crowdsourced annotator, combined with the image quality level, professional annotation difficulty coefficient and average image gap rate index, the professional annotation effective correction coefficient is obtained. Then, it is compared with the preset professional image feature annotation effect detection threshold to determine the overall annotation effectiveness of each effective crowdsourced annotator. If the overall annotation validity is valid, the corresponding features of the multiple images to be filled and the features of the multiple key error images of the valid crowdsourced annotators are extracted to fill and correct the images to obtain the repaired images. The repaired images are compared with the standardized images to obtain the image repair effect results, and the records of each valid crowdsourced annotator are updated.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a crowdsourcing-based image annotation method program, which, when executed by a processor, implements the steps of the crowdsourcing-based image annotation method as described in any one of claims 1 to 8.

Images