Data quality inspection method and device

Abstract

The invention discloses a data quality inspection method and device. The method comprises the steps that an image to be subjected to quality inspection and an original image are acquired, the image to be subjected to quality inspection is obtained by manually annotating the original image, the original image is annotated through an annotation model to obtain an annotated image, index values corresponding to quality inspection indexes of the image to be subjected to quality inspection are determined according to the image to be subjected to quality inspection and the annotated image, and determining a quality inspection result of the image to be subjected to quality inspection according to the index value. Therefore, quality inspection can be automatically carried out on the image to be subjected to quality inspection, manual quality inspection is not needed, and the quality inspection efficiency and precision can be improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a data quality inspection method and device. Background Art

[0002] The field of artificial intelligence (AI) requires a large amount of labeled data to train models. The accuracy of data labeling will directly affect the performance of the model. Therefore, the labeled data should be quality checked before use to avoid incorrect labeling behaviors that may occur during the manual labeling process and affect the performance of the model.

[0003] In the prior art, quality inspection is usually performed manually by checking the labeled data one by one. However, manual quality inspection is prone to errors, so the quality inspection accuracy is not high and the quality inspection efficiency is also relatively low. Summary of the invention

[0004] The embodiments of the present application provide a data quality inspection method and device, which can automatically perform quality inspection on images to be inspected without the need for manual quality inspection, thereby improving quality inspection efficiency and accuracy.

[0005] In a first aspect, an embodiment of the present application provides a data quality inspection method, the method comprising:

[0006] Obtain the image to be inspected and the original image. The image to be inspected is obtained by manually annotating the original image.

[0007] Use the annotation model to annotate the original image to obtain the annotated image.

[0008] According to the image to be inspected and the annotated image, determine the index value corresponding to the quality inspection index of the image to be inspected,

[0009] The quality inspection result of the image to be inspected is determined according to the indicator value.

[0010] In a second aspect, an embodiment of the present application provides a data quality inspection device, the device comprising:

[0011] The acquisition module is used to obtain the image to be inspected and the original image. The image to be inspected is obtained by manually annotating the original image.

[0012] The annotation module is used to annotate the original image using the annotation model to obtain an annotated image.

[0013] The first determination module is used to determine the index value corresponding to the quality inspection index of the image to be inspected according to the image to be inspected and the annotated image.

[0014] The second determination module is used to determine the quality inspection result of the image to be inspected according to the indicator value.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions,

[0016] When the processor executes the computer program instructions, the data quality inspection method as shown in any one of the embodiments of the first aspect is implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the data quality inspection method shown in any one of the embodiments of the first aspect is implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data quality inspection method shown in any one of the embodiments of the first aspect.

[0019] The data quality inspection method and device of the embodiment of the present application can automatically obtain the image to be inspected and the original image, and use the annotation model to annotate the original image to obtain the annotated image, and then determine the index value corresponding to the quality inspection index of the image to be inspected based on the image to be inspected and the annotated image, and then determine the quality inspection result of the image to be inspected based on the index value. In this way, the image to be inspected can be automatically inspected without manual inspection, thereby improving the efficiency and accuracy of quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is one of the flow charts of a data quality inspection method provided by an embodiment of the present application.

[0022] Figure 2 is a schematic diagram of an original image provided by an embodiment of the present application.

[0023] Figure 3 is a schematic diagram of annotating an image provided by an embodiment of the present application.

[0024] Figure 4 is a schematic diagram of a confusion matrix provided by an embodiment of the present application,

[0025] Figure 5 is a schematic diagram of an interactive interface provided by an embodiment of the present application,

[0026] Figure 6is a schematic diagram of a quality inspection report provided by an embodiment of the present application.

[0027] Figure 7 This is a second flowchart of a data quality inspection method provided by an embodiment of the present application.

[0028] Figure 8 is a structural diagram of a data quality inspection device provided by an embodiment of the present application,

[0029] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0031] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0032] As mentioned in the background technology, the most commonly used data quality inspection method currently is for quality inspectors to judge the labeling results of labelers. Therefore, the quality of quality inspection depends on the experience and working status of the quality inspectors, so high requirements are placed on the ability of the quality inspectors.

[0033] The embodiment of the present application provides a data quality inspection method, and the data quality inspection method provided by the embodiment of the present application is introduced below.

[0034] Figure 1 A flow chart of a data quality inspection method provided by an embodiment of the present application is shown.

[0035] like Figure 1 As shown, the execution subject of the data quality inspection method may be a data quality inspection device or a data quality inspection system, and the data quality inspection method may include the following steps:

[0036] S110, obtaining the image to be inspected and the original image,

[0037] S120, annotating the original image using the annotation model to obtain an annotated image,

[0038] S130, determining the index value corresponding to the quality inspection index of the image to be inspected according to the image to be inspected and the annotated image,

[0039] S140: Determine the quality inspection result of the image to be inspected according to the indicator value.

[0040] In this way, the image to be inspected and the original image can be automatically obtained, and the original image can be annotated using the annotation model to obtain the annotated image. Then, the index value corresponding to the quality inspection index of the image to be inspected can be determined based on the image to be inspected and the annotated image, and the quality inspection result of the image to be inspected can be determined based on the index value. In this way, the image to be inspected can be automatically inspected without manual inspection, thereby improving the efficiency and accuracy of quality inspection.

[0041] In S110, the image to be inspected may be obtained by manually annotating the original image. It may be a mask image. The image to be inspected may be the annotated data in the training sample required for model training. In order to avoid affecting the accuracy of model training due to errors in the annotated data in the training sample, the annotated data may be inspected, and the model training may be performed using the inspected annotated data.

[0042] For example, the original image may be a red, green, and blue three-channel image. Figure 2 shown.

[0043] In some implementations, the image to be inspected may be an image labeled with N categories, where N may be a positive integer.

[0044] In S120, the original image is input into the annotation model, and the original image is annotated using the annotation model, and annotated images can be output. The annotated image can be a mask image.

[0045] In some implementations, the annotated image may be an image annotated with N categories.

[0046] For example, the labeled image can be Figure 3As shown, region 310 may be labeled as category A, region 320 may be labeled as category B, region 330 may be labeled as category C, region 340 may be labeled as category D, and other unlabeled regions may be considered as background.

[0047] In some implementations, training samples may be constructed based on historical images, a labeling model may be trained in advance, and the trained labeling model may be used to label the original image.

[0048] In some embodiments, in order to improve the annotation accuracy of annotated images, the annotation model may be a high-precision semantic segmentation model (DeepLabV3).

[0049] The original image is annotated based on DeepLabV3, and the annotated image obtained has higher annotation accuracy.

[0050] In S130, the image to be inspected may be inspected based on the annotated image. The quality inspection index may include one or more. The index value of each quality inspection index corresponding to each annotation category may be determined.

[0051] In some implementations, in order to improve the efficiency and accuracy of quality inspection, S130 may include:

[0052] Based on the image to be inspected and the annotated image, the confusion matrix is ​​calculated.

[0053] According to the confusion matrix, determine the indicator values ​​corresponding to the quality inspection indicators.

[0054] Here, the image to be inspected can be used as the predicted value, the labeled image can be used as the true value, and the confusion matrix between the two can be calculated. Then, the indicator value corresponding to the quality inspection indicator can be determined based on the confusion matrix.

[0055] For example, the confusion matrix can be expressed as Figure 4 As shown in FIG. 1 , TN can represent a true negative example, FN can represent a false negative example, FP can represent a false positive example, and TP can represent a true positive example. The meaning of the i-th row and j-th column is that there are cm[i][j] pixels that should be labeled as category i but are labeled as category j.

[0056] In this way, the confusion matrix can be used to more quickly and accurately evaluate the annotation accuracy of the image to be inspected.

[0057] In some embodiments, in order to further improve the accuracy of quality inspection, the quality inspection index may include at least one of the probability of missing labels, the probability of multiple labels, and the probability of wrong labels. Both the image to be inspected and the labeled image may be images labeled with N categories, where N is a positive integer.

[0058] The probability of missing labels of the first category may be a ratio of the first number of pixels to the second number of pixels, the first number of pixels may be the number of pixels in the image to be inspected that should be marked as the first category but are not marked as such, and the second number of pixels may be the number of pixels in the image to be inspected that are marked as the first category.

[0059] The over-label probability of the second category may be a ratio of the third number of pixels to the fourth number of pixels, where the third number of pixels may be the number of pixels in the image to be inspected that should not be labeled as a category but are labeled as the second category, and the fourth number of pixels may be the number of pixels in the labeled image that are labeled as the second category.

[0060] The mislabeling probability of the third category may be a ratio of the fifth number of pixels to the sixth number of pixels, where the fifth number of pixels may be the number of pixels in the image to be inspected that should be labeled as the third category but are labeled as the fourth category, and the sixth number of pixels may be the number of pixels in the image to be inspected that are labeled as the third category.

[0061] The first category, the second category, and the third category may all be any one of the N categories, and the fourth category may be a category among the N categories except the third category.

[0062] In the prior art, the accuracy of quality inspection through a single quality inspection indicator is poor. The embodiment of the present application can perform quality inspection through multiple quality inspection indicators, so that the quality inspection image can be inspected more comprehensively and accurately, thereby improving the accuracy of quality inspection.

[0063] In some implementations, the calculation formula for the probability of missing a label may be:

[0064]

[0065] Among them, P 漏标i can be the probability of missing labels for category i, i = 1, 2...N, cm[i][0] can be the number of pixels in the image to be inspected that should be labeled as category i but are not labeled as such, It can be the number of pixels marked as category i in the image to be inspected,

[0066] The calculation formula of multiple label probability can be:

[0067]

[0068] Among them, P 多标i can be the multiple labeling probability of category i, cm[0][i] can be the number of pixels in the image to be inspected that should not be labeled with a category but are labeled as category i. It can be the number of pixels labeled as category i in the labeled image,

[0069] The calculation formula for the mislabeling probability can be:

[0070]

[0071] Among them, P 错标i It can be the mislabeling probability of category i, and cm[i][j] can be the number of pixels in the image to be inspected that should be labeled as category i but are labeled as category j.

[0072] In this way, through the above formula, the indicator values ​​corresponding to various quality inspection indicators of various labeling categories can be calculated more accurately.

[0073] In S140, the quality inspection result of the image to be inspected may be determined according to the indicator values ​​corresponding to the various quality inspection indicators. The quality inspection result may include passing the quality inspection or failing the quality inspection.

[0074] In some implementations, in order to more accurately determine the quality inspection result, S140 may include:

[0075] When the indicator values ​​corresponding to each quality inspection indicator of each category are all less than the first threshold value, it is determined that the image to be inspected has passed the quality inspection.

[0076] If the indicator value corresponding to at least one quality inspection indicator of at least one category is greater than the second threshold, it is determined that the image to be inspected has failed the quality inspection.

[0077] When there is at least one quality inspection indicator of at least one category whose corresponding indicator value is not less than the first threshold, and the indicator values ​​corresponding to each quality inspection indicator of each category are not greater than the second threshold, in response to the user's first input of the image to be quality inspected, it is determined whether the image to be quality inspected has passed the quality inspection or failed the quality inspection.

[0078] The second threshold may be greater than the first threshold.

[0079] Here, the first threshold and the second threshold can be determined according to actual needs. Appropriately lowering the first threshold and / or raising the second threshold can improve the accuracy of automatic quality inspection, but appropriately raising the first threshold and / or lowering the second threshold can reduce the workload of manual quality inspection. For example, when the first threshold is 0.2 and the second threshold is 0.8, the automatic quality inspection accuracy is higher than when the first threshold is 0.3 and the second threshold is 0.7, but more samples will require manual quality inspection.

[0080] Specifically, if the indicator values ​​corresponding to each quality inspection indicator of each category are all less than the first threshold value, it can be determined that the image to be inspected has passed the quality inspection, and the image to be inspected can be added to the passed list; if there is at least one quality inspection indicator of at least one category with an indicator value greater than the second threshold value, it can be determined that the image to be inspected has not passed the quality inspection, and the image to be inspected can be added to the failed list.

[0081] If there is at least one quality inspection indicator of at least one category whose corresponding indicator value is not less than the first threshold value, and the indicator values ​​corresponding to each quality inspection indicator of each category are not greater than the second threshold value, the image to be inspected can be added to the pending list, and the image to be inspected in the pending list can be manually inspected. Specifically, the image to be inspected and the original image can be displayed through a visual interactive interface, and the user can make a first input based on the displayed image to be inspected and the original image. If the user determines that the image to be inspected passes the quality inspection, the first input can be used to determine that the image to be inspected passes the quality inspection; if the user determines that the image to be inspected does not pass the quality inspection, the first input can be used to determine that the image to be inspected does not pass the quality inspection.

[0082] For example, the visual interactive interface can be as follows: Figure 5 As shown, the left side may be the original image 510, and the right side may be the image to be inspected 520. In the image to be inspected 520, area 521 may be marked as category A, area 522 may be marked as category B, area 523 may be marked as category C, area 524 may be marked as category D, and other unmarked areas may be considered as background. The user may quickly perform the first input through pre-set shortcut keys with different functions.

[0083] In addition, a rollback shortcut key can be set. If an incorrect operation is performed manually on the image to be inspected, the rollback shortcut key can be used to return to the image to be inspected that has already been inspected and correct the incorrect operation.

[0084] In this way, it is possible to automatically determine whether the image to be inspected has passed the quality inspection based on the quality inspection indicators. In the case that the automatic quality inspection cannot determine the result, manual quality inspection can be performed through human-computer interaction. This combines automatic quality inspection and manual quality inspection, so that the quality inspection results can be determined more accurately. Moreover, only the images to be inspected that cannot be determined by the automatic quality inspection need to be manually inspected, so the workload of manual quality inspection is less, and there is no need to waste too much energy of the quality inspector, thereby improving the quality inspection accuracy of the quality inspector.

[0085] The quality inspection results in the existing technology are only divided into passed and failed, and the reasons for failing the quality inspection cannot be accurately described. This requires additional communication between the data demander and the data labeler. Once the quality inspection results are not accurate enough, it may cause a trust crisis and contradictions, which is not conducive to the long-term cooperation between the data labeler and the data demander.

[0086] In some implementations, in order to reduce the communication cost between the data demander and the data annotator, the method may further include:

[0087] If the image to be inspected fails the quality inspection, a reason why the image to be inspected fails the quality inspection is determined.

[0088] Here, if the quality inspection result of the automatic quality inspection is that the image to be inspected fails the quality inspection, the reason why the image to be inspected fails the quality inspection can be automatically determined based on the quality inspection index corresponding to the index value greater than the second threshold.

[0089] The reasons for failing the quality inspection may be missing labels, wrong labels or multiple labels. Among them, missing labels can refer to the omission of objects that should be labeled, multiple labels can refer to the mislabeling of objects that should not be labeled, and wrong labels can refer to the correct labeling of the range of objects, but the incorrect category labeling.

[0090] Exemplarily, if the indicator value corresponding to the probability of missing labels is greater than the second threshold, it can be determined that the reason why the image to be inspected failed the quality inspection is missing labels; if the indicator value corresponding to the probability of multiple labels is greater than the second threshold, it can be determined that the reason why the image to be inspected failed the quality inspection is multiple labels; if the indicator value corresponding to the probability of wrong labels is greater than the second threshold, it can be determined that the reason why the image to be inspected failed the quality inspection is wrong labels.

[0091] If the quality inspection result of the manual quality inspection is that the image to be inspected has not passed the quality inspection, the reason why the image to be inspected has not passed the quality inspection can be determined based on the user input. Specifically, the input used to determine that the image to be inspected has not passed the quality inspection can include at least one of the following: missing label input, multiple label input, and wrong label input. Missing label input can be used to determine that the image to be inspected has not passed the quality inspection and the reason why the image to be inspected has not passed the quality inspection is missing label, multiple label input can be used to determine that the image to be inspected has not passed the quality inspection and the reason why the image to be inspected has not passed the quality inspection is multiple label, and wrong label input can be used to determine that the image to be inspected has not passed the quality inspection and the reason why the image to be inspected has not passed the quality inspection is wrong label.

[0092] Exemplarily, different shortcut keys may be pre-set to correspond to missing label input, multiple label input, and wrong label input, respectively, and the user may use the shortcut keys to perform quick input.

[0093] In some examples, shortcut keys such as pass, fail-missing mark, fail-wrong mark, fail-multiple marks, and return to previous sheet can be preset.

[0094] In this way, after the data quality inspection, the reason for failing the quality inspection can be automatically determined, avoiding the increase in communication costs between the data demander and the data labeler due to unclear reasons.

[0095] In some implementations, in order to more clearly identify incorrectly marked areas in the image to be inspected that has failed the quality inspection, the method may further include:

[0096] When it is determined that the image to be inspected fails the quality inspection, in response to a second input of the user on the image to be inspected, a target area in the image to be inspected is marked.

[0097] Here, the target area may include an area that is not labeled, is labeled multiple times, and / or is mislabeled.

[0098] Specifically, during the manual quality inspection process, if the user determines that the image to be inspected has not passed the quality inspection, a second input may be performed to mark areas of the image to be inspected that are missing, over-labeled, and / or mislabeled.

[0099] For example, the user may circle the areas with missing labels, multiple labels and / or wrong labels in the image to be inspected by using the mouse.

[0100] In this way, the incorrectly marked areas in the image to be inspected that failed the quality inspection can be more clearly identified.

[0101] In the prior art, quality inspectors are usually required to write quality inspection reports based on the quality inspection results, which consumes additional statistics and writing time, greatly increasing the cost of data quality inspection. In some embodiments, in order to save manpower and material resources, the method may also include:

[0102] Generate a quality inspection report based on the quality inspection results.

[0103] Here, a quality inspection report may be generated based on the quality inspection results of one or more images to be inspected, and the quality inspection report may include the quality inspection results, reasons for failing the quality inspection, and target areas.

[0104] Exemplarily, the quality inspection report may also include at least one of the quality inspection time, the total number of quality inspection images, the total number of images that failed the quality inspection, the total number of images that passed the quality inspection, the number of missing labels, the number of multiple labeled images, the number of incorrectly labeled images, images that failed the quality inspection, the reasons for failing the quality inspection, and the file names and numbers of images that failed the quality inspection.

[0105] For example, a partial schematic diagram of a quality inspection report may be as follows: Figure 6 The quality inspection report can be in word version, which is convenient for editing and improving the quality inspection report.

[0106] In addition, the quality inspection report can be sent to the data annotator so that the data annotator can correct the image to be inspected according to the quality inspection report, and the corrected image can be used as the image to be inspected for re-inspection.

[0107] In this way, quality inspection reports can be automatically generated without the need for manual writing, saving manpower and material resources.

[0108] In order to better describe the entire solution, based on the above embodiments, a specific example is given. Figure 7 As shown, the data quality inspection method may include S701-S710, which will be explained in detail below.

[0109] S701, obtaining an image to be inspected.

[0110] S702, obtaining an original image.

[0111] S703: Annotate the original image using the annotation model to obtain an annotated image.

[0112] The order of S701 and S702 - S703 is not limited here.

[0113] S704: According to the image to be inspected and the annotated image, an indicator value corresponding to the quality inspection indicator may be determined.

[0114] S705, determining the magnitude relationship between the indicator value and the first threshold and the second threshold.

[0115] If the indicator value is greater than the second threshold, execute S706; if the indicator value is less than the first threshold, execute S707; if the indicator value is not less than the first threshold and not greater than the second threshold, execute S708.

[0116] S706, adding the image to be inspected to the failed list, and then executing S710.

[0117] S707, adding the image to be inspected to the passed list, and then executing S710.

[0118] S708, adding the image to be inspected to the pending list, and then executing S709.

[0119] S709: Manual review of the image to be inspected.

[0120] If the review is passed, execute S707; if the review is not passed, execute S706.

[0121] S710, generating a quality inspection report.

[0122] The data quality inspection method provided in the embodiment of the present application can combine automatic quality inspection with manual quality inspection. For images that fail the quality inspection, the reasons for failing the quality inspection can be determined, and a quality inspection report in word format can be automatically formed in the end. Compared with simple manual quality inspection, the data quality inspection method provided in the embodiment of the present application can save a lot of time to achieve the effect of cost saving, while saving costs and ensuring the high quality of the quality inspection effect. Moreover, the quality inspection report can intuitively count and display the quality inspection results, and is also convenient for the connection and communication between the data annotation party and the data quality inspection party, which can speed up the speed of the entire quality inspection process and provide document records.

[0123] Based on the same inventive concept, the present application embodiment also provides a data quality inspection device. Figure 8 The data quality inspection device provided in the embodiment of the present application is described in detail.

[0124] Figure 8 A schematic structural diagram of a data quality inspection device provided in one embodiment of the present application is shown.

[0125] like Figure 8 As shown, the data quality inspection device may include:

[0126] The acquisition module 801 is used to acquire the image to be inspected and the original image. The image to be inspected is obtained by manually annotating the original image.

[0127] The annotation module 802 is used to annotate the original image using the annotation model to obtain an annotated image.

[0128] The first determination module 803 is used to determine the index value corresponding to the quality inspection index of the image to be inspected according to the image to be inspected and the annotated image.

[0129] The second determination module 804 is used to determine the quality inspection result of the image to be inspected according to the indicator value.

[0130] In this way, the image to be inspected and the original image can be automatically obtained, and the original image can be annotated using the annotation model to obtain the annotated image. Then, the index value corresponding to the quality inspection index of the image to be inspected can be determined based on the image to be inspected and the annotated image, and the quality inspection result of the image to be inspected can be determined based on the index value. In this way, the image to be inspected can be automatically inspected without manual inspection, thereby improving the efficiency and accuracy of quality inspection.

[0131] In some implementations, in order to improve the efficiency and accuracy of quality inspection, the first determination module 803 may include:

[0132] The calculation submodule is used to calculate the confusion matrix based on the image to be inspected and the annotated image.

[0133] The first determination submodule is used to determine the indicator value corresponding to the quality inspection indicator according to the confusion matrix.

[0134] In some embodiments, in order to further improve the accuracy of quality inspection, the quality inspection index includes at least one of the probability of missing labels, the probability of multiple labels, and the probability of wrong labels. The image to be inspected and the labeled image are both images labeled with N categories, where N is a positive integer.

[0135] The probability of missing labels of the first category is the ratio of the first number of pixels to the second number of pixels. The first number of pixels is the number of pixels in the image to be inspected that should be marked as the first category but are not marked as such. The second number of pixels is the number of pixels in the image to be inspected that are marked as the first category.

[0136] The over-labeling probability of the second category is the ratio of the third number of pixels to the fourth number of pixels. The third number of pixels is the number of pixels in the image to be inspected that should not be labeled but are labeled as the second category. The fourth number of pixels is the number of pixels in the labeled image that are labeled as the second category.

[0137] The mislabeling probability of the third category is the ratio of the fifth pixel number to the sixth pixel number. The fifth pixel number is the number of pixels in the image to be inspected that should be labeled as the third category but are labeled as the fourth category. The sixth pixel number is the number of pixels in the image to be inspected that are labeled as the third category.

[0138] The first category, the second category, and the third category are any one of the N categories, and the fourth category is a category among the N categories except the third category.

[0139] In some implementations, the calculation formula for the probability of missing a label is:

[0140]

[0141] Among them, P 漏标i is the probability of missing label of category i, i = 1, 2…N, cm[i][0] is the number of pixels in the image to be inspected that should be labeled as category i but are not labeled as such, is the number of pixels marked as category i in the image to be inspected,

[0142] The calculation formula of multiple-label probability is:

[0143]

[0144] Among them, P 多标i is the probability of multiple labeling of category i, cm[0][i] is the number of pixels in the image to be inspected that should not be labeled with a category but are labeled with category i, is the number of pixels labeled as category i in the labeled image,

[0145] The formula for calculating the probability of mislabeling is:

[0146]

[0147] Among them, P 错标i is the mislabeling probability of category i, cm[i][j] is the number of pixels in the image to be inspected that should be labeled as category i but are labeled as category j.

[0148] In some implementations, in order to more accurately determine the quality inspection result, the second determination module 804 may include:

[0149] The second determination submodule is used to determine that the image to be inspected has passed the quality inspection when the indicator values ​​corresponding to the various quality inspection indicators of each category are all less than the first threshold value.

[0150] The third determination submodule is used to determine that the image to be inspected fails the quality inspection if the indicator value corresponding to at least one quality inspection indicator of at least one category is greater than a second threshold, and the second threshold is greater than the first threshold.

[0151] The fourth determination submodule is used to determine whether the image to be inspected has passed the quality inspection or failed the quality inspection in response to the user's first input on the image to be inspected, when the indicator value corresponding to at least one quality inspection indicator of at least one category is not less than the first threshold, and the indicator values ​​corresponding to each quality inspection indicator of each category are not greater than the second threshold.

[0152] In some implementations, in order to reduce the communication cost between the data demander and the data annotator, the device may further include:

[0153] The third determination module is used to determine the reason why the image to be inspected fails the quality inspection if the image to be inspected fails the quality inspection.

[0154] In some embodiments, in order to more clearly identify the incorrectly marked areas in the image to be inspected that has failed the quality inspection, the device may further include:

[0155] The marking module is used to mark the target area in the image to be inspected in response to the user's second input on the image to be inspected when it is determined that the image to be inspected has not passed the quality inspection. The target area includes an area with missing marks, multiple marks and / or wrong marks.

[0156] In some embodiments, in order to save manpower and material resources, the device may further include:

[0157] The generation module is used to generate a quality inspection report based on the quality inspection results.

[0158] In some implementations, in order to improve the annotation accuracy of annotated images, the annotation model is DeepLabV3.

[0159] Fig. 9 A schematic structural diagram of an electronic device provided by an embodiment of the present application is shown.

[0160] like Fig. 9 As shown, the electronic device 9 is a block diagram of an exemplary hardware architecture of an electronic device that can implement the data quality inspection method and data quality inspection device in the embodiment of the present application. The electronic device may refer to the electronic device in the embodiment of the present application.

[0161] The electronic device 9 may include a processor 901 and a memory 902 storing computer program instructions.

[0162] Specifically, the processor 901 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0163] The memory 902 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 902 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 902 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 902 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 902 is a non-volatile solid-state memory. In a specific embodiment, the memory 902 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Therefore, typically, the memory 902 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present application.

[0164] The processor 901 implements any one of the data quality inspection methods in the above embodiments by reading and executing computer program instructions stored in the memory 902 .

[0165] In one example, the electronic device may further include a communication interface 903 and a bus 904. Fig. 9 As shown, the processor 901, the memory 902, and the communication interface 903 are connected via a bus 904 and communicate with each other.

[0166] The communication interface 903 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0167] Bus 904 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In suitable cases, bus 904 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0168] The electronic device can execute the data quality inspection method in the embodiment of the present application, thereby realizing the combination Figures 1 to 8 The data quality inspection method and device described.

[0169] In addition, in combination with the data quality inspection method in the above embodiment, the embodiment of the present application can provide a computer storage medium to implement. The computer storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, any one of the data quality inspection methods in the above embodiment is implemented.

[0170] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0171] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0172] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0173] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0174] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A data quality inspection method, characterized in that: The method comprises: Obtain an image to be inspected and an original image, wherein the image to be inspected is obtained by manually annotating the original image. Annotating the original image using the annotation model to obtain an annotated image, Determine the index value corresponding to the quality inspection index of the image to be inspected according to the image to be inspected and the annotated image, The quality inspection result of the image to be inspected is determined according to the indicator value.

2. The method according to claim 1, characterized in that The step of determining, according to the image to be inspected and the annotated image, an indicator value corresponding to a quality inspection indicator of the image to be inspected includes: Based on the image to be inspected and the annotated image, a confusion matrix is ​​calculated. According to the confusion matrix, the indicator value corresponding to the quality inspection indicator is determined.

3. The method according to claim 1, characterized in that The quality inspection index includes at least one of the probability of missing labels, the probability of multiple labels and the probability of wrong labels. The image to be inspected and the labeled image are both images labeled with N categories, where N is a positive integer. The probability of missing labels of the first category is the ratio of the first number of pixels to the second number of pixels, the first number of pixels is the number of pixels in the image to be inspected that should be marked as the first category but are not marked, and the second number of pixels is the number of pixels in the image to be inspected that are marked as the first category. The multiple labeling probability of the second category is the ratio of the third number of pixels to the fourth number of pixels, the third number of pixels is the number of pixels in the image to be inspected that should not be labeled as the second category but are labeled as the second category, and the fourth number of pixels is the number of pixels in the labeled image that are labeled as the second category. The mislabeling probability of the third category is the ratio of the fifth number of pixels to the sixth number of pixels, the fifth number of pixels being the number of pixels in the image to be inspected that should be labeled as the third category but are labeled as the fourth category, and the sixth number of pixels being the number of pixels in the image to be inspected that are labeled as the third category. The first category, the second category and the third category are any one of the N categories, and the fourth category is a category among the N categories except the third category.

4. The method according to claim 3, characterized in that The calculation formula of the missing label probability is: Among them, P 漏标i is the probability of missing label of category i, i=1, 2…N, cm[i][0] is the number of pixels in the image to be inspected that should be marked as category i but are not marked as such, is the number of pixels marked as category i in the image to be inspected, The calculation formula of the multiple label probability is: Among them, P 多标i is the probability of multiple labeling of category i, cm[0][i] is the number of pixels in the image to be inspected that should not be labeled with a category but are labeled as category i, is the number of pixels labeled as category i in the labeled image, The calculation formula of the mislabeling probability is: Among them, P 错标i is the mislabeling probability of category i, and cm[i][j] is the number of pixels in the image to be inspected that should be labeled as category i but are labeled as category j.

5. The method according to claim 1, characterized in that Determining the quality inspection result of the image to be inspected according to the indicator value includes: When the indicator values ​​corresponding to the various quality inspection indicators of each category are all less than the first threshold, it is determined that the image to be inspected has passed the quality inspection. In the case that the indicator value corresponding to at least one quality inspection indicator of at least one category is greater than a second threshold, it is determined that the image to be inspected fails the quality inspection, and the second threshold is greater than the first threshold. When the indicator value corresponding to at least one quality inspection indicator of at least one category is not less than the first threshold, and the indicator values ​​corresponding to each quality inspection indicator of each category are not greater than the second threshold, in response to the user's first input to the image to be quality inspected, it is determined whether the image to be quality inspected has passed the quality inspection or failed the quality inspection.

6. The method according to claim 5, characterized in that The method further comprises: In the case that the image to be inspected fails the quality inspection, a reason why the image to be inspected fails the quality inspection is determined.

7. The method according to claim 5, characterized in that The method further comprises: When it is determined that the image to be inspected fails the quality inspection, in response to a second input of the user on the image to be inspected, a target area in the image to be inspected is marked, wherein the target area includes an area that is missing, overly marked and / or mislabeled.

8. The method according to claim 1, characterized in that: The method further comprises: A quality inspection report is generated based on the quality inspection results.

9. The method according to claim 1, characterized in that: The annotation model is the high-precision semantic segmentation model DeepLabV3.

10. A data quality inspection device, characterized in that: The device comprises: The acquisition module is used to acquire the image to be inspected and the original image. The image to be inspected is obtained by manually annotating the original image. A labeling module is used to label the original image using a labeling model to obtain a labeled image. A first determination module is used to determine an indicator value corresponding to a quality inspection indicator of the image to be inspected according to the image to be inspected and the annotated image. The second determination module is used to determine the quality inspection result of the image to be inspected according to the indicator value.

Images