Quality control application evaluation method and device, computer equipment and readable storage medium

Abstract

The invention provides a quality control application evaluation method and device, computer equipment and a readable storage medium, and belongs to the technical field of application testing. The method comprises the following steps: acquiring a test image marked with a quality label; wherein the test image marked with the quality labels is a test image obtained by carrying out image marking based on a plurality of preset quality labels, and the plurality of quality labels are respectively used for representing at least one image quality condition; performing data enhancement processing on the test images marked with the quality labels to obtain test images which correspond to the plurality of quality labels and are distributed in a balanced manner; batch test processing is carried out on the test images with the balanced number distribution based on a quality control application, and an index evaluation result of the quality control application is obtained; wherein the index evaluation result at least comprises detailed business index evaluation results corresponding to the plurality of quality labels. According to the invention, the effectiveness and rationality of evaluating the quality control application can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of application program testing, and in particular to a method, apparatus, computer device, and readable storage medium for evaluating a quality control application. Background Art

[0002] In the field of smart insurance claims, users can upload images of claim materials to the quality control application provided by the insurance service provider, so that the quality of the claim material images can be automatically reviewed online through the quality control application. In related technologies, even if the evaluation of quality control applications can calculate the corresponding test indicators, the annotation results of the test data are too coarse (only the test data is marked with a data label of whether the quality control has passed), which makes it unclear about the specific distribution of the test data, and even more unclear about whether the distribution of the test data is reasonable, making it difficult to ensure the effectiveness and rationality of the evaluation of the quality control application based on the test data. Therefore, how to improve the effectiveness and rationality of evaluating quality control applications has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a quality control application evaluation method, device, computer equipment and readable storage medium, aiming to improve the effectiveness and rationality of evaluating quality control applications.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application provides a method for evaluating a quality control application, the method comprising:

[0005] Acquire a test image annotated with a quality label; wherein the test image annotated with the quality label is a test image obtained by annotating the image based on a plurality of preset quality labels, and each of the plurality of quality labels is used to characterize at least one image quality condition;

[0006] Performing data enhancement processing on the test image annotated with the quality labels to obtain test images with a balanced distribution of quantities corresponding to each of the multiple quality labels;

[0007] Based on the quality control application, batch testing is performed on the test images with a balanced distribution in quantity to obtain an indicator evaluation result of the quality control application; wherein the indicator evaluation result at least includes detailed business indicator evaluation results corresponding to the multiple quality labels.

[0008] In some embodiments, the multiple quality labels include at least two of quality compliance, poor integrity, poor exposure, and poor clarity; and the labeling of the test image based on the preset multiple quality labels includes:

[0009] Performing image quality detection processing on a test image belonging to a preset insurance claim material type to obtain a quality detection result of the test image; wherein the preset insurance claim material type includes an identity card page type, a household register type, an invoice type, and a document type;

[0010] In a case where the quality detection result of the test image is that the quality does not meet the standard, pre-marking the test image with a label value of poor integrity, a label value of poor exposure and / or a label value of poor clarity based on the quality detection result;

[0011] When the quality detection result of the test image is that the quality meets the standard and the test image passes the keyword verification, a label value indicating that the quality meets the standard is pre-marked on the test image.

[0012] In some embodiments, after annotating the image based on a plurality of preset quality labels, the method further includes at least one of the following:

[0013] In the case of receiving a quality label annotation error correction operation, switching the quality label annotated on the first target test image to the quality label indicated by the quality label annotation error correction operation; wherein the quality label annotation error correction operation is an interactive operation received by a visualization interface based on a preset annotation tool, and the first target test image is a test image annotated with the quality label and displayed in the visualization interface;

[0014] When the number of second target test images is less than a preset threshold value of the number of images that meet the quality standard, images that meet the quality standard are obtained, and label values ​​with quality standards are pre-marked for the images that meet the quality standard; wherein the second target test images are images with pre-marked label values ​​with quality standards in the test images after the quality labels are annotated, and the sum of the number of the second target test images and the number of the images that meet the quality standard is greater than or equal to the threshold value of the number of images that meet the quality standard.

[0015] In some embodiments, performing data enhancement processing on the test image annotated with the quality label includes:

[0016] Acquire a first number of test images corresponding to each of the plurality of quality labels in the test image annotated with the quality labels; wherein the sum of the first numbers is equal to the total number of the test images annotated with the quality labels;

[0017] Data enhancement processing is performed on the test images corresponding to each of the multiple quality labels until a second number of the test images corresponding to each of the multiple quality labels is the same as a maximum value of the first numbers.

[0018] In some embodiments, the quality control application includes a plurality of quality control application program interfaces, and the number of the plurality of quality control application program interfaces is greater than or equal to the number of categories to which the test image belongs;

[0019] The batch test processing is performed on the test images with balanced distribution in quantity based on the quality control application to obtain the indicator evaluation result of the quality control application, including:

[0020] Based on a target quality control application program interface, a batch test process is performed on the test images belonging to the target category among the test images with a balanced number of distributions; wherein the target quality control application program interface is any one of the multiple quality control application program interfaces, the target category is any one of the categories to which the test images belong, and the target quality control application program interface is dedicated to testing the test images of the target category;

[0021] Counting the test processing results of the test images of the target category; wherein the test processing results are used to characterize the quality control pass status of the test images corresponding to each of the multiple quality labels;

[0022] Based on the test processing results, detailed business indicator evaluation results of the quality control application corresponding to the multiple quality tags are calculated.

[0023] In some embodiments, the method further comprises:

[0024] Acquire historical image data to construct an initial test data set; wherein the initial test data set includes image data of multiple categories;

[0025] The image data in the initial test data set is classified to obtain test images of multiple categories.

[0026] In some embodiments, image annotation is performed based on a plurality of preset quality labels, including:

[0027] For a test image of a target category, image annotation is performed based on a plurality of preset quality labels; wherein the target category is any one of the plurality of categories;

[0028] Selecting test images of other categories from the initial test data set, and marking the test images of other categories with non-target category labels; wherein the other category is at least one category other than the target category among the multiple categories.

[0029] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an evaluation device for a quality control application, the device comprising:

[0030] An acquisition module, used to acquire a test image annotated with a quality label; wherein the test image annotated with the quality label is a test image obtained by annotating the image based on a plurality of preset quality labels, and each of the plurality of quality labels is used to characterize at least one image quality condition;

[0031] A data enhancement module, used to perform data enhancement processing on the test image annotated with the quality labels to obtain a test image with a balanced distribution of quantities corresponding to the multiple quality labels;

[0032] A quality control evaluation module is used to perform batch testing on the test images with a balanced distribution of quantity based on a quality control application to obtain an indicator evaluation result of the quality control application; wherein the indicator evaluation result at least includes detailed business indicator evaluation results corresponding to the multiple quality labels respectively.

[0033] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0034] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0035] In addition, to achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application also proposes a computer program product, which includes a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0036] The evaluation method, apparatus, computer equipment, computer-readable storage medium and computer program product for quality control applications proposed in the embodiments of the present application obtain a test image annotated with quality labels; wherein the test image annotated with quality labels is a test image obtained by annotating the image based on multiple preset quality labels, and each of the multiple quality labels is used to characterize at least one image quality condition; data enhancement processing is performed on the test image annotated with quality labels to obtain test images with a uniform distribution of quantities corresponding to each of the multiple quality labels; based on the quality control application, batch testing is performed on the test images with a uniform distribution of quantities to obtain indicator evaluation results of the quality control application; wherein the indicator evaluation results at least include detailed business indicator evaluation results corresponding to the multiple quality labels.

[0037] Compared with evaluating the quality control application by whether the quality control is annotated with test data through this kind of coarse dimensional data label, the embodiment of the present application obtains a test image with quality labels obtained by image annotation based on multiple quality labels (each quality label represents at least one image quality condition), and performs data enhancement processing on the test image to make the number of test images corresponding to the multiple quality labels evenly distributed, and then evaluates the quality control application through the test images with evenly distributed numbers, that is, based on the quality control application, the test images with evenly distributed numbers are run batch test processing to obtain the indicator evaluation results of the quality control application. Among them, the indicator evaluation results at least include detailed business indicator evaluation results corresponding to multiple quality labels.

[0038] In this way, the embodiment of the present application annotates the test images with multiple quality labels of finer dimensions, and makes the data distribution more balanced and reasonable by performing data enhancement on the test images annotated with different quality labels, and obtains detailed business indicator evaluation results corresponding to multiple quality labels by performing batch tests on the test images based on the quality control application, thereby performing statistics on the detailed business indicators of the quality control application according to the fine dimensions of multiple quality labels, which can improve the effectiveness and rationality of the evaluation of the quality control application, and is also conducive to the analysis of the reasons for the low indicator evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the hierarchical structure of the SDK quality control model involved in the quality control application evaluation method provided in the embodiment of the present application;

[0040] Figure 2 A schematic diagram of the steps of the quality control application evaluation method provided in some embodiments of the present application;

[0041] Figure 3 A schematic diagram of the steps of the evaluation method for the quality control application provided in the embodiment of the present application in other embodiments;

[0042] Figure 4 A schematic flow chart of the steps of image annotation involved in some embodiments of the quality control application evaluation method provided in the embodiments of the present application;

[0043] Figure 5 A schematic flow chart of the steps of marking quality labels involved in some embodiments of the quality control application evaluation method provided in the embodiments of the present application;

[0044] Figure 6 for Figure 2 Schematic diagram of detailed process flow of step S202;

[0045] Figure 7 for Figure 2 Schematic diagram of detailed process flow of step S203;

[0046] Figure 8 It is a structural schematic diagram of an evaluation device for quality control applications provided in an embodiment of the present application;

[0047] Fig. 9 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0051] First, several nouns involved in the embodiments of this application are analyzed:

[0052] Quality control applications.

[0053] In the field of smart insurance claims, quality control applications mainly represent applications with insurance claims material quality control functions. The insurance claims material quality control function means that when a customer files a claim after an accident, he or she needs to take photos with a mobile phone or other terminal device, and upload different types of claims material pictures through various material upload portals of the insurance institution / company quality control application, including the ID card portrait page, national emblem page, household register, invoice, medical records, etc. The quality control application calls the underlying software development kit (SDK) quality control model to perform quality inspection on the material pictures to be uploaded by the customer, intercepts and gives corresponding prompts for the uploaded pictures with the wrong type or substandard quality (such as blurred photos), so that the customer can re-take and upload pictures of the correct type and quality.

[0054] SDK quality control model.

[0055] like Figure 1As shown, the SDK quality control model of the lower layer of the quality control application provides an application programming interface (API) to the upper layer of the quality control application. The API can be called by the application layer to provide classification + quality control functions of material images. Only when the classification is correct and the quality meets the standard will the image quality control pass result be returned. Otherwise, if the classification is incorrect or the quality does not meet the standard, the quality control fail result will be returned.

[0056] AI.

[0057] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0058] Next, the overall concept of the embodiment of the present application is described.

[0059] In the field of smart insurance claims, users can upload images of claim materials to the quality control application provided by the insurance service provider, so that the quality control application can automatically review the quality of the claim material images online. In related technologies, the evaluation process for quality control applications is generally as follows:

[0060] 1. Collect the initial test data set, and manually classify, clean, and annotate the initial test data set. The images are divided into two sub-datasets: classification and quality-compliant, and classification or quality-uncompliant. The images are located in two folders respectively.

[0061] 2. Use the installed quality control application to read the data in batches twice and call the SDK quality control model API, once to test the quality standard data set folder, and once to test the quality standard data set folder, and obtain the quality control pass or fail results respectively.

[0062] 3. Based on the number of two sub-test sets and the number of test results output by the SDK quality control model, calculate the business indicators and general indicator results of the ID card portrait page quality control model.

[0063] 4. Similarly, test the evaluation indicators of other types of material images.

[0064] 5. If the indicator is too low, you need to analyze the bad cases in the test results one by one, find out the causes and optimize the SDK quality control model.

[0065] In this way, even if the evaluation of quality control applications can calculate the corresponding test indicators, the labeling result of the test data is too coarse (only the test data is marked with a data label of whether the quality control has passed), which makes it unclear about the specific distribution of the test data, and even more unclear about whether the distribution of the test data is reasonable, making it difficult to ensure the effectiveness and rationality of the evaluation of the quality control application based on the test data. Therefore, how to improve the effectiveness and rationality of evaluating quality control applications has become a technical problem that needs to be solved urgently.

[0066] Based on this, the embodiments of the present application provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for evaluating quality control applications, aiming to improve the effectiveness and rationality of evaluating quality control applications.

[0067] The embodiment of the present application obtains a test image after quality labels are annotated; wherein the test image after quality labels are obtained by annotating the image based on multiple preset quality labels, and each of the multiple quality labels is used to characterize at least one image quality condition; data enhancement processing is performed on the test image after quality labels are annotated to obtain test images with a balanced distribution of quantities corresponding to each of the multiple quality labels; based on a quality control application, a batch test is performed on the test images with a balanced distribution of quantities to obtain an indicator evaluation result of the quality control application; wherein the indicator evaluation result at least includes detailed business indicator evaluation results corresponding to the multiple quality labels.

[0068] Compared with evaluating the quality control application by whether the quality control is annotated with test data through this kind of coarse dimensional data label, the embodiment of the present application obtains a test image with quality labels obtained by image annotation based on multiple quality labels (each quality label represents at least one image quality condition), and performs data enhancement processing on the test image to make the number of test images corresponding to the multiple quality labels evenly distributed, and then evaluates the quality control application through the test images with evenly distributed numbers, that is, based on the quality control application, the test images with evenly distributed numbers are run batch test processing to obtain the indicator evaluation results of the quality control application. Among them, the indicator evaluation results at least include detailed business indicator evaluation results corresponding to multiple quality labels.

[0069] In this way, the embodiment of the present application annotates the test images with multiple quality labels of finer dimensions, and makes the data distribution more balanced and reasonable by performing data enhancement on the test images annotated with different quality labels, and obtains detailed business indicator evaluation results corresponding to multiple quality labels by performing batch tests on the test images based on the quality control application, thereby performing statistics on the detailed business indicators of the quality control application according to the fine dimensions of multiple quality labels, which can improve the effectiveness and rationality of the evaluation of the quality control application, and is also conducive to the analysis of the reasons for the low indicator evaluation results.

[0070] Next, based on the overall concept of the above-mentioned embodiments of the present application, specific embodiments of the evaluation method, device, computer equipment, computer readable storage medium and computer program product of the quality control application provided by the embodiments of the present application are proposed. First, each specific embodiment of the evaluation method of the quality control application in the embodiments of the present application is described in detail.

[0071] It should be noted that in each specific implementation of the evaluation method of the quality control application provided in the embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0072] In addition, the evaluation method of the quality control application provided in the embodiment of the present application relates to the field of application testing technology. The embodiment of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0073] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0074] The evaluation method of the quality control application provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the evaluation method of the quality control application, etc., but is not limited to the above forms.

[0075] Alternatively, the evaluation method of the quality control application provided by the embodiment of the present application can also be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor-based or microprocessor-based systems (such as insurance systems), set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0076] For ease of understanding and explanation, the following text will take the evaluation method of the quality control application provided by the embodiment of the present application as an example for a terminal device with a quality control application installed as an example for detailed explanation. The implementation of the evaluation method of the quality control application provided by the embodiment of the present application in any of the above-mentioned forms of the subject application can refer to the process of the evaluation method of the quality control application applied by the terminal device described later.

[0077] Please refer to Figure 2 , Figure 2 A schematic diagram of the steps of the quality control application evaluation method provided in some embodiments of the present application. It should be understood that although Figure 2 The execution order of some method steps is shown in the figure, but based on the different design requirements of actual applications, the evaluation method for quality control application provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. That is, Figure 1The order of the steps in the method shown does not constitute a limitation on the execution logic order of the evaluation method for the quality control application provided in the embodiment of the present application. Figure 1 Reasonable changes in the sequence of steps in the method shown should be included in the protection scope of the evaluation method for quality control applications provided in the embodiments of the present application.

[0078] like Figure 2 As shown, in some embodiments, the evaluation method for the quality control application provided in the embodiments of the present application may include but is not limited to steps S201 to S203.

[0079] Step S201: Acquire a test image annotated with a quality label; wherein the test image annotated with a quality label is a test image obtained by annotating the image based on a plurality of preset quality labels, and each of the plurality of quality labels is used to characterize at least one image quality condition.

[0080] It should be noted that the preset multiple quality labels are data labels used to characterize the quality conditions of the test image (for example, quality meets the standard, poor completeness, poor exposure, poor clarity, etc.), and each of the multiple quality labels is used to characterize at least one image quality condition.

[0081] When evaluating the quality control function of the insurance claim material of the quality control application, the terminal device first obtains a test image that has been annotated with a quality label. The test image obtained by the terminal device is a test image obtained by an image annotating operation based on multiple quality labels.

[0082] In some embodiments, the terminal device can configure a pre-labeling module and call the sub-classification optical character recognition (OCR), quality inspection and other service capabilities of the artificial intelligence (AI) system through the pre-labeling module to automatically pre-label one or more of the multiple quality labels for the image, and the pre-labeling result (the test image with the quality label) is saved locally in Doccano JSONL format. In this way, when the terminal device evaluates the quality control function of the insurance claim material of the quality control application, it can directly obtain the test image with the quality label locally.

[0083] In other embodiments, the terminal device can also receive manually uploaded test images with quality labels through a human-computer interaction interface. Among them, the staff who evaluate the quality control function of insurance claims materials for quality control applications can manually perform image annotation operations based on multiple quality labels in advance to obtain test images with quality labels, and then upload the test images to the terminal device. Alternatively, the staff can also collect test images marked with quality labels based on a big data platform / system (insurance platform / system) and upload the test images to the terminal device.

[0084] Step S202: performing data enhancement processing on the test image annotated with the quality labels to obtain test images with a balanced distribution of quantities corresponding to each of the plurality of quality labels.

[0085] After the terminal device obtains the test images labeled with quality labels, if the number distribution of the test images labeled with different quality labels is uneven, the terminal device performs data enhancement processing on the test images labeled with quality labels, so as to obtain the test images with balanced number distribution corresponding to the multiple quality labels. For example, assuming that among the test images labeled with quality labels, the number of test images labeled with the first quality label is 500, but the number of test images labeled with the second quality label is only 300, the terminal device determines that the number distribution of the test images labeled with the first quality label and the second quality label is uneven, so that the terminal device can perform data enhancement processing on the test images labeled with the second quality label, which is smaller in number, so that the number of test images labeled with the second quality label also reaches 500. In this way, the terminal device obtains the test images with balanced number distribution corresponding to the first quality label and the second quality label (500 test images labeled with the first quality label, and 500 test images labeled with the second quality label).

[0086] Step S203: performing batch testing on the test images with balanced distribution in quantity based on the quality control application to obtain the indicator evaluation results of the quality control application; wherein the indicator evaluation results at least include detailed business indicator evaluation results corresponding to the multiple quality labels.

[0087] For the test images that have been labeled with quality labels and are evenly distributed, the terminal device further divides the test images into multiple batches based on the different labeled quality labels, and then runs batch tests on the test data of multiple batches based on the quality control application, thereby obtaining the indicator evaluation results for the quality control function of the insurance claim materials of the quality control application. Since the terminal device tests each quality label as a batch, the final indicator evaluation results at least include detailed business indicator evaluation results corresponding to multiple quality labels one by one.

[0088] It should be noted that when the terminal device evaluates the insurance claim material quality control function of the quality control application based on the test images with well-labeled quality labels and balanced quantity distribution, it must at least measure the business evaluation indicators such as the quality control pass rate and interception rate of the quality control application SDK quality control model, or appropriate industry-wide evaluation indicators such as recall rate, precision rate, and accuracy rate, so as to evaluate the effectiveness of the tested SDK quality control model in multiple dimensions. Among them, the evaluation results of the insurance claim material quality control function based on the SDK quality control model are as follows:

[0089] The test image type is correct and the image quality meets the requirements (for example, the number is m), and the SDK quality control passes (for example, the number is n);

[0090] The test image type is correct and the image quality meets the standard m, but the SDK quality control fails mn;

[0091] The test image type is incorrect or the image quality does not meet the standard i, and the SDK quality control fails (i.e., is blocked) j;

[0092] The test image type is incorrect or the image quality does not meet the standard i, and the SDK quality control passes (that is, it is not intercepted) ij.

[0093] Based on this, in the indicator evaluation results, the business evaluation indicators are: pass rate = n / m (explanation: the ratio of the number of quality control that should pass to the actual predicted pass), interception rate = j / i (explanation: the ratio of the number of quality control interceptions that should be intercepted). The industry-wide evaluation indicators corresponding to each quality label (detailed business indicator evaluation results) are: recall rate of quality control pass = TP / (TP+FN) (explanation: TP is the number of quality control passes that are predicted and accurate n, TP+FN is the number of quality control that should pass m), precision rate of quality control pass = TP / (TP+FP) (explanation: TP is the number of quality control passes that are predicted and accurate n, TP+FP is the number of quality control passes predicted n+(ij)), and overall accuracy = (TP+TN) / (TP+TN+FP+FN) (explanation: TP+TN is the number of accurate predictions n+j, TP+TN+FP+FN is the number of all samples m+i).

[0094] In an embodiment of the present application, when the quality control function of the insurance claim materials of the quality control application is evaluated by the terminal device, a test image with a quality label is first obtained. Among them, the test image obtained by the terminal device is a test image obtained by an operation of image annotation based on multiple quality labels. Afterwards, in the case where the number distribution of test images annotated with different quality labels is uneven, the terminal device performs data enhancement processing on the test images annotated with the quality labels, thereby obtaining test images with a balanced number distribution corresponding to each of the multiple quality labels. Finally, the terminal device further divides the test images into multiple batches according to the different annotated quality labels, and then performs batch test processing on the test data of the multiple batches based on the quality control application, thereby obtaining the indicator evaluation results for the quality control function of the insurance claim materials of the quality control application. Since the terminal device performs the test with each quality label as a batch, the indicator evaluation results finally obtained at least include detailed business indicator evaluation results corresponding to multiple quality labels one by one.

[0095] Compared with evaluating the quality control application by whether the quality control is annotated with test data through this kind of coarse dimensional data label, the embodiment of the present application obtains a test image with quality labels obtained by image annotation based on multiple quality labels (each quality label represents at least one image quality condition), and performs data enhancement processing on the test image to make the number of test images corresponding to the multiple quality labels evenly distributed, and then evaluates the quality control application through the test images with evenly distributed numbers, that is, based on the quality control application, the test images with evenly distributed numbers are run batch test processing to obtain the indicator evaluation results of the quality control application. Among them, the indicator evaluation results at least include detailed business indicator evaluation results corresponding to multiple quality labels.

[0096] In this way, the embodiment of the present application annotates the test images with multiple quality labels of finer dimensions, and makes the data distribution more balanced and reasonable by performing data enhancement on the test images annotated with different quality labels, and obtains detailed business indicator evaluation results corresponding to multiple quality labels by performing batch tests on the test images based on the quality control application, thereby performing statistics on the detailed business indicators of the quality control application according to the fine dimensions of multiple quality labels, which can improve the effectiveness and rationality of the evaluation of the quality control application, and is also conducive to the analysis of the reasons for the low indicator evaluation results.

[0097] In addition, in an embodiment of the present application, a pre-labeling module is configured through the terminal device to call the AI ​​system's sub-classification OCR, quality inspection and other service capabilities to automatically pre-label the image with one or more of multiple quality labels to obtain a test image with labeled quality labels. This can also reduce the workload of manual labeling, thereby improving labeling efficiency.

[0098] In some embodiments, before the terminal device annotates the image based on multiple preset quality labels, it can first unify the label definition, that is, each picture is annotated with a maximum of three labels, one is the category label label1 (enumerated as the categories of claims materials supported by the SDK quality control module: ID card portrait page, ID card national emblem page, household registration book, invoice, document, and others); the other is the quality label label2 (enumerated as four values: quality meets the standard, poor completeness, poor exposure, and poor clarity). Among them, the SDK quality control module supports the picture categories of quality control (ID card portrait page, ID card national emblem page, household registration book, invoice, and document). Both label1 and label2 need to be marked. Pictures of categories (others) that the SDK quality control module does not support quality control do not need to be marked with label2. Another label label3 is a combination of multiple categories of data in label label1: non-xx category. Label3 enumerations are as follows: non-ID card portrait page, non-ID card national emblem page, non-household registration book, non-invoice, and non-document.

[0099] After that, the terminal device automatically pre-labels the image with quality labels through the pre-labeling module.

[0100] Please refer to Figure 3 , Figure 3 A schematic diagram of the steps of the evaluation method for the quality control application provided in the embodiment of the present application in other embodiments.

[0101] like Figure 3 As shown, in some embodiments, the evaluation method for quality control applications provided in the present application also includes steps S301 and S302 as shown below.

[0102] Step S301: Acquire historical image data to construct an initial test data set; wherein the initial test data set includes image data of multiple categories.

[0103] The terminal device collects historical image data of multiple categories (ID card portrait page, ID card national emblem page, household register, invoice, document, and others) from the big data platform / system (insurance platform / system) through a communication connection, and constructs the collected historical image data into an initial test data set. In this way, the initial test data set includes image data of multiple categories.

[0104] Step S302: classify the image data in the initial test data set to obtain test images of multiple categories.

[0105] After the terminal device constructs the initial test data set, it classifies the image data in the initial test data set according to the categories to which each image data belongs, thereby obtaining test images of multiple categories. For example, the terminal device classifies all image data belonging to the ID card portrait page in the initial test data set into the category of ID card portrait page, and the rest belong to the ID card national emblem page, household register, invoice, document and other image data, which are respectively classified into different categories.

[0106] In addition, when the terminal device classifies the image data, it can label each image data with a category label label1. For example, for all the image data classified as the ID card portrait page, the terminal device labels each of the image data with label1: ID card front / ID card portrait page. The same applies to the rest of the image data.

[0107] In some embodiments, the terminal device labels the image data in the initial test data set with a category label label1, which can also be done automatically through a pre-labeling module. For example, the terminal device reads the image data to be labeled in the initial test data set in a loop, and then calls the AI ​​system-claims sub-classification service for each image to perform detailed image classification, and maps the sub-classification results to the labeling of the category label label1 of the image (front and back of ID card, household register, invoice, document, other).

[0108] Please refer to Figure 4 , Figure 4 A flowchart of the steps for image annotation involved in some embodiments of the quality control application evaluation method provided in the embodiments of the present application.

[0109] like Figure 4 As shown, in some embodiments, when the terminal device applies the evaluation method of the quality control application provided in the embodiment of the present application, "image annotation based on preset multiple quality labels" may include steps S401 and S402 as shown below.

[0110] Step S401: for a test image of a target category, image annotation is performed based on a plurality of preset quality labels; wherein the target category is any one of the plurality of categories.

[0111] The terminal device further performs image annotation based on a plurality of preset quality labels for test images of any target category that have been annotated with category labels, thereby annotating each of all test images with a quality label label2. For example, for all test images that have been annotated with category label label1 as the front of an ID card, the terminal device further annotates each of the test images with a quality label2 as at least one of the following: quality compliance, poor integrity, poor exposure, and poor clarity. The same applies to the remaining test images of each category.

[0112] Step S402: selecting test images of other categories from the initial test data set, and marking the test images of other categories with non-target category labels; wherein the other category is at least one category other than the target category among the multiple categories.

[0113] During the image annotation process, the terminal device further selects test images of other categories from the initial test data set for any target category test image that has been annotated with category labels and quality labels, and annotates the selected test images of other categories with non-target category labels label3. Among them, the other categories to which the test images selected by the terminal device from the initial test data set belong are at least one category other than the target category in multiple categories. For example, for test images of all ID card portrait page categories that have been annotated with category labels label1 as the front of the ID card and quality labels label2 as meeting the quality standards, poor completeness, poor exposure or poor clarity, the terminal device evenly selects e images (test images of other categories) from the image data of label1 that are not in this category (non-ID card portrait page category, i.e.: ID card national micro page, household register, invoice document, and other categories), and marks the selected e images with non-target category labels label3: non-ID card portrait page. Here, e≥number, number is the maximum number of test images with the quality label label2 as meeting the quality standard, the number of test images with the quality label label2 as poor completeness, the number of test images with the quality label label2 as poor exposure, and the number of test images with the quality label label2 as poor clarity, among all test images with the category label label1 as the front of the ID card.

[0114] It should be noted that, to ensure data diversity, number can be the minimum number of images used for evaluation, such as 500.

[0115] In this embodiment, an initial test data set is constructed through a terminal device, and when classifying the image data in the initial test data set, a category label label1 is labeled for each image data, and then for any target category test image that has been labeled with a category label, the image labeling operation is further continued based on a plurality of preset quality labels, so that each of all the test images is labeled with a quality label label2, and, for any target category test image that has been labeled with a category label and a quality label, test images of other categories are further selected from the initial test data set, and the selected test images of other categories are labeled with a non-target category label label3.

[0116] In this way, based on the characteristics of claims material images, image quality control failure is subdivided into four data scenarios: incorrect classification, poor completeness, poor exposure, and poor clarity. Therefore, by unifying and refining the three data labels to replace the original label (whether the quality control passes the label), the effectiveness and rationality of evaluating the quality control function of insurance claims materials in quality control applications can be improved.

[0117] In some embodiments, when multiple quality labels include at least two of quality standards, poor completeness, poor exposure, and poor clarity, when the terminal device annotates the image based on the preset multiple quality labels, it can perform modular processing for the test data set construction process to annotate the image quality labels, thereby improving the annotation efficiency.

[0118] Please refer to Figure 5 , Figure 5 A flowchart of the steps for marking quality labels involved in some embodiments of the quality control application evaluation method provided in the embodiments of the present application.

[0119] like Figure 5 As shown, in some embodiments, the step of "annotating images based on preset multiple quality labels" in the evaluation method of the quality control application provided in the embodiment of the present application may specifically include steps S501 to S503 as shown below.

[0120] Step S501: Perform image quality detection processing on a test image belonging to a preset insurance claim material type to obtain a quality detection result of the test image; wherein the preset insurance claim material type includes an identity card page type, a household register type, an invoice type, and a document type.

[0121] In the process of image annotation for the image data in the initial test data set, the terminal device will perform image quality detection on the test images annotated with the category label label1 and the test images belonging to the preset insurance claim material type (that is, all the test images annotated with the category label label1 as the identity card page, household register, invoice and document) in turn to obtain the quality detection result of each test image.

[0122] Step S502: when the quality detection result of the test image is that the quality does not meet the standard, pre-mark the test image with a label value of poor integrity, a label value of poor exposure and / or a label value of poor clarity based on the quality detection result.

[0123] After the terminal device obtains the quality detection results of the test images belonging to the preset insurance claim material type, for each of the test images, if the quality detection result of the test image is that the quality does not meet the standard, the terminal device further pre-marks the test image with a label value of poor integrity, a label value of poor exposure, and / or a label value of poor clarity based on the specific content of the quality detection result. For example, assuming that the labeled category label label1 is a test image 1 of an ID card page, and its quality detection result is that the quality does not meet the standard and the specific content is poor integrity, the terminal device pre-marks the label value of poor integrity for the test image 1, so as to automatically mark the quality label label2 of the test image 1 as poor integrity. For another example, assuming that the labeled category label label1 is a test image 2 of an ID card page, and its quality detection result is also that the quality does not meet the standard but the specific content is poor exposure and poor clarity, the terminal device pre-marks the label value of poor exposure and the label value of poor clarity for the test image 2, so as to automatically mark the quality label label2 of the test image 2 as poor exposure and poor clarity.

[0124] Step S503: when the quality detection result of the test image is that the quality meets the standard and the test image passes the keyword verification, pre-mark the test image with a label value indicating that the quality meets the standard.

[0125] After the terminal device obtains the quality inspection result of the test image belonging to the preset insurance claim material type, for each of the test images, if the quality inspection result of the test image is that the quality meets the standard, the terminal device further performs keyword verification on the test image (calling OCR to recognize the characters in the image and performing keyword verification). Then, if the test image also passes the keyword verification, the terminal device pre-marks the test image with a label value that meets the quality standard, thereby automatically marking the quality label label2 of the test image as meeting the quality standard.

[0126] Exemplarily, in addition to marking the category label label1 as other test images, the terminal device continues to call the AI ​​system-image quality detection service or third-party open source image quality inspection capabilities for each test image to detect the quality of the test image, and then pre-mark the value of the quality label label2 (the label value of poor completeness, the label value of poor exposure and / or the label value of poor clarity) and mark other normal quality images according to the quality detection score result (quality detection result). Among them, for normal quality images, the terminal device continues to call OCR+keyword verification of each category of keywords, such as: the front category of the ID card contains the keyword citizen ID number, the back category of the ID card contains the keyword resident ID card, issuing authority, the household registration category contains the keyword permanent population registration card, household head relationship, native place, etc., the invoice category contains the keyword bill, amount, etc., and the document category contains the keyword discharge, admission, hospitalization, outpatient, medical record, examination report, etc. Afterwards, if the OCR+keyword verification passes, the quality label label2 is marked as meeting the quality standard, and if the OCR+keyword verification fails, the quality label label2 is temporarily marked as waiting for manual marking. In addition, the terminal device also saves the annotation results locally as a JSONL format file (the test image that has been annotated with quality labels).

[0127] In some embodiments, the terminal device can also perform manual sampling and error correction operations on the test images that have been annotated with quality labels. For example, the terminal device performs manual sampling on the test images that have been pre-annotated with quality labels based on the third-party open source annotation platform Doccano to verify the marking results and export the corrected annotation results.

[0128] Based on this, after annotating the image based on the preset multiple quality labels, the evaluation method of the quality control application provided in the embodiment of the present application may further include the following steps:

[0129] When a quality label annotation and error correction operation is received, the quality label that has been annotated with the first target test image is switched to the quality label indicated by the quality label annotation and error correction operation; wherein the quality label annotation and error correction operation is an interactive operation received by a visual interface based on a preset annotation tool, and the first target test image is the test image after the quality label is annotated and displayed in the visual interface.

[0130] It should be noted that the preset annotation tool may be an application installed on the terminal device for the staff to manually annotate the test image, such as the third-party open source annotation platform Doccano.

[0131] The terminal device visualizes the test image after the quality label is annotated based on the visualization interface of the preset annotation tool. Then, when the terminal device receives the quality label annotation and error correction operation performed by the staff on the first target test image after one or more quality labels are annotated currently displayed based on the visualization interface, the terminal device switches the quality label that has been annotated for the first target test image to the quality label that the quality label annotation and error correction operation performed by the staff indicates is to be re-annotated for the first target test image.

[0132] In some embodiments, after annotating the image based on a plurality of preset quality labels, the evaluation method for the quality control application provided in the embodiment of the present application may further include the following steps:

[0133] When the number of second target test images is less than a preset threshold value of the number of images that meet the quality standard, images that meet the quality standard are obtained, and label values ​​with quality standards are pre-marked for the images that meet the quality standard; wherein the second target test images are images with pre-marked label values ​​with quality standards in the test images after the quality labels are annotated, and the sum of the number of the second target test images and the number of the images that meet the quality standard is greater than or equal to the threshold value of the number of images that meet the quality standard.

[0134] For the second target test images pre-marked with the label value that meets the quality standard in the test images after the quality label is annotated, the terminal device obtains the images that meet the quality standard when the number of the second target test images is less than the preset threshold number of images that meet the quality standard (such as number = 500), and pre-marks the label value that meets the quality standard for the acquired images that meet the quality standard. Among them, the sum of the number of images that meet the quality standard obtained by the terminal device and the number of the second target test images that have been pre-marked with the label value that meets the quality standard needs to be greater than or equal to the threshold number of images that meet the quality standard of 500, so that the number of second target test images that are finally marked with the label value that meets the quality standard can be greater than or equal to the threshold number of images that meet the quality standard of 500.

[0135] Exemplarily, after the terminal device automatically pre-labels the image data in the initial test data set with the category label label1 and the quality label label2, it responds to the staff's operation and provides the staff with an interface for logging into the Doccano annotation platform. The staff then logs in to the Doccano annotation platform and chooses to import the test image after the automatic pre-labeling and its pre-labeling results (including the image file name or path, the labeling result values ​​of label1 and label2). Afterwards, the terminal device annotates the test images of "to be manually annotated label2" one by one based on the Doccano annotation platform in response to the staff's operation, and selects a value from the enumeration value of label2 for labeling. In addition, the test image after the automatic pre-labeling is displayed through the Doccano annotation platform, so as to respond to the staff's operation to manually sample the already annotated label1 and label2, verify the labeling results and correct errors. In addition, for any category of test images whose quality label label2 is the number of test images that meet the quality standards is insufficient (≤number=500), the terminal device supplements the data, that is, obtains images that meet the quality standards, and pre-marks the quality-compliant label values ​​for the acquired images that meet the quality standards.

[0136] In this embodiment, the construction process of the test image is modularly processed through the terminal device, wherein the pre-labeling module for image labeling effectively integrates the AI ​​system's image subclassification, image quality detection, OCR and other capabilities, and automatically labels the image data, thereby reducing the workload of manual labeling and improving the labeling efficiency of image data.

[0137] In some embodiments, the terminal device may configure a data balanced distribution module to make the data distribution of the test image more balanced and reasonable through data enhancement and other methods.

[0138] Please refer to Figure 6 , Figure 6 for Figure 2 Schematic diagram of the detailed step flow of step S202.

[0139] like Figure 6 As shown, in some embodiments, the "performing data enhancement processing on the test image annotated with the quality label" in the above-mentioned step S202 may include steps S601 and S602 as shown below.

[0140] Step S601: obtaining a first number of test images corresponding to each of the plurality of quality labels in the test image annotated with the quality labels; wherein the sum of the first numbers is equal to the total number of the test images annotated with the quality labels.

[0141] For each category of test images that have been labeled with quality labels, the terminal device obtains the first number of test images corresponding to each of the multiple quality labels in the test image. The sum of the first numbers is equal to the total number of test images labeled with quality labels. For example, for the 1500 test images of the front of the ID card category that have been labeled with quality label label2, the terminal device obtains the first number of 500 test images labeled with quality label label2 that meet the quality standard, the first number of 400 test images labeled with quality label label2 that are poor in completeness, the first number of 300 test images labeled with quality label label2 that are poor in clarity, and the first number of 300 test images labeled with quality label label2 that are poor in exposure (1500 = 500 + 400 + 300 + 300).

[0142] Step S602: performing data enhancement processing on the test images corresponding to the multiple quality labels respectively until the second number of the test images corresponding to the multiple quality labels is the same as the maximum value of the first numbers.

[0143] The terminal device configures a data balanced distribution module, and the data balanced distribution module performs data enhancement processing on the test images corresponding to each of the multiple quality labels in a data enhancement manner, until the second number of test images corresponding to each of the multiple quality labels after the data enhancement processing is the same as the maximum value of each first number. For example, in the case where the maximum value of each of the first numbers is 500, the terminal device performs data enhancement processing on the test images corresponding to each of the multiple quality labels through the data balanced distribution module, and the second number of test images corresponding to each of the multiple quality labels after the data enhancement processing is 500.

[0144] Exemplarily, the terminal device configures a data balanced distribution module, which reads the labeled data result file (test images with quality labels for each category), including the path of each test image and label1 and label2 labels (test images of the "other" category do not have label2).

[0145] Afterwards, the terminal device loops through the data balancing distribution module for various test images that have been marked with category label label1 (for example, the ID card portrait page category), and counts the number of images (the first number) of each value of its quality label label2. The counted numbers are: a images that meet the quality standard, b images with poor completeness, c images with poor exposure, and d images with poor clarity.

[0146] Then, the terminal device first determines whether the number of sheets that meet the quality standard a is ≤ number = 500. If so, data is supplemented so that a is greater than 500 sheets.

[0147] Afterwards, the terminal device processes the images that meet the quality standards through the data balance distribution module to increase the number of test images with poor quality (data enhancement is performed using methods such as taking the real features of poor quality samples + random positions, and simulating exposure / blur general algorithms):

[0148] The pixel assignment process of the random position block is performed until the number of test images with the quality label label2 being poor integrity (the second number) is e;

[0149] The image positions are randomly exposed until the number of test images with the quality label label2 being poorly exposed (the second number) is e;

[0150] The image direction is randomly blurred until the number of test images with the quality label label2 being poor clarity (the second number) is e;

[0151] The quality-compliant picture data is enhanced until the number of test images with the quality label label2 that meet the quality standard (the second number) is e.

[0152] Here, e=max(a, b, c, d, number).

[0153] In addition, the terminal device can also create poor quality pictures and annotation results through the data balanced distribution module to keep the number of test images labeled with each quality label label2 balanced.

[0154] In this embodiment, the data balanced distribution module is configured in the terminal device, and the data distribution of the test image is made more balanced and reasonable through data enhancement and other methods. Since the data balanced distribution module uses data enhancement technologies such as image processing, there are fewer real poor quality samples such as incomplete or overexposed samples due to imaging abnormalities, and data enhancement is performed by taking these real sample features + random position pixel processing and other methods, the distribution of positive and negative samples can be made more balanced and reasonable, and possible distortion caused by using all simulation algorithms for data enhancement can be avoided.

[0155] In some embodiments, considering that if the test images of each category are tested multiple times in batches, and the test results of different categories are statistically analyzed separately, the efficiency of test execution and indicator statistics will be reduced, the terminal device can sequentially call each quality control API supported by the SDK quality control model control by batch test results, read the corresponding image set according to the annotation results (annotated category label label1 and quality label label2) for batch test processing, and save the quality control test results of the quality control API. Finally, based on the annotation results and quality control test results, the overall business indicators and general indicators of each quality control API and the detailed evaluation indicator results of each label dimension are calculated.

[0156] Please refer to Figure 7 , Figure 7 for Figure 2 Schematic diagram of the detailed step flow of step S203.

[0157] like Figure 7 As shown, in some embodiments, when the quality control application includes multiple quality control application programming interfaces (each quality control API supported by the SDK quality control control), and the number of the multiple quality control application programming interfaces is greater than or equal to the number of categories to which the test images belong, the above-mentioned step S203: performing batch testing on the test images with a balanced distribution of the number based on the quality control application to obtain the indicator evaluation results of the quality control application, may include steps S701 to S703 as shown below.

[0158] Step S701: Based on a target quality control application program interface, batch testing is performed on the test images belonging to the target category among the test images with a balanced distribution in number; wherein the target quality control application program interface is any one of the multiple quality control application program interfaces, the target category is any one of the categories to which the test images belong, and the target quality control application program interface is dedicated to testing the test images of the target category.

[0159] For test images that have been annotated with data labels (including category label label1, quality label label2 and non-target category label label3), the terminal device calls a target quality control application program interface among multiple quality control application programs for any target category of test images, and distributes a balanced number of test images corresponding to each quality label label2 for batch testing.

[0160] For example, for test images whose category label label1 is household registration book (target category) and whose corresponding numbers of indicator labels label2 are evenly distributed (500 images meet the quality standard, 500 images have poor completeness, 500 images have poor exposure, and 500 images have poor clarity), the terminal device calls the household registration book API (target quality control application program interface) in the quality control API supported by the SDK quality control control, which is specifically used to perform quality control on household registration book category test images, and performs batch testing on the test images.

[0161] Step S702: Counting the test processing results of the test images of the target category; wherein the test processing results are used to characterize the quality control pass status of the test images corresponding to each of the multiple quality labels.

[0162] When the terminal device performs batch testing on the test images, it counts the test processing results of the test images of any target category in the test images. Among them, the test processing results of the test images of a certain target category are used to characterize the quality control pass status of the test images corresponding to multiple quality labels under the target category (such as: the quality control of the test images that meet the quality standards has passed f pieces and failed ef pieces, the quality control of the test images with poor integrity has passed g pieces and failed eg pieces, the quality control of the test images with poor exposure has passed h pieces and failed eh pieces, and the quality control of the test images with poor clarity has passed i pieces and failed ei pieces, and the quality control of the test images of non-target categories has passed j pieces and failed ej pieces).

[0163] Step S703: Calculate detailed business indicator evaluation results of the quality control application corresponding to the multiple quality tags respectively based on the test processing results.

[0164] After the terminal device obtains the test processing results of a test image of a certain target category, it can calculate the evaluation indicators based on the test processing results, thereby obtaining the detailed business indicator evaluation results (such as completeness difference interception rate, exposure blue interception rate, clarity difference interception rate, and other classification interception rates) of the quality control application for the test image of the target category corresponding to multiple quality labels.

[0165] Exemplarily, the terminal device annotates the test image data with data labels through batch running test and records the quality control results of each test image: image path-label [label1 value, label2 value, Iabe13 value]-API (any of the quality control APIs supported by SDK quality control) quality control result is pass / fail.

[0166] The terminal device counts the number of test results for each quality control API. For example, the ID card portrait page AP1 has the following statistics:

[0167] Label1 is the ID card portrait page, Label2 is the quality standard, and the quality control has passed f pages;

[0168] Label1 is the ID card portrait page, Label2 is the quality standard, and the quality control failed ef page;

[0169] Label1 is the ID card portrait page, Label2 is the completeness difference and passed the quality control g; Label1 is the ID card portrait page, Label2 is the completeness difference and failed the quality control eg;

[0170] Label1 is the ID card portrait page Label2 is the exposure difference quality control passed h photos;

[0171] Label1 is the ID card portrait page, Label2 is the exposure poor quality control failed eh;

[0172] Label1 is the ID card portrait page, Label2 is the i-th page that has passed the quality control of poor clarity;

[0173] Label1 is the ID card portrait page, Label2 is poor in clarity and failed the quality control. Label3 is the non-ID card portrait page and passed the quality control.

[0174] Label3 is not the ID card portrait page and has not passed quality control.

[0175] In this way, the terminal device calculates the indicator evaluation results based on the statistical quantity. Among them, 1. Business indicator evaluation results: pass rate = f / e, interception rate = (4e-ghij) / (4e); 2. Detailed business indicator evaluation results: completeness difference interception rate = (eg) / e, exposure blue interception rate = (eh) / e, clarity difference interception rate = (ei) / e, other classification interception rate = (ej) / e; 3. General indicator evaluation results: quality control pass recall rate recall = f / e, quality control pass precision rate Precision = f / (f+g+h+i+j), overall accuracy rate accuracy = (f+4e-ghij) / / (5e).

[0176] The calculation of statistics and indicator evaluation results for each other quality control API is similar to this and will not be repeated here.

[0177] In this embodiment, the terminal device uses the three types of label information (category label label1, quality label label2, and non-target category label label3) of each image to batch-run all the test image data at one time to evaluate the quality control APIs of all SDK quality control models and the corresponding indicator evaluation results of statistical calculations. There is no need to perform tests and statistically calculate the indicator evaluation results for different categories of test images in batches, thereby improving the efficiency of test execution and the efficiency of statistical indicator evaluation results. In addition, the terminal device calculates the four detailed business indicator evaluation results of completeness difference, exposure difference, clarity difference, and other classification interception rates. When a certain interception rate indicator is low, it is more convenient and quick to determine the effect distribution and optimization direction of the SDK quality control model in the quality control application, thereby improving the problem optimization efficiency of the SDK quality control model.

[0178] See also Figure 8 The embodiment of the present application also provides a quality control application evaluation device, which can implement the above-mentioned quality control application evaluation method, and the device includes:

[0179] An acquisition module, used to acquire a test image annotated with a quality label; wherein the test image annotated with the quality label is a test image obtained by annotating the image based on a plurality of preset quality labels, and each of the plurality of quality labels is used to characterize at least one image quality condition;

[0180] A data enhancement module, used to perform data enhancement processing on the test image annotated with the quality labels to obtain a test image with a balanced distribution of quantities corresponding to the multiple quality labels;

[0181] A quality control evaluation module is used to perform batch testing on the test images with a balanced distribution of quantity based on a quality control application to obtain an indicator evaluation result of the quality control application; wherein the indicator evaluation result at least includes detailed business indicator evaluation results corresponding to the multiple quality labels respectively.

[0182] In some embodiments, the plurality of quality labels include at least two of quality compliance, poor integrity, poor exposure, and poor clarity. The evaluation device for quality control applications provided in the embodiment of the present application further includes:

[0183] A pre-labeling module is used to perform image quality detection processing on a test image belonging to a preset insurance claim material type to obtain a quality detection result of the test image; wherein the preset insurance claim material type includes an identity card page type, a household register type, an invoice type, and a document type; when the quality detection result of the test image is that the quality does not meet the standard, the test image is pre-labeled with a label value of poor completeness, a label value of poor exposure, and / or a label value of poor clarity based on the quality detection result; and, when the quality detection result of the test image is that the quality meets the standard and the test image passes the keyword verification, the test image is pre-labeled with a label value of meeting the quality standard.

[0184] In some embodiments, the evaluation device for quality control applications provided in the embodiments of the present application further includes:

[0185] A sampling module is used to, upon receiving a quality label marking and error correction operation, switch the quality label already marked on the first target test image to the quality label indicated by the quality label marking and error correction operation; wherein the quality label marking and error correction operation is an interactive operation received by a visual interface based on a preset marking tool, and the first target test image is a test image marked with a quality label and displayed in the visual interface; and, in a case where the number of second target test images is less than a preset threshold number of images that meet the quality standard, obtain an image that meets the quality standard, and pre-mark the image that meets the quality standard with a label value that meets the quality standard; wherein the second target test image is an image with a label value that meets the quality standard pre-marked in the test image that meets the quality label, and the sum of the number of the second target test images and the number of the images that meet the quality standard is greater than or equal to the threshold number of images that meet the quality standard.

[0186] In some embodiments, the data enhancement module is further used to obtain a first number of test images corresponding to each of the multiple quality labels in the test image after the quality label is annotated; wherein the sum of the first numbers is equal to the total number of test images after the quality label is annotated; and data enhancement processing is performed on the test images corresponding to each of the multiple quality labels until the second number of test images corresponding to each of the multiple quality labels is the same as the maximum value of the first numbers.

[0187] In some embodiments, the quality control application includes multiple quality control application program interfaces, and the number of the multiple quality control application program interfaces is greater than or equal to the number of categories to which the test image belongs. The quality control evaluation module is also used to perform batch test processing on the test images belonging to the target category among the test images with balanced distribution of quantity based on the target quality control application program interface; wherein the target quality control application program interface is any one of the multiple quality control application program interfaces, the target category is any one of the categories to which the test image belongs, and the target quality control application program interface is dedicated to testing the test images of the target category; the test processing results of the test images of the target category are counted; wherein the test processing results are used to characterize the quality control pass status of the test images corresponding to each of the multiple quality labels; and, based on the test processing results, the detailed business indicator evaluation results of the quality control application corresponding to the multiple quality labels are calculated.

[0188] In some embodiments, the acquisition module is further used to acquire historical image data to construct an initial test data set; wherein the initial test data set includes image data of multiple categories; and, classify the image data in the initial test data set to obtain test images of multiple categories.

[0189] In some embodiments, the pre-labeling module is further used to label the test images of the target category based on a plurality of preset quality labels; wherein the target category is any one of the plurality of categories; and to select test images of other categories from the initial test data set, and label the test images of the other categories with non-target category labels; wherein the other category is at least one category other than the target category among the plurality of categories.

[0190] It should be noted that the specific implementation of the quality control application evaluation device provided in the embodiment of the present application is basically the same as the specific implementation of the quality control application evaluation method described above, and will not be repeated here.

[0191] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned quality control application evaluation method when executing the computer program. The computer device can be any intelligent terminal device including a tablet computer, a banking terminal, etc.

[0192] See also Fig. 9 , Fig. 9 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0193] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0194] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the evaluation method of the quality control application of the embodiment of this application;

[0195] Input / output interface 903, used to implement information input and output;

[0196] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0197] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0198] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0199] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the evaluation method of the quality control application is implemented.

[0200] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0201] An embodiment of the present application also provides a computer program product, including a computer-readable storage medium, the computer-readable storage medium storing a computer program, and the computer program, when executed by a processor, implements the above-mentioned quality control application evaluation method.

[0202] The specific implementation modes of the computer device, computer-readable storage medium, and computer program product provided in the embodiments of the present application are basically the same as the specific implementation modes of the evaluation method for the above-mentioned quality control application, and will not be repeated here.

[0203] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0204] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0205] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0207] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0208] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0209] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0213] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for evaluating a quality control application, characterized in that: The method comprises: Acquire a test image annotated with a quality label; wherein the test image annotated with the quality label is a test image obtained by annotating the image based on a plurality of preset quality labels, and each of the plurality of quality labels is used to characterize at least one image quality condition; Performing data enhancement processing on the test image annotated with the quality labels to obtain test images with a balanced distribution of quantities corresponding to each of the multiple quality labels; Based on the quality control application, batch testing is performed on the test images with a balanced distribution in quantity to obtain an indicator evaluation result of the quality control application; wherein the indicator evaluation result at least includes detailed business indicator evaluation results corresponding to the multiple quality labels.

2. The method according to claim 1, characterized in that The multiple quality labels include at least two of: quality reaching the standard, poor completeness, poor exposure, and poor clarity; Image annotation based on multiple preset quality labels, including: Performing image quality detection processing on a test image belonging to a preset insurance claim material type to obtain a quality detection result of the test image; wherein the preset insurance claim material type includes an identity card page type, a household register type, an invoice type, and a document type; In a case where the quality detection result of the test image is that the quality does not meet the standard, pre-marking the test image with a label value of poor integrity, a label value of poor exposure and / or a label value of poor clarity based on the quality detection result; When the quality detection result of the test image is that the quality meets the standard and the test image passes the keyword verification, a label value indicating that the quality meets the standard is pre-marked on the test image.

3. The method according to claim 1, characterized in that After annotating the image based on the preset multiple quality labels, the method further includes at least one of the following: In the case of receiving a quality label annotation error correction operation, switching the quality label annotated on the first target test image to the quality label indicated by the quality label annotation error correction operation; wherein the quality label annotation error correction operation is an interactive operation received by a visualization interface based on a preset annotation tool, and the first target test image is a test image annotated with the quality label and displayed in the visualization interface; When the number of second target test images is less than a preset threshold value of the number of images that meet the quality standard, images that meet the quality standard are obtained, and label values ​​with quality standards are pre-marked for the images that meet the quality standard; wherein the second target test images are images with pre-marked label values ​​with quality standards in the test images after the quality labels are annotated, and the sum of the number of the second target test images and the number of the images that meet the quality standard is greater than or equal to the threshold value of the number of images that meet the quality standard.

4. The method according to claim 1, characterized in that: The performing data enhancement processing on the test image annotated with the quality label comprises: Acquire a first number of test images corresponding to each of the plurality of quality labels in the test image annotated with the quality labels; wherein the sum of the first numbers is equal to the total number of the test images annotated with the quality labels; Data enhancement processing is performed on the test images corresponding to each of the multiple quality labels until a second number of the test images corresponding to each of the multiple quality labels is the same as a maximum value of the first numbers.

5. The method according to any one of claims 1 to 4, characterized in that The quality control application includes a plurality of quality control application program interfaces, and the number of the plurality of quality control application program interfaces is greater than or equal to the number of categories to which the test image belongs; The batch test processing is performed on the test images with balanced distribution in quantity based on the quality control application to obtain the indicator evaluation result of the quality control application, including: Based on a target quality control application program interface, a batch test process is performed on the test images belonging to the target category among the test images with a balanced number of distributions; wherein the target quality control application program interface is any one of the multiple quality control application program interfaces, the target category is any one of the categories to which the test images belong, and the target quality control application program interface is dedicated to testing the test images of the target category; Counting the test processing results of the test images of the target category; wherein the test processing results are used to characterize the quality control pass status of the test images corresponding to each of the multiple quality labels; Based on the test processing results, detailed business indicator evaluation results of the quality control application corresponding to the multiple quality tags are calculated.

6. The method according to claim 1, characterized in that The method further comprises: Acquire historical image data to construct an initial test data set; wherein the initial test data set includes image data of multiple categories; The image data in the initial test data set is classified to obtain test images of multiple categories.

7. The method according to claim 6, characterized in that Image annotation based on multiple preset quality labels, including: For a test image of a target category, image annotation is performed based on a plurality of preset quality labels; wherein the target category is any one of the plurality of categories; Selecting test images of other categories from the initial test data set, and marking the test images of other categories with non-target category labels; wherein the other category is at least one category other than the target category among the multiple categories.

8. An evaluation device for quality control applications, characterized in that: The device comprises: An acquisition module, used to acquire a test image annotated with a quality label; wherein the test image annotated with the quality label is a test image obtained by annotating the image based on a plurality of preset quality labels, and each of the plurality of quality labels is used to characterize at least one image quality condition; A data enhancement module, used to perform data enhancement processing on the test image annotated with the quality labels to obtain a test image with a balanced distribution of quantities corresponding to the multiple quality labels; A quality control evaluation module is used to perform batch testing on the test images with a balanced distribution of quantity based on a quality control application to obtain an indicator evaluation result of the quality control application; wherein the indicator evaluation result at least includes detailed business indicator evaluation results corresponding to the multiple quality labels respectively.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the evaluation method for the quality control application according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the evaluation method of the quality control application according to any one of claims 1 to 7 is implemented.

Images