Processing performance evaluation method and device, equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of artificial intelligence, is applied to a medical inspection document text extraction result evaluation scene, and relates to a processing performance evaluation method and device, equipment and a storage medium thereof. Inputting the extracted sample content information into a to-be-evaluated text recognition model to obtain the extracted sample content information; and comprehensively evaluating the processing performance of the text recognition model according to the sample content information, the annotation information in the evaluation sample set and a preset model evaluation strategy. By utilizing the method, the processing performance of the text recognition model can be recognized by evaluating the medical document text extraction result, so that the text recognition model with more excellent performance can be screened for a hospital in combination with the evaluation result subsequently, the medical document text content is extracted, the recognition accuracy is improved, and the medical document text extraction efficiency is improved. The business error rate in the aspect of hospitals is reduced, and a model service party can be helped to discover the defects of the service model in time and carry out upgrading processing.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and is applied to the scenario of evaluating the extraction results of medical inspection document texts. In particular, it relates to a processing performance evaluation method, device, equipment, and its storage medium. Background Art

[0002] With the rapid development of medical and health services, the amount of data in the business has been continuously increasing, and accordingly, various medical documents have also grown explosively. Due to the large variety of medical documents, or the large number of format categories of the same type of medical documents, and the inconsistent selection of processing models by all parties in the hospital, the performance of the models for identifying document content varies.

[0003] Therefore, with the rapid development of medical and health services, how to provide a more performant document extraction service model for hospitals has become an urgent problem to be solved. Summary of the Invention

[0004] The purpose of the embodiments of this application is to propose a processing performance evaluation method, device, equipment, and its storage medium, so as to provide a more performant document extraction service model for hospitals in combination with the processing performance evaluation results, which can not only improve the accuracy of text content recognition, reduce the business error rate of hospitals, but also help the model service provider timely discover the deficiencies of the service model and perform upgrade processing.

[0005] To solve the above technical problems, the embodiments of this application provide a processing performance evaluation method, which adopts the following technical solutions:

[0006] A processing performance evaluation method includes the following steps:

[0007] Obtain an evaluation sample set;

[0008] Input the evaluation sample set into the text recognition model to be evaluated, and obtain the sample content information extracted by the text recognition model to be evaluated;

[0009] Comprehensively evaluate the processing performance of the text recognition model according to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy.

[0010] Further, the step of obtaining the evaluation sample set specifically includes:

[0011] Obtain a batch of positive and negative samples from a multi-source database, where the positive samples are target processing samples and the negative samples are non-target processing samples;

[0012] Respectively count the positive and negative sample quantities, and calculate the sample quantity ratio relationship between the positive and negative samples according to the statistical results;

[0013] Add the batch of positive and negative samples to a preset sample set to generate the evaluation sample set;

[0014] After performing the step of obtaining the evaluation sample set, the method further includes:

[0015] Perform annotation processing on the samples in the evaluation sample set;

[0016] Obtain the annotation information in the evaluation sample set.

[0017] Further, the step of performing annotation processing on the samples in the evaluation sample set specifically includes:

[0018] Perform a first annotation on all samples in the evaluation sample set according to whether the sample is a target processing sample;

[0019] Filter out all positive samples according to the result of the first annotation;

[0020] Perform a second annotation on all the positive samples according to the layout of the samples to determine the sample layout information corresponding to each positive sample, where the layout of the samples includes web version, electronic version, and photographed version;

[0021] Perform a third annotation on all the positive samples according to the data content included in the samples to determine the text content information included in each positive sample.

[0022] Further, the sample content information includes sample positive and negative information, sample layout information, and text content information included in the sample. The step of inputting the evaluation sample set into the text recognition model to be evaluated and obtaining the sample content information extracted by the text recognition model to be evaluated specifically includes:

[0023] Use the classification and recognition component of the text recognition model to be evaluated to perform binary classification processing on the evaluation sample set to obtain all positive samples recognized by the text recognition model to be evaluated;

[0024] Use the layout recognition component of the text recognition model to be evaluated to perform layout recognition on all the positive samples to obtain the sample layout information corresponding to each positive sample;

[0025] Use the content extraction component of the text recognition model to be evaluated to extract the text content from all the positive samples to obtain the text content information included in each positive sample.

[0026] Further, the step of comprehensively evaluating the processing performance of the text recognition model according to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy specifically includes:

[0027] Calculate the binary classification ratio based on all positive samples identified by the text recognition model to be evaluated;

[0028] Calculate the classification accuracy of the classification recognition component based on the binary classification ratio and the sample quantity ratio;

[0029] Count the positive sample quantities of different formats identified by the format recognition component;

[0030] Compare and calculate the positive sample quantities of different formats identified by the format recognition component with the positive sample quantities of different formats marked, and calculate the format recognition accuracy of the format recognition component;

[0031] Perform consistency calculation on the text content information extracted by the content extraction component and the text content information of all marked positive samples, and calculate the text content extraction accuracy of the content extraction component;

[0032] Obtain the evaluation weights set in advance for the classification recognition component, the format recognition component, and the content extraction component respectively;

[0033] Perform weighted summation processing on the classification accuracy, the format recognition accuracy, and the text content extraction accuracy with the corresponding evaluation weights to obtain the comprehensive processing accuracy of the text recognition model.

[0034] Further, the step of comparing and calculating the positive sample quantities of different formats identified by the format recognition component with the positive sample quantities of different formats marked, and calculating the format recognition accuracy of the format recognition component specifically includes:

[0035] Based on the second marking result, identify the sample format information corresponding to all positive samples as the actual format information;

[0036] Determine the format weights corresponding to different formats based on the proportion of positive samples corresponding to different formats in the actual format information;

[0037] Compare and calculate the positive sample quantities of different formats identified by the format recognition component with the positive sample quantities of different formats marked, and calculate the format accuracy corresponding to different formats;

[0038] Perform weighted summation processing on the format accuracy corresponding to different formats and the format weights corresponding to different formats to obtain the format recognition accuracy of the format recognition component.

[0039] Further, the step of calculating the text content extraction accuracy rate of the content extraction component by calculating the consistency between the text content information extracted by the content extraction component and the text content information of all labeled positive samples specifically includes:

[0040] Delete the text content information included in non-target processing samples from the text content information extracted by the content extraction component, and only retain the text content information included in target processing samples;

[0041] For the text content information included in the current target processing sample and the text content information included in its corresponding positive sample, use the Euclidean distance method to calculate the character recognition accuracy rate, and obtain the character recognition accuracy rate corresponding to the current target processing sample;

[0042] Obtain the character recognition accuracy rates corresponding to all target processing samples respectively, and use the method of cumulative averaging to obtain the text content extraction accuracy rate of the content extraction component.

[0043] To solve the above technical problems, the embodiments of the present application also provide a processing performance evaluation device, which adopts the following technical solutions:

[0044] A processing performance evaluation device includes:

[0045] An evaluation sample set acquisition module, configured to acquire an evaluation sample set;

[0046] A sample content information extraction module, configured to input the evaluation sample set into a text recognition model to be evaluated, and obtain the sample content information extracted by the text recognition model to be evaluated;

[0047] A processing performance comprehensive evaluation module, configured to comprehensively evaluate the processing performance of the text recognition model according to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy.

[0048] To solve the above technical problems, the embodiments of the present application also provide a computer device, which adopts the following technical solutions:

[0049] A computer device includes a memory and a processor, and computer-readable instructions are stored in the memory. When the processor executes the computer-readable instructions, the steps of the above-mentioned processing performance evaluation method are implemented.

[0050] To solve the above technical problems, the embodiments of the present application also provide a computer-readable storage medium, which adopts the following technical solutions:

[0051] A computer-readable storage medium stores computer-readable instructions thereon, and when the computer-readable instructions are executed by a processor, the steps of the processing performance evaluation method as described above are implemented.

[0052] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:

[0053] In the processing performance evaluation method of the embodiments of the present application, an evaluation sample set is obtained; the evaluation sample set is input into a text recognition model to be evaluated, and sample content information extracted by the text recognition model to be evaluated is obtained; according to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy, the processing performance of the text recognition model is comprehensively evaluated. Applying the processing performance evaluation method of the present application to the evaluation scenario of the extraction result of medical document texts can evaluate the extraction result of medical document texts, thereby identifying the processing performance of the text recognition model, so as to subsequently select a text recognition model with better performance for the hospital side in combination with the evaluation result to extract the content of medical document texts, improve the recognition accuracy, reduce the business error rate of the hospital side, and also help the model service provider to timely discover the deficiencies of the service model and perform upgrade processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] To more clearly illustrate the solutions in the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0055] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

[0056] Figure 2 is a flowchart of an embodiment of the processing performance evaluation method according to the present application;

[0057] Figure 3 is Figure 2 a flowchart of a specific embodiment of step 201 shown;

[0058] Figure 4 is a flowchart of a specific embodiment of annotating the evaluation sample set in the processing performance evaluation method of the present application;

[0059] Figure 5 is Figure 4 a flowchart of a specific embodiment of step 401 shown;

[0060] Figure 6 is Figure 2Flow chart of a specific embodiment of step 202 shown;

[0061] Figure 7 is Figure 2 Flow chart of a specific embodiment of step 203 shown;

[0062] Figure 8 is Figure 7 Flow chart of a specific embodiment of step 704 shown;

[0063] Figure 9 is Figure 7 Flow chart of a specific embodiment of step 705 shown;

[0064] Figure 10 Structural schematic diagram of an embodiment of a processing performance evaluation device according to the present application;

[0065] Figure 11 Structural schematic diagram of an embodiment of a computer device according to the present application. Detailed implementation manners

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0067] Referring to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appearing at various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0068] In order to enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the drawings.

[0069] Such as Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102, and a server 103. The terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0070] Users can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications may be installed on the terminal device 101, such as a web browser application, a shopping application, a search application, an instant messaging tool, an email client, a social platform software, etc.

[0071] The terminal device 101 may be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012, or the mobile phone 1013, the terminal device 101 may also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop portable computer, and a desktop computer, etc.

[0072] The server 103 may be a server providing various services, such as a background server supporting the pages displayed on the terminal device 101.

[0073] It should be noted that the processing performance evaluation method provided by the embodiments of the present application is generally executed by the server. Correspondingly, the processing performance evaluation device is generally set in the server.

[0074] It should be understood that Figure 1 the numbers of the terminal device, the network, and the server in

[0075] are merely illustrative. According to the implementation requirements, there may be any number of terminal devices, networks, and servers. Figure 2 Continuing to refer to

[0076] Step 201, obtain an evaluation sample set.

[0077] In this embodiment, the evaluation sample set refers to a sample set composed of various medical documents, such as medical test reports, medical bills, medical registration forms, medical imaging reports, etc. Specifically, at least one type of document is selected from the documents as the target document, and the other unselected documents are used as confounding documents, so as to judge the text recognition processing effect of the target text recognition model.

[0078] By using the processing performance evaluation method described in this application to judge the text recognition processing effect of the target text recognition model, it is possible to automatically evaluate the text extraction effect of the document, ensuring that the text recognition model used by the hospital later meets the standards.

[0079] Step 202: Input the evaluation sample set into the text recognition model to be evaluated, and obtain the sample content information extracted by the text recognition model to be evaluated.

[0080] By obtaining the sample content information extracted by the text recognition model to be evaluated, it is convenient to evaluate the processing performance of the text recognition model based on the sample content information.

[0081] Step 203: Comprehensively evaluate the processing performance of the text recognition model according to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy.

[0082] Based on the sample content information, combined with the annotation information in the evaluation sample set and a preset model evaluation strategy, the processing performance of the text recognition model is comprehensively evaluated, ensuring the basis and persuasiveness of the evaluation results.

[0083] Applying the processing performance evaluation method described in this application to the evaluation scenario of the text extraction result of medical test documents, it is possible to evaluate the text extraction result of medical test documents, thereby identifying the processing performance of the text recognition model. This is convenient for subsequently selecting a text recognition model with better performance for the hospital based on the evaluation results to extract the text content of medical test documents, improving the recognition accuracy and reducing the business error rate of the hospital. Of course, the evaluation of the processing effect of medical test documents is only an example. It does not exclude combining medical bills, medical registration forms, medical imaging reports, etc. to separately evaluate the processing performance of the text recognition model, so as to screen out a comprehensive text recognition model with excellent performance to meet the requirement of selecting a target model for all documents and improving the accuracy of text recognition of all hospital documents. It can also help the model service provider promptly discover the deficiencies of the service model and perform upgrade processing.

[0084] In this embodiment, an evaluation sample set is obtained; the evaluation sample set is input into the text recognition model to be evaluated to obtain the sample content information extracted by the text recognition model to be evaluated; and the processing performance of the text recognition model is comprehensively evaluated based on the sample content information, the annotation information in the evaluation sample set, and the preset model evaluation strategy. The processing performance evaluation method described in this application is applied to the medical document text extraction result evaluation scenario. By evaluating the medical document text extraction results, the processing performance of the text recognition model can be identified, so that the evaluation results can be combined to select a text recognition model with better performance for the hospital, extract the medical document text content, improve the recognition accuracy, and reduce the business error rate of the hospital. It can also help the model service provider to promptly discover the deficiencies of the service model and perform upgrade processing.

[0085] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 201 shown includes the following steps:

[0086] Step 301, obtaining a batch of positive and negative samples from a multi-source database, wherein the positive samples are target processing samples and the negative samples are non-target processing samples;

[0087] In this embodiment, the multi-source database includes multiple medical document repositories of the same hospital, multiple medical document repositories of multiple hospitals, and a comprehensive medical management platform that can provide medical documents. The multi-source here refers to the different sources of samples, which can be obtained through multiple source channels.

[0088] Specifically, the positive samples are target processing samples, and the negative samples are non-target processing samples. For example, a medical examination form is pre-specified as a target processing sample. At this time, the medical bill, medical registration form, and medical imaging form can be used as non-target processing samples.

[0089] Step 302, respectively counting the number of positive and negative samples, and calculating the sample size ratio between the positive and negative samples based on the statistical results;

[0090] Continuing with the above embodiment, the number of medical test reports and the number of non-medical test reports are counted, and then the sample size ratio between positive and negative samples is calculated based on the number of medical test reports and the number of non-medical test reports, so as to facilitate subsequent model processing evaluation.

[0091] Step 303: Add the batch of positive and negative samples to a preset sample set to generate the evaluation sample set.

[0092] By combining obfuscated samples, an evaluation sample set is constructed, thus ensuring the credibility of subsequent processing performance evaluation of relevant models.

[0093] Continue to refer to Figure 4 , in some alternative implementation manners, after step 201, it further includes a step of annotating the evaluation sample set. Figure 4 FIG. is a flowchart of a specific embodiment of annotating the evaluation sample set in the processing performance evaluation method described in this application, including the following steps:

[0094] Step 401, perform annotation processing on the samples in the evaluation sample set;

[0095] Step 402, obtain the annotation information in the evaluation sample set.

[0096] By performing annotation processing on the samples in the evaluation sample set, it is convenient to subsequently determine the processing performance of the text recognition model in combination with the annotation information, ensuring the confidence of the processing performance evaluation result.

[0097] Continue to refer to Figure 5 , Figure 5 is Figure 4 A flowchart of a specific embodiment of step 401 shown in FIG., including the following steps:

[0098] Step 501, perform a first annotation on all samples in the evaluation sample set according to whether the sample is a target processing sample;

[0099] Specifically, that is, according to whether it is a medical inspection form, positive and negative sample annotations are performed. The Tesseract-OCR pre-annotation technology can be used to perform annotations by recognizing the form name. If the recognized form name contains the word "medical inspection", it is marked as a positive sample; otherwise, it is marked as a negative sample.

[0100] Step 502, screen out all positive samples according to the first annotation result;

[0101] Step 503, perform a second annotation on all the positive samples according to the format of the samples, and determine the sample format information corresponding to each positive sample, where the format of the sample includes web version, electronic version, and photographed version;

[0102] Specifically, since medical inspection forms may have different formats, for example: web version for convenient direct online browsing, electronic version for easy printing, and photographed version of the inspection pictures taken by patients for uploading.

[0103] Perform a second annotation on all the positive samples according to the layout of the samples, and determine the sample layout information corresponding to each of the positive samples, so as to evaluate the processing performance of the text recognition model from different versions, and ensure that a model with high generalization recognition ability for medical bills of different layouts can be selected.

[0104] Step 504: Perform a third annotation on all the positive samples according to the data content included in the samples, and determine the text content information included in each of the positive samples.

[0105] Specifically, annotate the specific data content included in the positive samples to facilitate the evaluation of the text extraction ability of the model in more detail, so as to ensure that the hospital and the service provider provide a more excellent bill text recognition model.

[0106] In this embodiment, the sample content information includes sample positive / negative information, sample layout information, and text content information included in the samples.

[0107] Continue to refer to Figure 6 , Figure 6 Yes Figure 2 is a flowchart of a specific embodiment of step 202 shown in

[0108] Step 601: Use the classification and recognition component of the text recognition model to be evaluated to perform binary classification on the positive and negative samples of the evaluation sample set, and obtain all the positive samples recognized by the text recognition model to be evaluated;

[0109] Specifically, the classification and recognition component only uses the text recognition ability of the text recognition model to be evaluated to predict positive and negative samples, so as to obtain the binary classification result of positive and negative samples. It should be understood that the actual annotation information is not used here.

[0110] Step 602: Use the layout recognition component of the text recognition model to be evaluated to perform layout recognition on all the positive samples, and obtain the sample layout information corresponding to each of the positive samples;

[0111] Similarly, the layout recognition component also only uses the layout recognition ability of the text recognition model to be evaluated to perform layout classification prediction, so as to obtain the sample layout information of all the positive samples predicted in step 601. It should be understood that the actual annotation information is not used here.

[0112] Step 603: Use the content extraction component of the text recognition model to be evaluated to extract the text content from all the positive samples, and obtain the text content information included in each of the positive samples.

[0113] Specifically, the evaluation sample set is processed by the text recognition model to be evaluated through positive and negative sample binary classification in sequence. The positive samples after binary classification are processed for layout classification and specific text content extraction, so as to obtain different processing evaluation results in different processing steps, which is convenient for comprehensively evaluating the processing performance of the text recognition model to be evaluated by combining the processing evaluation results in different processing steps later.

[0114] Continue to refer to Figure 7 , Figure 7 Yes Figure 2 Figure 203 is a flowchart of a specific embodiment of step 203, including the following steps:

[0115] Step 701, calculate the binary classification ratio relationship according to all positive samples recognized by the text recognition model to be evaluated;

[0116] Step 702, calculate the classification accuracy rate of the classification recognition component according to the binary classification ratio relationship and the sample quantity ratio relationship;

[0117] Step 703, count the positive sample quantities of different layouts recognized by the layout recognition component;

[0118] Step 704, compare and calculate the positive sample quantities of different layouts recognized by the layout recognition component with the positive sample quantities of different layouts marked, and calculate the layout recognition accuracy rate of the layout recognition component;

[0119] Step 705, calculate the consistency between the text content information extracted by the content extraction component and the text content information of all marked positive samples, and calculate the text content extraction accuracy rate of the content extraction component;

[0120] Step 706, obtain the evaluation weights set in advance for the classification recognition component, the layout recognition component, and the content extraction component respectively;

[0121] Step 707, perform weighted summation processing on the classification accuracy rate, the layout recognition accuracy rate, and the text content extraction accuracy rate with the corresponding evaluation weights to obtain the comprehensive processing accuracy rate of the text recognition model.

[0122] In this embodiment, the comprehensive processing accuracy rate of the text recognition model is obtained by performing weighted summation processing on the classification accuracy rate, the layout recognition accuracy rate, and the text content extraction accuracy rate with the corresponding evaluation weights, realizing the evaluation of the processing performance in multiple processing dimensions of the text recognition model, which is more scientific and reasonable, and ensuring that a text recognition model with better performance is selected for the hospital side.

[0123] Continue to refer to Figure 8 , Figure 8 is Figure 7 a flowchart of a specific embodiment of step 704 shown below, including the following steps:

[0124] Step 801, according to the second annotation result, identify the sample layout information corresponding to all positive samples respectively as the actual layout information;

[0125] Step 802, determine the layout weights corresponding to different layouts respectively according to the proportion of positive samples corresponding to different layouts in the actual layout information;

[0126] Step 803, compare and calculate the positive sample quantities of different layouts identified by the layout recognition component with the positive sample quantities of different layouts marked, and calculate the layout accuracy rates corresponding to different layouts respectively;

[0127] Step 804, perform weighted summation processing using the layout accuracy rates corresponding to different layouts respectively and the layout weights corresponding to different layouts respectively to obtain the layout recognition accuracy rate of the layout recognition component.

[0128] In this embodiment, first determine the layout weights corresponding to different layouts respectively according to the layout information corresponding to the actual positive samples in the evaluation sample set; then, combine the layout information corresponding to different positive samples identified by the text recognition model to calculate the layout accuracy rate. Finally, combine the layout accuracy rates corresponding to different layout information respectively and the layout weights to obtain the final layout recognition accuracy rate, which is more scientific and reasonable.

[0129] Continue to refer to Figure 9 , Figure 9 is Figure 7 a flowchart of a specific embodiment of step 705 shown below, including the following steps:

[0130] Step 901, delete the text content information contained in non-target processing samples from the text content information extracted by the content extraction component, and only retain the text content information contained in the target processing samples;

[0131] Specifically, since the text recognition model may identify negative samples as positive samples during binary classification of positive and negative samples, it may cause the text content information extracted by the content extraction component to contain the text content information of negative samples. Since step 504 only performs annotation processing on the specific data content in positive samples, through step 901, it is possible to first perform deletion processing on the text content information in negative samples and only retain the text content information contained in the target processing samples.

[0132] Step 902: Calculate the character recognition accuracy rate of the current target processing sample by using the Euclidean distance method for the text content information contained in the current target processing sample and the text content information contained in its corresponding positive sample, so as to obtain the character recognition accuracy rate corresponding to the current target processing sample.

[0133] Specifically, that is, by using the Euclidean distance method, calculate the proportion of correctly recognized characters in a specific sample, and determine the character recognition accuracy rate corresponding to the current target processing sample according to the proportion.

[0134] Step 903: Obtain the character recognition accuracy rates corresponding to all target processing samples respectively, and use the method of cumulative averaging to obtain the text content extraction accuracy rate of the content extraction component.

[0135] Specifically, use the processing method in Step 902 to obtain the character recognition accuracy rates corresponding to all target processing samples respectively. Then, in combination with the total number of positive samples, use the method of cumulative averaging to obtain the text content extraction accuracy rate of the content extraction component.

[0136] In this application, an evaluation sample set is obtained; the evaluation sample set is input into the text recognition model to be evaluated to obtain the sample content information extracted by the text recognition model to be evaluated; according to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy, the processing performance of the text recognition model is comprehensively evaluated. Applying the processing performance evaluation method described in this application to the evaluation scenario of the medical document text extraction result can evaluate the medical document text extraction result, so as to identify the processing performance of the text recognition model, so that in the subsequent combination with the evaluation result, a text recognition model with better performance can be selected for the hospital side to extract the medical document text content, improve the recognition accuracy rate, reduce the business error rate of the hospital side, and can also help the model service provider to timely discover the deficiencies of the service model and perform upgrade processing.

[0137] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results of theory, method, technology, and application systems.

[0138] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0139] In the embodiments of the present application, by obtaining an evaluation sample set; inputting the evaluation sample set into a text recognition model to be evaluated to obtain sample content information extracted by the text recognition model to be evaluated; and comprehensively evaluating the processing performance of the text recognition model according to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy. Applying the processing performance evaluation method of the present application to the evaluation scenario of the medical document text extraction result can evaluate the medical document text extraction result, thereby identifying the processing performance of the text recognition model, so as to subsequently screen out a text recognition model with better performance for the hospital side based on the evaluation result for medical document text content extraction, improve the recognition accuracy, reduce the business error rate of the hospital side, and also help the model service provider timely discover the deficiencies of the service model and perform upgrade processing.

[0140] Further referring to Figure 10 As an implementation of the method shown above Figure 2 In an embodiment of the present application, a processing performance evaluation device is provided. This device embodiment corresponds to the method embodiment shown in Figure 2 and can be specifically applied to various electronic devices.

[0141] As shown in Figure 10 the processing performance evaluation device 10 in this embodiment includes: an evaluation sample set acquisition module 10a, a sample content information extraction module 10b, and a processing performance comprehensive evaluation module 10c. Among them:

[0142] The evaluation sample set acquisition module 10a is used to obtain an evaluation sample set;

[0143] The sample content information extraction module 10b is used to input the evaluation sample set into a text recognition model to be evaluated to obtain sample content information extracted by the text recognition model to be evaluated;

[0144] The processing performance comprehensive evaluation module 10c is used to comprehensively evaluate the processing performance of the text recognition model according to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy.

[0145] This application obtains an evaluation sample set; inputs the evaluation sample set into a text recognition model to be evaluated to obtain sample content information extracted by the text recognition model to be evaluated; and comprehensively evaluates the processing performance of the text recognition model according to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy. Applying the processing performance evaluation method described in this application to the evaluation scenario of the extraction result of medical document text can evaluate the extraction result of medical document text, thereby identifying the processing performance of the text recognition model, so as to subsequently screen out a text recognition model with better performance for the hospital side based on the evaluation result to extract the text content of medical documents, improve the recognition accuracy, reduce the business error rate of the hospital side, and also help the model service provider promptly discover the deficiencies of the service model and perform upgrade processing.

[0146] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0147] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times, and their execution order does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0148] To solve the above technical problems, an embodiment of this application also provides a computer device. For details, please refer to Figure 11 , Figure 11 which is the basic structural block diagram of the computer device in this embodiment.

[0149] The computer device 11 includes a memory 11a, a processor 11b, and a network interface 11c that are communicatively connected to each other through a system bus. It should be noted that Figure 11Only a computer device 11 with a component memory 11a, a processor 11b, and a network interface 11c is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Among them, those skilled in the art of the present technology can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0150] The computer device may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device can interact with the user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device.

[0151] The memory 11a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11a may be an internal storage unit of the computer device 11, such as the hard disk or memory of the computer device 11. In other embodiments, the memory 11a may also be an external storage device of the computer device 11, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 11. Of course, the memory 11a may also include both the internal storage unit and the external storage device of the computer device 11. In this embodiment, the memory 11a is generally used to store the operating system and various application software installed on the computer device 11, such as computer-readable instructions for a processing performance evaluation method. In addition, the memory 11a can also be used to temporarily store various data that have been output or will be output.

[0152] In some embodiments, the processor 11b may be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 11b is generally used to control the overall operation of the computer device 11. In this embodiment, the processor 11b is used to run the computer-readable instructions stored in the memory 11a or process data, such as running the computer-readable instructions of the processing performance evaluation method.

[0153] The network interface 11c may include a wireless network interface or a wired network interface, and this network interface 11c is generally used to establish a communication connection between the computer device 11 and other electronic devices.

[0154] The computer device proposed in this embodiment belongs to the field of artificial intelligence technology and is applied to the scenario of evaluating the extraction results of medical inspection document texts. This application obtains an evaluation sample set; inputs the evaluation sample set into the text recognition model to be evaluated to obtain the sample content information extracted by the text recognition model to be evaluated; and comprehensively evaluates the processing performance of the text recognition model according to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy. Applying the processing performance evaluation method described in this application to the scenario of evaluating the extraction results of medical document texts can evaluate the extraction results of medical document texts, thereby identifying the processing performance of the text recognition model, so as to subsequently select a text recognition model with better performance for the hospital side in combination with the evaluation results to extract the text content of medical documents, improve the recognition accuracy, reduce the business error rate of the hospital side, and also help the model service provider timely discover the deficiencies of the service model and perform upgrade processing.

[0155] This application also provides another implementation manner, that is, to provide a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to execute the steps of the processing performance evaluation method as described above.

[0156] The computer-readable storage medium proposed in this embodiment belongs to the field of artificial intelligence technology and is applied to the scenario of evaluating the text extraction results of medical inspection documents. This application obtains an evaluation sample set, inputs the evaluation sample set into the text recognition model to be evaluated, and obtains the sample content information extracted by the text recognition model to be evaluated. According to the sample content information, the annotation information in the evaluation sample set, and a preset model evaluation strategy, the processing performance of the text recognition model is comprehensively evaluated. Applying the processing performance evaluation method of this application to the scenario of evaluating the text extraction results of medical documents can evaluate the text extraction results of medical documents, thereby identifying the processing performance of the text recognition model, so as to subsequently select a text recognition model with better performance for the hospital side based on the evaluation results to extract the text content of medical documents, improve the recognition accuracy, reduce the business error rate of the hospital side, and also help the model service provider promptly discover the deficiencies of the service model and perform upgrade processing.

[0157] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of this application.

[0158] Obviously, the above-described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. The accompanying drawings show the preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosed content of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of this application in other related technical fields is similarly within the scope of the patent protection of this application.

Claims

1. A processing performance evaluation method, characterized in that: The steps include: Obtain evaluation sample set; Inputting the evaluation sample set into the text recognition model to be evaluated to obtain sample content information extracted by the text recognition model to be evaluated; The processing performance of the text recognition model is comprehensively evaluated based on the sample content information, the annotation information in the evaluation sample set and a preset model evaluation strategy.

2. The processing performance evaluation method according to claim 1, characterized in that: The step of obtaining the evaluation sample set specifically includes: Obtain batches of positive and negative samples from a multi-source database, where positive samples are target processed samples and negative samples are non-target processed samples; Count the number of positive and negative samples respectively, and calculate the sample size ratio between positive and negative samples based on the statistical results; Adding the batch of positive and negative samples to a preset sample set to generate the evaluation sample set; After executing the step of obtaining the evaluation sample set, the method further includes: Performing labeling processing on samples in the evaluation sample set; Obtaining labeling information in the evaluation sample set.

3. The processing performance evaluation method according to claim 2, characterized in that: The step of labeling the samples in the evaluation sample set specifically includes: According to whether the sample is a target processing sample, all samples in the evaluation sample set are labeled for the first time; According to the first labeling result, all positive samples are screened out; Performing a second annotation on all the positive samples according to the sample layout to determine the sample layout information corresponding to all the positive samples, wherein the sample layout includes a web page version, an electronic version, and a photographed version; All positive samples are annotated for a third time according to the data content contained in the samples to determine the text content information respectively contained in all positive samples.

4. The processing performance evaluation method according to claim 2, characterized in that: The sample content information includes sample positivity information, sample format information, and text content information contained in the sample. The step of inputting the evaluation sample set into the text recognition model to be evaluated to obtain the sample content information extracted by the text recognition model to be evaluated specifically includes: Using the classification and recognition component of the text recognition model to be evaluated to perform binary classification processing of positive and negative samples on the evaluation sample set, and obtain all positive samples recognized by the text recognition model to be evaluated; Using the layout recognition component of the text recognition model to be evaluated to perform layout recognition on all the positive samples, and obtain sample layout information corresponding to all the positive samples respectively; The content extraction component of the text recognition model to be evaluated is used to extract the text content in all the positive samples to obtain the text content information respectively contained in all the positive samples.

5. The processing performance evaluation method according to claim 4, characterized in that: The step of comprehensively evaluating the processing performance of the text recognition model according to the sample content information, the annotation information in the evaluation sample set and the preset model evaluation strategy specifically includes: Calculate the binary classification ratio relationship according to all positive samples identified by the text recognition model to be evaluated; Calculating the classification accuracy of the classification recognition component according to the binary classification ratio relationship and the sample size ratio relationship; Counting the number of positive samples of different layouts identified by the layout recognition component; Compare and calculate the number of positive samples of different layouts identified by the layout recognition component and the number of positive samples of different layouts marked, and calculate the layout recognition accuracy of the layout recognition component; The text content information extracted by the content extraction component is calculated to be consistent with the text content information of all the positive samples annotated, and the accuracy of the text content extraction of the content extraction component is calculated; Obtaining evaluation weights pre-set for the classification recognition component, the layout recognition component, and the content extraction component; The classification accuracy, the format recognition accuracy and the text content extraction accuracy are weighted and summed with corresponding evaluation weights to obtain the comprehensive processing accuracy of the text recognition model.

6. The processing performance evaluation method according to claim 5, characterized in that: The step of comparing and calculating the number of positive samples of different layouts identified by the layout recognition component with the number of positive samples of different layouts marked to calculate the layout recognition accuracy of the layout recognition component specifically includes: According to the second labeling result, the sample layout information corresponding to all positive samples is identified as the actual layout information; Determining layout weights corresponding to different layouts according to the proportions of positive samples corresponding to different layouts in the actual layout information; Compare and calculate the number of positive samples of different layouts identified by the layout recognition component with the number of positive samples of different layouts marked, and calculate the layout accuracy rates corresponding to the different layouts; The layout accuracy rates corresponding to different layouts and the layout weights corresponding to different layouts are used to perform weighted summation processing to obtain the layout recognition accuracy rate of the layout recognition component.

7. The processing performance evaluation method according to claim 5, characterized in that: The step of calculating the consistency between the text content information extracted by the content extraction component and the text content information of all annotated positive samples to calculate the text content extraction accuracy of the content extraction component specifically includes: Deleting the text content information contained in the non-target processed samples from the text content information extracted by the content extraction component, and retaining only the text content information contained in the target processed samples; The text content information contained in the current target processing sample and the text content information contained in the corresponding positive sample are used to calculate the character recognition accuracy using the Euclidean distance method to obtain the character recognition accuracy corresponding to the current target processing sample; The character recognition accuracy rates corresponding to all target processing samples are obtained, and the text content extraction accuracy rate of the content extraction component is obtained by accumulating and averaging.

8. A processing performance evaluation device, characterized in that: include: An evaluation sample set acquisition module is used to acquire an evaluation sample set; A sample content information extraction module, used to input the evaluation sample set into the text recognition model to be evaluated, and obtain the sample content information extracted by the text recognition model to be evaluated; The processing performance comprehensive evaluation module is used to comprehensively evaluate the processing performance of the text recognition model based on the sample content information, the annotation information in the evaluation sample set and the preset model evaluation strategy.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the processing performance evaluation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the processing performance evaluation method according to any one of claims 1 to 7 are implemented.

Images