Report sheet detection method, device, electronic device and computer readable medium

Abstract

The present disclosure relates to a method, device, electronic device and computer-readable medium for detecting a report form, and belongs to the field of image processing technology. The method comprises: inputting the report form image of the report form into a pre-trained region detection model to obtain the region name and region coordinates of each detection region in the report form image; cropping the report form image according to the region name and region coordinates of each detection region to obtain a region feature map of each detection region; inputting the region feature map into a pre-trained region classification model to obtain the region category of each detection region; and detecting the report form according to the region category and region coordinates of each detection region in the report form image. By cropping the report form image, the present disclosure can ensure that the region classification model obtains more features, improves recognition accuracy, and thus improves the efficiency of report form detection.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a report detection method, a report detection device, an electronic device, and a computer-readable medium. Background Art

[0002] Filling out and archiving paper reports is an essential part of certain production processes. To ensure the efficient and safe operation of each production process, after manually filling out the report, an auditor must review the contents of the paper report.

[0003] Currently, the inspection process of paper reports mainly relies on manual review. Since manual inspection is time-consuming and labor-intensive, there is an urgent need in this field for a method that can improve the efficiency of report inspection.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The purpose of the present disclosure is to provide a report detection method, a report detection device, an electronic device and a computer-readable medium, thereby improving the detection efficiency of the report at least to a certain extent.

[0006] According to a first aspect of the present disclosure, a method for detecting a report is provided, comprising:

[0007] Inputting the report image of the report into a pre-trained region detection model to obtain the region name and region coordinates of each detection region in the report image;

[0008] cropping the report image according to the region name and region coordinates of each detection region to obtain a region feature map of each detection region;

[0009] Inputting the region feature map into a pre-trained region classification model to obtain the region category of each detection region;

[0010] The report is detected based on the area category and the area coordinates of each of the detection areas in the report image.

[0011] In an exemplary embodiment of the present disclosure, the training method of the region detection model includes:

[0012] Acquire a history report image, wherein the history report image includes area marking boxes corresponding to each of the detection areas;

[0013] Determine the detection areas corresponding to the area marking frames having the same relative position in each of the historical report images as having the same area name;

[0014] A region annotation dataset is generated according to the region annotation boxes and the region names corresponding to the respective detection regions, and the region detection model is obtained by training based on the region annotation dataset.

[0015] In an exemplary embodiment of the present disclosure, the step of training the region detection model based on the region annotation dataset includes:

[0016] Dividing the region annotation dataset into a region annotation training set, a region annotation validation set, and a region annotation test set;

[0017] An initial region detection model is obtained by training the region annotation training set, and a plurality of candidate region detection models are obtained based on the region annotation verification set and the initial region detection model;

[0018] The region detection model is determined from a plurality of candidate region detection models according to the region annotation test set.

[0019] In an exemplary embodiment of the present disclosure, the training method of the region classification model includes:

[0020] Acquire a history report image, wherein the history report image includes an area marking frame and an area category corresponding to each of the detection areas;

[0021] Cropping the historical report image according to the region marking frame to obtain region images corresponding to each detection region;

[0022] A category annotation data set is generated according to the area pictures and area categories corresponding to each of the detection areas in the historical report image, and the area classification model is obtained by training based on the category annotation data set.

[0023] In an exemplary embodiment of the present disclosure, generating a category annotation dataset based on the region images and region categories corresponding to each of the detection regions in the historical report image includes:

[0024] Obtaining the number of categories in each of the area categories, and determining the maximum number of categories and the minimum number of categories corresponding to each of the area categories according to the number of categories;

[0025] Obtaining a category quantity ratio of the regional category according to a ratio of the category quantity of the maximum category to the category quantity of the minimum category;

[0026] When the category number ratio is less than or equal to the category number target ratio, directly generating a category annotation dataset based on the region images and region categories corresponding to each of the detection regions in the historical report image;

[0027] When the category number ratio is greater than the category number target ratio, the category annotation dataset is generated after removing the area images in the category with the maximum number according to the category number target ratio.

[0028] In an exemplary embodiment of the present disclosure, the step of training the region classification model based on the category labeling dataset includes:

[0029] Dividing the category-labeled dataset into a category-labeled training set, a category-labeled validation set, and a category-labeled test set;

[0030] An initial region classification model is obtained by training the category-labeled training set, and a plurality of candidate region classification models are obtained based on the category-labeled validation set and the initial region classification model;

[0031] The region classification model is determined from the plurality of candidate region classification models according to the category labeling test set.

[0032] In an exemplary embodiment of the present disclosure, detecting the report according to the area category and the area coordinates of each detection area in the report image includes:

[0033] Determining a detection method corresponding to each detection area according to the area category of each detection area;

[0034] The report is detected according to the detection mode and the region coordinates of each detection region in the report image.

[0035] According to a second aspect of the present disclosure, a report detection device is provided, comprising:

[0036] A region detection module, configured to input the report image of the report into a pre-trained region detection model to obtain region names and region coordinates of each detection region in the report image;

[0037] A feature map acquisition module, configured to crop the report image according to the region name and region coordinates of each detection region to obtain a region feature map of each detection region;

[0038] A category detection module, configured to input the region feature map into a pre-trained region classification model to obtain a region category for each of the detection regions;

[0039] The report detection module is used to detect the report according to the area category and the area coordinates of each detection area in the report image.

[0040] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned report detection methods by executing the executable instructions.

[0041] According to a fourth aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for detecting a report sheet described in any one of the above is implemented.

[0042] The exemplary embodiments of the present disclosure may have the following beneficial effects:

[0043] In the report detection method of the example embodiment of the present disclosure, the report image is input into a pre-trained region detection model to obtain the region name and region coordinates of each detection region in the report image, and then the report image is cropped according to the region name and region coordinates to obtain the region feature map of each detection region. The region feature map is then input into a pre-trained region classification model to obtain the region category of each detection region, and finally the report is detected according to the region category and region coordinates of each detection region. The report detection method of the example embodiment of the present disclosure, by cropping the report image in units of regions to obtain the corresponding region feature map, can ensure that the region classification model obtains more image features, improves the accuracy of the region category recognition results, and thus helps to improve the efficiency of report detection.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0046] Figure 1 A schematic flow chart of a single-stage report detection method according to a related embodiment of the present disclosure is shown;

[0047] Figure 2 A schematic diagram showing a reduction in feature quantity due to scaling of an image according to a related embodiment of the present disclosure;

[0048] Figure 3 A schematic flow chart showing a method for detecting a report sheet according to an exemplary embodiment of the present disclosure is provided;

[0049] Figure 4 A schematic diagram showing a flow chart of a method for training a region detection model according to an exemplary embodiment of the present disclosure;

[0050] Figure 5 A schematic diagram of a process for obtaining a region detection model by training a region annotation dataset according to an exemplary embodiment of the present disclosure is shown;

[0051] Figure 6 A flowchart illustrating a method for training a region classification model according to an exemplary embodiment of the present disclosure is shown;

[0052] Figure 7 A schematic diagram of a process for generating a category annotation dataset according to an exemplary embodiment of the present disclosure is shown;

[0053] Figure 8 A schematic diagram of a process for obtaining a regional classification model by training a category-labeled dataset according to an exemplary embodiment of the present disclosure is shown;

[0054] Figure 9 A schematic diagram of a process for detecting a report sheet according to a specific embodiment of the present disclosure is shown;

[0055] Figure 10 Schematically showing the result of the average accuracy of the "1" category region category obtained according to the single-stage report single detection method in a related embodiment of the present disclosure;

[0056] Figure 11 Schematically showing the result graph of the average accuracy of the "1" category area category obtained according to the detection method of the report sheet in a specific embodiment of the present disclosure;

[0057] Figure 12 Schematically showing the result of the average precision of the “0” category region obtained by the single-stage report detection method in a related embodiment of the present disclosure;

[0058] Figure 13 Schematically showing the result of the average accuracy of the "0" category area category obtained by the detection method of the report sheet according to a specific embodiment of the present disclosure;

[0059] Figure 14 A block diagram showing a report detection device according to an exemplary embodiment of the present disclosure;

[0060] Figure 15A schematic structural diagram of a computer system suitable for implementing the electronic device according to the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0061] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0062] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0063] In some production processes, paper documentation is essential, making the completion and archiving of paper reports essential. To ensure the efficient and safe operation of each production process, after manually completing the report, an auditor must verify the contents of the paper report. Because manual verification is time-consuming and labor-intensive, some relevant embodiments utilize object detection techniques from deep learning to improve detection efficiency.

[0064] In some related embodiments, the detection of the report can be achieved through two target detection technologies, namely one-stage and two-stage. One-stage mainly uses the end-to-end idea and directly outputs coordinates and classifications. The main representatives are the YOLO (You only look once) series of models. One-stage has the characteristics of extremely fast detection speed and general classification effect. Two-stage first detects the candidate area and then outputs the category and coordinates of the target. The main representatives are the Faster R-CNN (Faster Regions with Convolutional Neural Network Features) model. Two-stage has the characteristics of high classification accuracy and poor detection speed.

[0065] Figure 1 A schematic diagram of a single-stage report detection method according to a related embodiment of the present disclosure is shown. The report 101 is scaled and then input into the YOLOv4 model. The YOLOv4 model maintains an extremely high detection rate while also having a high classification effect, and is therefore often given priority in actual production processes. However, when using the YOLOv4 model to detect the content of the report, the following problems may arise:

[0066] 1. The YOLOv4 model has a scaling behavior for the report image, and scaling the image during detection will lead to a decrease in the feature quantity. The specific situation is as follows Figure 2 As shown, picture 201 is a cropped original picture, and picture 202 is a cropped scaled picture.

[0067] 2. The imbalance of samples in the model training dataset resulted in low recognition accuracy for certain classes. Analysis of the dataset and model revealed that classes with a large proportion of samples in the dataset had higher AP values ​​(Average Precision), while classes with a small proportion had lower AP values. This situation seriously affected the detection of report content.

[0068] Based on the above problems, this exemplary embodiment first provides a method for detecting a report. Figure 3 As shown, the detection method of the above report sheet may include the following steps:

[0069] Step S310: Input the report image of the report into the pre-trained region detection model to obtain the region name and region coordinates of each detection region in the report image.

[0070] Step S320: Crop the report image according to the region name and region coordinates of each detection region to obtain a region feature map of each detection region.

[0071] Step S330: Input the region feature map into the pre-trained region classification model to obtain the region category of each detection region.

[0072] Step S340: Detect the report according to the area category and area coordinates of each detection area in the report image.

[0073] In the report detection method of the example embodiment of the present disclosure, the report image is input into a pre-trained region detection model to obtain the region name and region coordinates of each detection region in the report image, and then the report image is cropped according to the region name and region coordinates to obtain the region feature map of each detection region. The region feature map is then input into a pre-trained region classification model to obtain the region category of each detection region, and finally the report is detected according to the region category and region coordinates of each detection region. The report detection method of the example embodiment of the present disclosure, by cropping the report image in units of regions to obtain the corresponding region feature map, can ensure that the region classification model obtains more image features, improves the accuracy of the region category recognition results, and thus helps to improve the efficiency of report detection.

[0074] Next, combine Figures 4 to 8 The above steps of this exemplary embodiment are described in more detail.

[0075] In step S310, the report image of the report is input into a pre-trained region detection model to obtain the region name and region coordinates of each detection region in the report image.

[0076] In this example implementation, the report image can first be obtained by scanning the report, and then the report image can be input into a pre-trained region detection model to obtain the recognition result of the region detection model, wherein the recognition result of the region detection model includes two parts: the region name and the region coordinates corresponding to each detection area.

[0077] In this example implementation, Figure 4 As shown, the training method of the region detection model can specifically include the following steps:

[0078] Step S410: Acquire a history report image, wherein the history report image includes region marking boxes corresponding to each detection region.

[0079] In this example implementation, report images within a period of time can be extracted from the system, and then images with clear report content can be screened out, while images with incomplete report content can be removed, and the screened report images can be used as original report images.

[0080] After acquiring the original report images within a period of time, the content that needs to be detected in the original report images is annotated. The area annotation box is used to frame the content that needs to be detected in the report, and the category of the content is annotated. Then, a report dataset is generated based on the annotated original report images.

[0081] Step S420: Determine the detection areas corresponding to the area marking boxes with the same relative positions in each historical report image as having the same area name.

[0082] In this example implementation, for a batch of historical report images of the same format, the detection areas in the area annotation boxes with the same relative position in each historical report image can be merged into a group based on the annotation information in the report data set, and named with the same area name, such as area 1, area 2, area 3,..., area n.

[0083] For example, if Figure 1 As shown, the "XXXX Report" area can be named Area 1; the "Yes / No" check area, regardless of whether it is checked, no, or not checked, is named Area 2; the "Satisfied / Unsatisfied" check area, regardless of whether it is checked, unsatisfied, or not checked, is named Area 3; the filling area of ​​"Item 10" is named Area 4; the filling area of ​​"Item 11" is named Area 5; and the filling area of ​​"Item 12" is named Area 6.

[0084] Step S430: Generate a region annotation dataset based on the region annotation boxes and region names corresponding to each detection region, and train a region detection model based on the region annotation dataset.

[0085] In this example implementation, Figure 5 As shown in the figure, the region detection model is trained based on the region annotation dataset, which can specifically include the following steps:

[0086] Step S510: Divide the region annotation dataset into a region annotation training set, a region annotation validation set, and a region annotation test set.

[0087] For example, the region annotation dataset can be divided into a region annotation training set, a region annotation validation set, and a region annotation test set in a ratio of 7:1:2.

[0088] Step S520: An initial region detection model is obtained by training based on the region annotation training set, and multiple candidate region detection models are obtained based on the region annotation verification set and the initial region detection model.

[0089] In this example implementation, the region detection model is a target detection model, such as a YOLOv4 model. In addition, other types of target detection models can also be used. This example implementation does not specifically limit this, and only uses the YOLOv4 model as an example for specific description.

[0090] By inputting the region annotation training set into the YOLOv4 model for training, we can obtain an initial region detection model. Then, using the region annotation validation set, we select several sets of model parameters with the smallest validation set loss function to obtain candidate region detection models. For example, we can save three model parameters with the smallest validation set loss function to obtain three candidate region detection models.

[0091] Step S530: Determine a region detection model from multiple candidate region detection models based on the region annotation test set.

[0092] Finally, a better candidate region detection model is determined from multiple candidate region detection models through the region annotation test set as the final region detection model.

[0093] In step S320 , the report image is cropped according to the region name and region coordinates of each detection region to obtain a region feature map of each detection region.

[0094] In this example implementation, the region detection model is first used to obtain the region names and coordinates of each detection region. The original report image to be processed is then cropped using these region names and coordinates to generate the corresponding region feature map. Using the original image cropping method ensures that the region classification model captures more image features, improving the accuracy of subsequent region classification results.

[0095] In step S330, the region feature map is input into a pre-trained region classification model to obtain the region category of each detection region.

[0096] In this example implementation, after obtaining the regional feature maps of each detection area, the regional feature maps can be sequentially input into the pre-trained regional classification model to obtain the recognition results of the regional categories of each detection area. The regional category is the actual category of each detection area. For example, Figure 1 For the "yes / no" check area in the Figure 1 For the filling area of ​​"Item 12", the area categories can include "no content to fill in" and "content to fill in". Among them, "content to fill in" can be further divided into several specific types such as "descriptive content", "digital content", and "signature".

[0097] In this example implementation, Figure 6As shown, the training method of the region classification model can specifically include the following steps:

[0098] Step S610: Acquire a history report image, wherein the history report image includes region marking boxes and region categories corresponding to each detection region.

[0099] In this example implementation, report images within a period of time can be extracted from the system, and then images with clear report content can be screened out, while images with incomplete report content can be removed, and the screened report images can be used as original report images.

[0100] After acquiring the original report images within a period of time, the content that needs to be detected in the original report images is annotated. The area annotation box is used to frame the content that needs to be detected in the report, and the category of the content is annotated. Then, a report dataset is generated based on the annotated original report images.

[0101] Step S620: Crop the historical report image according to the region marking frame to obtain region images corresponding to each detection region.

[0102] The historical report image is cropped using the coordinates of the region annotation box to obtain the region images corresponding to the detection regions of each region category.

[0103] Step S630: Generate a category annotation dataset based on the region images and region categories corresponding to each detection region in the historical report image, and train a region classification model based on the category annotation dataset.

[0104] In this example implementation, Figure 7 As shown, generating a category annotation dataset based on the region images and region categories corresponding to each detection region in the historical report image may include the following steps:

[0105] Step S710: Obtain the category quantity of each category in each area category, and determine the maximum number of categories and the minimum number of categories corresponding to each area category according to the category quantity.

[0106] In this example implementation, when training a regional classification model, the category annotation dataset may be based on the region in the regional annotation dataset, and the number of categories with a larger proportion in the region may be reduced, thereby reducing the impact of sample imbalance on the model.

[0107] Specifically, we can first obtain the number of categories in each area category, and then determine the maximum and minimum number of categories corresponding to each area category based on the number of categories. Figure 1Region 6 contains two categories, labeled "0" and "1." The category corresponding to label "0" is "No content," and the category corresponding to label "1" is "Content is filled." Obtain the number of categories corresponding to each of the two categories labeled "0" and "1." The category with the larger number of categories is considered the category with the largest number of categories, and the category with the smaller number of categories is considered the category with the smallest number of categories.

[0108] Step S720: Obtain a ratio of the number of categories of the regional categories according to the ratio of the number of categories of the maximum number of categories to the number of categories of the minimum number of categories.

[0109] According to the ratio of the number of categories of the largest number of categories to the number of categories of the smallest number of categories, the ratio of the number of categories of regional categories n can be obtained. i For example, the ratio of the number of labels "0" to "1" in region 6 is 1:152, so the ratio of the number of categories in region 6 is n i It is 152.

[0110] Step S730: When the category quantity ratio is less than or equal to the category quantity target ratio, a category annotation dataset is directly generated based on the region images and region categories corresponding to each detection region in the historical report image.

[0111] In this example implementation, a target ratio of the number of categories may be preset, and the number of region images of different categories in each region of the category annotation dataset may be adjusted according to the target ratio of the number of categories.

[0112] When the ratio of the number of categories is less than or equal to the target ratio of the number of categories, it means that the number of samples of regional images of different categories is basically balanced. The category annotation dataset can be directly generated based on the regional images and regional categories corresponding to each detection area in the historical report image without adjustment.

[0113] Step S740. When the category number ratio is greater than the category number target ratio, the regional images in the largest number category are eliminated according to the category number target ratio to generate a category annotation dataset.

[0114] When the ratio of the number of categories is greater than the target ratio of the number of categories, it means that the number gap between different categories in the area is too large. Therefore, the area images in the largest number category can be eliminated according to the target ratio of the number of categories, and then the category annotation dataset can be generated based on the remaining area images.

[0115] For example, suppose the target ratio of the number of categories is 20, and the ratio of the number of categories in area 6 is n i is 152, so it is necessary to randomly remove a certain number of categories with label “1” in region 6 while maintaining the number of label “0”, so that n iThe value range reaches 1≤n i ≤20, for example, the ratio of the number of labels "0" to "1" is finally made 1:20, and the same is true for other areas. Then, a category annotation dataset is generated based on the remaining area images after the elimination.

[0116] In this example implementation, the class imbalance problem is solved in the regional classification model. The impact of sample imbalance in each region on the model is reduced, and the number of sample classes in each region is guaranteed to change without affecting the categories of other regions.

[0117] In this example implementation, Figure 8 As shown in the figure, the regional classification model is trained based on the category labeling dataset, which can specifically include the following steps:

[0118] Step S810: Divide the category-labeled dataset into a category-labeled training set, a category-labeled validation set, and a category-labeled test set.

[0119] For example, the category-labeled data can be divided into a category-labeled training set, a category-labeled validation set, and a category-labeled test set in a ratio of 7:1:2.

[0120] Step S820: An initial region classification model is obtained by training based on the category-labeled training set, and multiple candidate region classification models are obtained based on the category-labeled validation set and the initial region classification model.

[0121] In this example implementation, the region classification model can use ResNet50 (Deep residual network), remove the last fully connected layer of the ResNet50 model, and then add a Dense (fully connected) layer with 512 neurons and an activation function of "relu" to it, and then add a Dense layer with the total number of neurons in the category annotation dataset and an activation function of "softmax" to it. In this way, the ResNet50 model can output the corresponding region category.

[0122] By inputting the category-labeled training set into the ResNet50 model for training, an initial region classification model can be obtained. Then, several groups of model parameters with smaller validation set loss functions are selected through the category-labeled validation set to obtain the candidate region classification model.

[0123] Step S830: Determine a region classification model from multiple candidate region classification models based on the category-labeled test set.

[0124] Finally, a better candidate region classification model is determined from multiple candidate region classification models through the category labeling test set as the final region classification model.

[0125] In step S340, the report is detected according to the area category and area coordinates of each detection area in the report image.

[0126] In this example implementation, the report can be inspected based on the coordinates of the detection areas output by the region detection model and the categories output by the region classification model. Specifically, the detection method corresponding to each detection area can be determined based on the category of each detection area. The report can then be inspected based on the detection method and coordinates of each detection area in the report image.

[0127] For example, in Figure 1 In the example, the area category of "Item 12" in area 6 is "Signature". Therefore, it can be identified using OCR (Optical Character Recognition), and its position in the report can be determined based on the area coordinates. The identified content can be tested, for example, by comparing it with the information in the system to determine whether the content filled in the area is correct.

[0128] like Figure 9 The figure shows a complete flow chart of a method for detecting a report sheet in a specific embodiment of the present disclosure, which is an example of the above steps in this exemplary embodiment. The specific steps of the flow chart are as follows:

[0129] Step S902: Collect report images.

[0130] Step S904: Label the picture.

[0131] The content that needs to be detected in the report image is marked, and the area annotation box is used to frame the content that needs to be detected in the report, and the category of the content is marked. Then, a report dataset is generated based on the marked original report image.

[0132] Step S906: Obtain a region annotation dataset.

[0133] Generate a region annotation dataset based on the region annotation boxes and region names corresponding to each detection region.

[0134] Step S908: Train the region detection model.

[0135] The region annotation dataset is divided into a region annotation training set, a region annotation validation set and a region annotation test set, and the region detection model is trained using the region annotation training set and the region annotation validation set.

[0136] Step S910: Select a better region detection model.

[0137] Select a better region detection model through the region annotation test set.

[0138] Step S912: Obtain the recognition result of the region detection model.

[0139] The recognition results of the region detection model include two parts: the region name and region coordinates corresponding to each detection region.

[0140] Step S914: Obtain a category labeling dataset.

[0141] A category annotation dataset is generated based on the region images and region categories corresponding to each detection region in the historical report image.

[0142] Step S916: Reduce class imbalance.

[0143] Taking regions as units, the impact of sample imbalance in each region on the model is reduced.

[0144] Step S918: Train the region classification model.

[0145] The category annotation dataset is divided into a category annotation training set, a category annotation validation set and a category annotation test set, and the region classification model is trained using the category annotation training set and the category annotation validation set.

[0146] Step S920: Select a better regional classification model.

[0147] Select a better regional classification model through the category-labeled test set.

[0148] Step S922: Cutting the report.

[0149] The report image is cropped according to the region name and region coordinates of each detection region to obtain a region feature map of each detection region.

[0150] Step S924: Obtain the recognition result of the region classification model.

[0151] The regional feature map is input into the regional classification model to obtain the regional category of each detection area.

[0152] Step S926: Report sheet detection.

[0153] The report is detected according to the region category and region coordinates of each detection region in the report image.

[0154] In this example implementation, the quality inspection content is detected using the YOLOv4 model on a region-by-region basis, and the ResNet50 model is then used to accurately classify the quality inspection area content. Combined with a sample generation scheme based on regions, accurate classification of the report area to be inspected can be achieved, thereby improving the efficiency of report inspection.

[0155] In Figure 1 In the test experiment taking region 6 in the image as an example, the original ratio of the number of labels "0" to "1" in region 6 is 1:152. By keeping the number of label "0" and randomly removing the pictures with label "1", the ratio of the number of labels "0" to "1" is finally reduced to 1:20.

[0156] like Figure 10 The following is the result of the AP value of the "1" category obtained by directly using the YOLOv4 model, as shown in Figure 11 The figure shows the AP value for the "1" region, obtained using the report detection method according to a specific embodiment of the present disclosure. The AP value is the shaded area. Because the "1" region is relatively large, the AP values ​​obtained by the two methods are not significantly different.

[0157] like Figure 12 The following is the result of the AP value of the "0" category obtained by directly using the YOLOv4 model, as shown in Figure 13 The figure shows the AP value for the "0" region category obtained using the report detection method according to a specific embodiment of the present disclosure. It can be seen that compared to using the YOLOv4 model directly, the AP value for the "0" region category obtained using the method in this example embodiment increased from 15.83% to 45%, a 29.17% increase in AP value. This shows that the report detection method in this example embodiment can more accurately classify the report area to be inspected.

[0158] It should be noted that although the steps of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0159] Furthermore, the present disclosure also provides a detection device for a report sheet. Figure 14 As shown, the report detection device may include an area detection module 1410, a feature map acquisition module 1420, a category detection module 1430, and a report detection module 1440.

[0160] The region detection module 1410 may be used to input the report image of the report into a pre-trained region detection model to obtain the region name and region coordinates of each detection region in the report image;

[0161] The feature map acquisition module 1420 can be used to crop the report image according to the region name and region coordinates of each detection region to obtain a region feature map of each detection region;

[0162] The category detection module 1430 may be used to input the region feature map into a pre-trained region classification model to obtain the region category of each detection region;

[0163] The report detection module 1440 may be configured to detect the report based on the region category and region coordinates of each detection region in the report image.

[0164] In some exemplary embodiments of the present disclosure, the report detection device provided by the present disclosure may further include a region detection model training module, which may include a historical data acquisition unit, a region name determination unit, and a region detection model training unit. In particular:

[0165] The historical data acquisition unit may be used to acquire a historical report image, wherein the historical report image includes a region marking frame corresponding to each detection region;

[0166] The region name determination unit may be used to determine the detection regions corresponding to the region marking frames with the same relative positions in each historical report image as having the same region name;

[0167] The region detection model training unit can be used to generate a region annotation dataset based on the region annotation boxes and region names corresponding to each detection region, and train a region detection model based on the region annotation dataset.

[0168] In some exemplary embodiments of the present disclosure, the region detection model training unit may include a region data set division unit, a candidate detection model determination unit, and a region detection model determination unit. In particular:

[0169] The regional dataset division unit can be used to divide the regional annotation dataset into a regional annotation training set, a regional annotation verification set, and a regional annotation test set;

[0170] The candidate detection model determination unit can be used to train an initial region detection model based on the region annotation training set, and obtain multiple candidate region detection models based on the region annotation verification set and the initial region detection model;

[0171] The region detection model determination unit can be used to determine a region detection model from multiple candidate region detection models based on a region annotation test set.

[0172] In some exemplary embodiments of the present disclosure, the report detection device provided by the present disclosure may further include a region classification model training module, which may include a historical data acquisition unit, a region image acquisition unit, and a region classification model training unit.

[0173] The historical data acquisition unit may be used to acquire a historical report image, wherein the historical report image includes a region marking frame and region category corresponding to each detection region;

[0174] The regional image acquisition unit can be used to crop the historical report image according to the regional annotation frame to obtain the regional image corresponding to each detection area;

[0175] The regional classification model training unit can be used to generate a category annotation dataset based on the regional images and regional categories corresponding to each detection region in the historical report image, and train the regional classification model based on the category annotation dataset.

[0176] In some exemplary embodiments of the present disclosure, the regional classification model training unit may include a category number determination unit, a category number ratio determination unit, a first category data set generation unit, and a second category data set generation unit.

[0177] The category quantity determination unit can be used to obtain the category quantity of each category in each area category, and determine the maximum number of categories and the minimum number of categories corresponding to each area category according to the category quantity;

[0178] The category quantity ratio determination unit may be configured to obtain the category quantity ratio of the regional category according to the ratio of the category quantity of the maximum category to the category quantity of the minimum category;

[0179] The first category data set generating unit may be configured to generate a category annotation data set directly based on the region images and region categories corresponding to each detection region in the historical report image when the category number ratio is less than or equal to the category number target ratio;

[0180] The second category data set generating unit may be configured to generate a category annotation data set after removing area images in the maximum number of categories according to the category number target ratio when the category number ratio is greater than the category number target ratio.

[0181] In some exemplary embodiments of the present disclosure, the regional classification model training unit may further include a category data set division unit, a candidate classification model determination unit, and a regional classification model determination unit. In particular:

[0182] The category data set division unit can be used to divide the category annotation data set into a category annotation training set, a category annotation verification set and a category annotation test set;

[0183] The candidate classification model determination unit can be used to train an initial region classification model based on the category labeling training set, and obtain multiple candidate region classification models based on the category labeling verification set and the initial region classification model;

[0184] The region classification model determination unit can be used to determine a region classification model from multiple candidate region classification models based on a category annotation test set.

[0185] In some exemplary embodiments of the present disclosure, the report form detection module 1440 may include a detection mode determination unit and a report form detection unit.

[0186] The detection mode determination unit may be configured to determine a detection mode corresponding to each detection area according to the area category of each detection area;

[0187] The report detection unit can be used to detect the report according to the detection method and area coordinates of each detection area in the report image.

[0188] The specific details of each module / unit in the detection device of the above report sheet have been described in detail in the corresponding method embodiment part and will not be repeated here.

[0189] Figure 15 A schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present invention is shown.

[0190] It should be noted that Figure 15 The computer system 1500 of the electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0191] like Figure 15 As shown, computer system 1500 includes a central processing unit (CPU) 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage unit 1508 into a random access memory (RAM) 1503. Various programs and data required for system operation are also stored in RAM 1503. CPU 1501, ROM 1502, and RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to bus 1504.

[0192] The following components are connected to the I / O interface 1505: an input section 1506 including a keyboard, a mouse, and the like; an output section 1507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1508 including a hard disk; and a communication section 1509 including a network interface card such as a LAN card or a modem. The communication section 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to the I / O interface 1505 as needed. Removable media 1511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1510 as needed, so that computer programs read therefrom can be installed into the storage section 1508 as needed.

[0193] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1509, and / or installed from a removable medium 1511. When the computer program is executed by the central processing unit (CPU) 1501, the various functions defined in the system of the present application are executed.

[0194] It should be noted that the computer-readable medium described in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0195] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0196] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device implements the method described in the above embodiments.

[0197] It should be noted that although several modules of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.

[0198] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0199] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for detecting a report, characterized in that: include: Inputting the report image of the report into a pre-trained region detection model to obtain the region name and region coordinates of each detection region in the report image; cropping the report image according to the region name and region coordinates of each detection region to obtain a region feature map of each detection region; Inputting the region feature map into a pre-trained region classification model to obtain a region category of each of the detection regions, wherein the region category is a true category of each of the detection regions; The training method of the region classification model includes: obtaining a historical report image, wherein the historical report image contains region annotation boxes and region categories corresponding to each of the detection areas; cropping the historical report image according to the region annotation boxes to obtain region images corresponding to each of the detection areas; obtaining the number of categories of each category in each of the region categories, and determining the maximum number of categories and the minimum number of categories corresponding to each of the region categories according to the number of categories; obtaining the category number ratio of the region category according to the ratio of the number of categories of the maximum number of categories to the number of categories of the minimum number of categories; when the category number ratio is less than or equal to the category number target ratio, directly generating a category annotation dataset based on the region images and region categories corresponding to each of the detection areas in the historical report image; when the category number ratio is greater than the category number target ratio, generating the category annotation dataset after eliminating the region images in the maximum number of categories according to the category number target ratio; and training the region classification model based on the category annotation dataset. The report is detected based on the area category and the area coordinates of each of the detection areas in the report image.

2. The method for detecting a report according to claim 1, wherein: The training method of the region detection model includes: Acquire a history report image, wherein the history report image includes area marking boxes corresponding to each of the detection areas; Determine the detection areas corresponding to the area marking frames having the same relative position in each of the historical report images as having the same area name; A region annotation dataset is generated according to the region annotation boxes and the region names corresponding to the respective detection regions, and the region detection model is obtained by training based on the region annotation dataset.

3. The method for detecting a report according to claim 2, wherein: The step of training the region detection model based on the region annotation dataset includes: Dividing the region annotation dataset into a region annotation training set, a region annotation validation set, and a region annotation test set; An initial region detection model is obtained by training the region annotation training set, and a plurality of candidate region detection models are obtained based on the region annotation verification set and the initial region detection model; The region detection model is determined from a plurality of candidate region detection models according to the region annotation test set.

4. The method for detecting a report according to claim 1, wherein: The region classification model is obtained by training the category labeling dataset, including: Dividing the category-labeled dataset into a category-labeled training set, a category-labeled validation set, and a category-labeled test set; An initial region classification model is obtained by training the category-labeled training set, and a plurality of candidate region classification models are obtained based on the category-labeled validation set and the initial region classification model; The region classification model is determined from the plurality of candidate region classification models according to the category labeling test set.

5. The method for detecting a report according to claim 1, wherein: Detecting the report according to the area category and the area coordinates of each detection area in the report image includes: Determining a detection method corresponding to each detection area according to the area category of each detection area; The report is detected according to the detection mode and the region coordinates of each detection region in the report image.

6. A report detection device, characterized in that: include: A region detection module, configured to input the report image of the report into a pre-trained region detection model to obtain region names and region coordinates of each detection region in the report image; A feature map acquisition module, configured to crop the report image according to the region name and region coordinates of each detection region to obtain a region feature map of each detection region; A category detection module, configured to input the region feature map into a pre-trained region classification model to obtain a region category of each of the detection regions, wherein the region category is a true category of each of the detection regions; The training method of the region classification model includes: obtaining a historical report image, wherein the historical report image contains region annotation boxes and region categories corresponding to each of the detection areas; cropping the historical report image according to the region annotation boxes to obtain region images corresponding to each of the detection areas; obtaining the number of categories of each category in each of the region categories, and determining the maximum number of categories and the minimum number of categories corresponding to each of the region categories according to the number of categories; obtaining the category number ratio of the region category according to the ratio of the number of categories of the maximum number of categories to the number of categories of the minimum number of categories; when the category number ratio is less than or equal to the category number target ratio, directly generating a category annotation dataset based on the region images and region categories corresponding to each of the detection areas in the historical report image; when the category number ratio is greater than the category number target ratio, generating the category annotation dataset after eliminating the region images in the maximum number of categories according to the category number target ratio; and training the region classification model based on the category annotation dataset. The report detection module is used to detect the report according to the area category and the area coordinates of each detection area in the report image.

7. An electronic device, characterized in that: include: processor; as well as A memory for storing one or more programs, which, when executed by the processor, enables the processor to implement the report detection method as described in any one of claims 1 to 5.

8. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the report detection method according to any one of claims 1 to 5 is implemented.

Images