Text image region detection method, related device and readable storage medium

Abstract

The present application discloses a text image region detection method, related equipment and readable storage medium. In this solution, after obtaining a text image to be subjected to region detection, an instance segmentation process is performed on the first target in the text image to obtain a region detection result of the first target, a semantic segmentation process is performed on the second target in the text image to obtain a region detection result of the second target, and finally, based on the region detection result of the first target and the region detection result of the second target, the region detection result of the text image is determined. In this solution, different methods are used to perform region detection for different targets, so as to avoid the problem that one text image region detection method cannot cover the detection of various targets, resulting in missed detection or misdetection of some targets, thereby improving the accuracy of region detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a text image region detection method, related equipment and a readable storage medium. Background Art

[0002] In some scenarios, it is necessary to detect regions of text images and identify the content of the regions. For example, in the scenario of automatic correction of primary and secondary school homework, it is necessary to detect the question number, question stem and answer area of ​​each question in the homework image, and perform content recognition on the question number, question stem and answer area of ​​each question to achieve question type retrieval, answer acquisition, automatic correction, etc. In the scenario of document layout analysis and correction, it is necessary to detect each independent region of the document image and perform content recognition on the region to achieve analysis and correction of document layout content. In these scenarios, the accuracy of region detection has a crucial impact on the accuracy of region content recognition.

[0003] Currently, there are some text image region detection schemes, however, these text image region detection schemes cannot be applied to the detection of various targets, and often some targets are missed or misdetected, resulting in low accuracy of region detection.

[0004] Therefore, how to improve the accuracy of text image region detection has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] In view of the above problems, this application proposes a text image region detection method, related equipment and readable storage medium. The specific scheme is as follows:

[0006] A text image region detection method, the method comprising:

[0007] Obtain a text image to be detected in the region;

[0008] Performing instance segmentation processing on a first target in the text image to obtain a region detection result of the first target;

[0009] Performing semantic segmentation processing on the second target in the text image to obtain a region detection result of the second target;

[0010] Based on the region detection result of the first object and the region detection result of the second object, a region detection result of the text image is determined.

[0011] Optionally, performing instance segmentation processing on the first target in the text image to obtain a region detection result of the first target includes:

[0012] The text image is input into a trained instance segmentation model, and the instance segmentation model outputs the region detection result of the first target. The instance segmentation model is trained using the training text image as a training sample, and using the real region box contained in the first target annotated with the training text image and the type of each real region box as sample labels.

[0013] Optionally, the instance segmentation model includes a feature map extraction module, a region box generation module, a region box screening module and a region box detection module;

[0014] The training process of the instance segmentation model includes:

[0015] The feature map extraction module extracts the feature map of the training text image to obtain the feature map of the training text image;

[0016] The region frame generation module determines a plurality of region frames included in a first target in the training text image based on a feature map of the training text image;

[0017] The area box screening module determines, for each area box generated by the area box generation module, a non-maximum suppression threshold corresponding to the area box based on the shape information of the area box; and based on the non-maximum suppression thresholds corresponding to each area box, screens out an area box to be detected from the multiple area boxes generated by the area box generation module, and inputs the area box to be detected into the area box detection module.

[0018] Optionally, determining a non-maximum suppression threshold corresponding to the region box based on the shape information of the region box includes:

[0019] Calculating the inclination angle and aspect ratio of the minimum circumscribed rectangular frame of the area frame;

[0020] Obtain the preset benchmark non-maximum suppression threshold, tilt angle threshold, and aspect ratio threshold;

[0021] When the inclination angle of the minimum circumscribed rectangular frame of the area frame is greater than the inclination angle threshold, and the aspect ratio of the minimum circumscribed rectangular frame of the area frame is greater than the aspect ratio threshold, the benchmark non-maximum suppression threshold is adjusted, and the adjusted non-maximum suppression threshold is determined as the non-maximum suppression threshold corresponding to the area frame; the adjusted non-maximum suppression threshold is greater than the benchmark non-maximum suppression threshold;

[0022] Otherwise, the benchmark non-maximum suppression threshold is determined as the non-maximum suppression threshold corresponding to the area box.

[0023] Optionally, the filtering out the region frame to be detected from the multiple region frames generated by the region frame generation module based on the non-maximum suppression threshold corresponding to each region frame includes:

[0024] For each area box generated by the area box generation module, calculating an intersection-over-union ratio between the area box and the area box with the highest current confidence score;

[0025] When the intersection-and-union ratio of the region box and the region box with the highest current confidence score is greater than the non-maximum suppression threshold corresponding to the region box, the region box is discarded; otherwise, the region box is determined as the region box to be detected.

[0026] Optionally, the region box detection module includes three cascaded detection submodules, the output of the previous detection submodule is used as the input of the next detection submodule; the output of the last detection submodule is used as the output of the instance segmentation model; each detection submodule is provided with a reference intersection-over-union ratio threshold;

[0027] The training process of the instance segmentation model also includes:

[0028] For each detection submodule, for a region box input into the detection submodule, the detection submodule determines an IoU threshold corresponding to the region box based on the benchmark IoU threshold and the area of ​​the region box, and assigns a label to the region box based on the IoU threshold corresponding to the region box.

[0029] Optionally, determining the IoU threshold corresponding to the region box based on the benchmark IoU threshold and the area of ​​the region box includes:

[0030] If the area of ​​the region frame is smaller than the area threshold, the base IoU threshold is adjusted to obtain an adjusted IoU threshold as the IoU threshold corresponding to the region frame, wherein the adjusted IoU threshold is smaller than the base IoU threshold;

[0031] If the area of ​​the region frame is greater than or equal to the area threshold, the reference IoU threshold is used as the IoU threshold corresponding to the region frame.

[0032] Optionally, performing semantic segmentation processing on the second target in the text image to obtain a region detection result of the second target includes:

[0033] The text image is input into a trained semantic segmentation model, and the semantic segmentation model outputs the area detection result of the second target. The semantic segmentation model is trained using the training text image as a training sample, and the real area box contained in the second target annotated with the training text image and the type of each real area box as sample labels.

[0034] Optionally, the determining the region detection result of the text image based on the region detection result of the first target and the region detection result of the second target includes:

[0035] Optimizing the region detection result of the first target to obtain an optimized region detection result of the first target;

[0036] The region detection result of the first target after the optimization process and the region detection result of the second target are used to determine the region detection result of the text image.

[0037] Optionally, the optimizing the region detection result of the first target to obtain the optimized region detection result of the first target includes:

[0038] For each region box type in the first target detection result, calculating an intersection-over-union ratio of any two region boxes corresponding to the region box type;

[0039] Get the preset intersection-over-union ratio threshold of the same type of region boxes;

[0040] If the IoU of the two region frames is greater than the IoU threshold of the region frames of the same type, the region frame with the lower confidence score among the two region frames is discarded.

[0041] Optionally, the optimizing the region detection result of the first target to obtain the optimized region detection result of the first target includes:

[0042] For two different types of area frames whose positions meet a preset condition in the first target detection result, calculating an intersection-over-union ratio of the two area frames;

[0043] Get preset intersection-over-union ratio thresholds for different types of region boxes;

[0044] If the IoU of the two region frames is greater than the IoU threshold of region frames of different types, the region frame with the lower confidence score among the two region frames is discarded.

[0045] A text image area detection device, the device comprising:

[0046] A text image acquisition unit, used to acquire a text image to be subjected to area detection;

[0047] An instance segmentation processing unit, configured to perform instance segmentation processing on a first target in the text image to obtain a region detection result of the first target;

[0048] A semantic segmentation processing unit, configured to perform semantic segmentation processing on the second target in the text image to obtain a region detection result of the second target;

[0049] The area detection result determining unit is used to determine the area detection result of the text image based on the area detection result of the first object and the area detection result of the second object.

[0050] Optionally, the instance segmentation processing unit includes:

[0051] An instance segmentation model application unit is used to input the text image into a trained instance segmentation model, and the instance segmentation model outputs the area detection result of the first target. The instance segmentation model is trained using the training text image as a training sample, and the real area box contained in the first target annotated with the training text image and the type of each real area box as sample labels.

[0052] Optionally, the instance segmentation model includes a feature map extraction module, a region box generation module, a region box screening module and a region box detection module;

[0053] The device further comprises: an instance segmentation model training unit, wherein the instance segmentation model training unit comprises:

[0054] A feature map extraction unit, configured to extract a feature map from the training text image using the feature map extraction module to obtain a feature map of the training text image;

[0055] A region frame generating unit, configured to determine, by using the region frame generating module, a plurality of region frames included in a first target in the training text image based on a feature map of the training text image;

[0056] The area frame screening unit is used to use the area frame screening module to determine, for each area frame generated by the area frame generation module, a non-maximum suppression threshold corresponding to the area frame based on the shape information of the area frame; and based on the non-maximum suppression thresholds corresponding to each area frame, screen out an area frame to be detected from the multiple area frames generated by the area frame generation module, and input the area frame to be detected into the area frame detection module.

[0057] Optionally, the area frame screening unit includes:

[0058] A calculation unit, used to calculate the inclination angle and aspect ratio of the minimum circumscribed rectangular frame of the area frame;

[0059] A threshold acquisition unit, used to acquire a preset reference non-maximum suppression threshold, a tilt angle threshold, and an aspect ratio threshold;

[0060] A non-maximum suppression threshold determination unit is used to adjust the benchmark non-maximum suppression threshold when the inclination angle of the minimum circumscribed rectangular frame of the area frame is greater than the inclination angle threshold and the aspect ratio of the minimum circumscribed rectangular frame of the area frame is greater than the aspect ratio threshold, and determine the adjusted non-maximum suppression threshold as the non-maximum suppression threshold corresponding to the area frame; the adjusted non-maximum suppression threshold is greater than the benchmark non-maximum suppression threshold; otherwise, determine the benchmark non-maximum suppression threshold as the non-maximum suppression threshold corresponding to the area frame.

[0061] Optionally, the area frame screening unit includes:

[0062] an intersection-over-union ratio calculation unit, configured to calculate, for each region box generated by the region box generation module, an intersection-over-union ratio between the region box and the region box with the highest current confidence score;

[0063] The unit for determining the area frame to be detected is used to discard the area frame when the intersection-and-union ratio of the area frame and the area frame with the highest current confidence score is greater than the non-maximum suppression threshold corresponding to the area frame; otherwise, determine the area frame as the area frame to be detected.

[0064] Optionally, the region box detection module includes three cascaded detection submodules, the output of the previous detection submodule is used as the input of the next detection submodule; the output of the last detection submodule is used as the output of the instance segmentation model; each detection submodule is provided with a reference intersection-over-union ratio threshold;

[0065] The instance segmentation model training unit also includes:

[0066] A label assignment unit is used for, for each detection submodule, for a region frame input into the detection submodule, the detection submodule determines an IoU threshold corresponding to the region frame based on the benchmark IoU threshold and the area of ​​the region frame, and assigns a label to the region frame based on the IoU threshold corresponding to the region frame.

[0067] Optionally, the label allocating unit includes:

[0068] an IoU threshold adjustment unit, configured to adjust the base IoU threshold if the area of ​​the region frame is smaller than the area threshold, and obtain the adjusted IoU threshold as the IoU threshold corresponding to the region frame, wherein the adjusted IoU threshold is smaller than the base IoU threshold;

[0069] If the area of ​​the region frame is greater than or equal to the area threshold, the reference IoU threshold is used as the IoU threshold corresponding to the region frame.

[0070] Optionally, the semantic segmentation processing unit includes:

[0071] A semantic segmentation model application unit is used to input the text image into a trained semantic segmentation model, and the semantic segmentation model outputs the area detection result of the second target. The semantic segmentation model is trained using the training text image as a training sample and the real area box contained in the second target annotated with the training text image and the type of each real area box as sample labels.

[0072] Optionally, the area detection result determining unit includes:

[0073] an optimization unit, configured to optimize the region detection result of the first target to obtain the region detection result of the first target after optimization;

[0074] The area detection result determination subunit is used to determine the area detection result of the text image based on the area detection result of the first target after the optimization process and the area detection result of the second target.

[0075] Optionally, the optimization unit includes:

[0076] The same type region frame optimization unit is used to calculate, for each region frame type in the first target detection result, an intersection-and-union ratio of any two region frames corresponding to the region frame type; obtain a preset intersection-and-union ratio threshold of the same type region frames; if the intersection-and-union ratio of the two region frames is greater than the intersection-and-union ratio threshold of the same type region frames, discard the region frame with a lower confidence score among the two region frames.

[0077] Optionally, the optimization unit includes:

[0078] The different types of region frame optimization unit is used to calculate the intersection-and-union ratio of two region frames of different types whose positions meet the preset conditions in the first target detection result; obtain a preset intersection-and-union ratio threshold of different types of region frames; if the intersection-and-union ratio of the two region frames is greater than the intersection-and-union ratio threshold of different types of region frames, then discard the region frame with a lower confidence score among the two region frames.

[0079] A text image region detection device, comprising a memory and a processor;

[0080] The memory is used to store programs;

[0081] The processor is used to execute the program to implement the various steps of the text image area detection method as described above.

[0082] A readable storage medium stores a computer program, and when the computer program is executed by a processor, each step of the text image region detection method described above is implemented.

[0083] By means of the above technical solution, the present application discloses a text image region detection method, related equipment and readable storage medium. In this solution, after obtaining a text image to be subjected to region detection, an instance segmentation process is performed on the first target in the text image to obtain a region detection result of the first target, a semantic segmentation process is performed on the second target in the text image to obtain a region detection result of the second target, and finally, based on the region detection result of the first target and the region detection result of the second target, the region detection result of the text image is determined. In this solution, different methods are used to perform region detection for different targets, so as to avoid the problem that one text image region detection method cannot cover the detection of various targets, resulting in missed detection or misdetection of some targets, thereby improving the accuracy of region detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0085] Figure 1 A flowchart of a text image region detection method disclosed in an embodiment of the present application;

[0086] Figure 2 A schematic diagram of the structure of an instance segmentation model disclosed in an embodiment of the present application;

[0087] Figure 3 A schematic diagram of the structure of another instance segmentation model disclosed in an embodiment of the present application;

[0088] Figure 4 A schematic diagram of the structure of a text image region detection device disclosed in an embodiment of the present application;

[0089] Figure 5 This is a hardware structure block diagram of a text image area detection device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0091] In text images, the areas to be detected are often of different sizes and shapes. For example, in the scenario of automatic correction of primary and secondary school homework, the areas corresponding to the answer areas of different question types are often of different sizes. The answer areas of essay questions are larger and easier to detect, while the answer areas of multiple-choice questions and fill-in-the-blank questions are smaller and more closely distributed, making them more difficult to detect. In addition, in the process of generating text images, it is easy to cause distortion of the area to be detected. For example, when taking pictures of primary and secondary school homework, if the homework is bent, the area corresponding to the bent part in the text image will be distorted. These situations increase the difficulty of text image area detection.

[0092] There are some text image region detection schemes, however, these text image region detection schemes cannot be applied to the detection of various targets, and often some targets are missed or misdetected, resulting in low accuracy of region detection.

[0093] In order to improve the accuracy of text image region detection, the inventor of the present application, after in-depth research, proposed a text image region detection method. In this method, different methods are used to perform region detection for different targets, thereby avoiding the problem that one text image region detection method cannot cover the detection of various targets, resulting in missed detection or misdetection of some targets, thereby improving the accuracy of region detection.

[0094] Next, the text image region detection method provided by the present application is introduced through the following embodiments.

[0095] Reference Figure 1 , Figure 1 The following is a flow chart of a text image region detection method disclosed in an embodiment of the present application. The method may include:

[0096] Step S101: Acquire a text image to be subjected to region detection.

[0097] In the present application, the text image to be subjected to region detection may be any text image that has a region detection requirement, for example, it may be an image of primary and secondary school homework, an image of a test paper, etc. This application does not impose any limitation on this.

[0098] In the present application, the text can be photographed based on various image acquisition devices, such as cameras and scanners, to obtain the corresponding text image. The pre-stored text image can also be directly obtained from various storage devices. This application does not impose any limitation on this.

[0099] Step S102: performing instance segmentation processing on the first target in the text image to obtain a region detection result of the first target.

[0100] It should be noted that the instance segmentation method has better results for regions with regular areas and larger sizes. Therefore, in the present application, the instance segmentation method can be used to perform region detection on the first target. The first target can be a target in the text image that is suitable for instance segmentation processing, which can be specifically determined based on scene requirements.

[0101] As an implementable method, in the present application, the first target may be a target with a larger area in the text image, for example, a target with an area greater than a preset threshold. Taking the automatic grading scenario of primary and secondary school homework as an example, the first target may be a large question. The area detection result of the first target is used to indicate the different types of area boxes contained in the first target. Still taking the automatic grading scenario of primary and secondary school homework as an example, the area detection result of the first target may be an area box for indicating the question number of a large question, an area box for indicating the stem of a large question, and an area box for indicating the answer area of ​​a large question.

[0102] In the present application, a neural network model can be used to implement instance segmentation processing of the first target in the text image. The neural network model can adopt an existing general instance segmentation model, but the existing general instance segmentation model may not achieve good results for certain scenarios. Therefore, in the present application, an instance segmentation model can be retrained according to scenario requirements to implement instance segmentation processing of the first target in the text image, which will be explained in detail through subsequent embodiments.

[0103] Step S103: performing semantic segmentation processing on the second target in the text image to obtain a region detection result of the second target.

[0104] It should be noted that the instance segmentation method has a relatively poor detection effect on small, dense, distorted, and long regions. As a text segmentation method, the semantic segmentation method does not have high requirements on region size and shape. Therefore, in this application, the semantic segmentation method can be used to perform region detection on the second target. The second target can be a target in the text image that is suitable for semantic segmentation processing, which can be determined based on scene requirements.

[0105] As an implementable method, in the present application, the second target can be a target with a smaller area in the text image, for example, a target with an area smaller than a preset threshold. Taking the automatic grading scenario of primary and secondary school homework as an example, the second target can be a specific question type such as fill-in-the-blank questions and multiple-choice questions. The area detection result of the second target is used to indicate the different types of area boxes contained in the second target. Still taking the automatic grading scenario of primary and secondary school homework as an example, the area detection result of the second target can be an area box for indicating the stem of the fill-in-the-blank question and an area box for the answer area.

[0106] In the present application, a neural network model can be used to implement semantic segmentation processing of the second target in the text image. The neural network model can adopt an existing general semantic segmentation model, but the existing general semantic segmentation model may not be able to achieve good results for certain scenarios. Therefore, in the present application, a semantic segmentation model can be retrained according to scenario requirements to implement semantic segmentation processing of the second target in the text image, which will be explained in detail through subsequent embodiments.

[0107] Step S104: Determine the region detection result of the text image based on the region detection result of the first object and the region detection result of the second object.

[0108] In the present application, after obtaining the region detection result of the first target and the region detection result of the second target, as an implementation method, the region detection result of the first target and the region detection result of the second target can be directly used as the region detection result of the text image. As another implementation method, the region detection result of the text image can be determined after post-processing the region detection result of the first target and / or the region detection result of the second target.

[0109] This embodiment discloses a text image region detection method. In this scheme, after obtaining a text image to be subjected to region detection, an instance segmentation process is performed on the first target in the text image to obtain a region detection result of the first target, a semantic segmentation process is performed on the second target in the text image to obtain a region detection result of the second target, and finally, based on the region detection result of the first target and the region detection result of the second target, the region detection result of the text image is determined. In this method, different methods are used to perform region detection for different targets, so as to avoid the problem that one text image region detection method cannot cover the detection of various targets, resulting in missed detection or wrong detection of some targets. Therefore, the accuracy of region detection can be improved.

[0110] In another embodiment of the present application, a specific implementation method of performing instance segmentation processing on the first target in the text image in step S102 to obtain a region detection result of the first target is introduced. The method may include:

[0111] The text image is input into a trained instance segmentation model, and the instance segmentation model outputs the region detection result of the first target. The instance segmentation model is trained using the training text image as a training sample, and using the real region box contained in the first target annotated with the training text image and the type of each real region box as sample labels.

[0112] It should be noted that, for different scenarios, the training text image can be the text image in the current scene, and the real area frame and the type of each real area frame included in the first target annotated with the training text image are also adapted to the current scene. For ease of understanding, taking the primary and secondary school homework scenario as an example, the training text image can be a large number of primary and secondary school homework images, the first target can be a large question type (such as essay questions), and the types of real area frames included in the first target can include question number, question stem, answer area, etc.

[0113] As an implementation option, see Figure 2 , Figure 2 Schematic diagram of the structure of an instance segmentation model disclosed in the embodiment of the present application. Figure 2 As shown, the instance segmentation model includes a feature map extraction module, a region box generation module, a region box screening module and a region box detection module.

[0114] based on Figure 2 The instance segmentation model shown in the figure, the present application also discloses the training process of the instance segmentation model, which may include the following steps:

[0115] Step S201: the feature map extraction module extracts a feature map from the training text image to obtain a feature map of the training text image.

[0116] In the present application, as one feasible implementation method, the feature map extraction module can be implemented using a residual network (e.g., ResNet50). As another feasible implementation method, the feature map extraction module can be implemented using a residual network (e.g., ResNet50) plus a feature pyramid network (FPN) to enhance the expressive power of features and cover multi-scale information.

[0117] Step S202: the region frame generation module determines a plurality of region frames included in a first object in the training text image based on a feature map of the training text image.

[0118] In the present application, as an implementable method, the region box generation module can be implemented by using the RPN (Region Proposal Network) of Faster-RCNN. It can also be implemented by using the RPN of Cascade MaskRCNN.

[0119] In this step, a region frame of different sizes and different aspect ratios is generated in the feature map of the training text image corresponding to each position of the first object in the training text image.

[0120] Step S203: The area box screening module determines a non-maximum suppression threshold corresponding to each area box generated by the area box generation module; and based on the non-maximum suppression threshold corresponding to each area box, screens out an area box to be detected from the multiple area boxes generated by the area box generation module, and inputs the area box to be detected into the area box detection module.

[0121] In the region frames generated in step S202, there are often region frames with too many overlapping areas. If all of these region frames are processed, the training time will increase and the training effect will be affected. Therefore, in this application, the region frames generated by the region frame generation module are screened by the region frame screening module to determine the region frames to be detected for processing.

[0122] At present, in RPN, NMS (Non-maximum suppression) is performed on region boxes with too much overlapping area. Specifically, by setting a fixed non-maximum suppression threshold (usually 0.7), NMS is performed based on the non-maximum suppression threshold to achieve suppression of region boxes with too much overlapping area. However, among the multiple region boxes contained in the first target in the training text image generated in step S202, there are some region boxes that do not want to be suppressed. If NMS is performed using the above-mentioned fixed non-maximum suppression threshold, these region boxes will be suppressed, resulting in poor detection effect of the model for these region boxes. Therefore, in the present application, different non-maximum suppression thresholds are used to perform NMS for different region boxes generated by the region box generation module to reduce the suppression ability for some region boxes.

[0123] As an implementable method, the non-maximum suppression threshold corresponding to the area box can be determined based on the shape information of the area box. In the present application, the shape information of the area box can be characterized in a variety of ways. As an implementable method, the shape information of the area box can be characterized based on the inclination angle and aspect ratio of the minimum circumscribed rectangle of the area box. Based on this, the non-maximum suppression threshold corresponding to the area box can be determined based on the shape information of the area box, which can include the following steps: calculating the inclination angle and aspect ratio of the minimum circumscribed rectangle of the area box; obtaining a preset reference non-maximum suppression threshold, an inclination angle threshold, and an aspect ratio threshold; when the inclination angle of the minimum circumscribed rectangle of the area box is greater than the inclination angle threshold, and the aspect ratio of the minimum circumscribed rectangle of the area box is greater than the aspect ratio threshold, the reference non-maximum suppression threshold is adjusted, and the adjusted non-maximum suppression threshold is determined as the non-maximum suppression threshold corresponding to the area box; the adjusted non-maximum suppression threshold is greater than the reference non-maximum suppression threshold; otherwise, the reference non-maximum suppression threshold is determined as the non-maximum suppression threshold corresponding to the area box.

[0124] For ease of understanding, assume that angle is the inclination angle of the minimum circumscribed rectangular frame of the region box, thr a is the tilt angle threshold, which can be roughly obtained by the prior of the first target of the training text image, ratio is the aspect ratio of the minimum circumscribed rectangular box of the region box, thr r is the aspect ratio threshold, which can be roughly obtained by the prior of the first target of the training text image. IoU0 is the baseline non-maximum suppression threshold. For example, in the RPN training stage, this value is usually 0.7. Then the non-maximum suppression threshold thr corresponding to the region box is IoU It can be calculated based on the following formula:

[0125]

[0126] Based on the above, as an implementable method, based on the non-maximum suppression threshold corresponding to each area box, the area box to be detected is screened out from the multiple area boxes generated by the area box generation module, including: for each area box generated by the area box generation module, calculating the intersection and union ratio of the area box and the area box with the highest current confidence score; when the intersection and union ratio of the area box and the area box with the highest current confidence score is greater than the non-maximum suppression threshold corresponding to the area box, the area box is discarded; otherwise, the area box is determined as the area box to be detected.

[0127] As another possible implementation, see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of the structure of another instance segmentation model disclosed in an embodiment of the present application. Figure 3 As shown, the instance segmentation model includes a feature map extraction module, a region box generation module, a region box screening module and a region box detection module. The region box detection module includes three cascaded detection sub-modules, and the output of the previous detection sub-module is used as the input of the next detection sub-module; the output of the last detection sub-module is used as the output of the instance segmentation model.

[0128] It should be noted that each detection submodule is provided with a benchmark intersection-and-union ratio threshold; the benchmark intersection-and-union ratio threshold is used by the detection submodule to assign positive and negative labels to the input area box. Specifically, for each input area box, the intersection-and-union ratio of the area box and the annotated true area box is calculated. If the intersection-and-union ratio of the area box and the annotated true area box is greater than the benchmark intersection-and-union ratio threshold, a negative label is assigned to the area box; otherwise, a positive label is assigned to the area box.

[0129] As an implementable method, the structure of the three detection submodules can be implemented by using three cascaded classification regression heads in Cascade MaskRCNN. At present, in Cascade MaskRCNN, different classification regression heads set different baseline intersection-and-union thresholds, and the baseline intersection-and-union thresholds set by different classification regression heads are fixed. The baseline intersection-and-union threshold of the latter classification regression head is greater than the baseline intersection-and-union threshold of the former classification regression head. As the baseline intersection-and-union threshold continues to increase, small targets will gradually be judged as negative samples, resulting in poor learning effect of small targets during model training. In order to avoid this problem, in this application, Figure 3 During the training process of the instance segmentation model shown, for each detection sub-module, for the region box input into the detection sub-module, the detection sub-module determines the IoU threshold corresponding to the region box based on the benchmark IoU threshold and the area of ​​the region box, and assigns a label to the region box based on the IoU threshold corresponding to the region box.

[0130] As an implementable method, the method of determining the intersection-and-union ratio threshold corresponding to the area box based on the benchmark intersection-and-union ratio threshold and the area of ​​the area box includes: if the area of ​​the area box is smaller than the area threshold, adjusting the benchmark intersection-and-union ratio threshold to obtain the adjusted intersection-and-union ratio threshold as the intersection-and-union ratio threshold corresponding to the area box, and the adjusted intersection-and-union ratio threshold is smaller than the benchmark intersection-and-union ratio threshold; if the area of ​​the area box is greater than or equal to the area threshold, using the benchmark intersection-and-union ratio threshold as the intersection-and-union ratio threshold corresponding to the area box.

[0131] For ease of understanding, assuming that area is the area of ​​the region box, threshold is the area threshold, which can be roughly obtained through the prior of the first target of the training text image, and IoU0 is the benchmark intersection-over-union threshold, then the intersection-over-union threshold IoU corresponding to the region box can be calculated based on the following formula:

[0132]

[0133] Based on the above, as an implementable method, assigning a label to the area box based on the intersection-and-union ratio threshold corresponding to the area box may include: calculating the intersection-and-union ratio of the area box and the labeled true area box, and if the intersection-and-union ratio of the area box and the labeled true area box is greater than the intersection-and-union ratio threshold corresponding to the area box, assigning a negative label to the area box; if the intersection-and-union ratio of the area box and the labeled true area box is not greater than the intersection-and-union ratio threshold corresponding to the area box, assigning a positive label to the area box.

[0134] It should be noted that, by adopting the above scheme, the intersection-over-union ratio threshold of the small-sized region frame can be reduced, and the conditions for judging the small-sized region frame as a positive sample can be relaxed, thereby improving the learning ability of the small-sized region frame during the model training process.

[0135] In another embodiment of the present application, the implementation method of performing semantic segmentation processing on the second target in the text image in step S103 to obtain the region detection result of the second target is introduced, and the method may include:

[0136] The text image is input into a trained semantic segmentation model, and the semantic segmentation model outputs the area detection result of the second target. The semantic segmentation model is trained using the training text image as a training sample, and the real area box contained in the second target annotated with the training text image and the type of each real area box as sample labels.

[0137] It should be noted that, for different scenarios, the training text image can be the text image in the current scene, and the real area frame and the type of each real area frame included in the second target annotated with the training text image are also adapted to the current scene. For ease of understanding, taking the primary and secondary school homework scenario as an example, the training text image can be a large number of primary and secondary school homework images, the second target can be a small question type (such as fill-in-the-blank questions, multiple-choice questions, etc.), and the types of real area frames included in the second target can include question number, question stem, answer area, etc.

[0138] As an implementable embodiment, the structure of the semantic segmentation model can adopt the structure of a DBNet (Real-time Scene Text Detection with Differentiable Binarization) network.

[0139] In another embodiment of the present application, the implementation method of determining the region detection result of the text image based on the region detection result of the first target and the region detection result of the second target in step S104 is introduced as follows:

[0140] As an implementable method, the region detection result of the first target and the region detection result of the second target may be used to determine the region detection result of the text image.

[0141] However, there are often some similar area frames in the area detection result of the first target, and these similar area frames reduce the accuracy of the area detection result of the first target. In order to solve this problem. As another possible implementation, the area detection result of the text image is determined based on the area detection result of the first target and the area detection result of the second target, which may include: optimizing the area detection result of the first target to obtain the area detection result of the first target after optimization; and determining the area detection result of the text image by using the area detection result of the first target after optimization and the area detection result of the second target.

[0142] As an implementable method, the optimizing the region detection result of the first target to obtain the optimized region detection result of the first target may include the following steps:

[0143] Step S301: for each region box type in the first target detection result, calculating an intersection-over-union ratio of any two region boxes corresponding to the region box type.

[0144] For ease of understanding, assuming that the two area boxes corresponding to a certain area box type in the first target detection result are A and B respectively, the formula IoU = (A∩B) / min(A,B) can be used to calculate the intersection and union ratio of area box A and area box B, where min(A,B) represents the minimum area of ​​A and B, and A∩B represents the overlapping area of ​​A and B.

[0145] Step S302: obtaining a preset intersection-over-union ratio threshold of region frames of the same type;

[0146] Step S303: If the IoU of the two region frames is greater than the IoU threshold of the region frames of the same type, the region frame with the lower confidence score of the two region frames is discarded.

[0147] As another possible implementation, the optimizing process for the region detection result of the first target to obtain the optimized region detection result of the first target may include the following steps:

[0148] Step S401: for two different types of region frames whose positions meet a preset condition in the first target detection result, calculating an intersection-over-union ratio of the two region frames.

[0149] In this step, the position meets the preset condition, and the two area boxes may have an intersecting portion.

[0150] For ease of understanding, assuming that the two different types of area boxes whose positions meet the preset conditions in the first target detection result are A and B, the formula IoU=(A∩B) / (A+B-(A∩B)) can be used to calculate the intersection and union ratio of area box A and area box B, where A+B-(A∩B) represents the union of A and B, and A∩B represents the overlapping area of ​​A and B.

[0151] Step S402: Obtain preset IoU thresholds of different types of region frames.

[0152] Step S403: If the IoU of the two region frames is greater than the IoU threshold of the region frames of different types, the region frame with the lower confidence score of the two region frames is discarded.

[0153] It should be noted that the above two optimization methods can be performed either one or both. By optimizing the first target detection result, the region detection result of the text image can be made more accurate.

[0154] The text image region detection device disclosed in the embodiment of the present application is described below. The text image region detection device described below and the text image region detection method described above can be referenced to each other.

[0155] Reference Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a text image region detection device disclosed in an embodiment of the present application. Figure 4 As shown, the text image area detection device may include:

[0156] A text image acquisition unit 41, used to acquire a text image to be subjected to region detection;

[0157] An instance segmentation processing unit 42 is used to perform instance segmentation processing on the first target in the text image to obtain a region detection result of the first target;

[0158] A semantic segmentation processing unit 43, configured to perform semantic segmentation processing on the second target in the text image to obtain a region detection result of the second target;

[0159] The region detection result determining unit 44 is configured to determine the region detection result of the text image based on the region detection result of the first object and the region detection result of the second object.

[0160] Optionally, the instance segmentation processing unit includes:

[0161] An instance segmentation model application unit is used to input the text image into a trained instance segmentation model, and the instance segmentation model outputs the area detection result of the first target. The instance segmentation model is trained using the training text image as a training sample, and the real area box contained in the first target annotated with the training text image and the type of each real area box as sample labels.

[0162] Optionally, the instance segmentation model includes a feature map extraction module, a region box generation module, a region box screening module and a region box detection module;

[0163] The device further comprises: an instance segmentation model training unit, wherein the instance segmentation model training unit comprises:

[0164] A feature map extraction unit, configured to extract a feature map from the training text image using the feature map extraction module to obtain a feature map of the training text image;

[0165] A region frame generating unit, configured to determine, by using the region frame generating module, a plurality of region frames included in a first target in the training text image based on a feature map of the training text image;

[0166] The area frame screening unit is used to use the area frame screening module to determine, for each area frame generated by the area frame generation module, a non-maximum suppression threshold corresponding to the area frame based on the shape information of the area frame; and based on the non-maximum suppression thresholds corresponding to each area frame, screen out an area frame to be detected from the multiple area frames generated by the area frame generation module, and input the area frame to be detected into the area frame detection module.

[0167] Optionally, the area frame screening unit includes:

[0168] A calculation unit, used to calculate the inclination angle and aspect ratio of the minimum circumscribed rectangular frame of the area frame;

[0169] A threshold acquisition unit, used to acquire a preset reference non-maximum suppression threshold, a tilt angle threshold, and an aspect ratio threshold;

[0170] A non-maximum suppression threshold determination unit is used to adjust the benchmark non-maximum suppression threshold when the inclination angle of the minimum circumscribed rectangular frame of the area frame is greater than the inclination angle threshold and the aspect ratio of the minimum circumscribed rectangular frame of the area frame is greater than the aspect ratio threshold, and determine the adjusted non-maximum suppression threshold as the non-maximum suppression threshold corresponding to the area frame; the adjusted non-maximum suppression threshold is greater than the benchmark non-maximum suppression threshold; otherwise, determine the benchmark non-maximum suppression threshold as the non-maximum suppression threshold corresponding to the area frame.

[0171] Optionally, the area frame screening unit includes:

[0172] an intersection-over-union ratio calculation unit, configured to calculate, for each region box generated by the region box generation module, an intersection-over-union ratio between the region box and the region box with the highest current confidence score;

[0173] The unit for determining the area frame to be detected is used to discard the area frame when the intersection-and-union ratio of the area frame and the area frame with the highest current confidence score is greater than the non-maximum suppression threshold corresponding to the area frame; otherwise, determine the area frame as the area frame to be detected.

[0174] Optionally, the region box detection module includes three cascaded detection submodules, the output of the previous detection submodule is used as the input of the next detection submodule; the output of the last detection submodule is used as the output of the instance segmentation model; each detection submodule is provided with a reference intersection-over-union ratio threshold;

[0175] The instance segmentation model training unit also includes:

[0176] A label assignment unit is used for, for each detection submodule, for a region frame input into the detection submodule, the detection submodule determines an IoU threshold corresponding to the region frame based on the benchmark IoU threshold and the area of ​​the region frame, and assigns a label to the region frame based on the IoU threshold corresponding to the region frame.

[0177] Optionally, the label allocating unit includes:

[0178] an IoU threshold adjustment unit, configured to adjust the base IoU threshold if the area of ​​the region frame is smaller than the area threshold, and obtain the adjusted IoU threshold as the IoU threshold corresponding to the region frame, wherein the adjusted IoU threshold is smaller than the base IoU threshold;

[0179] If the area of ​​the region frame is greater than or equal to the area threshold, the reference IoU threshold is used as the IoU threshold corresponding to the region frame.

[0180] Optionally, the semantic segmentation processing unit includes:

[0181] A semantic segmentation model application unit is used to input the text image into a trained semantic segmentation model, and the semantic segmentation model outputs the area detection result of the second target. The semantic segmentation model is trained using the training text image as a training sample and the real area box contained in the second target annotated with the training text image and the type of each real area box as sample labels.

[0182] Optionally, the area detection result determining unit includes:

[0183] an optimization unit, configured to optimize the region detection result of the first target to obtain the region detection result of the first target after optimization;

[0184] The area detection result determination subunit is used to determine the area detection result of the text image based on the area detection result of the first target after the optimization process and the area detection result of the second target.

[0185] Optionally, the optimization unit comprises:

[0186] The same type region frame optimization unit is used to calculate, for each region frame type in the first target detection result, an intersection-and-union ratio of any two region frames corresponding to the region frame type; obtain a preset intersection-and-union ratio threshold of the same type region frames; if the intersection-and-union ratio of the two region frames is greater than the intersection-and-union ratio threshold of the same type region frames, discard the region frame with a lower confidence score among the two region frames.

[0187] Optionally, the optimization unit comprises:

[0188] The different types of region frame optimization unit is used to calculate the intersection-and-union ratio of two region frames of different types whose positions meet the preset conditions in the first target detection result; obtain a preset intersection-and-union ratio threshold of different types of region frames; if the intersection-and-union ratio of the two region frames is greater than the intersection-and-union ratio threshold of different types of region frames, then discard the region frame with a lower confidence score among the two region frames.

[0189] Reference Figure 5 , Figure 5 The hardware structure block diagram of the text image area detection device provided in the embodiment of the present application is shown in FIG. Figure 5 ,The hardware structure of the text image region detection device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0190] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0191] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0192] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0193] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0194] Obtain a text image for region detection;

[0195] Performing instance segmentation processing on a first target in the text image to obtain a region detection result of the first target;

[0196] Performing semantic segmentation processing on the second target in the text image to obtain a region detection result of the second target;

[0197] Based on the region detection result of the first object and the region detection result of the second object, a region detection result of the text image is determined.

[0198] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0199] The embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0200] Obtain a text image for region detection;

[0201] Performing instance segmentation processing on a first target in the text image to obtain a region detection result of the first target;

[0202] Performing semantic segmentation processing on the second target in the text image to obtain a region detection result of the second target;

[0203] Based on the region detection result of the first object and the region detection result of the second object, a region detection result of the text image is determined.

[0204] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0205] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0206] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0207] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text image region detection method, characterized in that: The method comprises: Obtain a text image to be detected in the region; Performing instance segmentation processing on a first target in the text image to obtain a region detection result of the first target; the first target is a target in the text image whose area is greater than a preset threshold, and the region detection result of the first target is used to indicate different types of region frames contained in the first target; Performing semantic segmentation processing on a second target in the text image to obtain a region detection result of the second target; the second target is a target in the text image whose area is smaller than a preset threshold, and the region detection result of the second target is used to indicate different types of region boxes contained in the second target; Based on the region detection result of the first object and the region detection result of the second object, a region detection result of the text image is determined.

2. The method according to claim 1, characterized in that The performing instance segmentation processing on the first target in the text image to obtain a region detection result of the first target includes: The text image is input into a trained instance segmentation model, and the instance segmentation model outputs the region detection result of the first target. The instance segmentation model is trained using the training text image as a training sample, and using the real region box contained in the first target annotated with the training text image and the type of each real region box as sample labels.

3. The method according to claim 2, characterized in that The instance segmentation model includes a feature map extraction module, a region frame generation module, a region frame screening module and a region frame detection module; The training process of the instance segmentation model includes: The feature map extraction module extracts the feature map of the training text image to obtain the feature map of the training text image; The region frame generation module determines a plurality of region frames included in a first target in the training text image based on a feature map of the training text image; The area box screening module determines, for each area box generated by the area box generation module, a non-maximum suppression threshold corresponding to the area box based on the shape information of the area box; and based on the non-maximum suppression thresholds corresponding to each area box, screens out an area box to be detected from the multiple area boxes generated by the area box generation module, and inputs the area box to be detected into the area box detection module.

4. The method according to claim 3, characterized in that The determining, based on the shape information of the region box, a non-maximum suppression threshold corresponding to the region box includes: Calculating the inclination angle and aspect ratio of the minimum circumscribed rectangular frame of the area frame; Obtain the preset benchmark non-maximum suppression threshold, tilt angle threshold, and aspect ratio threshold; When the inclination angle of the minimum circumscribed rectangular frame of the area frame is greater than the inclination angle threshold, and the aspect ratio of the minimum circumscribed rectangular frame of the area frame is greater than the aspect ratio threshold, the benchmark non-maximum suppression threshold is adjusted, and the adjusted non-maximum suppression threshold is determined as the non-maximum suppression threshold corresponding to the area frame; the adjusted non-maximum suppression threshold is greater than the benchmark non-maximum suppression threshold; Otherwise, the benchmark non-maximum suppression threshold is determined as the non-maximum suppression threshold corresponding to the area box.

5. The method according to claim 3, characterized in that: The step of selecting a region frame to be detected from the plurality of region frames generated by the region frame generation module based on the non-maximum suppression threshold corresponding to each region frame includes: For each area box generated by the area box generation module, calculating an intersection-over-union ratio between the area box and the area box with the highest current confidence score; When the intersection-and-union ratio of the region box and the region box with the highest current confidence score is greater than the non-maximum suppression threshold corresponding to the region box, the region box is discarded; otherwise, the region box is determined as the region box to be detected.

6. The method according to claim 3, characterized in that The region box detection module includes three cascaded detection submodules, the output of the previous detection submodule is used as the input of the next detection submodule; the output of the last detection submodule is used as the output of the instance segmentation model; Each detection submodule is provided with a reference intersection-over-union ratio threshold; The training process of the instance segmentation model also includes: For each detection submodule, for a region box input into the detection submodule, the detection submodule determines an IoU threshold corresponding to the region box based on the benchmark IoU threshold and the area of ​​the region box, and assigns a label to the region box based on the IoU threshold corresponding to the region box.

7. The method according to claim 6, characterized in that The determining, based on the reference IoU threshold and the area of ​​the region frame, the IoU threshold corresponding to the region frame includes: If the area of ​​the region frame is smaller than the area threshold, the benchmark IoU threshold is adjusted to obtain an adjusted IoU threshold as the IoU threshold corresponding to the region frame, wherein the adjusted IoU threshold is smaller than the benchmark IoU threshold; If the area of ​​the region frame is greater than or equal to the area threshold, the reference IoU threshold is used as the IoU threshold corresponding to the region frame.

8. The method according to claim 2, characterized in that: The performing semantic segmentation processing on the second target in the text image to obtain a region detection result of the second target includes: The text image is input into a trained semantic segmentation model, and the semantic segmentation model outputs the area detection result of the second target. The semantic segmentation model is trained using the training text image as a training sample, and the real area box contained in the second target annotated with the training text image and the type of each real area box as sample labels.

9. The method according to claim 1, characterized in that: The determining the region detection result of the text image based on the region detection result of the first target and the region detection result of the second target includes: Optimizing the region detection result of the first target to obtain an optimized region detection result of the first target; The region detection result of the first target after the optimization process and the region detection result of the second target are used to determine the region detection result of the text image.

10. The method according to claim 9, characterized in that The optimizing the region detection result of the first target to obtain the optimized region detection result of the first target includes: For each region box type in the first target detection result, calculating an intersection-over-union ratio of any two region boxes corresponding to the region box type; Get the preset intersection-over-union ratio threshold of the same type of region boxes; If the IoU of the two region frames is greater than the IoU threshold of the region frames of the same type, the region frame with the lower confidence score among the two region frames is discarded.

11. The method according to claim 9 or 10, characterized in that: The optimizing the region detection result of the first target to obtain the optimized region detection result of the first target includes: For two different types of area frames whose positions meet a preset condition in the first target detection result, calculating an intersection-over-union ratio of the two area frames; Get preset intersection-over-union ratio thresholds for different types of region boxes; If the IoU of the two region frames is greater than the IoU threshold of region frames of different types, the region frame with the lower confidence score among the two region frames is discarded.

12. A text image region detection device, characterized in that: The device comprises: A text image acquisition unit, used to acquire a text image to be subjected to area detection; an instance segmentation processing unit, configured to perform instance segmentation processing on a first target in the text image to obtain a region detection result of the first target; the first target is a target in the text image whose area is greater than a preset threshold, and the region detection result of the first target is used to indicate different types of region frames contained in the first target; a semantic segmentation processing unit, configured to perform semantic segmentation processing on a second target in the text image to obtain a region detection result of the second target; the second target is a target in the text image whose area is smaller than a preset threshold, and the region detection result of the second target is used to indicate different types of region boxes contained in the second target; The area detection result determining unit is used to determine the area detection result of the text image based on the area detection result of the first object and the area detection result of the second object.

13. A text image area detection device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the text image area detection method according to any one of claims 1 to 11.

14. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the text image area detection method according to any one of claims 1 to 11 is implemented.

Images