Text region detection method and device and storage medium

Abstract

The invention provides a text region detection method and device and a storage medium. According to the method, the probability graphs of at least two prediction networks are fused, so that the defects of each probability graph can be overcome, the finally obtained text region is more accurate, and the accuracy of subsequent text recognition can be improved.

Description

Technical Field

[0001] This application relates to the fields of Optical Character Recognition (OCR) and machine learning, specifically to a text region detection method, apparatus, and storage medium. Background Technology

[0002] Scene text detection aims to locate text instances in an image. For a given input image, it detects and returns text instances in the image. Text detection has a wide range of applications in the real world, including office automation, visual search, and geolocation, and is therefore of great significance.

[0003] For text detection tasks in scenes of arbitrary shapes, the most popular approach in the industry is currently to use segmentation-based methods, such as Differentiable Binarization (DB). Segmentation-based scene text detection transforms the probability map (heatmap) generated by the segmentation algorithm into bounding boxes and text regions, including a post-processing step of binarization.

[0004] Existing segmentation-based methods struggle to accurately locate text regions during post-processing. For example, when detecting non-rectangular text regions, issues such as missed detections and region consolidation sometimes occur, affecting subsequent text recognition. Figure 1 Taking the document image containing the seal image as an example, the seal image contains text content. When detecting the text region in the seal image, due to the characteristics of the seal text such as curved text, variable text length, and large background interference, common object detection methods such as SSD, YOLO, and Faster-RCNN are not ideal in seal text detection. Popular text detection methods such as the DB algorithm have problems such as missing edge regions in the detection of curved seal text. Summary of the Invention

[0005] At least one embodiment of this application provides a text region detection method, apparatus, and storage medium, which can improve the performance of text region detection and provide support for subsequent text recognition.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a text region detection method, including:

[0008] The first image to be detected is input into at least two prediction models respectively to obtain the probability map of the first image generated by each prediction model and the threshold map of the first image generated by the first prediction model, wherein the first prediction model is one of the at least two prediction models;

[0009] The probability maps of the first image generated by each prediction model are fused to obtain a fused probability map;

[0010] Based on the fusion probability map and the threshold map, an approximate binary map of the first image is calculated;

[0011] The text region in the first image is obtained based on the approximate binary image.

[0012] Optionally, the probability maps of the first image generated by each prediction model are fused to obtain a fused probability map, including:

[0013] The values ​​of the same pixel in the probability maps of the first image generated by each prediction model are weighted and summed to obtain the fusion value of the pixel; the fusion probability map is obtained based on the fusion values ​​of each pixel in the first image.

[0014] Optionally, the values ​​of the pixels in each probability map have the same weight.

[0015] Optionally, the at least two prediction models include a first prediction model and a second prediction model, wherein the first prediction model is the first feature pyramid network and the second prediction model is the first codec network.

[0016] Optionally, the values ​​of the same pixel in the probability maps of the first image generated by each prediction model are weighted and summed to obtain the fused value of the pixel, including:

[0017] Obtain the first value of the pixel in the first probability map generated by the first feature pyramid network, and the second value in the second probability map generated by the first codec network; calculate the absolute value of the difference between the first value and the second value.

[0018] If the absolute value is less than a preset threshold, the average of the first value and the second value is calculated to obtain the fused value of the pixel.

[0019] If the absolute value is greater than or equal to the preset threshold, the nearest distance between the pixel and the main region in the second probability map is determined, and the first value and the second value are weighted and summed according to the first weight and the second weight to obtain the fused value of the pixel. The first weight is a preset fixed value, and the second weight is negatively correlated with the nearest distance.

[0020] Optionally, based on the fusion probability map and the threshold map, an approximate binary map of the first image is calculated, including:

[0021] The fusion probability map and the threshold map are input into a pre-trained differentiable binarization DB network to obtain an approximate binary map of the first image output by the DB network.

[0022] Optionally, before inputting the first image to be detected into at least two prediction models, the method further includes:

[0023] The system receives a document image uploaded by a user, extracts a seal image containing a seal pattern from the document image, and obtains the first image, wherein the seal pattern contains text content.

[0024] Optionally, the method further includes:

[0025] Text recognition is performed on the text region in the first image to obtain the seal text;

[0026] Text recognition is performed on the remaining images in the document image excluding the first image to obtain the document text;

[0027] The project name and its corresponding project text are obtained from the document text and stored one by one in the project name and project text fields of the document recognition result;

[0028] The seal name is obtained from the document text, the seal text corresponding to the seal name is determined, and the seal text is stored in the seal name and seal text fields of the document recognition result.

[0029] Based on preset keywords, determine the first item name corresponding to the first seal name, and compare whether the first seal text is consistent with the first item text. Here, the first seal text is the seal text corresponding to the first seal name, and the first item text is the item text corresponding to the first item name. If they are inconsistent, generate a prompt message to indicate that the first seal text and the first item text are inconsistent.

[0030] The document recognition result is sent to the user, and the prompt message is also sent if the text content of the first stamp is inconsistent with the text of the first item.

[0031] Secondly, embodiments of this application provide a text region detection device, including:

[0032] The prediction module is used to input the first image to be detected into at least two prediction models respectively, and obtain the probability map of the first image generated by each prediction model and the threshold map of the first image generated by the first prediction model, wherein the first prediction model is one of the at least two prediction models.

[0033] The fusion module is used to fuse the probability maps of the first image generated by each prediction model to obtain a fused probability map;

[0034] The text region acquisition module is used to calculate an approximate binary map of the first image based on the fusion probability map and the threshold map; and to obtain the text region in the first image based on the approximate binary map.

[0035] According to a third aspect of this application, at least one embodiment provides a text region detection apparatus, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in any of the first aspects.

[0036] According to a fourth aspect of this application, at least one embodiment provides a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the method as described in any of the first aspects.

[0037] According to a fifth aspect of this application, at least one embodiment provides a computer program product including computer instructions that, when executed by a processor, implement the steps of the method as described in any of the first aspects.

[0038] Compared with the prior art, the text region detection method and apparatus provided in this application can improve the shortcomings of each probability map by fusing the probability maps of at least two prediction networks, making the final obtained text region more accurate, and thus improving the accuracy of subsequent text recognition. Attached Figure Description

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0040] Figure 1 This is an example image of a document containing a seal image;

[0041] Figure 2 This is a flowchart of a text region detection method according to an embodiment of this application;

[0042] Figure 3 This is a schematic diagram of the structure of the text region detection device according to an embodiment of this application;

[0043] Figure 4 This is another structural schematic diagram of the text region detection device according to an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of the structure of a document recognition system according to an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of the structure of a document recognition device according to an embodiment of this application;

[0046] Figure 7 The hardware structure of an exemplary computer in this application embodiment;

[0047] Figure 8 This is an exemplary hardware structure of a mobile terminal according to an embodiment of this application;

[0048] Figure 9 An example flowchart for document image recognition provided in this application embodiment;

[0049] Figure 10 This is an example image of a seal from an embodiment of this application;

[0050] Figure 11 This is an example diagram of the seal text area in an embodiment of this application;

[0051] Figure 12 This is an example image of a seal text image according to an embodiment of this application;

[0052] Figure 13 This is an example diagram of the seal text in an embodiment of this application;

[0053] Figure 14 This is an example diagram of document recognition results in an embodiment of this application;

[0054] Figure 15 This is another example diagram of the document recognition results in an embodiment of this application. Detailed Implementation

[0055] To make the technical problems, technical solutions, and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as particular configurations and components are provided merely to aid in a comprehensive understanding of the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Furthermore, for clarity and brevity, descriptions of known functions and structures have been omitted.

[0056] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The word "and / or" in the specification and claims indicates at least one of the connected objects.

[0057] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0058] The following description provides examples and is not intended to limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the function and arrangement of the elements discussed without departing from the spirit and scope of this disclosure. Various procedures or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to certain examples may be combined in other examples.

[0059] This application provides a text region detection method. The method utilizes at least two prediction models to generate probability maps of the image to be detected, then performs a fusion process to obtain a fused probability map, and finally obtains the text region based on the fused probability map. The fused probability map combines the probability maps from different prediction models, which can improve upon the shortcomings of each individual probability map, resulting in a more accurate final text region and thus improving the accuracy of subsequent text recognition.

[0060] Please refer to Figure 2 The text region detection method provided in this application includes the following steps:

[0061] Step 21: Input the first image to be detected into at least two prediction models respectively to obtain the probability map of the first image generated by each prediction model and the threshold map of the first image generated by the first prediction model, wherein the first prediction model is one of the at least two prediction models.

[0062] Here, one of the at least two prediction models is a first prediction model. These prediction models are all capable of generating a corresponding probability map based on the input first image. The probability map has the same size as the first image, and the value of each pixel in the probability map represents the probability that the position of the corresponding pixel in the first image belongs to a text region. In addition to generating the probability map, the first prediction model also generates a threshold map of the first image. The threshold map has the same size as the first image, and the value of each pixel in the threshold map represents a binarized threshold for the position of the corresponding pixel in the first image.

[0063] The at least two prediction models may include only two prediction models, or they may include three or more prediction models. For example, when two prediction models are included, they may specifically include a first prediction model and a second prediction model, wherein the first prediction model is the first feature pyramid network, and the second prediction model is the first encoding / decoding network.

[0064] The first feature pyramid network can employ a network with a similar structure, such as a feature pyramid, and may specifically include an image feature extraction module, a probability map, and a threshold map prediction module. The specific structure of these modules can be referenced from relevant modules in the DB++ network. For example, the image feature extraction module can be implemented using a Feature Pyramid Network (FPN) and an Adaptive Scale Fusion (ASF) module. The FPN generates pyramid features of the image, then upsamples these features to the same scale, concatenates them, and passes them to the ASF module to generate the final context features F. The probability map and threshold map prediction module generates a probability map (P) and a threshold map (T) based on the context features F. The first encoder-decoder network can employ an encoder-decoder network structure. For example, as one implementation, the encoder can be an EfficientNet network, and the decoder can be a UNet network. Of course, other encoder-decoder networks can also be used in this embodiment.

[0065] When the at least two prediction models may include three or more prediction models, specifically, they may include the first prediction model, the second prediction model, the third prediction model, etc. The first prediction model may employ a network with a similar structure such as a feature pyramid, while the second and third prediction models may employ encoding / decoding networks with different structures. This application does not specifically limit this aspect.

[0066] Step 22: Fuse the probability maps of the first image generated by each prediction model to obtain a fused probability map.

[0067] Here, in this embodiment of the application, the values ​​of the same pixel in the probability map of the first image generated by each prediction model are weighted and summed to obtain the fusion value of the pixel; the fusion probability map is obtained based on the fusion values ​​of each pixel in the first image.

[0068] As a fusion method, the probability maps generated by each prediction model have the same weights. Then, the values ​​of the same pixel in these probability maps are weighted and summed according to the same weights (the sum of these weights equals 1) to obtain the fused value of the pixel. Alternatively, an average value is calculated from the values ​​of the same pixel in these probability maps and used as the fused value of the pixel.

[0069] As another fusion method, embodiments of this application can perform different fusion processes based on the differences in values ​​of the same pixel in different threshold maps.

[0070] Taking the example that the at least two prediction models include only the first prediction model (first feature pyramid network) and the second prediction model (first codec network), for each pixel, the first value of the pixel in the first probability map generated by the first feature pyramid network is obtained, and the second value in the second probability map generated by the first codec network is obtained; the absolute value of the difference between the first value and the second value is calculated.

[0071] If the absolute value is less than a preset threshold, the first value and the second value are weighted and summed with the same weight (the sum of these weights equals 1) to obtain the fused value of the pixel. Alternatively, the average of the first value and the second value is calculated as the fused value of the pixel.

[0072] If the absolute value is greater than or equal to the preset threshold, the nearest distance between the pixel and the main body region in the second probability map is determined. The first and second values ​​are then weighted and summed according to a first weight and a second weight to obtain the fused value of the pixel. The first weight is a preset fixed value, and the second weight is negatively correlated with the nearest distance. In other words, the first weight corresponding to the first value is set to a fixed value, and the second weight of the second value is set according to the nearest distance between the pixel and the main body region in the second probability map. Here, the main body region in the second probability map is the same as the main body region in the first probability map. Since the two probability maps have the same size, the main body region in the second probability map can be determined based on the main body region in the first probability map. Specifically, the main body region in the first probability map refers to the region composed of pixels whose probability values ​​in the first probability map are greater than or equal to the preset probability threshold. It should also be noted that if the calculated fused value is greater than 1, the fused value is set to 1.

[0073] The following is a specific calculation example:

[0074] If the absolute value is less than a preset threshold, the first value and the second value are weighted and summed with a weight of 0.5 each to obtain the fused value of the pixel.

[0075] When the absolute value is greater than or equal to the preset threshold, the first value and the second value are weighted and summed according to the weight of the first value being 1 and the weight of the second value being 1-d / D, to obtain the fused value of the pixel; where d is the nearest distance from the pixel to the main area in the second probability map, and D is the larger of the width and height of the second probability map.

[0076] The above calculation example can be represented by the following formula:

[0077] I merge =w A *I mapA +w B *I mapB

[0078]

[0079] Among them, I merge w represents the blending value of a specific pixel (e.g., pixel i). A w is the first weight of pixel i. B The second weight of pixel i. t represents the preset threshold; I mapAiI represents the first value of pixel i in the first probability map; mapBi d represents the second value of pixel i in the second probability map; d represents the shortest distance from pixel i to the main area in the second probability map; D represents the larger of the width and height of the second probability map.

[0080] Step 23: Based on the fusion probability map and the threshold map, calculate an approximate binary map of the first image.

[0081] Here, the fusion probability map and the threshold map can be input into a pre-trained DB network to obtain an approximate binary map of the first image output by the DB network. For specific implementation details, refer to the existing DB++ network; this application will not elaborate further.

[0082] Step 24: Obtain the text region in the first image based on the approximate binary image.

[0083] Here, the bounding box of the text region can be determined using the approximate binary image, thereby obtaining the text region in the first image.

[0084] Through the above steps, the embodiments of this application integrate the probability graphs of at least two prediction networks, which can improve the shortcomings of each probability graph itself, making the final text region more accurate, and thus improving the accuracy of subsequent text recognition.

[0085] The method described in this application embodiment can be applied to document image detection containing seal images, obtaining text regions in the seal image, and then identifying the seal text content from the text regions using a text recognition module. Of course, the above is merely one application scenario of this application embodiment and is not intended to limit this application.

[0086] For example, prior to step 21 above, this embodiment of the application may receive a document image uploaded by a user, extract a seal image containing a seal pattern from the document image, and obtain the first image, wherein the seal pattern contains text content. The document image may be a JPEG, TIFF image, or PDF file obtained by scanning a paper contract.

[0087] Then, through Figure 2 The method shown identifies the text region in the first image (stamp image). Further, text recognition is performed on the text region in the first image to obtain the stamp text. Considering that the text region of a stamp is usually an arc-shaped or annular area, if the text region is not rectangular, polar coordinate transformation can be performed on the text region in the first image to obtain a rectangular text region, and then text recognition can be performed on the rectangular text region to obtain the stamp text.

[0088] Furthermore, text recognition can be performed on the remaining images in the document image excluding the first image to obtain the document text. Then, the project name and its corresponding project text are extracted from the document text and stored one-to-one in the project name and project text fields of the document recognition result. Additionally, the seal name is extracted from the document text, the seal text corresponding to the seal name is determined, and stored one-to-one in the seal name and seal text fields of the document recognition result.

[0089] The document recognition result adopts a preset document template structure, including multiple items such as a project name item (for storing project names), a project text item (for storing the project text corresponding to the project name), a seal name item (for storing seal names), and a seal text item (for storing the seal text corresponding to the seal name item). The project name item and seal name item can be identified using preset keywords, such as "contract number," "Party A," and "Party B," which allows extraction of various project names from the document text. The project text corresponding to a given project name can be determined by identifying the text content closest to the project name. Similarly, keywords such as "Party A (seal)" and "Party B (seal)" can be used as keywords for the seal name item, allowing extraction of various seal names from the document text. The seal text corresponding to a given seal name can be determined by identifying the text content closest to the seal name.

[0090] Additionally, based on preset keywords, the project name corresponding to a given seal name can be determined, for example, determining the first project name corresponding to the first seal name. Specifically, seal names and project names containing the same keywords (such as "Party A" and "Party B") can be considered to correspond to each other. Then, the text of the first seal is compared with the text of the first project, where the first seal text is the seal text corresponding to the first seal name, and the first project text is the project text corresponding to the first project name. If they do not match, a prompt message is generated to indicate that the first seal text and the first project text are inconsistent.

[0091] Furthermore, the document recognition result can be sent to the user, and the prompt message can also be sent if the text content of the first stamp is inconsistent with the text of the first item.

[0092] by Figure 1For example, the document recognition results obtained through the above processing are shown in Table 1. The first seal text item stores the company / unit name in the seal text, and the second seal text item stores the enterprise legal person code and other content in the seal text. The project name "Party A" and the seal name "Party A (Seal)" both share the same keyword "Party A," therefore they correspond to each other and are filled in the same row in Table 1.

[0093] Table 1

[0094] Project Name Item Project text item Seal Name Item First Seal Text Item Second Seal Text Item Contract Number 12345 - - - Party A A Technology Co., Ltd. Party A (Seal) A Technology Co., Ltd. 1234567890123 Party B b Co., Ltd. Party B (Seal) B Investment Co., Ltd. - ... ... ... ... ...

[0095] In Table 1, the project name "Party B" corresponds to the project text "B Co., Ltd.", while the first seal text item is "B Investment Co., Ltd." The two are inconsistent. At this time, the user can be prompted to check whether there is a stamping error.

[0096] Based on the above methods, this application also provides an apparatus for implementing the above methods. Please refer to [link / reference]. Figure 3 This application provides a text region detection device, including:

[0097] Prediction module 31 is used to input the first image to be detected into at least two prediction models respectively to obtain the probability map of the first image generated by each prediction model and the threshold map of the first image generated by the first prediction model, wherein the first prediction model is one of the at least two prediction models;

[0098] The fusion module 32 is used to fuse the probability maps of the first image generated by each prediction model to obtain a fused probability map;

[0099] The text region acquisition module 33 is used to calculate an approximate binary map of the first image based on the fusion probability map and the threshold map; and to obtain the text region in the first image based on the approximate binary map.

[0100] Optionally, the fusion module 32 is further configured to:

[0101] The values ​​of the same pixel in the probability maps of the first image generated by each prediction model are weighted and summed to obtain the fusion value of the pixel; the fusion probability map is obtained based on the fusion values ​​of each pixel in the first image.

[0102] Optionally, the values ​​of the pixels in each probability map have the same weight.

[0103] Optionally, the at least two prediction models include a first prediction model and a second prediction model, wherein the first prediction model is the first feature pyramid network and the second prediction model is the first codec network.

[0104] Optionally, the fusion module 32 is further configured to:

[0105] Obtain the first value of the pixel in the first probability map generated by the first feature pyramid network, and the second value in the second probability map generated by the first codec network; calculate the absolute value of the difference between the first value and the second value.

[0106] If the absolute value is less than a preset threshold, the average of the first value and the second value is calculated to obtain the fused value of the pixel.

[0107] If the absolute value is greater than or equal to the preset threshold, the nearest distance between the pixel and the main region in the second probability map is determined, and the first value and the second value are weighted and summed according to the first weight and the second weight to obtain the fused value of the pixel. The first weight is a preset fixed value, and the second weight is negatively correlated with the nearest distance.

[0108] Optionally, the text region acquisition module 33 is further configured to input the fusion probability map and the threshold map into a pre-trained differentiable binarization DB network to obtain an approximate binary map of the first image output by the DB network.

[0109] Optionally, the above-mentioned device further includes:

[0110] The image acquisition module is used to receive a document image uploaded by a user, extract a seal image containing a seal pattern from the document image to obtain the first image, wherein the seal pattern contains text content.

[0111] Optionally, the above-mentioned device further includes:

[0112] The first recognition module is used to perform text recognition on the text region in the first image to obtain the seal text;

[0113] The second recognition module is used to perform text recognition on the remaining images in the document image other than the first image to obtain the document text;

[0114] The saving module is used to obtain the project name and its corresponding project text from the document text, and save them one by one in the project name item and project text item in the document recognition result; to obtain the seal name from the document text, determine the seal text corresponding to the seal name, and save it one by one in the seal name item and seal text item in the document recognition result.

[0115] The matching module is used to determine the first item name corresponding to the first seal name based on preset keywords, and compare whether the first seal text is consistent with the first item text. The first seal text is the seal text corresponding to the first seal name, and the first item text is the item text corresponding to the first item name. If they are inconsistent, a prompt message is generated to indicate that the first seal text and the first item text are inconsistent.

[0116] The sending module is used to send the document recognition result to the user, wherein, in the case that the text content of the first seal is inconsistent with the text of the first item, the prompt information is also sent.

[0117] It should be noted that the various devices provided in the above embodiments are devices corresponding to the above text region detection method. The implementation methods in each of the above embodiments are applicable to the embodiments of this device and can achieve the same technical effect. The above-mentioned device provided in this application embodiment can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0118] Please refer to Figure 4 This application also provides a hardware structure block diagram of a text region detection device, such as... Figure 4 As shown, the text region detection device 400 includes:

[0119] Processor 402; and

[0120] Memory 404, in which computer program instructions are stored.

[0121] When the computer program instructions are executed by the processor, the processor 402 performs the following steps:

[0122] The first image to be detected is input into at least two prediction models respectively to obtain the probability map of the first image generated by each prediction model and the threshold map of the first image generated by the first prediction model, wherein the first prediction model is one of the at least two prediction models;

[0123] The probability maps of the first image generated by each prediction model are fused to obtain a fused probability map;

[0124] Based on the fusion probability map and the threshold map, an approximate binary map of the first image is calculated;

[0125] The text region in the first image is obtained based on the approximate binary image.

[0126] It should be noted that the systems provided in the above embodiments are devices corresponding to the above text region detection method. The implementation methods in the above embodiments are all applicable to the embodiments of this device and can achieve the same technical effect. The device provided in this application can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0127] Furthermore, such as Figure 4 As shown, the text area detection device 400 also includes a network interface 401, an input device 403, a hard disk 405, and a display device 406.

[0128] The various interfaces and devices described above can be interconnected via a bus architecture. The bus architecture can include any number of interconnecting buses and bridges. Specifically, various circuits representing one or more central processing units (CPUs) and / or graphics processing units (GPUs), as represented by processor 402, and one or more memories, as represented by memory 404, are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. It is understood that the bus architecture is used to implement communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are well known in the art and will not be described in detail herein.

[0129] The network interface 401 can be connected to a network (such as the Internet, local area network, etc.), receive data such as documents and images from the network, and save the received data to the hard disk 405.

[0130] The input device 403 can receive various instructions input by the operator and send them to the processor 402 for execution. The input device 403 may include a keyboard or a clicking device (e.g., a mouse, trackball, touchpad, or touchscreen).

[0131] The display device 406 can display the results obtained by the processor 402 executing instructions, such as displaying the model training progress.

[0132] The memory 404 is used to store programs and data necessary for the operation of the operating system, as well as intermediate results and other data during the calculation process of the processor 402.

[0133] It is understood that the memory 404 in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. The memory 404 of the apparatus and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0134] In some implementations, memory 404 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating system 4041 and application program 4042.

[0135] The operating system 4041 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 4042 includes various applications, such as a browser, used to implement various application functions. Programs implementing the methods of this application embodiment can be included in application program 4042.

[0136] The methods disclosed in the above embodiments of this application can be applied to processor 402, or implemented by processor 402. Processor 402 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 402 or by instructions in the form of software. The processor 402 may be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 404, and processor 402 reads the information in memory 404 and completes the steps of the above method in combination with its hardware.

[0137] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0138] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0139] Specifically, when the computer program is executed by the processor 402, it can also perform the following steps:

[0140] The values ​​of the same pixel in the probability maps of the first image generated by each prediction model are weighted and summed to obtain the fusion value of the pixel; the fusion probability map is obtained based on the fusion values ​​of each pixel in the first image.

[0141] Optionally, the values ​​of the pixels in each probability map have the same weight.

[0142] Optionally, the at least two prediction models include a first prediction model and a second prediction model, wherein the first prediction model is the first feature pyramid network and the second prediction model is the first codec network.

[0143] Specifically, when the computer program is executed by the processor 402, it can also perform the following steps:

[0144] Obtain the first value of the pixel in the first probability map generated by the first feature pyramid network, and the second value in the second probability map generated by the first codec network; calculate the absolute value of the difference between the first value and the second value.

[0145] If the absolute value is less than a preset threshold, the average of the first value and the second value is calculated to obtain the fused value of the pixel.

[0146] If the absolute value is greater than or equal to the preset threshold, the nearest distance between the pixel and the main region in the second probability map is determined, and the first value and the second value are weighted and summed according to the first weight and the second weight to obtain the fused value of the pixel. The first weight is a preset fixed value, and the second weight is negatively correlated with the nearest distance.

[0147] Specifically, when the computer program is executed by the processor 402, it can also perform the following steps:

[0148] The fusion probability map and the threshold map are input into a pre-trained differentiable binarization DB network to obtain an approximate binary map of the first image output by the DB network.

[0149] Specifically, when the computer program is executed by the processor 402, it can also perform the following steps:

[0150] Before inputting the first image to be detected into at least two prediction models, a document image uploaded by the user is received, and a seal image containing a seal pattern is extracted from the document image to obtain the first image, wherein the seal pattern contains text content.

[0151] Specifically, when the computer program is executed by the processor 402, it can also perform the following steps:

[0152] Text recognition is performed on the text region in the first image to obtain the seal text;

[0153] Text recognition is performed on the remaining images in the document image excluding the first image to obtain the document text;

[0154] The project name and its corresponding project text are obtained from the document text and stored one by one in the project name and project text fields of the document recognition result;

[0155] The seal name is obtained from the document text, the seal text corresponding to the seal name is determined, and the seal text is stored in the seal name and seal text fields of the document recognition result.

[0156] Based on preset keywords, determine the first item name corresponding to the first seal name, and compare whether the first seal text is consistent with the first item text. Here, the first seal text is the seal text corresponding to the first seal name, and the first item text is the item text corresponding to the first item name. If they are inconsistent, generate a prompt message to indicate that the first seal text and the first item text are inconsistent.

[0157] The document recognition result is sent to the user, and the prompt message is also sent if the text content of the first stamp is inconsistent with the text of the first item.

[0158] It should be noted that the device provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0159] Please refer to Figure 5 A document recognition system 80 is provided, comprising: a client 83, a document recognition device 82, and a seal text region detection model generation device 81. The document recognition device 82 further includes a text recognition module 821 and a text region detection module 822. The text region detection module 822 detects text regions in a seal image using a seal text region detection model 8221; the text recognition module 821 recognizes the text content within the text regions using a text recognition model 8211. The seal text region detection model generation device 81 generates the seal text region detection model 8221.

[0160] Figure 5 The document recognition system 80 shown can recognize similar documents. Figure 1 Text recognition is performed on document images to identify the text content, including but not limited to the text content in seal patterns. Figure 6 A specific structure of a document recognition device 82 is provided, which, in addition to a text recognition module 821 and a text region detection module 822, also includes an information sending and receiving module 823, an information storage module 824, and a user interface control module 825.

[0161] The seal text region detection model 8221 runs on a server system (document recognition device 82) consisting of one or more computers. The seal text region detection model generation device 81 and the document recognition device 82 can be a single device (such as a computer) or the various models / modules can be distributed across multiple devices.

[0162] The client 83 can be a personal computer or a mobile terminal, or an application running on any of the aforementioned terminals. A mobile terminal is a terminal device operated by a user. A mobile terminal can be a smartphone, a personal digital assistant (PDA) device, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, an in-vehicle device, a wearable device, or a terminal device in a next-generation communication system, such as a terminal device in an NR network or a terminal device in a future evolved Public Land Mobile Network (PLMN) network.

[0163] In the aforementioned document recognition system 80, the client 83 can connect to the document recognition device 82 via a wired and / or wireless network.

[0164] The functions of the client 83, document recognition device 82, and seal text area detection model generation device 81 can all be distributed across multiple computers.

[0165] The following describes the hardware or software structure of the relevant equipment, devices, or functions.

[0166] <Hardware Structure>

[0167] <<Computer>>

[0168] The client 83, document recognition device 82, and seal text region detection model generation device 81 are connected via, for example, having Figure 7 The hardware structure shown is then materialized into a computer. Figure 7 An exemplary hardware structure of a computer according to an embodiment of this application is shown.

[0169] refer to Figure 7 The computer 500 includes an input device 501, a display device 502, an external I / F 503, RAM 504, ROM 505, a CPU 506, a communication I / F 507, an HDD 508, etc., interconnected via a bus B. It is acceptable to form a structure in which the input device 501 and the display device 502 are connected when necessary.

[0170] Input device 501 includes a keyboard, mouse, touchpad, etc., through which users input various operation signals. Display device 502 includes a monitor, etc., to display the processing results obtained by computer 500.

[0171] Communication I / F 507 is an interface configured to enable computer 500 to connect to various networks. Therefore, computer 500 performs data communication via communication I / F 507.

[0172] HDD 508 is an exemplary non-volatile storage device for storing programs and data. The stored data includes the operating system (OS) that serves as the foundational software controlling the entire computer 500, application software (also referred to herein as "applications") that provides various functionalities within the OS, and so on. Computer 500 may use a drive device that uses flash memory (e.g., a solid-state drive (SSD)) as the storage medium instead of HDD 508.

[0173] External I / F 503 is an interface for external devices. These external devices include recording media 503a, etc. In this case, computer 500 reads information from and / or writes information to recording media 503a via external I / F 503. Recording media 503a may be a floppy disk, CD, DVD, SD memory card, USB storage device, etc.

[0174] ROM 505 is a non-volatile semiconductor memory (storage device) that retains programs and / or data even when the power is off. ROM 505 stores programs and data used to execute the Basic Input / Output System (BIOS), OS settings, network settings, etc., at the time of power-on of computer 500. RAM 504 is an example of a volatile semiconductor memory (storage device) for temporary storage of programs and / or data.

[0175] CPU 506 is an algorithmic device that reads programs and / or data from storage devices such as ROM 505 and HDD 508. The read program or read data performs a process, thereby materializing the control or functional capabilities of the entire computer 500.

[0176] The client 83, document recognition device 82, and seal text region detection model generation device 81 are, for example, as shown in... Figure 7 The hardware structure of the computer 500 shown is materialized.

[0177] <<Mobile Terminal>>

[0178] For example, the client 83 uses, for example, Figure 8 The hardware structure shown is then materialized. Figure 8 An exemplary hardware structure of the mobile terminal in this embodiment is shown. Figure 8 The mobile terminal 12 shown includes a CPU 601, ROM 602, RAM 603, EEPROM 604, CMOS sensor 605, acceleration and orientation sensor 606, and media driver 608.

[0179] The CPU 601 controls the entire operation of the mobile terminal 12. The ROM 602 stores basic input and output programs. The RAM 603 is used as the working area of ​​the CPU 601. The EEPROM 604 reads or writes data corresponding to the control of the CPU 601. The CMOS sensor 605 captures image data corresponding to the control of the CPU 601 to obtain image data. The acceleration and orientation sensor 606 is an electromagnetic compass, rotary compass, acceleration sensor, etc., that detects the Earth's magnetic field.

[0180] Media drive 608 controls the reading or writing (storage) of data from or sent to recordable medium 607, such as flash memory. Data already stored in recordable medium 607 is read out, or new data is written to recordable medium 607. Recordable medium 607 is freely attachable to or detachable from media drive 608.

[0181] EEPROM 604 stores the operating system (OS) executed by CPU 601, as well as related information necessary for network settings. Applications for performing various processes of the first embodiment are stored in EEPROM 604, recordable medium 607, etc.

[0182] The CMOS sensor 605 is a charge-coupled device that converts light into electrical charges and digitizes an image of an object. The CMOS sensor 605 can be materialized, for example, by a charge-coupled device (CCD) sensor, as long as it can capture an image of the object.

[0183] In addition, the mobile terminal 12 includes an audio input unit 609, an audio output unit 610, an antenna 611, a communication unit 612, a wireless LAN communication unit 613, a wireless communication antenna 614, a wireless communication unit 615, a display 616, a touchpad 617, and a bus 619.

[0184] Audio input unit 609 converts sound into audio signals. Audio output unit 610 converts audio signals into sound. Communication unit 612 uses antenna 611 to communicate with the nearest base station device via wireless communication signals. Wireless LAN communication unit 613 performs wireless LAN communication with the access point in accordance with the IEEE 80411 standard. Wireless communication unit 615 performs wireless communication using wireless communication antenna 614.

[0185] Display 616 is configured to display images, various icons, etc. Display 616 is made of liquid crystal, organic EL, etc. Touchpad 617 is mounted on display 616 and is formed of a pressure-sensitive plate or an electrostatic plate. Touch positions on display 616 are detected by touch with a finger or stylus. Bus 619 is an address bus, data bus, etc., electrically connecting the aforementioned units or components.

[0186] The client 83 includes a dedicated battery 618. The client 83 is powered by the battery 618. The audio input unit 609 includes a microphone for inputting sound. The audio output unit 610 includes a loudspeaker for outputting sound.

[0187] For example, the client 83 uses, for example, Figure 8 The hardware structure shown is then materialized.

[0188] Figure 9 This example identifies document images containing stamp images (such as...). Figure 1 An example flow (shown in the document image) includes:

[0189] S901, the information transceiver module 823 acquires a document image uploaded by the user through the operation interface displayed on the client 83. The document image may be similar to... Figure 1 The image contains a stamp image, and the stamp image contains text content.

[0190] S902, the text region detection module 822 acquires the seal image in the document image. For example, the seal region can be identified based on pixel color, thereby extracting the seal image. Figure 1 Taking the document image shown as an example, you can obtain, for instance... Figure 10 The image of the seal shown.

[0191] S903, the text region detection module 822 uses the seal text region detection model 8221 to detect the seal text region in the seal image and obtain the coordinates of the seal text region. Here, it can be done according to... Figure 2 The method shown retrieves the text area of ​​the seal. Figure 1 Taking the document image shown as an example, you can obtain, for instance... Figure 11 The text area of ​​the seal shown.

[0192] S904, the text region detection module 822 performs polar coordinate transformation on the seal text region based on its coordinates to obtain a seal text image. For example, for... Figure 11 Transforming the text area of ​​the seal shown can yield the following result: Figure 12 The image shown is a rectangular stamp text image.

[0193] S905. The text recognition module 821 uses the text recognition model 8211 to recognize the seal text in the seal text image. For example, for Figure 12 the rectangular seal text image shown, text recognition can obtain the seal text as shown in Figure 13 .

[0194] S906. The text area detection module 822 obtains the text images of other areas, and the text recognition module 821 uses the text recognition model 8211 to recognize the text in the text images of other areas.

[0195] S907. The text recognition module 821 obtains the project name and project text from the recognized result text, and instructs the information storage module 824 to save the obtained project name and project text one by one into the document recognition result.

[0196] Here, the project name can be obtained from the recognized result text according to the preset keywords, and then according to the relationship between the coordinates of the project name and the coordinates of other recognized result texts, the text closest to the project name is used as the corresponding project text. For example, if "Party A" in the recognized result text is the preset keyword, it is used as the project name, and "A Technology Co., Ltd." in the recognized result text is used as the project text corresponding to "Party A".

[0197] S908. The text recognition module 821 determines the project name corresponding to the seal text according to the coordinates of the seal image and the coordinates of each project name. For example, the project name with the coordinates closest to the upper left corner coordinates of the seal image can be used as the project name corresponding to the seal text.

[0198] S909. The text recognition module 821 instructs the information storage module 824 to save the recognized seal text and its corresponding project name into the document recognition result.

[0199] S910. Determine whether the seal text corresponding to the same project name is consistent with the project text. If it is consistent, go to step S912; if it is inconsistent, go to step S911. Figure 14 provides Figure 1 an example of the document recognition result corresponding to the document image, which contains a prompt for inconsistency.

[0200] Optionally, this example can also determine the unit name corresponding to the enterprise legal person code according to the enterprise legal person code in the seal text through network retrieval, compare it with the corresponding project text, and judge whether it is consistent. If it is inconsistent, a prompt as shown in Figure 15 can also be generated to prompt that the enterprise legal person code is inconsistent with the name in the project text.

[0201] S911, the text recognition module 821 controls the user interface control module 825 to display a message indicating that the stamp text and the project text are inconsistent.

[0202] S912, the user interface control module 825 displays the user interface containing the document recognition results on the client 83 through the information transceiver module 823.

[0203] In some embodiments of this application, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, performs the following steps:

[0204] The first image to be detected is input into at least two prediction models respectively to obtain the probability map of the first image generated by each prediction model and the threshold map of the first image generated by the first prediction model, wherein the first prediction model is one of the at least two prediction models;

[0205] The probability maps of the first image generated by each prediction model are fused to obtain a fused probability map;

[0206] Based on the fusion probability map and the threshold map, an approximate binary map of the first image is calculated;

[0207] The text region in the first image is obtained based on the approximate binary image.

[0208] When executed by the processor, this program can implement all the above-mentioned text region detection methods and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0209] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described text region detection method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0210] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0211] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0212] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0213] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0214] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0215] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0216] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A text region detection method, characterized in that, include: The first image to be detected is input into at least two prediction models respectively to obtain the probability map of the first image generated by each prediction model and the threshold map of the first image generated by the first prediction model, wherein the first prediction model is one of the at least two prediction models; The probability maps of the first image generated by each prediction model are fused to obtain a fused probability map; Based on the fusion probability map and the threshold map, an approximate binary map of the first image is calculated; The text region in the first image is obtained based on the approximate binary image.

2. The method as described in claim 1, characterized in that, The probability maps of the first image generated by each prediction model are fused to obtain a fused probability map, including: The values ​​of the same pixel in the probability maps of the first image generated by each prediction model are weighted and summed to obtain the fusion value of the pixel; the fusion probability map is obtained based on the fusion values ​​of each pixel in the first image.

3. The method as described in claim 2, characterized in that, The values ​​of each pixel in each probability map have the same weight.

4. The method as described in claim 2, characterized in that, The at least two prediction models include a first prediction model and a second prediction model, wherein the first prediction model is a first feature pyramid network and the second prediction model is a first encoding / decoding network.

5. The method as described in claim 4, characterized in that, The values ​​of the same pixel in the probability maps of the first image generated by each prediction model are weighted and summed to obtain the fused value of the pixel, including: Obtain the first value of the pixel in the first probability map generated by the first feature pyramid network, and the second value in the second probability map generated by the first codec network; calculate the absolute value of the difference between the first value and the second value. If the absolute value is less than a preset threshold, the average of the first value and the second value is calculated to obtain the fused value of the pixel. If the absolute value is greater than or equal to the preset threshold, the nearest distance between the pixel and the main region in the second probability map is determined, and the first value and the second value are weighted and summed according to the first weight and the second weight to obtain the fused value of the pixel. The first weight is a preset fixed value, and the second weight is negatively correlated with the nearest distance.

6. The method as described in claim 1, characterized in that, Based on the fusion probability map and the threshold map, an approximate binary map of the first image is calculated, including: The fusion probability map and the threshold map are input into a pre-trained differentiable binarization DB network to obtain an approximate binary map of the first image output by the DB network.

7. The method according to any one of claims 1 to 6, characterized in that, Before inputting the first image to be detected into at least two prediction models, the method further includes: The system receives a document image uploaded by a user, extracts a seal image containing a seal pattern from the document image, and obtains the first image, wherein the seal pattern contains text content.

8. A text region detection device, characterized in that, include: The prediction module is used to input the first image to be detected into at least two prediction models respectively, and obtain the probability map of the first image generated by each prediction model and the threshold map of the first image generated by the first prediction model, wherein the first prediction model is one of the at least two prediction models. The fusion module is used to fuse the probability maps of the first image generated by each prediction model to obtain a fused probability map; The text region acquisition module is used to calculate an approximate binary map of the first image based on the fusion probability map and the threshold map; and to obtain the text region in the first image based on the approximate binary map.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

Images