Text detection method and system

Abstract

The invention provides a text detection method and system, and the method comprises the steps: inputting a to-be-detected first image into a graph prediction model, obtaining a first probability graph and a first threshold graph of the first image generated by the graph prediction model, and obtaining a first probability graph and a first threshold graph of the first image based on the first probability graph and the first threshold graph; calculating to obtain a first approximate binary image of the first image; and according to the first approximate binary image, outwards searching a bounding box with the highest average threshold on the first threshold image to obtain a textbox in the first image. The position of the textbox can be accurately positioned, and the accuracy of subsequent text recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of optical character recognition (OCR) and machine learning technology, and specifically to a text detection method and system. Background Art

[0002] The scene text detection task aims to locate text instances in images. For the current input image, it detects and returns the text instances in the image. The text detection task has rich applications in the real world, including office automation, visual search, and geographic positioning, and is therefore of great significance.

[0003] For the task of detecting text in scenes of arbitrary shapes, the most popular method in the industry is to use a segmentation-based method, such as the Differentiable Binarization (DB) algorithm. Segmentation-based scene text detection converts the probability map (heat map) generated by the segmentation algorithm into bounding boxes and text areas, which includes a binarization post-processing process. The existing segmentation-based methods have difficulty accurately locating the position of the text box in the post-processing stage, and are prone to problems such as the text box being too close to the edge of the text or too far away from the edge of the text, thereby affecting the accuracy of subsequent text recognition. Summary of the Invention

[0004] At least one embodiment of the present application provides a text detection method and system that can accurately locate the position of a text box and improve the accuracy of subsequent text recognition.

[0005] According to a first aspect of the present application, at least one embodiment provides a text detection method, including:

[0006] Inputting a first image to be detected into an image prediction model, obtaining a first probability map and a first threshold map of the first image generated by the image prediction model, and calculating a first approximate binary map of the first image based on the first probability map and the first threshold map;

[0007] According to the first approximate binary image, a bounding box with the highest average threshold value is searched outward on the first threshold image to obtain a text box in the first image.

[0008] Optionally, searching outward on the first threshold map for a bounding box having a highest average threshold value based on the first approximate binary image to obtain a text box in the first image includes:

[0009] Determine a first bounding box according to the first approximate binary image;

[0010] determining a second bounding box on the first threshold map according to the position of the first bounding box, wherein the position of the second bounding box corresponds to the position of the first bounding box;

[0011] Taking the second bounding box as a reference, the first threshold map is expanded outward to obtain multiple bounding boxes, and the average threshold of the pixels on each bounding box is calculated, and the bounding box with the largest average threshold is used as the text box in the first image.

[0012] Optionally, the graph prediction model includes an image feature extraction module, a probability graph and threshold graph prediction module, and a binary graph calculation module, and the method further includes:

[0013] The steps of pre-training the graph prediction model specifically include:

[0014] According to the marked text boxes in the sample image, a true value probability map, a true value threshold map and a true value binary map of the sample image are generated, and a text box boundary weight matrix of the sample image is generated;

[0015] Inputting the sample image into the graph prediction model to obtain a prediction probability map, a prediction threshold map, and a prediction approximate binary map of the sample image generated by the graph prediction model;

[0016] Calculating a first loss of the probability map according to the true value probability map and the predicted probability map of the sample image;

[0017] Calculating a second loss of the binary image according to the true binary image and the predicted approximate binary image of the sample image;

[0018] performing a weighted summation of the losses between the true value threshold map and the predicted threshold map of the sample image according to the text box boundary weight matrix to obtain a third loss of the threshold map;

[0019] Constructing a loss function according to the first loss, the second loss and the third loss;

[0020] The graph prediction model is trained by backpropagation of the loss function to obtain a trained graph prediction model.

[0021] Optionally, generating a text box boundary weight matrix for the sample image includes:

[0022] For each pixel in the sample image, setting a weight for the pixel according to the closest distance between the pixel and the marked text box, wherein the weight is negatively correlated with the closest distance;

[0023] A text box boundary weight matrix of the sample image is obtained according to the weight of each pixel in the sample image.

[0024] Optionally, the loss between the true value threshold map and the predicted threshold map of the sample image is: the Manhattan distance between each pixel point in the true value threshold map and the predicted threshold map of the sample image.

[0025] Optionally, the first loss and the second loss are binary cross-entropy (BCE) losses.

[0026] According to the second aspect of the present application, at least one embodiment provides a text detection system, including: a graph prediction model and a text box prediction module, wherein:

[0027] The image prediction model is configured to receive a first image to be detected, generate a first probability map and a first threshold map for the first image, and calculate a first approximate binary map of the first image based on the first probability map and the first threshold map;

[0028] The text box prediction module is configured to search outward on the first threshold map for a bounding box with the highest average threshold value based on the first approximate binary image, so as to obtain a text box in the first image.

[0029] Optionally, the text box prediction module is further configured to:

[0030] Determine a first bounding box according to the first approximate binary image;

[0031] determining a second bounding box on the first threshold map according to the position of the first bounding box, wherein the position of the second bounding box corresponds to the position of the first bounding box;

[0032] Taking the second bounding box as a reference, the first threshold map is expanded outward to obtain multiple bounding boxes, and the average threshold of the pixels on each bounding box is calculated, and the bounding box with the largest average threshold is used as the text box in the first image.

[0033] Optionally, the graph prediction model includes an image feature extraction module, a probability graph and threshold graph prediction module, and a binary graph calculation module, and the text detection system further includes: a data preprocessing module and a training module;

[0034] The data preprocessing module is used to generate a true value probability map, a true value threshold map and a true value binary map of the sample image according to the marked text boxes in the sample image, and generate a text box boundary weight matrix of the sample image;

[0035] The training module is used to pre-train the graph prediction model, and the pre-training of the graph prediction model specifically includes: calculating the first loss of the probability map based on the true value probability map and the predicted probability map of the sample image; calculating the second loss of the binary map based on the true value binary map and the predicted approximate binary map of the sample image; performing weighted summation of the losses between the true value threshold map and the predicted threshold map of the sample image according to the text box boundary weight matrix to obtain the third loss of the threshold map; constructing a loss function based on the first loss, the second loss and the third loss; and using the loss function to back-propagate and train the graph prediction model to obtain a trained graph prediction model.

[0036] Optionally, the data preprocessing module is also used to: for each pixel point in the sample image, set the weight of the pixel point according to the nearest distance between the pixel point and the marked text box, wherein the weight is negatively correlated with the nearest distance; and obtain the text box boundary weight matrix of the sample image according to the weight of each pixel point in the sample image.

[0037] According to the third aspect of the present application, at least one embodiment provides a text detection system, comprising a processor, a memory, and a program or instruction stored on the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in any one of the first aspects.

[0038] According to the fourth aspect of the present application, at least one embodiment provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the steps of any method of the first aspect are implemented.

[0039] According to a fifth aspect of the present application, at least one embodiment provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in any one of the first aspects.

[0040] Compared with the prior art, the text detection method and system provided by the embodiments of the present application, in the post-processing stage of text box generation, uses an approximate binary image as a reference, expands outward on the threshold map to find the area with the highest threshold in the threshold map as the text box boundary, which can improve the boundary sensitivity of the model and reduce post-processing time. In addition, during the model training process, the embodiments of the present application construct a boundary weight matrix. When calculating the loss of the threshold map, the boundary weight matrix is ​​used to increase the boundary sensitivity of the model, helping the model to better find the text boundary box. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0042] Figure 1 A flowchart of a text detection method according to an embodiment of the present application is shown;

[0043] Figure 2 This is an example diagram of a training process of the graph prediction model according to an embodiment of the present application;

[0044] Figure 3 This is an example diagram of a text box in an embodiment of the present application;

[0045] Figure 4 A structural diagram of a text detection system according to an embodiment of the present application;

[0046] Figure 5 This is another structural diagram of the text detection system according to an embodiment of the present application;

[0047] Figure 6 This is another structural diagram of the text detection system according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following describes exemplary embodiments of the present application in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0049] It should be understood that references throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present application. Therefore, the appearance of "in one embodiment" or "in an embodiment" throughout this specification does not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. The terms "first," "second," and so on, used in the specification and claims of this application are used to distinguish similar items and are not necessarily used to describe a particular order or sequential sequence. It should be understood that such usage is interchangeable where appropriate, such that the embodiments of the present application described herein can, for example, be implemented in an order other than that illustrated or described herein. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus. The term "and / or" used in the specification and claims refers to at least one of the connected items.

[0050] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0051] The following description provides examples and does not limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the function and arrangement of the elements discussed without departing from the spirit and scope of this disclosure. The various examples may appropriately omit, substitute, or add various procedures or components. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0052] An embodiment of the present application provides a text detection method, which uses a pre-trained graph prediction model to generate a probability map, a threshold map, and an approximate binary map of the image to be detected, and then searches in the threshold map to generate a text box with a suitable position. The embodiment of the present application can accurately locate the position of the text box, thereby improving the accuracy of subsequent text recognition.

[0053] Please refer to Figure 1 The text detection method provided in the embodiment of the present application includes the following steps:

[0054] Step 11: Input the first image to be detected into the image prediction model, obtain the first probability map and the first threshold map of the first image generated by the image prediction model, and calculate the first approximate binary map of the first image based on the first probability map and the first threshold map.

[0055] The graph prediction model has three outputs: a probability map, a threshold map, and an approximate binary map. The value of each pixel in the probability map of an image represents the probability that the corresponding pixel belongs to a text region. The value of each pixel in the threshold map represents the binarization threshold for the corresponding pixel in the image. The approximate binary map is calculated from the probability map and threshold map using algorithms such as DB. The pixel values ​​in the approximate binary map are either 0 or 1.

[0056] Step 12: Based on the first approximate binary image, search outward on the first threshold image for a bounding box with the highest average threshold value to obtain a text box in the first image.

[0057] Here, the embodiment of the present application can determine a first bounding box on the first approximate binary image. Here, a first bounding box can be determined based on the first approximate binary image.

[0058] Then, based on the position of the first bounding box, a second bounding box is determined on the first threshold map, where the position of the second bounding box corresponds to the position of the first bounding box. That is, the position coordinates of the first bounding box on the first approximate binary map are the same as the position coordinates of the second bounding box on the first threshold map.

[0059] Then, the second bounding box is used as a reference, and the first threshold map is expanded outward to obtain multiple bounding boxes. The average threshold of the pixels on each bounding box is calculated, and the bounding box with the largest average threshold is used as the text box in the first image.

[0060] For example, with the center point of the second bounding box as the center, each point on the second bounding box is expanded away from the center to generate different bounding boxes. For each generated bounding box, the average threshold value of each pixel in the bounding box on the first threshold map is calculated to obtain an average threshold value. The bounding box with the largest average threshold value is then used as the final text box of the text area.

[0061] Through the above steps, in the post-processing process of determining the text box based on the first approximate binary image and the first threshold image, the embodiment of the present application uses the boundary box with the highest average threshold in the threshold image as the text box of the text area, which can improve the sensitivity of text box detection and more accurately locate the position of the text box, thereby reducing the error caused by inaccurate text box position in subsequent text recognition, and can improve the accuracy of subsequent text recognition.

[0062] Before the above step 11, the embodiment of the present application may also pre-train the graph prediction model. Specifically, Figure 2 As shown, the training includes the following steps:

[0063] Step A: Generate a true value probability map, a true value threshold map and a true value binary map of the sample image according to the marked text boxes in the sample image, and generate a text box boundary weight matrix of the sample image.

[0064] The above steps are data preprocessing steps. In this step, the positions of the labeled text boxes in the sample image, i.e., the text box ground truth, are used to generate the sample image's true value probability map, true value threshold map, true value binary map, and text box boundary weight matrix w.

[0065] (1) A generation process of the true value probability map is as follows:

[0066] Given an image (such as the sample image), the Vatti clipping algorithm is used to shrink the polygon G of the labeled text box to Gs to generate a positive region. Here, the shrinkage offset D is calculated from the length L and area A of the outer contour of the polygon G:

[0067]

[0068] Where r is the shrinkage ratio, which can be set to 0.4 based on experience.

[0069] Then, the pixels in the positive area of ​​the image are assigned a value of 1, and the other pixels are assigned a value of 0, thereby obtaining a true value probability map.

[0070] (2) A process for generating a true value threshold map is as follows:

[0071] First, the polygon G is expanded outward by an offset D to generate Gd. Then, the area between Gs and Gd is regarded as the boundary of the text area, where the true value of the true value threshold map can be generated by calculating the distance from each pixel between Gs and Gd to G. For example, the closer the pixel is to G, the larger the corresponding value in the true value threshold map. In the image, the other areas outside the area between Gs and Gd can be set to all 0. Figure 3 For example, assuming that the red box is the marked text box G, the green box is Gs, and the purple box is Gd, the value of each pixel in the area between the purple box and the green box is set according to the distance from the pixel to the red box; the value of the pixel in other areas can be set to 0.

[0072] (3) The process of generating the true binary map is the same as that of generating the true probability map, and will not be repeated here.

[0073] (4) The generation process of the text box boundary weight matrix w is similar to the generation process of the true value threshold map.

[0074] Here, for each pixel point in the sample image, the weight of the pixel point can be set according to the nearest distance between the pixel point and the marked text box, wherein the weight is negatively correlated with the nearest distance. In addition, the values ​​of the elements in the text box boundary weight matrix are all positive values ​​and are within a preset interval, for example, not less than 0.3 and not greater than 1. The closer a pixel point is to the marked text box, the greater the weight corresponding to the pixel point. In this way, the text box boundary weight matrix of the sample image can be obtained according to the weight of each pixel point in the sample image, that is, a weight matrix with the same dimension as the sample image is generated. Optionally, the upper and lower limits of the interval range can be set to configurable or modifiable values.

[0075] Step B: input the sample image into the graph prediction model to obtain a prediction probability map, a prediction threshold map, and a prediction approximate binary map of the sample image generated by the graph prediction model.

[0076] In an embodiment of the present application, the graph prediction model includes an image feature extraction module, a probability map and threshold map prediction module, and a binary map calculation module. The specific structure of the above modules can refer to the relevant modules in the DB++ network. For example, the image feature extraction module can be implemented using a feature pyramid network (FPN) and an adaptive scale fusion (ASF) module, wherein the feature pyramid network is used to generate pyramid features of the image, and then the pyramid features are upsampled to the same scale and passed to the ASF module after cascading to generate the final context feature F. Here, the sample image can be input into the feature pyramid trunk. Then, the pyramid features are upsampled to the same scale and passed to the ASF module after cascading to generate the final context feature F.

[0077] The probability map and threshold map prediction module is used to generate a predicted probability map (P) and a predicted threshold map (T) according to the context feature F. The binary map calculation module is used to calculate the predicted approximate binary map (B) by using the predicted probability map (P) and the predicted threshold map (T). Specifically, the value of the pixel at the coordinate (i, j) in the predicted approximate binary map (B) is One way to calculate is as follows:

[0078]

[0079] Among them, k represents the magnification factor, which can be set to 50 according to experience; i and j represent the horizontal and vertical coordinates of a pixel point respectively; P i,j and T i,j Represents the value of the pixel at coordinate (i, j) in the prediction probability map (P) and the prediction threshold map (T), respectively.

[0080] Step C: Calculate a first loss of the probability map based on the true value probability map and the predicted probability map of the sample image. Here, the first loss can be a binary cross-entropy (BCE) loss.

[0081] Step D: Calculate a second loss of the binary image based on the true binary image and the predicted approximate binary image of the sample image. Here, the second loss may be a BCE loss.

[0082] Step E: Perform a weighted summation of the losses between the true threshold map and the predicted threshold map of the sample image based on the text box boundary weight matrix to obtain a third loss of the threshold map. Here, the loss between the true threshold map and the predicted threshold map of the sample image may be the Manhattan distance (L1 distance) between each pixel in the true threshold map and the predicted threshold map of the sample image.

[0083] Step F: Construct a loss function based on the first loss, the second loss, and the third loss, and use the loss function to train the graph prediction model through backpropagation to obtain a trained graph prediction model.

[0084] In the training process of the embodiment of the present application, the prediction probability map (P), the prediction threshold map (T) and the prediction approximate binary map are The loss function L can be expressed as the loss Ls of the probability map and the loss L of the binary map. b and the loss L of the threshold map t The weighted sum of:

[0085] L=L s +α×L b +β×L t

[0086] α and β are preset constants, and according to the value of the loss, a value of α and β can be set to 1.0 and 10 respectively.

[0087] During the training process, iterative training is performed with the goal of minimizing the loss function L until the preset training end conditions are met, such as reaching convergence or reaching the maximum number of iterations.

[0088] Among them, the probability map and the approximate binary map share the same supervision and both use binary cross entropy (BCE) loss. For example:

[0089]

[0090] Among them, S l is a sampling set, where the ratio of positive and negative samples can be 1:3. i Represents the true value in the true value probability map or the true value binary map, y i Represents the values ​​in the predicted probability map or predicted binary map.

[0091] The loss L of the threshold map t One way to calculate d can be to calculate the weighted sum of the L1 distances between the predicted values ​​and the true values ​​of the prediction threshold map within the expanded polygon Gd:

[0092]

[0093] Among them, R d is a set of indices of pixels within the expanded polygon Gd (all pixels within the expanded polygon Gd are counted); i is the predicted value of the prediction threshold map, x i is the true value of the true threshold map, w i are the corresponding elements in the boundary weight matrix w generated in S10.

[0094] From the above, it can be seen that in the post-processing stage, the embodiment of the present application uses the approximate binary image as a reference, expands outward on the threshold map to find the area with the highest threshold in the threshold map as the text box boundary, which can reduce the post-processing time. In addition, in the model training process, the embodiment of the present application constructs a boundary weight matrix. When calculating the loss of the threshold map, the boundary weight matrix is ​​used to increase the boundary sensitivity of the model, helping the model to better find the text boundary box. The text detection system of the embodiment of the present application can improve the boundary sensitivity of the model, reduce the post-processing time, and reduce the error brought to the text recognition stage.

[0095] Based on the above method, the present application also provides a device for implementing the above method. Figure 4 The embodiment of the present application provides a text detection system 40, including: a graph prediction model 41 and a text box prediction module 42, wherein:

[0096] The image prediction model 41 is configured to receive a first image to be detected, generate a first probability map and a first threshold map for the first image, and calculate a first approximate binary map of the first image based on the first probability map and the first threshold map;

[0097] The text box prediction module 42 is configured to search outward on the first threshold map for a bounding box with the highest average threshold value based on the first approximate binary image, so as to obtain a text box in the first image.

[0098] Through the above modules, the text detection system of the embodiment of the present application can accurately locate the position of the text box and improve the accuracy of subsequent text recognition.

[0099] Optionally, the text box prediction module is further configured to:

[0100] Determine a first bounding box according to the first approximate binary image;

[0101] determining a second bounding box on the first threshold map according to the position of the first bounding box, wherein the position of the second bounding box corresponds to the position of the first bounding box;

[0102] Taking the second bounding box as a reference, the first threshold map is expanded outward to obtain multiple bounding boxes, and the average threshold of the pixels on each bounding box is calculated, and the bounding box with the largest average threshold is used as the text box in the first image.

[0103] like Figure 5 As shown, the graph prediction model may specifically include an image feature extraction module, a probability graph and threshold graph prediction module, and a binary graph calculation module. The text detection system 40 further includes: a data preprocessing module 43 and a training module 44;

[0104] The data preprocessing module 43 is configured to generate a true value probability map, a true value threshold map, and a true value binary map of the sample image based on the labeled text boxes in the sample image, and generate a text box boundary weight matrix of the sample image;

[0105] The training module 44 is used to pre-train the graph prediction model, and the pre-training of the graph prediction model specifically includes: calculating a first loss of the probability map based on the true value probability map and the predicted probability map of the sample image; calculating a second loss of the binary map based on the true value binary map and the predicted approximate binary map of the sample image; performing a weighted summation of the losses between the true value threshold map and the predicted threshold map of the sample image according to the text box boundary weight matrix to obtain a third loss of the threshold map; constructing a loss function based on the first loss, the second loss and the third loss; and using the loss function to back-propagate and train the graph prediction model to obtain a trained graph prediction model.

[0106] Optionally, the data preprocessing module is also used to: for each pixel point in the sample image, set the weight of the pixel point according to the nearest distance between the pixel point and the marked text box, wherein the weight is negatively correlated with the nearest distance; and obtain the text box boundary weight matrix of the sample image according to the weight of each pixel point in the sample image.

[0107] Optionally, the loss between the true value threshold map and the predicted threshold map of the sample image is: the Manhattan distance between each pixel point in the true value threshold map and the predicted threshold map of the sample image.

[0108] Optionally, the first loss and the second loss are binary cross-entropy (BCE) losses.

[0109] It should be noted that the various devices / systems provided in the above embodiments are devices corresponding to the above text detection method, and the implementation methods in the above embodiments are all applicable to the embodiments of the device and can achieve the same technical effects. The above-mentioned device provided in the embodiments of the present application can implement all the method steps implemented in the above method embodiments and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be specifically described here.

[0110] Please refer to Figure 6 , the embodiment of the present application also provides a hardware structure block diagram of the text detection system, such as Figure 6 As shown, the text detection system 600 includes:

[0111] Processor 602; and

[0112] a memory 604 in which computer program instructions are stored,

[0113] When the computer program instructions are executed by the processor, the processor 602 is caused to perform the following steps:

[0114] Inputting a first image to be detected into an image prediction model, obtaining a first probability map and a first threshold map of the first image generated by the image prediction model, and calculating a first approximate binary map of the first image based on the first probability map and the first threshold map;

[0115] According to the first approximate binary image, a bounding box with the highest average threshold value is searched outward on the first threshold image to obtain a text box in the first image.

[0116] It should be noted that the various systems provided in the above embodiments are devices corresponding to the above text detection methods, and the implementation methods in the above embodiments are all applicable to the embodiments of the devices and can achieve the same technical effects. The above-mentioned device provided in the embodiments of the present application can implement all the method steps implemented in the above-mentioned method embodiments and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those in the method embodiments will not be described in detail here.

[0117] Furthermore, if Figure 6As shown, the model training device 600 also includes a network interface 601, an input device 603, a hard disk 605, and a display device 606.

[0118] The above-mentioned various interfaces and devices can be interconnected through a bus architecture. The bus architecture can be a bus and bridge that can include any number of interconnected buses. Specifically, various circuits of one or more central processing units (CPUs) and / or graphics processing units (GPUs) represented by processor 602, and one or more memories represented by memory 604 are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together. It can be understood that the bus architecture is used to achieve connection and communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are all well known in the art and therefore will not be described in detail herein.

[0119] The network interface 601 can be connected to a network (such as the Internet, a local area network, etc.), receive data such as original training samples from the network, and save the received data in the hard disk 605.

[0120] The input device 603 can receive various instructions input by the operator and send them to the processor 602 for execution. The input device 603 can include a keyboard or a pointing device (e.g., a mouse, a trackball, a touchpad, or a touch screen).

[0121] The display device 606 can display the results obtained by the processor 602 executing instructions, such as displaying the model training progress.

[0122] The memory 604 is used to store programs and data necessary for the operation of the operating system, as well as data such as intermediate results during the calculation process of the processor 602.

[0123] It is understood that the memory 604 in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. The memory 604 of the apparatus and method described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0124] In some embodiments, the memory 604 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 6041 and application programs 6042 .

[0125] The operating system 6041 includes various system programs, such as a framework layer, a core library layer, and a driver layer, for implementing various basic services and handling hardware-based tasks. Application programs 6042 include various application programs, such as a browser, for implementing various application services. Programs implementing the methods of the embodiments of the present application may be included in application programs 6042.

[0126] The methods disclosed in the above embodiments of this application can be applied to or implemented by processor 602. Processor 602 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 602 or by software instructions. The above processor 602 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 604, and processor 602 reads the information in memory 604 and, in conjunction with its hardware, completes the steps of the above method.

[0127] It is understood that the embodiments described herein can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0128] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0129] Optionally, when the computer program is executed by the processor 602, the following steps may be further implemented:

[0130] Determine a first bounding box according to the first approximate binary image;

[0131] determining a second bounding box on the first threshold map according to the position of the first bounding box, wherein the position of the second bounding box corresponds to the position of the first bounding box;

[0132] Taking the second bounding box as a reference, the first threshold map is expanded outward to obtain multiple bounding boxes, and the average threshold of the pixels on each bounding box is calculated, and the bounding box with the largest average threshold is used as the text box in the first image.

[0133] Optionally, the image prediction model includes an image feature extraction module, a probability map and threshold map prediction module, and a binary map calculation module. When the computer program is executed by the processor 602, the following steps may be further implemented:

[0134] The steps of pre-training the graph prediction model specifically include:

[0135] According to the marked text boxes in the sample image, a true value probability map, a true value threshold map and a true value binary map of the sample image are generated, and a text box boundary weight matrix of the sample image is generated;

[0136] Inputting the sample image into the graph prediction model to obtain a prediction probability map, a prediction threshold map, and a prediction approximate binary map of the sample image generated by the graph prediction model;

[0137] Calculating a first loss of the probability map according to the true value probability map and the predicted probability map of the sample image;

[0138] Calculating a second loss of the binary image according to the true binary image and the predicted approximate binary image of the sample image;

[0139] performing a weighted summation of the losses between the true value threshold map and the predicted threshold map of the sample image according to the text box boundary weight matrix to obtain a third loss of the threshold map;

[0140] Constructing a loss function according to the first loss, the second loss and the third loss;

[0141] The graph prediction model is trained by backpropagation of the loss function to obtain a trained graph prediction model.

[0142] Optionally, when the computer program is executed by the processor 602, the following steps may be further implemented:

[0143] For each pixel in the sample image, setting a weight for the pixel according to the closest distance between the pixel and the marked text box, wherein the weight is negatively correlated with the closest distance;

[0144] A text box boundary weight matrix of the sample image is obtained according to the weight of each pixel in the sample image.

[0145] Optionally, the loss between the true value threshold map and the predicted threshold map of the sample image is: the Manhattan distance between each pixel point in the true value threshold map and the predicted threshold map of the sample image.

[0146] Optionally, the first loss and the second loss are binary cross-entropy (BCE) losses.

[0147] It should be noted here that the above-mentioned system provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0148] In some embodiments of the present application, a computer-readable storage medium is further provided, on which a program is stored. When the program is executed by a processor, the following steps are implemented:

[0149] Inputting a first image to be detected into an image prediction model, obtaining a first probability map and a first threshold map of the first image generated by the image prediction model, and calculating a first approximate binary map of the first image based on the first probability map and the first threshold map;

[0150] According to the first approximate binary image, a bounding box with the highest average threshold value is searched outward on the first threshold image to obtain a text box in the first image.

[0151] When the program is executed by the processor, it can implement all the implementation methods of the above text detection method and achieve the same technical effect. To avoid repetition, it will not be described here.

[0152] An embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned text detection method embodiment are implemented and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0153] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0154] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0155] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0156] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0158] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0159] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A text detection method, characterized in that: include: Inputting a first image to be detected into an image prediction model, obtaining a first probability map and a first threshold map of the first image generated by the image prediction model, and calculating a first approximate binary map of the first image based on the first probability map and the first threshold map; According to the first approximate binary image, a bounding box with the highest average threshold value is searched outward on the first threshold image to obtain a text box in the first image.

2. The method according to claim 1, wherein According to the first approximate binary image, searching outward on the first threshold map for a bounding box with the highest average threshold value to obtain a text box in the first image includes: Determine a first bounding box according to the first approximate binary image; determining a second bounding box on the first threshold map according to the position of the first bounding box, wherein the position of the second bounding box corresponds to the position of the first bounding box; Taking the second bounding box as a reference, the first threshold map is expanded outward to obtain multiple bounding boxes, and the average threshold of the pixels on each bounding box is calculated, and the bounding box with the largest average threshold is used as the text box in the first image.

3. The method according to claim 1, wherein The graph prediction model includes an image feature extraction module, a probability graph and threshold graph prediction module, and a binary graph calculation module. The method further includes: The steps of pre-training the graph prediction model specifically include: According to the marked text boxes in the sample image, a true value probability map, a true value threshold map and a true value binary map of the sample image are generated, and a text box boundary weight matrix of the sample image is generated; Inputting the sample image into the graph prediction model to obtain a prediction probability map, a prediction threshold map, and a prediction approximate binary map of the sample image generated by the graph prediction model; Calculating a first loss of the probability map according to the true value probability map and the predicted probability map of the sample image; Calculating a second loss of the binary image according to the true binary image and the predicted approximate binary image of the sample image; performing a weighted summation of the losses between the true value threshold map and the predicted threshold map of the sample image according to the text box boundary weight matrix to obtain a third loss of the threshold map; Constructing a loss function according to the first loss, the second loss and the third loss; The graph prediction model is trained by backpropagation of the loss function to obtain a trained graph prediction model.

4. The method according to claim 3, wherein Generating a text box boundary weight matrix of the sample image, including: For each pixel in the sample image, setting a weight for the pixel according to the closest distance between the pixel and the marked text box, wherein the weight is negatively correlated with the closest distance; A text box boundary weight matrix of the sample image is obtained according to the weight of each pixel in the sample image.

5. The method according to claim 3, wherein The loss between the true value threshold map and the predicted threshold map of the sample image is: the Manhattan distance between each pixel point in the true value threshold map and the predicted threshold map of the sample image.

6. The method according to claim 3, wherein The first loss and the second loss are binary cross entropy losses.

7. A text detection system, characterized in that: include: Graph prediction model and text box prediction module, where The image prediction model is configured to receive a first image to be detected, generate a first probability map and a first threshold map for the first image, and calculate a first approximate binary map of the first image based on the first probability map and the first threshold map; The text box prediction module is configured to search outward on the first threshold map for a bounding box with the highest average threshold value based on the first approximate binary image, so as to obtain a text box in the first image.

8. The system according to claim 7, wherein: The text box prediction module is further used to: Determine a first bounding box according to the first approximate binary image; determining a second bounding box on the first threshold map according to the position of the first bounding box, wherein the position of the second bounding box corresponds to the position of the first bounding box; Taking the second bounding box as a reference, the first threshold map is expanded outward to obtain multiple bounding boxes, and the average threshold of the pixels on each bounding box is calculated, and the bounding box with the largest average threshold is used as the text box in the first image.

9. The system according to claim 8, wherein The graph prediction model includes an image feature extraction module, a probability graph and threshold graph prediction module, and a binary graph calculation module. The text detection system also includes: a data preprocessing module and a training module; The data preprocessing module is used to generate a true value probability map, a true value threshold map and a true value binary map of the sample image according to the marked text boxes in the sample image, and generate a text box boundary weight matrix of the sample image; The training module is used to pre-train the graph prediction model, and the pre-training of the graph prediction model specifically includes: calculating the first loss of the probability map based on the true value probability map and the predicted probability map of the sample image; calculating the second loss of the binary map based on the true value binary map and the predicted approximate binary map of the sample image; performing weighted summation of the losses between the true value threshold map and the predicted threshold map of the sample image according to the text box boundary weight matrix to obtain the third loss of the threshold map; constructing a loss function based on the first loss, the second loss and the third loss; and using the loss function to back-propagate and train the graph prediction model to obtain a trained graph prediction model.

10. The system according to claim 9, wherein: The data preprocessing module is further used to: set a weight for each pixel in the sample image based on the nearest distance between the pixel and the marked text box, wherein the weight is negatively correlated with the nearest distance; and obtain a text box boundary weight matrix for the sample image based on the weight of each pixel in the sample image.

11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images