Seal character recognition method and device and electronic equipment

Abstract

The invention discloses a seal character recognition method and device and electronic equipment, and belongs to the technical field of character recognition. The method comprises the steps that a to-be-processed seal image is acquired, the seal image comprises a plurality of seal characters, and the outlines of the plurality of seal characters are arc-shaped; a plurality of character blocks of the seal image are recognized, a first seal image is obtained, and each character block corresponds to one seal character; based on the position information corresponding to the character blocks, the first seal image is segmented to obtain a second seal image, the second seal image comprises a plurality of character block images, and each character block image corresponds to one character block; inputting the second seal image into a character recognition model to obtain seal character information corresponding to the seal image output by the character recognition model; wherein the character recognition model is obtained by training a target seal image set. The method can improve the accuracy of the character recognition result of the arc-shaped seal.

Description

Technical Field

[0001] The present application belongs to the technical field of text recognition, and in particular, relates to a seal text recognition method, device and electronic equipment. Background Art

[0002] Curved seals are often used for authenticity identification and review of important documents such as official documents and certificates. Automatic recognition of curved seals can replace manual review and comparison, which can solve the problem of high time and labor costs of manual review in document management. Compared with text recognition in ordinary document scenarios, the recognition of curved seals is more complicated due to the angled arrangement of text, severe background occlusion, large differences in seal image quality, and difficulty in obtaining annotation data.

[0003] At present, the arc-shaped text on the seal is usually straightened by polar coordinate transformation for easy recognition, but this process may affect the shape of the arc-shaped text, resulting in reduced accuracy of the text recognition results. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a seal character recognition method, device and electronic device, which can reduce the influence of text deformation on seal character recognition and improve the accuracy of arc seal character recognition results.

[0005] In a first aspect, the present application provides a seal text recognition method, the method comprising:

[0006] Acquire a seal image to be processed, wherein the seal image includes a plurality of seal characters, and the contours of the plurality of seal characters are arc-shaped;

[0007] Recognize multiple text blocks of the seal image to obtain a first seal image, each of the text blocks corresponding to one seal character;

[0008] Based on the position information corresponding to the text block, the first seal image is segmented to obtain a second seal image, wherein the second seal image includes a plurality of text block images, each of which corresponds to one of the text blocks;

[0009] Inputting the second seal image into a text recognition model to obtain seal text information corresponding to the seal image output by the text recognition model;

[0010] Wherein, the text recognition model is obtained by training the target seal image set.

[0011] According to the seal text recognition method of the present application, a first seal image is obtained by identifying the text blocks corresponding to the seal text in the seal image, and the first seal image is segmented based on the position information corresponding to the text blocks to obtain a second seal image that can characterize the position distribution of the seal text in the seal. The second seal image including multiple text block images is input into a text recognition model, and the seal text information can be output according to the position distribution of the seal text through the text recognition model. There is no need to straighten the arc-shaped text on the seal image, which reduces the influence of text deformation on seal text recognition and can improve the accuracy of arc-shaped seal text recognition results.

[0012] According to an embodiment of the present application, the text recognition model includes a text recognition module and a text reasoning module, and the step of inputting the second seal image into the text recognition model to obtain seal text information corresponding to the seal image output by the text recognition model includes:

[0013] Inputting the second seal image into the text recognition module, and obtaining a text probability matrix corresponding to a plurality of seal characters output by the text recognition module based on the text information and position information corresponding to the character block image;

[0014] The text probability matrix is ​​input into a text reasoning module to obtain the seal text information output by the text reasoning module.

[0015] According to an embodiment of the present application, the text recognition module includes an average pooling layer, a global feature extraction layer and a prediction layer, and the second seal image is input into the text recognition module, and based on the text information and position information corresponding to the text block image, a text probability matrix corresponding to a plurality of seal characters output by the text recognition module is obtained, including:

[0016] Inputting the second seal image to the average pooling layer to obtain the second seal image with reduced dimension output by the average pooling layer;

[0017] Inputting the second seal image with reduced dimension into the global feature extraction layer, and obtaining global context information corresponding to the second seal image output by the global feature extraction layer based on text information and position information corresponding to the character block image;

[0018] The global context information is input into the prediction layer to obtain the text probability matrix output by the prediction layer.

[0019] According to an embodiment of the present application, the global feature extraction layer includes a first feature extraction layer and a second feature extraction layer, the second seal image with reduced dimension is input into the global feature extraction layer, and based on the text information and position information corresponding to the text block image, the global context information corresponding to the second seal image output by the global feature extraction layer is obtained, including:

[0020] Inputting the second seal image with reduced dimension into the first feature extraction layer, and obtaining first feature information output by the first feature extraction layer based on text information and position information corresponding to the character block image;

[0021] The first feature information is input into the second feature extraction layer to obtain the global context information output by the second feature extraction layer.

[0022] According to one embodiment of the present application, the prediction layer is a fully convolutional layer or a fully connected layer.

[0023] According to an embodiment of the present application, segmenting the first seal image based on the position information corresponding to the text block to obtain the second seal image includes:

[0024] Based on the position information corresponding to the character block, the first seal image is segmented to obtain a plurality of the character block images;

[0025] The plurality of character block images are rendered onto a reference image to obtain the second seal image.

[0026] According to one embodiment of the present application, the identifying of the multiple text blocks of the seal image includes:

[0027] A plurality of text blocks of the seal image are identified through a differentiable binarization network model.

[0028] According to an embodiment of the present application, the step of inputting the second seal image into a text recognition model includes:

[0029] performing masking processing on the area outside the area corresponding to the text block image in the second seal image;

[0030] The second seal image after masking is input into the text recognition model.

[0031] In a second aspect, the present application provides a seal character recognition device, the device comprising:

[0032] An acquisition module, used for acquiring a seal image to be processed, wherein the seal image includes a plurality of seal characters, and the contours of the plurality of seal characters are arc-shaped;

[0033] A first processing module, used for identifying a plurality of text blocks of the seal image to obtain a first seal image, wherein each of the text blocks corresponds to one seal character;

[0034] A second processing module is used to segment the first seal image based on the position information corresponding to the text block to obtain a second seal image, wherein the second seal image includes a plurality of text block images, each of which corresponds to one of the text blocks;

[0035] A third processing module, used for inputting the second seal image into a text recognition model to obtain seal text information corresponding to the seal image output by the text recognition model;

[0036] Wherein, the text recognition model is obtained by training the target seal image set.

[0037] According to the seal text recognition device of the present application, a first seal image is obtained by identifying the text blocks corresponding to the seal text in the seal image, and the first seal image is segmented based on the position information corresponding to the text blocks to obtain a second seal image that can characterize the position distribution of the seal text in the seal. The second seal image including multiple text block images is input into a text recognition model, and the seal text information can be output according to the position distribution of the seal text through the text recognition model. There is no need to straighten the arc-shaped text on the seal image, which reduces the influence of text deformation on seal text recognition and can improve the accuracy of arc-shaped seal text recognition results.

[0038] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the seal text recognition method as described in the first aspect above when executing the computer program.

[0039] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the seal text recognition method as described in the first aspect above is implemented.

[0040] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the seal text recognition method as described in the first aspect above.

[0041] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0043] Figure 1 This is one of the flow charts of the seal character recognition method provided in the embodiment of the present application;

[0044] Figure 2 is a schematic diagram of segmenting a seal image provided by an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of a seal image provided in an embodiment of the present application;

[0046] Figure 4 This is the second flow chart of the seal character recognition method provided in the embodiment of the present application;

[0047] Figure 5 is a structural diagram of a text recognition model provided in an embodiment of the present application;

[0048] Figure 6 It is a structural schematic diagram of a seal character recognition device provided in an embodiment of the present application;

[0049] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0050] Reference numerals:

[0051] Input module 510, image slicing module 520, average pooling layer 530, global feature extraction layer 540,

[0052] Prediction layer 550. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0054] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0055] The seal text recognition method, seal text recognition device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below in conjunction with the accompanying drawings through specific embodiments and their application scenarios.

[0056] The seal character recognition method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0057] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0058] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0059] The seal text recognition method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the seal text recognition method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, wearable devices, etc. The seal text recognition method provided in the embodiment of the present application is described below using the electronic device as an example of the execution subject.

[0060] like Figure 1 As shown, the seal text recognition method includes: steps 110 to 140.

[0061] Step 110: Obtain a seal image to be processed, wherein the seal image includes a plurality of seal characters, and the contours of the plurality of seal characters are arc-shaped.

[0062] The seal image is an image including a seal, the seal may include text and patterns, etc., the edge of the seal may be arc-shaped, and the seal may be printed on documents such as certificates and contracts.

[0063] In this step, an image of the portion corresponding to the seal on the document can be acquired by an image acquisition device to obtain the seal image to be processed.

[0064] In this embodiment, the seal image includes multiple seal characters, which are the characters in the seal. The seal characters can be arranged at the edge of the seal. The contours of the multiple seal characters of the arc seal are arc-shaped, that is, the edges where the multiple seal characters are arranged are curved arcs.

[0065] For example, for an elliptical seal, the outlines of the plurality of seal characters may be a portion of an ellipse, and for a circular seal, the outlines of the plurality of seal characters may be a portion of a circle.

[0066] In this embodiment, the multiple seal characters in the seal image may represent meanings such as a person's name, an organization name, a date, etc.

[0067] Step 120 , identifying multiple text blocks of the seal image to obtain a first seal image, where each text block corresponds to a seal character.

[0068] The character block is a block area including seal characters, one character block includes one seal character, the sizes of each character block can be the same, and there can be no overlap between the character blocks.

[0069] The first seal image is an image of a text block corresponding to each seal character marked.

[0070] It should be noted that one character block may correspond to one coordinate value, and the coordinate value corresponding to the character block may represent the position of the seal characters in the character block in the first seal image.

[0071] In this step, the multiple character blocks of the seal image can be identified by methods such as character segmentation or optical character recognition, or the seal image can be input into a corresponding recognition model to identify the multiple character blocks of the seal image.

[0072] Step 130 : based on the position information corresponding to the text blocks, segment the first seal image to obtain a second seal image, wherein the second seal image includes a plurality of text block images, each of which corresponds to a text block.

[0073] The position information corresponding to the text block may represent the position of the seal characters in the text block in the first seal image, and may be expressed as the coordinate value corresponding to the text block, which may be the coordinate value of the center point of the text block or the coordinate value of the vertices of the text block.

[0074] It should be noted that the position information may also include the rotation angle information of the character block in the seal image.

[0075] In this step, the area corresponding to the character block can be obtained based on the position information corresponding to the character block, and the area corresponding to the character block in the first seal image can be segmented.

[0076] In actual implementation, the arc length between the starting chord and the ending chord corresponding to the seal in the first seal image can be calculated. The area between the starting chord and the ending chord corresponds to the area where the text block is located. The arc between the starting chord and the ending chord is divided equally to obtain the correspondence between the text block and each arc segment, thereby determining the position distribution of the seal text in the seal.

[0077] According to the position distribution of the seal characters in the seal, a second seal image can be constructed, and the second seal image is an image that can be input into a character recognition model.

[0078] The second seal image includes a plurality of text block images, which are images including text blocks. Each text block image includes a text block, and the position distribution of the text blocks in the second seal image and the position distribution of the text blocks in the first seal image may correspond to each other.

[0079] Step 140: input the second seal image into a text recognition model to obtain seal text information corresponding to the seal image output by the text recognition model.

[0080] The text recognition model is trained by a target seal image set. The target seal image set is an image set used to train the text recognition model. The target seal image set includes multiple target seal images. The target seal image can be a seal image with corresponding annotations.

[0081] The shapes, colors, textures, sizes, lighting, backgrounds, angles, etc. of the stamp images included in the target stamp image set may be different.

[0082] In this embodiment, the text recognition model is a model for recognizing seal text information corresponding to a seal image, and the seal text information may be information representing the meaning of the seal text.

[0083] In this step, the second seal image is input into the text recognition model, and after the text recognition model performs recognition prediction, inference and other operations on the seal characters in the second seal image, it outputs the seal character information corresponding to the seal image to be processed.

[0084] According to the seal text recognition method provided in the embodiment of the present application, a first seal image is obtained by identifying the text blocks corresponding to the seal text in the seal image, and the first seal image is segmented based on the position information corresponding to the text blocks to obtain a second seal image that can characterize the position distribution of the seal text in the seal. The second seal image including multiple text block images is input into a text recognition model, and the seal text information can be output according to the position distribution of the seal text through the text recognition model. There is no need to straighten the arc-shaped text on the seal image, which reduces the influence of text deformation on seal text recognition and can improve the accuracy of arc-shaped seal text recognition results.

[0085] In some embodiments, the text recognition model includes a text recognition module and a text reasoning module. Inputting the second seal image into the text recognition model to obtain seal text information corresponding to the seal image output by the text recognition model includes:

[0086] Input the second seal image into a text recognition module, and obtain a text probability matrix corresponding to a plurality of seal characters output by the text recognition module based on text information and position information corresponding to the character block image;

[0087] The text probability matrix is ​​input into the text reasoning module to obtain the seal text information output by the text reasoning module.

[0088] The text recognition module is a module for recognizing the content corresponding to the seal text of the second seal image, and the text recognition module outputs a text probability matrix.

[0089] The text probability matrix can represent the possibility of the text content corresponding to each seal character, and the elements in the text probability matrix can represent the probability of a seal character being recognized as a certain character.

[0090] In this embodiment, the second seal image is input into the text recognition module, and according to the text information and position information corresponding to each text block image in the second seal image, the second seal image is subjected to feature extraction, prediction and other operations to obtain a text probability matrix.

[0091] Among them, the text information is the information corresponding to the text features of the seal text in the text block image. For example, the text information can be the stroke features of the seal text, etc. The position information can represent the position distribution, rotation angle, etc. of the seal text in the text block image in the seal.

[0092] In this embodiment, the text reasoning module is a module for determining the text content corresponding to the seal text. The text reasoning module can calculate the joint probability of the text probability matrix, obtain the optimal text content path corresponding to the text probability matrix, and output the corresponding text string. The output text string corresponds to the characters corresponding to each seal text, and the output text string is the seal text information.

[0093] For example, Figure 3 As shown, the second seal image is input into the text recognition model, and the text reasoning module can output the text string abcdefg, that is, the seal text information is abcdefg.

[0094] In some embodiments, the text recognition module includes an average pooling layer 530, a global feature extraction layer 540 and a prediction layer 550, and the second seal image is input to the text recognition module. Based on the text information and position information corresponding to the text block image, a text probability matrix corresponding to a plurality of seal characters output by the text recognition module is obtained, including:

[0095] Inputting the second seal image to the average pooling layer 530 to obtain the second seal image with reduced dimension output by the average pooling layer 530;

[0096] Input the second seal image with reduced dimension into the global feature extraction layer 540, and obtain the global context information corresponding to the second seal image output by the global feature extraction layer 540 based on the text information and position information corresponding to the character block image;

[0097] The global context information is input into the prediction layer 550 to obtain the text probability matrix output by the prediction layer 550 .

[0098] The average pooling layer 530 is used to divide the second seal image into different regions, and calculate the average value of each region as the value corresponding to the region, thereby reducing the spatial size of the second seal image and reducing the dimension of the second seal image.

[0099] In this embodiment, the global feature extraction layer 540 is used to extract global features corresponding to the overall structure of the second seal image with reduced dimension. The global feature extraction layer 540 can generate global context information that can reflect the characteristics of the entire second seal image based on the integration of text information and position information of multiple text block images.

[0100] The global context information is information that can reflect the overall features of the second seal image, and the global context information may include the relationship between the text blocks and the overall layout of the text blocks.

[0101] In this embodiment, the prediction layer 550 is used to predict the probability of the text content corresponding to the seal characters in the second seal image based on the global context information.

[0102] In this embodiment, the second seal image is input to the average pooling layer 530, the average pooling layer 530 performs dimensionality reduction processing on the second seal image, and outputs the second seal image with reduced dimensionality. The second seal image with reduced dimensionality is input to the global feature extraction layer 540, and the global feature extraction layer 540 extracts the global context information of the second seal image. The global context information is input to the prediction layer 550, and the prediction layer 550 predicts the probability of the text content corresponding to the seal text according to the global context information, and outputs a text probability matrix.

[0103] In this embodiment, the global feature extraction layer 540 is arranged behind the average pooling layer 530. The global feature extraction layer 540 can process the pooled second seal image. The pooled second seal image can include higher-level abstract information, which can ensure that the global feature extraction layer 540 can better capture global context information and long-distance dependencies.

[0104] In some embodiments, the global feature extraction layer 540 includes a first feature extraction layer and a second feature extraction layer. The second seal image with reduced dimension is input to the global feature extraction layer 540. Based on the text information and position information corresponding to the text block image, the global context information corresponding to the second seal image output by the global feature extraction layer 540 is obtained, including:

[0105] Inputting the second seal image with reduced dimension into the first feature extraction layer, and obtaining first feature information output by the first feature extraction layer based on text information and position information corresponding to the character block image;

[0106] The first feature information is input into the second feature extraction layer to obtain the global context information output by the second feature extraction layer.

[0107] The structures and functions of the first feature extraction layer and the second feature extraction layer may be the same, and the first feature extraction layer and the second feature extraction layer may extract features of the second seal image.

[0108] In this embodiment, the first feature extraction layer can extract feature information corresponding to each character block image in the second seal image, that is, first feature information, based on text information and position information corresponding to the character block image.

[0109] In this embodiment, the second feature extraction layer may perform further feature integration and feature extraction according to the first feature information to obtain global context information.

[0110] In this embodiment, the second seal image with reduced dimension is input into the first feature extraction layer, which can extract the first feature information corresponding to the text block image. The second feature extraction layer can perform further feature integration and feature extraction based on the first feature information to obtain global context information.

[0111] In this embodiment, the global feature extraction layer 540 includes two layers, namely a first feature extraction layer and a second feature extraction layer. Reducing the number of feature extraction layers can improve the calculation speed and reduce the risk of overfitting.

[0112] In some embodiments, the prediction layer 550 is a fully convolutional layer or a fully connected layer.

[0113] Among them, the full convolution layer can extract image features through convolution operations, and restore the feature map size through upsampling technology to achieve classification prediction.

[0114] The fully connected layer can output the prediction result by multiplying the input data with the corresponding weight matrix and adding the corresponding bias, and then processing it through the activation function.

[0115] In this embodiment, the prediction layer 550 may include two fully convolutional layers, the filters in the fully convolutional layers share weights across the entire input, and each filter may learn a set of weights, thereby reducing model parameters.

[0116] In some embodiments, segmenting the first seal image based on the position information corresponding to the text block to obtain the second seal image includes:

[0117] Based on the position information corresponding to the character blocks, the first seal image is segmented to obtain a plurality of character block images;

[0118] The plurality of text block images are rendered onto the reference image to obtain a second seal image.

[0119] The reference image may be a blank image in a format corresponding to the input text recognition model.

[0120] In this embodiment, the arc length between the starting chord and the ending chord corresponding to the seal in the first seal image can be calculated, and the area between the starting chord and the ending chord corresponds to the area where the text block is located, such as Figure 2 As shown, the arc between the starting and ending chords is divided equally.

[0121] After equally dividing the arc between the starting chord and the ending chord, the correspondence between the text block and each arc segment can be obtained, the position distribution of the seal text in the seal can be determined, and the text block image can be obtained.

[0122] After obtaining the text block image, the reference image can be segmented in the same way as the first seal image. According to the correspondence between the text block and each arc segment, the text block corresponding to the text block image is rendered on the reference image using the Cartesian coordinate system text rendering method. Then, according to the tangent angle corresponding to the text block image in the first seal image, the text block image is rotated on the reference image to determine the corresponding position of the text block image in the reference image to obtain the second seal image.

[0123] like Figure 3 As shown, in the second seal image, letters such as a, b, and c represent seal characters, and the blocks surrounded by the boxes around the letters are the text blocks corresponding to the seal characters.

[0124] It can be understood that the position of a text block image in the first seal image corresponds to the position in the second seal image.

[0125] In some embodiments, identifying multiple text blocks of a stamp image includes:

[0126] Through the differentiable binarization network model, multiple text blocks of seal images are recognized.

[0127] The differentiable binarization network model may be an improved version of the differentiable binarization network model (Differentiable Binarization Network++, DBNet++).

[0128] DBNet++ has the advantages of strong real-time performance, high detection accuracy, and the ability to effectively process multi-directional text.

[0129] In some embodiments, inputting the second seal image into a text recognition model includes:

[0130] Performing mask processing on the area outside the corresponding area of ​​the character block image in the second seal image;

[0131] The second seal image after masking is input into the text recognition model.

[0132] In this embodiment, the area corresponding to the character block image in the second seal image can be kept at the original value, and the area outside the area corresponding to the character block image can be assigned a value of 0 to implement mask processing.

[0133] In this embodiment, before inputting the second seal image into the text recognition model, the second seal image is subjected to mask processing, which can improve the accuracy of text recognition by the text recognition model, reduce interference in the recognition process, and improve recognition efficiency.

[0134] A specific embodiment of the seal character recognition method is given below.

[0135] like Figure 4 As shown, step one, input an elliptical seal image, and detect the text blocks of the seal image to be processed through DBNet++. DBNet++ can output the endpoint coordinates of the polygonal area of ​​the seal text corresponding to the text block in the seal image, for example, it can be [(x1, y1), (x2, y2), (x3, y3), (x4, y4) ... ], where (x1, y1) can be the coordinates corresponding to the first text block in the seal image, (x2, y2) can be the coordinates corresponding to the second text block in the seal image, and so on.

[0136] The training set of DBNet++ can be the ICDAR2023 dataset including 5,000 seal images.

[0137] Step 2: First, calculate the arc length between the starting chord and the ending chord of the seal image through the integral approximation method, divide the arc length into 1000 equal parts, determine the position of each seal text, and then use the Cartesian coordinate system text rendering method to fill in the text block on the reference image. Then, calculate the tangent angle of the output point of the text block, rotate the text block, obtain the rotated text block, and synthesize the second seal image.

[0138] Step 3: Mask the non-text area of ​​the elliptical second seal image, set the fan-shaped text area to the original value, and assign 0 to the area outside the fan-shaped text area.

[0139] Step 4. The mask image with a size of H×W×3 is scaled to 200×128×3 by padding while maintaining the aspect ratio. H is the height of the mask image and W is the width of the mask image. The mask image is fed into the scene text recognition model (Scene Text Recognition with a SingleVisual Model, SVTR) of the improved single visual model, that is, the text recognition module performs end-to-end text recognition. After the final linear fully connected layer prediction, a B×N probability matrix is ​​obtained, where B is the number of characters and N is the length of the dictionary. The dictionary represents the text content corresponding to the seal text.

[0140] Among them, in the improved SVTR model, the four feature extraction layers, namely the global fusion module (Global Mixing Block), are reduced to two, and the Global Mixing Block module is moved behind the average pooling layer 530. In addition, two full convolutional layers are used as the prediction layer 550, which can improve the recognition speed of SVTR.

[0141] like Figure 5 As shown, the first module in the SVTR model is the input module 510, which inputs the RGB image corresponding to the masked second seal image and outputs a 200×128×3 array representing the second seal image.

[0142] The second module is an image slicing module 520 (Patch Embedding), which slices the 200×128×3 seal image into n image blocks of 16×16×3, which are called an image block (Patch).

[0143] The third module is the average pooling layer 530, which performs an average operation on the local area of ​​the image block to reduce the size of the corresponding feature map, thereby reducing the computational complexity and memory requirements.

[0144] The fourth module is the global feature extraction layer 540, which may include two global mixing modules to integrate features from different positions after the average pooling layer 530 to capture global context information.

[0145] The fifth module is the prediction layer 550, which can be a fully connected layer. The input is the feature matrix output by the global mixing module, and the output is a B×N probability matrix.

[0146] Step 5: Through the connected temporal classification (CTC) decoding model, according to the dictionary file, calculate the joint probability of the B×N probability matrix output by SVTR, output the optimal path, and the output result is a text string of length B, that is, the output seal text information.

[0147] Step 6: Calculate the connection temporal classification loss (CTCLoss) for the predicted character sequence and the true text label, perform gradient backpropagation, and update the network weights.

[0148] The CTCLoss function is:

[0149] Among them, L(S) is the loss function, which is used to calculate the continuous time series classification loss of S text at time T. S is a set, which can include all input sequences and label sequence pairs (x, z) at time T, that is, the predicted text and the real text set.

[0150] (x,z) represents the input sequence and the corresponding label sequence in the text, where x represents a character sequence of the input and z represents the label sequence of the corresponding real characters.

[0151] p(x,z) is a joint probability distribution function that characterizes the label prediction accuracy of the model. In CTCLoss, the probability corresponding to p(x,z) can be calculated through a neural network model.

[0152] Among them, the CTCLoss function is used to train the SVTR model without prior alignment of the input labels and output result lengths.

[0153] In this embodiment, the parameter size of the SVTR model is 36Mb, and the seal text recognition method has an accuracy of 85.3% in the ICDAR2023 test set.

[0154] In the related art, the color feature map, edge features and original grayscale features of the seal image are fused and input into a deep learning model for recognition. This method has a low accuracy rate for arc seal recognition.

[0155] The YOLOv5 face detection model (YOLOv5-face) in the face field is used for seal detection, and the generative adversarial network is used to eliminate the interference of non-seal text. Then the polar coordinates of the detected single text are transformed to obtain the horizontal text area, which is recognized by the classification network. This method detects the seal text and then corrects it for recognition. The effect of text recognition is subject to the influence of the straightening of arc-shaped text, and there are many steps and low efficiency.

[0156] An image encoder and a feature decoder for seal text recognition model are pre-trained, the seal image is input into the image encoder to obtain the image features of the seal, and the image features are input into the feature decoder to decode the text corresponding to the seal. This method uses the encoder-decoder method of the natural speech processing task to directly recognize the seal. The reasoning is time-consuming, the model is difficult to quantize or accelerate, and it is not convenient to deploy on mobile terminals or edge devices.

[0157] The seal text recognition method of the embodiment of the present application identifies the text blocks corresponding to the seal text in the seal image through an improved differentiable binarization network model to obtain a first seal image, and based on the position information corresponding to the text blocks, the first seal image is segmented to obtain multiple text block images, and according to the positional relationship of the text block images in the first seal image, the multiple text block images are rendered to a reference image to obtain a second seal image, the second seal image includes multiple text block images, and the text block images correspond to text information and position information.

[0158] After masking the second seal image, the second seal image is input into the text recognition model. First, the second seal image is input into the average pooling layer 530. The average pooling layer 530 performs dimensionality reduction processing on the second seal image and outputs the second seal image with reduced dimension. The second seal image with reduced dimension is then successively input into the first feature extraction layer and the second feature extraction layer of the global feature extraction layer 540 to perform feature extraction on the second seal image with reduced dimension to obtain global context information corresponding to the second seal image. The global context information is then input into two full convolutional layers for prediction to obtain a text probability matrix. The text probability matrix is ​​input into the text reasoning module, and the text reasoning module outputs the seal text information.

[0159] The seal text recognition method of the embodiment of the present application can recognize the text blocks on the seal image and mask the area outside the corresponding area of ​​the text block image. The text recognition model can output the seal text information according to the position distribution of the seal text. There is no need to straighten the arc-shaped text on the seal image, thereby reducing the influence of text deformation on seal text recognition and improving the accuracy of arc-shaped seal text recognition results.

[0160] In addition, the improved SVTR model is used to perform end-to-end text recognition on the seal image, the feature extraction layer is reduced to two, and the global feature extraction layer 540 is moved behind the average pooling layer 530, and two full convolutional layers are used as the prediction layer 550 to ensure the accuracy of recognition while improving the speed, which is convenient for deployment on mobile terminals or embedded devices.

[0161] The seal character recognition method provided in the embodiment of the present application can be executed by a seal character recognition device. In the embodiment of the present application, a seal character recognition device executing the seal character recognition method is taken as an example to illustrate the seal character recognition device provided in the embodiment of the present application.

[0162] The embodiment of the present application also provides a seal text recognition device.

[0163] like Figure 6 As shown, the seal character recognition device comprises:

[0164] An acquisition module 610 is used to acquire a seal image to be processed, wherein the seal image includes a plurality of seal characters, and the contours of the plurality of seal characters are arc-shaped;

[0165] The first processing module 620 is used to identify multiple text blocks of the seal image to obtain a first seal image, where each text block corresponds to a seal character;

[0166] The second processing module 630 is used to segment the first seal image based on the position information corresponding to the text block to obtain a second seal image, wherein the second seal image includes a plurality of text block images, each of which corresponds to a text block;

[0167] The third processing module 640 is used to input the second seal image into the text recognition model to obtain seal text information corresponding to the seal image output by the text recognition model;

[0168] Among them, the text recognition model is trained by the target seal image set.

[0169] According to the seal text recognition device provided in the embodiment of the present application, a first seal image is obtained by identifying the text blocks corresponding to the seal text in the seal image, and the first seal image is segmented based on the position information corresponding to the text blocks to obtain a second seal image that can characterize the position distribution of the seal text in the seal. The second seal image including multiple text block images is input into a text recognition model, and the seal text information can be output according to the position distribution of the seal text through the text recognition model. There is no need to straighten the arc-shaped text on the seal image, which reduces the influence of text deformation on seal text recognition and can improve the accuracy of arc-shaped seal text recognition results.

[0170] In some embodiments, the text recognition model includes a text recognition module and a text reasoning module, and the third processing module 640 is used to input the second seal image into the text recognition module, and obtain a text probability matrix corresponding to a plurality of seal characters output by the text recognition module based on the text information and position information corresponding to the character block image;

[0171] The text probability matrix is ​​input into the text reasoning module to obtain the seal text information output by the text reasoning module.

[0172] In some embodiments, the text recognition module includes an average pooling layer 530, a global feature extraction layer 540 and a prediction layer 550, and a third processing module 640 is used to input the second seal image into the average pooling layer 530 to obtain a second seal image with reduced dimension output by the average pooling layer 530;

[0173] Input the second seal image with reduced dimension into the global feature extraction layer 540, and obtain the global context information corresponding to the second seal image output by the global feature extraction layer 540 based on the text information and position information corresponding to the character block image;

[0174] The global context information is input into the prediction layer 550 to obtain the text probability matrix output by the prediction layer 550 .

[0175] In some embodiments, the global feature extraction layer 540 includes a first feature extraction layer and a second feature extraction layer, and the third processing module 640 is used to input the second seal image with reduced dimension into the first feature extraction layer, and obtain the first feature information output by the first feature extraction layer based on the text information and position information corresponding to the text block image;

[0176] The first feature information is input into the second feature extraction layer to obtain the global context information output by the second feature extraction layer.

[0177] In some embodiments, the prediction layer 550 is a fully convolutional layer or a fully connected layer.

[0178] In some embodiments, the second processing module 630 is used to segment the first seal image based on the position information corresponding to the text block to obtain a plurality of text block images;

[0179] The plurality of text block images are rendered onto the reference image to obtain a second seal image.

[0180] In some embodiments, the first processing module 620 is used to identify multiple text blocks of the seal image through a differentiable binarization network model.

[0181] In some embodiments, the third processing module 640 is used to perform mask processing on the area outside the corresponding area of ​​the character block image in the second seal image;

[0182] The second seal image after masking is input into the text recognition model.

[0183] The seal character recognition device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0184] The seal character recognition device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0185] The seal character recognition device provided in the embodiment of the present application can realize Figure 1 and Figure 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0186] In some embodiments, Figure 7 As shown, the embodiment of the present application also provides an electronic device 700, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, the various processes of the above-mentioned seal text recognition method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0187] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0188] The embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned seal text recognition method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0189] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0190] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned seal text recognition method when executed by a processor.

[0191] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0192] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned seal text recognition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0193] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0194] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0196] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0197] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0198] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A seal character recognition method, characterized in that: include: Acquire a seal image to be processed, wherein the seal image includes a plurality of seal characters, and the contours of the plurality of seal characters are arc-shaped; Recognize multiple text blocks of the seal image to obtain a first seal image, each of the text blocks corresponding to one seal character; Based on the position information corresponding to the text block, the first seal image is segmented to obtain a second seal image, wherein the second seal image includes a plurality of text block images, each of which corresponds to one of the text blocks; Inputting the second seal image into a text recognition model to obtain seal text information corresponding to the seal image output by the text recognition model; Wherein, the text recognition model is obtained by training the target seal image set.

2. The seal character recognition method according to claim 1, characterized in that: The text recognition model includes a text recognition module and a text reasoning module. The step of inputting the second seal image into the text recognition model to obtain seal text information corresponding to the seal image output by the text recognition model includes: Inputting the second seal image into the text recognition module, and obtaining a text probability matrix corresponding to a plurality of seal characters output by the text recognition module based on the text information and position information corresponding to the character block image; The text probability matrix is ​​input into a text reasoning module to obtain the seal text information output by the text reasoning module.

3. The seal character recognition method according to claim 2, characterized in that: The text recognition module includes an average pooling layer, a global feature extraction layer and a prediction layer. The second seal image is input into the text recognition module, and based on the text information and position information corresponding to the text block image, a text probability matrix corresponding to a plurality of seal characters output by the text recognition module is obtained, including: Inputting the second seal image to the average pooling layer to obtain the second seal image with reduced dimension output by the average pooling layer; Inputting the second seal image with reduced dimension into the global feature extraction layer, and obtaining global context information corresponding to the second seal image output by the global feature extraction layer based on text information and position information corresponding to the character block image; The global context information is input into the prediction layer to obtain the text probability matrix output by the prediction layer.

4. The seal character recognition method according to claim 3, characterized in that: The global feature extraction layer includes a first feature extraction layer and a second feature extraction layer, the second seal image with reduced dimension is input to the global feature extraction layer, and based on the text information and position information corresponding to the text block image, global context information corresponding to the second seal image output by the global feature extraction layer is obtained, including: Inputting the second seal image with reduced dimension into the first feature extraction layer, and obtaining first feature information output by the first feature extraction layer based on text information and position information corresponding to the character block image; The first feature information is input into the second feature extraction layer to obtain the global context information output by the second feature extraction layer.

5. The seal character recognition method according to claim 3, characterized in that: The prediction layer is a fully convolutional layer or a fully connected layer.

6. The seal character recognition method according to any one of claims 1 to 5, characterized in that: The step of segmenting the first seal image based on the position information corresponding to the text block to obtain the second seal image includes: Based on the position information corresponding to the character block, the first seal image is segmented to obtain a plurality of the character block images; The plurality of character block images are rendered onto a reference image to obtain the second seal image.

7. The seal character recognition method according to any one of claims 1 to 5, characterized in that: The multiple text blocks of the seal image are identified, including: A plurality of text blocks of the seal image are identified through a differentiable binarization network model.

8. The seal character recognition method according to any one of claims 1 to 5, characterized in that: The step of inputting the second seal image into a text recognition model comprises: Performing mask processing on the area outside the area corresponding to the text block image in the second seal image; The second seal image after masking is input into the text recognition model.

9. A seal character recognition device, characterized in that: include: An acquisition module, used for acquiring a seal image to be processed, wherein the seal image includes a plurality of seal characters, and the contours of the plurality of seal characters are arc-shaped; A first processing module, used for identifying a plurality of text blocks of the seal image to obtain a first seal image, wherein each of the text blocks corresponds to one seal character; A second processing module is used to segment the first seal image based on the position information corresponding to the text block to obtain a second seal image, wherein the second seal image includes a plurality of text block images, each of which corresponds to one of the text blocks; A third processing module, used for inputting the second seal image into a text recognition model to obtain seal text information corresponding to the seal image output by the text recognition model; Wherein, the text recognition model is obtained by training the target seal image set.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the seal character recognition method as described in any one of claims 1-8 is implemented.

Images