Image recognition method and device, electronic equipment and computer readable medium

Abstract

Embodiments of the present disclosure disclose image recognition methods, apparatuses, electronic devices, and computer readable media. A specific implementation of the method includes: obtaining an image to be recognized; performing image preprocessing on the image to be recognized; performing non-handwritten text recognition on the preprocessed image; determining a handwritten text region of interest and a seal region of interest in the image to be recognized according to first text information; performing handwritten text recognition on the handwritten text region of interest in the image to be recognized; performing seal recognition on the seal region of interest in the image to be recognized; performing content verification on the first text information, second text information, and seal information; in response to the content verification passing, performing content extraction on the first text information, second text information, and seal information; and performing data storage on a data record and the image to be recognized. The implementation achieves effective and accurate content recognition for content contained in an image.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular, to an image recognition method and device, an electronic device, and a computer readable medium. BACKGROUND

[0002] With the development of computer-related technology, data electronicization has become a mainstream trend. However, there are a large number of text files (for example, authorization letters) that are stored in the form of images. How to effectively and accurately identify the content of a large number of images becomes a problem to be solved.

[0003] The above information disclosed in this BACKGROUND section is only for the purpose of enhancing the understanding of the background of the present inventive concepts, and therefore, it can contain information that does not form the prior art that is already known in this field to those skilled workers. SUMMARY

[0004] The summary of the present disclosure is used to introduce the concepts in a brief manner, which will be described in detail in the following detailed description. The summary of the present disclosure is not intended to identify key or essential features of the claimed technology nor is it intended to be used to limit the scope of the claimed technology.

[0005] Some embodiments of the present disclosure provide an image recognition method, device, electronic device, and computer readable medium to solve one or more of the technical problems mentioned in the background section.

[0006] In a first aspect, some embodiments of the present disclosure provide an image recognition method, comprising: obtaining an image to be recognized, wherein the image to be recognized is an image containing handwritten text, non-handwritten text, and a seal image; performing image preprocessing on the image to be recognized to generate a preprocessed image; performing non-handwritten text recognition on the preprocessed image to generate first text information; determining a handwritten text region of interest and a seal region of interest in the image to be recognized according to the first text information; performing handwritten text recognition on the handwritten text region of interest in the image to be recognized to generate second text information; performing seal recognition on the seal region of interest in the image to be recognized to generate seal information; performing content verification on the first text information, the second text information, and the seal information; in response to the content verification passing, performing content extraction on the first text information, the second text information, and the seal information to generate a data record for the image to be recognized; and performing data storage on the data record and the image to be recognized.

[0007] Optionally, the image preprocessing on the to-be-recognized image to generate a preprocessed image comprises: distortion correction on the to-be-recognized image to generate a distortion-corrected image; image size adjustment on the distortion-corrected image to obtain an image size-adjusted image; image enhancement on the image size-adjusted image to generate an image-enhanced image; character region of interest recognition on the image-enhanced image to determine a character region of interest set; and character shadow removal on the image-enhanced image according to the character region of interest set to obtain the preprocessed image.

[0008] Optionally, the content verification on the first text information, the second text information and the seal information comprises: tampering verification according to the first text information to generate a first verification result; format verification according to the second text information to generate a second verification result; seal anti-forgery verification according to the seal information to generate a third verification result; and generation of a content verification result according to the first verification result, the second verification result and the third verification result.

[0009] Optionally, the first text information comprises a first text description information set, and each first text description information in the first text description information set comprises character description information and text semantic features; and the non-handwritten text recognition on the preprocessed image to generate the first text information comprises: filtering, from the character region of interest set, character regions of interest satisfying a first filtering condition as first character regions of interest according to the region positions corresponding to the character regions of interest to obtain a first character region of interest set; filtering, from the character region of interest set, character regions of interest satisfying a second filtering condition as second character regions of interest according to the region positions corresponding to the character regions of interest to obtain a second character region of interest set; determining a first semantic correlation degree according to the first character region of interest set, wherein the first semantic correlation degree represents a character semantic correlation degree of a character set corresponding to the first character region of interest set; determining a second semantic correlation degree according to the second character region of interest set, wherein the second semantic correlation degree represents a character semantic correlation degree of a character set corresponding to the second character region of interest set; determining a text recognition direction as a horizontal recognition direction in response to the first semantic correlation degree being greater than a preset semantic correlation degree threshold and the first semantic correlation degree being greater than the second semantic correlation degree; and determining the text recognition direction as a vertical recognition direction in response to the second semantic correlation degree being greater than the preset semantic correlation degree threshold and the second semantic correlation degree being greater than the first semantic correlation degree.

[0010] Optionally, the non-handwritten text recognition on the pre-processed image to generate the first text information further includes: dividing the set of character regions of interest into a set of character region of interest groups according to the text recognition direction, wherein the character regions of interest in a character region of interest group are located in the same text line; and for each character region of interest group in the set of character region of interest groups, performing the following processing steps: determining, by using a pre-trained character feature extraction model, character features corresponding to characters in each character region of interest in the character region of interest group to obtain a character feature group, wherein the character features in the character feature group include: a font feature, a font size feature, a font color feature, and a font meaning feature; and determining, by using a character recognition model and a text semantic extraction model, first text description information corresponding to the character region of interest group in the set of first text description information according to the character feature group.

[0011] In a second aspect, some embodiments of the present disclosure provide an image recognition device, the device comprising: an acquisition unit configured to acquire an image to be recognized, wherein the image to be recognized is an image containing handwritten text, non-handwritten text, and a seal image; an image pre-processing unit configured to perform image pre-processing on the image to be recognized to generate a pre-processed image; a non-handwritten text recognition unit configured to perform non-handwritten text recognition on the pre-processed image to generate first text information; a determination unit configured to determine a handwritten text region of interest and a seal region of interest in the image to be recognized according to the first text information; a handwritten text recognition unit configured to perform handwritten text recognition on the handwritten text region of interest in the image to be recognized to generate second text information; a seal recognition unit configured to perform seal recognition on the seal region of interest in the image to be recognized to generate seal information; a content verification unit configured to perform content verification on the first text information, the second text information, and the seal information; a content extraction unit configured to, in response to the content verification passing, perform content extraction on the first text information, the second text information, and the seal information to generate a data record for the image to be recognized; and a data storage unit configured to store the data record and the image to be recognized.

[0012] In a third aspect, some embodiments of the present disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementations of the first aspect.

[0013] In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0014] The above various embodiments of the present disclosure have the following beneficial effects: through the image processing method of some embodiments of the present disclosure, effective and accurate content recognition for content contained in an image is achieved. Specifically, first, an image to be recognized is obtained, wherein the image to be recognized is an image containing handwritten text, non-handwritten text and seal image. That is, the image to be recognized contains various to-be-recognized contents such as handwritten text, non-handwritten text and seal image. Then, image preprocessing is performed on the image to be recognized to generate a preprocessed image. Through image preprocessing, the image quality is improved. Further, non-handwritten text recognition is performed on the preprocessed image to generate first text information. In this way, the non-handwritten text contained in the preprocessed image is automatically extracted. In addition, according to the first text information, a handwritten text region of interest and a seal region of interest in the image to be recognized are determined. In this way, according to the first text information, the region where the handwritten text and the seal image may exist is located. In addition, handwritten text recognition is performed on the handwritten text region of interest in the image to be recognized to generate second text information. Then, seal recognition is performed on the seal region of interest in the image to be recognized to generate seal information. In this way, automatic recognition of handwritten text and seal is achieved. Next, content verification is performed on the first text information, the second text information and the seal information. In this way, the effectiveness of the content is verified. Further, in response to the content verification passing, content extraction is performed on the first text information, the second text information and the seal information to generate a data record for the image to be recognized. In this way, attribute extraction is performed on the first text information, the second text information and the seal information to obtain the corresponding data record for data storage. Finally, data storage is performed on the data record and the image to be recognized. In this way, effective and accurate content recognition for content contained in an image is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0015] The above and other features, advantages and aspects of embodiments of the present disclosure will become more apparent upon consideration of the following detailed description, taken in conjunction with the accompanying drawings. In the drawings like or similar elements are referred to by like reference numbers throughout the various figures. It should be understood that the drawings are schematic and elements and features are not necessarily to scale.

[0016] Figure 1 is a flowchart of some embodiments of the image recognition method according to the present disclosure;

[0017] Figure 2is a schematic diagram of a to-be-recognized image of some embodiments of the image recognition method of the present disclosure;

[0018] Figure 3 is another schematic diagram of a to-be-recognized image of some embodiments of the image recognition method of the present disclosure;

[0019] Figure 4 is a position schematic diagram of a first character region of interest set and a second character region of interest set of some embodiments of the image recognition method of the present disclosure;

[0020] Figure 5 is a generation process schematic diagram of first text information of some embodiments of the image recognition method of the present disclosure;

[0021] Figure 6 is a position schematic diagram of a text barrier in a pre-processed image of some embodiments of the image recognition method of the present disclosure;

[0022] Figure 7 is a structural schematic diagram of some embodiments of the image recognition apparatus according to the present disclosure;

[0023] Figure 8 is a structural schematic diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided so as to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure.

[0025] It should also be noted that, for ease of description, only the parts related to the present application are shown in the drawings. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0026] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0027] It should be noted that the adjectives "one", "multiple" mentioned in the present disclosure are illustrative and not limiting, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0028] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0029] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0030] Reference Figure 1 , shows the flow 100 of some embodiments of the image recognition method according to the present disclosure. The image recognition method comprises the following steps:

[0031] Step 101, obtaining an image to be recognized.

[0032] In some embodiments, the execution subject (for example, a computing device) of the image recognition method can obtain the image to be recognized through wired connection or wireless connection. Wherein, the above-mentioned image to be recognized is an image containing handwritten text, non-handwritten text and seal image. In practice, the above-mentioned execution subject can obtain the image to be recognized uploaded by the user from the user end in real time.

[0033] As an example, see Figure 2 illustrated one schematic diagram of the image to be recognized, and Figure 3 illustrated another schematic diagram of the image to be recognized, wherein, Figure 2 and Figure 3 the solid line box in Figure 2 and Figure 3 is handwritten text, the dashed line box is seal image, and the outside of the dashed line box and the solid line box is non-handwritten text. It should be noted that,

[0034] It should be noted that the above-mentioned wireless connection mode can include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection modes.

[0035] It should be noted that the above-mentioned computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed in the above-mentioned hardware devices listed. It can be implemented as, for example, multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made here. It should be understood that the number of computing devices can have any number according to the needs of implementation.

[0036] Step 102, image pre-processing is performed on the image to be identified to generate a pre-processed image.

[0037] In some embodiments, the execution subject described above can perform image pre-processing on the image to be identified to generate a pre-processed image. In practice, the execution subject described above can perform image cropping on the image to be identified to obtain the pre-processed image.

[0038] In some optional implementations of some embodiments, the execution subject described above performs image pre-processing on the image to be identified to generate a pre-processed image, which can include the following steps:

[0039] First, distortion correction is performed on the image to be identified to generate a distortion-corrected image.

[0040] In practice, first, the execution subject described above can collect feature points from the image to be identified to obtain a feature point set. Then, by comparing the difference between the feature points in the feature point set and the feature points in the actual world, the distortion parameters are determined. Then, according to the distortion parameters, the image to be identified is corrected to obtain the distortion-corrected image.

[0041] Second, the image size of the distortion-corrected image is adjusted to obtain an image size-adjusted image.

[0042] In practice, the execution subject described above can adjust the image size of the distortion-corrected image to a preset size through stretching, cropping and other operations to obtain the image size-adjusted image. Specifically, when the image size of the distortion-corrected image is adjusted through cropping, the distortion-corrected image is cropped with the boundary points of the blank area around the image as the boundary to obtain the image size-adjusted image.

[0043] Third, the image size-adjusted image is enhanced to generate an image-enhanced image.

[0044] In practice, the execution subject described above can enhance the image size-adjusted image through a Laplacian-based image enhancement method to generate an image-enhanced image.

[0045] Fourth, the character region of interest of the image-enhanced image is identified to determine a set of character regions of interest.

[0046] In practice, the execution subject can identify the character region of interest in the image enhanced image through a character region of interest identification model to determine a character region of interest set. The character region of interest identification model includes a first downsampling network, a second downsampling network, a feature fusion layer, and a detection head. The first downsampling network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. The output of the first convolutional layer is the input of the second convolutional layer. The output of the second convolutional layer is the input of the third convolutional layer. The output of the third convolutional layer is the input of the fourth convolutional layer. The output of the fourth convolutional layer is the input of the fifth convolutional layer. The output size of the first convolutional layer is 1 / 2 of the image size of the image enhanced image. The output size of the second convolutional layer is 1 / 4 of the image size of the image enhanced image. The output size of the third convolutional layer is 1 / 4 of the image size of the image enhanced image. The output size of the third convolutional layer is 1 / 8 of the image size of the image enhanced image. The output size of the fourth convolutional layer is 1 / 16 of the image size of the image enhanced image. The output size of the fifth convolutional layer is 1 / 32 of the image size of the image enhanced image. The second downsampling network includes a sixth convolutional layer, a seventh convolutional layer, and an eighth convolutional layer. The input of the sixth convolutional layer is the superposition of the output of the second convolutional layer and the output of the seventh convolutional layer. The input of the seventh convolutional layer is the superposition of the output of the third convolutional layer and the output of the eighth convolutional layer. The input of the eighth convolutional layer is the superposition of the output of the fourth convolutional layer and the output of the fifth convolutional layer. The feature fusion layer is used to superimpose the output of the fifth convolutional layer, the output of the sixth convolutional layer, the input of the seventh convolutional layer, and the output of the eighth convolutional layer. Specifically, before superposition, the feature fusion layer up-samples the output of the fifth convolutional layer by 8 times, up-samples the output of the seventh convolutional layer by 2 times, and up-samples the output of the eighth convolutional layer by 4 times.

[0047] In the fifth step, the character shadow removal is performed on the image enhanced image according to the character region of interest set to obtain the preprocessed image.

[0048] In practice, the execution subject can perform binaryzation processing on the pixels in the character region of interest by setting a pixel threshold to eliminate the shadow part existing in the image acquisition process caused by shaking.

[0049] In step 103, non-handwritten text recognition is performed on the preprocessed image to generate first text information.

[0050] In some embodiments, the above-mentioned execution entity may perform non-handwritten text recognition on the preprocessed image to generate first text information. For example, the above-mentioned execution entity may perform non-handwritten text recognition on the preprocessed image through general OCR (Optical Character Recognition) to generate first text information.

[0051] Optionally, the first text information includes: a set of first text description information. The first text description information in the above set of first text description information includes: a group of character description information and text semantic features. The first text description information represents the text description at the sentence granularity. The character description information represents the description of characters. For example, the character description information may include: characters. The text semantic feature represents the sentence semantics of the sentence corresponding to the first text description information.

[0052] In some optional implementation manners of some embodiments, the above-mentioned execution entity performing non-handwritten text recognition on the preprocessed image to generate first text information may include the following steps:

[0053] First step, according to the region position corresponding to the character region of interest, screen out the character regions of interest that meet the first screening condition from the above set of character regions of interest as the first character regions of interest, and obtain a set of first character regions of interest.

[0054] Among them, the first character region of interest in the set of first character regions of interest is the character region of interest whose corresponding region position is the row where the middle region position is located.

[0055] Second step, according to the region position corresponding to the character region of interest, screen out the character regions of interest that meet the second screening condition from the above set of character regions of interest as the second character regions of interest, and obtain a set of second character regions of interest.

[0056] Among them, the second character region of interest in the set of second character regions of interest is the character region of interest whose corresponding region position is the column where the middle region position is located.

[0057] As an example, as Figure 4 shown in the schematic diagram of the positions of the set of first character regions of interest and the set of second character regions of interest, where the set of first character regions of interest 401 includes 8 second character regions of interest for "show" and 8 first character regions of interest for "example". The set of second character regions of interest 402 includes 3 second character regions of interest for "example".

[0058] Third step, determine the first semantic association degree according to the above set of first character regions of interest.

[0059] The first semantic correlation degree represents a character semantic correlation degree of a character set corresponding to the first character region of interest set.

[0060] In practice, the execution subject determines the first semantic correlation degree according to the first character region of interest set through a semantic correlation degree prediction model. The semantic correlation degree prediction model includes a recurrent neural network model and a predictor. The predictor is configured to output the semantic correlation degree of the character set corresponding to the first character region of interest set.

[0061] The second semantic correlation degree is determined according to the second character region of interest set.

[0062] In practice, the execution subject determines the second semantic correlation degree according to the second character region of interest set through a semantic correlation degree prediction model. The second semantic correlation degree represents a character semantic correlation degree of a character set corresponding to the second character region of interest set.

[0063] As an example, further referring to Figure 4 , the first character region of interest set 401 contains 8 second character regions of interest for "example" and 8 first character regions of interest for "example". There is a certain semantic relationship between them, so the corresponding first semantic correlation degree is greater than 0. The second character region of interest set 402 contains 3 second character regions of interest for "example", and there is no semantic relationship between the corresponding characters "example", so the corresponding second semantic correlation degree can be 0. Corresponding Figure 2 and Figure 3 The arrangement of the characters, when the characters are horizontally arranged, the first semantic correlation degree is greater than the second semantic correlation degree. When the characters are vertically arranged, the second semantic correlation degree is greater than the first semantic correlation degree.

[0064] Fourthly, in response to the first semantic correlation degree being greater than a preset semantic correlation degree threshold and the first semantic correlation degree being greater than the second semantic correlation degree, a horizontal recognition direction is determined as the text recognition direction.

[0065] Fifthly, in response to the second semantic correlation degree being greater than the preset semantic correlation degree threshold and the second semantic correlation degree being greater than the first semantic correlation degree, a vertical recognition direction is determined as the text recognition direction.

[0066] Sixthly, the character region of interest set is divided into a character region of interest group set according to the text recognition direction.

[0067] The character regions of interest in the character region of interest group are located in the same text line.

[0068] Seventhly, for each character region of interest group in the character region of interest group set, the following processing steps are performed:

[0069] The first sub-step involves using a pre-trained character feature extraction model to determine the character features corresponding to each character within the aforementioned character region of interest group, thus obtaining the character feature group.

[0070] The character features in the above character feature group include: character shape features, character size features, character color features, and character meaning features.

[0071] In practice, see Figure 5 The diagram illustrating the generation process of the first text information shows that the character feature extraction model 502 includes: a shallow character feature extraction model 5021, a character shape feature extraction model 5022, a character size feature extraction model 5023, a character color feature extraction model 5024, and a character meaning feature extraction model 5025. The shallow character feature extraction model 5021 uses a residual neural network model as its backbone. The character shape feature extraction model 5022, the character size feature extraction model 5023, the character color feature extraction model 5024, and the character meaning feature extraction model 5025 all have the same model structure, employing a convolutional neural network model with five convolutional layers connected in series.

[0072] The second sub-step involves using a character recognition model and a text semantic extraction model to determine the first text description information corresponding to the character region of interest group in the first text description information set, based on the aforementioned character feature groups.

[0073] In practice, further reference Figure 5The character region of interest group 501 can include a character region of interest corresponding to the character "zhi", a character region of interest corresponding to the character "xing", a character region of interest corresponding to the character "shi", a character region of interest corresponding to the character "wu", a character region of interest corresponding to the character "he", a character region of interest corresponding to the character "hui", and a character region of interest corresponding to the character "ren". Specifically, for each character region of interest, the character feature extraction model 502 performs corresponding character feature extraction, and inputs the extracted character features into the character recognition model 503. Specifically, the character recognition model includes a character recognizer, which is configured to determine the character corresponding to the character features and the confidence. The recognized character, character type (non-handwritten character or handwritten character), and character position (indicating the position of the character in the image to be recognized) are taken as the character description information 505. In addition, the character feature extraction model 502 inputs the character features corresponding to the character region of interest corresponding to the character "zhi", the character region of interest corresponding to the character "xing", the character region of interest corresponding to the character "shi", the character region of interest corresponding to the character "wu", the character region of interest corresponding to the character "he", the character region of interest corresponding to the character "hui", and the character region of interest corresponding to the character "ren" into the text semantic extraction model 504, respectively, to obtain corresponding text semantic features 506. In practice, the text semantic extraction model uses a recurrent neural network model.

[0074] In step 104, the handwritten text region of interest and the seal region of interest in the image to be recognized are determined according to the first text information.

[0075] In some embodiments, the above execution subject can determine the handwritten text region of interest and the seal region of interest in the image to be recognized according to the first text information. The handwritten text region of interest is a region that may contain handwritten text. The seal region of interest is a region that may contain a seal.

[0076] In some optional implementations of some embodiments, the above execution subject can determine the handwritten text region of interest and the seal region of interest in the image to be recognized according to the first text information, which can include the following steps:

[0077] First, determine the text raster according to the above first text information.

[0078] In practice, the above execution subject can take the region in the preprocessed image corresponding to the position of the character contained in the first text information as the text raster.

[0079] As an example, the position of the text raster in the preprocessed image is shown in the following figure: Figure 6

[0080] ​Secondly, a difference set image is obtained by determining the difference set between the text fence and the preprocessed image.

[0081] In practice, the difference set image = preprocessed image - preprocessed image containing text fence.

[0082] Thirdly, the difference set image is divided into a set of image blocks.

[0083] In the set of image blocks, the size of each image block is consistent.

[0084] Fourthly, for each image block in the set of image blocks, the handwritten text confidence and the seal confidence corresponding to the image block are determined according to the image block and the first text information.

[0085] In practice, the execution subject can determine the handwritten text confidence and the seal confidence corresponding to the image block by using binary classifier A and binary classifier B, respectively. Specifically, binary classifier A is used to classify whether the image block contains handwritten text and the corresponding confidence. Binary classifier B is used to classify whether the image block contains a seal and the corresponding confidence.

[0086] Fifthly, the set of image blocks is filtered according to the handwritten text confidence and the seal confidence corresponding to the image blocks, to obtain a first set of image blocks and a second set of image blocks.

[0087] In practice, the execution subject can set a handwritten text confidence threshold and a seal confidence threshold, respectively, and filter the set of image blocks according to the handwritten text confidence and the seal confidence corresponding to the image blocks, to obtain a first set of image blocks and a second set of image blocks. The first image block in the first set of image blocks is an image block containing handwritten text. The second image block in the second set of image blocks is an image block containing a seal.

[0088] Sixthly, the first set of image blocks is spliced to obtain the handwritten text region of interest.

[0089] Seventhly, the second set of image blocks is spliced to obtain the seal region of interest.

[0090] As an invention point, the above "in some optional implementations of some embodiments" specifically determines the text barrier, that is, the non-handwritten text corresponding area in the preprocessed image is shielded, specifically represented as pixel value is 0, which avoids the non-handwritten text area from participating in subsequent feature calculation, and greatly reduces the feature calculation amount. In addition, by dividing the image block, the recognition granularity is constrained, avoiding the problems of large data calculation amount due to too small granularity and inaccurate recognition due to too large granularity. Thus, the handwritten text region of interest and the seal region of interest are effectively and accurately recognized.

[0091] Step 105, performing handwritten text recognition on the handwritten text region of interest in the to-be-recognized image to generate second text information.

[0092] In some embodiments, the above execution subject can perform handwritten text recognition on the handwritten text region of interest in the to-be-recognized image to generate second text information. Wherein, the second text description information set, the second text description information in the above second text description information set includes: character description information group and text semantic feature. In practice, the above execution subject can generate second text information through shallow character feature extraction model 5021, glyph feature extraction model 5022 and character recognition model 503, and text semantic feature extraction model 504. Specifically, since there are differences between handwritten characters and non-handwritten characters, before performing handwritten text recognition, the shallow character feature extraction model 5021, the glyph feature extraction model 5022 and the character recognition model 503, and the text semantic feature extraction model 504 can be trained through the training sample corresponding to the non-handwritten text first, and after the training is completed, a small amount of training samples corresponding to the handwritten text are trained through the way of transfer learning to ensure the accuracy of the model in non-handwritten text recognition.

[0093] Step 106, performing seal recognition on the seal region of interest in the to-be-recognized image to generate seal information.

[0094] In some embodiments, the above execution subject can perform seal recognition on the seal region of interest in the to-be-recognized image to generate seal information.

[0095] In some optional implementations of some embodiments, the above execution subject performs seal recognition on the seal region of interest in the to-be-recognized image to generate seal information, which can include the following steps:

[0096] First, according to the above seal region of interest, determine the seal boundary, seal text direction and seal type.

[0097] Wherein, the seal type represents the boundary type corresponding to the seal boundary.

[0098] In practice, first, the execution subject can determine the seal boundary according to the seal region of interest by a Hough transform algorithm. Then, the seal type is determined according to the graphic type of the seal boundary. For example, if the seal boundary is circular or elliptical, the corresponding seal type can be a public seal type. For another example, if the seal boundary is rectangular or square, the corresponding seal type can be a private seal type. In addition, the execution subject can extract the characters in the seal boundary by the shallow character feature extraction model 5021, and fit a direction line corresponding to the seal text direction according to the position of the extracted characters in the seal region of interest.

[0099] Secondly, the seal boundary is divided into the first divided seal boundary and the second divided seal boundary according to the seal type and the seal text direction.

[0100] The first divided seal boundary and the second divided seal boundary are axis-symmetric closed boundaries.

[0101] Thirdly, the seal region of interest is regionally divided according to the first divided seal boundary to obtain the first divided seal region of interest.

[0102] The first divided seal region of interest is a region surrounded by the first divided seal boundary.

[0103] Fourthly, the seal region of interest is regionally divided according to the second divided seal boundary to obtain the second divided seal region of interest.

[0104] The second divided seal region of interest is a region surrounded by the second divided seal boundary.

[0105] Fifthly, the first divided seal region of interest is subjected to seal text recognition to obtain the first seal text information.

[0106] The first seal text information includes seal text content and seal text position. Specifically, the execution subject can perform character recognition on the first divided seal region of interest by a character recognition model to obtain the seal text content, and obtain the seal text position according to the position of the characters contained in the seal text content in the first divided seal region of interest.

[0107] Sixthly, the second divided seal region of interest is subjected to seal text recognition according to the seal text position included in the first seal text information to obtain the second seal text information.

[0108] The second seal text information includes seal text content and seal text position. Specifically, the execution subject can perform character recognition on the second segmented seal region of interest by using a character recognition model to obtain the seal text content, and determine the seal text position according to the position of the characters contained in the second segmented seal region of interest. Specifically, when recognizing the second segmented region of interest, the corresponding position of the first segmented seal region of interest containing text can be directly extracted as a feature to reduce the amount of feature processing.

[0109] In the seventh step, the first seal text information and the second seal text information are spliced to obtain the seal information.

[0110] In practice, the first seal text information and the second seal text information are spliced to obtain the seal information.

[0111] The above-mentioned "in some optional implementations of some embodiments" is an invention point of the present disclosure. Specifically, since the seal shape often has axial symmetry, the present disclosure achieves symmetrical division of the seal region of interest by determining the seal boundary. Meanwhile, considering that the seal may correspond to multiple symmetry axes, in order to perform axial symmetrical division on the contents (such as text, etc.) contained in the seal, the seal text direction is combined with the seal boundary to further perform axial symmetrical division. Then, the first segmented region of interest is recognized to recognize the part of the contents contained in the seal. Meanwhile, due to the axial symmetry, the symmetrical position in the second segmented region of interest can be directly recognized to reduce the feature processing amount and greatly improve the recognition efficiency.

[0112] In step 107, the first text information, the second text information, and the seal information are verified for content.

[0113] In some embodiments, the execution subject can verify the content of the first text information, the second text information, and the seal information in various ways.

[0114] In some optional implementations of some embodiments, the execution subject can verify the content of the first text information, the second text information, and the seal information, which can include the following steps:

[0115] In the first step, tampering verification is performed according to the first text information to generate a first verification result.

[0116] In practice, the execution subject can determine whether the character of the first text information has character smudging or tampering according to the glyph features of the character of the first text information and a text smudging classifier. The text smudging classifier is a binary classifier with the output of the character feature extraction model as input. The first verification result represents whether the first text information contains smudged or tampered characters.

[0117] Second, format verification is performed on the second text information to generate a second verification result.

[0118] In practice, the execution subject can perform format verification on the second text information by using a plurality of regular expressions related to the filling rules to generate the second verification result. The second verification result represents whether the format verification is passed. For example, the second text information can include "opening of a securities account" and "cancellation of a securities account", which means that the user applies for account opening and account cancellation at the same time, and therefore does not meet the actual business process. The format error can be filtered out by using the regular expression.

[0119] Third, seal anti-forgery verification is performed on the seal information to generate a third verification result.

[0120] In practice, when the seal is a public seal, the execution subject can search the seal database to determine whether there is a seal record matching the seal information, to generate a third verification result representing whether the seal is abnormal.

[0121] Fourth, the content verification result is generated according to the first verification result, the second verification result, and the third verification result.

[0122] In practice, the execution subject can generate the content verification result according to the first verification result, the second verification result, and the third verification result. Specifically, when any of the first verification result, the second verification result, and the third verification result represents a failed verification, the content verification result representing a failed content verification is generated. When the first verification result, the second verification result, and the third verification result all represent a passed verification, the content verification result representing a passed content verification is generated.

[0123] Optionally, the execution subject can also generate a fourth verification result according to the positional relationship between the seal information, the first text information, and the second text information, and generate the content verification result according to the first verification result, the second verification result, the third verification result, and the fourth verification result.

[0124] In practice, first, when the position relationship between the seal corresponding to the seal information and the handwritten text corresponding to the second text information is a covering relationship, a fourth verification result representing that the verification is passed is generated. Specifically, when the seal and the handwritten text are in a covering relationship, it means that the seal has been manually reviewed by the seal affixer and is correct. Second, when the position relationship between the seal corresponding to the seal information and the handwritten text corresponding to the second text information is a non-covering relationship, text attribute extraction is performed on the non-handwritten text corresponding to the first text information and the handwritten text corresponding to the second text information. For example, the non-handwritten text is “handling”, and the handwritten text is “business A, business B”. At this time, the text attributes of “handling” and “business A, business B” are different, and a corresponding fourth verification result representing that the verification is passed can be generated. For another example, when the non-handwritten text is “business A”, and the handwritten text is “business B, business C”. At this time, the text attribute of “business A” and the text attribute of “business A, business B” are both the business label of the noun attribute, and at this time it is considered that the verification is not passed, and a fourth verification result representing that the verification is not passed is generated. In addition, when any one of the first verification result, the second verification result, the third verification result, and the fourth verification result represents that the verification is not passed, a content verification result representing that the content is not verified is generated. When the first verification result, the second verification result, the third verification result, and the fourth verification result all represent that the verification is passed, a content verification result representing that the content is verified is generated.

[0125] In step 108, in response to the content verification being passed, content extraction is performed on the first text information, the second text information, and the seal information to generate a data record for the to-be-recognized image.

[0126] In some embodiments, in response to the content verification being passed, the above-mentioned execution subject can perform content extraction on the first text information, the second text information, and the seal information through NER (Named Entity Recognition) technology to generate a data record for the to-be-recognized image. For example, the first text information, the second text information, and the seal information are extracted through a BERT model to generate a data record for the to-be-recognized image.

[0127] In step 109, the data record and the to-be-recognized image are stored.

[0128] In some embodiments, the above-mentioned execution subject can store the data record and the to-be-recognized image. Specifically, the data record and the to-be-recognized image can be stored in different databases through foreign key association, or the data record and the to-be-recognized image can be stored in the same database, and the specific storage manner is not limited herein.

[0129] The above various embodiments of the present disclosure have the following beneficial effects: through the image processing method of some embodiments of the present disclosure, effective and accurate content recognition for content contained in an image is achieved. Specifically, first, an image to be recognized is obtained, wherein the image to be recognized is an image containing handwritten text, non-handwritten text and seal image. That is, the image to be recognized contains various to-be-recognized contents such as handwritten text, non-handwritten text and seal image. Then, image preprocessing is performed on the image to be recognized to generate a preprocessed image. Through image preprocessing, the image quality is improved. Further, non-handwritten text recognition is performed on the preprocessed image to generate first text information. In this way, the non-handwritten text contained in the preprocessed image is automatically extracted. In addition, according to the first text information, a handwritten text region of interest and a seal region of interest in the image to be recognized are determined. In this way, according to the first text information, the region where the handwritten text and the seal image may exist is located. In addition, handwritten text recognition is performed on the handwritten text region of interest in the image to be recognized to generate second text information. Then, seal recognition is performed on the seal region of interest in the image to be recognized to generate seal information. In this way, automatic recognition of handwritten text and seal is achieved. Next, content verification is performed on the first text information, the second text information and the seal information. In this way, the effectiveness of the content is verified. Further, in response to the content verification passing, content extraction is performed on the first text information, the second text information and the seal information to generate a data record for the image to be recognized. In this way, attribute extraction is performed on the first text information, the second text information and the seal information to obtain the corresponding data record for data storage. Finally, data storage is performed on the data record and the image to be recognized. In this way, effective and accurate content recognition for content contained in an image is achieved.

[0130] Further reference Figure 7 As an implementation of the above-mentioned method shown in the figures, the present disclosure provides some embodiments of an image recognition device, which device embodiments correspond to those method embodiments shown in Figure 1 , and the image recognition device can be specifically applied to various electronic devices.

[0131] As Figure 7As shown, the image recognition apparatus 700 of some embodiments includes an acquisition unit 701, an image preprocessing unit 702, a non-handwritten text recognition unit 703, a determination unit 704, a handwritten text recognition unit 705, a seal recognition unit 706, a content verification unit 707, a content extraction unit 708, and a data storage unit 709. The acquisition unit 701 is configured to acquire an image to be recognized, where the image to be recognized is an image containing handwritten text, non-handwritten text, and a seal image; the image preprocessing unit 702 is configured to perform image preprocessing on the image to be recognized to generate a preprocessed image; the non-handwritten text recognition unit 703 is configured to perform non-handwritten text recognition on the preprocessed image to generate first text information; the determination unit 704 is configured to determine a handwritten text region of interest and a seal region of interest in the image to be recognized according to the first text information; the handwritten text recognition unit 705 is configured to perform handwritten text recognition on the handwritten text region of interest in the image to be recognized to generate second text information; the seal recognition unit 706 is configured to perform seal recognition on the seal region of interest in the image to be recognized to generate seal information; the content verification unit 707 is configured to perform content verification on the first text information, the second text information, and the seal information; the content extraction unit 708 is configured to, in response to the content verification passing, perform content extraction on the first text information, the second text information, and the seal information to generate a data record for the image to be recognized; and the data storage unit 709 is configured to store the data record and the image to be recognized.

[0132] It can be understood that the units described in the image recognition apparatus 700 correspond to the respective steps in the method described with reference to Figure 1 Thus, the operations, features, and advantages described above for the method also apply to the image recognition apparatus 700 and the units included therein, which will not be described here again.

[0133] Reference is made below to Figure 8 which shows a structural schematic diagram of an electronic device (e.g., a computing device) 800 suitable for use to implement some embodiments of the present disclosure. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0134] As Figure 8As shown, the electronic device 800 can include a processing device (e.g., a central processor, a graphics processor, etc.) 801 that can perform various suitable actions and processes in accordance with programs stored in a read only memory 802 or loaded from a storage device 808 into a random access memory 803. Various programs and data required by the electronic device 800 to operate are also stored in the random access memory 803. The processing device 801, the read only memory 802, and the random access memory 803 are connected to each other by a bus 804. An input / output interface 805 is also connected to the bus 804.

[0135] Generally, the following devices can be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication devices 809 can allow the electronic device 800 to communicate wirelessly or wired with other devices to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present. Figure 8 Each block shown in the flowcharts can represent a device or multiple devices as needed.

[0136] In particular, processes described above with reference to the flowcharts can be implemented as a computer software program according to some embodiments of the present disclosure. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In some such embodiments, the computer program can be downloaded and installed from a network through the communication devices 809, or installed from the storage devices 808, or installed from the read only memory 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the methods of some embodiments of the present disclosure are performed.

[0137] Note that the computer readable medium in some embodiments of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by an instruction execution system, apparatus or device, or that can be used by or in connection with an instruction execution system, apparatus or device. In some embodiments of the present disclosure, the computer readable signal medium can include a computer readable program code that is carried in a baseband or as a part of a carrier wave, in which the computer readable program code can be loaded into a computer readable storage medium. Such a propagated computer readable signal medium can take many forms, including but not limited to, an electromagnetic signal, an optical pulse, or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport program code for use by or in connection with an instruction execution system, apparatus or device. Program code embodied on a computer readable medium can be transmitted using any suitable medium, including but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0138] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0139] The computer readable medium can be included in the electronic device; or can exist independently of the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire an image to be recognized, where the image to be recognized is an image containing handwritten text, non-handwritten text, and a seal image; perform image preprocessing on the image to be recognized to generate a preprocessed image; perform non-handwritten text recognition on the preprocessed image to generate first text information; determine a handwritten text region of interest and a seal region of interest in the image to be recognized according to the first text information; perform handwritten text recognition on the handwritten text region of interest in the image to be recognized to generate second text information; perform seal recognition on the seal region of interest in the image to be recognized to generate seal information; perform content verification on the first text information, the second text information, and the seal information; in response to the content verification passing, perform content extraction on the first text information, the second text information, and the seal information to generate a data record for the image to be recognized; and perform data storage on the data record and the image to be recognized.

[0140] Computer program code for carrying out operations of some embodiments of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0141] The computer program product of the first aspect can include one or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform the operations of the method of the first aspect. The computer program product of the first aspect can include a computer-readable medium storing instructions that, when executed, cause one or more processors to perform the operations of the method of the first aspect.

[0142] The units described in some embodiments of the present disclosure can be implemented by means of software, or can be implemented by hardware. The described units can also be arranged in a processor, for example, can be described as: a processor includes an acquisition unit, an image preprocessing unit, a non-handwritten text recognition unit, a determination unit, a handwritten text recognition unit, a seal recognition unit, a content verification unit, a content extraction unit, and a data storage unit. Among them, the name of these units does not constitute a limitation to the unit itself in some cases, for example, the content extraction unit can also be described as "a unit that, in response to the content verification passing, performs content extraction on the first text information, the second text information, and the seal information to generate a data record for the to-be-recognized image".

[0143] The functions described in the foregoing description can be implemented, at least in part, by one or more hardware logic components. For example, and without limitation, non-limiting examples of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0144] The above description is merely illustrative of the exemplary embodiments of the present disclosure and the principles of the technology employed. It is understood that modifications and variations of the disclosed embodiments are possible, and also fall within the scope of the present disclosure. For example, the above features can be replaced with other features disclosed in the embodiments of the present disclosure (but not limited to) having similar functions to form other technical solutions.

Claims

1. An image recognition method, comprising: obtaining an image to be recognized, wherein the image to be recognized is an image containing handwritten text, non-handwritten text and a seal image; performing image preprocessing on the image to be recognized to generate a preprocessed image; performing non-handwritten text recognition on the preprocessed image to generate first text information, comprising: according to the region position corresponding to the character region of interest, filtering out the character region of interest satisfying a first filtering condition from the character region of interest set as a first character region of interest to obtain a first character region of interest set, wherein the first character region of interest in the first character region of interest set is a character region of interest corresponding to a region position being a middle region position in a line; according to the region position corresponding to the character region of interest, filtering out the character region of interest satisfying a second filtering condition from the character region of interest set as a second character region of interest to obtain a second character region of interest set, wherein the second character region of interest in the second character region of interest set is a character region of interest corresponding to a region position being a middle region position in a column; determining a first semantic correlation degree according to the first character region of interest set; determining a second semantic correlation degree according to the second character region of interest set; determining a horizontal recognition direction as a text recognition direction in response to the first semantic correlation degree being greater than a preset semantic correlation degree threshold and the first semantic correlation degree being greater than the second semantic correlation degree; determining a vertical recognition direction as a text recognition direction in response to the second semantic correlation degree being greater than the preset semantic correlation degree threshold and the second semantic correlation degree being greater than the first semantic correlation degree; determining a handwritten text region of interest and a seal region of interest in the image to be recognized according to the first text information; performing handwritten text recognition on the handwritten text region of interest in the image to be recognized to generate second text information; performing seal recognition on the seal region of interest in the image to be recognized to generate seal information; performing content verification on the first text information, the second text information and the seal information; performing content extraction on the first text information, the second text information and the seal information in response to passing the content verification to generate a data record for the image to be recognized; performing data storage on the data record and the image to be recognized; wherein the performing seal recognition on the seal region of interest in the image to be recognized to generate seal information comprises: determining a seal boundary, a seal text direction and a seal type according to the seal region of interest; performing axisymmetric segmentation on the seal boundary according to the seal type and the seal text direction to obtain a first segmented seal boundary and a second segmented seal boundary, wherein the first segmented seal boundary and the second segmented seal boundary are axisymmetric closed boundaries. According to the first segmented seal boundary, the seal region of interest is regionally segmented to obtain a first segmented seal region of interest, wherein the first segmented seal region of interest is a region surrounded by the first segmented seal boundary; According to the second segmented seal boundary, the seal region of interest is regionally segmented to obtain a second segmented seal region of interest, wherein the second segmented seal region of interest is a region surrounded by the second segmented seal boundary; Through the character recognition model, the first segmented seal region of interest is subjected to seal text recognition to obtain first seal text information, wherein the first seal text information includes seal text content and seal text position; Through the character recognition model, the second segmented seal region of interest is subjected to seal text recognition according to the seal text position included in the first seal text information to obtain second seal text information; According to the first seal text information and the second seal text information, the seal information is generated; The content verification of the first text information, the second text information and the seal information includes: According to the character font features corresponding to the first text information and a text smearing classifier, tampering verification is performed to generate a first verification result. The text smearing classifier is a binary classifier taking the output of the character feature extraction model as input. The first verification result represents whether the first text information contains smearing or tampered characters. Through a plurality of regular expressions related to the pre-constructed filling rules, the second text information is subjected to format verification to generate a second verification result. According to the seal information, seal anti-forgery verification is performed to generate a third verification result. According to the positional relationship between the seal information, the first text information and the second text information, a fourth verification result is generated, including: in response to the positional relationship between the seal corresponding to the seal information and the handwritten text corresponding to the second text information being a covering relationship, a fourth verification result representing that the verification passes is generated; in response to the positional relationship between the seal corresponding to the seal information and the handwritten text corresponding to the second text information being a non-covering relationship, text attribute extraction is performed on the non-handwritten text corresponding to the first text information and the handwritten text corresponding to the second text information respectively; in response to the text attribute corresponding to the non-handwritten text being different from the text attribute corresponding to the handwritten text, a fourth verification result representing that the verification passes is generated; in response to the text attribute corresponding to the non-handwritten text being the same as the text attribute corresponding to the handwritten text, a fourth verification result representing that the verification fails is generated; According to the first verification result, the second verification result, the third verification result and the fourth verification result, a content verification result is generated.

2. The method of claim 1, wherein, The image preprocessing of the to-be-recognized image to generate a preprocessed image includes: The to-be-recognized image is subjected to distortion correction to generate a distortion-corrected image; The distortion-corrected image is subjected to image size adjustment to obtain an image size-adjusted image; The image size-adjusted image is subjected to image enhancement to generate an image-enhanced image; perform character region of interest recognition on the image enhanced image to determine a character region of interest set; perform character shadow removal on the image enhanced image according to the character region of interest set to obtain the preprocessed image.

3. The method of claim 1, wherein, The performing non-handwritten text recognition on the preprocessed image to generate first text information further includes: According to the text recognition direction, the character region of interest set is divided into a character region of interest group set, wherein the character regions of interest in a character region of interest group are located in the same text line. For each character region of interest group in the character region of interest group set, the following processing steps are performed: Determine the character features corresponding to the characters in each character region of interest in the character region of interest group through a pre-trained character feature extraction model to obtain a character feature group, wherein the character features in the character feature group include font features, font size features, font color features, and font meaning features. Determine the first text description information corresponding to the character region of interest group in the first text description information set according to the character feature group through a character recognition model and a text semantic extraction model.

4. An image recognition apparatus, comprising: an acquisition unit configured to acquire an image to be recognized, wherein the image to be recognized is an image containing handwritten text, non-handwritten text, and a seal image; an image preprocessing unit configured to perform image preprocessing on the image to be recognized to generate a preprocessed image; a non-handwritten text recognition unit configured to perform non-handwritten text recognition on the preprocessed image to generate first text information, including: filtering out character regions of interest that meet a first filtering condition from the character region of interest set as first character regions of interest according to the region positions corresponding to the character regions of interest to obtain a first character region of interest set, wherein the first character regions of interest in the first character region of interest set are character regions of interest located in the same line as the middle region position; filtering out character regions of interest that meet a second filtering condition from the character region of interest set as second character regions of interest according to the region positions corresponding to the character regions of interest to obtain a second character region of interest set, wherein the second character regions of interest in the second character region of interest set are character regions of interest located in the same column as the middle region position; determining a first semantic correlation degree according to the first character region of interest set; determining a second semantic correlation degree according to the second character region of interest set; determining a horizontal recognition direction as a text recognition direction in response to the first semantic correlation degree being greater than a preset semantic correlation degree threshold and the first semantic correlation degree being greater than the second semantic correlation degree; determining a vertical recognition direction as a text recognition direction in response to the second semantic correlation degree being greater than the preset semantic correlation degree threshold and the second semantic correlation degree being greater than the first semantic correlation degree; and The determining unit is configured to determine a handwritten text region of interest and a seal region of interest in the image to be recognized according to the first text information; The handwritten text recognizing unit is configured to perform handwritten text recognition on the handwritten text region of interest in the image to be recognized to generate second text information; The seal recognizing unit is configured to perform seal recognition on the seal region of interest in the image to be recognized to generate seal information, wherein the seal recognition on the seal region of interest in the image to be recognized to generate seal information comprises: determining a seal boundary, a seal text direction and a seal type according to the seal region of interest; performing axisymmetric segmentation on the seal boundary according to the seal type and the seal text direction to obtain a first segmented seal boundary and a second segmented seal boundary, wherein the first segmented seal boundary and the second segmented seal boundary are axisymmetric closed boundaries; performing region segmentation on the seal region of interest according to the first segmented seal boundary to obtain a first segmented seal region of interest, wherein the first segmented seal region of interest is a region enclosed by the first segmented seal boundary; performing region segmentation on the seal region of interest according to the second segmented seal boundary to obtain a second segmented seal region of interest, wherein the second segmented seal region of interest is a region enclosed by the second segmented seal boundary; performing seal text recognition on the first segmented seal region of interest by using a character recognition model to obtain first seal text information, wherein the first seal text information comprises seal text content and seal text position; performing seal text recognition on a corresponding position in the second segmented seal region of interest according to the seal text position included in the first seal text information by using the character recognition model to obtain second seal text information; and generating the seal information according to the first seal text information and the second seal text information. The content verification unit is configured to perform content verification on the first text information, the second text information, and the seal information. The content verification on the first text information, the second text information, and the seal information includes: performing tampering verification according to a glyph feature of a character corresponding to the first text information and a text smudging classifier to generate a first verification result. The text smudging classifier is a binary classifier taking an output of a character feature extraction model as input. The first verification result indicates whether the first text information contains smudged or tampered characters. The second text information is subjected to format verification through a plurality of regular expressions related to pre-constructed filling rules to generate a second verification result. The seal information is subjected to seal anti-forgery verification to generate a third verification result. A fourth verification result is generated according to a positional relationship among the seal information, the first text information, and the second text information. The fourth verification result includes: in response to a positional relationship between a seal corresponding to the seal information and a handwritten text corresponding to the second text information being a covering relationship, generating a fourth verification result indicating that the verification passes; in response to the positional relationship between the seal corresponding to the seal information and the handwritten text corresponding to the second text information being a non-covering relationship, performing text attribute extraction on a non-handwritten text corresponding to the first text information and the handwritten text corresponding to the second text information; in response to the text attribute of the non-handwritten text being different from the text attribute of the handwritten text, generating a fourth verification result indicating that the verification passes; and in response to the text attribute of the non-handwritten text being the same as the text attribute of the handwritten text, generating a fourth verification result indicating that the verification fails. The content verification result is generated according to the first verification result, the second verification result, the third verification result, and the fourth verification result. The content extraction unit is configured to, in response to the content verification passing, perform content extraction on the first text information, the second text information, and the seal information to generate a data record for the to-be-recognized image. The data storage unit is configured to store the data record and the to-be-recognized image. 5.An electronic device, comprising: one or more processors; a memory device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-3.

6. A computer readable medium having stored thereon a computer program, wherein, The computer program is executed by the processor to implement the method of any one of claims 1-3.

Images