A Natural Scene Text Recognition Method Based on Geometric Priors and Knowledge Graphs

Abstract

A natural scene text recognition method based on geometric prior and knowledge graph, which collects domain scene text images, detects and crops text line images, and performs feature extraction and columnar deformation correction through a deformation correction model based on geometric prior; for the corrected image, a visual recognition module based on the attention mechanism is used to perceive each character to obtain character-level aligned visual texture features; then a global semantic reasoning module based on the domain knowledge graph introduces scene domain knowledge, perceives context information, and encodes high-level semantic features; finally, the output of the visual and semantic modules is integrated to obtain the text recognition result. The present invention can be transferred and applied to natural scene text recognition in different application fields such as automatic control instruments, equipment manufacturing, numerical control machine tools, automobile manufacturing, and rail transit, solves the problem that the traditional text recognition technology has low recognition accuracy due to columnar text deformation and lack of relevant dictionaries in natural scenes, and realizes more accurate recognition of domain text.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and relates to feature extraction and character recognition of text images in fields such as electrical equipment, and is used for intelligent text recognition of natural scenes, and is a natural scene text recognition method based on geometric prior and knowledge graph. Background Art

[0002] With the development of technology and the needs of economic development, the work in various industrial scenarios has also begun to be upgraded and optimized in an intelligent and digital manner. A typical example is that front-line substation operation and maintenance personnel need to engage in a large amount of regular equipment inspections, which is time-consuming and laborious, and greatly reduces work efficiency. Moreover, due to the large number of substations in the jurisdiction, it is often difficult for operation and maintenance personnel to discover defects in a timely manner. The intelligent text recognition technology can quickly locate the potential risks of inconsistencies between the substation site and the as-built drawings by recognizing the text of the secondary circuit wiring and making an intelligent comparison with the CAD drawings, thereby greatly improving the operation and maintenance efficiency of the substation.

[0003] Optical Character Recognition, abbreviated as OCR, refers to the process in which an electronic device (such as a scanner or digital camera) examines the characters printed on paper, determines their shapes by detecting dark and bright patterns, and then translates the shapes into computer text using character recognition methods. In some simple environments such as electronic documents, the accuracy of OCR has reached a relatively high level, but the text recognition effect in complex natural scenes is still not ideal. Technical problems faced in the application of slanted characters, artistic characters, deformed characters, blurred characters, similar characters, incomplete characters, light and shadow occlusion, multi-language mixed text, etc. have not been completely solved.

[0004] According to the universality of the recognition scene, OCR can be classified into dedicated OCR for specific scenes and general OCR for multiple scenes. For scenarios such as intelligent substation operation and maintenance, first, there is a lack of dedicated OCR technology specifically for specific scenes; second, due to the diversity and complexity of the text representation forms in such scenarios, the general OCR is difficult to achieve high performance when used in scenarios such as substation terminal blocks because of the serious deformation; third, due to the extremely strong semantic professionalism of the text in such scenarios, semantic information such as general vocabulary tables is difficult to play a role in specific scenes. Therefore, it is of great significance to study a natural scene text recognition method based on geometric prior and knowledge graph applicable to specific scenes.

[0005] At present, natural scene optical character recognition technology mainly focuses on prominent text regions in open scenes. These texts usually have problems such as occlusion and blurring, insufficient light, etc. In addition to these common problems, cable markings in fields such as substations also have the problem of columnar deformation caused by being printed on a cylindrical surface. Moreover, cable markings in specific fields obviously have rich semantic knowledge. Therefore, on the one hand, the intelligent recognition technology for cable markings in specific fields needs to introduce an image correction module based on solid geometry knowledge to minimize the impact of image deformation on text recognition. On the other hand, it needs to introduce a semantic reasoning module based on a specific domain knowledge graph to improve the accuracy of text recognition. Summary of the Invention

[0006] The problem to be solved by the present invention is to solve the problem of low recognition accuracy that may occur in actual application scenarios such as substation terminal block inspection by traditional text recognition technology. The purpose is to comprehensively and accurately recognize text in domain-specific scene fields.

[0007] The technical solution of the present invention is as follows: A natural scene text recognition method based on geometric prior and knowledge graph, which collects domain scene text images. The domain refers to the scene field to which the recognition is to be applied. The text has a cylindrical bend. Use a text detection algorithm to process the image to obtain the regions of all text lines, cut out the text line images, and then input the text line images into a deformation correction model based on geometric prior for feature extraction and columnar deformation correction to obtain a corrected image; send the corrected image into a recognition network, perceive the key spatial regions that each character needs to focus on through a visual recognition module based on the attention mechanism, and obtain character-level aligned visual texture features. Then, introduce scene domain knowledge through a global semantic reasoning module based on the knowledge graph, perceive context information, and encode abstract high-level semantic features; finally, comprehensively obtain the text recognition result based on the outputs of the visual and semantic modules.

[0008] The present invention is applicable to specific domain scenes, especially for the recognition of text with cylindrical bends. It can be transferred and applied to the recognition of natural scene texts in different specific domains such as automatic control instruments, equipment manufacturing, numerical control machine tools, automobile manufacturing, and rail transit. It solves the problem of low recognition accuracy of traditional text recognition technology in natural scenes due to columnar text deformation and lack of relevant dictionaries, and realizes more accurate recognition of domain texts.

[0009] The beneficial effects of the present invention are as follows: By introducing a columnar deformation correction module based on solid geometry knowledge, the columnar deformation correction of scene texts in fields such as substations is realized. The existing correction preprocessing solutions usually perform spatial transformation by detecting the text edge anchor points in the text area. This processing method is difficult to solve the deformation problems of cylindrical texts such as cables, while the columnar deformation correction module can capture more concealed and deeper visual spatial information than the text edges; By introducing a global semantic reasoning module of the knowledge graph in the corresponding field, prior knowledge such as known domain scene text design rules can be used to incorporate context semantic information into the reasoning process that originally only has visual perception features, which helps to identify more comprehensively and accurately. Description of the Drawings

[0010] Figure 1 It is a flowchart of the implementation of the present invention.

[0011] Figure 2 It is a structural diagram of the deformation corrector based on geometric prior of the present invention.

[0012] Figure 3 It is a structural diagram of the semantic reasoner based on the knowledge graph of the present invention. Detailed Embodiment

[0013] The present invention proposes a natural scene text recognition method based on geometric prior and knowledge graph. As Figure 1 shown, collect domain scene text images. The domain refers to the scene domain to which the recognition is to be applied. The text has columnar bending. Use a text detection algorithm to process the image to obtain the regions of all text lines, crop out to obtain text line images, and then input the text line images into a deformation correction model based on geometric prior for feature extraction and columnar deformation correction to obtain a corrected image; Send the corrected image into the recognition network, perceive the key spatial regions that each character needs to focus on through a visual recognition module based on the attention mechanism, and obtain character-level aligned visual texture features. Then introduce domain scene knowledge through a global semantic reasoning module based on the knowledge graph, perceive context information, and encode abstract high-level semantic features; Finally, combine the outputs of the visual and semantic modules to obtain the text recognition result, such as the recognition of terminal block texts common in scenes such as substations.

[0014] Next, in combination with the embodiments and drawings of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. The described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. The specific implementation of the present invention includes the following steps.

[0015] 1. Process the image using the DB text detection algorithm. Pass the input image through the feature pyramid backbone network, then upsample the pyramid features to the same size. Next, use the feature map to predict the probability map and the threshold map simultaneously. Calculate the approximate binary map from the feature map and the threshold map, supervise the probability map, the threshold map, and the approximate binary map, and output the detection box coordinate information of the text line region. Crop the text line image according to the detection box.

[0016] 2. Crop the text line region according to the text line region detection box result, and identify the text line direction through a CNN-based method and rotate the text box to the correct direction.

[0017] 3. Use the deformation correction model based on geometric prior, CNN, and bilinear interpolation to extract features and perform columnar deformation correction on the text image, as Figure 2 shown, including:

[0018] 1) Extract features from the input image through ResNet. Represent the deformation of the observed cylindrical text using the cylindrical orthographic projection method, specifically referring to the method of mapping a certain area on the cylindrical tangent plane to the cylindrical surface. Then, pass the extracted features into the fully connected neural network to regress the three parameters cx, cy, and radius r of the cylindrical projection. Among them, (cx, cy) is the coordinate value of the upper left vertex of the text area on the tangent plane relative to the origin of the observation point, used to characterize the observation direction and perspective situation, and r is the radius of the cylinder, used to characterize the degree of bending;

[0019] 2) Further calculate the remapping coordinate values (x’, y’) of each pixel (x, y) in the image according to the deformation parameters obtained in step 1). The specific calculation formula is:

[0020]

[0021] 3) Define the cylindrical projection as a differentiable operation according to the remapping formula obtained in step 2) to enable the network to perform backpropagation. Denote the network input image as S, the target image after deformation correction by the geometric prior-based deformer as T, and the cylindrical projection transformation as ρ (cx,cy,r) , the original coordinate on the input image S is (x S , y S ), and the target coordinate on the target image T is (x T , y T ). The deformation correction process is defined as:

[0022]

[0023] 4) Use the bilinear interpolation method to collect pixel values from the input image according to the remapping coordinates, and complete the correction of the cylindrical deformed image through remapping.

[0024] 4. Locate the position of each character through the attention mechanism and use the obtained underlying visual features to classify the preliminary recognition results of each character. Specifically, use FPN to aggregate the hierarchical feature maps of the third, fourth, and fifth stages of ResNet50 as the backbone network to obtain the 2D visual features of the image Through the multi-head attention mechanism, given key-value pairs (k, v) and query q, both the key and the value are taken from the visual feature u ij , calculate the similarity between the query q and all keys k to obtain the attention value α, and use this as the weight to calculate the weighted sum of v, thereby generating multiple 2D visual features g aligned with each character in the text. The calculation formula is:

[0025]

[0026] where, e t,ij is the correlation between the t-th character and the visual feature u ij , O t is the order of the t-th character in the text, f o is the embedding function, W e 、W o 、W v are trainable weight matrices, α t,ij is the attention value of the t-th character relative to the visual feature u ij , and g t is the visual feature value aligned with the t-th character.

[0027] 5. Encode abstract high-level features through a global semantic reasoning module based on a specific scenario knowledge graph, as Figure 3 shown, including:

[0028] 1) Construct the knowledge graph ontology of the corresponding field. In this embodiment, taking the recognition of substation terminal blocks as an example, the knowledge graph refers to the "Standard Design Specification for Line Protection and Auxiliary Devices" issued by the State Grid. Construct a knowledge graph ontology with the three elements of interval number, cable number, and terminal number in the loop design as the core ontology, and then convert the knowledge graph ontology into the form of an undirected graph K for easy combination of the knowledge graph and the neural network. The specific definition of K is:

[0029]

[0030]

[0031] where, represents the number of nodes in the knowledge graph, n a represents that the character category at the l a -th position is c a , and each edge Used to encode a kind of knowledge between two nodes;

[0032] 2) Pass the input feature map through the fully connected layer and the softmax layer to obtain the preliminary class distribution probability P, and calculate L using the cross-entropy loss p to make it more focused on the target character:

[0033]

[0034] where N represents the length of the text, y t represents the true value of the t-th character, p t represents the distribution probability of the t-th character, and p() represents the conditional probability function;

[0035] 3) Multiply the undirected graph K containing domain knowledge with the class distribution probability graph P to obtain the high-level context features, splice them with the original result to obtain the enhanced features Q containing visual information and semantic information dependent on prior knowledge, input them into the neural network fitting model, and use L calculated using the cross-entropy loss q for supervision:

[0036]

[0037] where q t represents the enhanced feature of the t-th character;

[0038] 4) After embedding the result obtained in step 3), send it into the transformer encoder for re-encoding, perform semantic information learning independent of prior knowledge, implicitly model the first-order relationship and high-order semantic relationship through the stacked multi-layer transformer units, and finally output the context features M at the time step level, while adding the inference loss L m . The objective function can be defined as:

[0039]

[0040] where m t represents the context feature of the t-th character.

[0041] 6. Use the visual-semantic fusion decoder to add and fuse the visual and semantic-based feature values g t , q t , m t by introducing some trainable weights to balance the contributions of features from different domains, and decode the sequence into the final prediction of the character. The fusion method and the loss value of the decoder are specifically defined as:

[0042]

[0043]

[0044] Among them, z t is the weight for balancing the features in different domains, W z is the trainable weight, σ is the activation function, f t is the t-th fused feature vector, g t , q t , m t are the visual features g t corresponding to f t introduced in Step 4, and the semantic features based on prior knowledge and the semantic features independent of prior knowledge in Step 5, that is, the enhanced feature q t and the context feature m t . The final objective function of the overall network model is:

[0045] Loss = α p L p + α q L q + α m L m + α f L f

[0046] Among them, α p = 1, α q = 2, α m = 0.15, α f = 1, representing the proportions of the visual recognition loss, the reasoning loss based on the knowledge graph, the implicit semantic reasoning loss, and the final decoder loss respectively.

Claims

1. A natural scene text recognition method based on geometric prior and knowledge graph, characterized by collecting domain scene text images, where the domain refers to the scene domain to which the recognition is to be applied, the text has cylindrical curvature, using a text detection algorithm to process the image to obtain the regions of all text lines, cropping to obtain text line images, then inputting the text line images into a deformation correction model based on geometric prior for feature extraction and cylindrical deformation correction to obtain a corrected image; sending the corrected image into a recognition network, perceiving the key spatial regions that each character needs to focus on through a visual recognition module based on the attention mechanism, and obtaining character-level aligned visual texture features, and then introducing scene domain knowledge through a global semantic reasoning module based on the knowledge graph, perceiving context information, and encoding abstract high-level semantic features; finally, synthesizing the outputs of the visual and semantic modules to obtain the text recognition result; among them, The feature extraction and cylindrical deformation correction by the deformation correction model based on geometric prior are specifically as follows: 1.1) Use ResNet to extract features from the input image, and represent the deformation of the observed cylindrical text by the cylindrical orthographic projection method, specifically referring to the method of mapping a certain region on the cylindrical tangent plane to the cylindrical surface, and then input the extracted features into a fully connected neural network to regress the three parameters cx, cy and radius r of the cylindrical projection, where (cx, cy) is the coordinate value of the upper left vertex of the text region on the tangent plane relative to the origin of the observation point, used to characterize the observation direction and perspective situation, and r is the radius of the cylinder, used to characterize the degree of curvature; 1.2) Further calculate the remapping coordinate values (x’, y’) of each pixel (x, y) in the image according to the deformation parameters obtained in step 1.1), and the specific calculation formula is: 1.3) According to the remapping formula obtained in step 1.2), define the cylindrical projection as a differentiable operation so that the network can perform backpropagation. Denote the network input image as S and the target image after deformation correction by the geometric prior-based deformator as T. Define the cylindrical projection transformation as ρ (cx,cy,r) , the original coordinates on the input image S are (x S , y S ), and the target coordinates on the target image T are (x T , y T ). Define the deformation correction process as follows: 1.4) Pixel values are collected from the input image according to the remapping coordinates by bilinear interpolation, and the correction of the cylindrical deformation image is completed through remapping; The global semantic reasoning module based on the knowledge graph is specifically as follows: 2.1) Construct the ontology of the knowledge graph in the corresponding field, and then transform the ontology of the knowledge graph into the form of an undirected graph K, which is convenient for the combination of the knowledge graph and the neural network. The specific definition of K is: Among them, represents the number of nodes, n a represents the l-th a character category at the position is c a , and each edge is used to encode a kind of knowledge between two nodes; 2.2) The input feature map is passed through a fully connected layer and a softmax layer to obtain the preliminary class distribution probability P, and L calculated using the cross-entropy loss p is used to make it more focused on the target character: where N represents the length of the text, and y t represents the true value of the t-th character, and p t represents the distribution probability of the t-th character, and p() represents the conditional probability function; 2.3) Multiply the undirected graph K containing domain knowledge with the category distribution probability graph P to obtain high-level context features. After concatenating with the original result P, an enhanced feature Q containing visual information and semantic information dependent on prior knowledge is obtained, which is input into the neural network fitting model, and the loss L calculated using cross-entropy loss is used for supervision: q Perform supervision: where q t represents the enhanced feature of the t-th character; 2.4) Embed the result obtained in 2.3) and then send it to the Transformer encoder for re-encoding to learn semantic information independent of prior knowledge. Implicitly model first-order relationships and high-order semantic relationships through stacked multi-layer Transformer units, and finally output the context feature M at the time step level. At the same time, add the inference loss L m , which is defined as: where m t represents the context feature of the t-th character.

2. The natural scene text recognition method based on geometric prior and knowledge graph according to claim 1, characterized in that Use the DB text detection algorithm to process the image. The input image passes through the backbone network of the feature pyramid, and then the pyramid features are upsampled to the same size. Then, the probability map and the threshold map are predicted simultaneously by the feature map. An approximate binary map is obtained by calculation from the feature map and the threshold map. The probability map, the threshold map, and the approximate binary map are supervised, and the detection box coordinate information of the text line area is output. The text line image is cropped according to the detection box.

3. A natural scene text recognition method based on geometric prior and knowledge graph according to claim 1, characterized in that For the obtained text line image, first identify the text line direction by a method based on CNN, rotate the text to the correct direction, and then input it into the deformation correction model.

4. A natural scene text recognition method based on geometric prior and knowledge graph according to claim 1, characterized in that The key spatial regions that each character needs to focus on are perceived through an attention mechanism-based visual recognition module, and character-level aligned visual texture features are obtained. Specifically, the hierarchical feature maps of the third, fourth, and fifth stages of ResNet50 are aggregated using FPN as the backbone network to obtain the 2D visual feature U of the image = , Through the multi-head attention mechanism, given key-value pairs (k, v) and query q, both the key and the value are taken from the visual feature u ij , calculate the similarity between the query q and all keys k to obtain the attention value α, and use this as the weight to calculate the weighted sum of v, thereby generating multiple 2D visual features g aligned with each character in the text. The calculation formula is as follows: where, e t,ij is the correlation between the t-th character and the visual feature u ij , O t is the order of the t-th character in the text, f o is the embedding function, W e , W o , W v are trainable weight matrices, α t,ij is the attention value of the t-th character to the visual feature u ij , g t is the visual feature value aligned with the t-th character.

5. A natural scene text recognition method based on geometric prior and knowledge graph according to claim 1, characterized in that The output of the comprehensive vision and semantics module is used to obtain the text recognition result. Specifically, the obtained vision- and semantics-based feature values g t , q t , m t are added and fused. Weights are introduced to balance the contributions of features from different domains. The weights are obtained through training, and the sequence is decoded into the final prediction of characters. The fusion method and the loss value of the decoder are specifically defined as follows: Among them, z t is the weight for balancing the features in different domains, W z is the trainable weight, σ is the activation function, f t is the t-th fused feature vector, g t , q t , m t are the visual feature, the semantic feature based on prior knowledge, and the semantic feature independent of prior knowledge respectively. The final objective function of the overall recognition network model is as follows: Loss=α p L p +α q L q +α m L m +α f L f Among them, α p = 1, α q = 2, α m = 0.15, α f = 1, respectively representing the proportions of the visual recognition loss, the knowledge graph-based reasoning loss, the implicit semantic reasoning loss, and the final decoder loss.

Images