Method and apparatus for recognizing text

Abstract

Embodiments of the present disclosure disclose a method and device for recognizing text. A specific implementation of the method comprises: obtaining a feature map, the feature map being obtained by performing text instance segmentation on an image in which text to be recognized is present; constructing a relation graph according to the feature map, wherein a node in the relation graph represents a pixel point in the feature map, and an edge represents that the similarity of the spatial semantic features of the two connected nodes is greater than a target threshold, and the spatial semantic features include the position feature and the category feature of the pixel point indicated by the node; processing the relation graph by using a pre-trained graph convolution network to obtain a first text feature corresponding to the image; and generating a text recognition result of the image according to the first text feature. The implementation proposes a graph-based text recognition method.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, and particularly, to a method and apparatus for recognizing text. BACKGROUND

[0002] Existing text recognition can be divided into two categories, one is OCR (Optical Character Recognition), and the other is STR (Scene Text Recognition). Among them, OCR usually refers to recognizing text in high-quality scanned document images, and STR usually refers to recognizing text in natural scene images.

[0003] Generally, the recognition scene of OCR is simple in image background, orderly in text arrangement, and standard in font, while the recognition scene of STR is complex in image background, random in text arrangement, and various in font. Therefore, the recognition difficulty of STR is much greater than that of OCR.

[0004] STR has important practicality in many fields such as assisting navigation of visual impairment, automatic driving application, text reading and translation in augmented reality, and has attracted more and more attention in the field of computer vision. The current recognition method of STR usually locates the text region from the image first, and then recognizes the text in the text region. SUMMARY

[0005] Embodiments of the present disclosure provide a method and apparatus for recognizing text.

[0006] In a first aspect, embodiments of the present disclosure provide a method for recognizing text, the method comprising: obtaining a feature map, wherein the feature map is obtained by performing text instance segmentation on an image in which text to be recognized is presented; constructing a relation graph according to the feature map, wherein a node in the relation graph represents a pixel point in the feature map, and an edge represents that the similarity of spatial semantic features of two connected nodes is greater than a target threshold, and the spatial semantic features include position features and category features of the pixel points indicated by the nodes; processing the relation graph by using a pre-trained graph convolution network to obtain a first text feature corresponding to the image; and generating a text recognition result of the image according to the first text feature.

[0007] In a second aspect, embodiments of the present disclosure provide an apparatus for recognizing text, the apparatus comprising: a feature map obtaining unit configured to obtain a feature map, wherein the feature map is obtained by performing text instance segmentation on an image in which text to be recognized is presented; a relation graph constructing unit configured to construct a relation graph according to the feature map, wherein a node in the relation graph represents a pixel point in the feature map, and an edge represents that a spatial semantic feature of two connected nodes is greater than a target threshold, and the spatial semantic feature comprises a position feature and a category feature of the pixel points indicated by the nodes; a graph convolution processing unit configured to process the relation graph by using a pre-trained graph convolution network to obtain a first text feature corresponding to the image; and a recognizing unit configured to generate a text recognition result of the image according to the first text feature.

[0008] In a third aspect, embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage apparatus configured to store one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0009] In a fourth aspect, embodiments of the present disclosure provide a computer readable medium having stored thereon a computer program, which, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0010] The method and apparatus for recognizing text provided by the embodiments of the present disclosure can consider the two-dimensional spatial information of the text in the image, avoid directly compressing the text features in the image into one-dimensional features and ignoring the two-dimensional spatial information, and help improve the text recognition effect. BRIEF DESCRIPTION OF DRAWINGS

[0011] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments thereof as taken in conjunction with the accompanying drawings:

[0012] Figure 1 is an exemplary system architecture diagram to which an embodiment of the present disclosure can be applied;

[0013] Figure 2 is a flowchart of an embodiment of a method for recognizing text according to the present disclosure;

[0014] Figure 3 is a flow chart of one embodiment of a feature map generation manner in the method for identifying text according to the present disclosure;

[0015] Figure 4 is a flow chart of yet another embodiment of the method for identifying text according to the present disclosure;

[0016] Figure 5a 、 5b , 5c is a schematic diagram of one exemplary application scenario of the method for identifying text according to the present disclosure;

[0017] Figure 6 is a flow chart of one embodiment of a training method of a graph convolution network, a language model and a segmentation network in the method for identifying text according to the present disclosure;

[0018] Figure 7 is a structural schematic diagram of one embodiment of the apparatus for identifying text according to the present disclosure;

[0019] Figure 8 is a structural schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] The present disclosure will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings.

[0021] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0022] Figure 1 An exemplary architecture 100 of the embodiments of the method for identifying text or the apparatus for identifying text to which the present disclosure can be applied is shown.

[0023] As shown in Figure 1 , the system architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0024] Terminal devices 101, 102, and 103 interact with server 105 via network 104 to receive or send messages, etc. Various client applications can be installed on terminal devices 101, 102, and 103, such as browser applications, search applications, image processing applications, deep learning frameworks, etc.

[0025] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0026] Server 105 can be a server providing various services, such as a server providing backend support for client applications installed on terminal devices 101, 102, and 103. Server 105 can be hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0027] It should be noted that the method for recognizing text provided in the embodiments of this disclosure is generally executed by server 105, and correspondingly, the device for recognizing text is generally located in server 105.

[0028] It should also be noted that image processing applications can be installed on terminal devices 101, 102, and 103. Terminal devices 101, 102, and 103 can also process images containing text to be recognized based on these image processing applications. In this case, the method for recognizing the text can also be executed by terminal devices 101, 102, and 103, and correspondingly, the device for recognizing the text can also be located in terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may not include server 105 and network 104.

[0029] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0030] Continue to refer to Figure 2which shows a flow 200 of one embodiment of the method for identifying text according to the present disclosure. The method for identifying text comprises the following steps:

[0031] Step 201, obtaining a feature map.

[0032] In the present embodiment, the feature map can be obtained by performing text instance segmentation on an image presenting the text to be identified. The text to be identified can be text of various contents. For example, the text to be identified can include one or more characters (such as letters, numbers, special symbols, Chinese characters, etc.). The image presenting the text to be identified can be various types of images.

[0033] According to different actual application scenarios, the quality of the image presenting the text to be identified can be different, and various attributes of the text to be identified presented by the image, such as position and writing manner, can be different.

[0034] Text instance segmentation can refer to detecting and distinguishing each character included in the text to be identified from the image. Generally, the feature map (MASK) obtained after text instance segmentation can correspond to the characters included in the identified text. The number of feature maps can be flexibly set according to the actual application scenario. For example, if the text to be identified in the image includes only one character, the number of feature maps can be one.

[0035] Optionally, the number of feature maps is at least two. Alternatively, the number of feature maps can be specified by the technician in advance. Generally, the number of feature maps set can be greater than the estimated number of characters in the text to be identified in the image according to the actual application scenario, so as to avoid missing identification and the like.

[0036] Specifically, the above-mentioned execution subject or other electronic devices can use various existing instance segmentation methods to perform text instance segmentation on the image presenting the text to be identified to obtain the feature map.

[0037] In the present embodiment, the method for identifying text (such as the server 105 shown in Figure 1 The server 105 and the like) can pre-store the feature map obtained by performing text instance segmentation on the image presenting the text to be identified locally, at this time, the execution subject can directly obtain the feature map from the local. The execution subject can also obtain the feature map from other storage devices (such as connected databases, third-party data platforms, terminal devices 101, 102, 103, and the like shown in Figure 1

[0038] Step 202, constructing a relationship graph according to the feature map.

[0039] ​In this embodiment, after obtaining the feature map, each pixel point in the feature map can be taken as a node, and an edge can be constructed according to the similarity between the spatial semantic features of the pixel points, so as to obtain a relationship graph constructed based on the feature map.

[0040] The spatial semantic features of the pixel points can include position features and category features of the pixel points. The position features of the pixel points can be used to represent the positions of the pixel points in the feature map. The category features of the pixel points can be used to represent the categories of the text to which the pixel points belong.

[0041] The categories of the text can be set in advance according to actual application scenarios. For example, if the text to be recognized is a number, 11 categories of text can be pre-divided to represent 0-9 and the background.

[0042] Generally, the position features of the pixel points can be represented by the coordinates (such as the horizontal coordinates and the vertical coordinates) of the pixel points in the feature map. The category features of the pixel points can be represented by a vector, which can represent the probabilities of the pixel points belonging to each pre-set category of text.

[0043] When constructing the relationship graph, if the similarity of the spatial semantic features of two pixel points is greater than a target threshold, an edge can be established between the two pixel points. Otherwise, if the similarity of the spatial semantic features of the two pixel points is not greater than the target threshold, no edge is established between the two pixel points. The target threshold can be pre-set by a technician or can be flexibly determined in the relationship graph construction. For example, the target threshold can be determined according to the spatial semantic similarities between each pair of pixel points.

[0044] The similarity of the spatial semantic features can be flexibly determined by various methods. For example, the similarity of the position features and the similarity of the category features of the two pixel points can be calculated respectively, and then the weighted sum of the similarity of the position features and the similarity of the category features can be taken as the similarity of the spatial semantic features of the two pixel points.

[0045] Optionally, when the number of the feature maps is at least two, a corresponding relationship subgraph can be constructed for each feature map, and then the relationship subgraphs corresponding to each feature map can be merged to obtain a final relationship graph. The merging of the relationship graphs can be implemented by sequentially connecting the relationship subgraphs according to the positional relationship of each feature map in the image. Specifically, for two relationship subgraphs, a node can be selected from each of the two relationship subgraphs to connect the two relationship subgraphs. The method of selecting the node from the relationship subgraph can be flexibly set, such as selecting a root node.

[0046] In step 203, the pre-trained graph convolution network is used to process the relationship graph to obtain a first text feature corresponding to the image.

[0047] In this embodiment, the graph convolutional network (GCN) can be generally regarded as a model using Chebyshev first-order polynomial approximation spectral convolution operation of a graph Laplacian matrix. From the perspective of spectral graph convolution, the graph convolutional network can be regarded as a special form of graph Laplacian smoothing. The convolution operation of the graph convolutional network can be regarded as transmitting the feature information of each node to the neighbor nodes of the node after transformation, and then fusing the feature information of the neighbor nodes to update the feature information of each node.

[0048] Specifically, after updating the feature information (i.e., the spatial semantic feature) of each node by using the graph convolutional network, various methods can be used to generate the first text feature corresponding to the image according to the updated feature information of each node. The first text feature can be used to represent the feature of the to-be-recognized text in the image.

[0049] For example, the feature information of each node can be averaged or maximized, and the processing result can be taken as the first text feature. For another example, the updated feature information of the node corresponding to the target pixel point in the feature map can be taken as the first text feature. The target pixel point can be flexibly set.

[0050] As an example, the target pixel point can be pre-specified by a technician, such as the geometric center point of the feature map. As another example, the target pixel point can be a pixel point corresponding to a node whose similarity with each neighbor node is greater than a preset threshold.

[0051] In step 204, a text recognition result of the image is generated according to the first text feature.

[0052] In this embodiment, after obtaining the first text feature, various existing text recognition methods (such as a text recognition method based on a recurrent neural network and a connection time classifier) can be used to generate a text recognition result corresponding to the first text feature as the text recognition result corresponding to the image presenting the to-be-recognized text.

[0053] In some optional implementation modes of this embodiment, the feature map obtained by performing text instance segmentation on the image presenting the to-be-recognized text can represent the image features of the image region where each character in the to-be-recognized text is located and the sequential features between other feature maps. The sequential features can represent the sequential relationship between the feature maps, so that the context of each character can be more accurately combined for recognition in the subsequent recognition process, which helps to improve the accuracy of the recognition result.

[0054] Reference will be made to the following description Figure 3 , Figure 3A flowchart 300 illustrating one embodiment of a manner of generating a feature map in a method for recognizing text according to an embodiment of the present disclosure is shown. As shown in the flowchart 300, the method comprises the following steps. Figure 3 As shown, the feature map obtained by performing text instance segmentation on an image having text to be recognized can be generated by the following steps:

[0055] In step 301, the image is input into a pre-trained convolutional neural network to obtain an initial feature map.

[0056] In this embodiment, the convolutional neural network can be used to perform convolutional operations on the image to extract various features (such as texture features, color features, etc.) of the image to obtain the initial feature map.

[0057] Optionally, the convolutional neural network can be implemented based on a feature pyramid network (FPN) and a residual network (ResNet), and the stride of at least one convolutional layer before the output layer of the residual network can be set to 1, and the feature map output by the residual network can be used to generate the input feature map of the feature pyramid network through deformable convolution.

[0058] Taking ResNet50 as an example, ResNet50 generally processes an input image through five stages to obtain an output feature map. Assuming that the feature maps output by the five stages in sequence are S1-S5, the convolutional stride of stages 4 and 5 can be set to 1 to make S4 and S5 retain more underlying image information about the texture of the text and the boundary of the text.

[0059] Then, deformable convolution can be performed on the feature map S5, and the feature map after the deformable convolution is performed is input into the FPN, and the feature map finally output by the FPN is used as the initial feature map. The use of deformable convolution can make the convolutional neural network better adapt to the irregular boundary of the text to be processed, thereby improving the accuracy of subsequent text recognition.

[0060] In step 302, text instance segmentation is performed on the initial feature map to obtain an instance feature map.

[0061] In this embodiment, various existing instance segmentation methods can be used to perform text instance segmentation on the initial feature map to obtain the instance feature map.

[0062] Optionally, a network with a PPM (Pyramid Pooling Module) structure can be used to extract features of different receptive fields of the initial feature map first, and then the features of different receptive fields are concatenated and dimensionally converted. For example, parallel 1*1, 3*3 and 5*5 convolutional layers can be used to extract features of the initial feature map, and then the features extracted by the convolutional layers can be concatenated, and a 1*1 convolutional layer can be used for dimension conversion to facilitate subsequent processing.

[0063] Optionally, after the concatenation and dimension conversion of the features of different receptive fields, the features can be further converted using multiple stacked convolution layers based on the spatial attention mechanism to obtain features with enhanced spatial positions. For example, after the concatenation and dimension conversion of the features of different receptive fields, 3*3 and 1*1 convolution layers can be further used to obtain spatial attention feature maps. Then, text instance segmentation is performed based on the features with enhanced spatial positions.

[0064] In step 303, text sequential segmentation is performed on the initial feature map to obtain a sequential feature map.

[0065] In this embodiment, in order to maintain the reading order of each character in the text to be recognized in the image, various existing text sequential segmentation methods can be used to process the initial feature map to obtain a sequential feature map.

[0066] For example, the initial feature map can be input into a network constructed based on a convolutional encoder-decoder structure to perform simple convolution down-sampling and deconvolution up-sampling on the initial feature map, thereby obtaining a sequential feature map.

[0067] In step 304, the instance feature map and the sequential feature map are fused to obtain a feature map obtained by performing text instance segmentation on the image presenting the text to be recognized.

[0068] In this embodiment, the obtained instance feature map and the sequential feature map can be fused to obtain a feature map obtained by performing text instance segmentation on the image presenting the text to be recognized.

[0069] When there are at least two character instances in the instance feature map, the instance feature map can be fused with the sequential feature map corresponding to each character instance to obtain at least two fused feature maps.

[0070] Specifically, various feature fusion methods can be used to fuse the instance feature map and the sequential feature map. For example, the fusion of the instance feature map and the sequential feature map can be achieved by multiplying corresponding pixels in the instance feature map and the sequential feature map.

[0071] In some optional implementations of this embodiment, the spatial semantic feature of a node in the relationship graph constructed based on the feature map can further include the sequential feature of the feature map in which the pixel indicated by the node is located.

[0072] At this time, for a node in the relationship subgraph corresponding to each feature map, the spatial semantic feature of the node can be generated by the following steps:

[0073] Step one, obtaining the sequential value of the feature map corresponding to the node in each feature map.

[0074] In this step, since the feature maps are obtained by segmenting the image containing the text to be identified, there is an order relationship between the feature maps, that is, the order between the segmented instances.

[0075] The ordinal value can represent the ranking position of a feature map among the various feature maps. For example, if there are 20 feature maps, the ranking number (one of 1-20) of each feature map can be used as the ordinal value of that feature map.

[0076] It is understandable that the sequential values ​​of each pixel in the same feature map are the same.

[0077] Step 2: Determine the sequential features of the feature map corresponding to the node based on the sequential value of the feature map corresponding to the node.

[0078] In this step, the sequential features of the feature maps can be represented using vectors. Therefore, the sequential features corresponding to each feature map can be obtained by mapping each sequential value to the same vector space. Specifically, various existing mapping methods can be used.

[0079] For example, sine and cosine functions of different wavelengths can be used to represent the order values ​​as vectors. As an example, the vector representation of each order value can be obtained using the following formula:

[0080]

[0081]

[0082] Where z can represent the ordinal value. Cε represents the vector dimension, which can be preset by the technicians. i represents the index of the element in the vector. ε 2i ε represents the element in the vector whose index is even. 2i+1 This represents the element in the vector with an odd index.

[0083] Step 3: Combine the positional features, category features, and determined order features of the pixels indicated by the node to obtain the spatial semantic features of the node.

[0084] In this step, the spatial semantic features of the node are obtained by sequentially concatenating the positional features, category features, and determined order features of the pixels indicated by the node. Both the positional and category features of the pixels can be represented using vectors.

[0085] Optionally, for each node, the positional and category features of the pixel can be first represented as vectors, and then the vector representations of the positional and category features of the node and the vector representation of the order features of the node can be concatenated to obtain the spatial semantic features of the node.

[0086] As an example, for any node, the horizontal coordinate of the node is X, the vertical coordinate of the node is Y, and the category feature of the node is P, then X, Y and P are mapped into vector representations respectively, and then the vector representations corresponding to X, Y and P are concatenated as the vector representations corresponding to the position feature and the category feature of the node. The method of mapping X, Y and P into vector representations can adopt various existing mapping methods.

[0087] It should be noted that, in order to facilitate calculation, the dimensions of various vector representations can be kept consistent. For example, the dimensions of various vector representations can be flexibly adjusted using a 1*1 convolution.

[0088] Optionally, before concatenating the vector representations corresponding to the position feature and the category feature of the node and the vector representation of the sequential feature of the node, the vector representations corresponding to the position feature and the category feature of the node can be standardized by using various standardization methods, so as to reduce the vector elements and facilitate subsequent calculation.

[0089] As an example, the standardization can be achieved by subtracting the vector representations corresponding to the position feature and the category feature of the target node in the corresponding feature map from the vector representations of the node. It can be understood that the vector representations corresponding to the position feature and the category feature of the target node are 0 after the standardization.

[0090] The target node can be pre-established by the technician or flexibly set in the calculation process. For example, the target node can be the root node of the feature map. Generally, the geometric center point of the feature map can be set as the root node.

[0091] Optionally, the root node of the feature map can be determined by the following steps: for a node in the feature map, determining the intersection over union of the node with each of the other nodes, and in response to determining that each of the intersection over unions corresponding to the node is not greater than a preset threshold, determining the node as the root node. The intersection over union can represent the ratio of the number of elements included in the intersection of the neighbor nodes of two nodes to the number of elements included in the union of the neighbor nodes. The preset threshold can be pre-set by the technician. By this method, the pixel points of each text center can be effectively screened as the root node, and at the same time, the distribution of nodes in the relationship subgraph corresponding to the feature map can be balanced.

[0092] In some optional implementations of the embodiment, for two nodes, the similarity of the spatial semantic feature can be determined by the following steps:

[0093] Step one, determining the Euclidean distance between the two nodes, and determining a first similarity according to the determined Euclidean distance.

[0094] In this step, the Euclidean distance between two nodes is generally inversely proportional to the similarity of the spatial semantic features of the nodes. The first similarity can be determined based on this using various methods. As an example, the first similarity can be determined using the following formula:

[0095]

[0096] where p and q are two nodes respectively. D(p, q) represents the Euclidean distance between p and q. m and W m represent the height and width of the feature map respectively. E s (p, q) represents the first similarity.

[0097] Step two, determine the cosine similarity of the category features corresponding to the two nodes respectively.

[0098] In this step, the cosine similarity can be calculated using the vector representation of the category features corresponding to the two nodes respectively using existing cosine similarity calculation methods.

[0099] Step three, determine the similarity of the spatial semantic features of the two nodes according to the first similarity and the cosine similarity.

[0100] In this step, the spatial semantic feature similarity can generally be proportional to the first similarity and also proportional to the cosine similarity. Based on this, various methods can be used to determine the similarity of the spatial semantic features of the two nodes. For example, the product of the first similarity and the cosine similarity can be directly calculated as the similarity of the spatial semantic features of the two nodes. In this way, the similarity between the nodes in various aspects such as spatial position and category can be comprehensively considered, thereby helping to construct a more accurate relationship graph.

[0101] In some optional implementations of the present embodiment, when constructing the relationship graph corresponding to the feature map, for each node, the similarity of the spatial semantic features between the node and each of the other nodes can be calculated respectively, and then the target number of nodes can be selected as the neighbor nodes of the node in order from large to small according to the similarity, i.e. edges are established between the selected nodes and the node. The target number can be flexibly set according to the specific application scenario. For example, the target number can be 8. In this way, the complexity and accuracy of the constructed relationship graph can be flexibly controlled to assist subsequent calculations.

[0102] In some optional implementations of the embodiment, the graph convolution network can include a first graph convolution network and an output network. The first graph convolution network can be configured to transform the feature matrix of the constructed relation graph. The output network can be configured to select nodes from each relation subgraph based on the output of the first graph convolution network, and aggregate the transformed features of the selected nodes from each relation subgraph to obtain the first text feature.

[0103] For each relation subgraph, the feature matrix and the adjacency matrix of the relation graph can be used to represent the relation subgraph. The elements in the feature matrix can be used to represent the features of the nodes in the relation subgraph. The adjacency matrix can be used to represent the connection relationship (e.g., whether there is an edge, etc.) between the nodes in the relation subgraph.

[0104] The first graph convolution network can use various existing convolution network structures to transform the feature matrix of the relation graph.

[0105] Optionally, the first graph convolution network can include a first graph convolution subnetwork and a second graph convolution subnetwork. The first graph convolution subnetwork can be configured to transform the feature matrix of the relation graph. The second graph convolution subnetwork can be configured to transform the feature matrix output by the first graph convolution subnetwork based on the relation subgraph constructed based on the output of the first graph convolution subnetwork.

[0106] In this case, after the first graph convolution subnetwork transforms the feature matrix of the relation graph, the similarity between the nodes can be recalculated and the edges can be re-established according to the features of the nodes in the transformed feature matrix, i.e., the relation subgraph is updated. The second graph convolution subnetwork can process the updated relation subgraph.

[0107] The structures of the first graph convolution subnetwork and the second graph convolution subnetwork can be flexibly set by the technicians according to actual application requirements.

[0108] As an example, the processing process of the first graph convolution subnetwork can be shown in the following formula:

[0109]

[0110]

[0111]

[0112] wherein, l is the number of convolution layers of the first graph convolution subnetwork. Y l represents the output of the lth layer. X l represents the input of the lth layer. W l is the network parameter learned by the first graph convolution subnetwork. A represents the adjacency matrix of the relation graph. I Ndenotes a matrix of the same size as A with 1s on the main diagonal. is a diagonal matrix. i and j denote the row and column indices, respectively. σ denotes a nonlinear activation function. denotes concatenating matrices along dimensions.

[0113] The processing procedure of the second graph convolution subnetwork is shown in the following formula.

[0114]

[0115] where l is the number of convolution layers of the second graph convolution subnetwork. denotes the output of the l-th layer. is the network parameter learned by the second graph convolution subnetwork. It is to be noted that and are matrices obtained based on the updated relationship graph, and the specific calculation method is the same as and in the first graph convolution subnetwork.

[0116] Specifically, the output network can recalculate the distance between nodes according to the update of the feature information of each node in the relationship graph represented by the output of the first graph convolution network, and then discard part of the nodes according to the distance from small to large (such as discarding half of the nodes), so as to realize the pooling operation and reduce the size of the corresponding feature map.

[0117] Optionally, the feature update and node screening process of the first graph convolution network and the output network can be iteratively performed until only one node is left in each relationship subgraph. Then, the feature information of the screened nodes in each relationship subgraph can be spliced in sequence according to the order of the relationship subgraphs to form the first text feature.

[0118] Through the iteration of the feature update and node screening process of the first graph convolution network and the output network, the node in the spatial relationship that best represents the corresponding character instance can be screened from each relationship subgraph, so that the feature information of the node is used for subsequent text recognition, which helps to improve the efficiency and accuracy of text recognition.

[0119] The method provided by the above embodiments of the present disclosure provides a graph-based text recognition method. Specifically, a relationship graph is constructed based on the feature map obtained by segmenting the text instance, so as to use the graph structure to express the spatial semantic information of the text, and then the relationship graph is processed by graph convolution to extract the two-dimensional spatial feature information of the text to be recognized in the image, and the order relationship between each character instance is combined to realize the recognition of the text to be recognized in the image.

[0120] Reference will be made to the accompanying drawings Figure 4which shows a flow 400 of one of the further embodiments of the method for recognizing text according to the present disclosure. The method for recognizing text comprises the following steps:

[0121] Step 401, obtaining a feature map obtained by text instance segmentation on an image presenting text to be recognized.

[0122] Step 402, constructing a relation graph according to the feature map.

[0123] Step 403, processing the relation graph by using a pre-trained graph convolution network to obtain a first text feature corresponding to the image.

[0124] Step 404, processing the feature map by using a pre-trained language model to obtain a second text feature corresponding to the image.

[0125] In the present embodiment, the language model can be various existing language models for text recognition. For example, N-Gram model, HMM (Hidden Markov Model), BERT (Bidirectional Encoder Representations from Transformers), etc. Generally, the feature representation of the text sequence generated by the language model before the output layer can be selected as the second text feature.

[0126] Optionally, the feature map can be processed by using a pre-trained semantic feature extraction network to obtain a semantic feature corresponding to the feature map. Then the semantic feature is input into the language model to obtain the second text feature. The semantic feature extraction network can adopt the structure of various existing feature extraction networks.

[0127] As an example, the semantic feature extraction network can include a pooling layer for performing pooling processing on the feature map and a linear layer for performing linear transformation on the output of the pooling layer. Specifically, the input feature map is first subjected to a pooling operation, and then the pooling result is subjected to linear transformation, so as to reduce the resolution of the image space and generate the semantic feature.

[0128] The language model can adopt the structure of various existing natural language processing-based models. For example, a translation model and the like can be used to predict the semantics of each text instance by using several text instances before and after the text instance, so as to realize text recognition.

[0129] Step 405, generating a text recognition result of the image according to the first text feature and the second text feature.

[0130] In the embodiment, the first text feature and the second text feature can be combined, and various methods can be used to generate the text recognition result of the image. For example, the first text feature and the second text feature can be fused by using various existing feature fusion methods to obtain fused text features, and then the fused text features are recognized to obtain the recognition result.

[0131] Since the text recognition process of the language model uses one-dimensional compressed feature information and ignores two-dimensional spatial feature information, more reliable text recognition can be achieved by combining the graph-based text recognition method and the language model-based text recognition method to utilize richer feature information.

[0132] In some optional implementations of the embodiment, the text recognition result of the image is generated according to the first text feature, the second text feature, and the feature map.

[0133] Since the feature map itself can represent the image features of the text to be recognized, the representation ability of the text features can be further enhanced by combining the feature map obtained by text instance segmentation when the first text feature and the second text feature are used for text recognition, thereby improving the text recognition effect.

[0134] The specific recognition method can be flexibly set according to actual application requirements. For example, the first text feature, the second text feature, and the feature map can be fused by using various existing feature fusion methods, and then the fused features are used to predict the text recognition result.

[0135] Optionally, the first text feature, the second text feature, and the feature map can be input into a pre-trained feature fusion network to generate the text recognition result of the image.

[0136] The feature fusion network can be used to concatenate the first text feature, the second text feature, and the feature map, and then perform linear transformation on the obtained concatenation result to obtain the text recognition result of the image. The structure of the feature fusion network can be pre-set by the technician.

[0137] As an example, the processing process of the feature fusion network is shown in the following formula:

[0138] f t =Wt t ⊙(W f ·[v t ;l t ;g t ])

[0139] W t =Sigmod(w z ·[v t ;l t ;g t ])

[0140] wherein ft represents the fusion result of the feature fusion network. t , l t and g t represent the feature map, the second text feature and the first text feature respectively. W z and W f represent the linear transformation parameters to be learned by the feature fusion network. ⊙ represents the multiplication operation between elements. “;” represents the dimension concatenation operation. Sigmod is the activation function. t represents the number of feature maps obtained by performing text instance segmentation on the image presenting the text to be recognized.

[0141] It should be noted that the feature maps have a sequential relationship, and the first text feature and the second text feature are also the feature representations of the text sequence generated based on the sequential relationship, so each fusion is to fuse the corresponding text features in the feature map, the second text feature and the first text feature, thereby obtaining the final fusion result.

[0142] In addition, it should also be noted that since the corresponding text instances have a context sequential relationship, the feature maps also have a context sequential relationship, but in the processing of the feature maps, the text instances corresponding to each feature map can be recognized in parallel to improve the processing efficiency.

[0143] The contents not specifically explained in the present embodiment can be referred to the related explanations in the corresponding embodiments, which will not be repeated here. Figure 2 The contents not specifically explained in the present embodiment can be referred to the related explanations in the corresponding embodiments, which will not be repeated here.

[0144] With reference to Figure 5a , Figure 5b and Figure 5c are schematic diagrams of one exemplary application scenario of the method for recognizing text according to the present embodiment. As shown in Figure 5a , the image presenting the text to be recognized, i.e. the text image, can be obtained first, and then the text image is input into the pre-trained convolutional network to extract the initial feature map. Then, the initial feature map is input into the text instance segmentation network and the text sequential segmentation network respectively to extract the instance feature map and the sequential feature map. The obtained instance feature map and the sequential feature map are fused to form a plurality of feature maps corresponding to the text image.

[0145] Then, the relationship graph can be constructed according to the obtained plurality of feature maps, and the pre-trained graph convolutional network is used to process the constructed relationship graph to obtain the first text feature of the text image. At the same time, the obtained plurality of feature maps are processed by using the language model to obtain the second text feature of the text image. Then, the obtained plurality of feature maps, the first text feature and the second text feature are comprehensively used for text recognition to obtain the text recognition result corresponding to the text image.

[0146] The process of processing the plurality of feature maps by using the graph convolution network and the language model can refer to Figure 5b . Specifically, a relationship subgraph corresponding to each feature map can be constructed first, and then the relationship subgraphs are merged to obtain a relationship graph, and the graph convolution network is used to process the merged relationship graph to obtain the feature representation of the nodes in the relationship graph. At the same time, the feature map can be input into the semantic feature extraction network to extract semantic features, and then the translation model is used to form the feature representation of the character sequence corresponding to the text to be recognized according to the semantic features.

[0147] After that, in order to fuse the feature representation of the nodes in the relationship graph and the feature representation of the character sequence, a linear layer can be used for dimension transformation and other processing to fuse the feature representation of the nodes in the relationship graph and the feature representation of the character sequence, and generate the text recognition result of the character image according to the fusion result.

[0148] The specific process of constructing a corresponding relationship subgraph for each feature map can refer to Figure 5c . Specifically, the sequential feature of the feature map, as well as the position feature and the category feature of each pixel point in the feature map can be determined first. Then, the sequential feature and the position feature can be fused and mapped to form a vector representation of each node, so as to obtain a feature matrix of the relationship subgraph composed of the vector representations of the nodes. At the same time, the similarity between the nodes indicated by each pixel point can be determined according to the category feature, and neighbor nodes are searched for each node according to the similarity between the nodes to construct edges, so as to form an adjacency matrix of the relationship subgraph. Then, the obtained feature matrix and adjacency matrix can be used to represent the relationship subgraph corresponding to the feature subgraph.

[0149] The method provided by the above embodiments of the present disclosure dynamically fuses the graph-based text recognition method and the language model-based text recognition method, so that the two recognition methods can learn from each other to utilize more information for text recognition, thereby improving the text recognition effect and better adapting to various text recognition situations such as complex background, uneven illumination, blurred image, and various text forms in natural scene text recognition.

[0150] Reference is made to Figure 6 , which shows a flow 600 of one embodiment of a training method of a graph convolution network, a language model and a segmentation network in a method for recognizing text according to the present disclosure. As Figure 6 shown, the graph convolution network, the language model and the segmentation network can be trained by the following steps:

[0151] Step 601, obtaining a labeled training data set and an unlabeled training data set.

[0152] In this embodiment, it should be noted that the training of the graph convolution network, the language model and the segmentation network can be performed by the execution subject of the above method for identifying text, or can be performed by other electronic devices. The execution subject for training the graph convolution network, the language model and the segmentation network can obtain the labeled training data set and the unlabeled training data set from a local or other storage device.

[0153] The labeled training data set is composed of training data with labels, and the unlabeled training data set is composed of training data without labels. The labeled training data set can include labels of different granularities. For example, for character-based text recognition, the labels can include character set labels and word-level labels.

[0154] Step 602, constructing a teacher-student network corresponding to the graph convolution network, the language model and the segmentation network, and training the teacher-student network using the labeled training data set, the unlabeled training data set and a preset loss function.

[0155] In this embodiment, the teacher-student network is a network structure in transfer learning. The structures of the teacher network and the student network can be completely the same or different. When the structures of the teacher network and the student network are the same, the parameter sharing of the structures of the teacher network and the student network can be realized based on the EMA (exponential moving average) technology.

[0156] In this embodiment, the input of the student network can include the labeled training data and the unlabeled training data, and the input of the teacher network can only include the unlabeled training data. The loss function can include a first loss function, a second loss function and a third loss function.

[0157] The first loss function and the second loss function can respectively represent the difference between the output result of the student network for the labeled training data and the labels of different granularities. For example, the first loss function can be used to measure the difference between the character-level recognition result output by the student network and the real character-level label, and the second loss function can be used to measure the difference between the word-level recognition result output by the student network and the real word-level label.

[0158] The third loss function can represent the difference between the output results of the student network and the teacher network for the unlabeled training data. Specifically, the parameters of the graph convolution network, the language model and the segmentation network can be adjusted by comprehensively considering the first loss function, the second loss function and the third loss function (such as optimizing the sum of the three loss functions) to complete the training of the graph convolution network, the language model and the segmentation network.

[0159] In addition, it should be noted that the training of various networks or models in the present disclosure can be completed based on various existing machine learning training methods, and details are not repeated.

[0160] The training methods for graph convolutional networks, language models, and segmentation networks provided in the above embodiments of this disclosure utilize labeled and unlabeled training data to train the graph convolutional networks, language models, and segmentation networks together. This solves the problems of limited real training data and difficulty in labeling real data in practice. It can also improve the generalization and robustness of the overall network composed of graph convolutional networks, language models, and segmentation networks, thereby helping to improve the accuracy of text recognition results.

[0161] Further reference Figure 7 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a device for recognizing text, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0162] like Figure 7 As shown, the device 700 for recognizing text provided in this embodiment includes a feature map acquisition unit 701, a relationship graph construction unit 702, a graph convolution processing unit 703, and a recognition unit 704. The feature map acquisition unit 701 is configured to acquire a feature map, which is obtained by performing text instance segmentation on an image presenting text to be recognized. The relationship graph construction unit 702 is configured to construct a relationship graph based on the feature map, where nodes in the relationship graph represent pixels in the feature map, and edges indicate that the similarity of the spatial semantic features of two connected nodes is greater than a target threshold. The spatial semantic features include the positional and category features of the pixels indicated by the nodes. The graph convolution processing unit 703 is configured to process the relationship graph using a pre-trained graph convolutional network to obtain a first text feature corresponding to the image. The recognition unit 704 is configured to generate a text recognition result for the image based on the first text feature.

[0163] In this embodiment, the specific processing of the feature map acquisition unit 701, the relationship graph construction unit 702, the graph convolution processing unit 703, and the recognition unit 704 in the text recognition device 700, and the resulting technical effects, can be found in references to [reference needed]. Figure 2 The relevant descriptions of steps 201, 202, 203 and 204 in the corresponding embodiments will not be repeated here.

[0164] In some optional implementations of this embodiment, the device 700 for recognizing text further includes: a language model processing unit (not shown in the figure), configured to process the feature map using a pre-trained language model to obtain a second text feature corresponding to the image; and the recognition unit 704 is further configured to generate a text recognition result of the image based on the first text feature and the second text feature.

[0165] In some optional implementations of the present embodiment, the recognition unit 704 is further configured to generate a text recognition result of the image according to the first text feature, the second text feature, and the feature map.

[0166] In some optional implementations of the present embodiment, the feature map is at least two feature maps, and the relationship graph construction unit 702 is further configured to, for a feature map in the at least two feature maps, construct a relationship subgraph corresponding to the feature map, and merge the relationship subgraphs corresponding to the respective feature maps to obtain the relationship graph.

[0167] In some optional implementations of the present embodiment, a feature map in the at least two feature maps is used to represent image features of an image region where each character in the text to be recognized is located and sequential features with other feature maps.

[0168] In some optional implementations of the present embodiment, the feature map is generated by: inputting the image into a pre-trained convolutional neural network to obtain an initial feature map; performing text instance segmentation on the initial feature map to obtain an instance feature map; performing text sequential segmentation on the initial feature map to obtain a sequential feature map; and fusing the instance feature map and the sequential feature map to obtain a feature map obtained by performing text instance segmentation on the image presenting the text to be recognized.

[0169] In some optional implementations of the present embodiment, the convolutional neural network is implemented based on a feature pyramid network and a residual network, wherein at least one convolutional layer before an output layer of the residual network has a step length of 1, and an output feature map of the residual network is used to generate an input feature map of the feature pyramid network through deformable convolution.

[0170] In some optional implementations of the present embodiment, the language model processing unit is further configured to: process the feature map by using a pre-trained semantic feature extraction network to obtain semantic features corresponding to the feature map, wherein the semantic feature extraction network includes a pooling layer for performing pooling processing on the feature map and a linear layer for performing linear transformation on an output of the pooling layer; and input the semantic features into the language model to obtain the second text feature.

[0171] In some optional implementations of the present embodiment, the spatial semantic feature further includes sequential features of the feature map where the pixel point indicated by the node is located, and for a node in the relationship subgraph, the spatial semantic feature of the node is generated by: obtaining an order value of the feature map corresponding to the node in the at least two feature maps; determining the sequential features of the feature map corresponding to the node according to the order value; and splicing the position feature, the category feature, and the determined sequential features of the pixel point indicated by the node to obtain the spatial semantic feature of the node.

[0172] In some optional implementations of the present embodiment, the similarity of the spatial semantic features of the two nodes is determined by the following steps: determining the Euclidean distance between the two nodes, and determining a first similarity according to the determined Euclidean distance; determining the cosine similarity of the category features corresponding to the two nodes respectively; and determining the similarity of the spatial semantic features of the two nodes according to the first similarity and the cosine similarity.

[0173] In some optional implementations of the present embodiment, the relationship graph construction unit 702 is further configured to sequentially connect the root nodes of adjacent relationship subgraphs according to the order relationship between the relationship subgraphs.

[0174] In some optional implementations of the present embodiment, the root node of the feature graph is determined by the following steps: for a node in the feature graph, determining the intersection-over-union of the node and each of the other nodes, wherein the intersection-over-union represents the ratio of the number of elements included in the intersection of the neighbor nodes of the two nodes to the number of elements included in the union of the neighbor nodes; and in response to determining that each of the intersection-overs-union corresponding to the node is not greater than a preset threshold, determining the node as the root node.

[0175] In some optional implementations of the present embodiment, the graph convolutional network comprises a first graph convolutional network and an output network, wherein the first graph convolutional network is configured to convert the feature matrix of the relationship graph, and the output network is configured to select nodes from each relationship subgraph according to the output of the first graph convolutional network, and aggregate the converted features corresponding to the selected nodes from each relationship subgraph to obtain the first text feature.

[0176] In some optional implementations of the present embodiment, the first graph convolutional network comprises a first graph convolutional subnetwork and a second graph convolutional subnetwork, wherein the first graph convolutional subnetwork is configured to convert the feature matrix of the relationship graph, and the second graph convolutional subnetwork is configured to convert the feature matrix output by the first graph convolutional subnetwork according to a relationship graph constructed based on the output of the first graph convolutional subnetwork.

[0177] In some optional implementations of the present embodiment, the recognition unit 704 is further configured to input the first text feature, the second text feature, and the feature graph into a pre-trained feature fusion network to generate the text recognition result of the image, wherein the feature fusion network is configured to concatenate the first text feature, the second text feature, and the feature graph, and perform linear transformation on the obtained concatenation result to obtain the text recognition result of the image.

[0178] In some optional implementations of the embodiment, the initial feature map is subjected to text instance segmentation to obtain an instance feature map, including: performing text instance segmentation on the initial feature map by using a pre-trained text instance segmentation network to obtain the instance feature map; and the initial feature map is subjected to text sequential segmentation to obtain a sequential feature map, including: performing text sequential segmentation on the initial feature map by using a pre-trained text sequential segmentation network to obtain the sequential feature map; and the graph convolution network, the language model and the segmentation network are trained by the following steps, where the segmentation network includes a convolutional neural network, the text instance segmentation network and the text sequential segmentation network: obtaining a labeled training dataset and an unlabeled training dataset, where the training data in the labeled training dataset includes labels of different granularities; constructing a teacher-student network corresponding to the graph convolution network, the language model and the segmentation network, and training the teacher-student network by using the labeled training dataset, the unlabeled training dataset and a preset loss function, where the input of the student network includes the labeled training data and the unlabeled training data, the input of the teacher network includes the unlabeled training data, the loss function includes a first loss function, a second loss function and a third loss function, the first loss function and the second loss function respectively represent the difference between the output result of the student network for the labeled training data and the labels of different granularities, and the third loss function represents the difference between the output results of the student network and the teacher network for the unlabeled training data.

[0179] The apparatus provided by the above embodiments of the present disclosure acquires, by the feature map acquisition unit, a feature map obtained by performing text instance segmentation on an image presenting to-be-recognized text; the relationship graph construction unit constructs a relationship graph according to the feature map, where a node in the relationship graph represents a pixel point in the feature map, and an edge represents that the similarity of the spatial semantic features of the two nodes connected by the edge is greater than a target threshold, and the spatial semantic features include the position feature and the category feature of the pixel point indicated by the node; the graph convolution processing unit processes the relationship graph by using a pre-trained graph convolution network to obtain a first text feature corresponding to the image; and the recognition unit generates a text recognition result of the image according to the first text feature, thereby realizing graph-based text recognition. This way can take into account the two-dimensional spatial information of the text in the image, which is helpful to improve the text recognition effect.

[0180] Reference will now be made to Figure 8 , which shows a structural schematic diagram of an electronic device (for example, a server in a digital TV) 800 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals) and the like, and fixed terminals such as digital TVs, desktop computers and the like. Figure 1 , which shows a structural schematic diagram of an electronic device (for example, a server in a digital TV) 800 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals) and the like, and fixed terminals such as digital TVs, desktop computers and the like.Figure 8 The illustrated server is merely one example and should not be taken as limiting the scope of functionality or use of embodiments of the disclosure.

[0181] As Figure 8 shown, the electronic device 800 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or loaded into a random access memory (RAM) 803 from a storage device 808. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0182] In general, the following devices can be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication devices 809 can allow the electronic device 800 to communicate wirelessly or wired with other devices to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that not all of the illustrated devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present. Figure 8 Each block shown in the flowcharts can represent a device or multiple devices as needed.

[0183] In particular, processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the disclosure. For example, embodiments of the disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 809, or installed from the storage devices 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-described functions defined in the methods of embodiments of the disclosure are performed.

[0184] It should be noted that the computer readable medium described in the embodiments of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the embodiments of the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0185] The computer readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device. The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to: obtain a feature map, wherein the feature map is obtained by performing text instance segmentation on an image in which text to be recognized is presented; construct a relationship graph according to the feature map, wherein a node in the relationship graph represents a pixel point in the feature map, and an edge represents that the similarity of the spatial semantic features of the two nodes connected by the edge is greater than a target threshold, and the spatial semantic features include the position feature and the category feature of the pixel points indicated by the nodes; process the relationship graph using a pre-trained graph convolution network to obtain a first text feature corresponding to the image; and generate a text recognition result of the image according to the first text feature.

[0186] Computer program code for carrying out operations of embodiments of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0187] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It is also noted that each block of the block diagrams and / or flow diagrams and combinations of blocks in the block diagrams and / or flow diagrams can be implemented by a dedicated hardware-based system that carries out specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0188] The units described in the embodiments of the present disclosure can be implemented by software, or can be implemented by hardware. The described units can also be arranged in a processor, for example, can be described as: a processor includes a feature map obtaining unit, a relationship graph constructing unit, a graph convolution processing unit, and an identifying unit. Among them, the names of these units do not constitute a limitation to the units themselves in some cases, for example, the feature map obtaining unit can also be described as: "obtaining a feature map, wherein the feature map is obtained by performing text instance segmentation on an image in which text to be identified is presented".

[0189] The above description is merely that of the preferred embodiments of the present disclosure and a description of the technical principles of the present disclosure. It should be understood by those skilled in the art that the inventive scope of the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the above inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) form the technical solutions.

Claims

1. A method for recognizing text, comprising: A feature map is obtained by performing text instance segmentation on an image containing text to be recognized. A relationship graph is constructed based on the feature graph, wherein nodes in the relationship graph represent pixels in the feature graph, and edges represent the similarity of the spatial semantic features of two connected nodes being greater than a target threshold. The spatial semantic features include the positional features and category features of the pixels indicated by the nodes. The relationship graph is processed using a pre-trained graph convolutional network to obtain the first text feature corresponding to the image; The feature map is processed using a pre-trained language model to obtain the second text features corresponding to the image; Based on the first text feature and the second text feature, the text recognition result of the image is generated.

2. The method according to claim 1, wherein, The step of generating the text recognition result of the image based on the first text feature and the second text feature includes: Based on the first text features, the second text features, and the feature map, the text recognition result of the image is generated.

3. The method according to claim 2, wherein, The feature map consists of at least two feature maps; as well as The step of constructing a relationship graph based on the feature graph includes: For each feature map in the at least two feature maps, construct the corresponding relation subgraph for that feature map; The relationship subgraphs corresponding to each feature map are merged to obtain the relationship graph.

4. The method according to claim 3, wherein, The feature maps in the at least two feature maps are used to characterize the image features of the image regions where each character in the text to be identified is located and the order features between them and other feature maps.

5. The method according to claim 4, wherein, The feature map is generated through the following steps: The image is input into a pre-trained convolutional neural network to obtain an initial feature map; The initial feature map is segmented into text instances to obtain an instance feature map; The initial feature map is subjected to text sequence segmentation to obtain a sequence feature map; By fusing the instance feature map and the sequence feature map, the feature map obtained by performing text instance segmentation on the image presenting the text to be identified is obtained.

6. The method according to claim 5, wherein, The convolutional neural network is implemented based on a feature pyramid network and a residual network. The stride of at least one convolutional layer before the output layer of the residual network is 1. The feature map output by the residual network generates the input feature map of the feature pyramid network through deformable convolution.

7. The method according to claim 1, wherein, The process of processing the feature map using a pre-trained language model to obtain the second text features corresponding to the image includes: The feature map is processed using a pre-trained semantic feature extraction network to obtain the semantic features corresponding to the feature map. The semantic feature extraction network includes a pooling layer that performs pooling processing on the feature map and a linear layer that performs linear transformation on the output of the pooling layer. The semantic features are input into the language model to obtain the second text features.

8. The method according to claim 3, wherein, The spatial semantic features also include the order features of the feature maps where the pixels indicated by the nodes are located; and For a node in the relation subgraph, the spatial semantic features of that node are generated through the following steps: Obtain the order value of the feature map corresponding to the node in the at least two feature maps; Based on the sequence value, determine the sequence features of the feature map corresponding to the node; By concatenating the positional features, category features, and determined order features of the pixels indicated by the node, the spatial semantic features of the node are obtained.

9. The method according to claim 3, wherein, The similarity of the spatial semantic features of two nodes is determined through the following steps: Determine the Euclidean distance between two nodes, and determine the first similarity based on the determined Euclidean distance; Determine the cosine similarity of the category features corresponding to the two nodes; The similarity of spatial semantic features between two nodes is determined based on the first similarity and the cosine similarity.

10. The method according to claim 3, wherein, The merging of the relational subgraphs corresponding to each feature graph includes: Connect the root nodes of adjacent relational subgraphs in sequence according to the order of the relational subgraphs.

11. The method according to claim 10, wherein, The root node of the feature map is determined through the following steps: For a node in the feature graph, determine the intersection-union ratio of the node with each of the other nodes, where the intersection-union ratio represents the ratio of the number of elements in the intersection of the neighboring nodes of two nodes to the number of elements in the union of the neighboring nodes. In response to determining that the intersection-union ratios of the nodes are not greater than a preset threshold, the node is determined to be the root node.

12. The method according to claim 3, wherein, The graph convolutional network includes a first graph convolutional network and an output network. The first graph convolutional network is used to transform the feature matrix of the relational graph, and the output network is used to select nodes from each relational subgraph according to the output of the first graph convolutional network, and to aggregate the transformed features corresponding to the nodes selected from each relational subgraph to obtain the first text feature.

13. The method according to claim 12, wherein, The first graph convolutional network includes a first graph convolutional sub-network and a second graph convolutional sub-network. The first graph convolutional sub-network is used to transform the feature matrix of the relational graph, and the second graph convolutional network is used to transform the feature matrix output by the first graph convolutional network based on the relational graph constructed based on the output of the first graph convolutional network.

14. The method according to claim 2, wherein, The step of generating the text recognition result of the image based on the first text feature, the second text feature, and the feature map includes: The first text feature, the second text feature, and the feature map are input into a pre-trained feature fusion network to generate the text recognition result of the image. The feature fusion network is used to concatenate the first text feature, the second text feature, and the feature map, and to perform a linear transformation on the concatenated result to obtain the text recognition result of the image.

15. The method according to claim 5, wherein, The step of performing text instance segmentation on the initial feature map to obtain an instance feature map includes: performing text instance segmentation on the initial feature map using a pre-trained text instance segmentation network to obtain an instance feature map; and The step of performing text sequence segmentation on the initial feature map to obtain a sequence feature map includes: performing text sequence segmentation on the initial feature map using a pre-trained text sequence segmentation network to obtain a sequence feature map; and The graph convolutional network, language model, and segmentation network are trained through the following steps, wherein the segmentation network includes the convolutional neural network, the text instance segmentation network, and the text sequence segmentation network: Obtain labeled training datasets and unlabeled training datasets, where the training data in the labeled training datasets includes labels of different granularities; The teacher-student networks corresponding to the graph convolutional network, language model, and segmentation network are constructed, and the teacher-student networks are trained using the labeled training dataset, the unlabeled training dataset, and a preset loss function. The input of the student network includes labeled training data and unlabeled training data, and the input of the teacher network includes unlabeled training data. The loss function includes a first loss function, a second loss function, and a third loss function. The first and second loss functions represent the differences between the output results of the student network on labeled training data and labels of different granularities, respectively. The third loss function represents the differences between the output results of the student network and the teacher network on unlabeled training data, respectively.

16. An apparatus for recognizing text, wherein, The device includes: The feature map acquisition unit is configured to acquire a feature map, wherein the feature map is obtained by performing text instance segmentation on an image presenting text to be recognized; A relation graph construction unit is configured to construct a relation graph based on the feature graph, wherein nodes in the relation graph represent pixels in the feature graph, and edges represent that the similarity of the spatial semantic features of two connected nodes is greater than a target threshold, and the spatial semantic features include the positional features and category features of the pixels indicated by the nodes; The graph convolution processing unit is configured to process the relation graph using a pre-trained graph convolution network to obtain the first text feature corresponding to the image; The language model processing unit is configured to process the feature map using a pre-trained language model to obtain the second text features corresponding to the image; The recognition unit is configured to generate a text recognition result of the image based on the first text feature and the second text feature.

17. An electronic device comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-15.

18. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-15.

Images