A visual text detection method based on stroke region segmentation strategy

Abstract

A visual text detection method based on a stroke region segmentation strategy comprises the following steps: S1. A front-end processing module based on a convolutional neural network performs feature extraction and multi-level region prediction on an input text image; S2. Based on the multi-level prediction results of the text regions, text-level and stroke-level region candidate boxes are extracted, and a hierarchical local graph structure is constructed; S3. A back-end processing module based on a graph neural network performs node feature aggregation and relationship reasoning based on multi-level graph nodes in each local graph, infers the relationships between nodes at different levels of the graph, performs link prediction, and groups the nodes to form an overall text instance detection result. Experiments were conducted on a standard evaluation dataset widely used in the field of visual text detection research, verifying the effectiveness, high accuracy, and good generalization ability of the proposed visual text detection method.

Description

Technical Field

[0001] The present invention relates to a visual text detection technology, and in particular to a visual text detection method based on a stroke region segmentation strategy. Background Art

[0002] Visual text detection in complex real-world scenes aims to label each text instance in an input image as an enclosed region of arbitrary shape. This technique has been widely applied to tasks in multimedia signal processing, including image text editing, optical character recognition (OCR), and image-to-text translation. With the rapid development of convolutional neural network (CNN) models, current mainstream text detectors are primarily derived from object detection or object segmentation frameworks in computer vision, primarily including regression-based and segmentation-based text detection methods. Regression-based text detection methods are typically based on general-purpose object detectors and localize text boxes by predicting anchor boxes or pixel offsets. While effective to a certain extent, these methods often require complex anchor box configuration strategies and elaborate post-processing pipelines, which limit their ability to represent text of arbitrary shapes and hinder their large-scale application in real-world scenarios. Segmentation-based text detection methods typically combine pixel-level prediction and post-processing steps to extract text instances from the text regions predicted by segmentation. Compared to regression-based text detection methods, segmentation-based text detection methods are often able to more accurately localize text instances of arbitrary shapes. However, such methods [4,12] usually require time-consuming post-processing steps and have difficulty in effectively identifying and separating multiple text instances that are close to each other.

[0003] Recently, some hybrid text detection methods have been proposed in this research field to combine the core ideas of the above two methods. Hybrid text detection methods usually first perform pixel-level segmentation prediction to search for potential text regions, and then adopt a bounding box regression strategy to guide the final text detection results. In this research branch, DeepReg predicts offsets from the pixels of the text region to guide multi-directional text box regression prediction. Subsequently, some works have attempted to utilize the powerful capabilities of graph neural networks (GNNs) to improve text detection performance by modeling and reasoning about text regions. Among them, GraphText introduced a deep relational reasoning graph network as the backend network module in the text detection framework. In addition, StrokeNet first predicts a multi-level representation of each text region and then performs structural reasoning based on a hierarchical relational graph network model.

[0004] However, traditional methods usually have difficulty in accurately locating text instances of arbitrary shapes, and also have difficulty in effectively identifying and separating multiple text instances that are close to each other.

[0005] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0006] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide a high-precision visual text detection method based on stroke region segmentation strategy.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A visual text detection method based on a stroke region segmentation strategy comprises the following steps:

[0009] S1. Perform feature extraction and multi-level region prediction on an input text image through a front-end processing module based on a convolutional neural network; the front-end processing module includes a backbone image feature extraction network, a text region prediction network, and a stroke region prediction network, and performs multi-level prediction related to text regions through a series of convolutional layers stacked on top of the backbone image feature extraction network of a feature pyramid;

[0010] S2. Extract text-level and stroke-level region candidate boxes based on the multi-level prediction results of the text region. The image regions represented by each candidate box serve as graph nodes to form multiple local graph structures, thereby constructing a hierarchical local graph structure.

[0011] S3. Through the back-end processing module based on graph neural network, node feature aggregation and relationship reasoning based on multi-level graph nodes are performed in each local graph, the relationship between graph nodes at different levels is inferred and link prediction is performed, and the nodes are grouped according to the link relationship between each text-level node to form the overall text instance detection result.

[0012] Furthermore, in step S1, the multi-level prediction includes: obtaining the classification confidence of the text-level rectangular area corresponding to each text instance; regression prediction of relevant text-level attributes such as text rotation angle and centerline position in each text instance; and character segmentation prediction corresponding to the stroke level within the predicted bounding box of each text area.

[0013] Furthermore, in step S1, the text-level and stroke-level region candidate frames are extracted based on the multi-level prediction results of the text region, including: extracting corresponding multi-level candidate rectangular frames based on the multi-level prediction results of the text region, a local graph that only contains text-level or stroke-level nodes is a isomorphic graph, and a local graph that contains both text-level and stroke-level nodes is a heterogeneous graph.

[0014] Furthermore, in step S1, the text region prediction network predicts attributes related to the text instance region, including: predicting the classification probability of the text region TR and the text center region TCR, and then predicting the regression values ​​of h1, h2, cosθ and sinθ, where h1 and h2 represent the distances from the current pixel to the upper edge of TR and the lower edge of TR, respectively, the text instance height scale h is the sum of h1 and h2, and θ indicates the direction information of the text instance; estimating the text center line corresponding to TR based on the predicted TR potential region; using the feature outputs of two channels to guide the classification probability prediction of TR and TCR; during training, the first feature channel is used to predict the background, and the second channel is used to predict the foreground, i.e., the text instance; during testing, the foreground prediction result of the second feature channel is taken for subsequent processing; wherein, the output of one feature channel is used to predict each regression attribute value.

[0015] Furthermore, in step S1, the stroke region prediction network separates the character content in each text region from the complex background, wherein the low-level semantics and high-level semantic information of the image are combined to generate a fine stroke segmentation representation in the text region to guide the subsequent text detection process;

[0016] Preferably, the stroke region prediction network includes a two-stage prediction process;

[0017] 1) Extract text-related features from the high-level feature representation of the input image obtained by the backbone network; specifically, crop the outer rectangle OTR of the TR from the input image, and use a global pooling layer combined with continuous convolutional layers to extract the OTR region features obtained from the backbone network; use several pooling layers and a multi-layer perceptron network and related nonlinear activation functions to calculate the channel attention feature map of the input image to identify and measure the relative contribution of different network layers in the backbone network to the text region representation; during this period, the extracted input feature map is upsampled to the same resolution as the input image, and then multiplied by the obtained channel attention feature map to achieve the semantic information distillation operation of the input image; thereby, the semantic representation of the text image is obtained;

[0018] 2) Finely model the stroke representation of the text region and enhance the fine-grained stroke character segmentation representation by introducing an orthogonal convolutional network from the orthogonal direction; specifically, the 3-channel RGB raw input features of the text region circumscribed rectangle (OTR) are used as supplementary low-level image semantic information and fused with the obtained text image semantic representation; preferably, orthogonal convolutional layers with convolution kernel sizes of 1×7 and 7×1 are introduced to calculate the attention coefficient along the spatial direction, and the obtained attention value is then multiplied by the fused text feature map.

[0019] Furthermore, the front-end processing module uses a dataset with stroke-level segmentation map annotations as labels and a mean square error loss function to pre-train the stroke region prediction network.

[0020] Furthermore, in step S3, the node features and their connection structure are initialized, specifically including:

[0021] Initializing node features: Two complementary feature representations, geometric embedding and content embedding, are used for feature initialization of text-level and stroke-level nodes. For geometric embedding, the geometric properties of each predicted region candidate box are encoded into a high-dimensional space. For content embedding, the content features of each graph node are obtained by sending the predicted feature map of the geometric properties of each region candidate box to the RRoI-Align layer. The two feature embeddings are concatenated to form the final graph node feature representation. Preferably, the initial feature representations of all nodes are normalized by subtracting the features of the central node when generating the local graph network.

[0022] Adjacency matrix generation: The topological structure formed by each local graph network is encoded in the adjacency matrix A∈R N×N , where A(c,n)=1, if there is a connection between the central node c and each of its neighbor nodes n in the local graph; preferably, the method of generating the adjacency matrix specifically includes:

[0023] For isomorphic graphs, including the construction of isomorphic stroke-level graph networks and text-level graph networks; for isomorphic stroke graphs containing only stroke-level graph nodes, the KNN nearest neighbor algorithm based on Euclidean distance is adopted, and the 8 nearest neighbor nodes of each central node are selected as the 1-hop neighbor nodes of the central node to form the adjacency matrix A s For the isomorphic text graph containing only text-level graph nodes, the difference between the adjacency matrix construction method and the isomorphic stroke graph is that each central node in the isomorphic text graph only retains the four closest direct neighbor nodes to form the corresponding matrix A t ;

[0024] For heterogeneous text graph networks, graph nodes at both text and stroke levels are included. This type of graph network is constructed based on the Euclidean distance between the center positions of each extracted region candidate box. Specifically, each text-level region candidate box is regarded as the central node of the heterogeneous text graph, and the connection relationship within the 1-hop and 2-hop neighborhood of the central node is used to generate the adjacency matrix A of the heterogeneous graph. h ; The 1-hop neighborhood of the central node contains the 4 nearest neighbor nodes in the text-level graph, while its 2-hop neighborhood contains the additional 4 nearest neighbor nodes in the stroke-level graph.

[0025] Furthermore, in step S3, the backend processing module performs node relationship reasoning and link prediction in the generated multiple local graph networks through a hierarchical graph neural network reasoning model. The reasoning process based on the graph neural network includes the following three stages:

[0026] First, a weighted average approach is used to aggregate and update the stroke-level node features guided by the attention mechanism; the weight information in the weighted process comes from two parts, including the normalized adjacency matrix A s and the attention coefficient α between any two graph nodes v and u derived from the graph attention network GAT v,u ; The weighted aggregation process of the first stage is described as:

[0027]

[0028] Among them, σ is the activation function, W is the trainable weight parameter, s k Represents the features of the stroke-level graph node k;

[0029]

[0030] is derived from As, Fusr(·) represents the characteristic linear combination function;

[0031] If the center of a stroke-level graph node falls within the region of a text-level graph node, merging the updated representation of the stroke-level node into the corresponding text-level node representation;

[0032] The second stage fuses the features of two-level graph nodes by stacking two Transformer encoder modules. Specifically, the introduced Transformer encoder captures the attention coefficients between stroke-stroke, stroke-text, and text-text nodes to model and reason about the hierarchical structural relationships between heterogeneous graph nodes, which can be expressed as:

[0033]

[0034] in, represents all text (t) node features and stroke (s) node features of layer t; Attention (·) is the attention calculation operation in Transformer, Q, K, V represent the query matrix, key matrix and value matrix respectively, and W Q(,K,V) is a trainable weight parameter;

[0035] Using a graph reasoning network with expanded neighborhood scope, for each text-level graph node, the feature representations of its one-hop neighbor nodes containing only text neighbors are aggregated in the first layer of the designed graph network. Subsequently, the information of its two-hop neighbors containing both text neighbors and stroke neighbors is aggregated in subsequent layers. During this period, dynamic graph convolution is used to adaptively adjust the network structure of the heterogeneous graph, which is described as follows:

[0036] P=σ(M t,s ,A t,s (G(H t,s ))W)

[0037] Where W is a trainable weight matrix, G(·) represents the traditional information aggregation process on the graph network, and M t,s and A t,s They represent the cross-layer masking matrix and cross-hop attention matrix introduced in the dynamic graph network respectively;

[0038] Preferably, the cross-layer shielding matrix M t,s Further divided into M' s 、M' t and M′ t,s , respectively represent the self-masking matrix between the stroke-level graph nodes, the self-masking matrix between the text-level graph nodes, and the mutual-masking matrix between the stroke-level and text-level graph nodes; the masking result of the stroke-level graph node is finally based on M' s and M' t,s The linear combination of M' and the comparison of fixed thresholds are used, and the masking results of the text level graph nodes are finally based on M' t and M′ t,s Comparison of linear combinations and fixed thresholds;

[0039] After completing the above three stages, the output of the last graph network layer is used to predict the link relationship between text graph nodes and locate the text instance bounding box regression value.

[0040] Preferably, the cross entropy loss between the graphical model prediction results and the corresponding true category labels is used during the training process to guide the learning process of the entire detection framework.

[0041] Preferably, according to the classification and link prediction results of the graph nodes, the text-level nodes are grouped by a breadth-first search method and sorted by a minimum path algorithm.

[0042] Preferably, the boundary of the arbitrary shape text instance is obtained by sequentially connecting the midpoints of the top and bottom of the candidate boxes corresponding to the sorted text nodes.

[0043] A computer-readable storage medium, wherein when the computer program is executed by a processor, the visual text detection method based on the stroke region segmentation strategy is implemented.

[0044] The present invention has the following beneficial effects:

[0045] The present invention proposes a visual text detection method based on a stroke region segmentation strategy, which can effectively achieve high-precision visual text detection. First, by introducing a lightweight stroke segmentation prediction network, as an effective supplement to the current mainstream text detector that can only achieve text region prediction, the detection model can achieve a multi-level (text level, stroke level) representation of the text region. During this period, a visual image dataset (SceneText) can be introduced, in which the text instances in each image sample are annotated with stroke-level segmentation labels, that is, a binary stroke character segmentation map. This dataset will improve the prediction accuracy of the detection framework for the multi-level representation of text regions by pre-training the front-end processing module based on convolutional neural networks in the detection framework. At the same time, by introducing a graph neural network model as the back-end processing module in the constructed text detection framework, it can effectively perform feature aggregation and relationship reasoning on each part of the text region predicted by the front-end processing module, so that the improved graph model can better adapt to the text detection task scenario. The detection method of the present invention was experimented on a standard evaluation dataset widely used in the field of visual text detection research, verifying the effectiveness, high precision and good generalization ability of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The text detection framework and its processing flow chart of an embodiment of the present invention.

[0047] Figure 2 : (a) original image; (b) attribute prediction related to text region (TR); (c) outer rectangle (OTR) corresponding to TR in an embodiment of the present invention.

[0048] Figure 3 Schematic diagram of the network structure of the stroke region prediction network according to an embodiment of the present invention.

[0049] Figure 4 In an embodiment of the present invention, the introduced SceneText text image dataset and the mean square error loss function are used to pre-train the proposed stroke region prediction network, and then the trained stroke prediction network is used to online predict the stroke representation of text images in real scenes.

[0050] Figure 5 This figure visualizes the text detection results of an embodiment of the present invention. In the figure, the first column (a), the second column (b), and the third column (c) respectively represent the input image, the stroke segmentation predicted by the proposed method, and the final text detection result.

[0051] Figure 6An example of an OCR translation task (Chinese to English and English to French) guided by an embodiment of the present invention, including: input image (a), predicted stroke segmentation (b), text detection results (c), and translated image (d).

[0052] Figure 7 The front-end processing module in the embodiment of the present invention is used to perform the overall processing flow of the OCR translation task. DETAILED DESCRIPTION

[0053] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0054] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and coupling or communication.

[0055] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0056] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0057] In previous detection methods, the front-end processing modules based on convolutional neural networks have limited prediction accuracy for text regions, which greatly restricts the ability of the back-end processing modules based on graph neural networks to reason and generate final text detection results. Furthermore, the direct migration of classic graph neural networks or their variants to the field of text detection without the introduction of effective optimization strategies makes it difficult for the graph network modules constructed in these methods to be seamlessly compatible with text detection-related tasks. These shortcomings have, to a certain extent, limited the application potential of graph models in the field of visual text detection.

[0058] See Figure 1 The embodiment of the present invention provides a visual text detection method based on a stroke region segmentation strategy, comprising the following steps:

[0059] S1. Perform feature extraction and multi-level region prediction on an input text image through a front-end processing module based on a convolutional neural network; the front-end processing module includes a backbone image feature extraction network, a text region prediction network, and a stroke region prediction network, and performs multi-level prediction related to text regions through a series of convolutional layers stacked on top of the backbone image feature extraction network of a feature pyramid;

[0060] S2. Extract text-level and stroke-level region candidate boxes based on the multi-level prediction results of the text region. The image regions represented by each candidate box serve as graph nodes to form multiple local graph structures, thereby constructing a hierarchical local graph structure.

[0061] S3. Through the back-end processing module based on graph neural network, node feature aggregation and relationship reasoning based on multi-level graph nodes are performed in each local graph, the relationship between graph nodes at different levels is inferred and link prediction is performed, and the nodes are grouped according to the link relationship between each text-level node to form the overall text instance detection result.

[0062] The present invention proposes a visual text detection method based on a stroke region segmentation strategy, which can effectively achieve high-precision visual text detection. First, by introducing a lightweight stroke segmentation prediction network, as an effective supplement to the current mainstream text detector that can only achieve text region prediction, the detection model can achieve multi-level (text level, stroke level) representation of the text region. At the same time, by introducing a graph neural network model as a back-end processing module in the constructed text detection framework, it can effectively perform feature aggregation and relationship reasoning on each part of the text region predicted by the front-end processing module, so that the improved graph model can better adapt to the text detection task scenario. The detection method of the present invention was experimented on a standard evaluation dataset widely used in the field of visual text detection research, verifying the effectiveness, high precision and good generalization ability of the method of the present invention.

[0063] During this period, a visual image dataset (SceneText) can be introduced. Each text instance in its image sample is annotated with stroke-level segmentation labels, namely, binary stroke character segmentation maps. This dataset will be used to pre-train the convolutional neural network-based front-end processing module of the detection framework to improve the detection framework's prediction accuracy of multi-level representations of text regions.

[0064] Specific embodiments of the present invention are further described below.

[0065] (1) Overview of the detection framework

[0066] The processing flow of the technical framework proposed by the present invention is as follows Figure 1 As shown in the figure, it consists of two main network modules, including a front-end processing module based on convolutional neural network and a back-end processing module based on graph neural network. Figure 1 The technical details and processing flow of the text detection framework proposed in this invention are shown.

[0067] First, the present invention adopts a text region detector based on a convolutional neural network, which serves as a front-end processing module to perform feature extraction and multi-level region prediction for each input text image. Specifically, the front-end processing module uses a series of convolutional layers stacked on top of a feature pyramid backbone image feature extraction network to perform multi-level predictions related to text regions, including: 1. Classification confidence of the text-level rectangular region (bounding-box-level) corresponding to each text instance, 2. Regression prediction of relevant text-level attributes such as text rotation angle and centerline position within each text instance, and 3. Character segmentation prediction at the corresponding stroke level within the predicted bounding box of each text region. The present invention uses the introduced SceneText dataset to pre-train the front-end processing module.

[0068] Afterwards, the corresponding multi-level candidate rectangular boxes are extracted based on the multi-level prediction results of the text area, where the image area represented by each candidate rectangular box is regarded as a graph node, and all the generated graph nodes will form multiple local graph structures. Specifically, a local graph network that only contains text-level (stroke-level) nodes is called a homogeneous text (stroke) graph, while a local graph network that contains both text-level and stroke-level nodes is called a heterogeneous text graph. On this basis, the present invention proposes a back-end processing module based on a graph neural network to perform feature aggregation and relationship reasoning based on multi-level graph nodes in each local graph. The proposed module will infer the possibility of edge links between text-level graph nodes, and finally group the nodes according to the link relationship between each text-level node to form an overall text instance detection result. During the training process, the cross-entropy loss between the graph model prediction results and the corresponding true category labels is used to guide the learning process of the entire detection framework.

[0069] (2) Front-end processing module

[0070] Text Region Prediction Network: Because the input image-related features extracted from the backbone feature extraction network retain their spatial resolution and contain rich image semantic information, the proposed method further applies a series of convolutional layers stacked on top of the backbone network to predict attributes related to text instance regions. Specifically, this stage requires predicting the classification probabilities of the text region (denoted by TR) and the text center region (denoted by TCR), followed by the regression values ​​of h1, h2, cosθ, and sinθ. TR represents the region where the text instance is located. Based on the predicted TR potential region, the proposed method estimates the text centerline corresponding to TR and shrinks the end pixels of the centerline by 0.5 of the text instance width. The centerline region is then further expanded by 0.3 of the text instance height to obtain the final text centerline (TCR). The proposed network uses feature outputs from two channels to guide the classification probability prediction of TR and TCR, and both undergo subsequent processing such as softmax normalization and thresholding to obtain the final prediction results. Specifically, during training, the first feature channel is used to predict the background, and the second channel is used to predict the foreground (i.e., the text instance). During testing, the present invention directly takes the foreground prediction result of the second feature channel for subsequent processing. In addition, h1 and h2 represent the distance from the current pixel to the upper edge of TR and the lower edge of TR respectively, and the text instance height scale h mentioned above is the sum of h1 and h2, and θ is used to indicate the direction information of the text instance. The present invention uses the output of one feature channel to predict each regression attribute value. The representation of related attributes is as follows Figure 2 shown. Figure 2 In the figure, (a) original image; (b) attribute prediction related to text region (TR); (c) outer rectangle (OTR) corresponding to TR.

[0071] Stroke region prediction network: Based on the text region prediction, the present invention further explores a more fine-grained text stroke representation, thereby separating the character content in each text region from the complex background. The prediction process can be divided into two stages, such as Figure 3 shown.

[0072] Figure 3 The network structure of the stroke region prediction network is shown. Specifically, the present invention combines low-level semantics and high-level semantic information of the image to generate a fine stroke segmentation representation in the text region, and uses this to guide the subsequent text detection process.

[0073] First, the present invention extracts text-related features from the high-level feature representation of the input image obtained by the backbone network. Specifically, the method of the present invention crops the outer rectangle of TR (represented by OTR) from the input image, and uses a global pooling layer combined with a continuous convolutional layer to extract the OTR region features obtained from the backbone network. The present invention uses several pooling layers and a multi-layer perceptron network and related nonlinear activation functions to calculate the channel attention feature map of the input image to identify and measure the relative contribution of different network layers in the backbone network to the text region representation. During this period, the present invention upsamples the extracted input feature map to the same resolution size as the input image, and then multiplies it by the channel attention feature map obtained above to achieve the semantic information distillation operation of the input image. This process enables the method of the present invention to obtain rich semantic representations of text images, including information such as color, texture and edges of the text.

[0074] Secondly, the present invention introduces a stroke enhancement operator to finely model the stroke representation of the text region. Considering that the stroke content in each text instance can be understood as a closed area surrounded by a series of edges, inspired by the existing edge detection related methods, the method of the present invention effectively enhances the fine-grained stroke character segmentation representation by introducing an orthogonal convolutional network from the orthogonal direction. Specifically, the method of the present invention uses the 3-channel RGB original input features of the text region circumscribed rectangle (OTR) as supplementary low-level image semantic information, and fuses it with the previously obtained semantically rich text representation. During this period, the method of the present invention further introduces orthogonal convolution layers with convolution kernel sizes of 1×7 and 7×1 to calculate the attention coefficient along the spatial direction, and then multiplies the obtained attention value by the fused text feature map to suppress the interference of irrelevant information such as noise and other background. This process helps to enhance the representation of the stroke level of the text region, enabling the model to effectively generate more fine-grained stroke content containing complex texture details.

[0075] (3)Introduced SceneText dataset

[0076] The embodiment of the present invention also introduces a new text dataset with stroke-level segmentation map annotations as labels, namely SceneText, which is used to pre-train the front-end processing module based on convolutional neural network proposed in the present invention (such as Figure 4 The SceneText dataset contains approximately 200K text images and is an image dataset containing text in multiple languages, including English, Chinese, and Japanese.

[0077] Figure 4It is shown that the method of the present invention uses the introduced SceneText text image dataset and the mean square error loss function to pre-train the proposed stroke region prediction network, and then the trained stroke prediction network is used to online predict the stroke representation of text images in real scenes.

[0078] (4) Backend processing module

[0079] The method of the present invention constructs multiple local graph structure networks by extracting multiple levels of region candidates (text-level and stroke-level) within each predicted text region and treating each candidate as a graph node. For the text-level and stroke-level nodes obtained using the above method, the node features and their connection structure are then initialized using the following method.

[0080] Node feature initialization: The method proposed in the present invention adopts two complementary feature representations, including geometric embedding and content embedding, for feature initialization of text-level and stroke-level nodes. For geometric embedding, the method of the present invention applies trigonometric functions to encode the predicted geometric properties (including the center point coordinates, width, height and rotation angle of each region candidate box) into a high-dimensional space. For content embedding, the method of the present invention obtains the content features of each graph node by sending the predicted feature map of the geometric related attributes of each region candidate box to the RRoI-Align layer. In the above manner, the network connects the two feature embeddings obtained to form the final graph node feature representation. It is worth noting that the method of the present invention normalizes the initial feature representation of all nodes by subtracting the features of the central node when generating a local graph network.

[0081] Adjacency matrix generation: The topological structure formed by each local graph network is encoded in the adjacency matrix A∈R N×N , where A(c,n)=1 if there is a connection between the central node c and each of its neighbor nodes n in the local graph. Subsequently, the present invention explores two methods for generating adjacency matrices and applies different information aggregation functions in the constructed adjacency structure to verify their impact on text detection performance.

[0082] The first category is isomorphic graphs, including the construction of isomorphic stroke-level graph networks and text-level graph networks. For isomorphic stroke graphs containing only stroke-level graph nodes, the present invention adopts the KNN nearest neighbor algorithm based on Euclidean distance and selects the 8 nearest neighbor nodes of each central node as the direct (1-hop) neighbor nodes of the central node to form the adjacency matrix A. s For the isomorphic text graph containing only text-level graph nodes, the present invention adopts the same adjacency matrix construction method as the above (isomorphic stroke graph), but the difference is that each central node in the isomorphic text graph only retains the four closest direct neighbor nodes to form the corresponding matrix A t .

[0083] The second type is for heterogeneous text graph networks, which contain graph nodes at both text and stroke levels. The present invention constructs this type of graph network based on the Euclidean distance between the center positions of each extracted region candidate box. Specifically, each text-level region candidate box is regarded as the center node of the heterogeneous text graph, and the connection relationship within the 1-hop and 2-hop neighborhood of the center node is used to generate the adjacency matrix A of the heterogeneous graph. h In the present invention, the 1-hop neighborhood of a central node contains the 4 nearest neighbor nodes in the text-level graph, while its 2-hop neighborhood contains an additional 4 nearest neighbor nodes in the stroke-level graph. The advantage of this setting is that the limited number of neighbors facilitates effective relational reasoning and efficient feature learning, while the introduced high-order neighbor nodes (2-hop neighborhood) can provide auxiliary information beyond the local structure in the graph.

[0084] Graph Reasoning: Building on the aforementioned processing, our method considers graph nodes at both the text and stroke levels and further introduces a hierarchical graph neural network reasoning model to perform node relationship reasoning and link prediction within the generated multiple local graph networks. The graph neural network-based reasoning process can be broken down into three phases.

[0085] First, the method of the present invention adopts a weighted average approach to aggregate and update the stroke-level node features guided by the attention mechanism. The weight information in the weighting process comes from two parts, including the normalized adjacency matrix A s and the attention coefficient α between any two graph nodes v and u derived in the Graph Attention Network (GAT) v,u In this way, the weighted aggregation process in the first stage can be described as:

[0086]

[0087] Among them, σ is the activation function, W is the trainable weight parameter, s k Represents the features of stroke-level graph node k.

[0088] also,

[0089]

[0090] here From As, Fuse(·) represents the characteristic linear combination function.

[0091] Afterwards, if the center of the former (stroke-level graph node) falls in the area of ​​the latter (text-level graph node), the updated representation of the stroke-level node is merged into the corresponding text-level node representation. In addition, considering that different stroke-level graph nodes contain information from different parts of the text region and have different contributions to the corresponding text-level graph nodes, the above-mentioned stroke-level node feature update strategy can introduce more discriminative and expressive stroke representations for the subsequent two-level node feature fusion process. Therefore, the second stage fuses the features of the two-level graph nodes by stacking two Transformer encoder modules. Specifically, the introduced Transformer encoder captures the attention coefficients between stroke-stroke, stroke-text, and text-text nodes, thereby achieving effective modeling and reasoning of hierarchical structural relationships between heterogeneous graph nodes. The process can be expressed as:

[0092]

[0093] in, Represents all text (t) node features and stroke (s) node features of layer t. Attention (·) is the attention calculation operation in Transformer, Q, K, V represent the query matrix, key matrix and value matrix respectively, W Q(,K,V) are trainable weight parameters.

[0094] Based on the above two-stage processing, the present invention further proposes a graph reasoning network that includes a larger neighborhood range. Specifically, for each text-level graph node, the present invention aggregates the feature representations of its one-hop neighbor nodes (including only text neighbor nodes) in the first layer of the designed graph network, and then aggregates the information of its two-hop neighbors (including both text neighbor nodes and stroke neighbor nodes) in subsequent layers. During this process, the present invention method uses dynamic graph convolution to adaptively adjust the network structure of the heterogeneous graph. This process can be described as follows:

[0095] P=σ(M t,s ,A t,s (G(H t,s ))W)

[0096] Where W is a trainable weight matrix, G(·) represents the traditional information aggregation process on the graph network, that is, the representation update process of each central node is a linear combination of the features of its neighboring nodes. In addition, M t,s and A t,s where are the cross-layer masking matrix and the cross-hop attention matrix introduced in the dynamic graph network. The former restricts the aggregation process to the dynamic sub-portion of the entire graph, effectively eliminating irrelevant noise nodes during information aggregation and stabilizing the training process; while the latter performs feature aggregation updates more efficiently by recalculating the importance of each node in the receptive field captured by each layer.

[0097] Specifically, the cross-layer masking matrix masks some nodes so that they do not participate in the information aggregation process of the current layer, reducing the interference of noise nodes while making the information aggregation and relational reasoning process on the graph more discriminative; the cross-hop attention matrix adaptively adjusts the connection relationship between the retained nodes (not masked), making the feature update process of different nodes in each local neighborhood on the graph more expressive. It is worth noting that the cross-layer masking matrix M in the method of the present invention is t,s can be further divided into M′ s , M′ t and M′ t,s , which represent the self-masking matrix between stroke-level graph nodes, the self-masking matrix between text-level graph nodes, and the mutual-masking matrix between stroke-level and text-level graph nodes, respectively. Therefore, the masking result of the stroke-level graph nodes is ultimately based on M′ s and M' t,s The linear combination of M′ and the comparison of fixed thresholds are used, and the masking results of text-level graph nodes are finally based on M′ t and M′ t,s Comparison of linear combinations and fixed thresholds.

[0098] After completing the above three stages, the present invention uses the output of the last graph network layer to predict the link relationship between text graph nodes and locate the text instance bounding box regression value.

[0099] (5) Overview of the detection framework reasoning process

[0100] During the inference process of the constructed text detection framework, it first obtains multi-level prediction results for text regions based on the front-end processing module and extracts two levels of region candidate boxes by setting corresponding thresholds. It then constructs a hierarchical (including text-level and stroke-level) local graph structure network. On this basis, the back-end processing module performs hierarchical node feature aggregation and relationship reasoning to infer the relationships between graph nodes at different levels and perform link prediction. Based on the graph node classification and link prediction results, the text-level nodes are grouped using a breadth-first search method and sorted using a minimum path algorithm. Finally, the boundaries of arbitrary-shaped text instances are obtained by sequentially connecting the top and bottom midpoints of the candidate boxes corresponding to the sorted text nodes.

[0101] Experimental tests and their effects

[0102] (1) Evaluation dataset and implementation details

[0103] The proposed method is experimentally validated on mainstream evaluation datasets in the field of text detection, including CTW-1500, Total-Text, and ICDAR 2015. The experiments were conducted on a server equipped with four NVIDIA GeForce GTX 1080Ti GPUs and implemented based on the Pytorch deep learning framework. The backbone feature extraction network in the proposed method was pre-trained on the ImageNet dataset. A data augmentation strategy was adopted in the experiment, and the input images were resized to 640*640, and each image was randomly flipped with a probability of 0.5. During the training process, the batch size of the input data was set to 64, that is, each GPU processed 16 image samples. All evaluation experiments were conducted at a single image resolution.

[0104] (2) Experimental results

[0105] As shown in Table 1, the proposed method significantly outperforms existing methods on all text detection datasets. Furthermore, the table also lists the text detection speed (FPS) of each method, demonstrating that the proposed method achieves a good balance between accuracy and speed. First, to evaluate the performance of the proposed method in detecting text instances that are close to each other or of arbitrary shapes, the proposed method was experimentally evaluated and compared with existing mainstream text detection models on two text datasets containing numerous curved text instances: Total-Text and CTW-1500. As shown in Table 1, the proposed method performs significantly better than models designed specifically for detecting curved text instances, such as TextSnack and CRAFT. In particular, the proposed text detection method achieves an Hmean score of 87.5% on CTW-1500 and 89.1% on Total-Text, both of which significantly exceed the other compared methods. Benefiting from the introduced hierarchical graph reasoning network module, the proposed method achieves consistently superior performance in detecting text instances that are close to each other and have arbitrary shapes, especially for text instances with different curvatures, and has strong detection robustness.

[0106] Secondly, we evaluated our proposed method on the ICDAR 2015 dataset to verify its ability to detect tiny and low-resolution text instances. As shown in Table 1, our proposed method achieved recall, precision, and Hmea scores of 89.7%, 91.7%, and 90.7%, respectively, significantly outperforming relevant comparison models (including ReLaText, DRRG, and StrokeNet). The results demonstrate that the multi-level region prediction network introduced by our method excels at capturing tiny and low-resolution stroke representations, which plays a crucial role in enabling the detector to effectively model finer-grained text region representations.

[0107] Table 1 shows the evaluation results of the proposed method on the CTW-1500, Total-Text, and ICDAR 2015 datasets, and compares its performance with existing mainstream methods in this field. The top two performance values ​​are highlighted in bold. Furthermore, (ST) denotes the use of the introduced SceneText stroke segmentation dataset to pre-train the convolutional neural network-based front-end processing module in the detection framework. R: Recall (%), P: Precision (%), H: Hmean (%).

[0108] Table 1

[0109]

[0110] (3) Visualization

[0111] Figure 5 The visualization of text detection results in can also demonstrate the effectiveness, high accuracy and good generalization ability of the proposed method in visual image text detection.

[0112] Figure 5 The following figure shows the visualized text detection results. In the figure, the first column (a), the second column (b), and the third column (c) represent the input image, the stroke segmentation predicted by the proposed method, and the final text detection results, respectively.

[0113] Application examples:

[0114] (1) Example of stroke segmentation dataset

[0115] This paper first uses the SceneText dataset to pre-train the convolutional neural network-based front-end processing modules of existing mainstream text detectors to verify the effectiveness of the introduced stroke segmentation dataset and its contribution to improving the performance of other detectors in the field of text detection. The results in Table 2 demonstrate that the external dataset introduced by this paper can effectively improve the prediction accuracy of text regions in the front-end processing modules of existing mainstream text detectors, thereby further exploring and stimulating the potential of the graph model-based back-end processing modules of these methods, ultimately improving the detection performance of relevant mainstream methods in the field of text detection.

[0116] Table 2 shows a quantitative evaluation of the performance improvement of text detection detectors using the stroke segmentation dataset introduced in this paper. (ST) indicates the use of the SceneText stroke segmentation dataset to pre-train the convolutional neural network-based front-end processing module in the detection framework. R: Recall (%), P: Precision (%), H: Hmean (%).

[0117] Table 2

[0118]

[0119] (2) Example of the method of the present invention in OCR translation application

[0120] Based on the invention content, the present invention develops an OCR translation tool based on the proposed text detection method, such as Figure 6 This figure provides a detailed process flow of the proposed text detection method in Chinese-to-English and English-to-French translation examples. The sub-figures represent the input image (a), predicted stroke segmentation (b), text detection results (c), and the translated image (d). Figure 6 Examples of guided OCR translation (Chinese to English and English to French) tasks are shown.

[0121] On this basis, related applications based on OCR translation can be used as downstream tasks of the text detection model proposed in this invention. Figure 7 The overall processing flow of the proposed method in OCR translation application is demonstrated, and an example of Chinese to English OCR translation is given. Figure 7 The overall processing flow of performing OCR translation tasks using the front-end processing module in the method proposed by the present invention is shown.

[0122] In this application scenario, the front-end processing module of the method of the present invention is first called to process and then output the detection results at the stroke level and text level. The relevant results will be input into the text recognition module and the text erasure module respectively. For the image erasure module, the built-in OpenCV erasure algorithm is simply used here. Before applying the erasure operation, the present invention uses the dilation operation in the built-in OpenCV function to expand the predicted stroke segmentation area. In addition, the text color can be easily estimated by averaging the pixel values ​​in the predicted stroke area. This strategy can improve the visual effect of image erasure (restoration) in practice. The machine translation model is run by calling the Google Translate API, which includes language recognition related APIs.

[0123] The outstanding features of the embodiments of the present invention include: proposing a lightweight stroke segmentation prediction network to introduce a new graph neural network inference model, and achieving significant results. The stroke segmentation prediction network implements a multi-level (text level, stroke level) representation of the detection model for the text area. The graph neural network model, as the back-end processing module in the constructed text detection framework, can effectively perform feature aggregation and relationship reasoning on each part of the text area predicted by the front-end processing module, so that the improved graph model can better adapt to the text detection task scenario.

[0124] This method effectively addresses the difficulties faced by traditional methods in accurately locating text instances in images of arbitrary shapes, effectively distinguishing and separating multiple text instances within the same text image that are close to each other, and excessively time-consuming algorithms. The effectiveness, high accuracy, and good generalization capabilities of this visual text detection method have been fully demonstrated.

[0125] In some embodiments, a visual text image dataset (SceneText) is used to pre-train the stroke region prediction network, which helps improve the detection framework's prediction accuracy for multi-level representations of text regions. Specifically, each text instance in its image sample is annotated with stroke-level segmentation labels, namely, a binary stroke character segmentation map. This dataset is used to pre-train the detection framework's convolutional neural network-based front-end processing module to improve the detection framework's prediction accuracy for multi-level representations of text regions.

[0126] The text detection method proposed in the present invention can be used for front-end processing (text segmentation and detection) of text image restoration systems and front-end processing (text segmentation and detection) of OCR image translation systems, for example.

[0127] This invention can perform stroke-level segmentation prediction on text regions. This segmentation prediction result, combined with the text detection results of traditional methods, can meet the needs of more business functions, such as image restoration and OCR translation. It can be used for tasks such as OCR (text content recognition), image restoration, image translation, and text detection and recognition in visual text images such as electronic receipts and invoices, achieving intelligent and fully automated text image processing and improving information comprehension and processing efficiency in related fields.

[0128] The present invention has broad application prospects. For example, this method can effectively improve the efficiency and reliability of OCR technology by improving the accuracy of text content recognition and text instance detection, thereby greatly reducing the labor and resource costs of related businesses in enterprises.

[0129] An embodiment of the present invention further provides a storage medium for storing a computer program, which, when executed, at least performs the visual text detection method based on the stroke region segmentation strategy as described above.

[0130] An embodiment of the present invention further provides a control device comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the above-described visual text detection method based on the stroke region segmentation strategy when executing the computer program.

[0131] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0132] The storage medium may be implemented by any type of volatile or non-volatile storage device, or a combination thereof. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a magnetic disk memory or a magnetic tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM). The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0133] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0134] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0136] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0137] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0138] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0139] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0140] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0141] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A visual text detection method based on stroke region segmentation strategy, characterized in that: The steps include: S1. Perform feature extraction and multi-level region prediction on an input text image through a front-end processing module based on a convolutional neural network. The front-end processing module includes a backbone image feature extraction network, a text region prediction network, and a stroke region prediction network. The front-end processing module performs multi-level prediction related to text regions through a series of convolutional layers stacked on top of the backbone image feature extraction network in a feature pyramid. S2. Extract text-level and stroke-level region candidate boxes based on the multi-level prediction results of the text region. The image regions represented by each candidate box serve as graph nodes to form multiple local graph structures, thereby constructing a hierarchical local graph structure. S3. Through the back-end processing module based on the graph neural network, node feature aggregation and relationship reasoning based on multi-level graph nodes are performed on each local graph, the relationship between graph nodes at different levels is inferred and link prediction is performed. The nodes are grouped according to the link relationship between each text-level node to form the overall text instance detection result; In step S1, the stroke region prediction network separates the character content in each text region from the complex background, wherein the low-level semantics and high-level semantic information of the image are combined to generate a fine stroke segmentation representation in the text region to guide the subsequent text detection process; The stroke region prediction network includes a two-stage prediction process; 1) Extract text-related features from the high-level feature representation of the input image obtained by the backbone network; specifically, crop the outer rectangle OTR of the TR from the input image, and use a global pooling layer combined with continuous convolutional layers to extract the OTR region features obtained from the backbone network; use several pooling layers and a multi-layer perceptron network and related nonlinear activation functions to calculate the channel attention feature map of the input image to identify and measure the relative contribution of different network layers in the backbone network to the text region representation; during this period, the extracted input feature map is upsampled to the same resolution as the input image, and then multiplied by the obtained channel attention feature map to achieve the semantic information distillation operation of the input image; thereby, the semantic representation of the text image is obtained; 2) Finely model the stroke representation of the text region and enhance the fine-grained stroke character segmentation representation by introducing an orthogonal convolutional network from the orthogonal direction; specifically, the 3-channel RGB raw input features of the text region circumscribed rectangle (OTR) are used as supplementary low-level image semantic information and fused with the obtained text image semantic representation.

2. The visual text detection method based on stroke region segmentation strategy according to claim 1, characterized in that: In step S1, the multi-level prediction includes: obtaining the classification confidence of the text-level rectangular area corresponding to each text instance; regression prediction of relevant text-level attributes such as text rotation angle and centerline position in each text instance; and character segmentation prediction corresponding to the stroke level within the predicted bounding box of each text area.

3. The visual text detection method based on stroke region segmentation strategy according to claim 1 or 2, characterized in that: In step S1, the text-level and stroke-level region candidate frames are extracted based on the multi-level prediction results of the text region, including: extracting corresponding multi-level candidate rectangular frames based on the multi-level prediction results of the text region, a local graph containing only text-level or stroke-level nodes is a homogeneous graph, and a local graph containing both text-level and stroke-level nodes is a heterogeneous graph.

4. The visual text detection method based on the stroke region segmentation strategy according to any one of claims 1 to 2, characterized in that: In step S1, the text region prediction network predicts attributes related to the text instance region, including: predicting the classification probability of the text region TR and the text center region TCR, and then predicting the regression values ​​of h1, h2, cosθ and sinθ, where h1 and h2 represent the distances from the current pixel to the upper edge of TR and the lower edge of TR, respectively, the text instance height scale h is the sum of h1 and h2, and θ indicates the direction information of the text instance; estimating the text center line corresponding to TR based on the predicted TR potential region; using the feature outputs of two channels to guide the classification probability prediction of TR and TCR; during training, the first feature channel is used to predict the background, and the second channel is used to predict the foreground, i.e., the text instance; during testing, the foreground prediction result of the second feature channel is taken for subsequent processing; wherein, the output of one feature channel is used to predict each regression attribute value.

5. The visual text detection method based on stroke region segmentation strategy according to any one of claims 1 to 2, characterized in that: In step S1, stage 2 of the prediction process introduces orthogonal convolution layers with kernel sizes of 1×7 and 7×1 to calculate the attention coefficient along the spatial direction, and then multiplies the obtained attention value by the fused text feature map.

6. The visual text detection method based on stroke region segmentation strategy according to any one of claims 1 to 2, characterized in that: The front-end processing module uses a dataset with stroke-level segmentation map annotations as labels and a mean square error loss function to pre-train the stroke region prediction network.

7. The visual text detection method based on stroke region segmentation strategy according to any one of claims 1 to 2, characterized in that: In step S3, the node features and their connection structure are initialized, specifically including: Initializing node features: Two complementary feature representations, geometric embedding and content embedding, are used for feature initialization of text-level and stroke-level nodes. For geometric embedding, the geometric properties of each predicted region candidate box are encoded into a high-dimensional space. For content embedding, the content features of each graph node are obtained by sending the predicted feature map of the geometric properties of each region candidate box to the RRoI-Align layer. The two obtained feature embeddings are concatenated to form the final graph node feature representation. When generating the local graph network, the initial feature representation of all nodes is normalized by subtracting the features of the central node. Adjacency matrix generation: The topological structure formed by each local graph network is encoded in the adjacency matrix A∈R N×N Where A(c,n)=1, if there is a connection between the central node c and each of its neighbor nodes n in the local graph; the method of generating the adjacency matrix specifically includes: For isomorphic graphs, including the construction of isomorphic stroke-level graph networks and text-level graph networks; for isomorphic stroke graphs containing only stroke-level graph nodes, the KNN nearest neighbor algorithm based on Euclidean distance is adopted, and the 8 nearest neighbor nodes of each central node are selected as the 1-hop neighbor nodes of the central node to form the adjacency matrix A s For the isomorphic text graph containing only text-level graph nodes, the difference between the adjacency matrix construction method and the isomorphic stroke graph is that each central node in the isomorphic text graph only retains the four closest direct neighbor nodes to form the corresponding matrix A t ; For heterogeneous text graph networks, graph nodes at both text and stroke levels are included. This type of graph network is constructed based on the Euclidean distance between the center positions of each extracted region candidate box. Specifically, each text-level region candidate box is regarded as the central node of the heterogeneous text graph, and the connection relationship within the 1-hop and 2-hop neighborhood of the central node is used to generate the adjacency matrix A of the heterogeneous graph. h ; The 1-hop neighborhood of the central node contains the 4 nearest neighbor nodes in the text-level graph, while its 2-hop neighborhood contains an additional 4 nearest neighbor nodes in the stroke-level graph.

8. The visual text detection method based on stroke region segmentation strategy according to any one of claims 1 to 2, characterized in that: In step S3, the backend processing module performs node relationship reasoning and link prediction in the generated multiple local graph networks through a hierarchical graph neural network reasoning model. The reasoning process based on the graph neural network includes the following three stages: First, a weighted average approach is used to aggregate and update the stroke-level node features guided by the attention mechanism; the weight information in the weighted process comes from two parts, including the normalized adjacency matrix A s and the attention coefficient between any two graph nodes v and u derived in the graph attention network GAT; the weighted aggregation process in the first stage is described as: Among them, σ is the activation function, W is the trainable weight parameter, and s k represents the feature of the stroke-level graph node k, α′ v,k is the weight coefficient; is derived from As, Fuse(·) represents the characteristic linear combination function; If the center of a stroke-level graph node falls within the region of a text-level graph node, merging the updated representation of the stroke-level node into the corresponding text-level node representation; The second stage fuses the features of two-level graph nodes by stacking two Transformer encoder modules. Specifically, the introduced Transformer encoder captures the attention coefficients between stroke-stroke, stroke-text, and text-text nodes to model and reason about the hierarchical structural relationships between heterogeneous graph nodes, which can be expressed as: in, represents all text (t) node features and stroke (s) node features of layer t; Attention (·) is the attention calculation operation in Transformer, Q, K, V represent the query matrix, key matrix and value matrix respectively, and W Q ,W K ,W V is a trainable weight parameter; Using a graph reasoning network with expanded neighborhood scope, for each text-level graph node, the feature representations of its one-hop neighbor nodes containing only text neighbors are aggregated in the first layer of the designed graph network. Subsequently, the information of its two-hop neighbors containing both text neighbors and stroke neighbors is aggregated in subsequent layers. During this period, dynamic graph convolution is used to adaptively adjust the network structure of the heterogeneous graph, which is described as follows: P=σ(M t,s ,A t,s (G(H t,s ))W) Where W is a trainable weight matrix, G(·) represents the traditional information aggregation process on the graph network, and M t,s and A t,s They represent the cross-layer masking matrix and cross-hop attention matrix introduced in the dynamic graph network respectively; Cross-layer masking matrix M t,s Further divided into M′ s , M′ t and M′ t,s , respectively represent the self-masking matrix between the stroke-level graph nodes, the self-masking matrix between the text-level graph nodes, and the mutual-masking matrix between the stroke-level and text-level graph nodes; the masking result of the stroke-level graph node is finally based on M′ s and M' t,s The linear combination of M' and the comparison of fixed thresholds are used, and the masking results of the text level graph nodes are finally based on M' t and M' t,s By completing the above three stages, the output of the last graph network layer is used to predict the link relationship between text graph nodes and locate the text instance bounding box regression value.

9. The visual text detection method based on stroke region segmentation strategy according to any one of claims 1 to 2, characterized in that: During the training process, the cross-entropy loss between the graph model prediction results and the corresponding true category labels is used to guide the learning process of the entire detection framework; based on the classification and link prediction results of the graph nodes, the text-level nodes are grouped using the breadth-first search method and sorted using the minimum path algorithm; the boundaries of text instances of arbitrary shapes are obtained by sequentially connecting the midpoints of the top and bottom of the candidate boxes corresponding to the sorted text nodes.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the visual text detection method based on the stroke region segmentation strategy according to any one of claims 1 to 9 is implemented.

Images