Complex image-text detection and extraction method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a complex image-text detection and extraction method, device, electronic equipment and storage medium. The method comprises: performing query-based character-level detection on a target text image to obtain a character-level mask, a category prediction result and an updated query vector; based on the mask, the category prediction result and the query vector, obtaining a multi-modal embedding vector of each character, the multi-modal embedding vector comprising a position embedding vector, a semantic embedding vector and a visual embedding vector; taking the position embedding vector as a position constraint and the visual embedding vector as a visual constraint to construct a local subgraph of each character; combining the semantic embedding vector to infer the connection relationship of the characters based on the local subgraph; and obtaining a text detection result of the target text image based on the connection relationship of the characters. The method can effectively solve the problem of incorrect segmentation in overlapping, adhesion and mixed arrangement scenes, and effectively improve the text detection accuracy and precision.

Description

Technical Field

[0001] This application relates to the field of text and image processing technology, and in particular to a method for detecting and extracting complex text and images, a device for detecting and extracting complex text and images, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] In natural scenes, text nesting is common in billboards and street signs, manifesting as multiple lines of text closely arranged, mixed horizontal and vertical text, significant differences in font size, and even overlapping text. This phenomenon often leads to problems such as candidate box overlap, instance merging, or over-segmentation in text detection, severely affecting the accuracy of text detection. To alleviate this problem, existing technologies typically employ the following methods for text detection: First, the input image is processed through a convolutional network or feature pyramid network to extract multi-scale visual features, and then a large number of candidate regions that may contain text are generated based on these multi-scale visual features. Next, each candidate region undergoes further text discrimination. For example, the candidate region is input into a recognition module to obtain an output character sequence, which helps determine whether the region is valid text. Alternatively, a lightweight classification or discrimination mechanism can be used to directly determine whether a candidate region is text without character-by-character recognition, thus initially filtering out some invalid candidate regions. Then, a language model or semantic awareness module is introduced to use semantic consistency to determine the rationality of candidate regions. For instance, after recognition, candidate regions are reordered based on consistency using a language model or semantic prior to exclude semantically unreasonable combinations. Alternatively, an ambiguity detection mechanism can be used to filter candidate regions, calling the semantic discrimination module only on suspicious candidate regions to avoid redundant computation. Finally, the visual and semantic scores of candidate regions are fused, and candidate regions with semantic consistency and correct spatial boundaries are retained as the final detection result.

[0003] Existing text detection and extraction methods suffer from at least the following drawbacks: coarse-grained detection and insufficient parsing of nested text. For example, most detection frameworks still use text lines or text blocks as the basic detection unit. This coarse-grained design leads to misgrouping in nested text scenarios. Specifically, the traditional sliding window method scans the image using multi-scale windows; if nested text falls within the same window, it may be mistakenly identified as a single text block. The connected component method relies on geometric feature grouping; if the components of nested text have similar features, such as horizontally and vertically mixed text components with similar widths, "adjacent lines are mistakenly merged" or "single lines are mistakenly segmented." In datasets containing complex nested text, including circular nested text, text block-level detection methods may misclassify the text inside and outside the circle as a single unit.

[0004] It is evident that existing methods for detecting and extracting complex images and text still require improvement. Summary of the Invention

[0005] This application provides a method for detecting and extracting complex images and text, which can effectively improve the accuracy and precision of text detection.

[0006] Accordingly, embodiments of this application also provide a complex image and text detection and extraction device, an electronic device, a storage medium, and a computer program product to ensure the implementation and application of the above-mentioned complex image and text detection and extraction method.

[0007] To solve the above-mentioned technical problems, this application is implemented as follows:

[0008] In a first aspect, embodiments of this application provide a method for detecting and extracting complex images and text, the method comprising:

[0009] Perform query-based character-level detection on the target text image to obtain the character-level mask, category prediction results, and updated query vector;

[0010] Based on the mask, the category prediction result, and the query vector, a multimodal embedding vector for each character is obtained. The multimodal embedding vector includes: a positional embedding vector, a semantic embedding vector, and a visual embedding vector.

[0011] Using the positional embedding vector as a positional constraint and the visual embedding vector as a visual constraint, a local sub-image of each character is constructed.

[0012] By combining the semantic embedding vectors, the connection relationships of the characters are inferred based on the local subgraph;

[0013] Based on the connection relationship of each character, the text detection result of the target text image is obtained.

[0014] Secondly, embodiments of this application provide a complex image and text detection and extraction device, the device comprising:

[0015] The character-level detection module is used to perform query-based character-level detection on target text images, and obtain character-level masks, category prediction results and updated query vectors;

[0016] The multimodal embedding vector acquisition module is used to acquire the multimodal embedding vector of each character based on the mask, the category prediction result and the query vector. The multimodal embedding vector includes: positional embedding vector, semantic embedding vector and visual embedding vector.

[0017] The local subgraph construction module is used to construct local subgraphs for each character using the positional embedding vector as a positional constraint and the visual embedding vector as a visual constraint.

[0018] The connection relationship reasoning module is used to combine the semantic embedding vectors and infer the connection relationships of the characters based on the local subgraph.

[0019] The text detection module is used to obtain the text detection result of the target text image based on the connection relationship of each character.

[0020] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method described in the first aspect.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0022] Compared with the prior art, the embodiments of this application have the following advantages:

[0023] By performing query-based character-level detection on the target text image, character-level masks, category prediction results, and updated query vectors are obtained, refining the initial detection granularity and effectively solving erroneous segmentation problems in scenarios such as overlap, concatenation, and mixed text. Based on the mask, the category prediction results, and the query vector, multimodal embedding vectors for each character are obtained, including positional embedding vectors, semantic embedding vectors, and visual embedding vectors. Using the positional embedding vectors as positional constraints and the visual embedding vectors as visual constraints, local subgraphs for each character are constructed. The character-level detection results are further filtered by combining multimodal features to reduce redundancy and noise and improve text detection accuracy. The semantic embedding vectors are combined to infer the connection relationships of the characters based on the local subgraphs. Based on the connection relationships of the characters, the text detection results of the target text image are obtained, effectively improving the accuracy and precision of text detection. When using the complex image-text detection and extraction method disclosed in this application to detect nested text, the detection accuracy is significantly improved, and compared with traditional methods, it can effectively reduce ambiguity and errors in complex scenarios. Attached Figure Description

[0024] Figure 1 This is one of the flowcharts of the complex image and text detection and extraction method disclosed in the embodiments of this application;

[0025] Figure 2 This is a schematic diagram of the text detection model structure disclosed in the embodiments of this application;

[0026] Figure 3This is a schematic diagram of the target text image in the complex image and text detection and extraction method disclosed in the embodiments of this application;

[0027] Figure 4 This is a schematic diagram of the text detection results of the complex image and text detection and extraction method disclosed in the embodiments of this application;

[0028] Figure 5 This is a schematic diagram of the complex image and text detection and extraction device disclosed in the embodiments of this application;

[0029] Figure 6 A block diagram schematically illustrates an electronic device for performing the method according to this application; and

[0030] Figure 7 A storage unit for holding or carrying program code implementing the method according to this application is illustrated schematically. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] Reference Figure 1 The complex image and text detection and extraction method disclosed in this application includes steps 102 to 110.

[0033] In some alternative embodiments, it can be achieved through methods such as Figure 2 The neural network shown implements the complex image and text detection and extraction method disclosed in the embodiments of this application.

[0034] like Figure 2 As shown, the neural network includes: a feature extraction network, a semantic encoding network, a position encoding network, a local subgraph generation network, and a character relation reasoning network.

[0035] In some optional embodiments, the feature extraction network, the semantic encoding network, the position encoding network, the local subgraph generation network, and the character relation reasoning network can be trained jointly or independently. In this application embodiment, there are no restrictions on the training methods of the feature extraction network, the semantic encoding network, the position encoding network, the local subgraph generation network, and the character relation reasoning network.

[0036] In this embodiment, the term "character" refers to the smallest text unit in a text image, and does not specifically refer to a particular language or writing system. The character can be a character unit in any language or symbol system, such as Chinese characters, English letters, numbers, or symbol characters.

[0037] The following is combined with Figure 2 The specific implementation methods of each step of the complex image and text detection and extraction method disclosed in the embodiments of this application are described in detail.

[0038] Step 102: Perform query-based character-level detection on the target text image to obtain the character-level mask, category prediction results, and updated query vector.

[0039] The complex image and text detection and extraction method disclosed in this application can be applied to text detection of images containing text in various fields.

[0040] For example, it can automatically detect and locate text regions in natural scene images (such as street views and billboards). In the financial field, it can perform structured parsing of complex documents such as bills, contracts, and invoices, accurately extracting nested text, seal information, and multilingual elements. In the medical field, it can detect and parse text in medical records, prescriptions, and medical images, especially in scenarios with mixed handwriting, printed text, and symbols, accurately extracting key text information to provide technical support for assisted diagnosis and medical record management. In the educational field, it can perform refined parsing of elements such as question stems, answers, formulas, and tables in documents such as test papers, assignments, and annotations, effectively solving complex mixed text and image structures and nested structures, thus supporting intelligent grading, teaching resource organization, and knowledge point mining.

[0041] The target text image is an image that includes text, including but not limited to any of the following: natural scene image, invoice image, contract image, medical record image, prescription image, test paper image, and homework image.

[0042] The embodiments of this application do not limit the specific content and acquisition method of the target text image.

[0043] In some optional embodiments, the step of performing query-based character-level detection on the target text image to obtain a character-level mask, category prediction results, and an updated query vector includes: extracting multi-scale features from the target text image to obtain multi-scale visual features; introducing a character-level learnable query vector, and updating the query vector by decoding the multi-scale visual features; obtaining a character-level mask based on the updated query vector and the high-resolution multi-scale visual features in the multi-scale visual features; and performing classification prediction based on the updated query vector to obtain a character-level category prediction result.

[0044] Optionally, the feature extraction network can be built based on a deep residual network and a Transformer (a neural network model based on self-attention). For example, the feature extraction network can be built based on ResNet50 (a deep residual network structure for image feature extraction, containing 50 convolutional layers) and a Transformer Encoder and Decoder (an encoder and decoder of a neural network model based on self-attention). In the feature extraction stage, the deep residual network extracts multi-scale visual features of the target text image. In the decoding stage, a set of learnable query vectors is introduced. Each query vector does not correspond to a specific character at initialization, but gradually learns to respond to potential character instances as candidate representations of character instances through multiple rounds of interactive updates with the multi-scale visual features. The learnable query vectors are updated by the Transformer Decoder through multiple rounds of updates to the multi-scale visual features to obtain the updated query vectors.

[0045] Subsequently, based on the updated query vector and the high-resolution multi-scale visual features in the multi-scale visual features, a character-level mask is obtained.

[0046] Optionally, based on the updated query vector and the high-resolution multi-scale visual features in the multi-scale visual features, a character-level mask is obtained, including: multiplying each updated query vector by the high-resolution multi-scale visual features in the multi-scale visual features, and using the product as the mask of the character corresponding to the corresponding query vector.

[0047] On the other hand, classification prediction is performed based on the updated query vectors to obtain character-level category prediction results. Optionally, performing classification prediction based on the updated query vectors to obtain character-level category prediction results includes: performing classification prediction on each updated query vector using a pre-trained classification network to obtain the category prediction results of the characters corresponding to each query vector.

[0048] For a specific example, taking a street view image of an advertising sign as the target text image, assuming the advertising sign in the street view image contains the four characters "CAFE", firstly, the street view image is input into the feature extraction network, and multi-scale feature extraction is performed by ResNet50 and Transformer Encoder to obtain multi-scale feature maps C4, C3, C2, and C1; at the same time, a set of learnable query vectors are initialized as learnable embeddings. Each query does not initially correspond to a specific character. Assuming that the learnable query vectors use 10 queries, after the Transformer Decoder performs multiple rounds of decoding, these 10 query vectors are updated to refined queries. Some of the query vectors learn features corresponding to characters such as 'C', 'A', 'F', and 'E' on the advertising sign, while the remaining query vectors may correspond to non-text areas, background, or low-confidence character candidates.

[0049] In multi-scale feature maps, each feature map can be viewed as compressing a street view image into a small grid, with each grid containing the visual features of that image region. Taking a target text image of size 640×640 as an example, after multi-scale feature extraction, the resulting multi-scale feature maps are C4, C3, C2, and C1, with the spatial resolution of each feature map progressively increasing. For example, C4 is 40×40×256, C3 is 80×80×256, C2 is 160×160×256, and C1 is 320×320×256. C1 is a relatively high-resolution feature map. The above dimensions are merely examples and do not constitute a limitation of this application.

[0050] The high-resolution feature map C1 is viewed as a feature map composed of multiple spatial locations, where each spatial location corresponds to a 256-dimensional visual feature vector. The updated query vector is used as the visual representation of a character, and its correlation with the visual features of each spatial location in the high-resolution feature map C1 is calculated to measure the visual similarity between each spatial location and the character, thereby generating a probability mask map corresponding to the character, i.e., the mask for the character. The correlation calculation may include dot product, matrix multiplication, element-wise multiplication, or similarity calculation after linear mapping.

[0051] On the other hand, for each updated query vector, a small classification network is used to classify and predict whether the updated query vector is a valid character and roughly determine what category the updated query vector belongs to. This yields the category prediction result for the character corresponding to the updated query vector. Taking the four characters "CAFE" on an advertising sign as an example, the classification network will output a classification result for each candidate to indicate whether the character is text (e.g., outputting class 0 / 1, where class 0 indicates non-text or noise, and class 1 indicates text characters). If the classification result indicates that the character is text, the category prediction result will also output the probability that the character matches a preset character category.

[0052] In this embodiment, a query-based character-level detection mechanism is adopted in the character detection stage. This mechanism generates an updated query vector through multiple rounds of interactive updates between multi-scale visual features and learnable query vectors. A character-level mask is generated based on the correlation between the updated query vector and high-resolution visual features. The character category prediction result is obtained through a classification network based on the updated query vector. This overcomes the limitation of coarse detection granularity in existing methods, enabling higher accuracy when performing text detection based on the mask, the category prediction result, and the query vector output in this step.

[0053] Step 104: Based on the mask, the category prediction result, and the query vector, obtain the multimodal embedding vector of each character. The multimodal embedding vector includes: positional embedding vector, semantic embedding vector, and visual embedding vector.

[0054] Optionally, obtaining the multimodal embedding vector of each character based on the mask, the category prediction result, and the query vector includes: extracting geometric features based on the mask corresponding to each character to obtain the position embedding vector of the corresponding character; semantically encoding the category prediction result of each character to obtain the semantic embedding vector of the corresponding character; obtaining the visual embedding vector of the corresponding character based on the query vector of each character; and using the position embedding vector, the semantic embedding vector, and the visual embedding vector of each character as the multimodal embedding vector of the corresponding character.

[0055] Next, the mask is encoded using the positional encoding network to obtain the position embedding vector of each character. Optionally, geometric feature extraction is performed based on the mask corresponding to each character to obtain the position embedding vector of the corresponding character. This includes: for the mask corresponding to each character, extracting one or more of the following geometric features from the mask to obtain the feature representation corresponding to the mask: centroid, size, and shape; encoding the feature representation corresponding to the mask to obtain a specified dimension vector representing the spatial position and shape information of the character in the target text image, which serves as the position embedding vector of the corresponding character. For example, a multilayer perceptron can be used to encode the feature representation corresponding to the mask to obtain the specified dimension vector representing the spatial position and shape information of the character in the target text image.

[0056] On the other hand, the semantic encoding network encodes the category prediction results of each character to obtain the semantic embedding vector of each character. Optionally, the corresponding character or character candidate set is determined based on the category prediction results, and it is encoded as a token sequence and input into the semantic encoding network (e.g., BERT model) to obtain the semantic embedding vector corresponding to the character; or, the probability distribution obtained from the category prediction is weighted onto the character vocabulary embedding to form a soft embedding representation, which is then input into the semantic encoding network to obtain the semantic embedding vector of the character.

[0057] Optionally, the visual embedding vector of the corresponding character can be obtained by linearly mapping the query vector based on each character.

[0058] Finally, for each character, the positional embedding vector, semantic embedding vector, and visual embedding vector corresponding to that character are used as the multimodal embedding vector for that character.

[0059] In existing technologies, subsequent reasoning typically relies solely on visual or geometric information after character detection, lacking effective utilization of semantic features, resulting in limited parsing capabilities for nested text. This application combines three types of embeddings—visual features, positional information, and semantic information—to construct a complete multimodal feature representation, which helps improve the accuracy of text detection.

[0060] Step 106: Using the positional embedding vector as a positional constraint and the visual embedding vector as a visual constraint, construct a local sub-graph for each character.

[0061] Optionally, the step of constructing a local subgraph of each character using the positional embedding vector as a positional constraint and the visual embedding vector as a visual constraint includes: determining the spatial proximity relationship between each character based on the positional embedding vector, and determining the visual similarity between each character based on the visual embedding vector; for each first character, determining one or more second characters that are spatially adjacent to and visually similar to the first character, based on the spatial proximity relationship and the visual similarity, as a candidate combination character set for the first character, wherein the first character and the second characters are characters in different positions; establishing character nodes corresponding to the characters, and connecting each character with its candidate combination characters through edges to obtain a local subgraph of the character.

[0062] For example, a node, denoted as a "character node," is created for each instance of a character. Each character node is associated with at least one query vector (i.e., the query vector corresponding to the character in the character instance to which the character node belongs), used to characterize the visual representation of the character instance. Multimodal features are also bound to this character node, including the positional features, visual features, and semantic features of the corresponding character. The positional features are expressed through the positional embedding vector, the visual features through the visual embedding vector, and the semantic features through the semantic embedding vector. In this way, each character node is associated with a character, as well as the character's positional features, visual features, and semantic features.

[0063] Next, taking each character as the center and using spatial proximity and appearance similarity as constraints, a local subgraph is constructed for each character. Edges are then used to connect each node to a small number of related neighboring characters. Spatial proximity can be determined using geometric distance or k-nearest neighbors, while appearance similarity can be determined using the similarity metric between feature vectors.

[0064] In some optional embodiments, for each character (referred to as "first character" in this embodiment), the geometric distance between the character (i.e., first character) and other characters (referred to as "second character" in this embodiment) can first be measured based on the position embedding vector to determine the spatial proximity relationship between characters. Only the second characters whose geometric distance is within a preset distance range are retained as initial candidate characters. Then, the visual similarity between the first character and each initial candidate character (i.e., second character) is measured based on the visual embedding vector. Initial candidate characters with large appearance differences (such as visual similarity less than a preset similarity threshold) are filtered out. After that, the retained initial candidate characters are stored as candidate combination characters of the first character in the candidate combination character set.

[0065] Finally, for each character, a corresponding node is created, and edges are established to connect the character to its candidate combined characters (i.e., characters in the set of candidate combined characters), thereby constructing a local subgraph for each character.

[0066] In the embodiments of this application, by using positional embedding vectors and visual embedding vectors to limit the range of candidate edges, a local subgraph is generated for each character, thereby retaining only the adjacency relationships with high potential correlation between characters, thereby reducing invalid connections and improving the efficiency of constructing character relationship graphs.

[0067] The local subgraph constructed using this method reflects the proximity relationships between characters, and the multimodal features associated with character nodes can further accurately characterize the positional proximity and visual similarity between characters. Next, the connection relationships between the characters can be inferred based on the local subgraph.

[0068] Step 108: Combine the semantic embedding vectors and infer the connection relationships of the characters based on the local subgraph.

[0069] Optionally, the step of reasoning about the connection relationship of the character based on the local subgraph by combining the semantic embedding vector includes: obtaining the neighbor nodes of the character node corresponding to the current character based on the local subgraph, thus obtaining multiple neighbor node pairs of the current character composed of the corresponding character node and the neighbor nodes; obtaining the adjacency message of the current character based on the position embedding vector, the visual embedding vector, and the semantic embedding vector of the character corresponding to the character node in each neighbor node pair; fusing the adjacency message with the position embedding vector and the visual embedding vector of the current character, and using it as the input of the graph neural network; performing multi-layer message passing and update processing on the input through the graph neural network to obtain the connection relationship between the current character and other characters in the local subgraph, wherein, during the update processing, the semantic embedding vector participates in message calculation as static additional information, and its value remains unchanged during multi-layer message passing; the position embedding vector and the visual embedding vector participate in iterative update as dynamic update information. The fusion method may include concatenation, weighted summation, or a combination of mapped features.

[0070] In practice, for each local subgraph, the system traverses the subgraph to obtain the two character nodes connected by each edge, treating each as a neighbor node pair. Further, it obtains the multimodal features bound to the two character nodes in each neighbor node pair: positional embedding vector, visual embedding vector, and semantic embedding vector. Then, based on these vectors, the adjacency messages for the two characters in each neighbor node pair are determined, serving as one adjacency message for the character belonging to the current local subgraph. This method is used to determine all adjacency messages for the character belonging to the current local subgraph. Finally, based on all adjacency messages for the current character, the adjacency message for the current character is determined.

[0071] Optionally, obtaining the adjacency message of the current character based on the position embedding vector, visual embedding vector, and semantic embedding vector of the corresponding character in each of the neighbor node pairs includes: for each neighbor node pair, weighted and fused based on dynamically allocated differential weights on the position embedding vector, visual embedding vector, and semantic embedding vector of the corresponding character in the neighbor node pair to obtain the adjacency message of the neighbor node pair; aggregating the adjacency messages of all the neighbor node pairs of the current character to obtain the adjacency message of the current character. Here, the differential weights are edge-level weights calculated independently for each edge, enabling the multimodal feature contribution between different neighbor node pairs to adaptively change.

[0072] In the embodiments of this application, the features of the visual, positional, and semantic modalities are not simply superimposed in fixed proportions, but are dynamically adjusted through an "edge-level adaptive weighting mechanism." Optionally, based on dynamically allocated differential weights, the positional embedding vector, visual embedding vector, and semantic embedding vector of the corresponding characters in the neighbor node pair are weighted and fused to obtain the adjacency message of the neighbor node pair. This includes: determining the visual similarity feature expression, spatial positional relationship feature expression, and semantic consistency feature expression between the character nodes in the neighbor node pair based on the positional embedding vector, visual embedding vector, and semantic embedding vector of the corresponding characters in the neighbor node pair; dynamically allocating differential weights for the positional embedding vector, visual embedding vector, and semantic embedding vector based on the visual similarity feature expression, spatial positional relationship feature expression, and semantic consistency feature expression; and weighting and fusing the positional embedding vector, visual embedding vector, and semantic embedding vector based on the differential weights to obtain the adjacency message of the neighbor node pair. Following this method, the adjacency messages corresponding to all neighbor node pairs of the character to which the current local subgraph belongs are determined.

[0073] For example, for any pair of character nodes with connecting edges (i.e., neighboring node pairs), a relational feature vector is first constructed based on information such as visual similarity, spatial positional relationship (e.g., geometric distance), and semantic consistency between the two character nodes. This relational feature vector is then input into a lightweight neural network. The neural network outputs the original weight scores for each of the three modalities: visual, positional, and semantic. Subsequently, the original weight scores are normalized (e.g., using Softmax normalization) to obtain the weight coefficients for each of the three modalities, achieving an adaptive allocation of the contribution of different modalities. Afterward, the positional embedding vector, the visual embedding vector, and the semantic embedding vector are weighted and summed based on the adaptively allocated weights to obtain a sum vector, which is then used as the adjacency message for the corresponding neighboring node pair.

[0074] Finally, for each character node, the adjacency messages of all neighboring node pairs to which the character node belongs are aggregated to obtain the adjacency message of the character corresponding to that character node.

[0075] Next, the adjacency message is fused with the position embedding vector and the visual embedding vector of the current character, and the fused features are fed into a graph neural network (e.g., a graph convolutional network) for updating.

[0076] In the embodiments of this application, during the iterative update process of the graph neural network (e.g., graph convolutional network), differentiated weights are assigned to the features of the three modalities of vision, position, and semantics, and the modal contribution is dynamically adjusted according to different scenarios, thereby improving the robustness and adaptability of character detection in cross-language, complex background, or nested scenarios.

[0077] like Figure 2 As shown, during the iterative update process of a graph neural network (e.g., a graph convolutional network), the semantic embedding vector serves as static supplementary information. In each layer, it only participates in the calculation of adjacency messages. Afterward, it is concatenated with the updated messages (i.e., the positional embedding vector and the visual embedding vector) and input to the next layer. The semantic embedding vector remains unchanged during the multi-layer inference process of the graph neural network (e.g., a graph convolutional network) or does not participate in the parameter gradient updates for the graph neural network, thus ensuring the independence and stability of semantic features in multi-layer inference. The graph neural network only updates visual and positional features.

[0078] By keeping the semantic embedding vector unchanged during the iterative update process of the graph neural network (e.g., graph convolutional network), it is possible to effectively avoid secondary damage to semantics caused by "erroneous relations" and ensure that semantic features still represent "real language priors" in multi-layer inference. Especially when character recognition is not completely accurate, occlusion is severe, or there are slight errors in the graph structure, semantic features can still provide stable contextual constraints, improving the overall robustness of text detection.

[0079] After multi-layer inference through a graph neural network (e.g., a graph convolutional network), the connection confidence of each edge in the local subgraph can be obtained. The connection confidence is output by the graph neural network and is used to characterize the probability that there is a valid text combination relationship between two character nodes. For example, the higher the connection confidence, the greater the probability that the characters corresponding to the character nodes connected by the corresponding edge belong to the same text instance. Conversely, the lower the connection confidence, the greater the probability that the characters corresponding to the character nodes connected by the corresponding edge should not be classified as the same text instance. The text instance includes, but is not limited to, any of the following: word, text line, or text block.

[0080] Next, based on the connection relationship, the text detection results of the target text image are analyzed and determined.

[0081] Step 110: Based on the connection relationship of each character, obtain the text detection result of the target text image.

[0082] By combining the connections between characters obtained through inference from graph neural networks (such as graph convolutional networks), the characters are correctly combined into words, lines of text, or blocks of text, thus obtaining the text detection results.

[0083] Optionally, the text detection result of the target text image is obtained based on the connection relationship of each character, including: obtaining the adjacency confidence score between characters based on the connection relationship; performing connected component analysis on character pairs whose adjacency confidence score exceeds a preset threshold, and aggregating interconnected characters into the same candidate text instance; performing hierarchical clustering of the candidate text instances according to the spatial position and main direction of the characters to obtain the text detection result of the target text image, wherein the text detection result includes one or more of the following: words, text lines, and text blocks. The main direction is determined based on the character spatial distribution, character mask shape, or character centroid fitting result.

[0084] For a specific example, firstly, character-level adjacency relationships are constructed based on the adjacency confidence of edges output by a graph neural network (e.g., a graph convolutional network). Connectivity component analysis is performed on character pairs whose adjacency confidence exceeds a preset threshold, aggregating interconnected characters into the same candidate text instance. Within each candidate instance, characters are sorted according to their reading order based on their centroid coordinates and principal direction vectors, achieving a combination of character-level results and word-level results. Subsequently, multiple word instances with similar spatial locations and consistent principal directions are clustered to form text line-level combinations, i.e., text lines. Further, based on layout features such as line spacing and alignment, multiple lines of text are merged into text blocks. One or more of words, text lines, and text blocks can be used as text detection results. The principal direction vector of a character can be determined based on the spatial distribution of characters in the connected components, for example, through principal component analysis or line fitting of the character centroid coordinates, or based on the direction of the minimum bounding rectangle of the character mask. The reading order can be determined according to the specific application scenario, such as the usual reading order from left to right or from top to bottom.

[0085] by Figure 3 Taking the target text image shown as an example, the complex image and text detection and extraction method disclosed in the embodiments of this application is used to... Figure 3 Text detection can be performed on the target text image shown, resulting in the following: Figure 4 The text detection results are shown within the rectangle in the image. Figure 4 As shown, the text detection results include: words, text lines, and text blocks.

[0086] In summary, the complex image-text detection and extraction method disclosed in this application performs query-based character-level detection on the target text image to obtain character-level masks, category prediction results, and updated query vectors, refining the initial detection granularity and effectively solving erroneous segmentation problems in scenarios such as overlap, concatenation, and mixed text. Based on the mask, the category prediction results, and the query vector, multimodal embedding vectors for each character are obtained, including positional embedding vectors, semantic embedding vectors, and visual embedding vectors. Using the positional embedding vectors as positional constraints and the visual embedding vectors as visual constraints, local subgraphs for each character are constructed. The character-level detection results are further filtered by combining multimodal features to reduce redundancy and noise and improve text detection accuracy. The semantic embedding vectors are combined to infer the connection relationships of the characters based on the local subgraphs. Based on the connection relationships of each character, the text detection results of the target text image are obtained, effectively improving the accuracy and precision of text detection. When using the complex image and text detection and extraction method disclosed in the embodiments of this application to perform text detection on nested text, the detection accuracy is significantly improved, and compared with traditional methods, it can effectively reduce ambiguity and errors in complex scenarios.

[0087] Based on the above embodiments, this embodiment also provides a complex image and text detection and extraction device, such as... Figure 5 As shown, the device includes:

[0088] The character-level detection module 502 is used to perform query-based character-level detection on the target text image and obtain the character-level mask, category prediction results and updated query vector;

[0089] The multimodal embedding vector acquisition module 504 is used to acquire the multimodal embedding vector of each character based on the mask, the category prediction result and the query vector. The multimodal embedding vector includes: positional embedding vector, semantic embedding vector and visual embedding vector.

[0090] The local subgraph construction module 506 is used to construct local subgraphs for each character using the position embedding vector as a positional constraint and the visual embedding vector as a visual constraint.

[0091] The connection relationship reasoning module 508 is used to combine the semantic embedding vector and infer the connection relationship of the characters based on the local subgraph.

[0092] The text detection module 510 is used to obtain the text detection result of the target text image based on the connection relationship of each character.

[0093] Optionally, the step of constructing local sub-graphs for each character using the positional embedding vector as a positional constraint and the visual embedding vector as a visual constraint includes:

[0094] The spatial proximity relationship between each character is determined based on the positional embedding vector, and the visual similarity between each character is determined based on the visual embedding vector;

[0095] For each first character, based on the spatial proximity relationship and the visual similarity, one or more second characters that are spatially adjacent to and visually similar to the first character are determined as a candidate combination character set for the first character, wherein the first character and the second character are characters in different positions;

[0096] Establish the character node corresponding to the character, and connect each character with its candidate combination characters through edges to obtain a local subgraph of the character.

[0097] Optionally, the step of jointly inferring the connection relationships of the characters based on the local subgraph using the semantic embedding vector includes:

[0098] Based on the local subgraph, the neighbor nodes of the character node corresponding to the current character are obtained, and multiple neighbor node pairs of the current character are obtained, consisting of the corresponding character node and the neighbor node.

[0099] Based on the position embedding vector, visual embedding vector, and semantic embedding vector of the character corresponding to the character node in each of the neighbor node pairs, the adjacency message of the current character is obtained;

[0100] The adjacency message and the position embedding vector and visual embedding vector of the current character are fused together and used as input to a graph neural network. The graph neural network performs multi-layer message passing and update processing on the input to obtain the connection relationship between the current character and other characters in the local subgraph. In the update processing, the semantic embedding vector participates in message calculation as static additional information, and its value remains unchanged during multi-layer message passing. The position embedding vector and the visual embedding vector serve as dynamic update information.

[0101] Optionally, obtaining the adjacency message of the current character based on the position embedding vector, the visual embedding vector, and the semantic embedding vector of the corresponding character in each of the neighbor node pairs includes:

[0102] For each neighbor node pair, based on dynamically allocated differential weights, the position embedding vector, the visual embedding vector, and the semantic embedding vector of the character corresponding to the character node in the neighbor node pair are weighted and fused to obtain the adjacency message of the neighbor node pair;

[0103] The adjacency messages of all neighbor node pairs of the current character are aggregated to obtain the adjacency message of the current character.

[0104] Optionally, obtaining the text detection result of the target text image based on the connection relationship of each of the characters includes...

[0105] Based on the connection relationship, obtain the confidence level of the adjacency relationship between characters;

[0106] For character pairs whose adjacency confidence exceeds a preset threshold, perform connected component analysis to aggregate interconnected characters into the same candidate text instance;

[0107] Based on the spatial location and main direction of the characters, the candidate text instances are clustered step by step to obtain the text detection results of the target text image. The text detection results include one or more of the following: words, text lines, and text blocks.

[0108] Optionally, the step of performing query-based character-level detection on the target text image to obtain character-level masks, category prediction results, and updated query vectors includes:

[0109] Multi-scale feature extraction is performed on the target text image to obtain multi-scale visual features;

[0110] A character-level learnable query vector is introduced, and the updated query vector is obtained by decoding and updating the multi-scale visual features.

[0111] Based on the updated query vector and the high-resolution multi-scale visual features in the multi-scale visual features, a character-level mask is obtained; and, based on the updated query vector, classification prediction is performed to obtain a character-level category prediction result.

[0112] Optionally, obtaining the multimodal embedding vector of each character based on the mask, the category prediction result, and the query vector includes:

[0113] Geometric features are extracted based on the mask corresponding to each character to obtain the position embedding vector of the corresponding character.

[0114] Semantic encoding is performed on the category prediction results of each character to obtain the semantic embedding vector of the corresponding character;

[0115] Based on the query vector of each character, obtain the visual embedding vector of the corresponding character;

[0116] The positional embedding vector, semantic embedding vector, and visual embedding vector of each character are used as the multimodal embedding vector of the corresponding character. The complex image and text detection and extraction apparatus disclosed in this application is used to implement the above-described complex image and text detection and extraction method. For specific implementation methods of each module of the apparatus, please refer to the specific implementation methods of the corresponding steps in the foregoing method embodiments, which will not be repeated here.

[0117] In summary, the complex image and text detection and extraction device disclosed in this application performs query-based character-level detection on the target text image to obtain a character-level mask, category prediction results, and an updated query vector, refining the initial detection granularity and effectively solving erroneous segmentation problems in scenarios such as overlap, concatenation, and mixed text. Based on the mask, the category prediction results, and the query vector, it obtains multimodal embedding vectors for each character, including positional embedding vectors, semantic embedding vectors, and visual embedding vectors. Using the positional embedding vectors as positional constraints and the visual embedding vectors as visual constraints, it constructs local subgraphs for each character. Combining multimodal features, it further filters the character-level detection results, reducing redundancy and noise and improving text detection accuracy. Combined with the semantic embedding vectors, it infers the connection relationships of the characters based on the local subgraphs. Based on the connection relationships of the characters, it obtains the text detection results of the target text image, effectively improving the accuracy and precision of text detection. When using the complex image and text detection and extraction device disclosed in the embodiments of this application to perform text detection on nested text, the detection accuracy is significantly improved, and compared with traditional methods, it can effectively reduce ambiguity and errors in complex scenarios.

[0118] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus embodiments, since they are fundamentally similar to the method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0119] The above provides a detailed description of a complex image and text detection and extraction method and apparatus provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method of this application and its core idea. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the idea of ​​this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0121] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the electronic device according to the embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0122] For example, Figure 6 An electronic device is shown that can implement the methods according to this application. The electronic device may be a PC, mobile terminal, personal digital assistant, tablet computer, etc. The electronic device conventionally includes a processor 610 and a memory 620 communicatively connected to the processor, and program code 630 stored on the memory 620 and executable on the processor 610, which, when executing the program code 630, implements the methods described in the above embodiments. The memory 620 may be a computer program product or a computer-readable medium. The memory 620 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory 620 has a storage space 6201 for the program code 630 of a computer program for performing any of the method steps described above. For example, the storage space 6201 for the program code 630 may include various computer programs for implementing the various steps in the methods described above. The program code 630 is computer-readable code. These computer programs can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, compact discs (CDs), memory cards, or floppy disks. The computer program includes computer-readable code that, when executed on an electronic device, causes the electronic device to perform the methods according to the embodiments described above.

[0123] This application also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps described in this application.

[0124] Such a computer program product can be a computer-readable storage medium, which can have the same characteristics as... Figure 6The memory 620 in the illustrated electronic device is similarly arranged with storage segments, storage spaces, etc. Program code can be stored, for example, in a compressed form on the computer-readable storage medium. The computer-readable storage medium is typically as shown in the reference... Figure 7 The portable or fixed storage unit is described above. Typically, the storage unit includes computer-readable code 630', which is code read by a processor and, when executed by the processor, implements the various steps of the method described above.

[0125] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this application. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0126] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0127] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for detecting and extracting complex images and text, characterized in that, The method includes: Perform query-based character-level detection on the target text image to obtain the character-level mask, category prediction results, and updated query vector; Based on the mask, the category prediction result, and the query vector, a multimodal embedding vector for each character is obtained. The multimodal embedding vector includes: a positional embedding vector, a semantic embedding vector, and a visual embedding vector. Using the positional embedding vector as a positional constraint and the visual embedding vector as a visual constraint, a local sub-image of each character is constructed. By combining the semantic embedding vectors, the connection relationships of the characters are inferred based on the local subgraph; Based on the connection relationships of each character, the text detection result of the target text image is obtained; wherein, the step of jointly inferring the connection relationships of the characters based on the local subgraph by combining the semantic embedding vector includes: Based on the local subgraph, the neighbor nodes of the character node corresponding to the current character are obtained, and multiple neighbor node pairs of the current character are obtained, consisting of the corresponding character node and the neighbor node. Based on the position embedding vector, visual embedding vector, and semantic embedding vector of the character corresponding to the character node in each of the neighbor node pairs, the adjacency message of the current character is obtained; The adjacency message and the position embedding vector and visual embedding vector of the current character are fused together and used as input to a graph neural network. The graph neural network performs multi-layer message passing and update processing on the input to obtain the connection relationship between the current character and other characters in the local subgraph. In the update processing, the semantic embedding vector participates in message calculation as static additional information, and its value remains unchanged during multi-layer message passing. The position embedding vector and the visual embedding vector serve as dynamic update information.

2. The method according to claim 1, characterized in that, The step of constructing local sub-graphs for each character using the positional embedding vector as a positional constraint and the visual embedding vector as a visual constraint includes: The spatial proximity relationship between each character is determined based on the positional embedding vector, and the visual similarity between each character is determined based on the visual embedding vector; For each first character, based on the spatial proximity relationship and the visual similarity, one or more second characters that are spatially adjacent to and visually similar to the first character are determined as a candidate combination character set for the first character, wherein the first character and the second character are characters in different positions; Establish the character node corresponding to the character, and connect each character with its candidate combination characters through edges to obtain a local subgraph of the character.

3. The method according to claim 1, characterized in that, The process of obtaining the adjacency message of the current character based on the position embedding vector, visual embedding vector, and semantic embedding vector of the character corresponding to the character node in each of the neighbor node pairs includes: For each neighbor node pair, based on dynamically allocated differential weights, the position embedding vector, the visual embedding vector, and the semantic embedding vector of the character corresponding to the character node in the neighbor node pair are weighted and fused to obtain the adjacency message of the neighbor node pair; The adjacency messages of all neighbor node pairs of the current character are aggregated to obtain the adjacency message of the current character.

4. The method according to claim 1, characterized in that, The text detection result of the target text image is obtained based on the connection relationship of each character, including... Based on the connection relationship, obtain the confidence level of the adjacency relationship between characters; For character pairs whose adjacency confidence exceeds a preset threshold, perform connected component analysis to aggregate interconnected characters into the same candidate text instance; Based on the spatial location and main direction of the characters, the candidate text instances are clustered step by step to obtain the text detection results of the target text image. The text detection results include one or more of the following: words, text lines, and text blocks.

5. The method according to claim 1, characterized in that, The step of performing query-based character-level detection on the target text image to obtain character-level masks, category prediction results, and updated query vectors includes: Multi-scale feature extraction is performed on the target text image to obtain multi-scale visual features; A character-level learnable query vector is introduced, and the updated query vector is obtained by decoding and updating the multi-scale visual features. Based on the updated query vector and the high-resolution multi-scale visual features in the multi-scale visual features, a character-level mask is obtained; and, based on the updated query vector, classification prediction is performed to obtain a character-level category prediction result.

6. The method according to claim 1, characterized in that, The step of obtaining the multimodal embedding vector of each character based on the mask, the category prediction result, and the query vector includes: Geometric features are extracted based on the mask corresponding to each character to obtain the position embedding vector of the corresponding character. Semantic encoding is performed on the category prediction results of each character to obtain the semantic embedding vector of the corresponding character; Based on the query vector of each character, obtain the visual embedding vector of the corresponding character; The positional embedding vector, semantic embedding vector, and visual embedding vector of each character are used as the multimodal embedding vector of the corresponding character.

7. A complex image and text detection and extraction device, characterized in that, The device includes: The character-level detection module is used to perform query-based character-level detection on target text images, and obtain character-level masks, category prediction results and updated query vectors; The multimodal embedding vector acquisition module is used to acquire the multimodal embedding vector of each character based on the mask, the category prediction result and the query vector. The multimodal embedding vector includes: positional embedding vector, semantic embedding vector and visual embedding vector. The local subgraph construction module is used to construct local subgraphs for each character using the positional embedding vector as a positional constraint and the visual embedding vector as a visual constraint. The connection relationship reasoning module is used to combine the semantic embedding vectors and infer the connection relationships of the characters based on the local subgraph. A text detection module is used to obtain the text detection result of the target text image based on the connection relationships of each character; wherein, the step of jointly inferring the connection relationships of the characters based on the local subgraph by combining the semantic embedding vector includes: Based on the local subgraph, the neighbor nodes of the character node corresponding to the current character are obtained, and multiple neighbor node pairs of the current character are obtained, consisting of the corresponding character node and the neighbor node. Based on the position embedding vector, visual embedding vector, and semantic embedding vector of the character corresponding to the character node in each of the neighbor node pairs, the adjacency message of the current character is obtained; The adjacency message and the position embedding vector and visual embedding vector of the current character are fused together and used as input to a graph neural network. The graph neural network performs multi-layer message passing and update processing on the input to obtain the connection relationship between the current character and other characters in the local subgraph. In the update processing, the semantic embedding vector participates in message calculation as static additional information, and its value remains unchanged during multi-layer message passing. The position embedding vector and the visual embedding vector serve as dynamic update information.

8. An electronic device, comprising a memory, a processor, and program code stored in the memory and executable on the processor, characterized in that, When the processor executes the program code, it implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having program code stored thereon, characterized in that, When the program code is executed by the processor, it implements the steps of the method described in any one of claims 1 to 6.

Images