A multi-directional text positioning method based on transformer

Abstract

The application discloses a multi-directional text positioning method based on a Transformer, which comprises the following steps: preprocessing an initial text image to obtain an intermediate image; inputting the intermediate image into a text positioning model; the text positioning model is a Transformer text positioning model comprising a backbone network, an encoder, a feature fusion layer and a decoder; first image features are obtained through the backbone network; second image features are obtained through the encoder; the feature fusion layer is used for fusing the first image features and the second image features to obtain a rectangular frame containing target text; based on the end point order and the end point coordinate values corresponding to the multiple end points of the rectangular frame, the rotation angle corresponding to the initial text image is determined; and the rectangular frame is rotated based on the rotation angle to obtain a forward text image. The four-point marking frame is output through the decoder, and the text area of any direction and shape can be flexibly adapted, so that the problem that the rotated text and the inclined text cannot be accurately positioned is effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of text recognition technology, specifically relating to a multi-directional text localization method based on Transformer. Background Technology

[0002] With the development of information technology, Optical Character Recognition (OCR) technology has been widely used in document processing, information retrieval, and other fields. Especially in scenarios such as automated office work, financial document processing, and identity verification, accurately locating and recognizing text content is a key step in achieving efficient data processing. Traditional OCR technology mainly relies on fixed-directional text recognition, and traditional text localization methods primarily depend on convolutional neural networks (CNNs) or object detection frameworks (such as Faster R-CNN and SSD), whose output is typically a standard rectangular box defined by two points. In recent years, deep learning technology, especially models based on the Transformer architecture, has made significant progress in the field of natural language processing and is gradually being applied to computer vision tasks, providing new ideas for solving the aforementioned problems.

[0003] However, in real-world applications, text content often exhibits multiple orientations (e.g., 90°, 180°, 270° rotation), irregular layouts, or even non-rectangular structures (e.g., diagonal or curved text). Early OCR systems typically employed manual feature extraction methods such as edge detection and morphological operations to locate text regions. While these methods could achieve some success under specific conditions, their generalization ability was poor, making it difficult to handle multi-directional or irregularly shaped text in complex backgrounds. With the development of deep learning, many studies began using CNNs for text detection and recognition. For example, some methods utilize object detection frameworks such as Faster R-CNN and SSD to predict text box positions. However, these methods are limited by fixed-shape rectangular bounding boxes and cannot flexibly adapt to rotated or tilted text.

[0004] Therefore, there is an urgent need to develop a new text localization method that can adapt to multi-directional text and accurately capture text regions of arbitrary shapes in order to overcome the above challenges and improve the overall performance of OCR systems. Summary of the Invention

[0005] This invention provides a multi-directional text localization method based on Transformer to solve the problem of low text information extraction accuracy of OCR when the image orientation is incorrect and the text shape is irregular.

[0006] The technical solution adopted in this invention is as follows:

[0007] A multi-directional text localization method based on Transformer.

[0008] The initial text image is preprocessed to obtain an intermediate image;

[0009] The intermediate image is input into the text localization model, which is a Transformer text localization model including a backbone network, an encoder, a feature fusion layer and a decoder. The backbone network is used to obtain the first image features, the encoder is used to obtain the second image features, and the feature fusion layer is used to fuse the first image features and the second image features to obtain a rectangular frame containing the target text.

[0010] Based on the endpoint order and endpoint coordinate values ​​corresponding to the multiple endpoints of the rectangular frame, the rotation angle corresponding to the initial text image is determined.

[0011] The rectangular frame is rotated based on the rotation angle to obtain a forward-facing text image.

[0012] The multi-directional text localization method based on Transformer provided in this invention also has the following additional technical features:

[0013] The initial text image is preprocessed to obtain the intermediate image, specifically as follows:

[0014] Calculate the pixel mean and standard deviation of the three channels of the initial text image;

[0015] The initial text image is standardized based on the pixel mean and the standard deviation to form the intermediate image.

[0016] The first image features are obtained through the backbone network, and the second image features are obtained through the encoder, specifically as follows:

[0017] The backbone network extracts multi-layer feature maps from the intermediate image, and the multi-layer feature maps are obtained with different size parameters;

[0018] By downsampling, the multi-layer feature map is scaled up to match the output size of the Transformer encoder, resulting in multiple feature vector sequences.

[0019] The multiple feature vector sequences are specifically as follows:

[0020] A one-to-one corresponding query vector is obtained based on the feature vector sequence;

[0021] The confidence parameter is obtained using the Hungarian algorithm based on the query vector.

[0022] Multiple feature vector sequences are sorted according to the confidence parameter and then filtered based on the multiple feature vector sequences;

[0023] The filtered feature vector sequence is weighted using the confidence parameter.

[0024] The feature fusion layer is used to fuse the first image features and the second image features to obtain a rectangular frame that includes the target text, specifically:

[0025] The multi-layer feature maps output by the backbone network are concatenated with the feature vectors output by the encoder along the channel dimension to obtain the input tensor of the decoder.

[0026] The decoder outputs the four coordinates and confidence score of the initial image text based on the input tensor.

[0027] Based on the endpoint order and coordinate values ​​of the multiple endpoints of the rectangular frame, the rotation angle corresponding to the initial text image is determined, specifically as follows:

[0028] The multiple endpoint coordinates are the vertex coordinates of the rectangular frame. The vertex coordinates are ordered, and the text rotation angle is calculated based on the difference in the horizontal coordinates of the left and right vertices in the vertex coordinates.

[0029] An affine transformation is performed on the initial text image to generate a forward image rotated by the corresponding angle.

[0030] The multi-directional text localization method also includes:

[0031] The image text is rotated and annotated to form a pre-trained text sample, which includes the current coordinate values ​​and the true coordinate values ​​of the four endpoints.

[0032] The text localization model is determined by training based on the current coordinates and the true coordinates using a custom loss function.

[0033] The text localization model is determined by training based on the current coordinates and the true coordinates using a custom loss function, specifically as follows:

[0034] The text localization model generates the current coordinates of the four endpoints of the pre-trained text sample and the predicted rectangular frame.

[0035] Based on the L1 distance between the true coordinates and the current coordinates of the pre-trained text samples, calculate the sum of the absolute values ​​of the component differences in the XY direction for each endpoint;

[0036] Based on the overlapping area formed by the rectangular frame of the pre-trained text sample and the predicted rectangular frame, calculate the ratio of the overlapping area of ​​the two quadrilaterals to the total area.

[0037] The text positioning model is determined based on the sum of the absolute values ​​and the ratio.

[0038] The present invention also provides a storage medium,

[0039] The storage medium stores a computer program, which, when executed, implements the steps of any of the Transformer-based multi-directional text positioning methods described above.

[0040] The present invention further provides a processing apparatus, comprising:

[0041] Memory, used to store computer programs;

[0042] A processor, configured to implement the steps of any of the Transformer-based multi-directional text positioning methods when executing the computer program.

[0043] Due to the adoption of the above technical solution, the beneficial effects achieved by this invention are as follows:

[0044] The method proposed in this invention preprocesses an initial text image to obtain an intermediate image, inputs the intermediate image into a Transformer text localization model composed of a backbone network, an encoder, a feature fusion layer, and a decoder, extracts first image features and second image features respectively, and fuses the two through the feature fusion layer to generate a rectangular frame containing the target text; further, the rotation angle of the text is determined according to the endpoint order and coordinate values ​​of the rectangular frame, and the text image is rotated and corrected accordingly to obtain a forward text image.

[0045] By outputting a four-point annotation box through a decoder, it can flexibly adapt to text areas of any direction and shape, effectively solving the problem of inaccurate positioning of rotated or tilted text.

[0046] By employing the self-attention and cross-attention mechanisms in the Transformer architecture, and combining a multi-scale feature fusion strategy between the backbone network and the encoder output, the model's ability to understand the contextual relationships of text regions is improved, and its robustness to complex situations such as small characters, blurriness, and occlusion is enhanced.

[0047] By analyzing and predicting the coordinate order and positional relationship of the endpoints of the rectangular frame, the text rotation angle is automatically calculated, and affine transformation is performed to correct the text orientation. This allows the output positive text image to be directly used in subsequent recognition modules, improving the overall efficiency and recognition accuracy of the OCR system.

[0048] This method has a clear structure and a good modular design, making it easy to deploy on mobile or embedded devices. It is applicable to various real-world scenarios such as invoice recognition, document scanning, and form parsing, and has broad practical value and potential for widespread adoption. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0050] Figure 1 This is a flowchart illustrating the multi-directional text localization method based on Transformer according to one embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of the specific form of the annotation box of the Transformer-based multi-directional text positioning method according to one embodiment of the present invention. Detailed Implementation

[0052] To more clearly illustrate the overall concept of the present invention, a detailed description will be provided below with reference to the accompanying drawings and examples.

[0053] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0054] like Figure 1 , Figure 2 As shown, a multi-directional text localization method based on Transformer includes:

[0055] S100: Preprocess the initial text image to obtain an intermediate image.

[0056] The core objective of this step is to perform a series of preprocessing operations on the original image containing text. The original image may contain noise from the scanning / capturing process (such as salt-and-pepper noise), reflections, blur, or low resolution. Denoising (such as median filtering and Gaussian filtering) and sharpening processes can eliminate these interferences, preserve key text features, and prevent the model from misjudging due to noise and causing localization errors. Additionally, the contrast and brightness of the initial text image are enhanced. When extracting images in unevenly lit scenes (such as scanned documents or mobile phone photos), insufficient contrast between text and background can reduce recognition accuracy. Histogram equalization and contrast stretching operations make the text more clearly visible.

[0057] By employing image standardization, text localization becomes easier in complex scenarios, such as in business documents like invoices and licenses, where text may be difficult to locate due to scanning angle deviations or cluttered backgrounds. Preprocessing, through standardization and denoising, ensures the model can accurately capture text regions. In multilingual and multidirectional text processing, such as scenarios involving Arabic ligatures or mixed Hindi text, preprocessing needs to adjust parameters based on language characteristics (e.g., color space conversion, edge detection). For real-time OCR needs on mobile devices, where documents captured by mobile phones often suffer from tilt and blurriness, preprocessing combined with lightweight algorithms (such as MobileNetV3) enables rapid correction, ensuring real-time performance.

[0058] S200: The intermediate image is input into the text localization model, which is a Transformer text localization model including a backbone network, an encoder, a feature fusion layer and a decoder. The backbone network is used to obtain the first image features, the encoder is used to obtain the second image features, and the feature fusion layer is used to fuse the first image features and the second image features to obtain a rectangular frame containing the target text.

[0059] The text localization model is based on the Transformer architecture and includes core components: backbone network, encoder, feature fusion layer, and decoder. The backbone network extracts the basic features of the image, namely the first image features of the text image. Common backbone network structures include ResNet50 and ResNet18. The ResNet50 backbone network reduces the output feature map size by 32 times and increases the number of channels to 2048.

[0060] The encoder encodes the features output by the backbone network to generate second image features, capturing global contextual information through a self-attention mechanism. Based on the first image features output by the backbone network, these are input to the encoder part of the Transformer, where the image features of each channel are flattened into a feature vector. A feature query method is used to provide a high-quality initial query vector for the decoder.

[0061] Feature-based query methods are techniques for generating high-quality query vectors by analyzing image features (such as local or global features of text regions). Their core objective is to provide the decoder with an initial query vector strongly correlated with the text region, thereby guiding the decoder to locate the text region more efficiently. Specifically, features related to the text region (such as edges, textures, and shapes) are extracted from the first image features output by the backbone network (such as ResNet50). For example, local features of text lines are extracted through convolution operations, or the overall semantic information of the text region is obtained through global pooling. Through feature selection and aggregation, attention mechanisms (such as self-attention or channel attention) are used to select the features most relevant to the text region; through weighted summation or concatenation operations, multi-scale features (such as low-level edge features and high-level semantic features) are aggregated into a unified feature vector.

[0062] The aggregated feature vector can be used as the initial input to the query vector, or it can be mapped to the query vector through a linear transformation (such as a fully connected layer). For example, the feature vector of the center point of the text region can be used as the initial value of the query vector to ensure that the decoder focuses on potential text regions first.

[0063] By pre-screening features that are strongly correlated with the text, the decoder is prevented from generating a large number of irrelevant prediction boxes; at the same time, the initial query vector is directly associated with the key features of the text region, enabling the decoder to quickly focus on the target region.

[0064] The initial query vector is the first input vector used by the decoder during inference, guiding the interaction between the cross-attention mechanism and the encoder features. Initialization based on image features extracts the latent locations of text regions from the feature map output by the backbone network (e.g., through sliding windows or clustering algorithms), generating corresponding feature vectors as the initial query vector. Initialization based on prior knowledge combines prior information from the text detection task (e.g., typical height, width, or orientation of text lines) to design a query vector with a fixed pattern. For example, assuming horizontal text lines, the initial query vector can include horizontal attention weights. The training process automatically learns how to generate the initial query vector, adapting it to text region features in different scenarios.

[0065] Specifically, the self-attention mechanism captures the global contextual relationships of image features through multi-head self-attention, and computes multiple attention heads in parallel to focus on text region features in different directions or scales to obtain the second image features (high-dimensional features, i.e., vector sequences).

[0066] The self-attention mechanism generates query vectors, key vectors, and value vectors, calculates the correlation between each element in the input sequence and other elements, and dynamically adjusts the level of attention given to each position. The final detection result is generated by combining the query vector and encoded features through a cross-attention mechanism. The query vector is initialized using a feature query method to provide high-quality initial localization information.

[0067] The feature fusion layer is used to fuse the first and second image features to enhance multi-scale information. The outputs of the first and second layers of the backbone network are downsampled to the same size as the encoder output and then concatenated at the channel level. The feature fusion layer then combines these two sets of features to form the final feature representation. This feature representation helps the model identify regions containing the target text and confine them within a rectangular frame.

[0068] The second image features output by the decoder (encoder) are combined with the first image features of the backbone network through a feature fusion layer (such as channel splicing or weighted summation) to form a richer feature representation.

[0069] In the query vector input to the decoder, there is a one-to-one correspondence between the detected targets in the image and the classification and localization information contained in the query vector. To provide the decoder with higher quality query vectors, a confidence score is added to the encoder output and incorporated into the loss function. When the query vector does not contain an object present in the corresponding image, the confidence score is reduced. During inference, the top K queries with the highest confidence scores are selected. This effectively improves query quality.

[0070] Extract local features that are strongly correlated with text regions from the fused features (e.g., through max pooling or attention mechanisms).

[0071] S300: Based on the endpoint order and endpoint coordinate values ​​corresponding to the multiple endpoints of the rectangular frame, determine the rotation angle corresponding to the initial text image.

[0072] After identifying the text region, the orientation of the text—that is, the rotation angle relative to the forward-facing text image—is determined based on the positions and order of the four endpoints of the rectangular frame. This process is crucial for correctly understanding and processing text in different orientations.

[0073] S400: Rotate the rectangular frame based on the rotation angle to obtain a forward-facing text image.

[0074] Using the previously determined rotation angle, the identified text region is rotated back to the correct orientation (positive direction). This ensures that the output text image conforms to normal reading habits, facilitating subsequent text recognition. This not only improves the accuracy of text recognition but also reduces recognition errors caused by incorrect text orientation.

[0075] In summary, the multi-directional text localization method based on Transformer proposed in this invention includes the following steps: First, the input image is preprocessed by standardization, rotation enhancement, and other operations to construct a training dataset adapted for multi-directional text. Then, the image is input into a text localization model based on the Transformer architecture. This model consists of a backbone network (such as ResNet50), an encoder, a feature fusion layer, and a decoder. The encoder extracts global contextual information through a self-attention mechanism, and the decoder combines a high-quality initial query vector generated by a feature query method. Through multi-level feature extraction and multi-task joint optimization, the four-point coordinates of the text region are gradually predicted using a cross-attention mechanism, achieving accurate localization of text of arbitrary direction and shape, laying a solid foundation for subsequent text recognition and processing.

[0076] The Transformer-based multi-directional text localization method breaks through the limitation of traditional methods that can only output standard rectangular boxes, supports arbitrary quadrilateral annotations, and significantly improves the localization accuracy of rotated text and irregular text regions. At the same time, the initial query vector is optimized through feature query method, which improves the model convergence speed and detection efficiency. It has good generalization ability and practicality, and is suitable for OCR application needs in complex scenarios.

[0077] As a preferred embodiment of the invention, the method includes preprocessing the initial text image to obtain the intermediate image, specifically: calculating the pixel mean and standard deviation of the three channels of the initial text image; and standardizing the initial text image based on the pixel mean and standard deviation to form the intermediate image.

[0078] The initial text image is standardized by calculating the mean (μ) and standard deviation (σ) of the pixel values ​​in the three channels. The pixel values ​​are then normalized to zero mean and unit variance to make the distribution of each pixel value conform to a standard normal distribution (mean is 0, variance is 1).

[0079] The formula is: After standardization, the pixel values ​​of text images are adjusted to a specific range and possess specific statistical characteristics. When a model is trained on a standardized dataset, optimization algorithms such as gradient descent converge faster, the model's generalization ability is improved, overfitting due to differences in input data distribution is avoided, the values ​​are more stable, and gradient vanishing and gradient exploding phenomena are avoided to some extent.

[0080] Scale the image to a fixed size required by the model (e.g., 224×224 or 512×512) to ensure the input consistency of the intermediate image to the backbone network (e.g., ResNet50).

[0081] In a preferred embodiment of the present invention, a first image feature is obtained through the backbone network, and a second image feature is obtained through the encoder. Specifically, the backbone network extracts multi-layer feature maps from the intermediate image, and the multi-layer feature maps are obtained with different size parameters. The multi-layer feature maps are scaled down by downsampling to make the multi-layer feature maps consistent with the output size of the Transformer encoder, thereby obtaining multiple feature vector sequences.

[0082] The backbone network of this text localization model (such as ResNet50 or ResNet18) performs multi-level convolutional operations on the input intermediate image, extracting feature maps with different spatial resolutions and semantic levels layer by layer. These include shallow features, mid-level features, and high-level features. Shallow features extract low-level visual information, such as details like edges, corners, and colors, corresponding to a larger spatial size (e.g., H / 4 × W / 4) and fewer channels (e.g., 256 dimensions). These features help to accurately locate the boundaries of text lines. Mid-level features extract local structures with some semantic information, such as character strokes and connected regions, further reducing the spatial size (e.g., H / 8 × W / 8) and increasing the number of channels (e.g., 512 dimensions). High-level features output high-semantic-level global features, reflecting the overall structure and arrangement pattern of text lines, with the smallest spatial size (e.g., H / 32 × W / 32) and the most channels (e.g., 2048 dimensions). Finally, a multi-layer feature map sequence is obtained, denoted as F1, F2, ..., F... n Each layer represents image features at different scales.

[0083] To ensure compatibility between the multi-layer feature maps and the processing capabilities of the subsequent Transformer encoder, the spatial dimensions of these feature maps need to be adjusted to match the spatial dimensions of the encoder output. This is achieved by downsampling the multi-layer feature maps to ensure they match the output size of the Transformer encoder. Specifically, for each feature map F... iUse convolutional or pooling layers with a stride of 2 to progressively downsample until the spatial dimensions (height × width) match the features output by the Transformer encoder. Figure 1 To achieve (e.g., a resolution of H / 32×W / 32); each downsampled feature map F i The feature vector is converted into a one-dimensional feature vector sequence (in patch embedding form), which is expanded into multiple tokens according to spatial location, making it easier to input into the Transformer encoder for self-attention modeling.

[0084] After downsampling and vectorizing the multi-layer feature maps, these features are input into the Transformer encoder for global context modeling. Since the Transformer itself does not contain positional information, positional encoding needs to be added to each token to preserve its spatial relationships.

[0085] Multi-head self-attention mechanism: Each token calculates its relevance to other tokens, capturing long-distance dependencies in the image. Through this multi-head mechanism, the model can simultaneously focus on text features at different scales, orientations, and semantic levels.

[0086] Feedforward Neural Network (FFN) Processing: Each token goes through two fully connected layers for nonlinear transformation and feature enhancement.

[0087] The fusion methods include channel concatenation, weighted summation, and cross-layer residual connections. Channel concatenation combines backbone network features and encoder features at the same spatial location along the channel dimension, forming a richer feature representation. Weighted summation introduces attention weights to weightedly combine features from different sources, emphasizing the parts more meaningful for the current task. Cross-layer residual connections, similar to the idea of ​​ResNet, introduce skip connections in the decoder to preserve information from the original features and avoid gradient vanishing. This fusion method enhances the model's ability to perceive text regions, especially in scenes with small text, blurry images, or complex backgrounds, improving localization robustness.

[0088] This processing method not only preserves the rich information in the original image but also achieves effective fusion of features at different levels by adjusting the size of the feature maps. This helps improve the model's ability to understand target objects (such as text and objects) in the image, especially when dealing with complex scenes or images containing information at multiple scales. Furthermore, after such preprocessing, the Transformer encoder can focus more on capturing global contextual information and long-range dependencies, thereby further improving the performance of the final application (such as image recognition, object detection, etc.).

[0089] In a preferred embodiment of the present invention, the plurality of feature vector sequences specifically comprises: obtaining a one-to-one corresponding query vector based on the feature vector sequences; obtaining a confidence parameter based on the query vector using the Hungarian algorithm; sorting the plurality of feature vector sequences based on the confidence parameter and filtering the plurality of feature vector sequences; and weighting the filtered feature vector sequences using the confidence parameter.

[0090] After obtaining multiple feature vector sequences, further processing is performed to improve the model's accuracy and efficiency. First, a one-to-one query vector is generated for each feature vector sequence; these query vectors serve as the basis for specific information retrieval or matching tasks. Next, the Hungarian algorithm is used to calculate the confidence parameter between each query vector and the target. This algorithm is particularly well-suited for solving assignment problems, determining the optimal matching scheme by maximizing the overall quality of the matching. The confidence parameter reflects the reliability of the association between the query vector and the expected target.

[0091] By sorting and filtering multiple feature vector sequences based on the obtained confidence parameters, low-confidence feature vector sequences that may contain erroneous information can be removed, ensuring the efficiency and accuracy of subsequent processing. Furthermore, the filtered feature vector sequences are weighted using the confidence parameters, giving high-confidence feature vectors a greater proportion in the final feature representation. This enhances the expressiveness of key information and reduces the impact of noisy data. This series of operations not only optimizes the feature selection process but also significantly improves the model's performance in various application scenarios, such as image recognition and object detection, achieving higher accuracy and robustness.

[0092] In a preferred embodiment of the present invention, the feature fusion layer is used to fuse the first image features and the second image features to obtain a rectangular frame including the target text. Specifically, the multi-layer feature map output by the backbone network is concatenated with the feature vector output by the encoder in the channel dimension to obtain the input tensor of the decoder; the decoder outputs the four coordinates and confidence score of the initial image text according to the input tensor.

[0093] The feature fusion layer plays a crucial role in this technical solution, effectively integrating the first image features extracted by the backbone network with the second image features generated by the encoder. Specifically, the multi-layer feature map output by the backbone network contains rich local details, such as text edges and stroke structures, while the feature vector output by the encoder is rich in semantic information after global modeling. By concatenating these complementary features from different sources along the channel dimension, a more expressive input tensor can be formed, serving as the input to the decoder and providing a solid foundation for accurate localization of subsequent text regions.

[0094] Based on the fused input tensor and the initialized query vector, the decoder progressively decodes the location information of the target text using a cross-attention mechanism. The final output includes the coordinates of the four vertices of the text region (i.e., four-point bounding boxes) and the corresponding confidence scores, used to describe the reliability of the detection results. This four-point coordinate format can more accurately describe text regions of arbitrary orientation and shape compared to the traditional two-point bounding boxes, and is especially suitable for complex scenarios such as rotated text, diagonal text, or polygonal text.

[0095] Through the aforementioned feature fusion and decoding strategies, the model not only enhances its ability to perceive target text but also strengthens its robustness in complex backgrounds, low resolution, or occlusion conditions. This significantly improves the accuracy and practicality of text localization, meeting the high-precision requirements of OCR systems in practical applications such as document recognition, invoice processing, and document analysis.

[0096] In a preferred embodiment of the present invention, the rotation angle corresponding to the initial text image is determined based on the endpoint order and endpoint coordinate values ​​of the multiple endpoints of the rectangular frame. Specifically, the multiple endpoint coordinates are the vertex coordinates of the rectangular frame, and the vertex coordinates are ordered. The text rotation angle is calculated based on the difference in the horizontal coordinates of the left and right vertices in the vertex coordinates. An affine transformation is performed on the initial text image to generate a forward image rotated by the corresponding angle.

[0097] Based on the coordinates of the four vertices of the rectangular frame output by the decoder, the rotation angle of the text image can be further determined to achieve positive correction of subsequent text content. The vertex coordinates have a clear order, typically arranged clockwise or counterclockwise, for example, top left, top right, bottom right, and bottom left corners. By analyzing the geometric relationships between these vertices, especially the relative horizontal position differences between the left and right vertices (e.g., the first and second vertices), the tilt angle of the text region can be calculated. For example, the rotation angle θ can be derived by combining the ratio of the difference in the horizontal to the difference in the vertical coordinates of two vertices with the arctangent function.

[0098] After obtaining the rotation angle, an affine transformation is performed on the initial text image, rotating it by the corresponding angle to transform the originally tilted text area into a horizontally aligned "positive image." This process includes constructing an affine transformation matrix and interpolating and resampling the image to ensure that the rotated image is clear and distortion-free. This rotation correction mechanism is not only applicable to single-line text but can also be extended to multi-line text or entire documents, significantly improving the robustness and accuracy of the OCR system in recognizing tilted or rotated text.

[0099] This technical solution makes full use of the geometric information provided by the four-point bounding box, and achieves automatic judgment and correction of text direction without relying on an additional classification model, thereby enhancing the intelligence of the entire text positioning and recognition process.

[0100] As a preferred embodiment of the present invention, the multi-directional text localization method further includes: performing rotation annotation processing on the image text to form pre-trained text samples, wherein the pre-trained text samples include the current coordinate values ​​and the true coordinate values ​​of the four endpoints; and training and determining the text localization model based on the current coordinate values ​​and the true coordinate values ​​using a custom loss function.

[0101] A key step in achieving multi-directional text localization is to rotate and annotate the text regions in the image, thereby forming a pre-training text sample set. This pre-training sample set includes images from ICdar2013 and ICdar2015, as well as images collected from daily work and online sources. These samples not only contain the current coordinates of the four endpoints of the text region (i.e., coordinates after some transformation) but also record its true coordinates (i.e., the original position coordinates before any transformation). In this way, precise positional and rotation angle information can be provided for each text instance, enabling the model to learn text features from different directions and tilt levels, and to rewrite the detection head and loss function according to the task characteristics.

[0102] To ensure the model can accurately locate text regions and their orientations from the input image, a custom loss function is used to quantify the difference between predicted and actual coordinates, guiding the optimization of the text localization model. This loss function is typically designed based on the error of endpoint coordinates, such as mean squared error (MSE) or mean absolute error (MAE), to measure the difference between the predicted and actual coordinates of the four endpoints. As training progresses, the model gradually learns to adjust its internal parameters, reducing this error and thus improving the accuracy of text localization in various orientations. This method effectively enhances the model's adaptability to varying text orientations, significantly improving its recognition performance in complex backgrounds.

[0103] In a preferred embodiment of the present invention, the text localization model is trained and determined based on the current coordinate values ​​and the true coordinate values ​​using a custom loss function. Specifically, the text localization model generates the current coordinate values ​​of the four endpoints of the pre-trained text sample and a predicted rectangular frame; based on the L1 distance between the true coordinate values ​​and the current coordinate values ​​of the pre-trained text sample, the sum of the absolute values ​​of the component differences of each endpoint in the XY direction is calculated; based on the overlapping area formed by the rectangular frame of the pre-trained text sample and the predicted rectangular frame, the ratio of the overlapping area of ​​the two quadrilaterals to the total area is calculated; and the text localization model is determined based on the sum of the absolute values ​​and the ratio.

[0104] The training process of the text localization model incorporates a custom composite loss function to more accurately optimize the model's ability to localize text regions in multiple directions. Specifically, during the model's forward propagation, corresponding four-endpoint coordinates and predicted rectangular frames are generated based on the input pre-trained text samples. These predictions are compared with the labeled ground truth coordinates and ground truth rectangular frames to calculate the loss and backpropagate to optimize the model parameters.

[0105] The loss function consists of two key parts: the first is an endpoint coordinate error term based on L1 distance, which calculates the sum of the absolute values ​​of the coordinate differences between the predicted four endpoints and the actual four endpoints in the X and Y directions, measuring the accuracy of the endpoint positions; the second is an IoU (Intersection over Union) term based on geometric overlap, which evaluates the overall text region coverage accuracy by comparing the ratio of the overlapping area between the predicted and actual rectangular frames to the total area. These two parts together constitute a comprehensive loss value, guiding the model training towards more precise endpoint localization and better overall structure matching.

[0106] By combining L1 distance and quadrilateral IoU in the loss function design, this approach not only improves the model's regression accuracy for text region endpoint coordinates but also enhances its ability to perceive the overall structure of irregularly shaped and rotated text. Compared to a single loss function, this method is better suited to multi-directional text detection tasks in complex scenarios, thereby significantly improving the robustness and practicality of the entire OCR system.

[0107] The present invention also provides a storage medium,

[0108] The storage medium stores a computer program, which, when executed, implements the steps of any of the Transformer-based multi-directional text positioning methods described above.

[0109] Therefore, it is possible to achieve any effect using the Transformer-based multi-directional text positioning method, which will not be elaborated here.

[0110] The present invention further provides a processing apparatus, comprising:

[0111] Memory, used to store computer programs;

[0112] A processor, configured to implement the steps of any of the Transformer-based multi-directional text positioning methods when executing the computer program.

[0113] Therefore, it is possible to achieve any effect using the Transformer-based multi-directional text positioning method, which will not be elaborated here.

[0114] For any parts not mentioned in this invention, existing technologies can be used or referenced.

[0115] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0116] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A multi-directional text localization method based on Transformer, characterized in that, include: The initial text image is preprocessed to obtain an intermediate image; The intermediate image is input into the text localization model, which is a Transformer text localization model including a backbone network, an encoder, a feature fusion layer and a decoder. The backbone network is used to obtain the first image features, the encoder is used to obtain the second image features, and the feature fusion layer is used to fuse the first image features and the second image features to obtain a rectangular frame containing the target text. Based on the endpoint order and endpoint coordinate values ​​corresponding to the multiple endpoints of the rectangular frame, the rotation angle corresponding to the initial text image is determined. The rectangular frame is rotated based on the stated rotation angle to obtain a forward-facing text image; The image text is rotated and annotated to form a pre-trained text sample, which includes the current coordinate values ​​and the true coordinate values ​​of the four endpoints. The text localization model is determined by training based on the current coordinate values ​​and the true coordinate values ​​using a custom loss function. The text localization model generates the current coordinates of the four endpoints of the pre-trained text sample and the predicted rectangular frame. Based on the L1 distance between the true coordinates and the current coordinates of the pre-trained text samples, calculate the sum of the absolute values ​​of the component differences in the XY direction for each endpoint; Based on the overlapping area formed by the rectangular frame of the pre-trained text sample and the predicted rectangular frame, calculate the ratio of the overlapping area of ​​the two quadrilaterals to the total area. The text positioning model is determined based on the sum of the absolute values ​​and the ratio.

2. The multi-directional text localization method based on Transformer according to claim 1, characterized in that, The initial text image is preprocessed to obtain the intermediate image, specifically as follows: Calculate the pixel mean and standard deviation of the three channels of the initial text image; The initial text image is standardized based on the pixel mean and the standard deviation to form the intermediate image.

3. The multi-directional text localization method based on Transformer according to claim 1, characterized in that, The first image features are obtained through the backbone network, and the second image features are obtained through the encoder, specifically as follows: The backbone network extracts multi-layer feature maps from the intermediate image, and the multi-layer feature maps are obtained with different size parameters; By downsampling, the multi-layer feature map is scaled up to match the output size of the Transformer encoder, resulting in multiple feature vector sequences.

4. The multi-directional text localization method based on Transformer according to claim 3, characterized in that, The multiple feature vector sequences are specifically as follows: A one-to-one corresponding query vector is obtained based on the feature vector sequence; The confidence parameter is obtained using the Hungarian algorithm based on the query vector. Multiple feature vector sequences are sorted according to the confidence parameter and then filtered based on the multiple feature vector sequences; The filtered feature vector sequence is weighted using the confidence parameter.

5. The multi-directional text localization method based on Transformer according to claim 1, characterized in that, The feature fusion layer is used to fuse the first image features and the second image features to obtain a rectangular frame that includes the target text, specifically: The multi-layer feature maps output by the backbone network are concatenated with the feature vectors output by the encoder along the channel dimension to obtain the input tensor of the decoder. The decoder outputs the four coordinates and confidence score of the initial image text based on the input tensor.

6. The multi-directional text localization method based on Transformer according to claim 1, characterized in that, Based on the endpoint order and coordinate values ​​of the multiple endpoints of the rectangular frame, the rotation angle corresponding to the initial text image is determined, specifically as follows: The multiple endpoint coordinates are the vertex coordinates of the rectangular frame. The vertex coordinates are ordered, and the text rotation angle is calculated based on the difference in the horizontal coordinates of the left and right vertices in the vertex coordinates. An affine transformation is performed on the initial text image to generate a forward image rotated by the corresponding angle.

7. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed, implements the steps of the Transformer-based multi-directional text positioning method as described in any one of claims 1 to 6.

8. A processing apparatus, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the Transformer-based multi-directional text positioning method as described in any one of claims 1 to 6 when executing the computer program.

Images