Digital signage image text recognition method based on Transformer network model

Abstract

This paper discloses a method for text recognition in digital signage images based on a Transformer network model. Based on the Transformer network model and scene text recognition (STR) technology, this method uses relative position encoding to obtain the position encoding of the digital signage image and connects the feature representations of different subspaces to the encoder, thereby accurately identifying text in the digital signage image. This method uses relative position encoding to encode image feature maps in both horizontal and vertical dimensions, accurately capturing the position information between two-dimensional image blocks, better modeling the text information in the image, and more accurately identifying text in digital signage images.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and relates to image text recognition technology, in particular to a digital signage image text recognition method based on a Transformer network model. Background Art

[0002] Text recognition has many application scenarios, including document recognition, road sign recognition, license plate recognition, industrial serial number recognition, and more. Based on the actual scenario, text recognition tasks can be divided into two categories: regular text recognition and irregular text recognition. Regular text recognition primarily refers to printed fonts and scanned text, assuming that the text is roughly horizontal. Irregular text recognition often occurs in natural scenes, where text is often not horizontal and suffers from problems such as curvature, occlusion, and blur. Recognizing text in natural scenes is called scene text recognition (STR) and is generally considered a special form of optical character recognition (OCR), namely, camera-based OCR. Although OCR technology is relatively mature, STR remains challenging because the text in natural scene images has a wide variety of presentation formats and the backgrounds of natural scene images are extremely diverse, all of which pose additional challenges to recognition.

[0003] Text plays a vital role in human life, and automatic reading of text in natural scenes holds great practical value. Therefore, scene text recognition has become a key application in computer vision and pattern recognition. Currently, numerous image text recognition algorithms exist, primarily including CTC-based methods, correction-based methods, attention-based methods, and segmentation-based methods. CTC-based methods address sequence alignment by adding special "blank symbols," eliminating the need for manual alignment of label and input sequences and simplifying the modeling process for text recognition. Correction-based methods utilize visual transformation modules to convert irregular text into regular text as much as possible, then use conventional methods for recognition. Attention-based methods focus on the correlation between parts of a sequence. Segmentation-based methods locate each character in the input image, identify each character using a character classifier, and then combine all characters into a string sequence to obtain the final recognition result. However, most existing recognition algorithms are sensitive to environmental interference and struggle to handle the complexities of the real world. Consequently, they perform poorly for scene text recognition and suffer from low recognition accuracy. Summary of the Invention

[0004] To address the shortcomings of the aforementioned existing technologies, this paper proposes a method for digital signage image text recognition based on a Transformer network model. This method, based on the Transformer network model and scene text recognition (STR) technology, uses relative position encoding to obtain the position encoding of the digital signage image. The feature representations of different subspaces are then connected to the encoder, thereby accurately identifying text in digital signage images and improving recognition accuracy.

[0005] The technical solution of the present invention is: a digital signage image text recognition method based on a Transformer network model, comprising the following steps:

[0006] 1) Use the Connectionist Text Proposal Network (CTPN) to detect text in digital signage images and obtain candidate boxes for text regions in digital signage images;

[0007] 2) performing noise reduction processing on the digital signage image based on the candidate box of the text area obtained in step 1), and completely capturing the text information of the digital signage image as input data;

[0008] In specific implementation, the digital signage image is cut and denoised based on the candidate coordinate file for the text area. During the cutting (cropping) process, if the candidate box for the text area is a tilted rectangle, only the chord diagram of the candidate box is cut out to obtain the cut digital signage image; the chord diagram refers to the rectangular frame circumscribing the tilted rectangle. The designed algorithm takes the minimum of the upper left and lower left horizontal coordinates as the left parameter of the crop function, and the minimum of the upper left and upper right vertical coordinates as the upper parameter of the crop function. The same applies to right and lower. This ensures that the text information is fully captured and also solves the range problem of the candidate box coordinate parameters, ensuring that no parameters outside the range are passed in.

[0009] 3) Using Linear Projection to obtain the image patch vector Patch Embedding of the digital signage image obtained in step 2);

[0010] For each input 2D image x∈R H×W×C Reshape into a series of two-dimensional image patches, denoted as The image size is H×W, the number of channels is C, the patch size is P×P, the patch sequence length is N, and the patches are projected to the dimension D=P through linear transformation. 2 In the space of , the original two-dimensional image x of size H×W×C is flattened into N pieces of size P 2A one-dimensional vector of C, each patch is converted into an image block vector Patch Embedding with a dimension of D.

[0011] 4) Calculate the positional encoding between each two-dimensional image block of the digital signage image using the relative position encoding method; add it to the image block vector obtained in step 3) as the input of the Transformer encoder;

[0012] The Transformer model includes an encoder and a decoder. To capture the positional relationship between the patches of the two-dimensional image, the relative position encoding method is used to calculate the position encoding (Position) between the two-dimensional image patches of the digital signage image (indicating the positional relationship between the patches). This position encoding is then added to the PatchEmbedding obtained in step 3) as the input to the Transformer model's encoder.

[0013] The positional relationship between two-dimensional image blocks is captured by two-dimensional relative position encoding. The relative positions between the input two-dimensional image blocks are encoded into vectors and embedded into the self-attention module, which is expressed as:

[0014]

[0015]

[0016]

[0017] Where x=(x1,…,x n ) is the input sequence, Embedding of digital signage image blocks, z=(z1,...,z n ) is the output sequence, d x =d z , W V 、W Q 、W K is the parameter matrix, is the relative position encoding vector between input patches, α ij is the weight coefficient between image block i and image block j, e ij is the degree of attention of image block i to image block j after normalizing the exponential function softmax, b ij It is a two-dimensional relative position code, where the subscripts i and j correspond to two two-dimensional coordinates on the digital signage image.

[0018] Encode the digital signage image in two dimensions, horizontally and vertically. Calculate the horizontal and vertical codes separately, then summarize them and encode the relative positions between the input patches. The two-dimensional relative position codes are expressed as:

[0019] b ij =(x i W Q )r ij T

[0020]

[0021]

[0022]

[0023] in, is the relative position of image block i and image block j in the horizontal direction, is the relative position of image block i and image block j in the vertical direction, is the position code in the horizontal direction, is the position code in the vertical direction, r ij represents the relative position weight between image block i and image block j, and Both are learnable scalars.

[0024] Combine the Patch Embedding obtained by linear projection with the position encoding information obtained by relative position encoding and ) is added together to form the vector that is used as the input of the Transformer encoder.

[0025] 5) Using the Transformer encoder to extract the feature vectors of the text in the digital signage image from the input data, using the multi-head self-attention mechanism in the encoder to determine the relationship between the text feature vectors in the image, and using the feedforward neural network in the encoder to perform a nonlinear transformation on the output of the multi-head self-attention layer to output the recognized text; including:

[0026] The input image of each Transformer encoder is normalized using the layer normalization (LN) method, and the text-related information from different positions (different subspaces) is processed using multi-head self-attention (MSA) to determine the relationship between the text feature vectors. The output of the layer normalization LN is connected with the output of the MSA by residual connection, and the output result is normalized again using LN. A nonlinear transformation is performed using a feedforward neural network to output the recognized text.

[0027] Through the above steps, digital signage image text recognition based on the Transformer network model is realized.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] The present invention provides a digital signage image text recognition method based on the Transformer network model. Based on the excellent Transformer network model framework structure, relative position encoding (Relative Position Encoding) is used to optimize the original absolute position encoding (Position Encoding), and image feature maps are encoded in two dimensions, horizontally and vertically. This can more accurately capture the position information between two-dimensional patches, better model the text information in the image, and more accurately recognize the text in the digital signage image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of the method of the present invention.

[0031] Figure 2 This is the structural block diagram of the Transformer network model encoder.

[0032] Figure 3 This is the overall network model architecture diagram of the digital signage image text recognition method of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below by way of examples in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0034] The present invention provides a digital signage image text recognition method based on a Transformer network model. Figure 1 Shown is the process of the method of the present invention, comprising the following steps:

[0035] 1) Perform text detection on digital signage image data through CTPN network;

[0036] 2) Denoising the digital signage image data based on the generated candidate frames;

[0037] 3) Obtain the image patch embedding through linear projection;

[0038] 4) Capture the position information between patches through relative position encoding;

[0039] 5) Add the patch embedding and position encoding as the input of the encoder;

[0040] 6) Normalize each input through Layer Norm;

[0041] 7) Determine the relationship between feature vectors through a multi-head self-attention mechanism;

[0042] 8) Perform a nonlinear transformation on the output of the multi-head self-attention layer through a feedforward neural network and output the recognized text;

[0043] The present invention specifically implements outdoor commercial digital signage image data within the Sixth Ring Road area of ​​a certain city and adopts a digital signage image text recognition method based on a Transformer network model to perform digital signage image text recognition. The specific implementation steps are as follows:

[0044] 1) Text detection is performed on digital signage image data using a CTPN network. First, a vertical text box regression mechanism is used to detect small-scale text candidate boxes. Then, a recurrent neural network (RNN) is used to connect the detected small-scale text boxes to obtain text lines. A vertical text box has two ordinates: the center height (y-coordinate) and the height of the rectangular box, expressed as:

[0045] v h =log(h / h a )

[0046]

[0047] Where v = {v c ,v h} are the predicted coordinates, are the real coordinates, and h a They are the y coordinate center and height of the text box, c y and h are the predicted y-coordinate center and height in the input image, respectively. and h * are the true y-coordinate center and height of the input image respectively.

[0048] The CTPN network uses a convolutional neural network (CNN) for feature extraction. A deep convolutional neural network (Visual Geometry Group 16, VGG16) performs convolution and pooling operations on the image, converting the input image into a series of feature maps. These feature maps are then used to detect text lines within the image. The CTPN network takes the VGG16 conv5 feature map as input and models and predicts candidate text lines using an RNN-based sequence model. The feature vector of each pixel and its surrounding 3×3 region is considered a time step. The sequence of these time step feature vectors is fed into a bidirectional long short-term memory (BiLSTM) network, which generates a sequence of text line candidate regions, along with the text line's position and shape information, for each time step. An anchor-based detection algorithm is then used to classify and regress the text line candidate regions, resulting in the final text line detection results. The CTPN network outputs a series of text region candidate boxes, each of which represents a rectangular region that may contain text. Each candidate box is defined by eight coordinate values: the x and y coordinates of its four corners. Each candidate box also has a confidence score, which indicates the probability that text exists within the box. CTPN sorts the candidate boxes based on their confidence scores and selects the one with the highest score as the final output.

[0049] 2) Denoise the digital signage image data based on the generated candidate frames, and crop the digital signage image according to the candidate frame coordinates. During the cropping process, due to the characteristics of the PIL library function, only completely parallel frames are cropped, which will cause information distortion in the image. Therefore, for tilted candidate frames, their chord diagrams need to be cropped. Therefore, the algorithm is designed to take the minimum value of the upper left and lower left horizontal coordinates as the left parameter of the crop function, and take the minimum value of the upper left and upper right vertical coordinates as the upper parameter of the crop function. The same applies to right and lower. After this processing, it can be guaranteed that the text information is completely captured.

[0050] 3) Obtain the image’s Patch Embedding through linear projection. For each input image x∈R H×W×C is reshaped into a series of two-dimensional image patches, The image size is H×W, the number of channels is C, the patch size is P×P, the patch sequence length is N, and the patches are projected to the dimension D=P through linear transformation. 2 In the space of , the original two-dimensional image of size H×W×C is flattened into N two-dimensional images of size P 2A one-dimensional vector of C, each patch is converted to a PatchEmbedding of size D.

[0051] 4) The position information between patches is captured through relative position encoding. The relative position between input patches is encoded into a vector and embedded into the self-attention module, which is expressed as:

[0052]

[0053] Where x=(x1,…,x n ) is the input sequence, z i is the output sequence, W V is the parameter matrix, is the relative position encoding vector between input patches, α ij is the weight coefficient between image block i and image block j, expressed as:

[0054]

[0055] Among them, e ij It is the degree of attention of image block i to image block j after the normalized exponential function softmax, expressed as:

[0056]

[0057] Among them, W Q 、W K is the parameter matrix, is the relative position encoding vector between input patches, b ij It is a two-dimensional relative position encoding. It encodes the image in two dimensions, horizontally and vertically. It calculates the encoding in the horizontal and vertical directions respectively, and then summarizes them to encode the relative position between the input patches, which is expressed as:

[0058] b ij =(x i W Q )r ij T

[0059]

[0060]

[0061]

[0062] in, is the relative position of image block i and image block j in the horizontal direction, is the relative position of image block i and image block j in the vertical direction, is the position code in the horizontal direction, is the position code in the vertical direction, r ij represents the relative position weight between image block i and image block j, and Both are learnable scalars.

[0063] 5) Add the Patch Embedding obtained by linear projection and the Position obtained by relative position encoding as the input of the Transformer encoder, expressed as:

[0064]

[0065] in, Position code E pos ∈R (N+1)×D .

[0066] 6) Normalize the input of each Transformer encoder through Layer Norm. The specific calculation is as follows:

[0067]

[0068] Among them, x is the input, y is the output, ε is a small constant to prevent the denominator from being zero, and γ and β are trainable parameters.

[0069] 7) The relationship between feature vectors is determined through the multi-head self-attention mechanism. The multi-head self-attention mechanism makes each output related to all inputs and can effectively utilize context information. For each patch, three vectors q, k, and v are generated, where q = W q ×a,k=W k ×a,v=W v ×a, where W q 、W k 、W v is a three-parameter matrix, a is the input two-dimensional image patch, the q vector is analogous to the query, the k vector is analogous to the index, and the v vector is analogous to the answer. Given a task-related query q vector, the attention score is calculated by calculating the attention distribution with k and appending it to v. Multiplying all scores by v of each patch will obtain the vector of the extracted information. Adding these vectors, the final output of the multi-head self-attention mechanism is:

[0070]

[0071] Among them, Attention(Q, K, V) is the output of the multi-head self-attention mechanism; is the attention score matrix, multiplied by the factor After softmax, each row of the attention score matrix represents a two-dimensional image block, and each column represents the weight value (i.e., α value) of the two-dimensional image block and the two-dimensional image blocks at other positions.

[0072] 8) The output of Layer Norm is connected to the output of Multi-Head Self-Attention through a residual connection. The output result is normalized again using Layer Norm. The output result of the multi-head self-attention layer is nonlinearly transformed through a feedforward neural network to output the recognized text.

[0073] The present invention uses the outdoor commercial digital signage images in the Sixth Ring Road area of ​​a city as experimental data, and inputs the processed data into Figure 3 In the image text recognition model shown, text recognition results for digital signage images are obtained and then compared with other methods. Accuracy is used to evaluate the recognition results and verify the recognition accuracy. The specific calculation is as follows:

[0074]

[0075] Where N represents the total number of test samples, y i represents the true label of sample i, z i Denotes the predicted label of sample i, and δ denotes the Dirac function. The higher the accuracy, the better the recognition effect of the model.

[0076] The experimental results are shown in Table 1. The bold indicates the highest recognition accuracy in the same column.

[0077] Table 1 Comparison of experimental results

[0078]

[0079] As can be seen from Table 1, the Transformer-based digital signage image text recognition model proposed in the present invention achieves better recognition accuracy in comparison with the CTC-based CRNN and Rosetta models, the correction-based RARE model, and the attention-based R2AM model.

[0080] Finally, it should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments; the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A digital signage image text recognition method based on a Transformer network model, characterized by: Based on the Transformer network model and scene text recognition (STR) technology, relative position encoding is used to obtain the position encoding of the digital signage image. The feature expressions of different subspaces are connected to the encoder to accurately recognize the text in the digital signage image. The process includes the following steps: 1) Use the text detection network CTPN to detect text in the digital signage image and obtain candidate boxes of the text area in the digital signage image; 2) performing noise reduction processing on the digital signage image according to the candidate frame obtained in step 1) as input digital signage image data; 3) Each input digital signage image x∈R H×W×C It consists of a series of two-dimensional image blocks; the image size is H×W and the number of channels is C; Obtain image patch vectors of digital signage images using linear projection; 4) Calculate the position codes between the two-dimensional image blocks of the digital signage image using a relative position coding method; The vector obtained by adding it to the image block vector obtained in step 3) is used as the input of the Transformer model encoder; The positional relationship between two-dimensional image blocks is captured by two-dimensional relative position encoding. The relative positions between the input two-dimensional image blocks are encoded into vectors and embedded into the self-attention module, which is expressed as: Where x=(x1,…,x n ) is the input sequence, Embedding of digital signage image blocks, z=(z1,...,z n ) is the output sequence, d x =d z , W V 、W Q 、W K is the parameter matrix, is the relative position encoding vector between the input two-dimensional image blocks, α ij is the weight coefficient between image block i and image block j, e ij is the degree of attention of image block i to image block j after normalizing the exponential function softmax, b ij It is a two-dimensional relative position code, where the subscripts i and j correspond to two two-dimensional coordinates on the digital signage image; The digital signage image is encoded in two dimensions, horizontally and vertically. The codes in the horizontal and vertical directions are calculated separately and then summarized. The relative positions between the input two-dimensional image blocks are encoded. The two-dimensional relative position encoding is expressed as: b ij =(x i W Q )r ij T in, is the relative position of image block i and image block j in the horizontal direction, is the relative position of image block i and image block j in the vertical direction, is the position code in the horizontal direction, is the position code in the vertical direction, r ij represents the relative position weight between image block i and image block j, and Both are learnable scalars; The obtained image block vector is added to the position encoding information obtained by relative position encoding as the input of the Transformer model encoder; 5) Using the Transformer model encoder, extract the feature vectors of the text in the digital signage image from the input data. The encoder's multi-head self-attention mechanism determines the relationship between the text feature vectors. The encoder's feedforward neural network performs a nonlinear transformation on the output of the multi-head self-attention layer and outputs the recognized text. Specifically, the input of each Transformer model encoder is normalized using the layer normalization method, and multi-head self-attention is used to process information from different subspaces at different positions to determine the relationship between feature vectors. The output of the layer normalization is connected to the output of the multi-head self-attention with a residual connection. The output result is normalized again and a nonlinear transformation is performed using a feedforward neural network to output the recognized text. Through the above steps, digital signage image text recognition based on the Transformer network model is realized.

2. The digital signage image text recognition method based on the Transformer network model as claimed in claim 1, characterized in that: In step 1), text detection is performed on the digital signage image data using the CTPN network, including: First, a vertical text box regression mechanism is used to detect small-scale text candidate boxes; Then, a recurrent neural network (RNN) is used to connect the detected small-scale text boxes to obtain text lines; The vertical coordinates of the vertical text box are the height of the center position, i.e. the y coordinate, and the height of the rectangular box; The CTPN network uses a convolutional neural network (CNN) for feature extraction and a deep convolutional neural network (VGG16) to perform convolution and pooling operations on the image to convert the input image into a series of feature maps. The feature maps are then used to detect text lines in the image. The CTPN network takes the conv5 feature map of VGG16 as input, models and predicts candidate text lines through an RNN-based sequence model, takes the feature vector of each pixel and its surrounding 3×3 area as a time step, and inputs the feature vector sequence of the time step into the bidirectional long short-term memory network BiLSTM to obtain the binary classification probability of the text line and the position and shape information of the text line at each time step, thereby generating a sequence of text line candidate regions; then, an anchor box-based detection algorithm is used to classify and regress the text line candidate regions to obtain the final text line detection results; the CTPN network outputs a series of candidate boxes for the text region; each candidate box is defined by eight coordinate values, including the x and y coordinates of the four corners, the confidence score of each candidate box, and the probability that text may exist in the box; CTPN sorts the candidate boxes according to their confidence scores and selects the candidate box with the highest score as the final output.

3. The digital signage image text recognition method based on the Transformer network model as claimed in claim 2, characterized in that: The vertical coordinates of the vertical text box are the height of the center position and the height of the rectangular box, respectively expressed as: v h =log(h / h a ) Where v = {v c ,v h } are the predicted coordinates, are the real coordinates, and h a They are the y coordinate center and height of the text box, c y and h are the predicted y-coordinate center and height in the input image, respectively. and h * are the true y-coordinate center and height of the input image respectively.

4. The digital signage image text recognition method based on the Transformer network model as claimed in claim 1, characterized in that: In step 2), the digital signage image is subjected to noise reduction processing according to the candidate frame obtained in step 1), specifically, the digital signage image is cut according to the candidate frame coordinate file for noise reduction; during the cropping process, for tilted rectangles, only the chord diagram is cropped; the method is: take the minimum value of the upper left and lower left horizontal coordinates as the left parameter of the crop function, take the minimum value of the upper left and upper right vertical coordinates as the upper parameter of the crop function, and use the same method to process the right and lower parameters, so that the text information is completely captured and no parameters outside the interval are passed in.

5. The digital signage image text recognition method based on the Transformer network model as claimed in claim 1, characterized in that: Step 3) obtaining the image block vector of the digital signage image using linear projection; specifically comprising: For each input image x∈R H×W×C Reshape into a series of two-dimensional image blocks, The image size is H×W, the number of channels is C, the size of the two-dimensional image block is P×P, the length of the two-dimensional image block sequence is N, and the two-dimensional image block is projected to the dimension D=P by linear transformation 2 In the space of , the two-dimensional image of size H×W×C is flattened into N pieces of size P 2 A one-dimensional vector of C, each two-dimensional image block is converted to an image block vector of dimension D.

6. The digital signage image text recognition method based on the Transformer network model as claimed in claim 1, characterized in that: In step 4), the obtained image block vector is added to the position encoding information obtained by relative position encoding as the input of the Transformer model encoder; it is expressed as: in, Position code E pos ∈R (N+1)×D .

7. The digital signage image text recognition method based on the Transformer network model as claimed in claim 6, characterized in that: Step 5) Specifically, the input of each Transformer encoder is normalized through Layer Norm. The specific calculation is expressed as: Among them, x is the input, y is the output, ε is a small constant to prevent the denominator from being zero, and γ and β are trainable parameters.

8. The digital signage image text recognition method based on the Transformer network model as claimed in claim 7, characterized in that: Step 5) Determine the relationship between feature vectors through a multi-head self-attention mechanism; specifically, For each two-dimensional image block, three vectors q, k, v are generated, q = W q ×a,k=W k ×a,v=W v ×a, where W q 、W k 、W v are three parameter matrices, a is the input two-dimensional image block, q is the query vector, k is the index vector, and v is the answer vector; Given a task-related query vector q, the attention score is calculated by calculating the attention distribution with k and appending it to v. All scores are multiplied by v of each two-dimensional image patch to obtain the vector of the extracted information. The vectors are added together to obtain the result of the final two-dimensional image patch after the self-attention mechanism model, which is expressed as: in, is the attention score matrix; each row of the attention score matrix represents a two-dimensional image block, and each column represents the weight value of the two-dimensional image block and the two-dimensional image blocks at other positions.

Images