Irregular text recognition method based on deep learning and Raspberry Pi

Abstract

The invention discloses an irregular text recognition method based on deep learning and Raspberry Pi, and the method comprises the steps: firstly carrying out the data preprocessing, inputting an image and a corresponding text region label, and carrying out the standardization and data enhancement of the image; feature extraction is carried out through a backbone network, and an improved self-attention mechanism is added in front of an SPPF module, so that the quality of features is improved; performing text region proposal, inputting the extracted features into a text region proposal module, and outputting a predicted text region; identifying text content from the proposed text region using a sequence model; calculating the total loss of the text region detection loss and the text recognition loss; in the training process, two kinds of precision FP16 and FP32 are used for calculation; and finally, deploying an algorithm at a Raspberry Pi terminal to solve a real-time scene irregular text recognition task. And the performance of the whole text detection and recognition system can be remarkably improved, so that the text detection and recognition system has higher accuracy and robustness in various complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of text detection and recognition, and in particular to an irregular text recognition method based on deep learning and Raspberry Pi. Background Art

[0002] Deep learning-based text detection has made significant progress in the past few years, mainly due to the improvement of computing power, the emergence of large-scale annotated datasets, and the continuous innovation of deep learning model architecture.

[0003] Mainstream OCR algorithms include two-stage algorithms and end-to-end algorithms. Two-stage OCR algorithms generally consist of two parts: text detection and text recognition algorithms. The text detection algorithm locates the text area in the input image, and the recognition algorithm then recognizes the text content in the image.

[0004] The end-to-end OCR algorithm uses one model to complete both text detection and text recognition. The basic idea is to share the same Backbone network and design different detection modules and recognition modules, so that text detection and text recognition can be trained at the same time. The end-to-end algorithm simplifies the process, so the model is smaller and the processing speed is faster.

[0005] TongHe et al. proposed an end-to-end regular text recognition model TextSpotter, in which the detection branch uses the PVANet model as the backbone and the recognition branch uses the RNN model. This model integrates detection and recognition into one network for training, which improves the detection efficiency. The disadvantage is that it cannot recognize irregular text.

[0006] PengyuanLyu, MinghuiLiao and others proposed an end-to-end trainable scene text recognition model MaskTextSpotter. Compared with some algorithms that can only locate horizontal text or multi-directional text, it can detect and recognize text of any shape (horizontal, multi-directional, curved). The disadvantage is that it requires high accuracy in training data annotation and is not effective in detecting long text.

[0007] Based on this, an irregular text recognition method based on deep learning and Raspberry Pi is proposed, which is suitable for irregular text recognition tasks and can achieve the effect of real-time scene text recognition by being deployed to the Raspberry Pi terminal. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention discloses an irregular text recognition method based on deep learning and Raspberry Pi to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solution: an irregular text recognition method based on deep learning and Raspberry Pi, comprising the following steps:

[0010] S1. First, data preprocessing is performed, input images and corresponding text area annotations are input, and the images are standardized and data enhanced;

[0011] S2. Feature extraction is performed through the backbone network, and an improved SKSAttention self-attention mechanism is added before the SPPF module to improve the quality of features through the introduced SKSAttention self-attention mechanism;

[0012] S3, perform text region proposal, input the extracted features into the Transformer-based text region proposal module, and output the predicted text region;

[0013] S4, using sequence models to identify text content from proposed text regions;

[0014] 1) First, the target text image frames are arranged in time order to form a time series data sequence. Then, TCN is used to process the time series data and the mask sequence (X m ) Input the temporal convolutional network TCN, TCN uses a series of one-dimensional convolution operations to extract temporal features, where the output of TCN is a sequence containing temporal features, indicating the feature representation of each time step;

[0015] 2) Use a bidirectional long short-term memory network Bi-LSTM to process the time series features extracted by TCN. The output feature sequence of TCN is input into Bi-LSTM. Bi-LSTM processes the sequence data in both forward and backward directions to capture the bidirectional time dependency. The output of Bi-LSTM is the hidden state of each time step, which contains the contextual understanding of the sequence.

[0016] 3) Introduce a multi-head self-attention mechanism to extract information from different subspaces and capture dependencies globally, thereby improving the recognition accuracy and robustness of the image frame mask processing model based on temporal convolutional networks and bidirectional long short-term memory networks;

[0017] S5, calculating the total loss of text region detection loss and text recognition loss;

[0018] S6. Use FP16 and FP32 precision for calculation during training to speed up the training process and reduce video memory usage;

[0019] S7. Finally, the algorithm is deployed on the Raspberry Pi terminal to solve the irregular text recognition task in real-time scenarios.

[0020] Preferably, in step S2, the SKSAttention attention mechanism improves the performance of the model in processing multi-scale features by adaptively selecting a suitable convolution kernel size, and introduces a Swish activation function. The SKSAttention attention mechanism process steps include the following:

[0021] 1) Perform multi-scale convolution: The input feature map (X) is convolved with two convolution kernels of different scales to generate two feature maps of different scales (u1) and (u2); convolution kernels of (3×3) and (5×5) are used;

[0022] u1=conv 3×3 (x);u2=conv 5×5 (x);

[0023] 2) Feature fusion: Add two feature maps of different scales to obtain a fused feature map (U y ): U y =u1+u2;

[0024] 3) Perform global average pooling: The fusion feature map (U y ) to perform spatial compression and obtain the global eigenvector z: z = global_avg_pool (U y )

[0025] 4) Perform channel attention mechanism: perform a full connection operation on the global feature vector (z) to generate a selection vector (s); use two fully connected layers and a Swish activation function:

[0026] s intermediate =Swish(fc1(z))

[0027] s=fc2(sintermediate)

[0028] 5) Perform weighted summation: Normalize the selection vector (s) through the softmax function to obtain the selection weight (α): α = softmax (s); According to the selection weight (α), perform weighted summation on the feature maps (u1) and (u2) of two different scales to obtain the final output feature map (U x ): U x =α1·u1+α2·u2

[0029] Output the fused feature map U from the model backbone network x Enter the FPN and PAN feature pyramid modules for feature fusion operations and output three feature maps of different sizes U x1 , U x2 , U x3 .

[0030] Preferably, by introducing an improved FeatureFusion feature integration module, the three feature maps of different sizes output by the Neck part are input into the improved FeatureFusion feature integration module, a feature enhancement module and an adaptive feature fusion module are introduced, and the feature maps of different scales generated by FPN and PAN are effectively fused into a feature map U. The specific steps include:

[0031] 1) First, the three feature maps U of different sizes output by the Neck part x1 , U x2 , U x3 Perform feature alignment: Assuming the target size is H×W, use upsampling and downsampling operations to adjust all feature maps to the same size:

[0032] Upsampling operation: Taking the target size as the reference, for feature maps whose size is smaller than the target size, use bilinear interpolation or deconvolution operation to upsample them to the target size:

[0033] Downsampling operation: Taking the target size as the benchmark, for feature maps whose size is larger than the target size, use average pooling or convolution operation to downsample them to the target size:

[0034] 2) Introduce a feature enhancement module to extract contextual information; use multi-layer convolution and dilated convolution to capture different receptive fields:

[0035]

[0036] 3) Introduce an adaptive feature fusion module, use the attention mechanism to learn each feature map, and fuse multi-scale features, including:

[0037] Adaptive weight generation:

[0038] Weighted feature fusion:

[0039] 4) Perform feature optimization:

[0040] Add a normal convolution operation Conv(x)=W*x and ReLU activation function to the fused feature map for further feature optimization:

[0041] u optimized =Conv(ReLU(Conv(u fused )))

[0042] Feature map U optimizedis the final feature map U, which is input into the text region proposal module.

[0043] Preferably, in step S3, based on the segmentation proposal network SPN and the hardRoImasking method, it is proposed to add an IoU loss function to improve the regression result when generating the classification and regression output of the candidate region through the segmentation proposal network SPN, so that the generated candidate region is more accurate;

[0044] Among them, the IoU calculation formula represents the intersection-over-union ratio between two regions, the two regions include the predicted box (A) and the real box (B); the (IoU) calculation formula is:

[0045]

[0046] Among them: Area(A∩B) represents the intersection area of ​​the predicted box and the real box;

[0047] Area(A∪B) represents the union area of ​​the predicted box and the true box;

[0048] IoU loss is defined as 1 minus the IoU value: IoU Loss = 1-IoU(A,b)

[0049] The specific steps include:

[0050] 1) Candidate region generation: First, the candidate region is generated using the fused feature map (U): Among them, (R i ) represents the generated candidate region;

[0051] 2) Feature extraction: For each candidate region (R i ), extract the corresponding features from the feature map (U); use RoIAlign operation;

[0052] 3) Classification and regression: The features of each candidate region Through the fully connected layer, classification and bounding box regression are performed respectively; classification output: Regression output: Among them, (FCcls) and (FCreg) represent the fully connected layers for classification and regression;

[0053] 4) Add IoU loss: The IoU loss function is defined as:

[0054] in, is the candidate region (R i )The corresponding ground-truth bounding box;

[0055] The total loss function is expressed as a weighted sum of classification loss and regression loss:

[0056]

[0057] Among them, (L cls ) is the classification loss, (L reg ) is the standard regression SmoothL1 loss, (L IoU ) is the IoU loss, (λ) and (μ) are weight coefficients;

[0058] 5) Generate the final TestRoI feature map: Use the trained network to generate each candidate region (R i ) performs classification and regression to obtain the final bounding box and extracts the corresponding features from the feature map (U);

[0059] in, is the bounding box predicted in the test phase; the final TestRoI feature map It is expressed as:

[0060] Through the above steps, the fused feature map (U) is passed through the segmentation proposal network (SPN) to generate the classification and regression output of the candidate region, and the IoU loss is added to improve the regression result, and finally the TestRoI feature map is obtained.

[0061] Preferably, in step S3, the TestRoI feature map output by the text region proposal network is applied to each RoI through the introduced MaskBranch module, and the segmentation mask is predicted in a pixel-to-pixel manner to obtain the target text image frame mask, and the binary mask of the target text is output. The specific steps include:

[0062] 1) RoI feature map input:

[0063] Generate TestRoI feature map using text region proposal network As input;

[0064] 2) The MaskBranch structure module includes three convolutional layers and one upsampling layer:

[0065] Convolutional layer 1 performs convolution operation on the input feature map: Convolutional layer 2 further extracts features: Upsampling layer, upsampling the feature map to the input RoI size:

[0066]

[0067] Convolutional layer 3 convolves the upsampled feature map to obtain the binary mask prediction:

[0068]

[0069] Among them, (Conv2D) represents a two-dimensional convolution operation, (W1, W2, W3) is the convolution kernel, and (b1, b2, b3) is the bias;

[0070] 3) Loss function: Use the binary cross entropy loss function to optimize the output of MaskBranch, assuming that the true binary mask is Then the loss function is:

[0071]

[0072] Through the above steps, MaskBranch converts each TestRoI feature map Convert to binary mask The final output is a set of binary masks of the target text image frames.

[0073] Preferably, in step S4, the specific implementation steps are as follows:

[0074] 1) TCN feature extraction: Use the temporal convolutional network to extract the local features of the time series and obtain the feature matrix (F TCN ∈R T×dTN ); Formula: F TCN =TCN(X m ); where (X m ) is the input time series data, TCN is the temporal convolutional network;

[0075] 2) Bi-LSTM feature extraction: use bidirectional LSTM to extract the global features of the time series and obtain the hidden state matrix Formula: H Bi-LSTM =Bi-LSTM(X m ); where (X m ) is the input time series data, Bi-LSTM is a bidirectional long short-term memory network;

[0076] 3) Multi-head self-attention mechanism

[0077] 3.1) Linear transformation: Perform linear transformation on the features extracted by TCN and Bi-LSTM to obtain the query matrix Q, key matrix K and value matrix V:

[0078] Q=F TCN W Q ,K=F TCN W K ,V=H Bi-LSTM W V

[0079] in, is the linear transformation matrix;

[0080] 3.2) Attention score calculation, the attention score is calculated by dot product, and then scaled and Softmax normalized:

[0081] 3.3) Multi-head attention fusion, concatenate the outputs of multiple attention heads, and obtain the final fusion features through linear transformation: MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O ;

[0082] Among them, head i =Attention(Q i ,K i ,V i ), h is the number of attention heads, is the linear transformation matrix;

[0083] 4) Adaptive attention: By introducing an adaptive attention mechanism, the number and weight of attention heads are dynamically adjusted, and adaptive weights are added after the output of each attention head: i =α i Attention (Q i ,K i ,V i );

[0084] Among them, α i is the adaptive weight parameter obtained through learning;

[0085] 5) Multimodal fusion. When processing multimodal tasks, a special attention mechanism is set to fuse the features of different modalities, including visual features and text features. The hidden state of Bi-LSTM, visual features and output features of the attention mechanism are concatenated in the feature dimension to obtain a new feature matrix:

[0086] combined_features t =[H Bi-LSTM ,MultiHead(Q,K,V)];

[0087] 6) Text recognition after feature fusion;

[0088] 6.1) Fully connected layer: Input the concatenated feature matrix into one or more fully connected layers for feature transformation:

[0089] fc output =ReLU(W f +bf )

[0090] Among them, W f and b f is the weight matrix and bias vector of the fully connected layer;

[0091] 6.2) Softmax layer, input the output of the fully connected layer into the Softmax layer to obtain the final recognition result: Softmax output =Softmax(W s +b s ), where W s and b s is the weight matrix and bias vector of the Softmax layer.

[0092] Preferably, the loss scaling factor is dynamically adjusted in each training step to ensure numerical stability, specifically including the following steps:

[0093] 1) Initialize the loss scaling factor: Scale Factor = 1024

[0094] 2) Forward propagation: The input image (X) passes through the feature extraction module and the text region proposal module to obtain the predicted text region and recognition results;

[0095] Calculate the total loss (L total ): L total =L detection +L recognition ;

[0096] Multiply the total loss by the loss scaling factor: L scaled =Scale Factor×L total ;

[0097] 3) Back propagation: Calculate the loss (L scaled )’s gradient:

[0098] Scale the gradient back to its original value:

[0099] 4) Dynamically adjust the loss scaling factor: If numerical instability is detected in the current step (such as NaN in the gradient), reduce the loss scaling factor: Scale Factor = Scale Factor ÷ 2;

[0100] If the current step value is stable, gradually increase the loss scaling factor to take full advantage of higher precision: ScaleFactor = Scale Factor × 2;

[0101] 5) Parameter update: Use the stochastic gradient descent algorithm (SGD) as the optimizer to update the model parameters: Where α is the learning rate.

[0102] Compared with the prior art, the present invention has the following beneficial effects:

[0103] The present invention proposes an irregular text recognition method based on deep learning and Raspberry Pi. The video signal is collected by the Raspberry Pi terminal camera, and the end-to-end irregular text recognition algorithm is called. It can be used for real-time scene text detection tasks and has the following advantages:

[0104] 1. The SKSAttention attention mechanism in the present invention introduces the Swish activation function based on the SKAttention attention mechanism. Compared with the traditional ReLU activation function, the Swish activation function enhances the feature expression ability, has better numerical stability and gradient propagation, and in the end-to-end irregular text recognition task, SKSAttention can better adapt to and recognize various complex and irregular text shapes through the Swish activation function.

[0105] 2. Improvements to the text region proposal stage of the present invention. Improved regression accuracy: By adding the IoU loss function when generating the classification and regression output of the candidate regions, the accuracy of the regression results is significantly improved. Better region positioning: The IoU loss function can prompt the model to locate the text region more accurately, reduce false detections and missed detections, and improve detection results.

[0106] 3. The present invention improves the text recognition stage, including a multi-head self-attention mechanism based on the combination of TCN and Bi-LSTM, where:

[0107] Combining TCN with Bi-LSTM:

[0108] 1) Capturing temporal information: TCN is good at capturing long-distance temporal dependencies, while Bi-LSTM can capture information from both the front and back directions. The combination of the two can provide a more comprehensive understanding of text sequences.

[0109] 2) Reduce gradient disappearance: Compared with traditional RNN, TCN and Bi-LSTM can effectively reduce the gradient disappearance problem and improve training efficiency when processing long sequences.

[0110] Multi-head self-attention mechanism:

[0111] 1) Enhanced feature representation: The multi-head self-attention mechanism extracts information from different subspaces to improve the diversity and richness of feature representation.

[0112] 2) Capturing global dependencies: The self-attention mechanism can capture dependencies on a global scale and improve the ability to recognize complex text patterns.

[0113] 4. The present invention improves the end-to-end training stage, namely the adaptive loss scaling method:

[0114] 1) Dynamically adjust the loss scaling factor: The adaptive loss scaling method can automatically adjust the loss scaling factor according to the numerical stability during training to ensure the stability of the training process.

[0115] 2) Improve computational efficiency: By dynamically adjusting the loss scaling factor, while ensuring numerical stability, computing resources can be fully utilized to improve training efficiency.

[0116] 3) Enhanced numerical stability: Reduce the risk of gradient explosion and vanishing, and ensure the stability and scalability of the model in large-scale training.

[0117] The present invention combines the above improvements to significantly improve the performance of the entire text detection and recognition system, making it more accurate and robust in various complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0119] In the attached picture:

[0120] Figure 1 It is a network flow chart of the feature extraction stage of the present invention;

[0121] Figure 2 This is the architecture diagram of the SKSAttention mechanism of the present invention;

[0122] Figure 3 It is a flow chart of the improved Feature Fusion feature integration module of the present invention;

[0123] Figure 4 It is a flow chart of the segmentation suggestion network module of the present invention;

[0124] Figure 5 It is a flow chart of the text recognition module of the present invention;

[0125] Figure 6 It is a flow chart of the end-to-end adaptive loss scaling of the present invention. DETAILED DESCRIPTION

[0126] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0127] Example: Figure 1-Figure 6As shown in the figure, an irregular text recognition method based on deep learning and Raspberry Pi includes the following end-to-end training steps, uses a mixed precision training method, and proposes an adaptive loss scaling method to automatically adjust the loss scaling factor to further improve numerical stability:

[0128] ①Text recognition algorithm module process:

[0129] S1. Data preprocessing: First, data preprocessing is performed, input images and corresponding text area annotations, and the images are standardized and data enhanced;

[0130] S2, feature extraction stage: use CSPDarknet53 to extract features from the input image; further improve the quality of features through the self-attention mechanism.

[0131] Backbone network part: CSPDarknet53 is used as the backbone network for feature extraction, and the SKSAttention attention mechanism based on the SKAttention attention mechanism is added before the SPPF module. The input image is set to P, and the feature map X is output after the backbone network CSPDarknet53 extracts features. Before entering the SPPF module, it is first processed by the SKSAttention attention module based on the SKAttention attention mechanism.

[0132] The SKSAttention mechanism improves the performance of the model when processing multi-scale features by adaptively selecting the appropriate convolution kernel size. The Swish activation function is introduced, which is smoother and continuously differentiable than ReLU, avoiding the situation where the ReLU output is always zero. This smoothness helps the propagation of gradients and the training of the model.

[0133] The main process of SKSAttention mechanism is as follows:

[0134] 1. Multi-scale convolution: The input feature map (X) is convolved with two convolution kernels of different scales to generate two feature maps of different scales (u1) and (u2). Convolution kernels of (3×3) and (5×5) are usually used.

[0135] u1=conv 3×3 (x)

[0136] u2=conv 5×5 (x)

[0137] 2. Feature fusion: Add two feature maps of different scales to obtain a fused feature map (U y ).

[0138] U y=u1+u2

[0139] 3. Global average pooling: The fused feature map (Uy) is spatially compressed through global average pooling to obtain the global feature vector (z).

[0140] z = global_avg_pool(U y )

[0141] 4. Channel Attention Mechanism (CAM)

[0142] Perform a full connection operation on the global feature vector (z) to generate a selection vector (s). Use two fully connected layers and a Swish activation function.

[0143] s intermediate =Swish(fc1(z))

[0144] s=fc2(sintermediate)

[0145] 5. Weighted summation: Normalize the selection vector (s) through the softmax function to obtain the selection weight (α): α = softmax (s)

[0146] According to the selected weight (α), the feature maps (u1) and (u2) of two different scales are weighted and summed to obtain the final output feature map (U x ): U x =α1·u1+α2·u2

[0147] Neck network: Outputs the fused feature map U from the model backbone network x Enter the FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) feature pyramid modules for feature fusion operations, and output three feature maps of different sizes U x1 , U x2 , U x3 .

[0148] Head network: an improved Feature Fusion feature integration module is proposed. The three feature maps of different sizes output by the Neck part are input into an improved Feature Fusion feature integration module, and the Feature Enhancement Module (FEM) and Adaptive Feature Fusion Module (AFFM) are introduced to effectively fuse the feature maps of different scales generated by FPN and PAN into a feature map U.

[0149] ① In this step, firstly, the three feature maps U of different sizes output from the Neck part are x1 , U x2 , U x3 Perform feature alignment: Assuming the target size is H×W, use upsampling and downsampling operations to adjust all feature maps to the same size:

[0150] Upsampling operation: For feature maps with smaller size, use bilinear interpolation or deconvolution operation to upsample them to the target size:

[0151] Downsampling operation: For feature maps with larger sizes, use average pooling or convolution operations to downsample them to the target size:

[0152] ②Introduce Feature Enhancement Module (FEM) to extract more contextual information. Use multi-layer convolution and dilated convolution to capture different receptive fields:

[0153]

[0154] ③ In order to better fuse multi-scale features, the Adaptive Feature Fusion Module (AFFM) is introduced, and the attention mechanism is used to learn each feature map. Including:

[0155] Adaptive weight generation:

[0156] Weighted feature fusion:

[0157] ④ Feature optimization: Add a common convolution operation Conv(x)=W*x and ReLU activation function to the fused feature map for further feature optimization:

[0158] u optimized =Conv(ReLU(Conv(u fused )))

[0159] Feature map U optimized This is the final feature map U, which is input into the text region proposal module.

[0160] S3. Text region proposal: The extracted features are input into the Transformer-based text region proposal module (e.g., DETR).

[0161] Based on the Segmentation Proposal Network (SPN) and hard RoI masking methods, it is proposed to add an IoU loss function to improve the regression results when generating the classification and regression outputs of candidate regions through the segmentation proposal network (SPN), so that the generated candidate regions are more accurate.

[0162] IoU calculation formula: The calculation formula for the intersection over union (IoU) between two regions (prediction box (A) and real box (B)) is defined as:

[0163]

[0164] in:

[0165] Area(A∩B) represents the intersection area of ​​the predicted box and the true box.

[0166] Area(A∪B) represents the union area of ​​the predicted box and the true box.

[0167] IoU loss is defined as 1 minus the IoU value: IoU Loss = 1-IoU(A,B)

[0168] Specific steps:

[0169] ① Candidate region generation. First, the candidate region is generated using the fused feature map (U):

[0170]

[0171] Among them, (R i ) represents the generated candidate region.

[0172] ② Feature extraction. For each candidate region (R i ), extract the corresponding features from the feature map (U). Use the RoIAlign operation.

[0173]

[0174] ③Classification and regression. The features of each candidate region Through the fully connected layer, classification and bounding box regression are performed separately.

[0175] Classification output:

[0176] Regression output:

[0177] Among them, (FCcls) and (FCreg) represent the fully connected layers for classification and regression.

[0178] ④Add IoU loss.

[0179] The IoU loss function is defined as:

[0180] in, is the candidate region (R i ) corresponds to the ground-truth bounding box.

[0181] The total loss function is expressed as the weighted sum of classification loss and regression loss (including IoU loss):

[0182]

[0183] Among them, (L cls ) is the classification loss, (L reg ) is the standard regression SmoothL1 loss, (L IoU ) is the IoU loss, (λ) and (μ) are weight coefficients.

[0184] ⑤Generate the final TestRoI feature map.

[0185] Use the trained network to train each candidate region (R i ) to perform classification and regression, obtain the final bounding box, and extract the corresponding features from the feature map (U):

[0186] in, is the bounding box predicted during the test phase.

[0187] Final TestRoI feature map It is expressed as:

[0188]

[0189] Through the above steps, the fused feature map (U) is passed through the segmentation proposal network (SPN) to generate the classification and regression output of the candidate region, and the IoU loss is added to improve the regression result, and finally the TestRoI feature map is obtained. This method improves the accuracy of the candidate region and improves the performance of object detection and segmentation.

[0190] S3, text shape prediction: output the predicted text area.

[0191] Introduce a MaskBranch module. The TestRoI feature map output by the text region proposal network is passed through this module to output the binary mask of the target text. MaskBranch is a small fully convolutional network (FCN) applied to each RoI to predict the segmentation mask in a pixel-to-pixel manner to obtain the target text image frame mask. Specific steps:

[0192] ①RoI feature map input:

[0193] TestRoI feature map generated by text region proposal network as input.

[0194] ②MaskBranch structure: Here, the MaskBranch module includes three convolutional layers and one upsampling layer:

[0195] Convolutional layer 1 (convolution operation on input feature map): Convolutional layer 2 (further feature extraction): Upsampling layer (upsamples the feature map to the input RoI size):

[0196]

[0197] Convolutional layer 3 (convolution of the upsampled feature map to obtain binary mask prediction):

[0198] Among them, (Conv2D) represents a two-dimensional convolution operation, (W1, W2, W3) is the convolution kernel, and (b1, b2, b3) is the bias.

[0199] ③ Loss function: Use the binary cross entropy loss function to optimize the output of MaskBranch. Assume that the true binary mask is Then the loss function is:

[0200]

[0201] Through the above steps, MaskBranch converts each TestRoI feature map Convert to binary mask The final output is a set of binary masks of the target text image frames:

[0202] S4. Text recognition: Use a sequence model (such as LSTM or Transformer decoder) to recognize the text content from the proposed text regions.

[0203] A text recognition method based on the combination of Temporal Convolutional Network (TCN) and Bi-LSTM is proposed, and a multi-head self-attention mechanism is introduced.

[0204] TCN: Arrange the target text image frames in time order to form a time series data sequence X, and then use TCN to process these time series data. m ) is input to TCN, which uses a series of one-dimensional convolution operations to extract temporal features. These convolution operations have a long-range receptive field and capture long-term dependencies. The output of TCN is a sequence containing temporal features, representing the feature representation of each time step.

[0205] Bi-LSTM: Use Bi-LSTM to process the time series features extracted by TCN. Input the output feature sequence of TCN into Bi-LSTM. Bi-LSTM processes the sequence data in both forward and backward directions to capture the bidirectional time dependency. The output of Bi-LSTM is the hidden state of each time step, which contains the contextual understanding of the sequence.

[0206] Introducing attention mechanism: In order to improve the recognition accuracy and robustness of the image frame mask processing model based on temporal convolutional network (TCN) and bidirectional long short-term memory network (Bi-LSTM), a multi-head self-attention mechanism (Multi-Head Self-Attention) is proposed, and corresponding improvements are made in adaptive attention and multimodal fusion.

[0207] The specific implementation is as follows:

[0208] 1. TCN feature extraction: Use the temporal convolutional network to extract the local features of the time series and obtain the feature matrix (F TCN ∈R T×dTN ).

[0209] Formula: F TCN =TCN(X m ); where (X m ) is the input time series data, and TCN is the temporal convolutional network.

[0210] 2. Bi-LSTM feature extraction: Use bidirectional LSTM to extract the global features of the time series and obtain the hidden state matrix Formula: H Bi-LSTM =Bi-LSTM(X m ); where (X m ) is the input time series data, and Bi-LSTM is a bidirectional long short-term memory network.

[0211] 3. Multi-Head Self-Attention

[0212] 3.1 Linear transformation: Perform linear transformation on the features extracted by TCN and Bi-LSTM to obtain the query matrix Q, key matrix K and value matrix V:

[0213] Q=F TCN W Q ,K=F TCN W K ,V=H Bi-LSTM W V

[0214] in, is the linear transformation matrix.

[0215] 3.2 Attention score calculation: The attention score is calculated by dot product, and then scaled and Softmax normalized:

[0216]

[0217] 3.3 Multi-Head Attention Fusion: Concatenate the outputs of multiple attention heads and obtain the final fusion features through linear transformation: MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O

[0218] Among them, head i =Attention(Q i ,K i ,V i ), h is the number of attention heads, is the linear transformation matrix.

[0219] 4. Adaptive attention: By introducing an adaptive attention mechanism, the number and weight of attention heads are dynamically adjusted to better meet the needs of different tasks. Adaptive weights are added after each attention head output: head i =α i Attention (Q i ,K i ,V i ), where α i is the adaptive weight parameter obtained through learning.

[0220] 5. Multimodal fusion: When dealing with multimodal tasks, design a dedicated attention mechanism to fuse features of different modalities, such as visual features and textual features.

[0221] 5.1 Feature concatenation: Concatenate the hidden state, visual features, and attention mechanism output features of Bi-LSTM in the feature dimension to obtain a new feature matrix:

[0222] combined_features t =[H Bi-LSTM ,MultiHead(Q,K,V)]

[0223] 6. Text recognition after feature fusion

[0224] 6.1 Fully connected layer: Input the concatenated feature matrix into one or more fully connected layers for feature conversion:

[0225] fc output =ReLU(W f +b f )

[0226] Among them, W f and b f is the weight matrix and bias vector of the fully connected layer.

[0227] 6.1Softmax layer: Input the output of the fully connected layer to the Softmax layer to obtain the final recognition result: Softmax output =Softmax(W s +b s )

[0228] Among them, W s and b s is the weight matrix and bias vector of the Softmax layer.

[0229] S5. Loss calculation: Calculate the total loss of text region detection loss and text recognition loss.

[0230] S6. Mixed precision training: Use FP16 and FP32 precision for calculations during training to speed up the training process and reduce video memory usage.

[0231] S7. Adaptive loss scaling: Initialize the loss scaling factor and dynamically adjust the loss scaling factor in each training step to ensure numerical stability. Regarding adaptive loss scaling:

[0232] 1) Initialize the loss scaling factor: Scale Factor = 1024

[0233] 2) Forward propagation: The input image (X) passes through the feature extraction module and the text region proposal module to obtain the predicted text region and recognition results.

[0234] Calculate the total loss (L total ): L total =L detection +L recognition ;

[0235] Multiply the total loss by the loss scaling factor: L scaled =Scale Factor×L total ;

[0236] 3) Backward Propagation:

[0237] Calculate the loss (L scaled )’s gradient:

[0238] Scale the gradient back to its original value:

[0239] 4) Dynamically adjust the loss scaling factor: If numerical instability is detected in the current step (such as NaN in the gradient), reduce the loss scaling factor: Scale Factor = Scale Factor ÷ 2;

[0240] If the current step is numerically stable, gradually increase the loss scaling factor to take advantage of higher accuracy: ScaleFactor = Scale Factor × 2.

[0241] 5) Parameter update: Use the stochastic gradient descent algorithm (SGD) as the optimizer to update the model parameters: Where α is the learning rate.

[0242] ②Algorithm deployment:

[0243] Finally, the algorithm was deployed on the Raspberry Pi terminal to solve the irregular text recognition task in portable real-time scenarios.

[0244] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An irregular text recognition method based on deep learning and Raspberry Pi, characterized in that: The following steps are involved: S1. First, data preprocessing is performed, input images and corresponding text area annotations are input, and the images are standardized and data enhanced; S2. Feature extraction is performed through the backbone network, and an improved SKSAttention self-attention mechanism is added before the SPPF module to improve the quality of features through the introduced SKSAttention self-attention mechanism; S3, perform text region proposal, input the extracted features into the Transformer-based text region proposal module, and output the predicted text region; S4, using sequence models to identify text content from proposed text regions; 1) First, the target text image frames are arranged in chronological order to form a time series data sequence. Then, the time series data is processed using TCN, and the mask sequence is input into the time series convolution network TCN. TCN uses a series of one-dimensional convolution operations to extract time series features, where the output of TCN is a sequence containing time series features, indicating the feature representation of each time step; 2) Use a bidirectional long short-term memory network Bi-LSTM to process the time series features extracted by TCN. The output feature sequence of TCN is input into Bi-LSTM. Bi-LSTM processes the sequence data in both forward and backward directions to capture the bidirectional time dependency. The output of Bi-LSTM is the hidden state of each time step, which contains the contextual understanding of the sequence. 3) Introduce a multi-head self-attention mechanism to extract information from different subspaces and capture dependencies globally; S5, calculating the total loss of text region detection loss and text recognition loss; S6. Use FP16 and FP32 precision for calculation during training to speed up the training process and reduce video memory usage; S7. Finally, the algorithm is deployed on the Raspberry Pi terminal to solve the irregular text recognition task in real-time scenarios.

2. The irregular text recognition method based on deep learning and Raspberry Pi according to claim 1, characterized in that: In step S2, the SKSAttention attention mechanism improves the performance of the model in processing multi-scale features by adaptively selecting the appropriate convolution kernel size, and introduces the Swish activation function. The SKSAttention attention mechanism process steps include the following: 1) Perform multi-scale convolution: The input feature map (X) is convolved with two convolution kernels of different scales to generate two feature maps of different scales (u1) and (u2); convolution kernels of (3×3) and (5×5) are used; u1=conv 3×3 (x);u2=conv 5×5 (x) 2) Feature fusion: Add two feature maps of different scales to obtain a fused feature map (U y ): U y =u1+u2 3) Perform global average pooling: The fusion feature map (U y ) to perform spatial compression and obtain the global eigenvector z: z = global_avg_pool (U y ) 4) Perform channel attention mechanism: perform a full connection operation on the global feature vector (z) to generate a selection vector (s); use two fully connected layers and a Swish activation function: s intermediate =Swish(fc1(z)) s=fc2(sintermediate) 5) Perform weighted summation: Normalize the selection vector (s) through the softmax function to obtain the selection weight (α): α = softmax(s) According to the selected weight (α), the feature maps (u1) and (u2) of two different scales are weighted and summed to obtain the final output feature map (U x ): U x =α1·u1+α2·u2 Output the fused feature map U from the model backbone network x Enter the FPN and PAN feature pyramid modules for feature fusion operations and output three feature maps of different sizes U x1 , U x2 , U x3 .

3. The irregular text recognition method based on deep learning and Raspberry Pi according to claim 2, characterized in that: By introducing an improved FeatureFusion feature integration module, the three feature maps of different sizes output by the Neck part are input into the improved FeatureFusion feature integration module, and a feature enhancement module and an adaptive feature fusion module are introduced to effectively fuse the feature maps of different scales generated by FPN and PAN into a feature map U. The specific steps include: 1) First, the three feature maps U of different sizes output by the Neck part x1 , U x2 , U x3 Perform feature alignment operations: Assuming the target size is H×W, upsampling and downsampling operations are used to resize all feature maps to the same size: Upsampling operation: Use bilinear interpolation or deconvolution operation to upsample it to the target size: Downsampling operation: Use average pooling or convolution operation to downsample it to the target size: 2) Introduce a feature enhancement module to extract contextual information; use multi-layer convolution and dilated convolution to capture different receptive fields: 3) Introduce an adaptive feature fusion module, use the attention mechanism to learn each feature map, and fuse multi-scale features, including: Adaptive weight generation: Weighted feature fusion: 4) Perform feature optimization: Add a common convolution operation Conv(x)=W*x and ReLU activation function to the fused feature map for further feature optimization: you optimized =Conv(ReLU(Conv(u fused ))) Feature map U optimized is the final feature map U, which is input into the text region proposal module.

4. The irregular text recognition method based on deep learning and Raspberry Pi according to claim 1, characterized in that: In step S3, based on the segmentation proposal network SPN and the hardRoImasking method, it is proposed to add an IoU loss function to improve the regression results when generating the classification and regression outputs of the candidate regions through the segmentation proposal network SPN, so that the generated candidate regions are more accurate; Among them, the IoU calculation formula represents the intersection-over-union ratio between two regions, the two regions include the predicted box (A) and the real box (B); the (IoU) calculation formula is: Among them: Area(A∩B) represents the intersection area of ​​the predicted box and the real box; Area(A∪B) represents the union area of ​​the predicted box and the true box; IoU loss is defined as 1 minus the IoU value: IoU Loss = 1-IoU(A,B); the specific steps include: 1) Candidate region generation: First, the candidate region is generated using the fused feature map (U): Among them, (R i ) represents the generated candidate region; 2) Feature extraction: For each candidate region (R i ), extract the corresponding features from the feature map (U); use RoIAlign operation; 3) Classification and regression: The features of each candidate region Through the fully connected layer, classification and bounding box regression are performed respectively; Classification output: Regression output: Among them, (FCcls) and (FCreg) represent the fully connected layers for classification and regression; 4) Add IoU loss: The IoU loss function is defined as: in, is the candidate region (R i )The corresponding ground-truth bounding box; The total loss function is expressed as a weighted sum of classification loss and regression loss: Among them, (L cls ) is the classification loss, (L reg ) is the standard regression SmoothL1 loss, (L IoU ) is the IoU loss, (λ) and (μ) are weight coefficients; 5) Generate the final TestRoI feature map: Use the trained network to generate each candidate region (R i ) performs classification and regression to obtain the final bounding box and extracts the corresponding features from the feature map (U); in, is the bounding box predicted in the test phase; the final TestRoI feature map It is expressed as: Through the above steps, the fused feature map (U) is passed through the segmentation proposal network (SPN) to generate the classification and regression output of the candidate region, and the IoU loss is added to improve the regression result, and finally the TestRoI feature map is obtained.

5. The irregular text recognition method based on deep learning and Raspberry Pi according to claim 1, characterized in that: In step S3, the TestRoI feature map output by the text region proposal network is applied to each RoI through the introduced MaskBranch module, and the segmentation mask is predicted in a pixel-to-pixel manner to obtain the target text image frame mask, and the binary mask of the target text is output. The specific steps include: 1) RoI feature map input: Generate TestRoI feature map using text region proposal network As input; 2) The MaskBranch module consists of three convolutional layers and one upsampling layer: Convolutional layer 1 performs convolution operation on the input feature map: Convolutional layer 2 further extracts features: Upsampling layer, upsampling the feature map to the input RoI size: Convolutional layer 3 convolves the upsampled feature map to obtain the binary mask prediction: Among them, (Conv2D) represents a two-dimensional convolution operation, (W1, W2, W3) is the convolution kernel, and (b1, b2, b3) is the bias; 3) Loss function: Use the binary cross entropy loss function to optimize the output of MaskBranch, assuming that the true binary mask is Then the loss function is: Through the above steps, MaskBranch converts each TestRoI feature map Convert to binary mask The final output is a set of binary masks of the target text image frames.

6. The irregular text recognition method based on deep learning and Raspberry Pi according to claim 1, characterized in that: In step S4, the specific implementation steps are as follows: 1) TCN feature extraction: Use the temporal convolutional network to extract the local features of the time series and obtain the feature matrix (F TCN ∈R T ×dTN ); Formula: F TCN =TCN(X m ); Where (X m ) is the input time series data, TCN is the temporal convolutional network; 2) Bi-LSTM feature extraction: use bidirectional LSTM to extract the global features of the time series and obtain the hidden state matrix Formula: H Bi-LSTM =Bi-LSTM(X m ); where (X m ) is the input time series data, Bi-LSTM is a bidirectional long short-term memory network; 3) Multi-head self-attention mechanism 3.1) Linear transformation: Perform linear transformation on the features extracted by TCN and Bi-LSTM to obtain the query matrix Q, key matrix K and value matrix V: Q=F TCN W Q ,K=F TCN W K ,V=H Bi-LSTM W V ; in, is the linear transformation matrix; 3.2) Attention score calculation, the attention score is calculated by dot product, and then scaled and Softmax normalized: 3.3) Multi-head attention fusion, concatenate the outputs of multiple attention heads, and obtain the final fusion features through linear transformation: Multi Head(Q,K,V)=Concat(head1,head2,…,head h )W O ; in, head i =Attention(Q i ,K i ,V i ), h is the number of attention heads, is the linear transformation matrix; 4) Adaptive attention: By introducing an adaptive attention mechanism, the number and weight of attention heads are dynamically adjusted, and adaptive weights are added after the output of each attention head: head i =α i ·Attention(Q i ,K i ,V i ) Among them, α i is the adaptive weight parameter obtained through learning; 5) Multimodal fusion. When processing multimodal tasks, a special attention mechanism is set to fuse the features of different modalities, including visual features and text features. The hidden state of Bi-LSTM, visual features and output features of the attention mechanism are concatenated in the feature dimension to obtain a new feature matrix: combined_features t =[H Bi-LSTM ,MultiHead(Q,K,V)]; 6) Text recognition after feature fusion; 6.1) Fully connected layer: input the concatenated feature matrix into one or more fully connected layers for feature transformation: fc output =ReLU(W f +b f ); Among them, W f and b f is the weight matrix and bias vector of the fully connected layer; 6.2) Softmax layer, input the output of the fully connected layer into the Softmax layer to obtain the final recognition result: Softmax output =Softmax(W s +b s ); Among them, W s and b s is the weight matrix and bias vector of the Softmax layer.

7. The irregular text recognition method based on deep learning and Raspberry Pi according to claim 1, characterized in that: Dynamically adjust the loss scaling factor in each training step to ensure numerical stability. Specifically, the following steps are included: 1) Initialize the loss scaling factor: Scale Factor = 1024; 2) Forward propagation: The input image passes through the feature extraction module and the text region proposal module to obtain the predicted text region and recognition results; Calculate the total loss (L total ): L total =L detection +L recognition ; Multiply the total loss by the loss scaling factor: L scaled =Scale Factor×L total ; 3) Back propagation: Calculate the loss (L scaled )’s gradient: Scale the gradient back to its original value: 4) Dynamically adjust the loss scaling factor: If numerical instability is detected in the current step, reduce the loss scaling factor: Scale Factor = Scale Factor ÷ 2; If the current step value is stable, gradually increase the loss scaling factor to take full advantage of higher precision: Scale Factor = Scale Factor × 2; 5) Parameter update: Use the stochastic gradient descent algorithm as the optimizer to update the model parameters: Where α is the learning rate.

Images