Irregular text recognition method for flight instruments based on character localization and CRNN

Abstract

A method for irregular text recognition on flight instruments based on character localization and CRNN is proposed. Using a selected character-level annotated dataset, the parameters of a character localization-based detection network and a CRNN recognition network are trained. The trained detection and CRNN recognition networks are then used to perform online text recognition on real-world flight instrument images containing irregular text. Based on a two-stage recognition algorithm, the method independently designs the detection and recognition networks, significantly improving the irregular text recognition accuracy of the entire recognition network.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a method for recognizing irregular text on flight instruments based on character positioning and CRNN. Background Art

[0002] While flying an aircraft, pilots rely on the data displayed by flight instruments to make accurate decisions and operate the aircraft correctly. Due to the complexity of the flight environment, aircraft are equipped with a variety of flight instruments with different indications, which can sometimes display irregular text. Furthermore, aircraft control panels can also display irregular text.

[0003] Existing image recognition technology is highly dependent on text scenes and the characteristics of the text itself, making it difficult to perform feature fusion and lacking generalization. The one-stage recognition algorithm based on neural networks and deep learning uses the same neural network for text detection and text recognition, which makes network design and parameter debugging very difficult, and has poor migration performance for irregular text recognition tasks in different scenarios. The two-stage recognition algorithm based on neural networks and deep learning cannot solve the positioning deviation in irregular text scenes such as dense text and text adhesion in file detection, and pre-correction in text recognition cannot further improve recognition accuracy. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the present invention proposes a flight instrument irregular text recognition method based on character positioning and CRNN. ​​The detection network and the recognition network are designed separately based on the idea of ​​a two-stage recognition algorithm. For irregular text detection, a detection algorithm framework based on character positioning is adopted. Each character area is first detected, and then the characters are used to determine the key points on the boundary, and then the key points are connected to obtain the text area. For irregular text recognition, a CRNN recognition network with excellent performance in horizontal text recognition is adopted. The irregular text recognition accuracy of the entire recognition network is improved by pre-correction and the introduction of FReLU activation function.

[0005] The present invention is achieved through the following technical solutions:

[0006] The present invention relates to a method for recognizing irregular text on flight instruments based on character positioning and CRNN. ​​Parameters of a detection network based on character positioning and a CRNN recognition network are trained using a selected character-level annotation dataset. The trained detection network and CRNN recognition network are then used to perform online text recognition on real-shot flight instrument images containing irregular text.

[0007] The detection network based on character positioning, that is, the backbone network includes: a feature extraction module using a hierarchical deep aggregation structure and a feature fusion module using an iterative deep aggregation algorithm, wherein: the feature extraction module extracts shallow features and deep features, and the feature fusion module fuses the shallow features and the deep features, and outputs the character area heat map, the connection relationship heat map, the text area segmentation map and the text center area segmentation map to the CRNN recognition network.

[0008] The CRNN recognition network performs connected domain analysis on the superimposed mask based on the character area, connection area and text area processed by the feature fusion module, and determines the boundary of each text instance through the minimum circumscribed rectangle of each character area and the direction information provided by the text center area. The CRNN recognition network includes: a rotation pre-correction module and a CRNN recognition module, wherein: the rotation pre-correction module calculates the minimum rotation circumscribed rectangle of each polygonal text instance area based on the text area polygon box from the detection network, and then obtains the text area rotated to the horizontal through rotation transformation. The CRNN recognition module extracts character features from the text area, and obtains the recognized irregular text content through feature prediction and decoding.

[0009] The CRNN recognition module includes: a CNN character feature extraction unit, an RNN sequence feature prediction unit and a CTC decoding unit, wherein: the CNN character extraction unit inputs the corrected text area image into the ResNet50 network that introduces FReLU as the activation function to obtain a convolution feature sequence, the RNN sequence feature prediction unit obtains the sequence features in the predicted convolution features based on the convolution feature sequence extracted by CNN through BiLSTM prediction, and the CTC decoding unit decodes the input sequence features to obtain the recognized irregular text content.

[0010] The parameter training specifically includes: the weights and bias values ​​of the excitation functions of each neuron in the detection network and the recognition network, as well as the learning rate.

[0011] The online text recognition method involves cropping the flight instrument area to be identified from the actual flight instrument image containing irregular text, converting it into a grayscale image, filtering it to eliminate noise, and then inputting the trained detection network and CRNN recognition network to perform text detection and text recognition, and outputting the recognition results.

[0012] The irregular text recognition method for flight instruments specifically includes:

[0013] S1, select a scene dataset with character-level annotations and annotate it according to the true value, specifically including: the setting rule of the text area true value is to calculate the ratio r of the area after removing each vertex to the original area for each text instance with more than 4 vertices in the boundary annotation, when |r-1|<th ratio , then remove the point from the original vertex set; the setting rule of the true value of the text center area is to determine the upper and lower long sides and the left and right short sides of the text, and then calculate the total length of the upper and lower boundaries of the two long sides, and divide it into 20 equal parts, recorded as [up0, up1, ..., up 19 ], [dw0, dw1, ..., dw 19 ]. From up2 to up 17 and dw2 to dw 17 , connect the points in the upper and lower boundaries to form 15 line segments, obtain the midpoints of these 15 line segments, extend them from the midpoints in the positive and negative directions by 0.3 times the length of the line segments, and then connect the extended line segments.

[0014] S2 reads the dataset image and predicts the character area, the connection area between characters, the text area, and the text center area through the backbone network;

[0015] S3, determining the character area, connection area, and text center area using a preset threshold, connecting the character areas between each text instance, then superimposing the character area, connection area, and text center area, and performing connected domain analysis on the superimposed mask;

[0016] S4, determine the boundary of each text instance through the minimum bounding rectangle of each character region and the direction information provided by the text center region. The specific process is as follows: each character region C in the same text instance except the beginning and the end i The intersection points of each side of the minimum circumscribed rectangle and the upper and lower boundaries of the text center area are marked as CLi1, CLi2, CLi3, and CLi4 in clockwise order; find the midpoint L of the line segment CLi1CLi4 i and the midpoint R of line segment CLi2CLi3 i The midpoint CC of the connecting line i ; Find the minimum circumscribed rectangle of each side and line segment L i R i Angle; if the difference between the vertical angles of the two sides is greater than the threshold th θ , then len is the length of the side with the larger angle, otherwise len is the length of the line segment CC i+2 The longest line segment in the vertical direction; passing through the center point CC i Draw line segment L i R iThe vertical line, whose length is len, is the upper and lower boundary points BT i ,BB i .

[0017] S5, according to the calculated loss function, adjust the detection network parameters, specifically: L = L cls +λ hm L hm , where: the loss L of the text segmentation task cls =-[yα(1-p) γ ln(p)+(1-y)(1-α)p γ ln(1-p)], the loss of the heat map prediction task

[0018] S6, pre-correcting the text instance area determined by the detection network, and performing rotation correction on the irregular text area to be horizontal. Specifically, the minimum rotation circumscribed rectangle of each polygonal text instance area is obtained, and then the area is rotated to be horizontal through a rotation transformation.

[0019] S7, extracting corresponding features of characters for the text instance area to be recognized through CNN;

[0020] S8, predicting sequence features in convolutional features through RNN;

[0021] S9, decode the output of RNN through CTC (Connectionist Temporal Classification).

[0022] S10, calculate according to CTC Loss and adjust the parameters of the CRNN recognition network, specifically: Where: N is the number of images in the training set, is the true value of training, and y′ is the predicted result.

[0023] S11, repeat S2 to S10 until all the images of the selected training dataset are sent to the network for training.

[0024] Technical Effects

[0025] The present invention locates irregular text by obtaining the boundary points of irregular text based on the character area, connection area, text area and text center area. The pre-correction of the located irregular text is to obtain the minimum rotation circumscribed rectangle of each polygonal text instance area, and then rotate the area to the horizontal through rotation transformation. The present invention further improves the accuracy of irregular file detection and recognition by improving the two-stage algorithm respectively. Among them, the irregular text detection network based on character positioning adopts a detection algorithm based on character positioning, which increases the accuracy of irregular text positioning. The CRNN irregular text recognition network reduces the influence of the pre-correction module on the irregular text content area and improves the recognition accuracy of the CRNN recognition network. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart of the present invention;

[0027] Figure 2 This is an example diagram of irregular text recognition on a flight control panel that can be transferred and applied by the present invention;

[0028] Figure 3 This is a schematic diagram of the network structure of file detection based on character positioning of the present invention;

[0029] Figure 4 Schematic diagram of the post-processing algorithm for the detection network output of the present invention;

[0030] Figure 5 Schematic diagram of the CRNN recognition network structure of the present invention;

[0031] Figure 6 This is the BiLSTM structure diagram in the CRNN recognition network of the present invention;

[0032] Figure 7 Detection and recognition flow chart for flight instrument images;

[0033] Figure 8 This is an example diagram of flight instruments in the embodiment. DETAILED DESCRIPTION

[0034] like Figure 1 As shown, this embodiment relates to a detection and recognition method for irregular text recognition on flight instruments, including: a detection and recognition network training phase and a flight instrument image detection phase, specifically including:

[0035] A1. Select SyntText, a standard scene dataset with character-level annotations, and annotate it according to the ground truth. The rule for setting the ground truth of the text region is to calculate the ratio r of the area after removing each vertex to the original area for each text instance with more than 4 vertices in the boundary annotation. When |r-1| < th ratio, then remove the point from the original vertex set; the setting rule of the true value of the text center area is to determine the upper and lower long sides and the left and right short sides of the text, and then calculate the total length of the upper and lower boundaries of the two long sides, and divide it into 20 equal parts, recorded as [up0, up1, ..., up 19 ], [dw0, dw1, ..., dw 19 ]. From up2 to up 17 and dw2 to dw 17 , connect the points in the upper and lower boundaries to form 15 line segments, obtain the midpoints of these 15 line segments, extend them from the midpoints in the positive and negative directions by 0.3 times the length of the line segments, and then connect the extended line segments.

[0036] A2 reads in the labeled dataset image, cuts out each text instance from the image, and uses the backbone network to predict the character region, the connection region between characters, the text region, and the text center region. The backbone network outputs four components: a heat map of the character region, a heat map of the connection relationship, a text region segmentation map, and a text center region segmentation map.

[0037] A3: Superimpose the processed character area, connection area, and text center area, and perform connected component analysis on the superimposed mask;

[0038] A4,determines the boundary of each text instance by the minimum bounding rectangle of each character area and the direction information provided by the text center area;

[0039] like Figure 4 As shown, steps A2 to A4 specifically include:

[0040] 1) First, use the watershed algorithm to distinguish each character area on the predicted character area heat map;

[0041] 2) Using the superposition of three images of character area, connection area, and text center area, different text instances are determined through connected domain analysis;

[0042] 3) For each character region C in the same text instance except the beginning and the end i , find its minimum circumscribed rectangle;

[0043] 4) Find C i The intersection points of each side of the minimum circumscribed rectangle with the upper and lower boundaries of the text center area are marked as CLi1, CLi2, CLi3, and CLi4 in clockwise order;

[0044] 5) Find the midpoint L of line segment CLi1CLi4 i and the midpoint R of line segment CLi2CLi3 iThe midpoint CC of the connecting line i ;

[0045] 6) Find the minimum circumscribed rectangle and the line segment L i R i Angle;

[0046] 7) When the difference between the vertical angles of the two sides is greater than the threshold th θ , then len is the length of the side with the larger angle, otherwise len is the length of the line segment CC i+2 The longest line segment in the vertical direction;

[0047] 8) Passing the center point CC i Draw line segment L i R i The vertical line, whose length is len, is the upper and lower boundary points BT i ,BB i .

[0048] 9) Repeat steps 3 to 8 for the remaining character areas.

[0049] A5, calculate the loss function L and adjust the detection network parameters, specifically: L = L cls +λ hm L hm , where: L cls

[0050] represents the loss of the text segmentation task, L cls =-[yα(1-p) γ ln(p)+(1-y)(1-α)p γ ln(1-p)],L hm is the loss of the heat map prediction task, in

[0051] A6, pre-corrects the text instance area determined by the detection network and performs rotation correction on the irregular text area to make it horizontal. That is, the minimum rotational bounding rectangle of each polygonal text instance area is obtained, and then the area is rotated to the horizontal through a rotation transformation.

[0052] A7, through ResNet50, the input is mapped to the output by shortcut for the text instance area to be recognized to extract the corresponding features of the characters. The network uses FReLU as the activation function, specifically: y = max(x c，i，j , T(x c，i，j )), Where: x c，i，j For the feature pixel point with coordinates (i, j) on the two-dimensional feature map of the c-th channel, the function T(x c，i，j) is the Funnel condition, where: For feature points is the center and the size is kernel w ×kernel h Parameter pooling window to extract x c，i，j Surrounding space information. is the shared parameter pooling window coefficient on the c-th channel.

[0053] A8, using BiLSTM (Bi-directional Long Short-Term Memory) to predict the sequence features in the convolutional features: the output of the CNN in step A7 is regarded as 25 feature sequences with a dimension of 512. Each frame feature sequence corresponds to an equal-width area in the original image. These 25 feature sequences are regarded as 25 time steps, denoted as x = (x1, x2, ..., x t ,...,x T ) is input into the BiLSTM (Bi-directional Long Short-Term Memory) network, and the output after the softmax layer is y=(y 1 ,y 2 ,...,y t ,...,y T ),in: is the probability of the character corresponding to the t-th frame, and n is the number of categories of characters that can be predicted. t ,Pick The character with the highest probability is taken as the final output of BiLSTM, which is recorded as y=(c 1 , c 2 ,...,c T ).

[0054] A9 decodes the RNN output using CTC (Connectionist Temporal Classification). The transformation is β:

[0055] A10 calculates the CTC Loss and adjusts the parameters of the CRNN recognition network. Specifically: Where: N is the number of images in the training set, is the true value of training, and y′ is the predicted result.

[0056] A11, repeat steps A2 to A10 until all the images in the selected training dataset are sent to the network for training.

[0057] B1, perform flight instrument image acquisition, photograph the flight instruments of actual aircraft, and try to avoid text occlusion, text loss, etc. caused by human-selected incorrect perspectives when taking images.

[0058] B2, perform image preprocessing. First, crop the flight instrument area to be identified. In addition, the cropped image is converted into a grayscale image, and then a median filter with a kernel size of 3×3 is performed to eliminate noise.

[0059] B3, sending the pre-processed image obtained in step B2 to the trained detection and recognition network for text detection and text recognition, and outputting the recognition result.

[0060] After specific actual experiments, running on an NVIDIA GTX1080 GPU with parameters such as batch size 64, number of iterations 400k, number of BLSTM units 256, number of learning rate decay steps 200k, initial learning rate 0.0001, learning rate reduction ratio 0.1, and beamwidth 100, the experimental data obtained achieved an accuracy of 80.3% on the IIIT5K dataset.

[0061] Compared with the prior art, the present invention improves the accuracy of irregular file detection and recognition by improving the two-stage algorithm separately. In the detection stage, the detection and recognition network based on character positioning proposed in the present invention avoids the uncertainty and weak generalization of the artificial anchor point design of regional regression detection, while avoiding the shortcomings of semantic segmentation detection that cannot effectively deal with dense text, thereby improving the accuracy of text instance detection. In the recognition stage, the text recognition network based on CRNN proposed in the present invention only adopts simple adaptive pre-correction for the located text instances, and avoids the defect of poor recognition accuracy caused by over-correction, and introduces Funnel ReLU activation function and CTC Loss to improve the performance of the recognition network in irregular text recognition and solve the problem of misalignment between sequence recognition results and annotations. In general, the present invention avoids the complex network design of the one-stage detection and recognition algorithm, and improves the existing defects of the two-stage algorithm, with better recognition readiness and generalization.

[0062] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. A flight instrument irregular text recognition method based on character positioning and CRNN, characterized in that: Using a selected character-level annotation dataset, we train the parameters of a detection network based on character positioning and a CRNN recognition network. We then use the trained detection network and CRNN recognition network to perform online text recognition on real-world flight instrument images containing irregular text. The detection network based on character positioning, i.e., the backbone network, includes: a feature extraction module using a hierarchical deep aggregation structure and a feature fusion module using an iterative deep aggregation algorithm, wherein: the feature extraction module extracts shallow features and deep features, and the feature fusion module fuses the shallow features and deep features, and outputs a character area heat map, a connection relationship heat map, a text area segmentation map, and a text center area segmentation map to the CRNN recognition network; The CRNN recognition network performs connected domain analysis on the superimposed mask of the character area, connection area and text area processed by the feature fusion module, and determines the boundary of each text instance through the minimum circumscribed rectangle of each character area and the direction information provided by the text center area; The boundary is obtained as follows: S1: Select a scene dataset with character-level annotations and annotate it according to the true value. Specifically, the rule for setting the true value of the text area is to calculate the ratio of the area after removing each vertex to the original area for each text instance with more than 4 vertices in the boundary annotation. , when , then remove the vertex from the original vertex set; the setting rule of the true value of the text center area is to determine the upper and lower long sides and the left and right short sides of the text, and then calculate the total length of the upper and lower boundaries of the two long sides, and divide it into 20 equal parts, recorded as , ;from arrive and arrive , connect the points in the upper and lower boundaries to form 15 line segments, obtain the midpoints of these 15 line segments, extend them in the positive and negative directions by 0.3 times the length of the line segments, and then connect the extended line segments; S2 reads the dataset image and predicts the character area, the connection area between characters, the text area, and the text center area through the backbone network; S3, determining the character area, connection area, and text center area using a preset threshold, connecting the character areas between each text instance, then superimposing the character area, connection area, and text center area, and performing connected domain analysis on the superimposed mask; S4, determine the boundary of each text instance through the minimum bounding rectangle of each character area and the direction information provided by the text center area. The specific process is as follows: each character area except the beginning and end of the same text instance The intersection points of each side of the minimum circumscribed rectangle of the text with the upper and lower boundaries of the text center area are marked in clockwise order as ; Find the line segment midpoint With line segment midpoint midpoint of the connecting line ; Find the minimum circumscribed rectangle sides and line segments Angle; if the difference between the vertical angles of the two sides is greater than the threshold , then len is the length of the side with the larger angle, otherwise len is the line segment The longest line segment in the vertical direction; passing through the center point Draw a line segment The vertical line, whose length is len, is the upper and lower boundary points .

2. The flight instrument irregular text recognition method based on character positioning and CRNN according to claim 1 is characterized in that: The CRNN recognition network includes: a rotation pre-correction module and a CRNN recognition module, wherein: the rotation pre-correction module calculates the minimum rotation circumscribed rectangle of each polygonal text instance area based on the text area polygon box from the detection network, and then obtains the text area rotated to the horizontal through rotation transformation. The CRNN recognition module extracts character features from the text area, and obtains the recognized irregular text content through feature prediction and decoding.

3. The flight instrument irregular text recognition method based on character positioning and CRNN according to claim 2 is characterized in that: The CRNN recognition module includes: a CNN character feature extraction unit, an RNN sequence feature prediction unit and a CTC decoding unit, wherein: the CNN character feature extraction unit inputs the corrected text area image into the ResNet50 network that introduces FReLU as the activation function to obtain a convolution feature sequence, the RNN sequence feature prediction unit obtains the sequence features in the predicted convolution features based on the convolution feature sequence extracted by CNN through BiLSTM prediction, and the CTC decoding unit decodes the input sequence features to obtain the recognized irregular text content.

4. The flight instrument irregular text recognition method based on character positioning and CRNN according to claim 1 is characterized in that: The parameter training specifically includes: the weights and bias values ​​of the excitation functions of each neuron in the detection network and the recognition network, as well as the learning rate.

5. The flight instrument irregular text recognition method based on character positioning and CRNN according to claim 1 is characterized in that: The online text recognition method involves cropping the flight instrument area to be identified from the actual flight instrument image containing irregular text, converting it into a grayscale image, filtering it to eliminate noise, and then inputting the trained detection network and CRNN recognition network to perform text detection and text recognition, and outputting the recognition results.

6. The flight instrument irregular text recognition method based on character positioning and CRNN according to any one of claims 1 to 5, characterized in that: After step S4, the method further includes: S5, adjusting the detection network parameters according to the calculated loss function; S6, pre-correcting the text instance region determined by the detection network, and performing rotation correction on the irregular text region to be horizontal, specifically: finding the minimum rotation circumscribed rectangle of each polygonal text instance region, and then rotating the region to be horizontal through a rotation transformation; S7, extracting corresponding features of characters for the text instance area to be recognized through CNN; S8, predicting sequence features in convolutional features through RNN; S9, decode the output of RNN through CTC; S10, calculate according to CTCLoss and adjust the parameters of the CRNN recognition network, specifically: ,in: is the number of images in the training set, is the true value for training, To predict the results; S11, repeat S2 to S10 until all the selected training dataset images are sent to the network for training.

Images