Alarm method, device and equipment based on character bar recognition, medium and program product
Through the combination of deep convolutional neural network and convolutional long and short-term memory network, the problem of difficult character bar recognition in the existing technology is solved, efficient recognition of character bars and text content is achieved, and the accuracy of early warning is improved.
Patent Information
- Application Number
- CN202410236995.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-01
- Publication Date
- 2024-11-08
AI Technical Summary
Existing alarm methods based on character bar recognition cannot effectively recognize text when the character bar is blocked, the background color is not red, or the character bar color is close to the background color, and the training process is slow, and the text content in the character bar cannot be effectively recognized. .
Character bar recognition is used to use pre-trained deep convolutional neural network model, combined with dense convolutional neural network and convolutional long short-term memory neural network for text detection, and used Poisson loss function and time classification algorithm for text recognition, and through position correction Improve identification accuracy.
It improves the recognition rate of character bars and text content, improves the accuracy of character bar warning, can effectively identify multi-directional character bars in natural environment, and improves recognition efficiency in natural environment.
Smart Images

Figure CN118918593A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text recognition, and in particular to an alarm method, device, equipment, medium and program product based on character strip recognition. Background Art
[0002] At present, conventional alarm methods based on character strip recognition include: video-based alarm methods based on color threshold and corner point detection and alarm methods based on character strip recognition based on YOLO deep learning model; the former extracts "overview" features such as color and corner points of character strips, and cannot effectively recognize the text in the character strips when the character strips are blocked, the background color is not red, or the color is close to the background color. The latter uses a preset matching algorithm to calculate the intersection over union (IOU) of the intersection and union between the "predicted bounding box" and the "real bounding box". When the IoU value between the predicted bounding box and the real bounding box is low, the gradient of the loss function may be small, resulting in a slow training process; the character strips are classified and judged by judging whether the category confidence of the object to be tested meets the preset character strip category confidence requirements, and the content of the character strips is not effectively recognized and analyzed. Summary of the invention
[0003] In view of the problems existing in the prior art, the embodiments of the present invention provide an alarm method, device, equipment, medium and program product based on character bar recognition, which can improve the recognition rate of character bars and the text content therein, thereby improving the accuracy of warning of character bars containing target content.
[0004] In a first aspect, an embodiment of the present invention provides an alarm method based on character bar recognition, comprising:
[0005] Perform character strip recognition on the target image to be recognized by using a pre-trained character strip recognition model to obtain a character strip feature map;
[0006] Performing character detection on the character bar feature map using a preset character bar text detection model to obtain a text sequence in the character bar feature map;
[0007] Performing text recognition on the text sequence through a preset text recognition model to obtain a text label sequence corresponding to the text sequence;
[0008] When there is a preset target word or sentence in the text label sequence, an alarm prompt is issued.
[0009] As an improvement of the above-mentioned scheme, the character bar recognition model is a deep convolutional neural network model constructed based on several layers of convolutional layers, several layers of convolutional blocks, several upsampling layers, several layers of feature splicing layers and an output layer; wherein the feature splicing layer is used to splice the features of the output of the previous layer and the features of the output of a set layer above the previous layer; the upsampling layer is used to perform double upsampling processing on the features of the output of the previous layer.
[0010] As an improvement of the above solution, the convolution layer adopts a PReLU activation function, and the output layer adopts a Poisson loss function.
[0011] As an improvement of the above solution, the training of the character strip recognition model includes the following steps:
[0012] Collect multiple sample images with character bar information;
[0013] Marking the position of the character bar information on each of the sample images to obtain a label image of the corresponding sample image;
[0014] Recombining and classifying the sample images and their corresponding label images to construct a training data set and a verification data set;
[0015] Using the training data set to perform model training on a pre-built character strip recognition model;
[0016] The trained character stripe recognition model is verified using the verification data set, and when the character stripe recognition model meets a preset model verification condition, a trained character stripe recognition model is obtained.
[0017] As an improvement of the above solution, the character strip feature map is subjected to character detection by a preset character strip text detection model to obtain a text sequence in the character strip feature map, including:
[0018] Performing character detection on the character bar feature map by using the character bar text detection model to determine a text area in the character bar feature map;
[0019] Using a text line construction algorithm to splice each character in the text area to obtain a text sequence in the character bar feature map;
[0020] The text sequence is marked with a text box according to the anchor point of the text sequence.
[0021] As an improvement of the above solution, the character bar text detection model includes a dense convolutional neural network, a first convolutional long short-term memory neural network and a first fully connected layer;
[0022] The performing character detection on the character bar feature map by using the character bar text detection model to determine the text area in the character bar feature map includes:
[0023] Performing character extraction on the input character bar feature map through the dense convolutional neural network to obtain a character sequence;
[0024] Extracting context information from the character sequence using the first convolutional long short-term memory neural network to obtain context features of the character sequence;
[0025] The first fully connected layer performs text classification judgment on the context features of the character sequence to determine the text area of the character bar feature map.
[0026] As an improvement of the above solution, the text recognition model includes a convolutional neural network, a second convolutional long short-term memory neural network, a feature sequence extraction layer and a second fully connected layer; wherein the second fully connected layer adopts a time classification algorithm loss function;
[0027] The performing text recognition on the text sequence by using a preset text recognition model to obtain a text label sequence corresponding to the text sequence includes:
[0028] Performing feature extraction on the input text sequence through the convolutional neural network to obtain a feature map;
[0029] The feature map is converted into a feature vector sequence through a feature sequence extraction layer;
[0030] Learning each feature vector in the feature vector sequence by using the second convolutional long short-term memory neural network to obtain a predicted label distribution of the feature vector sequence;
[0031] The predicted label distribution is converted into the text label sequence through the second fully connected layer.
[0032] As an improvement of the above solution, before performing character detection on the character bar feature map by using a preset character bar text detection model to obtain a text sequence in the character bar feature map, the method further includes:
[0033] Performing position correction on the character bar feature map.
[0034] In a second aspect, an embodiment of the present invention provides an alarm device based on character bar recognition, comprising:
[0035] A character strip recognition module, used to perform character strip recognition on a target image to be recognized by using a pre-trained character strip recognition model to obtain a character strip feature map;
[0036] A character detection module, used to perform character detection on the character bar feature map by using a preset character bar text detection model to obtain a text sequence in the character bar feature map;
[0037] A text recognition model, used to perform text recognition on the text sequence through a preset text recognition model to obtain a text label sequence corresponding to the text sequence;
[0038] The alarm module is used to issue an alarm prompt when a preset target word or sentence exists in the text label sequence.
[0039] In a third aspect, an embodiment of the present invention provides an alarm device based on character bar recognition, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the alarm method based on character bar recognition as described in any one of the first aspects.
[0040] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the alarm method based on character strip recognition as described in any one of the first aspects.
[0041] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the alarm method based on character strip recognition as described in any one of the first aspects.
[0042] Compared with the prior art, an alarm method, device, equipment, medium and program product based on character bar recognition in an embodiment of the present invention performs character bar recognition on a target image to be recognized through a pre-trained character bar recognition model to obtain a character bar feature map; performs character detection on the character bar feature map through a preset character bar text detection model to obtain a text sequence in the character bar feature map; performs text recognition on the text sequence through a preset text recognition model to obtain a text label sequence corresponding to the text sequence; and issues an alarm prompt when a preset target word or phrase exists in the text label sequence. The embodiment of the present invention can improve the recognition rate of character bars and the text content therein, and enhance the accuracy of warnings for character bars containing target content. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solution of the present invention, the drawings used in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0044] Figure 1 is a flow chart of an alarm method based on character bar recognition provided by an embodiment of the present invention;
[0045] Figure 2 is a schematic diagram of a target image provided by an embodiment of the present invention;
[0046] Figure 3 is a character strip feature map output by the character strip recognition model provided by an embodiment of the present invention;
[0047] Figure 4 is a character bar feature map after position correction provided by an embodiment of the present invention;
[0048] Figure 5 is a curve diagram of the accuracy of the character bar recognition model and the number of training times during the training process provided by the embodiment of the present invention;
[0049] Figure 6 The embodiment of the present invention provides Figure 5 The curve of mean average degree corresponding to medium precision and number of training times;
[0050] Figure 7 is a curve diagram of the recall rate and the number of training times of the character bar recognition model during the training process provided by the embodiment of the present invention;
[0051] Figure 8 The embodiment of the present invention provides Figure 7 The curve of the mean average degree corresponding to the recall rate and the number of training times;
[0052] Fig. 9 It is a structural block diagram of an alarm device based on character bar recognition provided by an embodiment of the present invention;
[0053] Fig.10 It is a structural block diagram of an alarm device based on character bar recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] See also Figure 1 , Figure 1 1 is a flow chart of an alarm method based on character bar recognition provided by an embodiment of the present invention. The alarm method based on character bar recognition specifically includes:
[0056] S1: Perform character strip recognition on the target image to be recognized through the pre-trained character strip recognition model to obtain a character strip feature map;
[0057] S2: performing character detection on the character bar feature map using a preset character bar text detection model to obtain a text sequence in the character bar feature map;
[0058] S3: performing text recognition on the text sequence through a preset text recognition model to obtain a text label sequence corresponding to the text sequence;
[0059] S4: When there is a preset target word or sentence in the text tag sequence, an alarm is issued.
[0060] It should be noted that the alarm method based on character strip recognition described in the embodiment of the present invention can be executed by a server, a computer, or a computing device with data and image processing functions. In the embodiment of the present invention, a character strip recognition model, a character strip text detection model, and a text recognition model are pre-built; wherein the character strip recognition model is built based on a deep convolutional neural network model, the character strip text detection model is built based on a dense convolutional neural network and a convolutional long short-term memory network, and the text recognition model is built based on a convolutional neural network and a convolutional long short-term memory network.
[0061] For example, Figure 2 The target image to be recognized is input into a pre-trained character strip recognition model. The character strip recognition model uses an anchor frame to mark the character strip position of the input target image and determines the character strip anchor frame in the target image. The position information of the character strip anchor frame is determined by the pixel coordinates of the target image. Then, the target image is cropped according to the position information of the character strip anchor frame to obtain a character strip feature map, such as Figure 3 As shown, the character bar recognition model then outputs the character bar feature map.
[0062] Furthermore, before performing character detection on the character bar feature map by using a preset character bar text detection model to obtain a text sequence in the character bar feature map, the method further includes:
[0063] Performing position correction on the character bar feature map.
[0064] The embodiment of the present invention uses a perspective transformation function to perform position correction on the character strip feature map output by the character strip recognition model, so as to correct the tilted and deflected character strips into positive character strips, reduce the error between the character strips in the captured target image and the character strips in the actual scene, improve the readability and recognizability of the character strip content, and intuitively display the character strip information as an input variable of the text detection model.
[0065] For example, first select the pixel coordinates corresponding to the vertices of the character strip anchor box in the character strip feature map; then use the getPerspectiveTransform function in the opencv tool to perform matrix transformation on the pixel coordinates of the selected vertices; then use the warpPerspective function in the opencv (Open computer Vision Library) to perform perspective transformation on the transformed matrix to obtain the character strip feature map after position correction, such as Figure 4 It should be noted that opencv is a cross-platform open source computer vision and machine learning software library. Opencv and the getPerspectiveTransform function and warpPerspective function configured therein belong to the existing technology in the field of image processing and will not be described in detail here.
[0066] Specifically, the character bar recognition model is a deep convolutional neural network model constructed based on several layers of convolutional layers, several layers of convolutional blocks, several upsampling layers, several layers of feature splicing layers and an output layer; wherein the feature splicing layer is used to splice the features of the output of the previous layer and the features of the output of a set layer above the previous layer; the upsampling layer is used to perform double upsampling processing on the features of the output of the previous layer.
[0067] The convolution layer adopts the PReLU activation function, and the output layer adopts the Poisson loss function.
[0068] It should be noted that, in the embodiment of the present invention, the number of convolutional layers, convolutional blocks, upsampling layers, and feature splicing layers in the character strip recognition model is not specifically limited and can be set as needed. It is understandable that, in the character strip recognition model, one convolutional layer is used as the input layer, and the remaining convolutional layers and convolutional blocks are connected in sequence in a layer-by-layer manner, and the last convolutional block is connected to the output layer; the upsampling layer is connected between adjacent convolutional layers and convolutional blocks, and the feature splicing layer is connected between the upsampling layer and the convolutional block at the next layer, or between adjacent convolutional layers and convolutional blocks.
[0069] Exemplarily, the network structure of the character bar recognition model is as follows:
[0070] The first convolutional layer: 32 output channels, 6×6 convolution kernel, stride 2, padding 2, activation function PReLU;
[0071] The second convolutional layer: 64 output channels, 3×3 convolution kernel, stride 2, activation function PReLU;
[0072] The third convolution block: 1 convolution block, 64 output channels, including 3 convolution layers, and the convolution kernels are 1×1, 3×3, and 1×1 respectively;
[0073] The fourth convolutional layer: 128 output channels, 3×3 convolution kernel, stride 2, activation function PReLU;
[0074] The fifth convolution block: 2 convolution blocks, 128 output channels, each convolution block contains 3 convolution layers, and the convolution kernel sizes are 1×1, 3×3, and 1×1 respectively;
[0075] The sixth convolutional layer: 256 output channels, 3×3 convolution kernel, stride 2, activation function PReLU;
[0076] The seventh convolution block: 3 convolution blocks, 256 output channels, each convolution block contains 3 convolution layers, and the convolution kernel sizes are 1×1, 3×3, and 1×1 respectively;
[0077] The eighth convolutional layer: 512 output channels, 3×3 convolution kernel, stride 2, activation function PReLU;
[0078] The ninth convolution block: 1 convolution block, 512 output channels, including 3 convolution layers, and the convolution kernel sizes are 1×1, 3×3, and 1×1 respectively;
[0079] Tenth convolutional layer: 512 output channels, pooling layer size is 5×5, spatial pyramid pooling;
[0080] Eleventh convolutional layer: 256 output channels, 1×1 convolution kernel, stride 1, activation function PReLU;
[0081] The 12th upsampling layer: 256 output channels, two-fold upsampling, nearest neighbor method;
[0082] The thirteenth feature concatenation layer: concatenates the features of the seventh and twelfth layers;
[0083] The 14th convolution block: 1 convolution block, 256 output channels, including 3 convolution layers, and the convolution kernels are 1×1, 3×3, and 1×1 respectively;
[0084] The fifteenth convolutional layer: 128 output channels, 1×1 convolution kernel, stride 1, activation function PReLU (Parametric Rectified Linear Unit, ReLU with parameters);
[0085] The 16th upsampling layer: 128 output channels, double upsampling, nearest neighbor method;
[0086] The seventeenth feature concatenation layer: concatenates the features of the fifth and sixteenth layers;
[0087] The eighteenth convolution block: 1 convolution block, 128 output channels, including 3 convolution layers, and the convolution kernels are 1×1, 3×3, and 1×1 respectively;
[0088] The nineteenth convolutional layer: 128 output channels, 3×3 convolution kernel, stride 2, activation function PReLU.
[0089] The 20th feature concatenation layer: concatenates the features of the 15th and 19th layers;
[0090] The 21st convolution block: 1 convolution block, 256 output channels, including 3 convolution layers, and the convolution kernels are 1×1, 3×3, and 1×1 respectively;
[0091] The 22nd convolutional layer: 256 output channels, 1×1 convolution kernel, stride 1, activation function PReLU.
[0092] The 23rd feature splicing layer: the feature splicing of the 11th and 22nd layers;
[0093] The 24th convolution block: 1 convolution block, 512 output channels, including 3 convolution layers, and the convolution kernels are 1×1, 3×3, and 1×1 respectively;
[0094] The 25th output layer: Use convolution to modify the output channel to 255, and output the feature results of three sampling ratios, with sizes of 80, 40, and 20 respectively. The target loss function is the Poisson loss function.
[0095] In an embodiment of the present invention, the depth and breadth of the deep convolutional neural network structure are fully considered when constructing a character bar recognition model to enhance the model's powerful learning ability. At the same time, the PReLU activation function and the Poisson function are introduced. Compared with the prior art recognition method using general features such as the color and corner points of the character bar, the embodiment of the present invention uses the above-mentioned character bar recognition model to effectively identify various character bar situations in a natural environment (for example, the character bar is blocked, the background color of the character bar is not red, or the color of the character bar is close to the background color, etc.), thereby improving the recognition rate of character bars in multiple directions such as horizontal, vertical, and oblique directions in a natural environment.
[0096] Furthermore, the training of the character bar recognition model includes the following steps:
[0097] Collect multiple sample images with character bar information;
[0098] Marking the position of the character bar information on each of the sample images to obtain a label image of the corresponding sample image;
[0099] Recombining and classifying the sample images and their corresponding label images to construct a training data set and a verification data set;
[0100] Using the training data set to perform model training on a pre-built character strip recognition model;
[0101] The trained character stripe recognition model is verified using the verification data set, and when the character stripe recognition model meets a preset model verification condition, a trained character stripe recognition model is obtained.
[0102] For example, firstly, sample images with character stripe information are searched through collection engines such as X.de, XX headlines, and Xying; and a character stripe picture set is constructed based on the searched sample images; wherein the character stripe picture set includes a number of sample images of different sizes and different character stripe inclinations or deflection directions; then, the labelimg tool is used to annotate the position of the character stripe information of each sample image in the character stripe picture set, and a label image carrying the character stripe position annotation information is obtained; then, a sample image and its corresponding label image are used as a training sample, and the sample image and its corresponding label image are reorganized and classified to construct a training data set and a verification data set, which are respectively used for model training of the character stripe recognition model and evaluation of model parameters and performance. Model training conditions and model verification conditions can be pre-set, for example, the model training conditions include the number of training times reaching a preset number threshold, or the model accuracy reaching a preset precision threshold; the model verification conditions include the model's character bar recognition rate being greater than a preset recognition rate threshold, or the model's character bar recognition rate average value being greater than a preset recognition rate threshold, or the model's horizontal character bar recognition rate, vertical character bar recognition rate, and oblique character bar recognition rate are respectively greater than the first recognition rate threshold, the second recognition rate threshold, and the third recognition rate threshold. In an embodiment of the present invention, there is no specific limitation on the values of the number threshold, the precision threshold, and the recognition rate threshold, and the user can set them as needed, for example, the number threshold is 500 times, the precision threshold is 0.9, the first recognition rate threshold is 90%, the second recognition rate threshold is 80%, and the third recognition rate threshold is 60%.
[0103] The model is trained 500 times based on the training data, and the accuracy and mean average of the model are obtained as follows: Figure 5 , Figure 6 As shown in the figure, the recall rate of the model and its mean average are as follows Figure 7 , Figure 8As shown. It can be seen that after 500 trainings, the model's precision and its mean average, recall and its mean average have reached a high level, meeting the model's accuracy requirements, thereby determining the optimal model parameters. After completing 500 trainings, the model was verified using a validation data set, and the recognition rate of horizontal character strips was 92%, the recognition rate of vertical character strips was 86% and 82%, and the recognition rate of oblique character strips was 68%. It can be seen that the trained character strip recognition model can effectively recognize character strips in multiple directions such as horizontal, vertical, and oblique in the image.
[0104] Specifically, performing character detection on the character bar feature map by using a preset character bar text detection model to obtain a text sequence in the character bar feature map includes:
[0105] Performing character detection on the character bar feature map by using the character bar text detection model to determine a text area in the character bar feature map;
[0106] Using a text line construction algorithm to splice each character in the text area to obtain a text sequence in the character bar feature map;
[0107] The text sequence is marked with a text box according to the anchor point of the text sequence.
[0108] Further, the character bar text detection model includes a dense convolutional neural network, a first convolutional long short-term memory neural network and a first fully connected layer;
[0109] The performing character detection on the character bar feature map by using the character bar text detection model to determine the text area in the character bar feature map includes:
[0110] Performing character extraction on the input character bar feature map through the dense convolutional neural network to obtain a character sequence;
[0111] Extracting context information from the character sequence using the first convolutional long short-term memory neural network to obtain context features of the character sequence;
[0112] The first fully connected layer performs text classification judgment on the context features of the character sequence to determine the text area of the character bar feature map.
[0113] Exemplarily, a position-corrected character strip feature map is input into the character strip text detection model; in the character strip text detection model: a dense convolutional neural network is used to extract characters from the input character strip feature map to obtain a character sequence and input the character sequence into a first convolutional long short-term memory neural network; wherein, in the dense convolutional neural network, the network is divided into a plurality of dense blocks, and the feature maps in each block are of the same size; the first convolutional long short-term memory neural network is used to calculate the correlation between the input of the current time step and the hidden layer of the previous time step, and context feature extraction is performed on the input character sequence based on the calculated correlation to obtain context feature of the character sequence; the context feature of the character sequence output by the first convolutional long short-term memory neural network is subjected to text classification judgment through the first fully connected layer, and characters with classification scores lower than a preset score threshold are judged as non-text, and the anchor frame area corresponding to the non-text is removed from the character sequence to obtain the text area and output it. It should be noted that the character strip text detection model uses anchor frames to mark the positions of each character in the input character strip feature map to obtain the anchor frames of each character. The position information of the anchor points at the four corners of the anchor frame is determined according to the pixel coordinates of the image.
[0114] For the text area output by the character bar text detection model, a text line construction algorithm is used to splice the text areas according to the anchor points at the four top corners of the anchor boxes of each character in the text area, so as to splice together the anchor points with similar distances, similar heights, and similar vertical positions of the center points to form a text sequence; then, the text sequence is marked as a text box according to the anchor points at the four top corners of the text sequence to complete the character bar text detection.
[0115] In an embodiment of the present invention, a character strip text detection model is pre-built based on a dense convolutional neural network (DenseNet) and a convolutional long short-term memory neural network (ConvLSTM). By extracting features through a dense convolutional neural network, the gradient vanishing problem can be alleviated and the propagation of features between networks can be enhanced; by performing convolution operations through a convolutional long short-term memory neural network to capture spatial features, a more accurate text box can be obtained, thereby improving the accuracy of text area recognition in a character strip feature map.
[0116] Specifically, the text recognition model includes a convolutional neural network, a second convolutional long short-term memory neural network, a feature sequence extraction layer, and a second fully connected layer; wherein the second fully connected layer adopts a time classification algorithm loss function;
[0117] The performing text recognition on the text sequence by using a preset text recognition model to obtain a text label sequence corresponding to the text sequence includes:
[0118] Performing feature extraction on the input text sequence through the convolutional neural network to obtain a feature map;
[0119] The feature map is converted into a feature vector sequence through a feature sequence extraction layer;
[0120] Learning each feature vector in the feature vector sequence by using the second convolutional long short-term memory neural network to obtain a predicted label distribution of the feature vector sequence;
[0121] The predicted label distribution is converted into the text label sequence through the second fully connected layer.
[0122] Exemplarily, the above-mentioned text sequence marked with a text box is input into the text recognition model; in the text recognition model: a convolutional neural network (CNN) is used to extract character features of the input text sequence to obtain a feature map; wherein the convolutional neural network adopts a PReLU activation function; a feature map to sequence model (Map-to-Sequence) is used to extract a feature vector sequence from the feature map; a second convolutional long short-term memory neural network is used to learn each feature vector in the feature vector sequence to obtain a predicted label distribution of the feature vector sequence, that is, a real character distribution corresponding to the feature vector sequence; the predicted label distribution output by the second convolutional long short-term memory neural network is converted into the text label sequence and output through a second fully connected layer, wherein the second fully connected layer adopts a connectionist temporal classification (CTC) loss function to convert a series of predicted label distributions obtained from the recurrent layer of the second convolutional long short-term memory neural network into a final text label sequence.
[0123] Finally, the obtained text label sequence is compared and matched with the target words and sentences in the preset database; the target words and sentences can be preset target content; when any target word and sentence exists in the text label sequence, it indicates that the target content exists in the character bar in the target image, and an alarm is issued in the form of text message or email.
[0124] In an embodiment of the present invention, an improved CRNN model is pre-built based on a convolutional neural network and a convolutional long short-term memory neural network ConvLSTM to obtain the text recognition model, which can realize end-to-end recognition of text label sequences of indefinite length without cutting individual characters first, and converting text recognition into a time-dependent sequence, thereby improving the accuracy of horizontal text recognition.
[0125] Compared with the prior art, the embodiment of the present invention adopts a deep convolutional neural network based on the PReLU activation function and the poission loss function to perform character strip recognition on the input image, which can improve the recognition of character strips in multiple directions such as horizontal, vertical, and oblique in a natural environment, and improve the recognition rate and efficiency of character strips in a natural environment; based on the use of DenseNet+ConvLSTM to build a character strip text detection model, by extracting the contextual features of a large range of characters in the character strip, the text detection in the character strip image is realized. Compared with the traditional OCR recognition algorithm, it can be better applied to the detection and recognition of large font image text such as character strips, and improve the accuracy of character strip text detection; the text recognition model based on CNN+Map-to-Sequence+ConvLSTM is built to realize semantic analysis of the text, improve the recognition rate of the character strip text content, thereby improving the accuracy of the warning of the character strip containing the target content, and avoiding the character strip containing the target content.
[0126] See also Fig. 9 , Fig. 9 The present invention provides a structural block diagram of an alarm device based on character bar recognition, wherein the alarm device based on character bar recognition includes:
[0127] A character strip recognition module 1 is used to perform character strip recognition on a target image to be recognized by using a pre-trained character strip recognition model to obtain a character strip feature map;
[0128] A character detection module 2, used to perform character detection on the character bar feature map by using a preset character bar text detection model to obtain a text sequence in the character bar feature map;
[0129] A text recognition model 3 is used to perform text recognition on the text sequence through a preset text recognition model to obtain a text label sequence corresponding to the text sequence;
[0130] The alarm module 4 is used to issue an alarm prompt when a preset target word or sentence exists in the text label sequence.
[0131] In an optional embodiment, the character bar recognition model is a deep convolutional neural network model constructed based on several layers of convolutional layers, several layers of convolutional blocks, several upsampling layers, several layers of feature splicing layers and an output layer; wherein the feature splicing layer is used to splice the features of the output of the previous layer and the features of the output of a set layer located above the previous layer; the upsampling layer is used to perform double upsampling processing on the features of the output of the previous layer.
[0132] In an optional embodiment, the convolution layer adopts a PReLU activation function, and the output layer adopts a Poisson loss function.
[0133] In an optional embodiment, the device further:
[0134] A sample image acquisition module, used for acquiring multiple sample images with character bar information;
[0135] An image annotation module is used to annotate the position of character strip information on each sample image to obtain a label image of the corresponding sample image;
[0136] An image reorganization and classification module is used to reorganize and classify the sample images and their corresponding label images to construct a training data set and a verification data set;
[0137] A model training module, used to perform model training on a pre-built character strip recognition model using the training data set;
[0138] The model verification module is used to perform model verification on the trained character stripe recognition model using the verification data set, and obtain the trained character stripe recognition model when the character stripe recognition model meets the preset model verification conditions.
[0139] In an optional embodiment, the character detection module 2 includes:
[0140] A character detection unit, configured to perform character detection on the character bar feature map by using the character bar text detection model, and determine a text area in the character bar feature map;
[0141] A character splicing unit, used for splicing each character in the text area by using a text line construction algorithm to obtain a text sequence in the character bar feature map;
[0142] The text box marking unit is used to mark the text sequence with a text box according to the anchor point of the text sequence.
[0143] In an optional embodiment, the character bar text detection model includes a dense convolutional neural network, a first convolutional long short-term memory neural network, and a first fully connected layer;
[0144] The character detection unit comprises:
[0145] A character feature extraction subunit, used for performing character extraction on the input character bar feature map through the dense convolutional neural network to obtain a character sequence;
[0146] a context feature extraction subunit, configured to extract context information from the character sequence through the first convolutional long short-term memory neural network to obtain context features of the character sequence;
[0147] The text classification subunit is used to perform text classification judgment on the context features of the character sequence through the first fully connected layer to determine the text area of the character bar feature map.
[0148] In an optional embodiment, the text recognition model includes a convolutional neural network, a second convolutional long short-term memory neural network, a feature sequence extraction layer, and a second fully connected layer; wherein the second fully connected layer adopts a time classification algorithm loss function;
[0149] The text recognition model 3 includes:
[0150] A feature map extraction unit, used for performing feature extraction on the input text sequence through the convolutional neural network to obtain a feature map;
[0151] A feature sequence conversion unit, used for converting the feature map into a feature vector sequence through a feature sequence extraction layer;
[0152] a label distribution prediction unit, configured to learn each feature vector in the feature vector sequence through the second convolutional long short-term memory neural network to obtain a predicted label distribution of the feature vector sequence;
[0153] A label sequence conversion unit is used to convert the predicted label distribution into the text label sequence through the second fully connected layer.
[0154] In an optional embodiment, the device further includes:
[0155] A position correction module is used to perform position correction on the character bar feature map.
[0156] It should be noted that the working process of each module in the alarm device based on character bar recognition described in the embodiment of the present invention can refer to the working process of the alarm method based on character bar recognition described in the above embodiment, and the technical effect achieved is also the same as the alarm method based on character bar recognition described in the above embodiment, which will not be repeated here.
[0157] See also Fig.10 , Fig.101 is a structural block diagram of an alarm device based on character strip recognition provided by an embodiment of the present invention. The alarm device based on character strip recognition includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above-mentioned alarm method embodiments based on character strip recognition are implemented, such as steps S1 to S4.
[0158] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the alarm device based on character bar recognition.
[0159] The alarm device based on character bar recognition may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of an alarm device based on character bar recognition and does not constitute a limitation on the alarm device based on character bar recognition. The alarm device based on character bar recognition may include more or fewer components than shown in the diagram, or may combine certain components, or different components. For example, the alarm device based on character bar recognition may also include input and output devices, network access devices, buses, etc.
[0160] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the alarm device based on character bar recognition, and uses various interfaces and lines to connect various parts of the entire alarm device based on character bar recognition.
[0161] The memory 22 can be used to store the computer program and / or module. The processor 21 realizes various functions of the alarm device based on character bar recognition by running or executing the computer program and / or module stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0162] Wherein, if the module / unit integrated in the alarm device based on character bar recognition is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, the steps of the above-mentioned method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0163] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.
[0164] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, many improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. An alarm method based on character bar recognition, characterized in that: include: Perform character strip recognition on the target image to be recognized by using a pre-trained character strip recognition model to obtain a character strip feature map; Performing character detection on the character bar feature map using a preset character bar text detection model to obtain a text sequence in the character bar feature map; Performing text recognition on the text sequence through a preset text recognition model to obtain a text label sequence corresponding to the text sequence; When there is a preset target word or sentence in the text label sequence, an alarm prompt is issued.
2. The alarm method based on character bar recognition as claimed in claim 1, characterized in that: The character strip recognition model is a deep convolutional neural network model constructed based on several convolution layers, several convolution blocks, several upsampling layers, several feature splicing layers and an output layer; wherein the feature splicing layer is used to splice the features of the output of the previous layer and the features of the output of a set layer above the previous layer; the upsampling layer is used to perform double upsampling processing on the features of the output of the previous layer.
3. The warning method based on character strip recognition as claimed in claim 2, characterized in that: The convolution layer adopts the PReLU activation function, and the output layer adopts the Poisson loss function.
4. The alarm method based on character strip recognition as claimed in any one of claims 1 to 3, characterized in that: The training of the character strip recognition model comprises the following steps: Collect multiple sample images with character bar information; Marking the position of the character bar information on each of the sample images to obtain a label image of the corresponding sample image; Recombining and classifying the sample images and their corresponding label images to construct a training data set and a verification data set; Using the training data set to perform model training on a pre-built character strip recognition model; The trained character stripe recognition model is verified using the verification data set, and when the character stripe recognition model meets a preset model verification condition, a trained character stripe recognition model is obtained.
5. The alarm method based on character strip recognition as claimed in claim 1, characterized in that: The step of performing character detection on the character bar feature map by using a preset character bar text detection model to obtain a text sequence in the character bar feature map includes: Performing character detection on the character bar feature map by using the character bar text detection model to determine a text area in the character bar feature map; Using a text line construction algorithm to splice each character in the text area to obtain a text sequence in the character bar feature map; The text sequence is marked with a text box according to the anchor point of the text sequence.
6. The warning method based on character strip recognition as claimed in claim 5, characterized in that: The character strip text detection model includes a dense convolutional neural network, a first convolutional long short-term memory neural network and a first fully connected layer; The performing character detection on the character bar feature map by using the character bar text detection model to determine the text area in the character bar feature map includes: Performing character extraction on the input character bar feature map through the dense convolutional neural network to obtain a character sequence; Extracting context information from the character sequence using the first convolutional long short-term memory neural network to obtain context features of the character sequence; The first fully connected layer performs text classification judgment on the context features of the character sequence to determine the text area of the character bar feature map.
7. The alarm method based on character strip recognition as claimed in claim 1, characterized in that: The text recognition model includes a convolutional neural network, a second convolutional long short-term memory neural network, a feature sequence extraction layer and a second fully connected layer; wherein the second fully connected layer adopts a time classification algorithm loss function; The performing text recognition on the text sequence by using a preset text recognition model to obtain a text label sequence corresponding to the text sequence includes: Performing feature extraction on the input text sequence through the convolutional neural network to obtain a feature map; The feature map is converted into a feature vector sequence through a feature sequence extraction layer; Learning each feature vector in the feature vector sequence by using the second convolutional long short-term memory neural network to obtain a predicted label distribution of the feature vector sequence; The predicted label distribution is converted into the text label sequence through the second fully connected layer.
8. The alarm method based on character strip recognition as claimed in claim 1, characterized in that: Before performing character detection on the character bar feature map by using a preset character bar text detection model to obtain a text sequence in the character bar feature map, the method further includes: Performing position correction on the character bar feature map.
9. An alarm device based on character bar recognition, characterized in that: include: A character strip recognition module, used to perform character strip recognition on a target image to be recognized by using a pre-trained character strip recognition model to obtain a character strip feature map; A character detection module, used to perform character detection on the character bar feature map by using a preset character bar text detection model to obtain a text sequence in the character bar feature map; A text recognition model, used to perform text recognition on the text sequence through a preset text recognition model to obtain a text label sequence corresponding to the text sequence; The alarm module is used to issue an alarm prompt when a preset target word or sentence exists in the text label sequence.
10. An alarm device based on character bar recognition, characterized in that: include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the warning method based on character strip recognition as described in any one of claims 1 to 8 when executing the computer program.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the alarm method based on character strip recognition as described in any one of claims 1 to 8.
12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the alarm method based on character strip recognition described in any one of claims 1 to 8 is implemented.
