Environmental monitoring equipment image text recognition method and model based on neural network
By using the feature pyramid structure of residual attention convolution and hollow convolution in image text recognition in environmental monitoring equipment, combined with the correction network, the problems of inconsistent character areas, complex lighting changes and text angle tilt are solved, and the recognition accuracy is improved , suitable for complex industrial scenarios.
Patent Information
- Application Number
- CN202411020773.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-29
AI Technical Summary
In the image text recognition of environmental monitoring equipment, the prior art problems such as inconsistent character areas, complex lighting changes, uneven image quality and text angle tilt, resulting in low recognition accuracy and difficult to meet actual needs.
Using segmentation-based text detection network, the feature pyramid structure of residual attention convolution and hollow convolution is designed, and a correction network and SVTR sequence recognition network is combined to form an end-to-end network model, through multi-scale feature fusion and receptive field expansion. , solve small text areas and multi-scale problems, and integrate correction networks to deal with uneven lighting and blurring to improve recognition accuracy.
It improves the robustness and recognition accuracy of image text detection of environmental monitoring equipment, and can accurately identify text information in complex scenarios, which is suitable for actual industrial environments.
Smart Images

Figure CN118552945B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image text recognition, and in particular to an image text recognition method and model for environmental monitoring equipment based on a neural network. Background Art
[0002] In recent years, with the continuous development of industrialization and urbanization in my country, a large amount of industrial wastewater, waste gas and garbage have polluted the air, water resources and soil, destroyed the ecological balance, led to the reduction of biodiversity and the frequent occurrence of natural disasters, so the efficient operation and maintenance of environmental protection work has become increasingly important. Among them, the equipment in environmental protection work not only records its parameters, models and other important information, but also often feedback the operating status of the equipment itself. However, there are common problems such as data loss and low accuracy in the current equipment ledgers. Therefore, the use of image text recognition technology to quickly and accurately identify equipment text information can improve the efficiency of environmental protection equipment statistics, ledger verification and other work, thereby improving the equipment management level of the environmental protection system.
[0003] With the rapid development of image and text recognition technology based on neural networks, it is imperative to realize real-time recognition of text information of equipment in order to accelerate the realization of intelligent integrated operation of environmental protection sites. At present, the common means of identifying image and text information of environmental protection site equipment is regular manual inspection by operation and maintenance personnel, and judging whether there are abnormalities in the equipment through visual observation or ledger verification. This traditional inspection method relies too much on the concentration and responsibility of operation and maintenance personnel. If there are too many inspection points, omissions are inevitable.
[0004] Guo Maoqin, Xie Hongwei and others published a method for character detection and recognition in the nameplate of sewage treatment equipment in the journal Computer Engineering and Design with the publication number CN11-1775 / TP. In order to solve the problem of low recognition accuracy when identifying the nameplate information on sewage treatment equipment, this method uses the DenseNet+BLSTM+CTC algorithm to recognize the text sequence. However, when the text line is too long and multiple characters need to be recognized, the recognition effect is difficult to meet the on-site needs. In the real environmental monitoring station environment, due to the complexity of different equipment, there are often situations where the character area is not uniform, the lighting changes are complex, and the image quality is uneven. In addition, during the robot's photo shooting process, if it is affected by the external environment, the acquired image will also be tilted, resulting in a change in the angle of the text on the image. These characteristics make the existing algorithm unable to adapt to the scene well, and in actual industrial scenes, its accuracy is also difficult to meet the needs. Summary of the invention
[0005] In order to solve the above technical problems, the purpose of the present invention is to provide an environmental monitoring equipment image text recognition method and model based on neural network.
[0006] In a first aspect, the present invention provides a method for image text recognition of environmental monitoring equipment based on a neural network, the method comprising the following steps:
[0007] Get the image training dataset;
[0008] Constructing an image text recognition model, the image text recognition model includes a text detection module and a text recognition module, the text detection module is composed of a segmentation-based text detection network, and the segmentation-based text detection network includes a feature pyramid structure designed by residual attention convolution and void convolution;
[0009] Using an image training data set to train the image text recognition model to obtain a trained image text recognition model;
[0010] The trained image text recognition model is used to perform image text recognition.
[0011] In combination with the first aspect above, in some possible implementations, in the feature pyramid structure designed with residual attention convolution and dilated convolution, the input image is sequentially generated into four scale feature maps through the backbone network, and the four scale feature maps are composed of five feature maps. Mapping generation, Generating feature maps via residual attention convolution , and is fused with other features at lower levels through a top-down propagation pathway , obtained through residual attention convolution Generated feature map .
[0012] In combination with the first aspect above, in some possible implementations, in the feature pyramid structure designed with residual attention convolution and dilated convolution, the tree-shaped hierarchical structure provides receptive fields of different sizes for the input feature map through dilated convolution, and uses the add operation to fuse feature information from different branches. The dilated convolutions of different branches have the same convolution kernel and different expansion rates.
[0013] In combination with the above-mentioned first aspect, in some possible implementations, the different branches are 3 branches, the convolution kernel is 3×3, and the expansion rates of the 3 branches are 1, 3, and 5 respectively.
[0014] In combination with the above-mentioned first aspect, in some possible implementations, the segmentation-based text detection network is a DBNet network.
[0015] In combination with the above first aspect, in some possible implementations, the text recognition module includes a correction network and a text recognition network.
[0016] In combination with the above first aspect, in some possible implementations, the correction network includes a SPIN network and a correction structure.
[0017] In combination with the above first aspect, in some possible implementations, the text recognition network is a SVTR network.
[0018] In combination with the first aspect above, in some possible implementations, obtaining an image training data set includes:
[0019] Preprocess all collected real images to obtain preprocessed images;
[0020] Annotate the preprocessed image to obtain an annotated image;
[0021] Data augmentation is performed on the annotated image to obtain an augmented image, and the set of augmented images is used as an image training data set.
[0022] In the second aspect, the present invention also provides an image text recognition model based on a neural network, wherein the image text recognition model includes a text detection module and a text recognition module, wherein the text detection module is composed of a segmentation-based text detection network, and the segmentation-based text detection network includes a feature pyramid structure designed using residual attention convolution and hole convolution.
[0023] In a third aspect, the present invention further provides a computer program product, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.
[0024] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a computer program code. When the computer program code runs on a computer, the computer executes the method in the first aspect or any possible implementation of the first aspect.
[0025] The present invention has the following beneficial effects: the present invention adopts a feature pyramid structure designed by residual attention convolution and void convolution in the segmentation-based text detection network design, so as to solve the problem of small and multi-scale text areas in the image of environmental monitoring station equipment by multi-scale feature fusion and expanded receptive field, thereby improving the robustness of image text detection, and finally effectively improving the accuracy of image text detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0027] Figure 1 A basic flow chart of a method for image text recognition of environmental monitoring equipment based on a neural network according to an embodiment of the present invention;
[0028] Figure 2 A schematic diagram of the structure of an improved DBNet network according to an embodiment of the present invention;
[0029] Figure 3 A schematic diagram of the structure of a BackBone network according to an embodiment of the present invention;
[0030] Figure 4 A schematic diagram of the network structure of AD-PAN according to an embodiment of the present invention;
[0031] Figure 5 A schematic diagram of the network structure of the residual attention convolution RSE according to an embodiment of the present invention;
[0032] Figure 6 A schematic diagram of a tree-shaped hierarchical structure according to an embodiment of the present invention;
[0033] Figure 7 A schematic diagram of the structure of a SPIN network composed of SPN and AIN according to an embodiment of the present invention;
[0034] Figure 8 A schematic diagram of the network structure of an SVTR according to an embodiment of the present invention;
[0035] Fig. 9 Schematic diagram of the network structure of the correction structure of an embodiment of the present invention. DETAILED DESCRIPTION
[0036] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.
[0037] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.
[0038] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0039] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0040] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more". In the description of the present invention, unless otherwise specified, "multiple" refers to two or more than two, and other quantifiers are similar; "at least one item", "one or more items" or similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one item a can represent any number of a; for another example, one or more items of a, b and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple; "and / or" is a kind of association relationship that describes the associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.
[0041] Although operations or steps are described in a specific order in the drawings in the embodiments of the present invention, it should not be understood that it is required to perform these operations or steps in the specific order shown or in a serial order, or that all the operations or steps shown must be performed to obtain the desired results. In the embodiments of the present invention, these operations or steps may be performed in series; these operations or steps may also be performed in parallel; or some of these operations or steps may be performed.
[0042] At the same time, it is understood that the data involved in the technical solution of the present invention (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the relevant laws, regulations and relevant provisions. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meanings as those generally understood by technicians in the technical field of the present invention, and all parameters or indicators in the formulas involved in the present invention are normalized values that eliminate the influence of dimensions.
[0043] In order to solve the problem in the prior art that the text recognition of environmental monitoring equipment images is not accurate due to the problems of non-uniform character areas, complex illumination changes, uneven image quality, and tilted angles, the embodiment of the present invention provides an environmental monitoring equipment image text recognition method and model based on a neural network. An AD-PAN (Attention and Dilation ConvolutionsPath Aggregation Network) feature pyramid structure is designed by using residual attention convolution and dilation convolution on the basis of a DBNet network, so as to solve the problems of small text areas and multi-scales of environmental monitoring station equipment images through multi-scale feature fusion and expanded receptive field, improve the robustness of equipment image text detection, and thus improve the accuracy of equipment image text recognition; at the same time, the correction network and the SVTR sequence recognition network are integrated to form an end-to-end network for model training, so as to overcome the problems of uneven illumination, blur, and tilted text angles in the character area, and further improve the accuracy of equipment image text recognition.
[0044] A method for image text recognition of environmental monitoring equipment based on a neural network provided by an embodiment of the present invention will be described in detail below in conjunction with the accompanying drawings.
[0045] Figure 1 A basic flow chart of a neural network-based environmental monitoring equipment image text recognition method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the method specifically comprises the following steps:
[0046] Step S1: Obtain image training data set.
[0047] Collect image training data sets of environmental monitoring station equipment, specifically, collect images of real environmental monitoring station equipment. The images can be inspected by a rail-mounted robot equipped with a pan-tilt platform according to a predetermined inspection route, and the network camera collects on-site image data to obtain. Preprocess the collected images, and use image annotation tools to annotate all preprocessed images. The preprocessing process refers to the process of filtering images using a filtering algorithm. The filtering algorithm can be reasonably selected as needed, such as Gaussian filtering, average filtering, or median filtering. Considering that the number of collected images may be small and it is difficult to meet the data volume requirements for model training, relevant images can also be collected from the Internet, or corresponding data augmentation can be performed, such as random cropping, rotation, flipping, contrast enhancement, etc. Finally, the images with data augmentation are divided into training set, verification set, and test set, and the division ratio can be reasonably set as needed. In the embodiment of the present invention, the PPOCRLabel annotation tool of PaddleOCR open source is used to annotate the images, and the division ratio is 8:1:1, that is, 80% of the images are used as the training data set of the model, 10% of the images are used as the verification data set of the model to adjust the model parameters and hyperparameters, and 10% of the images are used as the test data set of the model to evaluate the model detection accuracy and false detection rate, so as to objectively describe the detection ability of the model. These training sets, verification sets and test sets constitute the image training data set of the environmental monitoring station equipment.
[0048] Step S2: construct an image text recognition model, wherein the image text recognition model includes a text detection module and a text recognition module, wherein the text detection module is composed of a segmentation-based text detection network, and the segmentation-based text detection network includes a feature pyramid structure designed using residual attention convolution and hole convolution.
[0049] Figure 2 The schematic diagram of the structure of the improved DBNet network is shown. The improved DBNet network uses residual attention convolution and dilated convolution to design an AD-PAN feature pyramid structure based on the existing DBNet algorithm, so as to solve the problem of small and multi-scale text areas in environmental monitoring station equipment images by multi-scale feature fusion and expanded receptive field, and further improve the robustness of equipment image text detection. The improved DBNet network consists of BackBone network, AD-PAN network and Head network.
[0050] Among them, the BackBone network uses the industry-mature Res2Net50 network for feature extraction, which consists of 5 levels, namely Stage 1, Stage 2, Stage 3, Stage 4, and Stage 5. Figure 3 A schematic diagram of the structure of the BackBone network is shown.
[0051] The main function of the AD-PAN network is to perform path enhancement and aggregation on image features, thereby making it easier to propagate the underlying positioning information. Figure 4 The network structure diagram of AD-PAN is shown. In the AD-PAN network, the input image generates four scale feature maps through the backbone network. These four scale feature maps have a total of 5 features. The mapping is generated, i.e. , Mapping generates a scale feature map, , Mapping generates a scale feature map, , Mapping generates a scale feature map, The mapping generates a scale feature map; in order to solve the problem of insufficient feature expression of the backbone network output, the residual squeeze excitation (RSE) mechanism is introduced, and the convolution layer in the horizontal link is replaced by the residual attention convolution; Generating feature maps via residual attention convolution , and is fused with other features at lower levels through a top-down propagation pathway ;use To represent the corresponding The newly generated feature map, here and Exactly the same. In addition, PANet (Path Aggregation Network) shortens the information path between the bottom-level and top-level features.
[0052] Figure 5 Shows Figure 3 The network structure of the residual attention convolution RSE in the AD-PAN network structure. Since the sizes of the convolution kernels are different when upsampling and downsampling in the AD-PAN network structure, the i-th scale feature map Perform 1×1 convolution to obtain the same channel dimension (256); for the i-th feature map A 3×3 convolution is performed, and the number of channels is also 256. The feature map then passes through a 1×1 convolution layer, a ReLU activation layer, a 1×1 convolution layer, and a sigmoid activation layer in sequence to generate corresponding spatial weights for each feature map. The generated weight map and the feature map after merging the channels are subjected to a Hadamard product operation, and the obtained features are added to the input feature map to obtain the final feature map. Since the feature map has rich multi-scale contextual information, it alleviates the problem of information loss caused by the reduction in the number of channels to a certain extent.
[0053] In AD-PAN network, use Figure 6 The tree-like hierarchical structure shown in the figure is used to integrate multi-scale and contextual information. The specific implementation process is as follows: the tree-like hierarchical structure uses dilated convolution and then BN+ReLU to provide receptive fields of different sizes for the input feature map. The role of BN is to standardize the samples, and then ReLU is used for activation. The add operation is used to fuse the feature information from the three branches to improve the multi-scale image prediction ability. Among them, the dilated convolutions in the three parallel branches have the same convolution kernel size, but different expansion rates. Specifically, the kernel of each dilated convolution is 3×3, and the expansion rates of different branches are They are 1, 3, and 5 respectively. Multi-branch dilated convolution feature fusion can help the model effectively extract the edge features of the target and enhance the clarity and accuracy of the boundary, which is critical for the positioning of small targets.
[0054] The head network cascades the features extracted by the AD-PAN network, and then uses a 3×3 convolution and two deconvolution operations with a stride of 2 to upsample to generate a probability map and a threshold map. The probability map represents the probability that each pixel belongs to the text category, and the threshold map represents the dynamic setting of the threshold of each pixel. Through the differentiable binarization operation, the binary segmentation result is obtained to obtain an approximate binary map; finally, the box formation operation is performed to obtain the final text detection result.
[0055] The improved correction recognition network consists of SPN (Structure Preserving Network), AIN (Auxiliary Inner-offset Network), correction structure and SVTR. Figure 7 The diagram shows the structure of the SPIN (Structure-Preserving Inner Offset Network) network composed of SPN and AIN. Figure 8 A schematic diagram of the network structure of SVTR is shown; Fig. 9 The diagram of the network structure of the correction structure is shown. SPN can handle the color distortion problem caused by the dispersion of text pattern brightness due to light or high temperature in character images; AIN is an auxiliary network that can be used to distinguish the color shift problem caused by noise patterns such as character blur in character images; the correction structure first further scales the image, and then uses the Localization Network to extract the Control Points , and then use the prior knowledge to get the corrected Control Points ,use and Construct the transformation matrix based on the position relationship .
[0056] In SPN, given an input image , represents the image set, let represents the transformed image, represents the transformed image set, then to Transformation The definition is as follows:
[0057] ;
[0058] In the formula, and Represents the coordinate points on the input and output images The brightness on.
[0059] The specific formula of SPN image conversion is as follows:
[0060] ;
[0061] In the formula, and They represent the partial weights and feature extractor of the last fully connected layer in SPN respectively; Represents the output of the last fully connected layer dimensional feature vector; Represents the exponent of the ith elementary power function.
[0062] Since SPN does not take into account that the brightness of the noise pattern may be similar to that of the text pattern, it may cause pattern confusion. Color shift on To alleviate pattern confusion, AIN first divides the image into small blocks and then predicts the offset value of each block. It should be noted that all offset values are activated by the sigmoid function and mapped to the size of the input image by a general upsampling operation. In AIN, SPN will generate the following color offset at each coordinate point:
[0063] ;
[0064] In the formula, Represents some weights of the last fully connected layer in SPN; Represents the weight parameter of AIN; represents a feature extractor; represents the Hadamard product; represents a learnable update gate that receives signals from SPN and perceives the difficulty of different tasks while taking into account the input image Balance with predicted color shift; represents the input image; represents the updated image. With the help of AIN, the updated image undergoes the following enhancement transformation:
[0065] ;
[0066] In the formula, represents the corresponding weight of the i-th; Represents the exponent of the basic power function.
[0067] Using Localization Network to implicitly learn Control Points ,in Indicates The coordinates of the control points are similarly preset. Control Points , and define the TPS transformation matrix ,in ; as well as Both represent matrices Then according to and The linear mapping relationship calculates the transformation matrix, where , Represents the kernel function, which is used to calculate the Euclidean distance between pixel points ci and ck:
[0068] ;
[0069] In the formula, ; Indicates The horizontal coordinates of the control points; ; Indicates The ordinate coordinates of the control points; and Both represent boundary values.
[0070] So we can get:
[0071] 𝐶 = ;
[0072] 𝐶 = ;
[0073] In the formula, ; = ; It represents the result of calculating the Euclidean distance between pixels ci and cj as the input of the kernel function.
[0074] From the above formula we can get = .
[0075] Therefore, for Each pixel in , we can get the pixel coordinates after the correction direction:
[0076] .
[0077] The SVTR algorithm is a character recognition network based on transformer encoding and CTC (Connectionist Temporal Classification) decoding introduced in PaddleOCRV3. It obtains multi-scale features through multi-level step-by-step downsampling, and uses self-attention mechanism and multi-layer perceptron to process and fuse features. The character image of the SPIN network composed of SPN and AIN is first extracted through the Patch Embedding layer, and the feature information of the character image is gradually transmitted and aggregated through the multi-layer Mix Block stacking. It should be noted that the Merging downsampling operation will be performed in the MixBlock stacking to reduce the height of the feature map and reduce the computational complexity. MixBlock is divided into two parts: Local Mix Block and Global Mix Block. Local Mix Block performs self-attention calculation through a sliding window to encode character morphological features and related features between different parts of the character; while Global Mix Block is a global self-attention mechanism used to evaluate the dependency between all character features and weaken the influence of non-character features. Finally, the fully connected layer outputs the character features processed by Mix Block as a feature sequence. Furthermore, during the CTC decoding process, the consecutively repeated labels in the feature sequence are merged into a single label, and the blank labels are removed, and finally the character sequence with the highest score is selected as the output.
[0078] Step S3: using the image training data set to train the image text recognition model to obtain a trained image text recognition model.
[0079] Load the image text dataset obtained in the above step S1 and the image text recognition model constructed in step S2 on the training server with GPU server, adjust the size of the images in the training set and the validation set to 640×640, set the learning rate to 0.00001, and the input batch size to 8. The optimization algorithm uses the Adam optimizer, uses the gradient descent method, sets the maximum number of iterations to 1200, inputs the adjusted images to the image text recognition model, and performs multiple rounds of iterative training on the image text recognition model on the training server with GPU. During the iterative training process, the parameters of the model are continuously iteratively updated through the reverse gradient propagation and gradient descent method of the image text recognition model, so that the loss function value of the model inference result and the actual annotation result of the training data gradually decreases and tends to be stable. After the iterative training is completed, the size of the images in the test set of the image text dataset is uniformly adjusted to 640×640, and the adjusted images are input into the image text recognition model obtained after the training is completed. The improved DBNet network outputs the text detection area, and then the text area is cropped as the input of the improved correction recognition network. The output is the predicted text result, and finally the image text recognition model test is completed.
[0080] Step S4: Use the trained image text recognition model to perform image text recognition.
[0081] The final image text recognition model and the corresponding algorithm are deployed on the industrial computer, and the software interacts with the algorithm through the MQTT protocol. When image text recognition is required for the environmental monitoring station equipment, the image to be recognized is sent to the algorithm, and the algorithm calls the image text recognition model to perform image text recognition on the image to be recognized, and returns the recognition result to the software through the MQTT protocol. The software then interacts with the platform to output the result image and display it on the monitor of the main control computer.
[0082] The embodiment of the present invention provides an environmental monitoring equipment image text recognition method based on a neural network. On the one hand, it can accurately identify situations when the image text has inconsistent character areas, complex lighting changes, uneven image quality, and tilted text angles; on the other hand, it can be deployed to the on-site industrial computer in a timely manner without making any changes to the on-site equipment. The overall detection process adopts non-contact detection, which is fast and accurate, and the detection process is an end-to-end process. The detection results can be displayed in real time on the display screen during the rail-hanging robot inspection task, and the computer can also be remotely controlled to view the results.
[0083] Based on the same inventive concept, the present invention also provides an image text recognition model based on a neural network, the image text recognition model includes a text detection module and a text recognition module, the text detection module is composed of a segmentation-based text detection network, and the segmentation-based text detection network includes a feature pyramid structure designed using residual attention convolution and hole convolution. Since the specific network structure of the image text recognition model has been similarly introduced in the above-mentioned neural network-based environmental monitoring equipment image text recognition method, it will not be repeated here.
[0084] Based on the same inventive concept, an embodiment of the present invention also provides a computer program product, which includes: computer program code, when the computer program code runs on a computer, the computer executes any one of the environmental monitoring equipment image text recognition methods based on neural network introduced above.
[0085] Based on the same inventive concept, an embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program code. When the computer program code runs on a computer, the computer executes any one of the environmental monitoring equipment image text recognition methods based on neural network introduced above.
[0086] It should be noted that the above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for image text recognition of environmental monitoring equipment based on neural network, characterized in that: The method comprises the following steps: Get the image training dataset; Constructing an image text recognition model, the image text recognition model includes a text detection module and a text recognition module, the text detection module is composed of a segmentation-based text detection network, and the segmentation-based text detection network includes a feature pyramid structure designed by residual attention convolution and void convolution; Using an image training data set to train the image text recognition model to obtain a trained image text recognition model; Performing image text recognition using the trained image text recognition model; In the feature pyramid structure designed with residual attention convolution and dilated convolution, the input image is sequentially generated into four scale feature maps through the backbone network, and the four scale feature maps are composed of five feature Mapping generation, Generating feature maps via residual attention convolution , and is fused with other features at lower levels through a top-down propagation pathway , obtained through residual attention convolution Generated feature map ; In the feature pyramid structure designed with residual attention convolution and dilated convolution, the tree-shaped hierarchical structure provides receptive fields of different sizes for the input feature map through dilated convolution, and uses the add operation to fuse feature information from different branches. The dilated convolutions of different branches have the same convolution kernel and different expansion rates. The text recognition module includes a correction network and a text recognition network; the correction network includes a SPIN network and a correction structure; In SPN, given an input image , represents a set of images, let represents the transformed image, represents the transformed image set, then to Transformation The definition is as follows: ; In the formula, and Represents the coordinate points on the input and output images respectively Brightness on; The specific formula of SPN image conversion is as follows: ; In the formula, and They represent the partial weights and feature extractor of the last fully connected layer in SPN respectively; Represents the output of the last fully connected layer dimensional feature vector; represents the exponent of the ith basic power function; AIN at each coordinate point Color shift on Specifically, AIN first divides the image into small blocks and then predicts the offset value of each block; all offset values are activated by the sigmoid function and mapped to the size of the input image by a general upsampling operation; in AIN, SPN will generate the following color offset at each coordinate point: ; In the formula, Represents some weights of the last fully connected layer in SPN; Represents the weight parameter of AIN; represents a feature extractor; represents the Hadamard product; represents a learnable update gate; represents the input image; represents the updated image; with the assistance of AIN, the updated image undergoes the following enhancement transformation: ; In the formula, represents the corresponding weight of the i-th; Represents the exponent of the basic power function; Using Localization Network to implicitly learn Control Points ,in Indicates The coordinates of the control points are similarly preset. Control Points , and define the TPS transformation matrix ,in ; as well as Both represent matrices The numerical variable of and The linear mapping relationship calculates the transformation matrix, where , Represents the kernel function, which is used to calculate the Euclidean distance between pixel points ci and ck: ; In the formula, ; Indicates The horizontal coordinates of the control points; ; Indicates The ordinate coordinates of the control points; and Both represent boundary values; You can get: 𝐶= ; 𝐶= ; In the formula, ; = ; It represents the result of calculating the Euclidean distance between pixel points ci and cj as the input of the kernel function; From the above formula we can get = ; for Each pixel in , get the pixel coordinates after the correction direction: 。 2. According to the neural network-based environmental monitoring equipment image text recognition method of claim 1, it is characterized in that: There are three different branches, the convolution kernel is 3×3, and the expansion rates of the three branches are 1, 3, and 5 respectively.
3. The method for image text recognition of environmental monitoring equipment based on neural network according to claim 1 is characterized in that: The segmentation-based text detection network is a DBNet network.
4. The method for image text recognition of environmental monitoring equipment based on neural network according to claim 1 is characterized in that: The text recognition network is a SVTR network.
5. The method for image text recognition of environmental monitoring equipment based on neural network according to claim 1 is characterized in that: Obtaining an image training dataset includes: Preprocess all collected real images to obtain preprocessed images; Annotate the preprocessed image to obtain an annotated image; Data augmentation is performed on the annotated image to obtain an augmented image, and the set of augmented images is used as an image training data set.
6. A neural network-based image text recognition model, characterized in that: The image text recognition model includes a text detection module and a text recognition module. The text detection module is composed of a segmentation-based text detection network. The segmentation-based text detection network includes a feature pyramid structure designed with residual attention convolution and hole convolution. In the feature pyramid structure designed with residual attention convolution and dilated convolution, the input image is sequentially generated into four scale feature maps through the backbone network, and the four scale feature maps are composed of five feature Mapping generation, Generating feature maps via residual attention convolution , and is fused with other features at lower levels through a top-down propagation pathway , obtained through residual attention convolution Generated feature map ; In the feature pyramid structure designed with residual attention convolution and dilated convolution, the tree-shaped hierarchical structure provides receptive fields of different sizes for the input feature map through dilated convolution, and uses the add operation to fuse feature information from different branches. The dilated convolutions of different branches have the same convolution kernel and different expansion rates. The text recognition module includes a correction network and a text recognition network; the correction network includes a SPIN network and a correction structure; In SPN, given an input image , represents a set of images, let represents the transformed image, represents the transformed image set, then to Transformation The definition is as follows: ; In the formula, and Represents the coordinate points on the input and output images respectively Brightness on; The specific formula of SPN image conversion is as follows: ; In the formula, and They represent the partial weights and feature extractor of the last fully connected layer in SPN respectively; Represents the output of the last fully connected layer dimensional feature vector; represents the exponent of the ith basic power function; AIN at each coordinate point Color shift on Specifically, AIN first divides the image into small blocks and then predicts the offset value of each block; all offset values are activated by the sigmoid function and mapped to the size of the input image by a general upsampling operation; in AIN, SPN will generate the following color offset at each coordinate point: ; In the formula, Represents some weights of the last fully connected layer in SPN; Represents the weight parameter of AIN; represents a feature extractor; represents the Hadamard product; represents a learnable update gate; represents the input image; represents the updated image; with the assistance of AIN, the updated image undergoes the following enhancement transformation: ; In the formula, represents the corresponding weight of the i-th; Represents the exponent of the basic power function; Using Localization Network to implicitly learn Control Points ,in Indicates The coordinates of the control points are similarly preset. Control Points , and define the TPS transformation matrix ,in ; as well as Both represent matrices The numerical variable of and The linear mapping relationship calculates the transformation matrix, where , Represents the kernel function, which is used to calculate the Euclidean distance between pixel points ci and ck: ; In the formula, ; Indicates The horizontal coordinates of the control points; ; Indicates The ordinate coordinates of the control points; and Both represent boundary values; You can get: 𝐶= ; 𝐶= ; In the formula, ; = ; It represents the result of calculating the Euclidean distance between pixel points ci and cj as the input of the kernel function; From the above formula we can get = ; for Each pixel in , get the pixel coordinates after the correction direction: 。
Citation Information
Patent Citations
Image local feature description method based on pixel location arrangement column diagrams
CN103295014A
Scene text detection method and system based on multi-scale feature pyramid
CN113673506A
Pedestrian target detection method, system and device and storage medium
CN115100594A
Medical image segmentation method based on multiple attention fusion
CN116152492A
Invoice text recognition method and system based on multi-scale feature fusion
CN116189199A
