A multi-modal meter single identification method and device

Abstract

The application provides a multi-modal electric power form recognition method and device. In the execution of the method, first, the image of the electric power form to be recognized is obtained, then the image of the electric power form to be recognized is preprocessed to obtain a preprocessed image, then the CTPN neural network model is used to extract the text line position information of the preprocessed image to obtain the position information of the text line of the preprocessed image; and based on the position information of the text line, the CRNN model is used to recognize the preprocessed image to obtain the recognition result of the preprocessed image; finally, the multi-modal feature information extraction algorithm is used to extract the recognition result of the preprocessed image, and the multi-modal feature information is taken as the recognition result of the electric power form to be recognized. The application can automatically recognize the multi-modal feature information of the electric power form with the nature of the table in multiple scenes, solve the problem of low work efficiency of manual extraction, improve the electric power form recognition efficiency, and has strong robustness.

Description

Technical Field

[0001] This application relates to the field of image recognition, and in particular to a method and apparatus for recognizing multimodal power form data. Background Technology

[0002] Due to the long-standing structural staff shortage at the grassroots level in the power industry, there is an insufficient number of personnel in key grassroots positions and the workload is complicated. The test reports of the operation and maintenance department, which are work orders, all need to be manually identified and entered into the business system. Manual entry is not only demanding, but also has a large workload and is highly repetitive. It cannot be efficiently converted into digital files, and the power form recognition efficiency is low.

[0003] Currently, most traditional template matching methods require the creation and adaptation of templates for different scenarios, which is cumbersome and not robust enough. Summary of the Invention

[0004] In view of this, this application provides a multimodal electricity form recognition method and apparatus, which aims to solve the problem of low efficiency in manual data extraction, improve the efficiency of electricity form recognition, and has strong robustness.

[0005] In a first aspect, embodiments of this application provide a multimodal electricity form recognition method, the method comprising:

[0006] Obtain an image of the electricity form to be identified;

[0007] The image of the power form to be identified is preprocessed to obtain a preprocessed image;

[0008] The text line position information of the preprocessed image is extracted using a pre-trained CTPN neural network model to obtain the position information of the text lines in the preprocessed image.

[0009] Based on the position information of the text lines, the pre-trained CRNN model is used to recognize the pre-processed image to obtain the recognition result of the pre-processed image;

[0010] A multimodal feature information extraction algorithm is used to extract the recognition results of the preprocessed image to obtain the corresponding multimodal feature information, and the multimodal feature information is used as the recognition result of the power form to be recognized.

[0011] Optionally, the preprocessing of the image of the electricity form to be identified includes:

[0012] The image of the power form to be identified is subjected to grayscale transformation based on the weighted value method to obtain the grayscale image corresponding to the power form to be identified.

[0013] The grayscale image is binarized using a binarization function to obtain the corresponding binarized image.

[0014] Optionally, after obtaining the binarized image corresponding to the grayscale image, the method further includes:

[0015] The binarized image is subjected to orientation detection to obtain the detection result;

[0016] Based on the detection results, the position of the binarized image is corrected.

[0017] Optionally, the pre-trained CTPN neural network model includes a VGG16 network layer and a fully connected layer;

[0018] The step of extracting text line position information from the preprocessed image using a pre-trained CTPN neural network model to obtain the position information of text lines in the preprocessed image includes:

[0019] The preprocessed image is input into the VGG16 network layer for feature extraction to generate a first feature map.

[0020] A sliding window operation is performed on the first feature map to generate a second feature map;

[0021] The second feature map is extracted using the fully connected layer to obtain the feature vector corresponding to the second feature map;

[0022] The feature vector is input into the three branches of the fully connected layer for prediction to obtain the vertical coordinate regression, classification score, and horizontal translation amount corresponding to the feature vector.

[0023] After converting the vertical coordinate regression, classification score, and horizontal translation, the position information of the text lines in the preprocessed image is obtained.

[0024] Optionally, the pre-trained CRNN model includes convolutional layers and RNN recurrent layers;

[0025] The recognition result of the preprocessed image obtained by using a pre-trained CRNN model based on the position information of the text line includes:

[0026] The convolutional layer extracts the preprocessed image based on the position information of the text line to obtain the feature information corresponding to the preprocessed image;

[0027] The RNN recurrent layer is used to predict the feature information corresponding to the preprocessed image to obtain the corresponding prediction probability distribution sequence;

[0028] The predicted probability distribution sequence is transformed using the RNN recurrent layer to obtain the corresponding label sequence;

[0029] The labeled sequence is used as the recognition result of the preprocessed image.

[0030] Optionally, the step of using a multimodal feature information extraction algorithm to extract the recognition result of the preprocessed image to obtain the corresponding multimodal feature information, and using the multimodal feature information as the recognition result of the power form to be recognized, includes:

[0031] The multimodal feature information extraction algorithm is used to extract the recognition results of the preprocessed image to obtain the corresponding multimodal feature information;

[0032] A multimodal attention algorithm is used to associate and fuse the multimodal feature information to obtain multimodal target feature information. The multimodal target feature information is used as the recognition result of the power form image to be recognized. The multimodal target feature information includes the text content of the power form image to be recognized and the category attribute corresponding to the text content.

[0033] Secondly, embodiments of this application provide a multimodal electricity form recognition device, the device comprising:

[0034] The acquisition module is used to acquire images of the electricity form to be identified;

[0035] The preprocessing module is used to preprocess the image of the power form to be identified to obtain a preprocessed image;

[0036] The first extraction module is used to extract text line position information from the preprocessed image using a pre-trained CTPN neural network model, so as to obtain the position information of the text lines in the preprocessed image.

[0037] The recognition module is used to recognize the preprocessed image based on the position information of the text line using a pre-trained CRNN model, and obtain the recognition result of the preprocessed image;

[0038] The second extraction module is used to identify the preprocessed image based on the position information of the text line using a pre-trained CRNN model, and obtain the recognition result of the preprocessed image.

[0039] Optionally, the preprocessing module includes a grayscale processing submodule and a binarization processing submodule;

[0040] The grayscale processing submodule is used to perform grayscale transformation processing on the image of the power form to be identified based on the weighted value method to obtain the grayscale image corresponding to the power form to be identified.

[0041] The binarization processing submodule is used to perform binarization processing on the grayscale image to obtain the binarized image corresponding to the grayscale image.

[0042] Optionally, after obtaining the binarized image corresponding to the grayscale image, the apparatus further includes:

[0043] The correction module is used to perform orientation detection on the binarized image and obtain the detection result;

[0044] Based on the detection results, the position of the binarized image is corrected.

[0045] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods described above.

[0046] The above technical solution has the following beneficial effects:

[0047] This application provides a multimodal electricity form recognition method and apparatus. When executing the method, firstly, an image of the electricity form to be recognized is acquired; then, the image of the electricity form to be recognized is preprocessed to obtain a preprocessed image; next, a pre-trained CTPN neural network model is used to extract text line position information from the preprocessed image to obtain the position information of the text lines in the preprocessed image; based on the position information of the text lines, a pre-trained CRNN model is used to recognize the preprocessed image to obtain the recognition result of the preprocessed image; finally, a multimodal feature information extraction algorithm is used to extract the recognition result of the preprocessed image to obtain the corresponding multimodal feature information, and the multimodal feature information is used as the recognition result of the electricity form to be recognized. In this way, this application uses a pre-trained CTPN neural network model to extract the position information of text lines in the image of the electricity form to be identified. Based on the position information of the text lines, a pre-trained CRNN model is used to identify the text information of the preprocessed image. A multimodal feature information extraction algorithm is used to extract the recognition results of the preprocessed image to obtain the corresponding multimodal feature information. This can automatically identify the multimodal feature information of table-like electricity forms in multiple scenarios, solve the problem of low efficiency of manual data extraction, improve the efficiency of electricity form recognition, and has strong robustness. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart of one method of the multimodal power form recognition method provided in this application embodiment;

[0050] Figure 2 This is a schematic diagram of a multimodal power form recognition device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0052] The relevant terminology in this application embodiment is described as follows:

[0053] CTPN Neural Network Model: A neural network model for detecting text in natural images with connectionist text proposal networks (CTPN). This model primarily focuses on accurately locating lines of text in images. Its basic approach involves directly generating a series of appropriately sized text proposals (pre-selected boxes) on the feature map obtained through convolution to detect text lines.

[0054] CRNN model: short for Convolutional Recurrent Neural Network, is mainly used for end-to-end recognition of text sequences of variable length. It does not require the first segmentation of individual characters, but transforms text recognition into a time-dependent sequence learning problem, which is image-based sequence recognition.

[0055] Multimodal: In the field of artificial intelligence, this often refers to the collaboration of perceived information, such as images, text, and voice, to help artificial intelligence understand the external world more accurately.

[0056] To facilitate understanding of the technical solutions provided in this application, the background technology involved in this application will be explained below.

[0057] The inventors discovered through research that due to the long-term structural shortage of personnel at the grassroots level in the power industry, there is an insufficient number of personnel in key grassroots positions and the tasks are complicated. The test reports of the operation and maintenance department, as work orders, all require manual identification of the content and manual operation of the business system to complete the data entry. Manual data entry is not only demanding, but also has a large workload and is highly repetitive. It cannot be efficiently converted into digital files, and the power form recognition efficiency is low.

[0058] Furthermore, traditional template-based matching methods require the creation and adaptation of templates for different scenarios, which is cumbersome, not robust enough, and unable to solve recognition problems for different forms, page layouts, and formats in power test reports.

[0059] To overcome the above-mentioned technical problems, this application provides a multimodal electricity form recognition method. The method of this application can be executed by a multimodal electricity form recognition device, which can be implemented by software and / or hardware, and can generally be integrated into a server or terminal device.

[0060] Please see Figure 1 , Figure 1 A flowchart of a multimodal power form recognition method provided in this application embodiment, the method may include:

[0061] Step S101: Obtain an image of the power form to be identified.

[0062] In this embodiment of the application, an image of the power form to be identified is first obtained to facilitate subsequent preprocessing of the image of the power form to be identified.

[0063] Step S102: Preprocess the image of the power form to be identified to obtain the preprocessed image.

[0064] In one possible implementation, the preprocessing of the image of the electricity form to be identified includes: performing grayscale transformation on the image of the electricity form to be identified based on a weighted method to obtain a grayscale image corresponding to the electricity form to be identified; and performing binarization processing on the grayscale image using a binarization function to obtain a binarized image corresponding to the grayscale image.

[0065] Specifically, in this embodiment of the application, the image of the power form to be identified is processed by grayscale transformation based on the weighted value method to obtain the grayscale image corresponding to the power form to be identified.

[0066] The specific method is to perform grayscale transformation on the image based on the weighted value method, where the weights of red (R), green (G), and blue (B) are weighted and averaged according to the ratios of 0.299, 0.587, and 0.144 respectively to obtain a reasonable grayscale image. The formula is Gray (grayscale value) = R*0.299 + G*0.587 + B*0.144.

[0067] After obtaining the grayscale image corresponding to the electricity form to be identified, the grayscale image is binarized using a binarization function to obtain the corresponding binarized image.

[0068] Specifically, the binarization function is:

[0069]

[0070] Among them, e x,y f represents the pixel value at coordinates (x, y) in the processed image. x,y is the original pixel value at coordinates (x, y) in the image, and T is the binarization threshold.

[0071] It should be noted that binarization can reduce the dimensionality of the data, eliminate the interference of noise in the original image, highlight the outline of the effective area, and significantly improve the accuracy of recognition.

[0072] In one possible implementation, after obtaining the binarized image corresponding to the grayscale image, the method further includes: performing orientation detection on the binarized image to obtain a detection result; and correcting the position of the binarized image based on the detection result.

[0073] Since the process of capturing images of the electricity meter to be identified involves manual operation, the acquired images will inevitably have some degree of tilt. Therefore, after obtaining the binarized image corresponding to the grayscale image, orientation detection is performed on the binarized image to obtain the detection result; based on the detection result, the position of the binarized image is corrected.

[0074] Specifically: the image tilt angle is determined by finding the angle at which the maximum projection value is found through fixed-direction projection superposition. This specifically refers to: for a binary image matrix I(k... y ,j x ) matrix coordinates (k y ,j x Using (N,1) as the origin, establish a Cartesian coordinate system, and transform the matrix coordinates into Cartesian coordinates X = j x Y = Nk y Establish (k) relative to the X-axis θ y ,j xThe Cartesian coordinates of the image are projected onto the X' axis (X', Y'). The projection values ​​are statistically analyzed, with θ ranging from 0 to 90 degrees. The angle θ corresponding to the maximum projection value is found; 90 - θ is the matrix image I(k). y ,j x The required counterclockwise rotation angle essentially involves tracking the intersection of curves corresponding to each point in the image. If the number of curves intersecting at a single point exceeds a threshold, then the parameter pair represented by this intersection point can be considered as a straight line in the original image.

[0075] Step S103: Use a pre-trained CTPN neural network model to extract text line position information from the preprocessed image to obtain the position information of the text lines in the preprocessed image.

[0076] Specifically, after obtaining the preprocessed image through step S102, the pre-trained CTPN neural network model is used to extract the text line position information of the preprocessed image, thereby obtaining the position information of the text lines in the preprocessed image.

[0077] It should be noted that the training process of the CTPN neural network model is as follows:

[0078] A training set for training the CTPN neural network model is obtained, comprising multiple historical electricity data sheets. Specifically, multiple historical electricity data sheets are first collected, then labeled using the labelimg annotation tool to obtain an XML-formatted dataset, which is then converted into the VOCdevkit dataset.

[0079] Then, the CTPN neural network was built using Python combined with the Tensorflow framework, OpenCV and other third-party toolkits. Finally, the CTPN neural network model was trained using the dataset.

[0080] In one possible implementation, the pre-trained CTPN neural network model includes a VGG16 network layer and a fully connected layer.

[0081] The step of extracting text line position information from the preprocessed image using a pre-trained CTPN neural network model to obtain the position information of text lines in the preprocessed image includes:

[0082] The preprocessed image is input into the VGG16 network layer for feature extraction to generate a first feature map; a sliding window operation is performed on the first feature map to generate a second feature map; the second feature map is extracted using the fully connected layer to obtain the feature vector corresponding to the second feature map; the feature vector is input into the three branches of the fully connected layer for prediction to obtain the vertical coordinate regression, classification score, and horizontal translation amount corresponding to the feature vector; after converting the vertical coordinate regression, classification score, and horizontal translation amount, the position information of the text lines in the preprocessed image is obtained.

[0083] Specifically, the preprocessed image is input into the VGG16 network layer of a pre-trained CTPN neural network model for feature extraction, generating a first feature map. Then, a series of sliding window operations are performed on the first feature map to generate a second feature map. Next, the second feature map is input into the fully connected layer of the pre-trained CTPN neural network model for extraction, obtaining the corresponding feature vector. This feature vector is then input into the three branches of the fully connected layer in the pre-trained CTPN neural network model for prediction, obtaining the vertical coordinate regression, classification score, and horizontal translation amount corresponding to the feature vector. Finally, the vertical coordinate regression, classification score, and horizontal translation amount are converted to obtain the position information of the text lines in the preprocessed image.

[0084] In practical applications, the VGG16 network layer consists of a W*H*C conv5 feature map. A 3*3 spatial window slides across the feature map of the last convolutional layer (VGG16's conv5). The sequential windows in each row are cyclically connected by a BLSTM (bi-directional long short-term memory). The convolutional features (3*3*C) of each window are used as the input to the BLSTM, which then implements a bidirectional BLSTM to enhance the learning of information from associated sequences. Finally, the feature map output from the last convolutional layer of VGG16 is converted into a vector form to obtain the feature vector.

[0085] Step S104: Based on the position information of the text line, the pre-trained CRNN model is used to recognize the pre-processed image to obtain the recognition result of the pre-processed image.

[0086] Specifically, after obtaining the position information of the text lines in the preprocessed image through step S103, the preprocessed image is recognized based on the position information of the text lines using a pre-trained CRNN model to obtain the recognition result of the preprocessed image.

[0087] It should be noted that the training process of the CRNN model is as follows:

[0088] Obtain a training set, which includes text characters from historical electricity forms. The data is generated based on the invoice character information of historical electricity forms. Use the training set to train the CRNN model.

[0089] In one possible implementation, the pre-trained CRNN model includes convolutional layers and recurrent RNN layers; the step of using the pre-trained CRNN model to recognize the preprocessed image based on the position information of the text lines, and obtaining the recognition result of the preprocessed image, includes: the convolutional layer extracting the preprocessed image based on the position information of the text lines to obtain feature information corresponding to the preprocessed image; using the recurrent RNN layer to predict the feature information corresponding to the preprocessed image to obtain a corresponding predicted probability distribution sequence; using the recurrent RNN layer to transform the predicted probability distribution sequence to obtain a corresponding label sequence; and using the label sequence as the recognition result of the preprocessed image.

[0090] It should be noted that the pre-trained CRNN model uses improved VGG convolutional layers. These layers are primarily based on the VGG16 convolutional network design, which consumes more computational resources and has a larger number of parameters. Therefore, improvements were made, reducing the original 13 convolutional layers to 7 and eliminating 3 fully connected layers. This improvement reduces the number of training parameters and increases training efficiency. Each convolutional layer uses a 3x3 kernel with a stride of 1.

[0091] In this embodiment, an improved convolutional layer is used to extract the preprocessed image based on the position information of the text lines, thereby obtaining the feature information corresponding to the preprocessed image. Then, an RNN recurrent layer is used to predict the feature information corresponding to the preprocessed image to obtain the corresponding prediction probability distribution sequence. The RNN recurrent layer is then used to transform the prediction probability distribution sequence to obtain the corresponding label sequence. Finally, the label sequence is used as the recognition result of the preprocessed image.

[0092] Step S105: Using a multimodal feature information extraction algorithm, extract the recognition result of the preprocessed image to obtain the corresponding multimodal feature information, and use the multimodal feature information as the recognition result of the power form to be recognized.

[0093] In this embodiment of the application, after obtaining the recognition result of the preprocessed image through step S104, the recognition result of the preprocessed image is extracted using a multimodal feature information extraction algorithm to obtain the corresponding multimodal feature information. Finally, the multimodal feature information is used as the recognition result of the power form to be recognized.

[0094] In one possible implementation, the step of using a multimodal feature information extraction algorithm to extract the recognition result of the preprocessed image to obtain corresponding multimodal feature information, and using the multimodal feature information as the recognition result of the power form to be recognized, includes: using the multimodal feature information extraction algorithm to extract the recognition result of the preprocessed image to obtain corresponding multimodal feature information; using a multimodal attention algorithm to associate and fuse the multimodal feature information to obtain multimodal target feature information; and using the multimodal target feature information as the recognition result of the power form image to be recognized, wherein the multimodal target feature information includes the text content of the power form image to be recognized and the category attribute corresponding to the text content.

[0095] Specifically, the multimodal feature information extraction algorithm is first used to extract the recognition results of the preprocessed image to obtain the corresponding multimodal feature information.

[0096] In practical applications, the VI-LayoutXLM multimodal feature information extraction algorithm is used, employing a BERT pre-trained language representation model as the backbone network and incorporating 2D absolute position information and image information to capture visual information features such as position, font, text direction, and color from the recognition results of the preprocessed image. The VI-LayoutXLM multimodal semantic entity recognition method in the PaddleOCR component is used to fuse text, position, and layout information, introducing a text line sorting method that conforms to the reading order to obtain the text features of the preprocessed image. Combining the text features and visual information features of the preprocessed image yields the corresponding multimodal feature information.

[0097] Next, a multimodal attention algorithm is used to associate and fuse the multimodal feature information to obtain multimodal target feature information, which is then used as the recognition result for the power form image to be identified. The multimodal target feature information includes the text content of the power form image to be identified and the category attributes corresponding to the text content, thus enabling the extraction of key fields.

[0098] It should be noted that after obtaining the multimodal target feature information, a multilayer perceptron (MLP) can be used to predict the multimodal target feature information and obtain the predicted answer.

[0099] It should be noted that traditional template matching methods lack generalization for recognizing power test reports, especially when the format and layout of the test report change, causing the corresponding template to become ineffective. This application, however, captures visual information features such as relative position, font, text direction, and color within the image text. By fusing text, position, and layout information and introducing a text line sorting method that conforms to the reading order, the preprocessed image's text features are obtained. This enables the key information extraction module to possess robustness and handle the recognition of different report formats.

[0100] As can be seen from the above technical solution, the embodiments of this application first acquire an image of the electricity form to be identified, then preprocess the image of the electricity form to be identified to obtain a preprocessed image, then use a pre-trained CTPN neural network model to extract the text line position information of the preprocessed image to obtain the position information of the text lines in the preprocessed image; and based on the position information of the text lines, use a pre-trained CRNN model to identify the preprocessed image to obtain the identification result of the preprocessed image; finally, use a multimodal feature information extraction algorithm to extract the identification result of the preprocessed image to obtain the corresponding multimodal feature information, and use the multimodal feature information as the identification result of the electricity form to be identified. In this way, this application uses a pre-trained CTPN neural network model to extract the position information of text lines in the image of the electricity form to be identified. Based on the position information of the text lines, a pre-trained CRNN model is used to identify the text information of the preprocessed image. A multimodal feature information extraction algorithm is used to extract the recognition results of the preprocessed image to obtain the corresponding multimodal feature information. This can automatically identify the multimodal feature information of table-like electricity forms in multiple scenarios, solve the problem of low efficiency of manual data extraction, improve the efficiency of electricity form recognition, and has strong robustness.

[0101] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0102] The above are some specific implementations of a fault detection method for a filter in a CVT transmission provided in this application. Based on this, this application also provides a corresponding device. The device provided in this application will be described below from the perspective of functional modularity.

[0103] See Figure 2 The diagram shows a multimodal power form recognition device. The device may include an acquisition module 100, a preprocessing module 200, a first extraction module 300, an identification module 400, and a second extraction module 500.

[0104] The acquisition module 100 is used to acquire an image of the electricity form to be identified;

[0105] The preprocessing module 200 is used to preprocess the image of the power form to be identified to obtain a preprocessed image;

[0106] The first extraction module 300 is used to extract text line position information from the preprocessed image using a pre-trained CTPN neural network model, so as to obtain the position information of the text lines in the preprocessed image.

[0107] The recognition module 400 is used to recognize the preprocessed image based on the position information of the text line using a pre-trained CRNN model, and obtain the recognition result of the preprocessed image;

[0108] The second extraction module 500 is used to identify the preprocessed image based on the position information of the text line using a pre-trained CRNN model, and obtain the recognition result of the preprocessed image.

[0109] Optionally, the preprocessing module includes a grayscale processing submodule and a binarization processing submodule;

[0110] The grayscale processing submodule is used to perform grayscale transformation processing on the image of the power form to be identified based on the weighted value method to obtain the grayscale image corresponding to the power form to be identified.

[0111] The binarization processing submodule is used to perform binarization processing on the grayscale image to obtain the binarized image corresponding to the grayscale image.

[0112] Optionally, after obtaining the binarized image corresponding to the grayscale image, the apparatus further includes:

[0113] The correction module is used to perform orientation detection on the binarized image and obtain the detection result;

[0114] Based on the detection results, the position of the binarized image is corrected.

[0115] As can be seen from the above technical solution, the embodiments of this application first acquire an image of the electricity form to be identified, then preprocess the image of the electricity form to be identified to obtain a preprocessed image, then use a pre-trained CTPN neural network model to extract the text line position information of the preprocessed image to obtain the position information of the text lines in the preprocessed image; and based on the position information of the text lines, use a pre-trained CRNN model to identify the preprocessed image to obtain the identification result of the preprocessed image; finally, use a multimodal feature information extraction algorithm to extract the identification result of the preprocessed image to obtain the corresponding multimodal feature information, and use the multimodal feature information as the identification result of the electricity form to be identified. In this way, this application uses a pre-trained CTPN neural network model to extract the position information of text lines in the image of the electricity form to be identified. Based on the position information of the text lines, a pre-trained CRNN model is used to identify the text information of the preprocessed image. A multimodal feature information extraction algorithm is used to extract the recognition results of the preprocessed image to obtain the corresponding multimodal feature information. This can automatically identify the multimodal feature information of table-like electricity forms in multiple scenarios, solve the problem of low efficiency of manual data extraction, improve the efficiency of electricity form recognition, and has strong robustness.

[0116] This application also provides a storage medium storing a program that, when executed by a processor, implements the multimodal power form recognition method described in the above embodiments.

[0117] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0118] Those skilled in the art will understand that the flowchart shown is merely an example in which the embodiments of this application can be implemented, and the scope of application of the embodiments of this application is not limited by any aspect of the flowchart.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and devices can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0121] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-modal electricity meter single identification method, characterized in that, The method includes: Obtain an image of the electricity form to be identified; The image of the power form to be identified is preprocessed to obtain a preprocessed image; The text line position information of the preprocessed image is extracted using a pre-trained CTPN neural network model to obtain the position information of the text lines in the preprocessed image. Based on the position information of the text lines, the pre-trained CRNN model is used to recognize the pre-processed image to obtain the recognition result of the pre-processed image; The recognition results of the preprocessed image are extracted using a multimodal feature information extraction algorithm to obtain the corresponding multimodal feature information; A multimodal attention algorithm is used to associate and fuse the multimodal feature information to obtain multimodal target feature information, which is then used as the recognition result of the electricity form to be identified. The multimodal target feature information includes the text content of the electricity form image to be identified and the category attribute corresponding to the text content. The step of using a multimodal feature information extraction algorithm to extract the recognition result of the preprocessed image to obtain the corresponding multimodal feature information includes: The multimodal feature information extraction algorithm is used to capture the position, font, text direction, and color visual information features in the recognition results of the preprocessed image; Using a multimodal semantic entity recognition method, by fusing text, location and layout information, and introducing a text line sorting method that conforms to the reading order, the text features of the preprocessed image are obtained; The multimodal feature information is obtained by combining the text features of the preprocessed image and the visual information features.

2. The method of claim 1, wherein, The preprocessing of the image of the electricity form to be identified includes: The image of the power form to be identified is subjected to grayscale transformation based on the weighted value method to obtain the grayscale image corresponding to the power form to be identified. The grayscale image is binarized using a binarization function to obtain the corresponding binarized image.

3. The method of claim 2, wherein, After obtaining the binarized image corresponding to the grayscale image, the method further includes: The binarized image is subjected to orientation detection to obtain the detection result; Based on the detection results, the position of the binarized image is corrected.

4. The method of claim 1, wherein, The pre-trained CTPN neural network model includes a VGG16 network layer and a fully connected layer; The step of extracting text line position information from the preprocessed image using a pre-trained CTPN neural network model to obtain the position information of text lines in the preprocessed image includes: The preprocessed image is input into the VGG16 network layer for feature extraction to generate a first feature map. A sliding window operation is performed on the first feature map to generate a second feature map; The second feature map is extracted using the fully connected layer to obtain the feature vector corresponding to the second feature map; The feature vector is input into the three branches of the fully connected layer for prediction to obtain the vertical coordinate regression, classification score, and horizontal translation amount corresponding to the feature vector. After converting the vertical coordinate regression, classification score, and horizontal translation, the position information of the text lines in the preprocessed image is obtained.

5. The method of claim 1, wherein, The pre-trained CRNN model includes convolutional layers and RNN recurrent layers; The recognition result of the preprocessed image obtained by using a pre-trained CRNN model based on the position information of the text line includes: The convolutional layer extracts the preprocessed image based on the position information of the text line to obtain the feature information corresponding to the preprocessed image; The RNN recurrent layer is used to predict the feature information corresponding to the preprocessed image to obtain the corresponding prediction probability distribution sequence; The predicted probability distribution sequence is transformed using the RNN recurrent layer to obtain the corresponding label sequence; The labeled sequence is used as the recognition result of the preprocessed image.

6. A multi-modal electricity meter single identification device, characterized in that, The device includes: The acquisition module is used to acquire images of the electricity form to be identified; The preprocessing module is used to preprocess the image of the power form to be identified to obtain a preprocessed image; The first extraction module is used to extract text line position information from the preprocessed image using a pre-trained CTPN neural network model, so as to obtain the position information of the text lines in the preprocessed image. The recognition module is used to recognize the preprocessed image based on the position information of the text line using a pre-trained CRNN model, and obtain the recognition result of the preprocessed image; The second extraction module is used to extract the recognition results of the preprocessed image using a multimodal feature information extraction algorithm to obtain the corresponding multimodal feature information. The fusion module is used to associate and fuse the multimodal feature information using a multimodal attention algorithm to obtain multimodal target feature information, and to use the multimodal target feature information as the recognition result of the power form to be recognized. The multimodal target feature information includes the text content of the power form image to be recognized and the category attribute corresponding to the text content. The step of using a multimodal feature information extraction algorithm to extract the recognition result of the preprocessed image to obtain the corresponding multimodal feature information includes: using the multimodal feature information extraction algorithm to capture the position, font, text direction, and color visual information features in the recognition result of the preprocessed image; using a multimodal semantic entity recognition method, by fusing text, position, and layout information, and introducing a text line sorting method that conforms to the reading order, to obtain the text features of the preprocessed image; and combining the text features of the preprocessed image and the visual information features to obtain the multimodal feature information.

7. The apparatus of claim 6, wherein, The preprocessing module includes a grayscale processing submodule and a binarization processing submodule; The grayscale processing submodule is used to perform grayscale transformation processing on the image of the power form to be identified based on the weighted value method to obtain the grayscale image corresponding to the power form to be identified. The binarization processing submodule is used to perform binarization processing on the grayscale image to obtain the binarized image corresponding to the grayscale image.

8. The apparatus according to claim 7, characterized in that, After obtaining the binarized image corresponding to the grayscale image, the device further includes: The correction module is used to perform orientation detection on the binarized image and obtain the detection result; Based on the detection results, the position of the binarized image is corrected.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images