Electric power multi-task document classification method based on cross-modal attention mechanism

Abstract

The invention relates to an electric power multi-task document classification method based on a cross-modal attention mechanism, and the method comprises the following steps: collecting an electric power multi-task document, and carrying out the preprocessing of the electric power multi-task document; constructing an electric power multi-task document classification model based on a cross-modal attention mechanism; inputting the preprocessed power multi-task document into a power multi-task document classification model for training to obtain a trained power multi-task document classification model; classifying the electric power multi-task documents through the trained electric power multi-task document classification model; according to the method, a cross-modal attention mechanism and a transformer encoder are integrated, so that text and visual information are effectively utilized to improve the performance of a document classification task.

Description

TECHNICAL FIELD

[0001] The application relates to a power multi-task document classification method based on a cross-modal attention mechanism and belongs to the technical field of machine learning. BACKGROUND

[0002] In recent years, with the development of deep learning technology, the transformer encoder has become an important tool in the field of natural language processing. However, when dealing with complex technical documents, the transformer encoder still has some limitations; for example, the traditional transformer encoder mainly relies on text information, while ignoring the importance of visual information.

[0003] Some recent research works try to combine visual information with text information to improve the performance of the transformer encoder. For example, the VisualBERT and ViLT models both use cross-modal attention mechanisms to combine text and visual information. These models have achieved good performance in multi-modal tasks, but still have some limitations.

[0004] In the field of power multi-task document understanding, documents usually contain a large amount of technical information, including text and visual information. However, existing transformer encoders are difficult to effectively utilize these information to improve the performance of tasks such as document classification and information extraction. Therefore, a new method is needed to combine text and visual information to improve the understanding ability of transformer encoders for complex technical documents.

[0005] Although the existing cross-modal attention mechanism can combine text and visual information, it still has some limitations. For example, the existing cross-modal attention mechanism usually relies on pre-trained models, and these models may not be suitable for specific tasks or domains. In addition, the existing cross-modal attention mechanism usually requires a large amount of training data, while in the field of power multi-task document understanding, training data may be limited. SUMMARY

[0006] In order to solve the problems existing in the prior art, the application proposes a power multi-task document classification method based on a cross-modal attention mechanism.

[0007] The technical solution of the application is as follows: On the one hand, the application proposes a power multi-task document classification method based on a cross-modal attention mechanism, including the following steps: Collecting power multi-task documents and preprocessing the power multi-task documents; Constructing a power multi-task document classification model based on a cross-modal attention mechanism; The preprocessed power multitask document is input into the power multitask document classification model for training, and a trained power multitask document classification model is obtained. The power multitask document is classified by the trained power multitask document classification model.

[0008] Preferably, each power multitask document comprises image data and text data.

[0009] Preferably, the power multitask document classification model comprises a visual feature extraction layer, a text-visual alignment layer, a cross-modal attention layer, and an output layer.

[0010] Preferably, the visual feature extraction layer is constructed based on a convolutional neural network and is used to extract image data features of the power multitask document.

[0011] Preferably, the text-visual alignment layer is used to extract text data features of the power multitask document, and similarity calculation is performed on the corresponding image data features to generate a similarity matrix.

[0012] Preferably, the cross-modal attention layer is used to generate cross-modal attention weights based on the similarity matrix of the current power multitask document, and the cross-modal attention weights are fused with the image data features to output weighted image data features.

[0013] Preferably, the output layer is used to output the type of the current power multitask document according to the weighted image data features.

[0014] In another aspect, the present application also provides a power multitask document classification system based on a cross-modal attention mechanism, comprising a data acquisition module, a power multitask document classification model construction module, a power multitask document classification model training module, and a power multitask document classification module. The data acquisition module is used to acquire power multitask documents and preprocess the power multitask documents. The power multitask document classification model construction module is used to construct a power multitask document classification model based on a cross-modal attention mechanism. The power multitask document classification model training module is used to input the preprocessed power multitask document into the power multitask document classification model for training, and obtain a trained power multitask document classification model. The power multitask document classification module is used to classify the power multitask document by the trained power multitask document classification model.

[0015] In still another aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to the present application when executing the program.

[0016] In still another aspect, the present application provides a computer readable storage medium having stored thereon a computer program, wherein the program is executable on a processor to implement the method according to the present application.

[0017] The present application has the following beneficial effects: 1. The present application proposes a cross-modal attention mechanism for enhancing power multi-task document understanding, which can effectively combine text and visual information to improve the understanding of complex technical content by the transformer encoder, and effectively utilize text and visual information to improve the performance of the document classification task by integrating the cross-modal attention mechanism with the transformer encoder. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The present application has the following beneficial effects: DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0020] It should be understood that the step numbers used herein are only for the convenience of description, and are not limited to the execution sequence of the steps.

[0021] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clearly indicated by the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0022] The terms "comprise" and "include" indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0023] The term "and / or" means any combination of one or more of the associated listed items and all possible combinations thereof, and includes these combinations.

[0024] Reference is made toFigure 1 A power multi-task document classification method based on a cross-modal attention mechanism, comprising the following steps: Collecting power multi-task documents and preprocessing the power multi-task documents; Building a power multi-task document classification model based on a cross-modal attention mechanism; Inputting the preprocessed power multi-task documents into the power multi-task document classification model for training to obtain a trained power multi-task document classification model; Classifying power multi-task documents through the trained power multi-task document classification model.

[0025] In some embodiments, each power multi-task document includes image data and text data.

[0026] In a specific embodiment, the power multi-task documents include maintenance work orders, equipment account books, inspection reports, hidden danger rectification forms, etc. Each document sample includes image data obtained by scanning or photographing and text data obtained by OCR text recognition.

[0027] In some embodiments, the power multi-task document classification model includes a visual feature extraction layer, a text-visual alignment layer, a cross-modal attention layer, and an output layer.

[0028] In some embodiments, the visual feature extraction layer is constructed based on a convolutional neural network and is used to extract image data features of power multi-task documents.

[0029] In a specific embodiment, the visual feature extraction layer is constructed based on a ResNet-50 network.

[0030] In a specific embodiment, the visual feature extraction layer is constructed based on a VGG-16 network model.

[0031] In some embodiments, the text-visual alignment layer is used to extract text data features of power multi-task documents and perform similarity calculation with corresponding image data features to generate a similarity matrix.

[0032] In a specific embodiment, the text-visual alignment layer extracts text data features based on a Transformer encoder and performs similarity calculation with corresponding image data features to generate a similarity matrix.

[0033] In the similarity matrix, each element represents the similarity between each text word feature in the image data and the visual feature at the corresponding position in the image data, where the corresponding position is the image region corresponding to the current text word.

[0034] In some embodiments, the cross-modal attention layer is configured to generate cross-modal attention weights based on the similarity matrix of the current power multi-task document, and fuse the cross-modal attention weights with the image data features to output weighted image data features.

[0035] In some embodiments, the cross-modal attention layer is configured to generate cross-modal self-attention weights based on the similarity matrix of the current power multi-task document by using a self-attention mechanism, and fuse the cross-modal self-attention weights with the image data features to output weighted image data features.

[0036] In some embodiments, the cross-modal attention layer is integrated into a multi-task learning framework. Specifically, a multi-task learning model is designed to handle both document classification and information extraction tasks simultaneously. By sharing the visual feature extraction layer and the text-visual alignment layer, the model can effectively utilize text and visual information to improve the performance of multiple tasks.

[0037] In some embodiments, an incremental learning method is used to update the cross-modal attention layer. Specifically, an incremental learning framework is designed to update model parameters gradually as new data arrives without the need to retrain the entire model. This can effectively adapt to changes in data in the field of power multi-task document understanding, improving the robustness and adaptability of the model.

[0038] In some embodiments, a distributed training method is used to accelerate the training process of the cross-modal attention layer. Specifically, the training data is distributed across multiple computing nodes, each node is responsible for processing a portion of the data. Through parallel computing, the training time can be significantly reduced, and the training efficiency of the model can be improved.

[0039] In some embodiments, a lightweight convolutional neural network (such as MobileNet) is used as the visual feature extraction layer, and a simplified version of the Transformer model is used as the text-visual alignment layer. This can significantly reduce the computational and storage costs of the model while maintaining high performance.

[0040] In some embodiments, the power multi-task document classification model also introduces an audio feature extraction layer and a video feature extraction layer.

[0041] In a specific embodiment, the cross-modal attention layer maps each element in the similarity matrix to a weight value through a Softmax function.

[0042] In some embodiments, the output layer is configured to output the type of the current power multi-task document based on the weighted image data features.

[0043] In some embodiments, a power multi-task document classification system based on a cross-modal attention mechanism is proposed, including a data acquisition module, a power multi-task document classification model construction module, a power multi-task document classification model training module, and a power multi-task document classification module; The data acquisition module is used to acquire power multi-task documents and preprocess the power multi-task documents; The power multi-task document classification model construction module is used to construct a power multi-task document classification model based on a cross-modal attention mechanism; The power multi-task document classification model training module is used to input the preprocessed power multi-task document into the power multi-task document classification model for training, so as to obtain the trained power multi-task document classification model. The power multitasking document classification module is used to classify power multitasking documents using a trained power multitasking document classification model.

[0044] In some embodiments, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any embodiment of the present invention.

[0045] In some embodiments, a computer-readable storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the method as described in any embodiment of the present invention.

[0046] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0047] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0048] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0049] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0050] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A power multi-task document classification method based on cross-modal attention mechanism, characterized in that, Includes the following steps: Collect power multi-task documents and preprocess them; Construct a multi-task document classification model for electricity based on a cross-modal attention mechanism; The preprocessed power multitasking documents are input into the power multitasking document classification model for training, resulting in the trained power multitasking document classification model. The trained power multi-task document classification model is used to classify power multi-task documents.

2. The power multi-task document classification method based on cross-modal attention mechanism according to claim 1, characterized in that, Each power multitasking document includes both image and text data.

3. The power multi-task document classification method based on cross-modal attention mechanism according to claim 2, characterized in that, The power multi-task document classification model includes a visual feature extraction layer, a text visual alignment layer, a cross-modal attention layer, and an output layer.

4. The power multi-task document classification method based on cross-modal attention mechanism according to claim 3, characterized in that, The visual feature extraction layer is built on a convolutional neural network and is used to extract features from power multitasking document image data.

5. The power multi-task document classification method based on cross-modal attention mechanism according to claim 4, characterized in that, The text visual alignment layer is used to extract text data features from the power multitasking document, calculate the similarity between the text and the corresponding image data features, and generate a similarity matrix.

6. The power multi-task document classification method based on cross-modal attention mechanism according to claim 5, characterized in that, The cross-modal attention layer is used to generate cross-modal attention weights based on the similarity matrix of the current power multi-task document, and to fuse the cross-modal attention weights with image data features to output weighted image data features.

7. The power multi-task document classification method based on cross-modal attention mechanism according to claim 6, characterized in that, The output layer is used to output the type of the current power multitasking document based on the weighted image data features.

8. A power multi-task document classification system based on a cross-modal attention mechanism, characterized in that, It includes a data acquisition module, a power multi-task document classification model construction module, a power multi-task document classification model training module, and a power multi-task document classification module; The data acquisition module is used to acquire power multi-task documents and preprocess the power multi-task documents; The power multi-task document classification model construction module is used to construct a power multi-task document classification model based on a cross-modal attention mechanism; The power multi-task document classification model training module is used to input the preprocessed power multi-task document into the power multi-task document classification model for training, so as to obtain the trained power multi-task document classification model. The power multitasking document classification module is used to classify power multitasking documents using a trained power multitasking document classification model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images