Intelligent transformer operation and maintenance detection method based on multilevel neural network

Abstract

The invention relates to an intelligent transformer operation and maintenance detection method based on a multi-level neural network, which belongs to the field of transformer operation and maintenance, and comprises the following steps: carrying out preprocessing, feature extraction and pre-training sample construction on corpora collected in a transformer operation and maintenance scene, and completing a large model network architecture suitable for power industry knowledge; according to the method, hierarchical knowledge pre-training is carried out on a transformer operation inspection cognition large model, the method has a very strong prediction capability on texts and images of transformer operation inspection, the processing capability on the texts and the images in the transformer operation inspection is improved through various pre-training means, and the working efficiency of workers on transformer operation inspection is improved.

Description

Technical Field

[0001] The invention relates to an intelligent transformer operation and maintenance detection method based on a multi-level neural network, and belongs to the field of transformer operation and maintenance. Background Art

[0002] As a key equipment in the power system, the monitoring and maintenance of the operating status of the transformer is crucial to ensure the safe and stable operation of the power system. Transformer operation and maintenance, that is, the operation, maintenance and overhaul of the transformer, involves regular inspection, monitoring, fault diagnosis and necessary maintenance work on the transformer to prevent the occurrence of faults and promptly repair the existing faults. Using multi-level training can more effectively extract features from raw data, especially when processing large-scale, multi-source heterogeneous data. Through different levels of models, we can gradually learn deeper feature representations from simple to complex, better understand the operating status and potential problems of the transformer, and thus provide a more accurate basis for the formulation of predictive maintenance and condition maintenance plans. It helps the model learn more generalized features, reduce dependence on specific data sets, and improve the applicability of the model in different environments and conditions. Each layer in the multi-level training can provide different interpretations of the data, which helps to understand the internal causes and mechanisms of the fault, thereby improving the interpretability of the model, reducing manual intervention, improving the automation level of transformer operation and inspection, and reducing labor costs. Combined with real-time data streams, the multi-level training model can monitor the status of the transformer in real time and provide early warnings before potential problems occur, thereby achieving preventive maintenance. Summary of the invention

[0003] According to the problem described in the background, the problem to be solved by the present invention is: to provide an intelligent transformer operation and maintenance detection method based on a multi-level neural network, to pre-process, feature extract and construct pre-trained samples of the corpus collected in the transformer operation and inspection scene, and to complete a large model network architecture suitable for the knowledge of the power industry, to perform hierarchical knowledge pre-training on the transformer operation and inspection cognitive large model, so that this method has a strong predictive ability for the text and images of the transformer operation and inspection, and to improve the processing ability of the text and images in the transformer operation and inspection through various pre-training means, thereby improving the work efficiency of the staff in the transformer operation and inspection.

[0004] To achieve the above object, the present invention provides the following technical solution: providing an intelligent transformer operation and maintenance detection method based on a multi-level neural network, comprising the following steps:

[0005] (1) Large-scale corpus collection in transformer operation and maintenance scenarios;

[0006] (2) Preprocessing, feature extraction and construction of training samples for large-scale predictions collected during transformer operation and maintenance;

[0007] (3) Construct the network architecture of the encoding and decoding layers of the transformer operation and maintenance cognitive model;

[0008] (4) Conduct hierarchical knowledge pre-training on the transformer operation and maintenance cognitive model.

[0009] The step (1) is further specifically divided into the following steps:

[0010] (1.1) Detailed analysis of transformer multimodal data range;

[0011] (1.2) Establish a semantic hierarchical classification system for multimodal information;

[0012] (1.3) Abstract the transformer state law into knowledge;

[0013] (1.4) Collect the three common levels of concept (state), description, and knowledge in multimodal data.

[0014] The step (2) includes corpus preprocessing, feature extraction and training sample construction;

[0015] The corpus preprocessing can use different levels of knowledge extraction strategies for different types of documents. For transformer information, common knowledge is mainly used, and the basic parameters of the associated equipment product information are used as associated information to provide information support for subsequent equipment alignment. Other document types should distinguish between common and individual knowledge parts based on document content characteristics, and extract them in layers;

[0016] The key technology of feature extraction is Transformer, which uses two technical means: relative position encoding and full-word masking mechanism based on the electric power field dictionary. The relative position embedding of relative position encoding does not use a fixed embedding for each position, but generates different learning embeddings according to different position relationships, thereby enhancing the expression ability of relative position information and making the word-segmented corpus input into the model more accurate; the full-word masking mechanism based on the electric power field dictionary will mask the entire word, so that the model can predict the entire word when making MLM predictions, thereby better grasping the entity information;

[0017] The training sample construction reduces the semantic gap between image and text information to a certain extent.

[0018] The step (3) includes visual modality, language modality and cross-visual-language mixed modality;

[0019] The network architecture design of the visual modality encoding and decoding layers includes two methods: Transformer and convolutional neural network;

[0020] The main operation steps of the Transformer are to first divide the input transformer image into blocks, then input each block into the linear mapping layer to obtain a preliminary visual encoding, concatenate (or add) the preliminary visual encoding obtained after passing the linear layer through a position encoding vector, input the concatenated model into the Transformer encoder for further encoding, and finally train it using the cross entropy loss function;

[0021] The convolutional neural network supplements the scenario where the data volume in the above-mentioned Transformer model is not so sufficient. The specific operation steps are as follows: the input image is a transformer knowledge image. First, preliminary image features are obtained through a convolution layer and a maximum pooling layer. The subsequent 4-stage model structure consists of a downsampling layer and a residual module. The residual module consists of a superposition of convolution layers with different convolution kernel sizes and channel numbers, and the input will be directly connected to the output for the last accumulation. The downsampling layer will be the maximum pooling layer, which will attenuate the size of the feature map by a factor of 2. Feature extraction starts from low-level texture features and gradually goes to high-level semantic features. Finally, the output layer is fully connected and mapped to different categories for training using a cross entropy loss function.

[0022] The network architecture design of the language modality encoding and decoding layers includes BERT technology and T5 technology. BERT technology adds different classification heads for specific downstream tasks and can also be used for tasks such as entity extraction of power field knowledge in texts. T5 technology can quickly solve technical problems through question-and-answer, which can save a lot of time and manpower.

[0023] The network architecture design of the cross-visual-language hybrid modal encoding and decoding layers includes two models: a single-tower fusion cross-modal model and a dual-tower fusion cross-modal model. The single-tower fusion cross-modal model can capture a more fine-grained semantic alignment relationship between vision and language, and is divided into three tasks: language mask prediction task, image-text matching task, and image description task to complete the purpose of the single-tower fusion cross-modal model; the dual-tower fusion cross-modal model can flexibly use the scenarios of the vision and language towers.

[0024] The step (4) includes pre-training of general knowledge masking, discrimination, substitution, etc. for transformer operation and inspection, and three-stage pre-training of transformer operation and inspection based on characters, phrases, and knowledge;

[0025] The transformer operation and inspection general knowledge masking, discrimination, substitution and other pre-training include three types of training: text, multi-modal pre-training basis and operation and inspection multi-stage progressive pre-training;

[0026] The three-stage pre-training of transformer operation and inspection based on characters, phrases and knowledge includes three stages and transformer operation and inspection cognitive pre-training integrating multiple basic strategies and multiple semantic levels.

[0027] The text training includes MLM tasks and NSP tasks;

[0028] The MLM task reduces the bias of network learning, which is conducive to the text pre-training model learning the representation distribution of each input word context, and the model restores the original category information according to the power equipment operation and inspection image features corresponding to the image end;

[0029] The NSP task can better understand the relationship between two sentences, complete text training, and better process text data in transformer operation and inspection tasks;

[0030] The multimodal pre-training basis includes a text mask task, a mask image feature regression task, and a mask object classification task;

[0031] The text masking task has different settings in the transformer inspection multimodal end pre-training task MLM and the text end pre-training MLM; the mask image feature regression task, the model restores the masked image features according to the surrounding information of the transformer image and the description of the text end; the mask object classification task is that the model predicts the category label of the mask feature according to the surrounding information of the transformer inspection image data and the description of the text end;

[0032] The multi-stage progressive pre-training of operation and inspection includes a single-stream model and a dual-stream model;

[0033] The single-stream model fuses the image information and text information at the beginning of the input and then inputs it into the BERT-based model; the dual-stream model encodes the transformer inspection image information and text information separately first, and then fuses the encoding to obtain the final image text representation, which can greatly improve the benchmark performance of various NLP tasks;

[0034] The first stage is the image feature random shuffling task. Based on the existing pre-training tasks, this stage designs a corresponding pre-training task to fully learn the correspondence between images and texts. In this stage, the image end inputs the image object features, and the text end inputs the transformer information and technical standard category label characters and words in the corresponding order. The main image feature random shuffling task is to randomly select three consecutive unmasked image features and randomly shuffle their order with a fixed probability.

[0035] The second stage is a sentence-based subject selection task, which helps the model learn rich attribute information and effectively improves the accuracy of the model on downstream tasks. In this stage, the image side inputs the image object features, and the text side inputs the phrase-level transformer status evaluation task knowledge text description of the image. This stage mainly uses the sentence-based subject selection task to combine the image and text information to predict the content described by both.

[0036] The third stage is an image-text matching task based on difficult samples. Sentences contain the description information of the entire image. Therefore, sentence-based pre-training can enable the model to learn the correspondence between the image and text at the semantic level. This stage mainly uses image-text matching tasks based on difficult samples. The purpose is to reduce the noise brought to the model by the image-text pair when the image and text do not match and increase the difficulty of the image-text matching task.

[0037] The transformer operation and inspection cognitive pre-training integrating multiple basic strategies and multiple semantic levels includes D-UniTAB technology and MQA technology;

[0038] The D-UniTAB technology first performs visual language pre-training, then fine-tunes the model for multiple tasks, merges multiple downstream task data sets of power equipment inspection with supervision information, fine-tunes D-UniTAB for different tasks, and shares parameters, and finally fine-tunes specific sub-tasks of transformer inspection.

[0039] The MQA technology includes six methods: visual QA, region-guided visual QA, position-guided visual QA, pure spatial QA, region-guided spatial QA and spatial QA guided by visual attributes; the goal of visual QA (Pure Visual QA, PVQA) is to help the model establish the connection between the power equipment ID and the corresponding visual attributes when providing scene images; region-guided visual QA (Region-Guided Visual QA, RVQA) is to improve the model's ability to locate power equipment and describe its visual attributes through regional visual context; position-guided visual QA (Position-Guided Visual QA, PoVQA) in the dialogue, the agent must locate the power equipment through its spatial information whenever it understands the user's speech or makes suggestions; pure spatial QA (Pure Spatial QA, PSQA) is the most basic spatial QA task. The goal of PSQA is to help the model establish the connection between the power equipment ID and the corresponding spatial relationship when providing scene images; region-guided spatial QA (Region-Guided Spatial QA, RVQA) is to improve the model's ability to locate power equipment and describe its visual attributes through regional visual context; position-guided visual QA (PoVQA) in the dialogue, the agent must locate the power equipment through its spatial information whenever it understands the user's speech or makes suggestions; pure spatial QA (Pure Spatial QA, PSQA) is the most basic spatial QA task. The goal of PSQA is to help the model establish the connection between the power equipment ID and the corresponding spatial relationship when providing scene images; QA (RSQA) is to improve the model's ability to locate power devices and describe their spatial relationships through regional visual context; Visual Attribute-Guided Spatial QA (VSQA) In a conversation, the agent must locate power devices through their visual attributes whenever it understands the user's utterance or makes suggestions.

[0040] The beneficial effects of the present invention are:

[0041] 1. Complete the information fusion of text and image, effectively utilize the visual attributes and spatial relationships of the transformer operation and inspection to recognize the text and image information, and improve the quality of response generation.

[0042] 2. The network framework of the transformer operation and maintenance cognitive large model is developed from the visual modality, language modality, and cross-visual language mixed modality. Various methods in each modality complement and combine with each other, making this network framework more solid, more practical, and more efficient.

[0043] 3. The convolutional neural network architecture assists Transformer to make up for the lack of data. Transformer has strong applicability, flexibility and sufficient data in the framework structure;

[0044] 4. BERT and T5 language modalities can be used not only for classification tasks, but also for tasks such as entity extraction of power field knowledge in texts. By answering questions, technical problems can be quickly solved, saving a lot of time and manpower.

[0045] 5. The cross-modality of single-tower fusion and dual-tower fusion complement each other, which can capture more fine-grained semantic alignment relationships between vision and language, and can also meet scenarios that require flexible use of vision and language towers.

[0046] 6. The pre-training of the transformer operation and maintenance cognitive large model strengthens the semantic expression between different modalities from three aspects: text, characters, phrases, knowledge, multi-basic strategies and multi-semantic levels, so that the model can adapt to each specific task in order to further improve performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the method of the present invention;

[0048] Figure 2 is a flow chart of the method of step (1);

[0049] Figure 3 Schematic diagram of the module of step (2);

[0050] Figure 4 Schematic diagram of the module of step (3);

[0051] Figure 5 Schematic diagram of the module of step (4);

[0052] Figure 6 Schematic diagram of the modules of BERT technology and MLM tasks;

[0053] Figure 7 It is a module diagram of the MQA method; DETAILED DESCRIPTION

[0054] The embodiments of the present invention are further described below in conjunction with the accompanying drawings:

[0055] Example 1

[0056] like Figure 1 As shown, the present invention comprises the following steps:

[0057] (1) Large-scale corpus collection in transformer operation and maintenance scenarios;

[0058] (2) Preprocessing, feature extraction and construction of training samples for large-scale predictions collected during transformer operation and maintenance;

[0059] (3) Construct the network architecture including the encoding and decoding layers of the transformer operation and maintenance cognitive model;

[0060] (4) Conduct hierarchical knowledge pre-training on the transformer operation and maintenance cognitive model.

[0061] like Figure 2 As shown, the step (1) is further specifically divided into the following steps:

[0062] (1.1) Detailed analysis of transformer multimodal data range;

[0063] (1.2) Establish a semantic hierarchical classification system for multimodal information;

[0064] (1.3) Abstract the transformer state law into knowledge;

[0065] (1.4) Collect the three common levels of concept (state), description, and knowledge in multimodal data.

[0066] like Figure 3 As shown, the step (2) includes corpus preprocessing, feature extraction and training sample construction;

[0067] The corpus preprocessing can use different levels of knowledge extraction strategies for different types of documents. For transformer information, common knowledge is mainly used, and the basic parameters of the associated equipment product information are used as associated information to provide information support for subsequent equipment alignment. Other document types should distinguish between common and individual knowledge parts based on document content characteristics, and extract them in layers;

[0068] The key technology of feature extraction is Transformer, which uses two technical means: relative position encoding and full-word masking mechanism based on the electric power field dictionary. The relative position embedding of relative position encoding does not use a fixed embedding for each position, but generates different learning embeddings according to different position relationships, thereby enhancing the expression ability of relative position information and making the word-segmented corpus input into the model more accurate; the full-word masking mechanism based on the electric power field dictionary will mask the entire word, so that the model can predict the entire word when making MLM predictions, thereby better grasping the entity information;

[0069] The training sample construction reduces the semantic gap between image and text information to a certain extent.

[0070] like Figure 4 As shown, the step (3) includes visual modality, language modality and cross-visual-language mixed modality;

[0071] The network architecture design of the visual modality encoding and decoding layers includes two methods: Transformer and convolutional neural network;

[0072] The main operation steps of the Transformer are to first divide the input transformer image into blocks, then input each block into the linear mapping layer to obtain a preliminary visual encoding, concatenate (or add) the preliminary visual encoding obtained after passing the linear layer through a position encoding vector, input the concatenated model into the Transformer encoder for further encoding, and finally train it using the cross entropy loss function;

[0073] The convolutional neural network supplements the scenario where the data volume in the above-mentioned Transformer model is not so sufficient. The specific operation steps are as follows: the input image is a transformer knowledge image. First, preliminary image features are obtained through a convolution layer and a maximum pooling layer. The subsequent 4-stage model structure consists of a downsampling layer and a residual module. The residual module consists of a superposition of convolution layers with different convolution kernel sizes and channel numbers, and the input will be directly connected to the output for the last accumulation. The downsampling layer will be the maximum pooling layer, which will attenuate the size of the feature map by a factor of 2. Feature extraction starts from low-level texture features and gradually goes to high-level semantic features. Finally, the output layer is fully connected and mapped to different categories for training using a cross entropy loss function.

[0074] The network architecture design of the language modality encoding and decoding layers includes BERT technology and T5 technology. BERT technology adds different classification heads for specific downstream tasks and can also be used for tasks such as entity extraction of power field knowledge in texts. T5 technology can quickly solve technical problems through question-and-answer, which can save a lot of time and manpower.

[0075] The network architecture design of the cross-visual-language hybrid modal encoding and decoding layers includes two models: a single-tower fusion cross-modal model and a dual-tower fusion cross-modal model. The single-tower fusion cross-modal model can capture a more fine-grained semantic alignment relationship between vision and language, and is divided into three tasks: language mask prediction task, image-text matching task, and image description task to complete the purpose of the single-tower fusion cross-modal model; the dual-tower fusion cross-modal model can flexibly use the scenarios of the vision and language towers.

[0076] like Figure 5 As shown, the step (4) includes pre-training of general knowledge masking, discrimination, substitution, etc. of transformer operation and inspection and three-stage pre-training of transformer operation and inspection based on characters, phrases, and knowledge;

[0077] The transformer operation and inspection general knowledge masking, discrimination, substitution and other pre-training include three types of training: text, multi-modal pre-training basis and operation and inspection multi-stage progressive pre-training;

[0078] The three-stage pre-training of transformer operation and inspection based on characters, phrases and knowledge includes three stages and transformer operation and inspection cognitive pre-training integrating multiple basic strategies and multiple semantic levels.

[0079] The text training includes MLM tasks and NSP tasks;

[0080] The MLM task reduces the bias of network learning, which is conducive to the text pre-training model learning the representation distribution of each input word context, and the model restores the original category information according to the power equipment operation and inspection image features corresponding to the image end;

[0081] The NSP task can better understand the relationship between two sentences, complete text training, and better process text data in transformer operation and inspection tasks;

[0082] The multimodal pre-training basis includes a text mask task, a mask image feature regression task, and a mask object classification task;

[0083] The text masking task has different settings in the transformer inspection multimodal end pre-training task MLM and the text end pre-training MLM; the mask image feature regression task, the model restores the masked image features according to the surrounding information of the transformer image and the description of the text end; the mask object classification task is that the model predicts the category label of the mask feature according to the surrounding information of the transformer inspection image data and the description of the text end;

[0084] The multi-stage progressive pre-training of operation and inspection includes a single-stream model and a dual-stream model;

[0085] The single-stream model fuses the image information and text information at the beginning of the input and then inputs it into the BERT-based model; the dual-stream model encodes the transformer inspection image information and text information separately first, and then fuses the encoding to obtain the final image text representation, which can greatly improve the benchmark performance of various NLP tasks;

[0086] The first stage is the image feature random shuffling task. Based on the existing pre-training tasks, this stage designs a corresponding pre-training task to fully learn the correspondence between images and texts. In this stage, the image end inputs the image object features, and the text end inputs the transformer information and technical standard category label characters and words in the corresponding order. The main image feature random shuffling task is to randomly select three consecutive unmasked image features and randomly shuffle their order with a fixed probability.

[0087] The second stage is a sentence-based subject selection task, which helps the model learn rich attribute information and effectively improves the accuracy of the model on downstream tasks. In this stage, the image side inputs the image object features, and the text side inputs the phrase-level transformer status evaluation task knowledge text description of the image. This stage mainly uses the sentence-based subject selection task to combine the image and text information to predict the content described by both.

[0088] The third stage is an image-text matching task based on difficult samples. Sentences contain the description information of the entire image. Therefore, sentence-based pre-training can enable the model to learn the correspondence between the image and text at the semantic level. This stage mainly uses image-text matching tasks based on difficult samples. The purpose is to reduce the noise brought to the model by the image-text pair when the image and text do not match and increase the difficulty of the image-text matching task.

[0089] The transformer operation and inspection cognitive pre-training integrating multiple basic strategies and multiple semantic levels includes D-UniTAB technology and MQA technology;

[0090] The D-UniTAB technology first performs visual language pre-training, then fine-tunes the model for multiple tasks, merges multiple downstream task data sets of power equipment inspection with supervision information, fine-tunes D-UniTAB for different tasks, and shares parameters, and finally fine-tunes specific sub-tasks of transformer inspection.

[0091] The MQA technology includes six methods: visual QA, region-guided visual QA, position-guided visual QA, pure spatial QA, region-guided spatial QA and visual attribute-guided spatial QA;

[0092] like Figure 6 As shown, Figure 5The multimodal pre-training text mask (Masked Language Modeling, MLM) task in the transformer operation and inspection multimodal end pre-training task MLM and text end pre-training MLM settings are different. Figure 2 MLM task in Chinese text processing: Assuming that the sentence in the pre-training corpus is "Oil-immersed transformer body", MLM will randomly select 15% of the tokens in the sentence for masking. Assuming that the fourth token position is randomly selected to be masked, that is, "change" is masked. The masking process can be described as follows: 80%: Replace the target word with [MASK], for example: Oil-immersed transformer body becomes Oil-immersed [MASK] pressure device body. 10%: Replace the target word with a random word, for example: Oil-immersed transformer body becomes Oil-immersed positive pressure device body. 10%: Do not change the target word, for example: Oil-immersed transformer body becomes Oil-immersed transformer body. (The purpose of this is to make the representation biased towards the words actually observed.) The above mask mechanism is implemented as follows in the specific training process: Each epoch represents the completion of learning all the samples of the power equipment operation and inspection text, so each sample will be repeatedly input into the model during the entire training process. Each time a sample is input into the model, the probability of replacing the target word with [MASK] is 80%; the probability of replacing the target word with a random word is 10%; the probability of not changing the target word is 10%. For the transformer operation and inspection multimodal end pre-training task MLM, MLM replaces the text with the special word "[MASK]" with a probability of 80%, replaces the current text randomly with a probability of 10%, and keeps the text unchanged with a probability of 10%. The model restores the original category information based on the power equipment operation and inspection image features corresponding to the image end.

[0093] like Figure 7As shown in the figure, the goal of visual QA (Pure Visual QA, PVQA) in MQA technology is to help the model establish the connection between the power equipment ID and the corresponding visual attributes when providing scene images; Region-Guided Visual QA (RVQA) is to improve the model's ability to locate power equipment and describe its visual attributes through regional visual context; Position-Guided Visual QA (PoVQA) In the dialogue, the agent must locate the power equipment through its spatial information whenever it understands the user's words or makes suggestions; Pure Spatial QA (PSQA) is the most basic spatial QA task. The goal of PSQA is to help the model establish the connection between the power equipment ID and the corresponding spatial relationship when providing scene images; Region-Guided Spatial QA (RSQA) is to improve the model's ability to locate power equipment and describe its spatial relationship through regional visual context; Visual Attribute-Guided Spatial QA (Visual Attribute-Guided Spatial QA) is to improve the model's ability to locate power equipment and describe its spatial relationship through regional visual context; QA, VSQA) In a conversation, the agent must locate power devices by their visual attributes whenever it understands the user's utterance or makes suggestions.

Claims

1. An intelligent transformer operation and maintenance detection method based on multi-level neural network, It is characterized in that The following steps are involved: (1) Large-scale corpus collection in transformer operation and maintenance scenarios; (2) Preprocessing, feature extraction and construction of training samples for large-scale predictions collected during transformer operation and maintenance; (3) Construct the network architecture of the encoding and decoding layers of the transformer operation and maintenance cognitive model; (4) Conduct hierarchical knowledge pre-training on the transformer operation and maintenance cognitive model.

2. According to claim 1, a smart transformer operation and maintenance detection method based on a multi-level neural network, It is characterized in that The step (1) is further specifically divided into the following steps: (1.1) Detailed analysis of transformer multimodal data range; (1.2) Establish a semantic hierarchical classification system for multimodal information; (1.3) Abstract the transformer state law into knowledge; (1.4) Collect the three common levels of concept (state), description, and knowledge in multimodal data.

3. According to the intelligent transformer operation and maintenance detection method based on multi-level neural network as claimed in claim 1, It is characterized in that The step (2) includes corpus preprocessing, feature extraction and training sample construction; The corpus preprocessing can use different levels of knowledge extraction strategies for different types of documents to provide information support for subsequent device alignment; The key technology of feature extraction is Transformer, which uses two technical means, relative position encoding and full word masking mechanism based on the electric power field dictionary, to enhance the ability to express relative position information and make the word-segmented corpus of the input model more accurate. The training sample construction reduces the semantic gap between image and text information to a certain extent.

4. According to the intelligent transformer operation and maintenance detection method based on multi-level neural network as claimed in claim 1, It is characterized in that The step (3) includes visual modality, language modality and cross-visual-language mixed modality; The network architecture design of the visual modality encoding and decoding layers includes two methods: Transformer and convolutional neural network; The network architecture design of the language modality encoding and decoding layers includes BERT technology and T5 technology; The network architecture design of the cross-visual language hybrid modal encoding and decoding layers includes two types of models: a single-tower fusion cross-modal model and a dual-tower fusion cross-modal model.

5. According to the intelligent transformer operation and maintenance detection method based on multi-level neural network as claimed in claim 1, It is characterized in that The step (4) includes pre-training of general knowledge masking, discrimination, substitution, etc. for transformer operation and inspection, and three-stage pre-training of transformer operation and inspection based on characters, phrases, and knowledge; The transformer operation and inspection general knowledge masking, discrimination, substitution and other pre-training include three types of training: text, multi-modal pre-training basis and operation and inspection multi-stage progressive pre-training; The three-stage pre-training of transformer operation and inspection based on characters, phrases and knowledge includes three stages and transformer operation and inspection cognitive pre-training integrating multiple basic strategies and multiple semantic levels.

6. According to claim 5, a smart transformer operation and maintenance detection method based on a multi-level neural network, It is characterized in that The text training includes MLM tasks and NSP tasks; The multimodal pre-training basis includes a text mask task, a mask image feature regression task, and a mask object classification task; The multi-stage progressive pre-training of operation and inspection includes a single-stream model and a dual-stream model.

7. According to claim 5, a method for intelligent transformer operation and maintenance detection based on a multi-level neural network, It is characterized in that The first stage is a task of randomly shuffling image features; The second stage is a sentence-based subject selection task; The third stage is an image-text matching task based on difficult samples; The transformer operation and inspection cognitive pre-training integrating multiple basic strategies and multiple semantic levels includes D-UniTAB technology and MQA technology.

8. According to claim 6, a smart transformer operation and maintenance detection method based on a multi-level neural network, It is characterized in that The MLM task reduces the bias of network learning, which is conducive to the text pre-training model learning the representation distribution of each input word context. The model restores the original category information according to the power equipment operation and inspection image features corresponding to the image end.

9. According to claim 7, a smart transformer operation and maintenance detection method based on a multi-level neural network, It is characterized in that The MQA technology includes six methods: visual QA, region-guided visual QA, position-guided visual QA, pure spatial QA, region-guided spatial QA and visual attribute-guided spatial QA.

Images