A multi-modal fault understanding and auxiliary labeling method for intelligent operation and maintenance of instruments and meters

Abstract

The application provides a kind of multi-modal fault understanding and auxiliary labeling method for intelligent operation and maintenance of instrument, steps are: collecting the image, sound and text data feedback when instrument fault, respectively to the image, audio and text data collected are preprocessed, the image data, audio data, text data after pre-processing are extracted, the image, audio, text features extracted are fused using multi-modal quantum heuristic neural network;Image, audio, text features extracted are used to train multi-modal quantum heuristic neural network model;The trained model is loaded into the instrument intelligent operation and maintenance platform backend, and the multi-modal fault data is transmitted into API, and the annotation set is returned after model inference;The application fuses image, speech, text features through multi-modal quantum heuristic deep network model, understands the fault of instrument through fused coding features, and automatically labels fault feedback information, and has strong practicability.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for instruments and meters, and in particular to a method for understanding and assisting in labeling multimodal faults in intelligent operation and maintenance of instruments and meters. Background Technology

[0002] With the rapid development of artificial intelligence technology, traditional industries are gradually transforming towards intelligent and unmanned operations, significantly improving social productivity and efficiency. Instrumentation is a fundamental facility ensuring the safe and reliable operation of factories. However, as factories expand, large-scale instrumentation equipment is being deployed on production lines. Maintaining this vast amount of equipment is a significant and challenging task. Fortunately, the rapid development of artificial intelligence technology has brought new opportunities for the intelligent operation and maintenance of instruments.

[0003] Looking at the current architecture of intelligent operation and maintenance platforms for instruments and meters, there is no efficient method for processing fault feedback information from the edge. Given that the feedback information received by the operation and maintenance platform terminal has multimodal characteristics, such as images, voice and text, and the received fault feedback information comes from different instruments or equipment, most of the current processing methods are manual classification and labeling, which obviously consumes a lot of manpower. Summary of the Invention

[0004] The purpose of this invention is to provide a method for understanding and assisting in labeling multimodal faults in intelligent operation and maintenance of instruments and meters.

[0005] The objective of this invention is achieved through the following technical solution, the specific steps of which are as follows:

[0006] 1) Data Acquisition: Collect image, sound, and text data fed back when instruments malfunction, and organize them into a dictionary {'image':[img1,img2,…],'voice':[v1,v2,…],'text':[t1,t2,…]}, where: image, voice, and text represent the three modalities of image, audio, and text, respectively;

[0007] 2) Data preprocessing: The image, audio and text data collected in step 1) are preprocessed respectively to obtain preprocessed image, audio and text data;

[0008] 3) Feature extraction: Image feature extraction is performed on the preprocessed image data in step 2) using the PatchConvNet model; audio feature extraction is performed on the preprocessed audio data in step 2) using one-dimensional dilated convolution; and text feature extraction is performed on the preprocessed text in steps 2-3) using a bidirectional recurrent network based on GRU units.

[0009] 4) Model Construction: The image features, audio features, and text features extracted in step 3) are fused using a multimodal quantum-inspired neural network to obtain fused encoded features;

[0010] 5) Model training: The image, audio, and text feature extraction model is trained using autoencoder training and multi-task training. The image, audio, and text features extracted in step 3) are used to train the multimodal quantum-inspired neural network model.

[0011] 6) Auxiliary annotation: The feature extraction network model of the three modal data and the multimodal quantum-inspired deep network model are loaded into the backend of the instrumentation intelligent operation and maintenance platform. The multimodal fault data of the instrumentation are input into the API. After model inference, the annotation set is returned to realize auxiliary annotation.

[0012] Furthermore, the specific steps of data preprocessing in step 2) are as follows:

[0013] 2-1) Adaptive cropping and square filling are performed on the image data collected in step 1), then the images are scaled and normalized to keep the image ratio unchanged, and finally the images in the list are deduplicated and filtered.

[0014] 2-2) The audio data collected in step 1) is denoised using a heuristic dual-optimization PCD algorithm, and the denoised audio segments are then centrally cropped or symmetrically filled.

[0015] 2-3) Based on a complete and professional word vector space, extract keywords from the text data collected in step 1), and convert the keywords into vector form to obtain a keyword vector matrix.

[0016] Furthermore, the specific steps for feature extraction in step 3) are as follows:

[0017] 3-1) Image Feature Extraction: Image data is input into the PatchConvNet model, recurrent forward propagation is used, and then the output is averaged and fused to obtain image features with a dimension of 512.

[0018]

[0019] In the formula, images represents image data, and N1 represents the number of image data segments in a single sample;

[0020] 3-2) Audio Feature Extraction: Audio features are extracted from the audio data processed in step 2) using a one-dimensional dilated convolution preprocessing method. The outputs are then averaged and fused to obtain the audio features.

[0021]

[0022] In the formula, voices represents audio data, and N2 represents the number of audio data segments in a single sample;

[0023] 3-3) Text Feature Extraction: A bidirectional recurrent network based on GRU units is used to extract features from the keyword vectors obtained in step 2), and the output of the last time step is used as the subsequent features.

[0024]

[0025] In the formula, texts is the keyword vector, and N3 is the number of keyword vectors in a single sample.

[0026] Furthermore, the specific steps for constructing the model in step 4) are as follows:

[0027] 4-1) Input the feature vectors of the three modes into the fully connected layer of the multimodal quantum-inspired neural network:

[0028]

[0029] In the formula, These represent image feature vectors, audio feature vectors, and text feature vectors, respectively.

[0030] Projecting the features of the three modalities into a d-dimensional space:

[0031] a j ,t j ,v j =transform(u j )

[0032] Where transform represents linear projection transformation;

[0033] The pure states of each modality are fused:

[0034] ρ j =λ a |a j > j |+λ t |t j > <t j |+λ v |v j > <v j |

[0035] In the formula, λ a ,λ t ,λ v These represent three modes respectively;

[0036] 4-2) The feature matrices of the three modes are fused using a quantum recurrent neural network (QRNN) model, and the density matrix sequence is then fused.​ Inputting into the QRNN network model, through the density matrix ρ of the hidden states h To represent the information of the sequence, the iterative calculation process of the time step is as follows:

[0037]

[0038]

[0039] In the formula, the update function f(·) is the parameterized unitary matrix U. x U h and real value λ; U x U h , All are unitary matrices defined in Hilbert space and satisfy UU H =I 2 , I is the identity matrix, t is the virtual time variable, and ρ t Let be the density matrix at time t. Let represent the hidden state density matrix at time t-1.

[0040] The density matrix is ​​obtained through Where |φ j ><φ j | indicates the ground state.

[0041] The output of the QRNN network model is the d-dimensional state of the entire sequence. That is, the fusion feature matrix of the three modalities;

[0042] 4-3) The state of each complete sequence is measured by a global observable. A d-dimensional unitary matrix M, i.e., a probability distribution matrix, is constructed by mutually orthogonal feature states to represent the probability of the state collapsing to the corresponding feature state.

[0043] 4-4) A d-dimensional probability distribution is obtained through step 4-3). The d-dimensional probability distribution vector is then passed through a fully connected layer to calculate the predicted probability of the corresponding label, thus obtaining the classification label.

[0044] e i =argmax(σ) k

[0045]

[0046] In the formula, x is the output of the previous neural node. Let s be the parameter matrix of the fully connected layer. j For the output of the j-th node, the σ function returns the class probability, where k represents the output dimension, and e i The category with the highest probability.

[0047] Furthermore, the specific steps for model training in step 5) are as follows:

[0048] 5-1) Use autoencoder training and multi-task training to train the image, audio, and text feature extraction model;

[0049] 5-2) Utilizing static features Complete the heuristic search process for the multimodal quantum-inspired neural network model, save the optimal unitary matrix, and then use the Adam algorithm to train the weight parameters of the fully connected layers of the model to obtain the trained multimodal quantum-inspired deep network model.

[0050] Furthermore, the specific steps for auxiliary annotation in step 6) are as follows:

[0051] 6-1) Load the trained feature extraction network model of the three modalities and the multimodal quantum-inspired deep network model into the backend of the instrumentation intelligent operation and maintenance platform;

[0052] 6-2) Check whether the input instrument multimodal fault data meets the format requirements:

[0053] {'image':[img1,img2,…],'voice':[v1,v2,…],'text':[t1,t2,…]},

[0054] If the format is correct, proceed to step 6-3); if the format is incorrect, process the multimodal fault data of the instrument into a format that meets the requirements before proceeding to step 6-3.

[0055] 6-3) Perform data preprocessing on the multimodal fault data of instruments and meters, and then input the processed multimodal fault data into the API of the operation and maintenance platform. After model inference, return the annotation set to complete the auxiliary annotation.

[0056] Because of the adoption of the above technical solution, the present invention has the following advantages:

[0057] 1. This invention fuses image, speech, and text features through a multimodal quantum-inspired deep network model, understands instrument faults by fusing coded features, and automatically labels and triages fault feedback information, making it highly practical.

[0058] 2. The multimodal quantum-inspired deep network model of this invention utilizes quantum computing to enhance the information processing capabilities of neural computing. Combining artificial neural networks with quantum theory can better simulate the information processing process of the human brain and achieve higher accuracy in multimodal information fusion.

[0059] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained from the following description and claims. Attached Figure Description

[0060] The accompanying drawings of this invention are described below.

[0061] Figure 1 This is a flowchart of the present invention.

[0062] Figure 2 This is a diagram of the PatchConvNet model used for image feature extraction in this invention.

[0063] Figure 3 This is a diagram of the one-dimensional dilated convolution model used for audio feature extraction in this invention.

[0064] Figure 4 This is a diagram of the multimodal quantum-inspired neural network model used for feature fusion in this invention.

[0065] Figure 5 This is a schematic diagram of the modality fusion process of the multimodal quantum-inspired neural network model of the present invention. Detailed Implementation

[0066] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0067] like Figure 1 The method for understanding and auxiliary annotation of multimodal faults in intelligent operation and maintenance of instruments and meters is shown below. The specific steps are as follows:

[0068] 1) Data Acquisition: Collect image, sound, and text data fed back when instruments malfunction, and organize them into a dictionary {'image':[img1,img2,…],'voice':[v1,v2,…],'text':[t1,t2,…]}, where: image, voice, and text represent the three modalities of image, audio, and text, respectively;

[0069] In this invention, the VALUE corresponding to the three modal data are all organized in list form. The lengths of the three lists are not aligned, and the minimum length is 0. The image list is a collection of images of the faulty equipment acquired from multiple angles and distances, and it has the characteristics of inconsistent image sizes, off-center target objects, and redundant edges. The sound list is a collection of abnormal audio segments of the faulty equipment, and it has the characteristics of strong noise and inconsistent audio segment durations. The text list is a collection of subjective textual descriptions of the fault phenomena by on-site inspection personnel, and it has the characteristics of non-standard text, unclear logic, and insufficient professional expression.

[0070] 2) Data Preprocessing: The image, audio, and text data acquired in step 1) are preprocessed to obtain preprocessed image, audio, and text data; the specific steps are as follows:

[0071] 2-1) Adaptively crop and fill the image data acquired in step 1), then scale and normalize the images while maintaining their aspect ratio, and finally filter out duplicate images from the list; the specific steps are as follows:

[0072] 2-1-1) An adaptive cropping algorithm is used to remove edge redundancy from the image data acquired in step 1):

[0073] Convert the original image to a grayscale image:

[0074] gray=(R*19595+G*38469+B*7472)>>16

[0075] In the formula, R, G, and B represent the original... Figure 3 A pixel matrix with multiple channels;

[0076] Binarization of the grayscale image is performed using an adaptive local thresholding method to obtain a binary image matrix:

[0077] IF:f(x,y)>T,gray(x,y)=255; Else:gray(x,y)=0

[0078] In the formula, T is the global threshold, and f(x,y) is the pixel value at image coordinates (x,y);

[0079] Search for the target boundary vector based on the binary image matrix [b] up ,b down ,b left ,b right Then, the original image is cropped based on the obtained boundary vectors. The shape of the cropped image is as follows:

[0080] shape(gray) = (b down -bup ,b right -b left );

[0081] 2-1-2) Using the long side of the cropped image from step 2-1-1) as a reference, perform symmetrical filling on the short side to obtain a square image with the object centered:

[0082] IF:w>h,shape(gray)=(w,w); Else:shape(gray)=(h,h)

[0083] In the formula, w and h represent the width and height of the image, respectively;

[0084] 2-1-3) Scale the filled image from step 2-1-2) to a standard size and normalize the pixel values ​​to the [0,1] range:

[0085]

[0086] In the formula, x i Represents the value of a pixel;

[0087] 2-1-4) Iterate through the image list after the above processing, calculate the pairwise cosine similarity, and based on the set similarity threshold T', keep only one sample from each pair whose cosine similarity is greater than the similarity threshold T':

[0088]

[0089] In the formula, A and B represent the pixel matrices of a pair of images.

[0090] 2-2) The audio data acquired in step 1) is denoised using a heuristic dual-optimization PCD algorithm, and the denoised audio segments are then centrally cropped or symmetrically filled; the specific steps are as follows:

[0091] 2-2-1) The parameters of the PCD algorithm are iteratively optimized using the differential evolution algorithm, and then the audio data is denoised based on the optimal parameter set. The objective function for this optimization is:

[0092]

[0093] In the formula, The log-likelihood term describes the relationship between the clean audio signal x and the noisy audio signal y; ρ(x) represents the prior signal of the unknown clean audio signal x.

[0094] 2-2-2) The denoised audio list is trimmed and padded one by one to ensure that the audio signals in the list have the same length. The processed audio can be represented as x = [x1, x2, ..., x...]. m ,pm+1 ,...,p n ], where m is the effective audio length, n is the standard length, and p is the padding value.

[0095] 2-3) Based on a complete and professional word vector space, extract keywords from the text data collected in step 1), and convert the keywords into vector form to obtain a keyword vector matrix.

[0096] In this invention example, the complete professional terminology database contains the names and models of all instruments and meters in use. A Word2Vector model is trained using this database and saved, resulting in a complete professional term vector space [v1, v2, v3, ..., v]. N ], where N represents the size of the word vector space, i.e., the number of all specialized words. Each vector corresponds to a specialized word, and the i-th word is represented as v. i =[v i1 ,...,v id ], where d represents the word vector dimension; the word vector matrix SHAPE is L*D, where L represents the number of keywords extracted from the sample, and D represents the vector dimension of each keyword.

[0097] 3) Feature Extraction: Image features are extracted from the preprocessed image data in step 2) using a PatchConvNet model; audio features are extracted from the preprocessed audio data in step 2) using a one-dimensional dilated convolution model; and text features are extracted from the preprocessed text in steps 2-3) using a bidirectional recurrent network based on GRU units. The specific steps are as follows:

[0098] 3-1) Image Feature Extraction: The single image sample data contains multiple images:

[0099] shape(images) = (N,C,W,H)

[0100] In the formula, N is the number of images in a single sample, and C, W, and H are the number of image channels, image width, and image height, respectively.

[0101] Image data is input into the PatchConvNet model, and a recurrent forward propagation is performed. The output is then averaged and fused to obtain image features with a dimension of 512.

[0102]

[0103] In the formula, images represents image data, and N1 represents the number of image data segments in a single sample;

[0104] In an example of the present invention, such as Figure 2As shown, the PatchConvNet model includes a convolutional network for segmenting image pixels and mapping them into a set of vectors. The output of the convolutional network is connected to a backbone module consisting of N stacked residual convolutional layers, a squeeze-and-excitation layer, and a 1*1 convolutional layer. Each of the N stacked residual convolutional layers has a normalization layer, a 1*1 convolution, and a 3*3 convolution for spatial processing. The preprocessed vectors at the output of the backbone module are fused using a Transformer-like cross attention layer. Each weight in the attention layer depends on the similarity between the predicted patch and the trainable vector (CLS). The resulting d-dimensional vector is then added to the CLS vector and processed by a feedforward network.

[0105] 3-2) Audio Feature Extraction: Audio features are extracted from the audio data processed in step 2) using a one-dimensional dilated convolution preprocessing method. The outputs are then averaged and fused to obtain the audio features.

[0106]

[0107] In the formula, voices represents audio data, and N2 represents the number of audio data segments in a single sample.

[0108] In an example of this invention, for a one-dimensional audio sequence input x∈R n And with filter f:{0,…,k-1}→R, the dilated convolution operation F of audio sequence elements s is: In the formula, d is the expansion factor, k is the filter size, and sd·i is the past direction. When the value of d is greater than 1, the output of the top layer can represent a larger range of inputs, thereby effectively expanding the receptive field of one-dimensional convolution.

[0109] 3-3) Text Feature Extraction: A bidirectional recurrent network based on GRU units is used to extract features from the keyword vectors obtained in step 2), and the output of the last time step is used as the subsequent features.

[0110]

[0111] In the formula, texts is the keyword vector, and N3 is the number of keyword vectors in a single sample.

[0112] 4) Model Construction: The image features, audio features, and text features extracted in step 3) are fused using a multimodal quantum-inspired neural network to obtain fused encoded features; the specific steps are as follows:

[0113] 4-1) Input the feature vectors of the three modes into the fully connected layer of the multimodal quantum-inspired neural network:

[0114]

[0115] In the formula, These represent image feature vectors, audio feature vectors, and text feature vectors, respectively.

[0116] Projecting the features of the three modalities into a d-dimensional space:

[0117] a j ,t j ,v j =transform(u j )

[0118] Where transform represents linear projection transformation;

[0119] The pure states of each modality are fused:

[0120] ρ j =λ a |a j > j |+λ t |t j > <t j |+λ v |v j > <v j |

[0121] In the formula, λ a ,λ t ,λ v These represent three modes respectively;

[0122] 4-2) The feature matrices of the three modes are fused using a quantum recurrent neural network (QRNN) model, and the density matrix sequence is then fused. Inputting into the QRNN network model, through the density matrix ρ of the hidden states h To represent the information of the sequence, the iterative calculation process of the time step is as follows:

[0123]

[0124]

[0125] In the formula, the update function f(·) is the parameterized unitary matrix U. x U h and real value λ; U x U h , All are unitary matrices defined in Hilbert space and satisfy UU​H =I 2 , I is the identity matrix, t is the virtual time variable, and ρ t Let be the density matrix at time t. Let represent the hidden state density matrix at time t-1.

[0126] The density matrix is ​​obtained through Where |φ j ><φ j | indicates the ground state.

[0127] The output of the QRNN network model is the d-dimensional state of the entire sequence. That is, the fusion feature matrix of the three modalities;

[0128] 4-3) The state of each complete sequence is measured by a global observable. A d-dimensional unitary matrix M, i.e., a probability distribution matrix, is constructed by mutually orthogonal feature states to represent the probability of the state collapsing to the corresponding feature state.

[0129] 4-4) A d-dimensional probability distribution is obtained through step 4-3). The d-dimensional probability distribution vector is then passed through a fully connected layer to calculate the predicted probability of the corresponding label, thus obtaining the classification label.

[0130] e i =argmax(σ) k

[0131]

[0132] In the formula, x is the output of the previous neural node. Let s be the parameter matrix of the fully connected layer. j For the output of the j-th node, the σ function returns the class probability, where k represents the output dimension, and e i The category with the highest probability.

[0133] In this invention example, during the heuristic search process, in order to satisfy the unitary constraint throughout the training process, the Riemann method is used to update the unitary matrix:

[0134]

[0135]

[0136] In the formula, G is the gradient, lr is the learning rate, I is the identity matrix, L is the cost loss, and X is the state matrix.

[0137] 5) Model Training: The image, audio, and text feature extraction model is trained using autoencoder training and multi-task training methods. The multimodal quantum-inspired neural network model is trained using the image, audio, and text features extracted in step 3). The specific steps are as follows:

[0138] 5-1) Train the image, audio, and text feature extraction models using autoencoder training and multi-task training methods. Treat the feature extraction network as an Encoder and build an auxiliary Decoder based on the Encoder structure:

[0139]

[0140]

[0141]

[0142] The Adam optimization algorithm was used to train the feature extraction networks for the three modalities.

[0143] The feature extraction network is trained using a multi-task training method, and a task adaptation layer is added to the feature extraction network. The last layer of the classification task is the classifier layer, and the output is represented as p(y|x)=softmax(f), where f represents the output layer features and the dimension is equal to the number of classification categories.

[0144] 5-2) Utilizing static features Complete the heuristic search process for the multimodal quantum-inspired neural network model, save the optimal unitary matrix, and then use the Adam algorithm to train the weight parameters of the fully connected layers of the model to obtain the trained multimodal quantum-inspired deep network model.

[0145] 6) Auxiliary Labeling: The trained feature extraction network model for the three modalities and the multimodal quantum-inspired deep network model are loaded into the backend of the instrumentation intelligent operation and maintenance platform. Multimodal fault data from the instruments is input into the API. After model inference, a label set is returned to achieve auxiliary labeling. The specific steps are as follows:

[0146] 6-1) Load the trained feature extraction network model of the three modalities and the multimodal quantum-inspired deep network model into the backend of the instrumentation intelligent operation and maintenance platform;

[0147] 6-2) Check whether the input instrument multimodal fault data meets the format requirements:

[0148] {'image':[img1,img2,…],'voice':[v1,v2,…],'text':[t1,t2,…]},

[0149] If the format is correct, proceed to step 6-3); if the format is incorrect, process the multimodal fault data of the instrument into a format that meets the requirements before proceeding to step 6-3.

[0150] 6-3) Perform data preprocessing on the multimodal fault data of instruments and meters, and then input the processed multimodal fault data into the API of the operation and maintenance platform. After model inference, return the annotation set to complete the auxiliary annotation.

[0151] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for understanding and auxiliary labeling multimodal faults in intelligent operation and maintenance of instruments and meters, characterized in that, The specific steps are as follows: 1) Data Acquisition: Collect image, sound, and text data fed back when instruments malfunction, and organize them into a dictionary {'image':[img1,img2,…],'voice':[v1,v2,…],'text':[t1,t2,…]}, where: image, voice, and text represent the three modalities of image, audio, and text, respectively; 2) Data preprocessing: The image, audio and text data collected in step 1) are preprocessed respectively to obtain preprocessed image, audio and text data; 3) Feature extraction: Image feature extraction is performed on the preprocessed image data in step 2) using the PatchConvNet model; audio feature extraction is performed on the preprocessed audio data in step 2) using one-dimensional dilated convolution; and text feature extraction is performed on the preprocessed text in steps 2-3) using a bidirectional recurrent network based on GRU units. 4) Model Construction: The image features, audio features, and text features extracted in step 3) are fused using a multimodal quantum-inspired neural network to obtain fused encoded features; 5) Model training: The image, audio, and text feature extraction model is trained using autoencoder training and multi-task training. The image, audio, and text features extracted in step 3) are used to train the multimodal quantum-inspired neural network model. 6) Auxiliary annotation: The trained feature extraction network model of three modal data and the multimodal quantum-inspired deep network model are loaded into the backend of the instrumentation intelligent operation and maintenance platform. The multimodal fault data of the instrumentation is input into the API, and after model inference, the annotation set is returned to realize auxiliary annotation. The specific steps of data preprocessing in step 2) are as follows: 2-1) Adaptive cropping and square filling are performed on the image data collected in step 1), then the images are scaled and normalized to keep the image ratio unchanged, and finally the images in the list are deduplicated and filtered. 2-2) The audio data collected in step 1) is denoised using a heuristic dual-optimization PCD algorithm, and the denoised audio segments are then centrally cropped or symmetrically filled. 2-3) Based on a complete and professional word vector space, extract keywords from the text data collected in step 1), and convert the keywords into vector form to obtain a keyword vector matrix; The specific steps for feature extraction in step 3) are as follows: 3-1) Image Feature Extraction: Image data is input into the PatchConvNet model, recurrent forward propagation is used, and then the output is averaged and fused to obtain image features with a dimension of 512. In the formula, images represents image data, and N1 represents the number of image data segments in a single sample; 3-2) Audio Feature Extraction: Audio features are extracted from the audio data processed in step 2) using a one-dimensional dilated convolution preprocessing method. The outputs are then averaged and fused to obtain the audio features. In the formula, voices represents audio data, and N2 represents the number of audio data segments in a single sample; 3-3) Text Feature Extraction: A bidirectional recurrent network based on GRU units is used to extract features from the keyword vectors obtained in step 2), and the output of the last time step is used as the subsequent features. In the formula, texts is the keyword vector, and N3 is the number of keyword vectors in a single sample; The specific steps for building the model in step 4) are as follows: 4-1) Input the feature vectors of the three modes into the fully connected layer of the multimodal quantum-inspired neural network: In the formula, These represent image feature vectors, audio feature vectors, and text feature vectors, respectively. Projecting the features of the three modalities into a d-dimensional space: a j ,t j ,v j =transform(u j ) Where transform represents linear projection transformation; The pure states of each modality are fused: p j =λ a |a j ><a j |+λ t |t j ><t j |+λ v |v j ><v j | In the formula, a j ,t j ,v j λ represents the three modalities: audio, text, and visual. a ,λ t ,λ v These are fusion coefficients, all of which are non-negative and sum to 1; 4-2) The feature matrices of the three modes are fused using a quantum recurrent neural network (QRNN) model, and the density matrix sequence is then fused. Inputting into the QRNN network model, through the density matrix ρ of the hidden states h To represent the information of the sequence, the iterative calculation process of the time step is as follows: In the formula, the update function f(·) is the parameterized unitary matrix U. x U h and real value λ; U x U h , All are unitary matrices defined in Hilbert space and satisfy UU H =I 2 , I is the identity matrix, t is the virtual time variable, and ρ t Let be the density matrix at time t. Let represent the hidden state density matrix at time t-1; The density matrix is ​​obtained through Where |φ j > < φ j | indicates the ground state; The output of the QRNN network model is the d-dimensional state of the entire sequence. That is, the fusion feature matrix of the three modalities; 4-3) The state of each complete sequence is measured by a global observable. A d-dimensional unitary matrix M, i.e., a probability distribution matrix, is constructed by mutually orthogonal feature states to represent the probability of the state collapsing to the corresponding feature state. 4-4) A d-dimensional probability distribution is obtained through step 4-3). The d-dimensional probability distribution vector is then passed through a fully connected layer to calculate the predicted probability of the corresponding label, thus obtaining the classification label. e i =argmax(σ) k In the formula, x is the output of the previous neural node. Let s be the parameter matrix of the fully connected layer. j For the output of the j-th node, the σ function returns the class probability, where k represents the output dimension, and e i The category with the highest probability.

2. The method for multimodal fault understanding and auxiliary annotation in intelligent operation and maintenance of instruments and meters as described in claim 1, characterized in that, The specific steps for model training in step 5) are as follows: 5-1) Use autoencoder training and multi-task training to train the image, audio, and text feature extraction model; 5-2) Utilizing static features Complete the heuristic search process for the multimodal quantum-inspired neural network model, save the optimal unitary matrix, and then use the Adam algorithm to train the weight parameters of the fully connected layers of the model to obtain the trained multimodal quantum-inspired deep network model.

3. The method for multimodal fault understanding and auxiliary annotation in intelligent operation and maintenance of instruments and meters as described in claim 2, characterized in that, The specific steps for auxiliary annotation in step 6) are as follows: 6-1) Load the trained feature extraction network model of the three modalities and the multimodal quantum-inspired deep network model into the backend of the instrumentation intelligent operation and maintenance platform; 6-2) Check whether the input instrument multimodal fault data meets the format requirements: {'image':[img1,img2,…],'voice':[v1,v2,…],'text':[t1,t2,…]}, If the format is correct, proceed to step 6-3); if the format is incorrect, process the multimodal fault data of the instrument into a format that meets the requirements before proceeding to step 6-3. 6-3) Perform data preprocessing on the multimodal fault data of instruments and meters, and then input the processed multimodal fault data into the API of the operation and maintenance platform. After model inference, return the annotation set to complete the auxiliary annotation.

Images