Nuclear power operation and maintenance data cross-modal retrieval method, device, equipment and medium

Abstract

The invention relates to a nuclear power operation and maintenance data cross-modal retrieval method, device and equipment and a medium, and belongs to the technical field of nuclear power operation and maintenance management.The nuclear power operation and maintenance data cross-modal retrieval method comprises the steps that nuclear power operation and maintenance image data and text data are obtained; performing feature extraction on the image data and the text data, performing modal similarity calculation on image features, global pairing text features, pairing generated text features and non-pairing generated texts to obtain a similarity distance, determining a target loss function based on the similarity distance, and adjusting parameters of a cross-modal retrieval model by using the target loss function to obtain a cross-modal retrieval result. Inputting to-be-retrieved operation and maintenance data into the trained cross-modal retrieval model to obtain an image feature matrix and a text feature matrix, performing quantification processing on the image feature matrix and the text feature matrix to obtain a hash code, performing retrieval based on the hash code to obtain a text or image corresponding to the hash code, and storing the text or image in the cross-modal retrieval model. And the retrieval precision of the nuclear power operation and maintenance data is improved.

Description

Technical Field

[0001] This invention relates to the field of nuclear power operation and maintenance management technology, and in particular to a method, apparatus, equipment and medium for cross-modal retrieval of nuclear power operation and maintenance data. Background Technology

[0002] Nuclear power plant operation and maintenance processes accumulate a large amount of multimodal data. This data is usually from complex sources, is massive in quantity, and has problems such as "heterogeneous gap" and "semantic gap". Traditional single-modal retrieval, whether based on text or image, strictly limits the data types of query items and retrieval results to be the same. However, this retrieval method has certain limitations and is difficult to meet the diverse retrieval needs of operation and maintenance personnel. Compared with single-modal retrieval, cross-modal hash retrieval has received widespread attention due to its advantages such as low storage cost and fast query speed.

[0003] Most existing cross-modal hash retrieval methods typically employ a "one-step hashing" learning strategy, which considers both the training of hash codes and hash functions simultaneously during model training. While this learning strategy is intuitive, it has certain limitations. It does not adequately mine the correlations between multimodal data during hash learning, and it lacks integration of intramodal consistency and intermodal complementarity. This leads to significant information loss in the quantization process of hash codes, reducing their quality and consequently affecting retrieval accuracy. Summary of the Invention

[0004] In view of this, it is necessary to provide a method, device, equipment and medium for cross-modal retrieval of nuclear power operation and maintenance data to solve the technical problem of low retrieval accuracy of nuclear power operation and maintenance data.

[0005] To address the aforementioned problems, in a first aspect, the present invention provides a method for cross-modal retrieval of nuclear power plant operation and maintenance data, comprising: Acquire historical data on the operation and maintenance of multimodal nuclear power plants, including image data and text data; Feature extraction is performed on the image data and text data to obtain image features and text features. The text features include global paired text features corresponding to the image features, as well as paired generated text features and unpaired generated text features generated by a text variational autoencoder. The image features are compared with the global paired text features and the unpaired generated text features to calculate intermodal similarity, obtaining a first similarity distance and a third similarity distance. The paired generated text features are compared with the unpaired generated text features to calculate intramodal similarity, obtaining a second similarity distance. A target loss function is determined based on the first similarity distance, the second similarity distance, and the third similarity distance. The parameters of the cross-modal retrieval model are adjusted using the target loss function to obtain a trained cross-modal retrieval model. The operation and maintenance data to be retrieved is input into the trained cross-modal retrieval model to obtain image feature matrix and text feature matrix. The image feature matrix and text feature matrix are quantized to obtain hash code. Based on the hash code, the text or image in the database is retrieved to obtain the text or image corresponding to the hash code.

[0006] In one possible implementation, the cross-modal retrieval model includes an image feature extraction network, a text feature extraction network, and a text variational autoencoder; the feature extraction of the image data and text data to obtain image features and text features includes: The image data is preprocessed to obtain the original image, the main image, and the local images; The original image, the main image, and the local image are input into the image feature extraction network to obtain global image features, main image features, and local image features. The global image features, main image features, and local image features are then weighted and fused to obtain image features. Based on the image data and text data, the paired text, unpaired text, and unpaired local keyword text of the original image are obtained. The paired text, unpaired text, and unpaired local keyword text are input into the text feature extraction network to obtain global paired text features, unpaired global text features, and unpaired local text features. The unpaired global text features and unpaired local text features are weighted and fused to obtain unpaired text features. The global paired text features and unpaired text features are input into the text variational autoencoder to obtain paired generated text features and unpaired generated text features.

[0007] In one possible implementation, the network architecture of the image feature extraction network is a Fast-RCNN network, which includes an input layer, a convolutional backbone network, a region of interest pooling layer, a fully connected layer, and a parallel output layer; the network architecture of the text feature extraction network is a BERT model, which includes an input layer, a multi-layer Transformer encoder, and a pre-trained output layer; the text variational autoencoder includes an encoder, a latent space, and a decoder.

[0008] In one possible implementation, the step of performing inter-modal similarity calculations between the image features and the globally paired text features and the unpaired generated text features to obtain a first similarity distance and a third similarity distance includes: The image features are mapped to the global paired text features through a cross-modal attention mechanism, and the cosine similarity between the mapped image features and the global paired text features is calculated to obtain the first similarity distance. The image features are mapped to the unpaired generated text features using a cross-modal attention mechanism. The cosine similarity between the mapped image features and the unpaired generated text features is then calculated to obtain the third similarity distance.

[0009] In one possible implementation, the step of calculating the intra-modal similarity between the paired generated text features and the unpaired generated text to obtain the second similarity distance includes: The Euclidean distance between the paired generated text features and the unpaired generated text is calculated to obtain the second similarity distance.

[0010] In one possible implementation, the target loss function is: , in, Let be the target loss function. The first similarity distance, For the second similarity distance, The third similarity distance, , These are the weighting coefficients.

[0011] In one possible implementation, the step of quantizing the image feature matrix and the text feature matrix to obtain a hash code, and then retrieving text or images from the database based on the hash code to obtain the text or image corresponding to the hash code, includes: The image feature matrix and the text feature matrix are binarized respectively to obtain the binary image feature matrix and the binary text feature matrix; The binary image feature matrix and the binary text feature matrix are ORed to obtain a fused binary matrix. The fused binary matrix is ​​then encoded to obtain a hash code. Obtain the hash code of text or image in the database, calculate the distance between the hash code and the hash code of the text or image to obtain the hash code of the nearest neighbor in the database, and obtain the text or image corresponding to the hash code based on the hash code of the nearest neighbor.

[0012] Secondly, the present invention also provides a nuclear power plant operation and maintenance data cross-modal retrieval device, comprising: The data acquisition module is used to acquire historical data on the operation and maintenance of multimodal nuclear power plants, which includes image data and text data. The feature extraction module is used to extract features from the image data and text data based on the constructed cross-modal retrieval model to obtain image features and text features. The text features include global paired text features corresponding to the image features, as well as paired generated text features and unpaired generated text features generated by the text variational autoencoder. An adjustment module is used to perform inter-modal similarity calculations between the image features and the global paired text features and the unpaired generated text features to obtain a first similarity distance and a third similarity distance; to perform intra-modal similarity calculations between the paired generated text features and the unpaired generated text features to obtain a second similarity distance; to determine a target loss function based on the first similarity distance, the second similarity distance, and the third similarity distance; and to adjust the parameters of the cross-modal retrieval model using the target loss function to obtain a trained cross-modal retrieval model. The retrieval module is used to input the operation and maintenance data to be retrieved into the trained cross-modal retrieval model to obtain image feature matrices and text feature matrices, perform quantization processing on the image feature matrices and text feature matrices to obtain hash codes, and retrieve text or images in the database based on the hash codes to obtain the text or images corresponding to the hash codes.

[0013] Thirdly, the present invention also provides an electronic device, comprising: a processor and a memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the cross-modal retrieval method for nuclear power plant operation and maintenance data as described above.

[0014] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, wherein when the program or instruction is executed by a processor, it is able to implement the steps in the nuclear power plant operation and maintenance data cross-modal retrieval method described in any one of the above method items.

[0015] The beneficial effects of this invention are as follows: Feature extraction is performed on image data and text data to obtain image features and text features. By extracting image features and text features, the accuracy of mutual retrieval between text and images is improved. Intermodal similarity calculations are performed between image features and globally paired text features and unpaired generated text features to obtain a first similarity distance and a third similarity distance. Intramodal similarity calculations are performed between paired generated text features and unpaired generated text features to obtain a second similarity distance. A target loss function is determined based on the first, second, and third similarity distances. The parameters of the cross-modal retrieval model are adjusted using the target loss function to obtain a trained cross-modal retrieval model. The target loss function utilizes graph... The inter-modal and intra-modal similarity constraints between images and text align inter-modal similarities, thereby learning a better cross-modal retrieval model. Through inter-modal and intra-modal similarity constraint mechanisms, the semantic alignment capability between images and text is improved, achieving more efficient and accurate cross-modal retrieval. The operation and maintenance data to be retrieved is input into the trained cross-modal retrieval model to obtain image feature matrices and text feature matrices. The image feature matrices and text feature matrices are quantized to obtain hash codes. Based on the hash codes, the text or images in the database are retrieved to obtain the text or images corresponding to the hash codes. By extracting features through the trained cross-modal retrieval model and retrieving through hash codes, the retrieval accuracy of nuclear power operation and maintenance data is improved. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of an embodiment of the cross-modal retrieval method for nuclear power plant operation and maintenance data provided by the present invention; Figure 2 A schematic diagram of the Fast-RCNN network structure of the cross-modal retrieval method for nuclear power operation and maintenance data provided by the present invention; Figure 3 This is a schematic diagram of the BERT model for the cross-modal retrieval method for nuclear power plant operation and maintenance data provided by the present invention. Figure 4This is a schematic diagram of the text variational autoencoder used in the cross-modal retrieval method for nuclear power plant operation and maintenance data provided by the present invention. Figure 5 A schematic diagram of an embodiment of the nuclear power plant operation and maintenance data cross-modal retrieval device provided by the present invention; Figure 6 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0018] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0019] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] This invention discloses a method, apparatus, device, and medium for cross-modal retrieval of nuclear power plant operation and maintenance data, which can be used in a computer. The method, apparatus, or computer-readable storage medium involved in this invention can be integrated with the aforementioned apparatus or can be relatively independent.

[0021] A specific embodiment of the present invention discloses a cross-modal retrieval method for nuclear power plant operation and maintenance data, which can be executed by a computer, specifically by one or more processors of the computer. For example... Figure 1 As shown, the cross-modal retrieval method for nuclear power plant operation and maintenance data includes: S101. Obtain historical data on the operation and maintenance of multimodal nuclear power plants. The historical data on the operation and maintenance of multimodal nuclear power plants includes image data and text data. It should be noted that historical nuclear power plant operation and maintenance data includes maintenance plans, training documents, manufacturer documents, equipment manuals, meeting documents, summary reports, risk analysis sheets, experience feedback, spare parts information, basic equipment information, work orders, equipment operation data, and other image and text data.

[0022] S102. Perform feature extraction on image data and text data to obtain image features and text features. The text features include global paired text features corresponding to image features, as well as paired generated text features and unpaired generated text features generated by a text variational autoencoder. It should be noted that the cross-modal retrieval model includes an image feature extraction network, a text feature extraction network, and a text variational autoencoder. The image feature extraction network extracts image features, and the text feature extraction network and the text variational autoencoder extract text features.

[0023] S103. Perform intermodal similarity calculations between the image features and the global paired text features and the unpaired generated text features to obtain the first similarity distance and the third similarity distance. Perform intramodal similarity calculations between the paired generated text features and the unpaired generated text features to obtain the second similarity distance. Determine the target loss function based on the first similarity distance, the second similarity distance, and the third similarity distance. Use the target loss function to adjust the parameters of the cross-modal retrieval model to obtain the trained cross-modal retrieval model. It should be noted that the objective loss function utilizes similarity constraints between and within modes, aligning the similarity between modes, thereby learning a better image feature extraction network and a text variational autoencoder network.

[0024] S104. Input the maintenance data to be retrieved into the trained cross-modal retrieval model to obtain the image feature matrix and the text feature matrix. Quantize the image feature matrix and the text feature matrix to obtain the hash code. Retrieve the text or image in the database based on the hash code to obtain the text or image corresponding to the hash code.

[0025] In some embodiments, in step S101, multimodal nuclear power plant operation and maintenance historical data is acquired. This multimodal nuclear power plant operation and maintenance historical data includes image data and text data. A large amount of historical operation and maintenance data from the nuclear power plant is acquired, including maintenance plans, training documents, manufacturer documents, equipment manuals, meeting documents, summary reports, risk analysis sheets, experience feedback, spare parts information, equipment basic information, work orders, equipment operation data, and other image and text data. Preprocessing operations are performed on all nuclear power plant operation and maintenance historical data, including data cleaning, format conversion, desensitization, classification, and modality supplementation. Specifically, the preprocessing steps are as follows: Data cleaning: In nuclear power plant DCS operation and maintenance, data sources are complex and may include data streams from different devices, sensors, and systems. Data cleaning technology ensures data consistency and quality by identifying and repairing erroneous, incomplete, or inaccurate parts of the data. Automated data cleaning can eliminate outliers or erroneous data to ensure the accuracy of subsequent analysis and decision-making. Security and reliability; Data format conversion: Data in nuclear power DCS systems may be stored and transmitted in different formats or structures. Data format conversion technology can unify data from various formats into the standard format required by the nuclear power system, so as to effectively integrate and analyze data, ensure data interoperability and consistency, and enable information from different data sources to be effectively utilized by the system; Data anonymization: When processing nuclear power DCS data, especially when it involves security and privacy-sensitive information, data anonymization technology plays a key role. Sensitive data is processed through obfuscation, encryption or other methods to protect personal privacy and corporate secrets, and ensure the security and compliance of data in the analysis and sharing process; Data classification and modal supplementation: Data is divided into image data and text data. Text information is added to all image data, and image data and corresponding text data are represented as image and paired text data. Other text data without image data is unpaired text data.

[0026] In some embodiments, in step S102, feature extraction is performed on the image data and text data to obtain image features and text features. The text features include global paired text features corresponding to the image features, paired generated text features generated by a text variational autoencoder, and unpaired generated text features. The cross-modal retrieval model includes an image feature extraction network, a text feature extraction network, and a text variational autoencoder. The image data is preprocessed to obtain an original image, a main image, and local images. A main image containing the foreground target region and randomly cropped local images are extracted from the original image. The original image, main image, and local images are input into the image feature extraction network to obtain global image features, main image features, and local image features, respectively. The global image features, main image features, and local image features are weighted and fused to obtain image features, i.e., the global features, main image features, and local image features are weighted and fused in a 4:2:1 ratio. The network architecture of the image feature extraction network is a Fast-RCNN network. For a schematic diagram of the Fast-RCNN network structure, please refer to [link to relevant documentation]. Figure 2 ,like Figure 2 As shown, the Fast-RCNN network includes an input layer, a convolutional backbone, region-of-interest (ROI) pooling layers, fully connected layers, and a parallel output layer. The input layer of the Fast-RCNN network consists of two parts: the entire image to be processed and a set of target proposals. The image is input into the convolutional backbone, where multiple convolutional pooling operations are performed on the entire input image to obtain a convolutional feature map. For each target proposal, the Fast-RCNN network extracts several candidate regions from the convolutional feature map through the RIO pooling layers. Since there are multiple candidate regions, the RIO pooling layers judge these candidate regions, select the region of interest, and extract a fixed-length feature vector from the convolutional feature map. Each feature vector is fed into the fully connected layer, where these features are combined, and then fed from the fully connected layer to the parallel output layer. The parallel output layer has two peer outputs: softmax and bounding box regression. The softmax function outputs the softmax probability estimate of the object category corresponding to each RIO; the bounding box regression... regression outputs four real values ​​for each object category, with each set of four real values ​​corresponding to the precise bounding box position for each category; The purpose of image feature extraction is to extract valuable information from raw image data, and then transform this information into a feature vector representation that computers can understand and analyze. Inputting the original image, the main image, and randomly cropped local images into the image feature extraction network will extract more accurate and complete image feature information, thereby improving the accuracy of text and image retrieval. Based on image and text data, paired text, unpaired text, and unpaired local keyword text of the original image are obtained. These are then input into a text feature extraction network to obtain global paired text features, global unpaired text features, and unpaired local text features. The unpaired global and local text features are then weighted and fused in a 2:1 ratio to obtain the unpaired text features. The network architecture of this text feature extraction network is the BERT model. For a schematic diagram of the BERT model, please refer to [link / reference needed]. Figure 3 ,like Figure 3 As shown, the BERT model consists of an input layer, a multi-layer Transformer encoder, and a pre-trained output layer. The input of the BERT model is composed of word embeddings, segment embeddings, and position embeddings, which can convert basic text units into vectors containing semantics, sentence boundaries, and word order. Each layer of the multi-layer Transformer encoder contains two sub-layers: a self-attention mechanism and a feedforward neural network. The former is used to capture global semantic associations, and the latter is used to achieve non-linear feature transformation. Each Transformer encoder layer is paired with layer normalization and residual connections for stable training. The pre-trained output layer contains a masked language model head and a next-sentence prediction head. The masked language model head is used to predict the masked text units, and the next-sentence prediction head is used to determine sentence coherence. The masked language model head and the next-sentence prediction head optimize the model parameters through unsupervised tasks. Text feature extraction aims to extract key information from raw text data and then transform this information into feature vectors that computers can recognize and process. Inputting local keyword texts of paired text, unpaired text, and unpaired text into the text feature extraction network is to extract more accurate and complete text feature information, thereby improving the accuracy of text and image retrieval. Global paired and unpaired text features are input into a text variational autoencoder to obtain paired and unpaired generated text features. The global paired text features and the weighted fused unpaired text features are then fed into the text variational autoencoder to learn and generate paired and unpaired generated text features, respectively. For a schematic diagram of the text variational autoencoder structure, please refer to [link to schematic diagram]. Figure 4 ,like Figure 4As shown, the text variational autoencoder includes an encoder, a latent space, and a decoder. The encoder transforms a discrete text sequence into continuous probability distribution parameters. The latent space is the core of the text variational autoencoder. It achieves probability sampling through the "reparameterization technique" and uses KL divergence to constrain the latent distribution to approximate a standard normal distribution, ensuring the continuity and interpretability of the latent space. The decoder maps the latent vectors back to the discrete text sequence to reconstruct the text sequence. Text variational autoencoders are deep probabilistic models designed specifically for text data generation and feature learning. Their core is to combine the probabilistic modeling capabilities of variational autoencoders with the characteristics of text data. While learning text features, text variational autoencoders map the text features to a continuous and interpretable latent space, thus achieving the task of text feature generation.

[0027] In some embodiments, in step S103, intermodal similarity calculations are performed between image features and both global paired text features and unpaired generated text features to obtain a first similarity distance and a third similarity distance. Intramodal similarity calculations are then performed between paired generated text features and unpaired generated text features to obtain a second similarity distance. A target loss function is determined based on the first, second, and third similarity distances. Intermodal similarity calculations are then performed between image features and global paired text features to obtain the first similarity distance. That is, a cross-modal attention mechanism is used to map image features to global paired text features, and cosine similarity calculations are performed between the mapped image features and global paired text features to obtain... The first similarity distance is calculated by comparing the intermodal similarity between image features and globally paired text features, denoted as similarity distance 1. This distance quantifies the content matching degree of paired features in the image and text modalities. Globally paired text refers to text that has a clear association with the image. Due to the modal gap between image and text features, it is necessary to first map them to a unified dimensional space through linear layers, cross-modal attention, etc., before calculating indices such as cosine distance or Euclidean distance. The smaller the calculated similarity distance, the better the indices, indicating a better match between the image and text content. The second similarity distance is obtained by calculating the intramodal similarity between paired generated text features and unpaired generated text. The Euclidean distance of the generated text is used to obtain the second similarity distance. The intra-modal similarity between paired and unpaired generated text features is then calculated, denoted as similarity distance 2, or the second similarity distance. This distance quantifies the semantic association between two types of generated text features within the same text modality. Paired generated text refers to generated text associated with the image, while unpaired generated text refers to generated text unrelated to the image. Since both paired and unpaired generated text features are within the text modality, their feature dimensions and distributions are more consistent, eliminating the need for complex cross-modal mapping. Cosine distance or Euclidean distance can be directly calculated in the text feature space. Finally, inter-modal similarity calculation is performed between image features and unpaired generated text to obtain the third similarity distance. Distance; calculate the intermodal similarity between image features and unpaired generated text features, denoted as similarity distance 3, or the third similarity distance, which is directly proportional to similarity distance 2. This distance is used to quantify the content mismatch between unpaired features in the image modality and the text modality, where unpaired generated text refers to generated text that is not related to the current image. Since there is a modal gap between image features and text features, it is necessary to first map the image features and unpaired generated text features to a unified dimensional space through a linear layer or cross-modal attention mechanism, and then calculate indicators such as cosine similarity or Euclidean distance. The larger the calculated similarity distance, the better the indicator, indicating that the model can effectively distinguish unrelated images and text and avoid incorrectly associating irrelevant content. The target loss function is determined based on the first, second, and third similarity distances. The parameters of the cross-modal retrieval model are then adjusted using this target loss function to obtain a trained cross-modal retrieval model. Since the order of magnitude difference between inter-modal and intra-modal similarity distances is unavoidable, and the first and third similarity distances can differ significantly, to improve retrieval accuracy, the target loss function is optimized by minimizing the target loss function as follows: , in, Let be the target loss function. The first similarity distance, For the second similarity distance, The third similarity distance, , As the weighting coefficient, the initial weighting coefficient is set to... , The parameters of the cross-modal retrieval model are adjusted using a target loss function, which can be the weights of the loss function. The target loss function utilizes inter-modal and intra-modal similarity constraints to align the similarity between modalities, thereby learning a better image feature extraction network and a text variational autoencoder network, thus obtaining a well-trained cross-modal retrieval model.

[0028] In some embodiments, in step S104, the maintenance data to be retrieved is input into a trained cross-modal retrieval model to obtain an image feature matrix and a text feature matrix. The image feature matrix and text feature matrix are then quantized to obtain a hash code. Based on the hash code, text or images in the database are retrieved to obtain the text or image corresponding to the hash code. The trained cross-modal retrieval model is used to extract features from the maintenance data to be retrieved, obtaining image features and text features. The image features are quantized to obtain a quantized image feature matrix, and the text features are quantized to obtain a quantized text feature matrix. The quantized image feature matrix and text feature matrix are then compared. The feature matrix is ​​ORed to obtain a hash code. Specifically, the image feature matrix and the text feature matrix are binarized to obtain binary image feature matrix and binary text feature matrix, respectively. The binary image feature matrix and the binary text feature matrix are then ORed to obtain a fused binary matrix. After sorting the fused binary matrix, the final hash code is obtained, which is the hash code to be retrieved. The hash codes of the text or image in the database are obtained. The distance between the hash code to be retrieved and the hash codes of the text or image in the database is calculated to obtain the nearest neighbor hash code, and the text or image corresponding to the final hash code is obtained.

[0029] In summary, the cross-modal retrieval method for nuclear power plant operation and maintenance data provided by this invention acquires multimodal historical nuclear power plant operation and maintenance data, including image data and text data. Feature extraction is performed on the image data and text data to obtain image features and text features. The text features include globally paired text features corresponding to the image features, as well as paired generated text features and unpaired generated text features generated by a text variational autoencoder. Intermodal similarity calculations are performed between the image features and the globally paired text features and the unpaired generated text features to obtain a first similarity distance and a third similarity distance. The paired generated text features are then compared with the unpaired generated text features. Paired text features are generated for intra-modal similarity calculation to obtain a second similarity distance. Based on the first, second, and third similarity distances, a target loss function is determined. The parameters of the cross-modal retrieval model are adjusted using the target loss function to obtain a trained cross-modal retrieval model. The operation and maintenance data to be retrieved is input into the trained cross-modal retrieval model to obtain image feature matrices and text feature matrices. The image feature matrices and text feature matrices are quantized to obtain hash codes. Based on the hash codes, text or images in the database are retrieved to obtain the text or image corresponding to the hash codes, thus improving the retrieval accuracy of nuclear power operation and maintenance data.

[0030] To better implement the cross-modal retrieval method for nuclear power plant operation and maintenance data in this embodiment of the invention, based on the cross-modal retrieval method for nuclear power plant operation and maintenance data, the corresponding method is as follows: Figure 5 As shown, this embodiment of the invention also provides a nuclear power plant operation and maintenance data cross-modal retrieval device 500, comprising: The data acquisition module 501 is used to acquire historical data of multimodal nuclear power plant operation and maintenance, which includes image data and text data. The feature extraction module 502 is used to extract features from image data and text data to obtain image features and text features. The text features include global paired text features corresponding to image features, as well as paired generated text features and unpaired generated text features generated by a text variational autoencoder. The adjustment module 503 is used to perform intermodal similarity calculations between image features and global paired text features and unpaired generated text features to obtain a first similarity distance and a third similarity distance; to perform intramodal similarity calculations between paired generated text features and unpaired generated text features to obtain a second similarity distance; to determine a target loss function based on the first similarity distance, the second similarity distance, and the third similarity distance; and to adjust the parameters of the cross-modal retrieval model using the target loss function to obtain a trained cross-modal retrieval model. The retrieval module 504 is used to input the operation and maintenance data to be retrieved into the trained cross-modal retrieval model to obtain the image feature matrix and the text feature matrix, perform quantization processing on the image feature matrix and the text feature matrix to obtain the hash code, and retrieve the text or image in the database based on the hash code to obtain the text or image corresponding to the hash code.

[0031] like Figure 6 As shown, the present invention also provides an electronic device 600, which can be a mobile terminal, desktop computer, laptop, handheld computer, server, or other computing device. The electronic device 600 includes a processor 601, a memory 602, and a display 603. Figure 6 Only some components of the electronic device 600 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0032] In some embodiments, memory 602 may be an internal storage unit of the electronic device 600, such as a hard disk or memory of the electronic device 600. In other embodiments, memory 602 may be an external storage device of the electronic device 600, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 600. Further, memory 602 may include both internal and external storage units of the electronic device 600. Memory 602 is used to store application software and various types of data installed on the electronic device 600, such as program code installed on the electronic device 600. Memory 602 may also be used to temporarily store data that has been output or will be output. In one embodiment, memory 602 stores a nuclear power plant operation and maintenance data cross-modal retrieval program, which can be executed by processor 601 to implement the nuclear power plant operation and maintenance data cross-modal retrieval method of various embodiments of the present invention.

[0033] In some embodiments, processor 601 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 602 or process data, such as a cross-modal retrieval method for nuclear power plant operation and maintenance data.

[0034] In some embodiments, display 603 may be an LED display, a liquid crystal display, a touch-screen liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 603 is used to display identification information from the nuclear power plant operation and maintenance data cross-modal retrieval program and to display a visual user interface. Components 601-603 of electronic device 600 communicate with each other via a system bus.

[0035] In some embodiments, when the processor 601 executes the nuclear power operation and maintenance data cross-modal retrieval program in the memory 602, it implements each step of the nuclear power operation and maintenance data cross-modal retrieval method as described in the above embodiments. Since the nuclear power operation and maintenance data cross-modal retrieval method has been described in detail above, it will not be repeated here.

[0036] Accordingly, the present invention also provides a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps or functions of the nuclear power operation and maintenance data cross-modal retrieval method provided in the above-described method embodiments.

[0037] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for cross-modal retrieval of nuclear power plant operation and maintenance data, characterized in that, include: Acquire historical data on the operation and maintenance of multimodal nuclear power plants, including image data and text data; Feature extraction is performed on the image data and text data to obtain image features and text features. The text features include global paired text features corresponding to the image features, as well as paired generated text features and unpaired generated text features generated by a text variational autoencoder. The image features are compared with the global paired text features and the unpaired generated text features to calculate intermodal similarity, obtaining a first similarity distance and a third similarity distance. The paired generated text features are compared with the unpaired generated text features to calculate intramodal similarity, obtaining a second similarity distance. A target loss function is determined based on the first similarity distance, the second similarity distance, and the third similarity distance. The parameters of the cross-modal retrieval model are adjusted using the target loss function to obtain a trained cross-modal retrieval model. The operation and maintenance data to be retrieved is input into the trained cross-modal retrieval model to obtain image feature matrix and text feature matrix. The image feature matrix and text feature matrix are quantized to obtain hash code. Based on the hash code, the text or image in the database is retrieved to obtain the text or image corresponding to the hash code.

2. The cross-modal retrieval method for nuclear power plant operation and maintenance data according to claim 1, characterized in that, The cross-modal retrieval model includes an image feature extraction network, a text feature extraction network, and a text variational autoencoder; the feature extraction of the image data and text data to obtain image features and text features includes: The image data is preprocessed to obtain the original image, the main image, and the local images; The original image, the main image, and the local image are input into the image feature extraction network to obtain global image features, main image features, and local image features. The global image features, main image features, and local image features are then weighted and fused to obtain image features. Based on the image data and text data, the paired text, unpaired text, and unpaired local keyword text of the original image are obtained. The paired text, unpaired text, and unpaired local keyword text are input into the text feature extraction network to obtain global paired text features, unpaired global text features, and unpaired local text features. The unpaired global text features and unpaired local text features are weighted and fused to obtain unpaired text features. The global paired text features and unpaired text features are input into the text variational autoencoder to obtain paired generated text features and unpaired generated text features.

3. The cross-modal retrieval method for nuclear power plant operation and maintenance data according to claim 2, characterized in that, The image feature extraction network architecture is a Fast-RCNN network, which includes an input layer, a convolutional backbone network, a region of interest pooling layer, a fully connected layer, and a parallel output layer; the text feature extraction network architecture is a BERT model, which includes an input layer, a multi-layer Transformer encoder, and a pre-trained output layer; the text variational autoencoder includes an encoder, a latent space, and a decoder.

4. The cross-modal retrieval method for nuclear power plant operation and maintenance data according to claim 2, characterized in that, The step of performing intermodal similarity calculations between the image features and the globally paired text features and the unpaired generated text features to obtain a first similarity distance and a third similarity distance includes: The image features are mapped to the global paired text features through a cross-modal attention mechanism, and the cosine similarity between the mapped image features and the global paired text features is calculated to obtain the first similarity distance. The image features are mapped to the unpaired generated text features using a cross-modal attention mechanism. The cosine similarity between the mapped image features and the unpaired generated text features is then calculated to obtain the third similarity distance.

5. The cross-modal retrieval method for nuclear power plant operation and maintenance data according to claim 4, characterized in that, The step of calculating the intra-modal similarity between the paired generated text features and the unpaired generated text to obtain the second similarity distance includes: The Euclidean distance between the paired generated text features and the unpaired generated text is calculated to obtain the second similarity distance.

6. The cross-modal retrieval method for nuclear power plant operation and maintenance data according to claim 5, characterized in that, The target loss function is: ， in, Let be the target loss function. The first similarity distance, For the second similarity distance, The third similarity distance, , These are the weighting coefficients.

7. The cross-modal retrieval method for nuclear power plant operation and maintenance data according to claim 5, characterized in that, The step of quantizing the image feature matrix and text feature matrix to obtain a hash code, and then retrieving text or images from the database based on the hash code to obtain the text or image corresponding to the hash code, includes: The image feature matrix and the text feature matrix are binarized respectively to obtain the binary image feature matrix and the binary text feature matrix; The binary image feature matrix and the binary text feature matrix are ORed to obtain a fused binary matrix. The fused binary matrix is ​​then encoded to obtain a hash code. Obtain the hash code of text or image in the database, calculate the distance between the hash code and the hash code of the text or image to obtain the hash code of the nearest neighbor in the database, and obtain the text or image corresponding to the hash code based on the hash code of the nearest neighbor.

8. A cross-modal retrieval device for nuclear power plant operation and maintenance data, characterized in that, include: The data acquisition module is used to acquire historical data on the operation and maintenance of multimodal nuclear power plants, which includes image data and text data. The feature extraction module is used to extract features from the image data and text data based on the constructed cross-modal retrieval model to obtain image features and text features. The text features include global paired text features corresponding to the image features, as well as paired generated text features and unpaired generated text features generated by the text variational autoencoder. An adjustment module is used to perform inter-modal similarity calculations between the image features and the global paired text features and the unpaired generated text features to obtain a first similarity distance and a third similarity distance; to perform intra-modal similarity calculations between the paired generated text features and the unpaired generated text features to obtain a second similarity distance; to determine a target loss function based on the first similarity distance, the second similarity distance, and the third similarity distance; and to adjust the parameters of the cross-modal retrieval model using the target loss function to obtain a trained cross-modal retrieval model. The retrieval module is used to input the operation and maintenance data to be retrieved into the trained cross-modal retrieval model to obtain image feature matrices and text feature matrices, perform quantization processing on the image feature matrices and text feature matrices to obtain hash codes, and retrieve text or images in the database based on the hash codes to obtain the text or images corresponding to the hash codes.

9. An electronic device, characterized in that, Including memory and processor; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the cross-modal retrieval method for nuclear power plant operation and maintenance data as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the nuclear power plant operation and maintenance data cross-modal retrieval method according to any one of claims 1-7.

Images