An intelligent operation and maintenance picture-text retrieval method and system based on multi-modal cognitive enhancement

Abstract

The application discloses an intelligent operation and maintenance picture-text retrieval method and system based on multi-modal cognitive enhancement, relates to the technical field of multi-modal retrieval, and comprises the following steps: acquiring a query request, judging the modal type of the query request and performing corresponding retrieval processes according to the type; inputting the query text / image into a trained feature projection network to obtain query text / image features; performing similarity retrieval on the query text / image features in a vector database to obtain first text / visual recall results; using a multi-dimensional category label library to perform label matching on the query text / image, obtaining image / picture-text records associated with the matched labels as second text recall results; and aggregating the first text recall results and the second text recall results, outputting the final retrieval results after weighted fusion and sorting. The stability and accuracy of picture-text data quality semantic matching and retrieval precision are improved.

Description

Technical Field

[0001] This application belongs to the field of multimodal retrieval technology, and specifically relates to an intelligent operation and maintenance image and text retrieval method and system based on multimodal cognitive enhancement. Background Technology

[0002] With the continuous advancement of digital transformation, the demand for accurate retrieval of multimodal information (text and images) in intelligent operation and maintenance is becoming increasingly prominent. For example, this involves locating equipment images through text descriptions or querying related text records by combining image features. Traditional information retrieval methods are limited to a single modality and cannot uncover the value of the correlation between images and text. Cross-modal text-image retrieval technology aims to establish semantic mapping between images and text, achieving accurate matching and cross-referencing of information from different modalities, thus becoming a key direction for solving this problem. Early cross-modal retrieval methods mostly used statistical learning methods to construct a common representation space. Subsequent pre-trained models learned deep semantic features through large-scale data, significantly improving modality alignment accuracy. Typical architectures include a single-stream architecture that directly fuses text and image features and a dual-stream architecture that fuses them after using independent encoders.

[0003] However, applying the above methods to intelligent operation and maintenance scenarios still faces a series of challenges. First, existing cross-modal retrieval models rely on general image and text datasets and internet data during the pre-training stage. Such data lacks professional knowledge in the operation and maintenance domain, and the correlation between images and text descriptions is often weak and semantically noisy, making it difficult to provide fine-grained, highly relevant domain text descriptions. This results in the model being unable to fully learn the detailed correspondence between equipment images and precise text records. Second, there are cross-modal semantic differences between images and text. The visual features and text features output by the pre-trained model still retain strong modality specificity, causing a shift in their distribution within the same semantic space. This leads to insufficient stability and accuracy of cross-modal semantic alignment, affecting retrieval matching results. Furthermore, current retrieval models mainly focus on similarity calculation at the feature level, lacking a deep understanding of the operation and maintenance business logic. They cannot effectively integrate multi-dimensional conditions such as time range, equipment status, and equipment type for constrained retrieval, making it difficult to meet complex query needs such as finding images of transformer equipment with abnormally high temperatures within the past three months. These problems severely restrict the practical application of cross-modal retrieval in intelligent operation and maintenance, and there is an urgent need for image and text retrieval methods that can integrate knowledge from the operation and maintenance domain, achieve fine cross-modal semantic alignment, and possess business cognitive capabilities. Summary of the Invention

[0004] To address the aforementioned issues, this application provides an intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement, comprising the following steps: Based on the modality type of the query request, the query data is input into the trained feature projection network to obtain query features; similarity retrieval is performed in the vector database based on the query features to obtain the first recall result; the query data is processed using a multi-dimensional category label library to obtain the image and text records associated with matching labels as the second recall result; the first recall result and the second recall result are aggregated, weighted, fused, and sorted before the final retrieval result is output. The construction steps of the feature projection network and vector database include: acquiring image data; generating basic and enhanced text descriptions for each image using a multimodal large model and processing the enhanced text descriptions; constructing an image-text pair dataset with the corresponding images after processing; extracting initial features of images and text in the image-text pair dataset using a pre-trained multimodal feature extractor with frozen parameters, and inputting them into a trainable feature projection network for mapping, outputting visual projection features and text projection features; calculating cross-modal contrast loss based on visual projection features and text projection features, and calculating modal adversarial loss through a modal discriminator; training the feature projection network by jointly using the cross-modal contrast loss and modal adversarial loss, and using the trained feature projection network to extract image visual features and store them in the vector database.

[0005] Furthermore, a multimodal large model is used to generate basic and enhanced text descriptions for each image, including: Image data is input into a multimodal large model, and basic text descriptions are generated through preset first cue constraints. The basic text descriptions are used to summarize the scene, device type and external state that can be directly observed in the image. Image data and basic text descriptions are input into a multimodal large model. Enhanced text descriptions are generated through preset second cue constraints. The enhanced text descriptions include supplementary descriptions of key component details, standardized expressions of device status, and structural reorganization of scene semantics.

[0006] Further processing of the enhanced text description includes: The enhanced text description, the corresponding basic text description, and the operation and maintenance inspection images are input into the evaluation model to obtain a factual consistency score. When the factual consistency score reaches a preset threshold, the current enhanced text description is retained; otherwise, the current enhanced text description is discarded. The retained enhanced text description is cleaned using rules, removing sentences containing speculative, uncertain, and suggestive words, and filtering out repetitive semantic content to obtain the cleaned enhanced text description.

[0007] Furthermore, the steps for building a multidimensional category tag library include: The inspection images are input into a trained multimodal large model, and status labels are generated through the trained multimodal large model. The text embedding model is used to convert the state labels into state label vectors, and it is compared with the preset enumerated label library to determine whether they match. If they match, they are directly mapped to the existing labels and output. State label vectors that fail to match the enumerated label library are classified as anomaly descriptions and included in the anomaly description pool. The anomaly descriptions are clustered using a clustering algorithm to extract new label candidates. The new label candidates are manually reviewed. If the review fails, the state label vector corresponding to the new label candidate is reclassified into the anomaly description pool. If the review passes, the new label candidate is written into the enumerated label library and output.

[0008] Furthermore, the feature projection network is a multilayer perceptron consisting of fully connected layers, normalization layers, and nonlinear activation functions.

[0009] Furthermore, the cross-modal contrastive loss is a symmetric InfoNCE loss, and the modal adversarial loss is a cross-entropy loss; the joint optimization objective function is: ; in, , These are the cross-modal contrastive loss weighting coefficients and the modal adversarial loss weighting coefficients, respectively. For cross-modal contrast loss, For modal adversarial loss, To jointly optimize the objective function.

[0010] Furthermore, during the training of the feature projection network, the parameters of the pre-trained multimodal feature extractor are frozen, and only the parameters of the feature projection network and the modality discriminator are updated.

[0011] Furthermore, the vector database is the Milvus vector database.

[0012] Furthermore, the weighted fusion ranking includes: aggregating text and image information originating from the same event based on the event identifiers carried in the first and second recall results, and then ranking and outputting the aggregated results after calculating a comprehensive score according to preset weights.

[0013] This application also provides an intelligent operation and maintenance image and text retrieval system based on multimodal cognitive enhancement, which is used to execute any of the above-mentioned intelligent operation and maintenance image and text retrieval methods based on multimodal cognitive enhancement, including: The modality determination module is used to obtain the query request and determine the modality type of the query request; The text description generation and enhancement module is used to generate text descriptions for operation and maintenance inspection images in stages using a multimodal large model, resulting in basic text descriptions and enhanced text descriptions. The enhanced text descriptions are then subjected to consistency verification and rule-based cleaning. The cleaned enhanced text descriptions are retained and together with the basic text descriptions and operation and maintenance inspection images to form a graph-text pair dataset. The tag library construction and update module is used to generate state label descriptions for images in the image-text pair dataset using a multimodal large model, and to build a multidimensional category tag library by supplementing it with semantic vector mapping and dynamic clustering. The vector library construction and update module is used to extract visual features of images in the image-text pair dataset using the trained feature projection network and store them in the vector database. The text retrieval module is used to input the query text into the trained feature projection network to obtain the query text features; perform similarity retrieval in the vector database based on the query text features to obtain the first text recall result; use a multi-dimensional category label library to perform label matching on the query text to obtain the image and text records associated with the matching labels as the second text recall result; aggregate the first text recall result and the second text recall result, and output the final retrieval result after weighted fusion and sorting. The visual retrieval module is used to input the query image into the trained feature projection network to obtain the query image features; perform similarity retrieval in the vector database based on the query image features to obtain the first visual recall result; use a multi-dimensional category label library to perform label matching on the query image to obtain the image and text records associated with the matching labels as the second visual recall result; aggregate the first visual recall result and the second visual recall result, and output the final retrieval result after weighted fusion and sorting.

[0014] The beneficial effects of the technical solutions provided in this application include at least the following: 1. By adopting a phased text description generation and enhancement strategy, combined with consistency verification and rule-based cleaning mechanisms, the operation and maintenance inspection images are transformed into structured text descriptions that conform to the domain semantics. At the same time, a multi-dimensional operation and maintenance category label system provides fine-grained business prior knowledge for retrieval tasks, effectively solving the problems of high noise and poor domain adaptability of general image and text data, and improving the accuracy and data quality of retrieval model training. 2. By performing secondary mapping and semantic decoupling on the initial features output by the pre-trained multimodal model through a feature projection network, and combining cross-modal contrastive learning and adversarial modality elimination mechanism for joint optimization, deep semantic alignment of visual features and text features is achieved without modifying the parameters of the pre-trained base model. This effectively reduces the semantic gap in cross-modal retrieval and improves the stability and accuracy of semantic matching. 3. Through a multi-path recall fusion ranking architecture that combines multimodal vector retrieval and tag retrieval, it supports matching retrieval of text and images in a unified semantic space. Combined with the dual-dimensional tag system of device-anomaly and personnel-behavior-risk, it can respond to multi-dimensional composite retrieval conditions such as device type, status features and time dimension. It solves the defects of existing methods that can only perform single feature matching and have weak business logic adaptability, and improves the retrieval accuracy and practicality in complex business scenarios.

[0015] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of an intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement is shown; Figure 2 This document illustrates a flowchart of the process for generating and enhancing text descriptions of operation and maintenance inspection images according to an embodiment of this application. Figure 3 A flowchart illustrating the process of constructing a multidimensional category label library according to an embodiment of this application is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Please see Figure 1 A method for intelligent operation and maintenance image and text retrieval based on multimodal cognitive enhancement includes the following steps: Step 1: The user inputs text / image to trigger the search; Step 2: Obtain the query request and determine its modality type; if the query request is text, execute the text modality retrieval process; if the query request is an image, execute the visual modality retrieval process. (1) The text modality retrieval process includes: inputting the query text into the trained feature projection network to obtain the query text features; performing similarity retrieval in the vector database based on the query text features to obtain the first text recall result; using a multi-dimensional category label library to perform label matching on the query text and obtaining the image and text records associated with the matching labels as the second text recall result; aggregating the first text recall result and the second text recall result, and outputting the final retrieval result after weighted fusion and sorting. (2) The visual modality retrieval process includes: inputting the query image into the trained feature projection network to obtain the query image features; performing similarity retrieval in the vector database based on the query image features to obtain the first visual recall result; using a multi-dimensional category label library to perform label matching on the query image and obtaining the image and text records associated with the matching labels as the second visual recall result; aggregating the first visual recall result and the second visual recall result, and outputting the final retrieval result after weighted fusion and sorting.

[0020] It should be noted that the weighted fusion ranking specifically involves aggregating images and text information originating from the same event based on the event identifiers carried in the first and second recall results, and then ranking and outputting the aggregated results by calculating a comprehensive score according to a preset weighted fusion algorithm, thereby improving the relevance and accuracy of the search results.

[0021] It should also be noted that since the input in text modality retrieval is query text, tag matching directly relies on keyword / tag matching technology to filter out relevant image and text records from the multidimensional category tag library; while the input in visual modality retrieval is query image, so it is necessary to first generate corresponding state labels through a trained multimodal large model, then use a text embedding model to convert the state labels into state label vectors before performing keyword / tag matching to obtain relevant image and text records.

[0022] The text modality retrieval process and the visual modality retrieval process share a pre-built feature projection network and vector database for feature extraction and similarity retrieval; the construction steps of the feature projection network and vector database include: (1) Obtain operation and maintenance inspection images, and use multimodal large model to generate phased text descriptions of operation and maintenance inspection images to obtain basic text descriptions and enhanced text descriptions; (2) such as Figure 2 As shown, a multimodal large model is used to generate phased text descriptions for operation and maintenance inspection images, resulting in basic text descriptions and enhanced text descriptions, including: The operation and maintenance inspection images are input into a pre-trained multimodal large model. In this embodiment, the Qwen3-VL-30B-A3B-Thinking model is used to generate basic text descriptions through preset first prompt constraints. The basic text descriptions are used to summarize the scenes, equipment types and external states that can be directly observed in the images. The operation and maintenance inspection images and basic text descriptions are input together into the pre-trained multimodal large model. In this embodiment, the Qwen3-VL-30B-A3B-Thinking model is used to generate enhanced text descriptions through preset second cue constraints. The enhanced text descriptions include supplementary descriptions of key component details, standardized expressions of equipment status, and structural reorganization of scene semantics.

[0023] (3) Perform consistency verification and rule-based cleaning on the enhanced text description, retain the cleaned enhanced text description, and form a text-image pair dataset together with the basic text description and the operation and maintenance inspection image. The consistency verification and rule-based cleaning are as follows: input the enhanced text description, the corresponding basic text description and the operation and maintenance inspection image into the evaluation model to obtain the factual consistency score; when the factual consistency score reaches the preset threshold, retain the current enhanced text description, otherwise discard the current enhanced text description; perform rule-based cleaning on the retained enhanced text description, delete the statements containing speculative words, uncertain words and suggestive words, and filter duplicate semantic content to obtain the cleaned enhanced text description.

[0024] (4) Input the images and corresponding text descriptions in the image-text pair dataset into the pre-trained multimodal feature extractor with frozen parameters to extract initial visual features and initial text features. In the cross-modal retrieval scenario for tower defect detection, the pre-trained multimodal feature extractor is first used as the base model, and the initial feature representations of the visual modality (image) and the text modality (text description) are obtained through interface calls. Let the visual input features be... Text input features are The base model output is: ; in, , A multimodal feature extraction interface for parameter freezing, wherein As the initial visual features, These are the initial features of the text.

[0025] (5) Input the initial visual features and initial text features into a trainable feature projection network for secondary mapping, and output visual projection features and text projection features. Since the base model is trained for a general scene, its output features still contain obvious modality-specific information. Therefore, a trainable feature projection layer is introduced after the initial features to perform secondary mapping and semantic decoupling of the features. The calculation formula is as follows: ; in, , For the feature projection networks acting on the visual modality and the text modality respectively, the parameters are as follows: , , For visual projection features, This refers to text projection features. It should be noted that the preferred feature projection network is a multilayer perceptron structure consisting of fully connected layers, normalization layers, and nonlinear activation functions.

[0026] (6) To achieve effective alignment of visual features and textual features within the same semantic space, a cross-modal contrastive learning mechanism is adopted. First, pooling is performed on the projected features to obtain global features, calculated as follows: ; Where Pool represents the pooling operation. For visual global features, These are global features of the text.

[0027] (7) During the training phase, the symmetric InfoNCE loss function is used as the cross-modal contrastive loss to narrow the distance between semantically matching samples and widen the distance between non-matching samples. The loss function of the cross-modal contrastive loss is defined as: ; ; ; Where N represents the number of samples, Represents the cosine similarity function. For temperature coefficient, It is a natural exponential function. For cross-modal contrast loss, For image-to-text one-way contrast loss, For text-to-image one-way contrast loss, and These are the visual global features of the i-th and j-th images, respectively. and These are the global text features describing the i-th and j-th texts, respectively.

[0028] (8) To further eliminate the statistical difference between the visual modality and the text modality, a modality discriminator is introduced after the feature projection layer to determine the source modality of the input features. Let the modality discriminator be... Its output is the probability value of the feature belonging to the text modality.

[0029] The modal adversarial loss of the modal discriminator is defined as follows: ; in, The cross-entropy loss function is used, where 0 represents the visual modality and 1 represents the text modality. This is for modal adversarial loss.

[0030] (9) During training, a gradient reversal layer (GRL) is set between the feature projection layer and the modality discriminator. The gradient reversal layer maintains the identity mapping during forward propagation and reverses the sign of the gradient signal during backward propagation, thereby maximizing the classification error of the modality discriminator and suppressing the retention of modality-specific features during the optimization process of the feature projection layer.

[0031] (10) Combining the above cross-modal contrast loss and modal adversarial loss, the joint optimization objective function is constructed as follows: ; in, , These are the cross-modal contrastive loss weighting coefficients and the modal adversarial loss weighting coefficients, respectively. For cross-modal contrast loss, For modal adversarial loss, The objective function is jointly optimized. Through the above joint optimization mechanism, only the parameters of the feature projection layer and the modality discriminator are updated, achieving deep alignment of cross-modal semantic features without changing the base model parameters.

[0032] It should be noted that during the training of the feature projection network, the parameters of the pre-trained multimodal feature extractor are frozen, and only the parameters of the feature projection network and the modality discriminator are updated.

[0033] (11) The trained feature projection network is used to extract the visual global features of the images in the newly obtained image-text pair dataset and store them in the vector database. The vector database is the Milvus vector database. The generated high-dimensional feature vectors are stored in the Milvus vector database to provide fast retrieval.

[0034] It should be noted that each visual global feature is associated with corresponding image information, text description, and an event identifier ID.

[0035] like Figure 3 As shown, the steps for building a multidimensional category tag library include: Step 101: Input the inspection image into the trained multimodal large model, and generate status labels through the trained multimodal large model (Qwen3-VL-30B-A3B-Thinking model is used in this embodiment); Step 102: Use a text embedding model (Qwen3-embedding model in this embodiment) to convert the state labels into state label vectors, and determine whether they match the preset enumerated label library. If they match, directly map them to the existing labels and output them. Step 103: State label vectors that fail to match the enumerated label library are classified as anomaly descriptions and included in the anomaly description pool. The anomaly descriptions are clustered using a clustering algorithm (SinglePass clustering algorithm is used in this embodiment) and new label candidates are extracted. The new label candidates are manually reviewed. If the review fails, the state label vector corresponding to the new label candidate is reclassified into the anomaly description pool. If the review passes, the new label candidate is written into the enumerated label library and output.

[0036] It should be noted that each tag in the multidimensional category tag library is associated with an image and text record that includes an event identifier ID as well as the corresponding image information and text description.

[0037] This embodiment constructs a multidimensional category tag library with rich, fine-grained semantic description information using the multidimensional category tag library construction method described above, as shown in Table 1: Table 1. Multidimensional Category Tag Library for Operation and Maintenance Scenarios

[0038] An intelligent operation and maintenance image and text retrieval system based on multimodal cognitive enhancement, used to execute the aforementioned intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement, includes: The modality determination module is used to obtain the query request and determine the modality type of the query request; The text description generation and enhancement module is used to generate text descriptions for operation and maintenance inspection images in stages using a multimodal large model, resulting in basic text descriptions and enhanced text descriptions. The enhanced text descriptions are then subjected to consistency verification and rule-based cleaning. The cleaned enhanced text descriptions are retained and together with the basic text descriptions and operation and maintenance inspection images to form a graph-text pair dataset. The tag library construction and update module is used to generate state label descriptions for images in the image-text pair dataset using a multimodal large model, and to build a multidimensional category tag library by supplementing it with semantic vector mapping and dynamic clustering. The vector library construction and update module is used to extract visual features of images in the image-text pair dataset using the trained feature projection network and store them in the vector database. The text retrieval module is used to input the query text into the trained feature projection network to obtain the query text features; perform similarity retrieval in the vector database based on the query text features to obtain the first text recall result; use a multi-dimensional category label library to perform label matching on the query text to obtain the image and text records associated with the matching labels as the second text recall result; aggregate the first text recall result and the second text recall result, and output the final retrieval result after weighted fusion and sorting. The visual retrieval module is used to input the query image into the trained feature projection network to obtain the query image features; perform similarity retrieval in the vector database based on the query image features to obtain the first visual recall result; use a multi-dimensional category label library to perform label matching on the query image to obtain the image and text records associated with the matching labels as the second visual recall result; aggregate the first visual recall result and the second visual recall result, and output the final retrieval result after weighted fusion and sorting.

[0039] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for intelligent operation and maintenance image and text retrieval based on multimodal cognitive enhancement, characterized in that, Includes the following steps: Based on the modality type of the query request, the query data is input into the trained feature projection network to obtain the query features; Based on the query features, a similarity search is performed in the vector database to obtain the first recall result; The query data is processed using a multi-dimensional category tag library to obtain the image and text records associated with matching tags as the second recall result; The first and second recall results are aggregated, weighted, and sorted before the final search results are output. The construction steps of the feature projection network and vector database include: acquiring image data; generating basic and enhanced text descriptions for each image using a multimodal large model and processing the enhanced text descriptions; constructing an image-text pair dataset with the corresponding images after processing; extracting initial features of images and text in the image-text pair dataset using a pre-trained multimodal feature extractor with frozen parameters, and inputting them into a trainable feature projection network for mapping, outputting visual projection features and text projection features; calculating cross-modal contrast loss based on visual projection features and text projection features, and calculating modal adversarial loss through a modal discriminator; training the feature projection network by jointly using the cross-modal contrast loss and modal adversarial loss, and using the trained feature projection network to extract image visual features and store them in the vector database.

2. The intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement according to claim 1, characterized in that, The method of generating basic and enhanced text descriptions for each image using a multimodal large model includes: Image data is input into a multimodal large model, and basic text descriptions are generated through preset first cue constraints. The basic text descriptions are used to summarize the scene, device type and external state that can be directly observed in the image. Image data and basic text descriptions are input into a multimodal large model. Enhanced text descriptions are generated through preset second cue constraints. The enhanced text descriptions include supplementary descriptions of key component details, standardized expressions of device status, and structural reorganization of scene semantics.

3. The intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement according to claim 1, characterized in that, The processing of the enhanced text description includes: The enhanced text description, the corresponding basic text description, and the operation and maintenance inspection images are input into the evaluation model to obtain a factual consistency score. When the factual consistency score reaches a preset threshold, the current enhanced text description is retained; otherwise, the current enhanced text description is discarded. The retained enhanced text description is cleaned using rules, removing sentences containing speculative, uncertain, and suggestive words, and filtering out repetitive semantic content to obtain the cleaned enhanced text description.

4. The intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement according to claim 1, characterized in that, The construction steps of the multidimensional category label library include: The inspection images are input into a trained multimodal large model, and status labels are generated through the trained multimodal large model. The text embedding model is used to convert the state labels into state label vectors, and it is compared with the preset enumerated label library to determine whether they match. If they match, they are directly mapped to the existing labels and output. State label vectors that fail to match the enumerated label library are classified as anomaly descriptions and included in the anomaly description pool. The anomaly descriptions are clustered using a clustering algorithm to extract new label candidates. The new label candidates are manually reviewed. If the review fails, the state label vector corresponding to the new label candidate is reclassified into the anomaly description pool. If the review passes, the new label candidate is written into the enumerated label library and output.

5. The intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement according to claim 1, characterized in that, The feature projection network is a multilayer perceptron consisting of fully connected layers, normalization layers, and nonlinear activation functions.

6. The intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement according to claim 1, characterized in that, The cross-modal contrastive loss is a symmetric InfoNCE loss, and the modal adversarial loss is a cross-entropy loss; the joint optimization objective function is: ； in, , These are the cross-modal contrastive loss weighting coefficients and the modal adversarial loss weighting coefficients, respectively. For cross-modal contrast loss, For modal adversarial loss, To jointly optimize the objective function.

7. The intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement according to claim 1, characterized in that, During the training of the feature projection network, the parameters of the pre-trained multimodal feature extractor are frozen, and only the parameters of the feature projection network and the modality discriminator are updated.

8. The intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement according to claim 1, characterized in that, The vector database is the Milvus vector database.

9. The intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement according to claim 1, characterized in that, The weighted fusion sorting includes: aggregating text and image information originating from the same event based on the event identifiers carried in the first and second recall results, and then sorting and outputting the aggregated results according to preset weights after calculating a comprehensive score.

10. An intelligent operation and maintenance image and text retrieval system based on multimodal cognitive enhancement, characterized in that, It is used to execute the intelligent operation and maintenance image and text retrieval method based on multimodal cognitive enhancement as described in any one of claims 1-9, comprising: The modality determination module is used to obtain the query request and determine the modality type of the query request; The text description generation and enhancement module is used to generate text descriptions for operation and maintenance inspection images in stages using a multimodal large model, resulting in basic text descriptions and enhanced text descriptions. The enhanced text descriptions are then subjected to consistency verification and rule-based cleaning. The cleaned enhanced text descriptions are retained and together with the basic text descriptions and operation and maintenance inspection images to form a graph-text pair dataset. The tag library construction and update module is used to generate state label descriptions for images in the image-text pair dataset using a multimodal large model, and to build a multidimensional category tag library by supplementing it with semantic vector mapping and dynamic clustering. The vector library construction and update module is used to extract visual features of images in the image-text pair dataset using the trained feature projection network and store them in the vector database. The text retrieval module is used to input the query text into the trained feature projection network to obtain the query text features; perform similarity retrieval in the vector database based on the query text features to obtain the first text recall result; use a multi-dimensional category label library to perform label matching on the query text to obtain the image and text records associated with the matching labels as the second text recall result; aggregate the first text recall result and the second text recall result, and output the final retrieval result after weighted fusion and sorting. The visual retrieval module is used to input the query image into the trained feature projection network to obtain the query image features; perform similarity retrieval in the vector database based on the query image features to obtain the first visual recall result; use a multi-dimensional category label library to perform label matching on the query image to obtain the image and text records associated with the matching labels as the second visual recall result; aggregate the first visual recall result and the second visual recall result, and output the final retrieval result after weighted fusion and sorting.

Images