Water conservancy project file retrieval method and system

Abstract

The invention relates to the technical field of electronic data processing, and discloses a hydraulic engineering archive retrieval method and system.The hydraulic engineering archive retrieval method comprises the steps that a hydraulic engineering archive containing text data and image data is obtained, and the text data and the image data are mapped into an initial text feature vector and an initial image feature vector; the initial text feature vector and the initial image feature vector are projected to a shared semantic subspace, cross-modal representation is obtained, a joint loss function comprises a local feature alignment loss item and a semantic structure maintenance loss item, for any query request, the query request is converted into a query vector, and in the shared semantic subspace, the local feature alignment loss item and the semantic structure maintenance loss item are obtained. And sorting by calculating the cosine similarity of the query vector and the cross-modal representation of all archives in the archive library, and outputting a sorting result as retrieval. According to the method, fine-grained semantic alignment between the local key information of the image and the core terms of the text is realized, the understanding depth and accuracy of the core associated content of the image and text are improved, and the file retrieval precision is improved.

Description

Technical Field

[0001] This invention relates to the field of electronic data processing technology, specifically to a method and system for retrieving water conservancy project archives. Background Technology

[0002] Water conservancy project archives are crucial documents recording the entire process of project planning, design, construction, management, and maintenance. These archives come in various forms, typically including text reports, design specifications, and meeting minutes, as well as accompanying engineering drawings, site photographs, and other image data. Therefore, the archive data is characterized by its massive scale, complex structure, and heterogeneous modality. Traditional retrieval methods, such as keyword-based text retrieval or content-based image retrieval, often fragment data across different modalities, making it difficult to meet cross-modal retrieval needs such as "searching for images by text" or "searching for text by images." The core idea of ​​cross-modal retrieval technology is to construct a shared semantic subspace, mapping data from different modalities to a unified feature vector within this space. This bridges the semantic gap between heterogeneous data, thereby achieving unified information representation and efficient retrieval.

[0003] However, existing cross-modal retrieval methods still have some problems in application: First, in terms of feature alignment granularity, most methods tend to use coarse-grained global alignment, that is, simply matching the visual features of the entire image with the semantic features of the entire text. This approach ignores the fine-grained correspondence between specific key components in water conservancy engineering images and specific engineering terms in archival texts, making it difficult for the model to accurately understand the core association between the image and text content. Second, in terms of utilizing semantic relationships, existing models usually lack a deep integration of professional knowledge in the field of water conservancy engineering. There are a large number of professional terms with complex hierarchical and relational relationships in the field of water conservancy engineering, which constitute a specialized knowledge system. General models cannot effectively capture and utilize the deep semantic relationships between these terms, resulting in insufficient ability to understand user query intent and expand the search scope, making it difficult to retrieve those documents that are semantically related but whose textual expressions are not entirely consistent. Summary of the Invention

[0004] This invention provides a method and system for retrieving water conservancy project archives to solve the problem in the prior art that the existing technology is not capable enough in understanding the user's query intent and expanding the search scope, resulting in low archive retrieval accuracy.

[0005] In a first aspect, the water conservancy project archive retrieval method of the present invention includes the following steps: A water conservancy project archive containing both text and image data is acquired. A text encoder and an image encoder are used to map the text and image data into initial text feature vectors and initial image feature vectors, respectively. A shared semantic subspace is constructed, and the initial text and image feature vectors are projected onto the shared semantic subspace by optimizing a joint loss function to obtain a cross-modal representation. The joint loss function includes a local feature alignment loss term and a semantic structure preservation loss term. The local feature alignment loss term is calculated based on the local correspondence between the regional visual features of key components in the image and the textual features of specific engineering terms in the archive text. The alignment weight of the alignment loss term is adjusted according to the number of key components and the number of specific engineering terms; the semantic structure preservation loss term is constructed based on the water conservancy engineering ontology knowledge graph; when the semantic similarity of text terms of different archives in the water conservancy engineering ontology knowledge graph is higher than a preset threshold, the semantic structure preservation loss term makes the cross-modal representations of the different archives closer to each other in the shared semantic subspace; for any query request, the query request is converted into a query vector, and in the shared semantic subspace, the query vector is sorted by calculating the cosine similarity between the cross-modal representations of all archives in the archive database, and the sorting result is used as the retrieval output.

[0006] Preferably, the step of using a text encoder and an image encoder to map text data and image data into initial text feature vectors and initial image feature vectors, respectively, includes: The BERT model is used as the text encoder. Texts longer than 512 are truncated and input into the BERT model to obtain an initial text feature vector of 768 dimensions. The ResNet-50 model was used as the image encoder to uniformly adjust the image size. The pixels are input into the ResNet-50 model to obtain an initial image feature vector of 2048 dimensions.

[0007] Preferably, the regional visual features of the key components are obtained through the following methods: The image is input into the target detection model, which identifies and locates the key components and determines their category. For each identified key component, extract the visual features of the corresponding region in the feature map output by the intermediate layer of the image encoder.

[0008] Preferably, the local correspondence is established in the following way: Extract specific engineering terms that match the category of the key component from the archive text, and extract the text features corresponding to the specific engineering terms in the text encoder; The local correspondence is established by calculating the dot product similarity between the textual features of each specific engineering term and the visual features of all regions.

[0009] Preferably, the alignment weight of the local feature alignment loss term is adjusted according to the number of key components and the number of specific engineering terms, including: The total number of key components identified by the object detection model in a single image is denoted as . ; The total number of specific engineering terms in the statistical archive text that match the categories of the identified key components is denoted as . ; Alignment weight The calculation is performed using the following formula: ; in, These are the preset weight hyperparameters.

[0010] Preferably, the semantic similarity is calculated in the following way: The TransE model is used to train the knowledge graph of the water conservancy project ontology, generating a 128-dimensional embedding vector for each entity term in the knowledge graph of the water conservancy project ontology. Semantic similarity is calculated by taking the cosine similarity of the entity term embedding vectors corresponding to the text terms in different archives.

[0011] Preferably, the semantic structure preservation loss term is obtained by constructing a triplet loss function, wherein the cross-modal representation of the current file is used as the anchor point, the cross-modal representation of files in the water conservancy engineering ontology knowledge graph with a semantic similarity higher than 0.85 with the current file is used as positive samples, and the cross-modal representation of files in the water conservancy engineering ontology knowledge graph with a semantic similarity lower than 0.3 with the current file is used as negative samples, minimizing the distance between the anchor point and the positive samples, and maximizing the distance between the anchor point and the negative samples.

[0012] Preferably, the joint loss function further includes a global alignment loss term, which is a contrastive loss function. The global alignment loss term is used to measure the similarity of the global feature representations of matched text-image pairs within a batch. The goal of the global alignment loss term is to bring the global feature representations of matched pairs closer together and push away the global feature representations of unmatched pairs.

[0013] Preferably, the target detection model is the YOLOv5 model.

[0014] Secondly, the water conservancy project archive retrieval system of the present invention includes a memory and a processor. The memory stores computer instructions, and when the processor executes the computer instructions, it implements the above-mentioned water conservancy project archive retrieval method.

[0015] The beneficial effects of this invention are as follows: The method proposed in this invention identifies key components in an image using a target detection model and establishes a local correspondence between the visual features of the component regions and specific engineering terms in the archival text. This achieves fine-grained semantic alignment between key local information in the image and core terms in the text, improving the model's understanding depth and accuracy of the core related content of the image and text. Simultaneously, by utilizing a water conservancy engineering ontology knowledge graph, domain knowledge is integrated into the construction of a shared semantic subspace. This allows for the mining and utilization of deep semantic relationships between professional terms in different archives, enabling semantically related but differently expressed archives to aggregate in the feature space, thereby enhancing the generalization ability and recall rate of the retrieval. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the water conservancy project archive retrieval method provided in an embodiment of the present invention. Detailed Implementation

[0017] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0018] like Figure 1 As shown, an embodiment of the water conservancy project archive retrieval method provided by the present invention includes the following steps: S1: Obtain water conservancy project archives containing text data and image data, and use a text encoder and an image encoder to map the text data and image data into initial text feature vectors and initial image feature vectors, respectively.

[0019] Specifically, when processing text data from water conservancy project archives, such as construction reports, a pre-trained BERT model is used as the text encoder. The text is first segmented and marked with special tags before being input into the BERT model. The output vector corresponding to the CLS classification tag in the last hidden state of the model is extracted as the initial text feature vector. For image data from the archives, such as photos of dam sites, a pre-trained ResNet model is used as the image encoder. The images are first uniformly resized to a predetermined size before being input into the ResNet network, where the output of the global average pooling layer is extracted as the initial image feature vector.

[0020] S2, construct a shared semantic subspace, and project the initial text feature vector and the initial image feature vector to the shared semantic subspace by optimizing the joint loss function to obtain a cross-modal representation. The joint loss function includes: a local feature alignment loss term and a semantic structure preservation loss term.

[0021] The local feature alignment loss term is calculated based on the local correspondence between the regional visual features of key components in the image and the textual features of specific engineering terms in the archival text. The alignment weight of the local feature alignment loss term is adjusted according to the number of key components and the number of specific engineering terms.

[0022] Specifically, object detection models such as YOLOv5 are used to process images of hydraulic engineering projects to identify bounding boxes of key components such as gates, pumping stations, and cracks. Next, visual features of the regions corresponding to these bounding boxes are extracted from the feature map of the last convolutional layer of the image encoder. Simultaneously, word vectors of engineering terms (such as "gate," "pumping station," and "crack") corresponding to these key components are extracted from the output of the text encoder. A local alignment loss term is constructed by calculating the cosine similarity between the visual features of the regions and the corresponding word vectors, aiming to maximize the similarity of the matching pairs. If the number of matching term pairs between the identified key components in the image and the text is N, the weight coefficient of this loss term in the joint loss function is set to a monotonically increasing function of N, such as log(1+N), so that when the local details of the matching are richer, the model will focus on learning fine-grained correspondences.

[0023] The semantic structure preservation loss term is constructed based on the knowledge graph of the water conservancy project ontology. When the semantic similarity of the text terms of different archives in the knowledge graph of the water conservancy project ontology is higher than a preset threshold, the semantic structure preservation loss term makes the cross-modal representations of the different archives closer to each other in the shared semantic subspace.

[0024] For example, a knowledge graph embedding model such as TransE is used to train a pre-constructed knowledge graph of hydraulic engineering ontology, generating entity vectors for each engineering term. In each batch of model training, for any two different files, the system extracts the core engineering terms from their texts and calculates the average semantic similarity between these terms using the trained entity vectors. If the similarity value is higher than a preset threshold (e.g., 0.8), the two files are considered to have a strong correlation at the domain knowledge level. At this point, a semantic structure preservation loss term is introduced into the joint loss function. This loss term aims to minimize the Euclidean distance between the cross-modal representations of the two files in the shared semantic subspace, thereby promoting the clustering of semantically related files in the feature space. Finally, in the constructed shared semantic subspace, texts and images with similar content meanings are mapped to points (vectors) that are close to each other in the space, while those with unrelated meanings are far apart.

[0025] S3. For any query request, the query request is converted into a query vector. In the shared semantic subspace, the query vector is sorted by calculating the cosine similarity between the query vector and the cross-modal representations of all files in the archive, and the sorting result is used as the retrieval output.

[0026] Specifically, when the query request is text, such as a query for "corrosion records of a reservoir gate," the same text encoder and projection network used in the document processing stage are used to convert the query text into a query vector. Similarly, when the query request is an image, such as a picture showing a specific type of valve, an image encoder and projection network are used to convert it into a query vector. This query vector is then compared with the cosine similarity of each pre-calculated and stored cross-modal representation of the documents in the database. The system sorts all documents from highest to lowest similarity score and returns the top-ranked documents as search results to the user.

[0027] In an optional embodiment, the step of using a text encoder and an image encoder to map text data and image data into initial text feature vectors and initial image feature vectors, respectively, includes: The BERT model is used as the text encoder. Texts longer than 512 are truncated and input into the BERT model to obtain an initial text feature vector of 768 dimensions. The ResNet-50 model was used as the image encoder to uniformly adjust the image size. The pixels are input into the ResNet-50 model to obtain an initial image feature vector of 2048 dimensions.

[0028] For example, consider a construction log text containing several thousand characters. First, the text is segmented. If the length of the segmented sequence (i.e., the number of tokens) exceeds 512, the sequence is truncated, retaining only the first 512 tokens; the excess is discarded. This 512-to-1 token sequence is then input into the BERT model. After computation through multiple Transformer layers, the model generates a 768-dimensional floating-point vector at the output layer. This vector is the initial text feature vector, encoding the overall semantic information of the text.

[0029] The processing of image data is similar. For example, an image with an original size of... High-resolution photos of the dam, taken at pixel counts, will have their dimensions uniformly adjusted using an image scaling algorithm. Pixels. The adjusted standard-sized image is input into the ResNet-50 model. After a series of convolutional, pooling, and fully connected layers, the model finally outputs a 2048-dimensional feature vector. This vector is the initial image feature vector, which contains rich visual patterns and information from the image.

[0030] In an optional embodiment, the regional visual features of the key component are obtained in the following manner: The image is input into a target detection model, specifically the YOLOv5 model. The target detection model identifies and locates the key components, determining their categories, such as "dam," "spillway," and "gate." For each identified key component, the visual features of the corresponding region in the feature map output from the intermediate layer of the image encoder are extracted.

[0031] The local correspondence is established in the following way: Extract specific engineering terms that match the category of the key component from the archive text, and extract the text features corresponding to the specific engineering terms in the text encoder; The local correspondence is established by calculating the dot product similarity between the textual features of each specific engineering term and the visual features of all regions.

[0032] For example, consider an image containing a water conservancy project. When this image is input into a pre-trained YOLOv5 model, the model outputs a series of bounding boxes and corresponding category labels. For instance, the model might identify the "dam" in the center of the image and provide its coordinates; simultaneously, it might identify the "spillway" on the right side of the image. In this way, these two key components in the image are successfully identified and located.

[0033] Then, this location information is used to extract local visual features. An intermediate convolutional layer in the image encoder generates a feature map that preserves spatial information. Based on the coordinates of the "dam" detection box output by YOLOv5, the system can locate the corresponding region on this feature map and extract the feature data within that region, using it as the regional visual features of the "dam". The regional visual features of the "spillway" are extracted in the same way. Simultaneously, the system searches for the engineering terms "dam" and "spillway" in the accompanying archival text and extracts their respective textual feature vectors from the output of the text encoder.

[0034] To establish local correspondences, the textual features of the term "dam" are calculated, and their dot product similarity is measured with the visual features of the "dam" region and the visual features of the "spillway" region in the image, respectively. Typically, the similarity score for the former ("dam" to "dam") is much higher than that for the latter ("dam" to "spillway"), for example, 0.9 compared to 0.1. By comparing these similarity scores, the system can accurately correlate the term "dam" in the text with the "dam" region in the image, thus achieving fine-grained image-text alignment.

[0035] To adjust the intensity of local alignment during model training, in an optional embodiment, the alignment weights of the local feature alignment loss term are adjusted based on the number of key components and the number of specific engineering terms, including: The total number of key components identified by the object detection model in a single image is denoted as . ; The total number of specific engineering terms in the statistical archive text that match the categories of the identified key components is denoted as . ; Alignment weight The calculation is performed using the following formula: ; in, These are the preset weight hyperparameters.

[0036] For example, suppose we process an image-text pair where, after analysis by an object detection model, five key components are identified: a dam, two gates, and two spillways. Therefore, the total number of key components in the image is... The total number of matching terms in the text is 5. Then, the archival text paired with the image is analyzed, and the number of engineering terms that match the previously identified component categories is counted. For example, if the text mentions a dam once, a gate three times, but no spillway, then the total number of matching terms in the text is [number missing]. The value is 4. Take... and The smaller value in the formula is 4. The alignment weights are calculated according to the given formula. Assume the preset weight hyperparameters. If it is 0.2, then the local alignment weight is... The calculated result is 0.3218. During the backpropagation process of model training, the calculated alignment weight of 0.3218 is used to multiply the loss caused by local feature alignment, thereby adjusting its contribution to the total loss function. If the number of matches for another image-text pair is less, such as only one, the weight will be smaller, and its impact on model parameter updates will be correspondingly weakened.

[0037] In an optional embodiment, the semantic similarity is calculated in the following manner: The TransE model is used to train the knowledge graph of the water conservancy project ontology, generating a 128-dimensional embedding vector for each entity term in the knowledge graph of the water conservancy project ontology. Semantic similarity is calculated by taking the cosine similarity of the entity term embedding vectors corresponding to the text terms in different archives.

[0038] For example, leveraging external knowledge to guide model learning involves first acquiring a knowledge graph of hydraulic engineering ontology. This graph contains various concepts within the domain and their relationships (e.g., "concrete dam" is a type of "dam," and "spillway" is a component of "dam"). Subsequently, models such as TransE are used to learn from this knowledge graph. This model can learn a 128-dimensional vector representation, or embedding vector, for each entity in the graph (e.g., "concrete dam," "earth-rock dam," "gate," etc.). The relative positions of these vectors in the vector space reflect the semantic relationships between entities.

[0039] When comparing the semantic similarity of two different files, the system first identifies the core engineering terms from their respective file texts. For example, the core term for file A is "concrete dam," and the core term for file B is "gravity dam." Next, the system retrieves the 128-dimensional embedding vectors for "concrete dam" and "gravity dam" from the pre-trained TransE model. By calculating the cosine similarity between these two vectors, a value between -1 and 1 is obtained. In this example, the calculated similarity is assumed to be 0.91.

[0040] The calculated similarity score of 0.91 is compared with a pre-set threshold of 0.85. Since 0.91 is higher than 0.85, file A and file B are determined to be highly related semantically. This determination will be used as a constraint in subsequent steps to guide the image-text fusion representations of the two files to be close to each other in the feature space.

[0041] To enable the model to learn the semantic structure contained in the knowledge graph, in an optional embodiment, the semantic structure preservation loss term is obtained by constructing a triplet loss function, wherein the cross-modal representation of the current file is used as the anchor point, the cross-modal representations of files in the water conservancy engineering ontology knowledge graph with a semantic similarity greater than 0.85 with the current file are used as positive samples, and the cross-modal representations of files in the water conservancy engineering ontology knowledge graph with a semantic similarity less than 0.3 with the current file are used as negative samples, minimizing the distance between the anchor point and the positive sample, while maximizing the distance between the anchor point and the negative sample.

[0042] For example, during training, a file is randomly selected as an "anchor point." For instance, file A, whose content is about concrete dams, has its cross-modal representation designated as the anchor point. Then, a "positive sample" and a "negative sample" need to be found for this anchor point. Based on the semantic similarity calculation from the previous step, the system finds file B in the dataset, whose content is about "gravity dams," and its semantic similarity to file A (anchor point) is 0.91. Therefore, the cross-modal representation of file B is selected as a positive sample. Simultaneously, the system also finds file C, whose content is about "pump station management," and its semantic similarity to file A is only 0.2, below the threshold of 0.3. Therefore, the cross-modal representation of file C is selected as a negative sample.

[0043] Using the three samples mentioned above—anchor A, positive sample B, and negative sample C—a triplet loss is calculated. The goal of this loss function is to adjust the model parameters to minimize the distance between the representations of anchor A and positive sample B in the shared semantic subspace, while maximizing the distance between the representations of anchor A and negative sample C, and ensuring that the distance difference (i.e., the margin) between the positive and negative samples and the anchor point is greater than a certain set value. By training on a large number of such triples, the model can eventually map semantically related different files to neighboring regions in the feature space.

[0044] In an optional embodiment, the joint loss function further includes a global alignment loss term, which is a contrastive loss function. The global alignment loss term measures the similarity of the global feature representations of matched text-image pairs within a batch. The goal of the global alignment loss term is to bring the global feature representations of matched pairs closer together and push away the global feature representations of unmatched pairs.

[0045] For example, the global alignment loss term ensures that the image and text match at the overall semantic level. In each iteration of model training, a batch of data is loaded, for example, a batch containing 8 image-text pairs, denoted as ( , ), ( , ), ..., ( , ).in, and These are images and text from the same archive with matching content. For any single image in this batch, for example... It and its corresponding text This constitutes a positive sample pair. And all other text in that batch, i.e. , ,..., This forms 7 negative sample pairs. Similarly, for text... It and the image Forming positive sample pairs, and all other images , ,..., Form negative sample pairs. Then calculate... The similarity score between the global feature representation of the sample and the global feature representation of all 8 texts in the batch. The objective of the contrastive loss function is to maximize the similarity score between positive sample pairs ( , The similarity score of ) while minimizing all negative sample pairs ( , ), ( , Similarity scores, etc. By performing this comparative learning on all possible image-text combinations within a batch, the model learns global feature representations that can effectively distinguish between matching and non-matching image-text pairs, thereby achieving overall alignment across modalities.

[0046] The implementation principle of the water conservancy engineering archive retrieval method in this invention is as follows: On the one hand, this invention utilizes a target detection model to identify key components in images and establishes local correspondences between the regional visual features of these components and specific engineering terms in the archive text. This achieves fine-grained semantic alignment between key local information in the image and core terms in the text, significantly improving the model's understanding depth and accuracy of the core related content of the image and text. On the other hand, this invention utilizes a water conservancy engineering ontology knowledge graph to integrate professional domain knowledge into the construction process of a shared semantic subspace. This helps to mine and utilize the deep semantic relationships between professional terms in different archives, enabling archives that are semantically related but have different expressions to aggregate in the feature space, thereby effectively enhancing the generalization ability and recall rate of the retrieval model.

[0047] An embodiment of the water conservancy project archive retrieval system provided by the present invention includes a memory and a processor. The memory stores computer instructions, and when the processor executes the computer instructions, it implements the water conservancy project archive retrieval method in the above embodiment.

[0048] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for retrieving water conservancy project archives, characterized in that, The process includes the following steps: acquiring water conservancy engineering archives containing text and image data, and using a text encoder and an image encoder to map the text and image data into initial text feature vectors and initial image feature vectors, respectively; constructing a shared semantic subspace, and projecting the initial text feature vectors and initial image feature vectors onto the shared semantic subspace by optimizing a joint loss function to obtain a cross-modal representation. The joint loss function includes a local feature alignment loss term and a semantic structure preservation loss term. The local feature alignment loss term is calculated based on the local correspondence between the regional visual features of key components in the image and the textual features of specific engineering terms in the archive text. The alignment weight of the feature alignment loss term is adjusted according to the number of key components and the number of specific engineering terms; the semantic structure preservation loss term is constructed based on the water conservancy engineering ontology knowledge graph; when the semantic similarity of text terms in different archives in the water conservancy engineering ontology knowledge graph is higher than a preset threshold, the semantic structure preservation loss term makes the cross-modal representations of the different archives closer to each other in the shared semantic subspace; for any query request, the query request is converted into a query vector, and in the shared semantic subspace, the query vector is sorted by calculating the cosine similarity between the cross-modal representations of all archives in the archive database, and the sorting result is used as the retrieval output.

2. The water conservancy project archive retrieval method according to claim 1, characterized in that, The process of mapping text data and image data into initial text feature vectors and initial image feature vectors using a text encoder and an image encoder, respectively, includes: The BERT model is used as the text encoder. Texts longer than 512 are truncated and input into the BERT model to obtain an initial text feature vector of 768 dimensions. The ResNet-50 model was used as the image encoder to uniformly adjust the image size. The pixels are input into the ResNet-50 model to obtain an initial image feature vector of 2048 dimensions.

3. The water conservancy project archive retrieval method according to claim 1, characterized in that, The regional visual features of the key components were obtained through the following methods: The image is input into the target detection model, which identifies and locates the key components and determines their category. For each identified key component, extract the visual features of the corresponding region in the feature map output by the intermediate layer of the image encoder.

4. The water conservancy project archive retrieval method according to claim 3, characterized in that, The local correspondence is established in the following way: Extract specific engineering terms that match the category of the key component from the archive text, and extract the text features corresponding to the specific engineering terms in the text encoder; The local correspondence is established by calculating the dot product similarity between the textual features of each specific engineering term and the visual features of all regions.

5. The water conservancy project archive retrieval method according to claim 4, characterized in that, The alignment weights of the local feature alignment loss term are adjusted based on the number of key components and the number of specific engineering terms, including: The total number of key components identified by the object detection model in a single image is denoted as . ; The total number of specific engineering terms in the statistical archive text that match the categories of the identified key components is denoted as . ; Alignment weight The calculation is performed using the following formula: ； in, These are the preset weight hyperparameters.

6. The water conservancy project archive retrieval method according to claim 1, characterized in that, The semantic similarity is calculated in the following way: The TransE model is used to train the knowledge graph of the water conservancy project ontology, generating a 128-dimensional embedding vector for each entity term in the knowledge graph of the water conservancy project ontology. Semantic similarity is calculated by taking the cosine similarity of the entity term embedding vectors corresponding to the text terms in different archives.

7. The water conservancy project archive retrieval method according to claim 1, characterized in that, The semantic structure preservation loss term is obtained by constructing a triplet loss function, in which the cross-modal representation of the current file is used as the anchor point, the cross-modal representation of files in the water conservancy engineering ontology knowledge graph with a semantic similarity greater than 0.85 with the current file is used as positive samples, and the cross-modal representation of files in the water conservancy engineering ontology knowledge graph with a semantic similarity less than 0.3 with the current file is used as negative samples. The distance between the anchor point and the positive sample is minimized, while the distance between the anchor point and the negative sample is maximized.

8. The water conservancy project archive retrieval method according to claim 1, characterized in that, The joint loss function also includes a global alignment loss term, which is a contrastive loss function. The global alignment loss term is used to measure the similarity of the global feature representations of matched text-image pairs within a batch. The goal of the global alignment loss term is to bring the global feature representations of matched pairs closer together and push away the global feature representations of unmatched pairs.

9. The water conservancy project archive retrieval method according to claim 3, characterized in that, The target detection model is the YOLOv5 model.

10. A water conservancy project archive retrieval system, characterized in that, It includes a memory and a processor. The memory stores computer instructions. When the processor executes the computer instructions, it implements the water conservancy project archive retrieval method as described in any one of claims 1-9.

Images