A method for image-text retrieval graph network based on model reuse

Abstract

The application discloses a kind of based on model reuse image-text retrieval graph network method, first according to user demand from internet public data source constructs multi-field image-text pair data set, utilizes pre-training model and carries out multistage data cleaning to ensure data quality, secondly, reuse large-scale pre-training multimodal model as feature extractor, the deep semantic representation of image text pair is efficiently obtained, and the model training cost is significantly reduced.Then design heterogeneous topological structure, wherein image text node forms homogeneous subgraph respectively, and cross-modal edge is dynamically generated through learnable attention weight.Finally, the image text semantic relation graph is constructed, and the context information of retrieval target is supplemented by graph sampling and aggregation, and then the retrieval feature is generated, and the retrieval task of multimodal combination is completed.The method makes the process more efficient by reusing pre-training model features, and greatly improves the model convergence speed.

Description

Technical Field

[0001] This invention relates to a graph network method for text and image retrieval based on model reuse, which is applicable to intelligent retrieval and model reuse scenarios of massive heterogeneous data in industrial settings, especially suitable for scenarios such as image retrieval, e-commerce recommendation, and multimedia content analysis, and belongs to the field of computer vision and cross-modal retrieval technology. Background Technology

[0002] Image-text cross-modal retrieval systems represent a core technological breakthrough in the field of intelligent recommendation. By deeply integrating visual and semantic features, they demonstrate powerful application value in scenarios such as precise e-commerce recommendations and social media content understanding. This system can efficiently handle complex query needs involving both image search and text enhancement. For example, based on user-uploaded clothing images combined with text enhancement conditions such as "pure cotton material" and "new spring style," it can accurately locate target products from a massive pool of candidate items. Simultaneously, it can dynamically adapt to the retrieval needs of newly added categories and trending elements. Its core technology lies in constructing a unified multimodal representation space, achieving fine-grained alignment between local image features and text keywords through a hierarchical attention mechanism, and leveraging an efficient near-nearest neighbor search algorithm. While ensuring millisecond-level response speeds, it filters the most relevant search results from a library of tens of millions of images, significantly improving conversion rates and the user shopping experience on e-commerce platforms.

[0003] Current mainstream image-text cross-modal retrieval technologies are mainly based on large-scale pre-trained multimodal models. These models, pre-trained on massive image-text datasets on the internet, can learn general visual-language alignment representations. However, this "big data-driven" training paradigm has obvious domain adaptability problems: the image features extracted using general pre-trained models are usually fixed, making it difficult to focus on image details based on text information. This leads to "representation conflicts" when directly reusing pre-trained models, making them unsuitable for image-text retrieval tasks. Optimizing models according to the "pre-training-fine-tuning" paradigm not only destroys the original cross-modal association capabilities of the pre-trained model encoder but may also introduce noise or get trapped in local optima by adjusting the model's initial weights. As a result, the final image-text retrieval graph network loses the robustness of general models in specialized domain retrieval tasks and struggles to form accurate feature representations for specialized domain image-text retrieval tasks. In industrial image and text retrieval scenarios, it is necessary to focus on the local features of images using text information. Therefore, it is necessary to effectively model the fine-grained topological associations between images and conditional text within a professional domain, so as to avoid the problem of semantic detail loss when the model organizes the professional domain knowledge of image and text retrieval tasks (such as the correspondence between pathological description nodes and visual feature nodes of lesion areas in medical image reports). Summary of the Invention

[0004] Purpose of the Invention: Previous methods lacked the ability to structure and model image-text retrieval tasks within specialized domains, and struggled to address the "representational conflict" dilemma faced by general-purpose models in these tasks. This invention provides an image-text retrieval graph network method based on model reuse, enabling "efficient reuse of pre-trained models for specific retrieval tasks." Specifically, firstly, a domain-specific dataset is collected according to user needs. The pre-trained model is then reused to extract meta-features from the dataset, avoiding the high computational costs of retraining from scratch or full-parameter fine-tuning. Next, a graph network is used to model the topological structure of images and automatically focus on key image regions relevant to textual descriptions, generating fine-grained image features. Semantic information from similar images is used to supplement the contextual information for retrieval, enhancing the fine-grained features on the domain-specific dataset and thus assisting in the completion of the retrieval task. Throughout this process, no fine-tuning of the pre-trained model is required; only a graph neural network with a few parameters is needed to efficiently complete downstream tasks.

[0005] Technical Solution: A graph network method for image-text retrieval based on model reuse, comprising three steps: image-text data acquisition, construction of a text-adaptive image feature extraction model, and neighbor image retrieval. First, image data of different distributions and categories are collected from image-text datasets in publicly available data sources. Each image is accompanied by a semantically relevant title, tag, or descriptive text. A multi-level data preprocessing pipeline is built to denoise and filter the data in the original image-text dataset. Next, a text-adaptive image feature extraction model is constructed. This step aims to selectively extract relevant features of the image based on the input text conditions. The specific process of constructing the text-adaptive image feature extraction model is as follows: First, the meta-features of the image and text are extracted using an open-source pre-trained model. Initial node representations are constructed, and the topological structure of a single image is designed. The image's segmented regions constitute homoprime graphs, and cross-modal edges are dynamically generated through a graph attention mechanism. Conditional text-adaptive image features are generated. Finally, neighbor image retrieval is performed. The specific process involves using image nodes and text nodes to construct homoprime graphs, and based on graph sampling and clustering algorithms, obtaining similar images to the query image to supplement contextual information, generating target image features to complete the retrieval.

[0006] The specific steps for collecting image-text datasets are as follows: Step 100: Construct an image and text data acquisition system to obtain raw image information from public data sources and record the image's metadata.

[0007] Step 101: Obtain the labeled text of the image from the same network source address as the original image information. Based on semantic association rules, synchronously obtain multi-dimensional labeled text that is strongly related to the image from the same network source address, including but not limited to user annotations, titles, descriptive text, and scene tags.

[0008] Step 102 employs a three-level data cleaning and filtering mechanism: automated data extraction, pre-trained model filtering of semantic relationships, and expert review. Specifically, the pre-trained model filtering mechanism deploys a pre-trained multimodal model based on the Transformer architecture. It achieves automated semantic matching by calculating cross-modal similarity scores for image-text pairs and sets thresholds to initially filter low-quality samples. The expert review process involves domain experts sampling and inspecting the results to ensure semantic alignment between the images and text and that the image data does not contain low-pixel counts or low-quality issues.

[0009] Step 103: Hierarchical clustering algorithm is used to perform multi-granular semantic clustering on the image dataset. Based on intra-class diversity and inter-class discriminant index, the sampling is optimized to ensure the balance and representativeness of the final dataset in both visual semantics and text description dimensions.

[0010] After steps 100-103 above, a valid training image-text dataset is obtained.

[0011] The specific steps in building a text-adaptive image feature extraction model are as follows: Step 200: Directly use the open-source industrial pre-trained model to extract the block features and text features of the images in the training image-text dataset obtained in the previous step. The block features are non-overlapping image blocks divided into 16*16 segments. This step aims to provide more fine-grained image meta-features for this adaptive image feature extraction model. Step 201: Construct an image block topology graph based on spatial relationships, where each image block is a graph node, and edge relationships are established based on a 4-neighborhood connection strategy to form a regular grid graph structure; Step 202: Introduce text features as global context nodes into the graph structure, and establish cross-modal associations with all image block nodes through a fully connected approach to construct a heterogeneous base graph.

[0012] Step 203: A multi-head image attention mechanism is adopted, setting both unimodal and bimodal dual-constrained contrastive learning loss functions as training objectives, and using image unimodal intra-contrast loss and image-text bimodal contrastive loss as objectives to complete training. Based on the heterogeneous base map generated in step 202, the parameters of the multi-head image attention module are updated to generate a text-conditionally adaptive image feature representation.

[0013] Based on the image features extracted in the previous step, the specific steps for retrieving neighboring images are as follows: Step 300: Construct an image semantic relationship graph based on feature similarity, and use cosine similarity to measure the semantic association strength of the fused features; Step 301: Use an importance-based random walk sampling algorithm to obtain the k nearest neighbors of the query image starting from the query node.

[0014] Step 302: Design a multi-channel feature aggregation module: a neighboring feature attention weighted aggregation module and a max pooling capture salient feature aggregation module. Finally, the features extracted by weighted aggregation and the features captured by max pooling are dynamically fused through a gating mechanism.

[0015] Step 303: Construct the dataset required for sample evaluation, including: 1) candidate image library; 2) reference image set; 3) multimodal query conditions.

[0016] Step 304: Use the approximate nearest neighbor search algorithm to quickly match similar images in the feature library.

[0017] The text-adaptive image feature model in step 203 and the graph sampling and aggregation modules in steps 302 and 303 both adopt a hierarchical feature fusion architecture: First, a heterogeneous multi-layer graph attention network is used to perform cross-modal interaction between the original image block features and text features to generate text-adaptive image features. Then, a graph sampling and aggregation algorithm is designed to complete the context supplementation of the query by neighboring samples. In particular, as mentioned in step 203, a dual-constraint contrastive learning objective function is designed, including intra-modal contrastive loss and cross-modal contrastive loss, which respectively maintain the consistency of image structural characteristics and text semantics, and finally output joint embedded features.

[0018] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the graph network method for graph-text retrieval based on model reuse as described above.

[0019] A computer-readable storage medium storing a computer program that performs the graph network method for text retrieval based on model reuse as described above. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the image and text dataset acquisition process according to an embodiment of the present invention. Figure 2 Flowchart for building the text adaptive image feature extraction model according to an embodiment of the present invention; Figure 3 This is a flowchart of the nearest image retrieval process according to an embodiment of the present invention. Detailed Implementation

[0021] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0022] The following examples illustrate the application of image and text description in industrial quality inspection scenarios for target image retrieval. The image-text pairs processed by the method have typical industrial feature dimensions. Taking semiconductor wafer defect detection as an example, each high-resolution microscopic image corresponds to a structured inspection report text. This text includes defect type (such as scratches, particle contamination, lithographic anomalies, etc.), defect location (precisely marked in wafer coordinates), size parameters (micrometer-level measurement data), and severity classification (critical defect, major defect, minor defect). By constructing a deep learning-based multimodal retrieval model, the method can achieve intelligent matching between defect images and historical case databases, providing quality inspection engineers with references for handling similar defects, significantly improving the efficiency and accuracy of defect analysis. This application scenario places higher demands on the fine-grained alignment of image and text features, requiring the model to accurately understand the complex mapping relationship between microscopic features in microscopic images and professional inspection terminology.

[0023] like Figure 1 The data collection process begins by clearly defining the input data, which includes the queried images and the text of interest. To ensure the consistency of multimodal data quality, a rigorous three-level data filtering mechanism is implemented: first, an automated cleaning pipeline removes obviously unqualified samples such as low-resolution images and incomplete text; second, a pre-trained multimodal model is used for semantic matching verification, retaining only high-quality data pairs with an image-text similarity exceeding 0.85; finally, domain expert sampling audits ensure the accuracy of key attribute annotations. This quality control process ensures that the final training set maintains high semantic consistency across different domains, laying a reliable data foundation for subsequent graph neural network modeling. The images obtained after this filtering step typically reach tens of millions, covering an extremely wide range of domains and feature distributions. To effectively handle this large-scale, high-dimensional data, a hierarchical clustering algorithm is employed. First, PCA dimensionality reduction technology is used to compress image features from their original dimensions to 256 dimensions, significantly improving computational efficiency while maintaining over 95% of the feature information. Subsequently, an improved K-means++ algorithm was applied for cluster analysis. Simultaneously, different clusters were sampled, representing the intra-cluster diversity index, to ensure that the characteristics of the sampled data distribution naturally belong to different clusters, thus completing multi-granular semantic clustering of the image. During the clustering process, a balance constraint was specifically designed, namely, setting a threshold to ensure that the samples are distributed as evenly as possible across as many clusters as possible, representing the inter-cluster discriminancy index. This prevents situations where some clusters have too many samples while others have too few. After clustering, the system randomly selects representative samples proportionally from each cluster, ultimately obtaining a training sample set of moderate quantity but balanced distribution.

[0024] After completing the dataset collection, a text-adaptive image feature extraction model is constructed that can autonomously adjust the focus of image analysis according to different detection needs. Its core technical architecture is as follows: Figure 2 As shown, this scheme first extracts multi-scale features from microscopic images based on large-scale or industrial-grade pre-trained models: the high-resolution detection image is divided into several detection unit regions, and each region is encoded into a feature vector with clear physical meaning through a deep network, i.e., image block features are extracted; at the same time, textual information such as standard process documents and defect description reports are transformed into structured semantic features through a domain-specific language model. Based on this, the system constructs a heterogeneous feature map for industrial quality inspection, in which the block features of the images in the test samples maintain the actual physical spatial arrangement relationship, and process text features are introduced as quality judgment nodes to establish interpretable connections with each detection unit, thus establishing a multimodal heterogeneous map. On this heterogeneous map, the system uses a multi-layer graph attention neural network module for feature extraction and learning. Through adaptive calculation of the attention mechanism, the attention weight for different image regions can be dynamically adjusted according to the semantic content of the conditional text. For example, when processing the detection task of "locating all particulate contamination larger than 5 micrometers", the system will automatically enhance the attention weight for contamination-sensitive areas; while when executing the instruction of "identifying uneven defects in photoresist coating", the system will focus on analyzing the texture features of the coating area. This intelligent feature-focusing mechanism enables the system to accurately capture the key features most relevant to quality inspection requirements, achieving a dual optimization effect of improved defect detection rate and reduced false positive rate in actual production line applications.

[0025] Finally, based on the image feature extraction with enhanced Chinese text conditions in the previous step, the system further employs a refined retrieval strategy based on graph structure to improve result quality and complete the test. For example... Figure 3The system demonstrates the process of retrieving neighboring images and testing samples from a target image. First, based on the constructed image semantic relationship graph, the system uses an optimized approximate nearest neighbor search algorithm, such as measuring distance using Euclidean distance or cosine similarity, to quickly locate the k candidate image nodes most similar to the query features. This process considers not only global feature similarity but also the degree of matching between local image regions and text conditions. Subsequently, the system innovatively designs a multi-channel feature aggregation mechanism. On the one hand, it retains the most relevant semantic information through attention-weighted feature fusion; on the other hand, it uses multi-scale pooling operations to capture important features at different granularities. Simultaneously, based on the characteristics of deep neural network training, it adds the input of the previous layer to the next layer, introducing residual connections to ensure the integrity of the original features. During the actual retrieval process, the system dynamically adjusts the aggregation weights using a gating mechanism. The user-input test samples are processed through the above steps to obtain the final enhanced features, which are then used to select the target image from the candidate images. This composite feature enhancement method effectively integrates visual similarity and semantic relevance, ultimately generating a more discriminative target image representation to complete the retrieval task. It demonstrates significant advantages in handling complex and compound queries, providing a reliable solution for scenarios requiring high-precision cross-modal search, such as e-commerce platforms and medical image retrieval.

[0026] Obviously, those skilled in the art should understand that the steps of the graph network method for text retrieval based on model reuse described in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

Claims

1. A graph network method for text retrieval based on model reuse, characterized in that, The process comprises three steps: image-text data acquisition, construction of a text-adaptive image feature extraction model, and neighbor image retrieval. First, image data of different distributions and categories is collected from publicly available image-text datasets. Each image is accompanied by a semantically relevant title, label, or descriptive text. A multi-level data preprocessing pipeline is built to denoise and filter the data in the original image-text dataset. Next, a text-adaptive image feature extraction model is constructed. This step aims to selectively extract relevant image features based on the input text conditions. The specific process of constructing the text-adaptive image feature extraction model involves first extracting meta-features of images and text using a pre-trained model, constructing initial node representations, designing the topological structure of a single image, and forming homogeneous subgraphs from image regions. Cross-modal edges are dynamically generated using a graph attention mechanism. Conditional text-adaptive image features are then generated. Finally, neighbor image retrieval is performed. The specific process involves using image nodes and text nodes to form homogeneous subgraphs, and using graph sampling and clustering algorithms to obtain similar images to supplement contextual information, generating target image features to complete the retrieval. The specific steps in building the text-adaptive image feature extraction model are as follows: Step 200: Use an industrial pre-trained model to extract image block features and text features from the image-text dataset. The block features are non-overlapping image blocks that are divided into several segments, providing more fine-grained image meta-features for the adaptive image feature extraction model. Step 201: Construct an image block topology graph based on spatial relationships, where each image block is a graph node, and edge relationships are established based on a 4-neighborhood connection strategy to form a regular grid graph structure; Step 202: Introduce text features as global context nodes into the graph structure, and establish cross-modal associations with all image patch nodes through a fully connected approach to construct a heterogeneous base graph; Step 203: A multi-head image attention mechanism is adopted, and the dual-constraint contrastive learning loss function of unimodal and bimodal modes is set as the training objective. The training is completed with the image intramodal contrastive loss and the image-text bimodal contrastive loss as the objectives. Based on the heterogeneous base map generated in step 202, the parameters of the multi-head image attention module are updated to generate a text conditionally adaptive image feature representation.

2. The graph network method for text retrieval based on model reuse according to claim 1, characterized in that, The specific steps for acquiring the image-text dataset are as follows: Step 100: Construct an image and text data acquisition system to obtain raw image information from public data sources and record the image's metadata. Step 101: Obtain the labeled text of the image from the same network source address as the original image information; based on semantic association rules, synchronously obtain multi-dimensional labeled text related to the image from the same network source address. Step 102 adopts a three-level data cleaning and filtering mechanism, namely, automatic data extraction, pre-trained model filtering of semantic relationships, and expert review; the pre-trained model filtering of semantic relationships mechanism specifically involves deploying a pre-trained multimodal model based on the Transformer architecture, realizing automatic semantic matching by calculating the cross-modal similarity score of image-text pairs, and setting a threshold to complete the initial filtering of low-quality samples. Step 103: Hierarchical clustering algorithm is used to perform multi-granular semantic clustering on the image dataset. Based on intra-class diversity and inter-class discriminant index, the sampling is optimized to ensure the balance and representativeness of the final dataset in both visual semantics and text description dimensions. After steps 100-103 above, a valid training image-text dataset is obtained.

3. The graph network method for text retrieval based on model reuse according to claim 1, characterized in that, Based on image features, the specific steps for retrieving neighboring images are as follows: Step 300: Construct an image semantic relationship graph based on feature similarity, and use cosine similarity to measure the semantic association strength of the fused features; Step 301: Use an importance-based random walk sampling algorithm to obtain the k nearest neighbors of the query image starting from the query node; Step 302: Design a multi-channel feature aggregation module, which consists of a neighboring feature attention weighted aggregation module and a max pooling capture salient feature aggregation module. Finally, the features extracted by weighted aggregation and the features captured by max pooling are dynamically fused through a gating mechanism. Step 303: Construct the dataset required for sample evaluation, including: 1) a candidate image library; 2) a reference image set; 3) multimodal query conditions; Step 304: Use the approximate nearest neighbor search algorithm to quickly match similar images in the feature library.

4. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the graph network method for graph retrieval based on model reuse as described in any one of claims 1-3.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that performs the graph network method for text retrieval based on model reuse as described in any one of claims 1-3.

Images