An image retrieval method and device based on multi-modal feature fusion

Abstract

The application provides a kind of image retrieval method and device based on multimodal feature fusion, the method comprises: extracting the global feature and local feature of retrieval text, retrieval sketch and candidate image respectively;The global feature of the retrieval text and the retrieval sketch is fused, and the global similarity between the fused feature and the global feature of the candidate image is calculated;Based on the local feature of retrieval sketch and candidate image, the corresponding target area feature is determined respectively;The local feature of retrieval text and the target area feature of retrieval sketch are interactively fused, and the fusion result is matched with the target area feature of candidate image, to obtain local similarity;According to global similarity and local similarity, determine the retrieval result from candidate image, not only give full play to the complementary advantage of text semantic and sketch visual information, but also improve the accuracy and robustness of image retrieval.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image retrieval method and apparatus based on multimodal feature fusion. Background Technology

[0002] Image Retrieval with Text and Sketch (TSBIR) is a novel retrieval technology that combines semantic description with visual intuitive expression. This technology utilizes text to describe image attributes such as color and scene, and uses sketches to express the image's shape, texture, and layout. By extracting features from each modality and building an index, it aims to overcome the limitations of single-modality retrieval and meet users' more accurate and diverse image search needs.

[0003] Existing TSBIR schemes typically employ a global feature matching strategy. For example, convolutional neural networks (CNNs) or basic attention models are used to extract global feature vectors for text, sketches, and images, respectively. Subsequently, metric learning (such as Quadruplet Loss) or feature interaction networks are used to map the features of the text and sketches into the same feature space as the image. The retrieval process primarily relies on calculating the overall similarity between the query vector and candidate image vectors in the image database to determine the most relevant target image.

[0004] However, existing technical solutions mainly focus on global feature alignment between different modalities, making it difficult to effectively capture and utilize local detail features in images and queries. When processing images with cluttered backgrounds or rich details, global features alone cannot accurately match the fine-grained relationship between specific words in text or local structures in sketches and corresponding targets in images. This results in insufficient model perception of details, thus limiting the accuracy and robustness of image retrieval. Summary of the Invention

[0005] This invention provides an image retrieval method based on multimodal feature fusion (invention title) to address the shortcomings of existing technologies that rely solely on global feature alignment and struggle to capture local detail features, thus limiting retrieval accuracy.

[0006] This invention provides an image retrieval method based on multimodal feature fusion, comprising: Global and local features of the search text, search sketch, and candidate images are extracted respectively; The global features of the search text and the search sketch are fused, and the global similarity between the fused features and the global features of the candidate image is calculated. Based on the retrieval sketch and the local features of the candidate images, the corresponding target region features are determined respectively; The local features of the retrieved text and the target region features of the retrieved sketch are interactively fused, and the fusion result is matched with the target region features of the candidate image to obtain the local similarity. The retrieval results are determined from the candidate images based on the global similarity and the local similarity.

[0007] According to the method provided by the present invention, the extraction of global and local features of the retrieval text, retrieval sketch, and candidate image respectively includes: The retrieved text is segmented to obtain a word sequence, and a first special marker is inserted at the beginning of the sequence. The word sequence containing the first special marker is input into a text encoding model, and the output vector of the first special marker is extracted as the global feature of the text. The output vector of the word sequence is extracted as the local feature of the text. The retrieved sketch is divided into sub-block sequences, and a second special marker is inserted at the beginning of the sequence. The sub-block sequence containing the second special marker is input into the sketch coding model. The output vector of the second special marker is extracted as the global feature of the sketch, and the output vector of the sub-block sequence is extracted as the local feature of the sketch. The candidate image is divided into sub-block sequences, and a third special marker is inserted at the beginning of the sequence. The sub-block sequence containing the third special marker is input into the image coding model. The output vector of the third special marker is extracted as the global feature of the image, and the output vector of the sub-block sequence is extracted as the local feature of the image.

[0008] According to the method provided by the present invention, the step of fusing the global features of the search text and the search sketch, and calculating the global similarity between the fused features and the global features of the candidate image, includes: vector-adding the global features of the search text and the global features of the search sketch to obtain joint global features; and using a contrastive learning mechanism to calculate the similarity value between the joint global features and the global features of the candidate image based on the cosine similarity function, as the global similarity.

[0009] According to the method provided by the present invention, determining the corresponding target region features based on the local features of the retrieval sketch and the candidate image includes: The target detection network identifies the sketch target bounding box in the retrieval sketch and the image target bounding box in the candidate image, respectively; the target detection network is used to determine the region range where the target object contained in the retrieval sketch and the candidate image is located. Establish a positional mapping relationship between the bounding box of the sketch target and the local features of the retrieved sketch, and fuse the local features mapped within the bounding box of the sketch target to obtain the sketch target region features; Establish a positional mapping relationship between the bounding box of the image target and the local features of the candidate image, and fuse the local features mapped within the bounding box of the image target to obtain the image target region features.

[0010] According to the method provided by the present invention, the local features of the retrieved text include multiple text sub-features; the interactive fusion based on the local features of the retrieved text and the target region features of the retrieved sketch includes: Calculate the similarity distribution between each of the text sub-features and the target region features; Based on the similarity distribution, at least one of the target region features with a correlation higher than a threshold is selected for each text sub-feature; The selected target region features are aggregated and concatenated with the corresponding text sub-features. The concatenated features are then subjected to feature transformation processing to generate text-guided text-sketch fine-grained fusion features.

[0011] According to the method provided by the present invention, the step of matching the fusion result with the target region features of the candidate image to obtain local similarity includes: By utilizing the cross-attention mechanism, feature interaction calculations are performed on the text-sketch fine-grained fusion features and the image target region features to obtain the global representation features after interaction. Based on the global representation features, the probability that the search text and the search sketch form a matching relationship with the candidate image is predicted, and the probability is used as the local similarity between the search text and the search sketch and the candidate image.

[0012] According to the method provided by the present invention, the method further includes the steps of constructing a training set and training and optimizing the model, specifically including: A training batch containing multiple sets of training samples is constructed. Each set of training samples includes retrieval sample text, retrieval sample sketch, and candidate sample image with corresponding relationships. The retrieval sample text, retrieval sample sketch, and candidate sample image within the same set constitute positive samples, and the combination of retrieval sample text, retrieval sample sketch, and candidate sample image with other sets of candidate sample images within the training batch constitutes negative samples. Before feature extraction, a mask is randomly added to the search sample text and the search sample sketch according to a preset ratio; Features are extracted using text encoding model, sketch encoding model and image encoding model respectively. The global similarity and local similarity are calculated based on the positive samples and the negative samples. The overall optimization objective is constructed by combining the global loss function and the local loss function. Backpropagation is performed based on the overall optimization objective to update the parameters of the text encoding model, the sketch encoding model, and the image encoding model. When updating the parameters of the text encoding model and the sketch encoding model, the difference in their contributions to the learning objective is monitored, the intermodal difference ratio is calculated, and the parameter update is performed using the gradient descent rate adaptively adjusted according to the difference ratio.

[0013] The present invention also provides an image retrieval device based on multimodal feature fusion, comprising: The extraction module is used to extract global and local features from the search text, search sketch, and candidate images, respectively. The fusion module is used to fuse the global features of the search text and the search sketch, and calculate the global similarity between the fused features and the global features of the candidate image. The selection module is used to determine the corresponding target region features based on the local features of the retrieval sketch and the candidate image, respectively. The matching module is used to interactively fuse the local features of the searched text with the target region features of the search sketch, and to match the fusion result with the target region features of the candidate image to obtain the local similarity. The determination module is used to determine the retrieval results from the candidate images based on the global similarity and the local similarity.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image retrieval method based on multimodal feature fusion as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image retrieval method based on multimodal feature fusion as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image retrieval method based on multimodal feature fusion as described above.

[0017] This invention provides an image retrieval method and apparatus based on multimodal feature fusion. It extracts global and local features from the retrieval text, retrieval sketch, and candidate image respectively. Based on the fusion of global features from the retrieval text and retrieval sketch to achieve coarse-grained semantic alignment with the overall candidate image, it further locks the target region of the retrieval sketch and candidate image based on local features. Through the interactive fusion of local text features and sketch target region features, it generates fusion features rich in fine-grained semantics. These fusion features are then precisely matched with the target region features of the candidate image to obtain local similarity. Finally, based on global and local similarity, the retrieval result is determined from the candidate image. This dual matching mechanism, combining global overview and local fine-grained interaction, not only fully leverages the complementary advantages of text semantics and sketch visual information but also significantly improves the ability to distinguish subtle targets and complex backgrounds in the image through feature interaction targeting the target region, thereby greatly improving the accuracy and robustness of image retrieval. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the image retrieval method based on multimodal feature fusion provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the network structure of the image retrieval method based on multimodal feature fusion provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the structure of the image retrieval device based on multimodal feature fusion provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] With the rapid expansion of image data and the increasing diversity of application scenarios, users are demanding higher accuracy, flexibility, and naturalness in image retrieval. Traditional text-based image retrieval (TBIR) relies on keyword matching or semantic understanding, which can reflect the overall semantics of an image well. However, it is limited by the abstract and subjective nature of text descriptions, making it difficult to accurately capture the spatial structure, object shape, and detailed texture of an image. Sketch-based image retrieval (SBIR), on the other hand, matches user-drawn outlines, which can intuitively express the geometric layout and relative position of objects. However, its retrieval performance is often constrained by the user's drawing skills, the simplification level of the sketch, and the robustness of the feature extraction algorithm. To integrate the complementary advantages of text and sketches, text-based sketch image retrieval (TSBIR) has gradually become a research hotspot. It aims to simultaneously utilize the high-level semantics of text and the visual structural information of sketches to achieve more accurate image retrieval that better matches the user's intent.

[0025] However, most existing TSBIR methods focus on global feature alignment between different modalities. For example, they extract global feature vectors for each modality using convolutional neural networks (CNNs) or basic Transformers, then map them to a shared feature space using metric learning or feature interaction networks, and finally complete the retrieval based on global similarity. While these methods can establish macroscopic associations across modalities, they often struggle to achieve fine-grained semantic correspondences when dealing with images with complex backgrounds, multi-object details, or prominent local structures. Specifically, existing solutions typically lack explicit modeling of the local associations between specific words in the text and specific regions in the sketch, as well as the corresponding objects in the image. This results in insufficient perception of detailed features, thus limiting further improvements in retrieval accuracy.

[0026] Based on the aforementioned technical background, and to overcome the bottleneck of existing retrieval schemes in detail capture capabilities, this application proposes an image retrieval scheme based on multimodal feature fusion. This scheme fully leverages the advantages of the Transformer architecture in extracting sequential local features, and is no longer limited to a single global matching dimension. By introducing an object detection mechanism to explicitly locate key regions and constructing a fine-grained feature interaction network based on an attention mechanism, this scheme aims to achieve deep alignment of text semantics, sketch structure, and image content at both the global overview and local detail levels, thereby accurately responding to the user's complex search intent.

[0027] Before introducing the technical solutions of the embodiments of the present invention, the terminology involved in the embodiments of the present invention will be described illustratively.

[0028] Text-Based Image Retrieval (TBIR) is a technique that uses text as query input to retrieve images. Users input descriptive keywords or sentences, and the system analyzes the semantic content of the text to search for semantically matching images in an image database.

[0029] Sketch-Based Image Retrieval (SBIR) is a technique that uses hand-drawn sketches as query input to retrieve images. Users draw simple line drawings or outlines, and the system analyzes the shape, structure, and topological features of the sketches to find visually similar natural images.

[0030] Image Retrieval with Text and Sketch (TSBIR) refers to a comprehensive retrieval technology that combines the two modalities mentioned above. This technology allows users to simultaneously input semantic descriptive text and visual sketches, aiming to utilize text to describe semantic attributes such as color and scene, while using sketches to describe visual attributes such as object shape and posture, thereby achieving more accurate image retrieval that better matches the user's intent.

[0031] Search text: refers to the textual information entered by the user to describe the content of the target image. In this embodiment, the search text is mainly used to provide semantic clues about the target object's category, color, material, or background environment.

[0032] Retrieval sketch: refers to a line drawing input by the user that depicts the visual structure of a target image. In this embodiment, the retrieval sketch is mainly used to provide geometric clues about the target object's location, shape, orientation, and spatial layout.

[0033] Candidate images: refer to natural images stored in an image database that are to be processed and sorted by the retrieval system. The method in this embodiment aims to calculate the similarity between the retrieval text and retrieval sketch and the candidate images in order to find the best-matching target image.

[0034] Global features: These are high-dimensional feature vectors that can characterize the overall semantics or visual appearance of input data (text, sketches, or images). In Transformer-based models, the output vector corresponding to a specific classification label (such as [CLS] Token) is usually taken as the global feature for coarse-grained overall matching.

[0035] Local features are sequences of feature vectors that characterize fine-grained details in input data. For text, they correspond to the features of individual tokens; for sketches and images, they correspond to the features of individual segments. Local features preserve rich spatial information and detailed semantics, enabling fine-grained interactions and matching.

[0036] Text encoding models refer to deep neural networks used to process retrieved text, typically employing a Transformer-based architecture (such as BERT). Their function is to convert discrete text symbols into continuous feature vectors that can be processed by a computer, including global semantic vectors and local word vectors.

[0037] Sketch encoding model: refers to a deep neural network used to process and retrieve sketches, typically employing the Visual Transformer (ViT) architecture. Its function is to extract visual features from sparse line drawings, capturing the overall structure and local line details of the sketch.

[0038] Image coding model: refers to a deep neural network used to process candidate images, often employing the Visual Transformer (ViT) architecture. Its function is to extract features from natural images and generate global and local visual representations for subsequent matching calculations.

[0039] Text-Sketch Fine-Grained Fusion Features: These are composite features generated by injecting textual semantics into the visual features of a sketch by calculating the correlation between local text features and target region features of the sketch. This feature integrates the semantic description of the text with the corresponding region information of the sketch, serving as a key intermediate representation connecting abstract semantics and visual shape.

[0040] Image target region features: These refer to the feature representations extracted and aggregated from local features of candidate images based on target detection results, representing only the specific target object region. They eliminate background interference and focus on the main part of the image most relevant to the retrieval intent.

[0041] The implementation of the method in this embodiment of the invention relies on electronic devices and network environments with certain computing and storage capabilities. Wherein: Computing devices can be deployed on cloud servers, edge computing nodes, or personal computing devices (such as high-end PCs and workstations). Because forward inference involving multiple Transformer models (text encoding, visual encoding, object detection, feature fusion and matching) is involved, certain requirements are placed on the device's CPU / GPU computing power and memory. A powerful GPU cluster is typically required during the training phase.

[0042] The storage device needs to store a large-scale image database for retrieval, pre-trained model parameters, and intermediate feature indexes generated at runtime.

[0043] Network environment (for client-server architecture): The user side needs a stable network connection to upload input text and sketch data to the server. The server side needs network bandwidth and computing resources to handle concurrent requests.

[0044] Input / output devices: Input devices need to support text input (such as keyboards, speech-to-text devices) and sketching (such as touch screens, graphics tablets, mice); output needs to be displayed on a screen to show the search results.

[0045] Figure 1 This is one of the flowcharts illustrating the image retrieval method based on multimodal feature fusion provided by the present invention, such as... Figure 1 As shown, the method includes: Step 101: Extract the global and local features of the search text, search sketch, and candidate image respectively.

[0046] This step forms the basis for constructing a multimodal joint retrieval representation. Its core task is to transform and deconstruct the diverse original queries (text, sketches) and the retrieval targets (images) into a unified, machine-understandable representation with a two-layer structure. Specifically, for each modality of data, a feature extraction network processes the data in parallel, generating a global feature set and a local feature set respectively.

[0047] Global features aim to provide a highly abstract and concise summary of the overall semantics or overall picture of the entire data sample. For example, for retrieved text, its global features should represent the overall intent or theme of the description; for sketches or images, its global features should capture the overall layout, main scene, or high-level visual concept of the core object. This feature provides a compact semantic anchor for subsequent cross-modal coarse-grained alignment.

[0048] Local features aim to preserve and structurally express the detailed components and relationships within the data. It is not a single vector, but rather a set or sequence of features. For text, local features may correspond to the contextual semantics of keywords or phrases in a sentence; for sketches or images, local features are associated with their segmented sub-regions or potential points of interest, characterizing the detailed visual patterns of local shapes, textures, edges, or specific objects. This set of features provides rich material for subsequent fine-grained, targeted cross-modal interaction and matching.

[0049] Through this step, the originally heterogeneous text, sketch, and image data are uniformly mapped to a two-level feature space: a global layer for overall semantic overview and a local layer for detailed component analysis, laying the data foundation for implementing a phased and refined retrieval strategy in subsequent steps.

[0050] Step 102: Fuse the global features of the search text and the search sketch, and calculate the global similarity between the fused features and the global features of the candidate image.

[0051] After step 101, both the retrieved text and the retrieved sketch possess global features representing their overall content. However, the user-inputted text and sketch have inherent semantic and visual complementarity and consistency, together constituting a complete composite query intent. The core of this step is, firstly, to integrate these two global features from different modalities into a unified and more expressive joint global feature representation through a fusion operation. This fusion process essentially combines the abstract semantics of the text with the overall visual structural information of the sketch, generating a composite feature vector that more comprehensively and robustly represents the user's complete query intent.

[0052] Subsequently, this step utilizes the generated joint global features to compare their relevance with the global features of each candidate image in the image database, calculating a global similarity score. This score quantitatively measures the degree of semantic matching between the overall content of the candidate image (such as scene, subject, and main object composition) and the user's composite query intent expressed through text and sketches. This global similarity, as a preliminary, macro-level matching indicator, functions to quickly filter and coarsely sort massive amounts of candidate images, effectively narrowing the search scope for subsequent fine-grained matching, thereby improving the efficiency of the entire retrieval system. It establishes the most basic semantic association between the text-sketch query and the candidate images, and is the first key matching step in the entire hierarchical retrieval process.

[0053] Step 103: Based on the local features of the retrieval sketch and the candidate image, determine the corresponding target region features respectively.

[0054] After step 101, detailed sets of local features for both the retrieval sketch and the candidate image are obtained. However, not all local features are equally important. The sketch lines drawn by the user are usually intended to outline a specific object or structure of interest; similarly, an image often contains one or more main targets. Therefore, the core task of this step is to identify and extract aggregated features corresponding to salient visual entities or key regions from the cluttered or redundant local features, i.e., target region features.

[0055] For retrieving sketches, the system analyzes their local features (such as image patch features) to identify areas in the sketch where brushstrokes are concentrated, structures are closed, or features are prominent. These areas often correspond to specific objects or parts that the user wants to query. These related local features are merged and integrated to form target region features representing different sketch objectives. These features go beyond raw pixels or low-level edges, carrying the user's structured visual intent expressed through the sketch regarding the shape, position, or relative relationships of the target object.

[0056] For candidate images, determining target region features is a content extraction process. The system analyzes its local features to automatically discover regions in the image that may constitute independent semantic entities (such as objects, animals, buildings, etc.). By aggregating multiple local features describing the same entity or the same coherent visual structure, a set of target region features is generated for the image. Each such feature represents a latent semantic unit or visual center of interest in the image, providing a meso-level structured summary of the image content.

[0057] Through this step, the high-dimensional, fine-grained local features of the retrieved sketches and candidate images are transformed into a more representative and semantically rich set of target region features, enabling the matching process to shift from a coarse-grained global-to-global approach to a precise, key-local-to-key-local comparison.

[0058] Step 104: Based on the local features of the retrieved text and the target region features of the retrieved sketch, perform interactive fusion, and match the fusion result with the target region features of the candidate image to obtain the local similarity.

[0059] The goal of this step is to establish precise correspondences at the key detail level, building upon global semantic association. This process consists of two closely linked stages: fine-grained cross-modal fusion and goal-oriented local matching.

[0060] In the first stage, fine-grained cross-modal fusion, the system deeply interacts with the local features of the retrieved text (containing word-level semantics) and the target region features of the retrieved sketch (representing the visual entities of interest to the user). The purpose of this interaction is to semantically associate and bind specific attributes or object terms in the text description with specific shapes or structural regions depicted in the sketch. Through this interaction, the discrete semantic information of the text is injected into or anchored to the corresponding visual structure of the sketch, thereby generating fine-grained joint features that fuse the semantic details of the text and the visual details of the sketch. This feature is no longer a simple stacking of text features or sketch features, but constitutes a more interpretable structured representation of the user's query intent, clearly identifying which word in the text describes which part of the sketch.

[0061] In the second stage, target-oriented local matching, the system performs a one-to-one comparison between the generated fine-grained joint features and the target region features (representing latent semantic entities in the image) of the candidate image. This matching calculation is not a simple overall similarity measure, but rather a targeted evaluation of whether the text-sketch local association patterns contained in the fine-grained joint features match the feature patterns of the corresponding regions in the candidate image. This calculation generates a local similarity score for each candidate image. This score quantitatively measures the degree to which the candidate image satisfies the user's complex query (specific text attributes corresponding to specific sketch structures) at a detailed level, and is a key indicator for judging whether the image content deeply matches the user's complex and concrete intentions.

[0062] This step enhances retrieval capabilities, moving from overall relevance to detailed correspondence. Local similarity and global similarity complement each other; the former ensures that search results accurately target user intent in key details, while the latter ensures that results do not deviate from the topic in overall semantics. Together, they constitute a comprehensive and hierarchical evaluation system for candidate images.

[0063] Step 105: Determine the retrieval results from the candidate images based on the global similarity and the local similarity.

[0064] After steps 102 and 104, the system has generated two types of core evaluation metrics for each candidate image: global similarity and local similarity. The former reflects the degree of matching between the image and the query in terms of overall semantics and macroscopic visual layout; the latter reflects the depth of matching between the image and the query in terms of key details, specific object attributes, and their spatial / visual relationships.

[0065] The decision-making logic in this step is not a simple summation of the two scores, but rather an organic fusion of them through a comprehensive evaluation function to calculate a final comprehensive matching score. This function is designed to balance overall relevance and local accuracy. For example, a weighted summation can be used to assign different weights to global and local similarity, where the weight parameters can be configured or learned based on the emphasis on overall matching and detail matching in different application scenarios. This fusion mechanism ensures that an image only achieves the highest comprehensive score when the overall subject is correct (high global similarity) and the key details are also largely matched (high local similarity), thus effectively avoiding false detections caused solely by background similarity or local coincidence.

[0066] Finally, the system sorts all candidate images in descending order based on their comprehensive matching scores and outputs the top-ranked image or group of images as the retrieval result. This result represents the system's optimal response to the complex, multi-layered query intent of a user simultaneously inputting text and sketches. Through this dual filtering and ranking mechanism that combines breadth (global) and depth (local) approaches, this method can efficiently and accurately locate targets in massive image databases, significantly improving the accuracy and practicality of complex cross-modal retrieval tasks.

[0067] The image retrieval method based on multimodal feature fusion provided in this invention extracts global and local features from the retrieval text, retrieval sketch, and candidate image respectively. Based on the coarse-grained alignment of the overall semantics of the retrieval text and retrieval sketch with the candidate image through global feature fusion, it further locks the target region of the retrieval sketch and candidate image based on local features. Through the interactive fusion of local text features and sketch target region features, it generates fused features rich in fine-grained semantics, which are then used for precise local matching calculations with the target region features of the candidate image. This dual matching mechanism, combining global overview and local fine-grained interaction, not only fully leverages the complementary advantages of text semantics and sketch visual information but also significantly improves the ability to distinguish subtle targets and complex backgrounds in the image through feature interaction targeting the target region, thereby greatly improving the accuracy and robustness of image retrieval.

[0068] Furthermore, to more clearly illustrate the specific execution process of feature extraction, the following will describe the processing details for three modalities of data: text, sketch, and image.

[0069] For the retrieved text, it is first segmented into words, resulting in a sequence of lexical units composed of basic semantic units. To obtain a representation that summarizes the intent of the entire text, a specific first-class special tag (e.g., the [CLS] tag in the BERT model) is inserted at the beginning of this sequence. The embedded complete sequence is then fed into a pre-trained text encoding model. This model models and transforms the contextual relationships of all elements in the sequence using a deep neural network. In the model's final output, the output vector corresponding to the first-class special tag is extracted as a global text feature, which encapsulates the core semantics of the entire input text. Simultaneously, the output vectors corresponding to each lexical unit in the original lexical sequence are extracted to form a set of local text features, where each vector contains the specific meaning of the corresponding lexical unit in the sentence.

[0070] For retrieving a sketch, it is first segmented into multiple regular image sub-blocks, and these sub-blocks are arranged into a sequence. At the beginning of this sequence, a second-class special label specifically for the sketch is added. The entire sequence is then fed into a pre-trained visual encoding model. This model understands the visual content of the image blocks and their spatial relationships. From the model output, the output vector corresponding to the second-class special label is extracted as the global feature of the sketch, reflecting the main outline and layout of the entire sketch. Simultaneously, the output vector corresponding to each original sketch sub-block is extracted from the model to form a set of local sketch features, which describe the fine structure of each part of the sketch.

[0071] For candidate images, the processing flow is similar to that of the sketch, but a separate encoding model is used. First, the image is divided into image sub-blocks and formed into a sequence, with a third-class special label inserted at the beginning of the sequence. This sequence is then input into another pre-trained visual encoding model. Finally, the output vector corresponding to the third-class special label is extracted as the global image feature to characterize the overall scene and content of the image. Extracting the output vectors corresponding to each image sub-block from the model constitutes the local image feature set, capturing the visual details of different regions in the image.

[0072] Through the parallel and structurally consistent processing flow described above, text, sketch, and image data are transformed into representations with the same paradigm (a global feature vector and a local feature sequence), providing a standardized and information-complete data foundation for subsequent cross-modal fusion and matching.

[0073] The training of the text coding model, sketch coding model, and image coding model is a systematic process aimed at enabling them to learn to work collaboratively and accurately establish cross-modal associations through a large amount of data. The training process mainly includes three stages: data organization, forward model computation and loss construction, and backward parameter update and optimization.

[0074] First, in the data organization and augmentation phase, structured training batches need to be constructed. Each batch contains multiple sets of training samples. Each set consists of semantically corresponding retrieval sample text, retrieval sample sketch, and candidate sample image, forming a positive sample triplet. Within the same batch, combining the text and sketch of one set of samples with images from other sets of samples can construct a large number of negative samples. This intra-batch negative sample construction method is efficient and meets the needs of contrastive learning. To improve the model's robustness to incomplete or noisy inputs, before inputting the text and sketch into their respective encoding models, some words or image sub-blocks are randomly replaced with uniform mask symbols at a preset ratio to simulate the actual situation of missing information, enabling the model to learn based on contextual reasoning.

[0075] Secondly, in the feature extraction and loss calculation stage, the processed batch data is input into three encoding models respectively. Global and local features are extracted according to the aforementioned steps, and target region features are further calculated. Subsequently, two core calculations are performed: First, the global features of the text and sketch are fused, and the global feature distance of positive sample triples is narrowed by a comparative loss function (e.g., cross-entropy loss based on cosine similarity), while the distance of negative sample combinations is widened, thus obtaining the global loss. Second, fine-grained feature fusion of text and sketch is performed, and then matched with the image target region features. The correctness of local matching is evaluated using a binary classification matching loss function (e.g., binary cross-entropy loss), thus obtaining the local loss. Finally, the global and local losses are weighted and summed to form the overall optimization objective guiding the model's learning.

[0076] Finally, in the parameter update and balancing optimization phase, the gradients of the three encoding models are calculated and the parameters are updated based on the overall optimization objective using the backpropagation algorithm. A key optimization strategy is adaptive gradient adjustment for the text and sketch query modalities. Since the information density and learning difficulty of text and sketches typically differ, the difference in their contributions to the reduction of the overall loss is monitored during training, and an inter-modal difference ratio is calculated. Based on this ratio, the gradient magnitude or learning rate used to update the parameters of the text encoding model and the sketch encoding model during backpropagation is dynamically adjusted. For example, when one modality converges too quickly and its contribution decreases, its gradient is appropriately increased to encourage the other modality to learn faster; conversely, it is decreased to prevent the dominant modality from dominating training. This dynamic balancing mechanism ensures that the features of both the text and sketch modalities are learned fully and collaboratively, thereby generating a more balanced and effective cross-modal joint representation.

[0077] Through the above multi-stage training and optimization, the system can finally obtain a well-trained model, laying the foundation for accurate image retrieval.

[0078] Furthermore, in order to achieve the initial integration and coarse-grained screening of query intent, step 102 includes two key operations: feature fusion and similarity measurement.

[0079] First, global feature fusion is performed. The global feature vectors representing the overall content extracted from the text encoding model and the sketch encoding model, respectively, are combined. A direct and effective fusion method is to add these two feature vectors together to obtain a unified joint global feature. This addition operation is equivalent in the feature space to linearly superimposing and integrating the abstract semantics of the text and the overall visual structure of the sketch. The resulting joint feature can more comprehensively reflect the complete query intent expressed by the user through both text description and sketch drawing.

[0080] Subsequently, similarity measurement and calculation are performed based on this joint global feature. This step is implemented using a contrastive learning mechanism within a batch training framework. Specifically, a cosine similarity function is used as a metric to calculate the directional consistency between the joint global feature and the global features of each candidate image in the batch, resulting in an initial similarity value. To more effectively distinguish between positive and negative samples during training, a learnable temperature parameter is typically introduced to scale this value. Within a batch, candidate images that truly correspond to the current joint feature constitute positive sample pairs, while combinations with other images constitute negative sample pairs. Through algorithm optimization, the model is encouraged to bring the similarity values ​​between positive sample pairs closer together, while simultaneously pushing away the similarity values ​​between negative sample pairs.

[0081] During the retrieval inference phase, the final similarity score calculated by the model after optimization using this mechanism, between the joint global features and the global features of a candidate image, is defined as the global similarity between the two. This score quantifies the degree to which the candidate image matches the user's complex query intent at the overall semantic and macroscopic visual levels, providing a ranking basis for subsequent retrieval.

[0082] After obtaining the local feature sequences of the aforementioned multimodal data, in order to further remove the interference of background noise on retrieval accuracy and ensure that subsequent feature interactions can focus on the main objects that users are truly concerned about, the system further executes the target region feature extraction process.

[0083] First, the system uses a pre-built object detection network to process the input retrieval sketch and candidate images. This object detection network is specifically trained and has the ability to recognize common objects (such as bags, shoes, and vehicles). During processing, the object detection network analyzes the geometric shapes formed by the sketch lines and the forms of objects in the image, outputting the coordinates of the bounding box that encloses the target object. These coordinates precisely define the position and size of the target bounding box in the sketch and the target bounding box in the image, thus dividing the image into foreground target areas and background areas. For example, in the retrieval sketch, the object detection network can ignore noise at the edges of the canvas and select the core object area drawn by the user; in the candidate image, it can isolate the target object from complex natural backgrounds (such as grass and streets).

[0084] Next, the system constructs a mapping bridge between pixel coordinates and feature space indices to establish a correspondence between bounding boxes and local feature sequences. Since the extracted local features are generated based on image patches, they spatially correspond to the regular grid positions of the original image. Therefore, the system calculates which patch indices in the original image are covered by the bounding box coordinates output by the object detection. Through this positional mapping, the system can filter out a set of local feature vectors falling within the sketch target bounding box and a set of local feature vectors falling within the image target bounding box from the complete local feature sequence. This step logically achieves feature-level pruning, ensuring that the selected features only represent the visual information of the target object itself.

[0085] Finally, to obtain a compact and representative region representation, the system fuses the selected local feature set. Specific fusion methods can employ strategies such as average pooling or max pooling to aggregate multiple local feature vectors within the bounding box into a single feature vector. In this way, sketch target region features and image target region features are generated separately. These two feature vectors condense the core visual attributes of the target object (such as the structural topology of the sketch and the entity texture of the image), shielding it from interference from irrelevant background features outside the bounding box, and providing accurate data support for subsequent calculations of fine-grained similarity between text and sketch, and between sketch and image.

[0086] Furthermore, after extracting target region features from the sketch and candidate images and identifying their respective key visual ranges, this embodiment proceeds to the feature deep fusion and matching stage. To compensate for the lack of semantic attributes such as color and material in a single sketch modality, the system utilizes local fine-grained information from the retrieved text to semantically enhance the sketch features and calculates the degree of matching with the candidate images based on this.

[0087] Specifically, for the fine-grained fusion process of text and sketch, the system first processes the local feature sequence of the retrieved text. This sequence consists of multiple text sub-features (i.e., token vectors corresponding to each word or phrase), each carrying specific semantic attributes such as "red," "leather," and "long and narrow." The system calculates the similarity distribution matrix between these text sub-features and various local visual features within the target region of the sketch. This process essentially evaluates the degree of association between each semantic word and a specific line region in the sketch.

[0088] To reduce interference from irrelevant semantics or background noise, the system performs a filtering mechanism based on the calculated similarity distribution. For each text sub-feature, the system sets a relevance threshold, retaining only sketch target region features with similarity scores higher than this threshold. This means that only when a part of the sketch's structure has a strong correlation with the text description will it be selected for fusion. Subsequently, the system aggregates these filtered, highly correlated sketch features (e.g., through weighted summation) and concatenates them to the corresponding text sub-features. To make the concatenated heterogeneous features more uniform in numerical distribution, the system performs feature transformation processing on the concatenated vector (e.g., through nonlinear mapping via fully connected layers or multilayer perceptrons), ultimately generating a text-guided, fine-grained text-sketch fusion feature. This feature not only preserves the semantic instructions of the text but also embeds the structural information of the sketch.

[0089] Building upon this, the system further performs matching calculations between the fused features and candidate images to derive local similarity. This process does not employ simple vector distance calculations, but instead introduces a cross-attention mechanism capable of capturing complex dependencies.

[0090] During the computation, the system uses the generated text-sketch fine-grained fusion features as the query vector and the target region feature sequence of the candidate image as the key and value vectors. Through cross-attention operations, the model enables the composite query vector to actively scan the target region of the candidate image, focusing on pixel blocks in the image that match the text description (e.g., color matching) and are similar to the sketch structure (e.g., shape matching). After multiple layers of interactive computation, the output is aggregated into a global representation feature that represents the overall matching status. Finally, this global representation feature is input into a pre-trained prediction module (e.g., a binary classifier or scoring network), which outputs a probability value between 0 and 1, representing the confidence that the current combination of the retrieved text and the retrieved sketch matches the candidate image. This probability value is determined as the local similarity between the two and is used to combine the global similarity for the final image ranking.

[0091] To further illustrate the method of the embodiments of the present invention, Figure 2 This diagram illustrates the network structure of the image retrieval method based on text and sketch multimodal fusion according to an embodiment of the present invention. This embodiment, in conjunction with the accompanying drawings, details the complete process of the image retrieval method proposed in this invention. This process mainly includes the following steps S1 to S7: Step S1: Multimodal data preparation and organization.

[0092] First, structured data units for training and retrieval are constructed. As shown on the left and upper middle of the figure, each data unit contains three modalities of semantically aligned data: a descriptive text T (e.g., "A man in white holding an abat on the field playing"), a hand-drawn sketch S corresponding to the text, and a target image I. The text excels at describing the overall scene, attributes, and actions, while the sketch visually expresses the spatial layout, shape, and outline of an object. These three elements constitute a positive sample triple (T, S, I), serving as the basis for the model to learn cross-modal associations. The data can be organized as {"text":"text description", "sketch":path / to / sketch.png", "image":path / to / image.png}. The text and sketch serve as retrieval inputs, and the image is the target to be retrieved.

[0093] Step S2: Multimodal feature encoding and extraction.

[0094] This step corresponds to Figure 2 The three parallel encoder branches are designed to extract global and local features for each modality.

[0095] Text feature extraction: such as Figure 2 As shown in the leftmost branch, the text T is segmented to obtain a word sequence, and a special marker [CLS] is inserted at the beginning of the sequence. The sequence is input into a text encoder (e.g., a pre-trained BERT model), and the feature vector t0 corresponding to the [CLS] marker in the final layer output is extracted as the global feature of the text; the feature vector sequence {t1,t2,...,t} corresponding to the remaining word units is extracted. Mt As a local feature of the text, it can be represented as: ; in, It is the text encoder output layer and Symbol-related feature vectors are used to represent the global semantic features of the entire text. It represents the set of local feature vectors corresponding to each word in the input sequence, used to describe the local feature information of different words in the context semantic environment.

[0096] Sketch feature extraction: such as Figure 2 As shown in the middle branch, the sketch S is divided into Ms sub-blocks and formed into a sequence, with a [CLS] marker inserted at the beginning of the sequence. The sequence is input into a sketch encoder (such as a visual Transformer), and the feature vector s0 corresponding to the [CLS] marker is extracted as the global feature of the sketch; the feature vector sequence {s1,s2,...,sMs} corresponding to each sketch sub-block is extracted as the local feature of the sketch. This can be represented as: ; in, The feature vector associated with the special identifier [CLS] Token at the beginning of the sequence is used to represent the global semantic information of the entire sketch; while These correspond to the feature vectors of each sketch sub-block in the input sequence.

[0097] Image feature extraction: such as Figure 2 As shown in the right branch, the candidate image I is divided into Mv image blocks and formed into a sequence, with a [CLS] marker inserted at the beginning of the sequence. The sequence is input into an image encoder (such as a visual Transformer), and the feature vector i0 corresponding to the [CLS] marker is extracted as the global feature of the image; the feature vector sequence {i1,i2,...,i...} corresponding to each image block is extracted. Mv As a local feature of the image, it can be represented as: I = {i0; i1, i2, ..., i...} Mv}

[0098] Step S3: Global feature fusion and alignment learning.

[0099] This step corresponds to Figure 2 The "global alignment learning" section is highlighted in the text. In embodiments of the invention, the training input for the model is processed in batches, with each batch containing... A set of multimodal samples. In a set of samples (T, S, I), the text T and the sketch S both correspond to the target image I, forming a positive sample relationship, while other samples outside this set... The combination of text and sketch is considered a negative sample independent of image I, used to enhance the model's discriminative ability. In the feature representation stage, the global token output of the last layer is extracted from the Transformer structures of the text, sketch, and image encoders, respectively, serving as the global feature description for each modality. Subsequently, the text feature vector and sketch feature vector are additively fused to obtain a joint feature representation. Furthermore, the correlation between the fused feature F and the image features is calculated, and the cosine similarity function is used for measurement.

[0100] The correlation between the fused modal features and the corresponding image features within the current batch is calculated to obtain the similarity measurement results of the text-sketch joint features in the image feature space: in, This represents the temperature control parameters optimized through training. Similarly, similarity values ​​can be obtained to characterize the correspondence between image features and text-sketch fusion features. Finally, the global loss is obtained by calculating the similarity between text-sketch fusion features and image features and evaluating it using the cross-entropy function.

[0101] Step S4: Target detection and local region feature generation.

[0102] To achieve fine-grained matching, features of key regions need to be located and extracted from the sketch and image. Based on the local features extracted by the sketch encoder and image encoder, an object detection network (such as a Transformer-based detection head) is added. This network identifies Zs bounding boxes for objects in the sketch S. And the Zi target bounding boxes in image I .

[0103] For the sketch, based on the position mapping, find the boxes that fall within each bounding box. Within the sketch sub-blocks, obtain their corresponding local feature sets. The target region features of the sketch object are fused together using average pooling. .

[0104] Similarly, for images, the target region features of each image object are obtained. Thus, a structured set of regional features is obtained.

[0105] Step S5: Fine-grained cross-modal feature fusion.

[0106] First, calculate the fine-grained association matrix. To establish the correspondence between text terms and sketch targets, calculate the cosine similarity matrix between the local feature sequences of the text and the feature sets of the sketch target regions: In this matrix, each row corresponds to a text word, and each column corresponds to a sketch target region. This matrix quantifies the potential semantic-visual association strength between each text word and each sketch target.

[0107] Secondly, a two-way correlation filtering process is performed. Based on the aforementioned similarity matrix, a two-way refined filtering process is conducted.

[0108] For each text term T k Select the K with the highest similarity from the corresponding rows of the similarity matrix. t2s Each sketch target is represented by a set of related targets; simultaneously, for each sketch target, the K most relevant targets are selected from the corresponding columns of the similarity matrix. s2t Each text term forms a set of related texts. This two-way mechanism ensures the robustness and comprehensiveness of the association relationships.

[0109] Next, matching pairs are constructed and features are aggregated. A detailed explanation is given using a text-driven path as an example: For text lexicon T... k From its set of related targets, the most relevant K is further selected. sp There are several sketch targets. The features of these targets are aggregated through average pooling to generate a unified sketch context feature. Subsequently, the text lexical features t k With aggregated sketch features Combining these elements forms a fine-grained cross-modal matching pair (t k , Performing this operation on all text lexics yields K. t A set of such matching pairs .

[0110] Finally, fine-grained fusion features are generated. Each matching pair ( The two feature vectors in the diagram are concatenated, and the concatenated combined feature is input into a multilayer perceptron for nonlinear transformation and information fusion. The network's function is to deeply integrate textual semantics and sketch visual information, and output a new feature vector. All text terms undergo this processing to obtain a fine-grained feature sequence based on the text. and the corresponding fine-grained feature sequence based on the sketch .

[0111] Through this step, the model achieves precise correlation and deep fusion of text description and sketch drawing at the local component level. The resulting fine-grained fusion features lay a direct foundation for the next step of detailed matching with the image.

[0112] Step S6: Local feature matching and alignment learning.

[0113] To achieve a close correspondence between the fused text-sketch fine-grained features and image fine-grained features in the feature space, this embodiment of the invention prioritizes selecting difficult examples within the batch processing. The text-sketch fine-grained features and image fine-grained features are fused via a six-layer cross-attention module, and the [CLS] representation of the last layer of this cross-attention module is used as input to predict the matching probability of corresponding sample pairs. and The loss function is expressed as: in, It is represented as a two-dimensional one-hot encoded vector, used to identify the corresponding true category label.

[0114] Step S7: Joint optimization and training strategy integration.

[0115] The model optimizes its final training objective by integrating global and local feature alignment between text, sketches, and images. During training, to enhance the model's robustness to incomplete text and sketch inputs, masks of varying scales are randomly added to text and sketch samples.

[0116] During the training of a multimodal model, the convergence speed of each modality differs, which can lead to the dominant modality having an advantage in the early stages of training. As the dominant modality converges, the model's loss function gradually decreases, but the features of other modalities may not be fully learned. Therefore, the intermodal difference ratio is used to adaptively adjust the gradient descent rate of each modality by monitoring the contributions of the text and sketch modalities to the learning objective.

[0117] Reinforcement strategies are used during training: Random modal masking: Randomly masks text words or sketch blocks at a preset ratio to simulate incomplete input and improve model robustness.

[0118] Adaptive gradient adjustment: Monitor the difference in contribution of text and sketch modalities to the reduction of total loss, calculate the difference ratio between modalities, and dynamically adjust the gradient descent rate of both in backpropagation to balance the multimodal learning process and avoid single modality dominance.

[0119] In summary, this invention constructs a complete technical framework from data to features, from global to local, and from fusion to matching through the coherent steps of S1-S7, ultimately achieving accurate and efficient multimodal image retrieval based on text and sketches.

[0120] The image retrieval device based on multimodal feature fusion provided in the embodiments of the present invention will be described below. The image retrieval device based on multimodal feature fusion described below can be referred to in correspondence with the image retrieval method based on multimodal feature fusion described above.

[0121] This invention provides an image retrieval device based on multimodal feature fusion, see [link to relevant documentation]. Figure 3 ,include: The extraction module 310 is used to extract global and local features from the retrieval text, retrieval sketch, and candidate image, respectively. The fusion module 320 is used to fuse the global features of the search text and the search sketch, and calculate the global similarity between the fused features and the global features of the candidate image. The selection module 330 is used to determine the corresponding target region features based on the local features of the retrieval sketch and the candidate image, respectively. The matching module 340 is used to perform interactive fusion based on the local features of the searched text and the target region features of the search sketch, and to perform matching calculation with the target region features of the candidate image to obtain local similarity. The determination module 350 is used to determine the retrieval results from the candidate images based on the global similarity and the local similarity.

[0122] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an image retrieval method based on multimodal feature fusion. This method includes: extracting global and local features from the retrieval text, the retrieval sketch, and the candidate image, respectively; fusing the global features of the retrieval text and the retrieval sketch, and calculating the global similarity between the fused features and the global features of the candidate image; determining corresponding target region features based on the local features of the retrieval sketch and the candidate image, respectively; interactively fusing the local features of the retrieval text with the target region features of the retrieval sketch, and matching the fused result with the target region features of the candidate image to obtain a local similarity; and determining the retrieval result from the candidate image based on the global similarity and the local similarity.

[0123] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image retrieval method based on multimodal feature fusion provided by the above methods. The method includes: extracting global and local features of retrieval text, retrieval sketch, and candidate image respectively; fusing the global features of the retrieval text and the retrieval sketch, and calculating the global similarity between the fused features and the global features of the candidate image; determining corresponding target region features based on the local features of the retrieval sketch and the candidate image respectively; interactively fusing the local features of the retrieval text and the target region features of the retrieval sketch, and matching the fusion result with the target region features of the candidate image to obtain a local similarity; and determining the retrieval result from the candidate image based on the global similarity and the local similarity.

[0125] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an image retrieval method based on multimodal feature fusion provided by the above methods. The method includes: extracting global and local features of retrieval text, retrieval sketch, and candidate image respectively; fusing the global features of the retrieval text and the retrieval sketch, and calculating the global similarity between the fused features and the global features of the candidate image; determining corresponding target region features based on the local features of the retrieval sketch and the candidate image respectively; interactively fusing the local features of the retrieval text and the target region features of the retrieval sketch, and matching the fusion result with the target region features of the candidate image to obtain a local similarity; and determining a retrieval result from the candidate image based on the global similarity and the local similarity.

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image retrieval method based on multimodal feature fusion, characterized in that, include: Global and local features of the search text, search sketch, and candidate images are extracted respectively; The global features of the search text and the search sketch are fused, and the global similarity between the fused features and the global features of the candidate image is calculated. Based on the retrieval sketch and the local features of the candidate images, the corresponding target region features are determined respectively; The local features of the retrieved text and the target region features of the retrieved sketch are interactively fused, and the fusion result is matched with the target region features of the candidate image to obtain the local similarity. The retrieval results are determined from the candidate images based on the global similarity and the local similarity.

2. The method according to claim 1, characterized in that, The extraction of global and local features from the search text, search sketch, and candidate image includes: The retrieved text is segmented to obtain a word sequence, and a first special marker is inserted at the beginning of the sequence. The word sequence containing the first special marker is input into a text encoding model, and the output vector of the first special marker is extracted as the global feature of the text. The output vector of the word sequence is extracted as the local feature of the text. The retrieved sketch is divided into sub-block sequences, and a second special marker is inserted at the beginning of the sequence. The sub-block sequence containing the second special marker is input into the sketch coding model. The output vector of the second special marker is extracted as the global feature of the sketch, and the output vector of the sub-block sequence is extracted as the local feature of the sketch. The candidate image is divided into sub-block sequences, and a third special marker is inserted at the beginning of the sequence. The sub-block sequence containing the third special marker is input into the image coding model. The output vector of the third special marker is extracted as the global feature of the image, and the output vector of the sub-block sequence is extracted as the local feature of the image.

3. The method according to claim 1, characterized in that, The step of fusing the global features of the search text and the search sketch, and calculating the global similarity between the fused features and the global features of the candidate image, includes: The global features of the retrieved text and the global features of the retrieved sketch are vector-added to obtain the joint global features; A contrastive learning mechanism is employed to calculate the similarity value between the joint global features and the global features of the candidate image based on the cosine similarity function, which is then used as the global similarity.

4. The method according to claim 1, characterized in that, The step of determining the corresponding target region features based on the local features of the retrieval sketch and the candidate image includes: The target detection network identifies the sketch target bounding box in the retrieval sketch and the image target bounding box in the candidate image, respectively; the target detection network is used to determine the region range where the target object contained in the retrieval sketch and the candidate image is located. Establish a positional mapping relationship between the bounding box of the sketch target and the local features of the retrieved sketch, and fuse the local features mapped within the bounding box of the sketch target to obtain the sketch target region features; Establish a positional mapping relationship between the bounding box of the image target and the local features of the candidate image, and fuse the local features mapped within the bounding box of the image target to obtain the image target region features.

5. The method according to claim 4, characterized in that, The local features of the retrieved text include multiple text sub-features; The interaction and fusion of local features of the retrieved text and target region features of the retrieved sketch includes: Calculate the similarity distribution between each of the text sub-features and the target region features; Based on the similarity distribution, at least one of the target region features with a correlation higher than a threshold is selected for each text sub-feature; The selected target region features are aggregated and concatenated with the corresponding text sub-features. The concatenated features are then subjected to feature transformation processing to generate text-guided text-sketch fine-grained fusion features.

6. The method according to claim 5, characterized in that, The step of matching the fusion result with the target region features of the candidate image to obtain local similarity includes: By utilizing the cross-attention mechanism, feature interaction calculations are performed on the text-sketch fine-grained fusion features and the image target region features to obtain the global representation features after interaction. Based on the global representation features, the probability that the search text and the search sketch form a matching relationship with the candidate image is predicted, and the probability is used as the local similarity between the search text and the search sketch and the candidate image.

7. The method according to claim 2, characterized in that, The method further includes the steps of constructing a training set and training and optimizing the model, specifically including: A training batch containing multiple sets of training samples is constructed. Each set of training samples includes retrieval sample text, retrieval sample sketch, and candidate sample image with corresponding relationships. The retrieval sample text, retrieval sample sketch, and candidate sample image within the same set constitute positive samples, and the combination of retrieval sample text, retrieval sample sketch, and candidate sample image with other sets of candidate sample images within the training batch constitutes negative samples. Before feature extraction, a mask is randomly added to the search sample text and the search sample sketch according to a preset ratio; Features are extracted using text encoding model, sketch encoding model and image encoding model respectively. The global similarity and local similarity are calculated based on the positive samples and the negative samples. The overall optimization objective is constructed by combining the global loss function and the local loss function. Backpropagation is performed based on the overall optimization objective to update the parameters of the text encoding model, the sketch encoding model, and the image encoding model. When updating the parameters of the text encoding model and the sketch encoding model, the difference in their contributions to the learning objective is monitored, the intermodal difference ratio is calculated, and the parameter update is performed using the gradient descent rate adaptively adjusted according to the difference ratio.

8. An image retrieval device based on multimodal feature fusion, characterized in that, include: The extraction module is used to extract global and local features from the search text, search sketch, and candidate images, respectively. The fusion module is used to fuse the global features of the search text and the search sketch, and calculate the global similarity between the fused features and the global features of the candidate image. The selection module is used to determine the corresponding target region features based on the local features of the retrieval sketch and the candidate image, respectively. The matching module is used to interactively fuse the local features of the searched text with the target region features of the search sketch, and to match the fusion result with the target region features of the candidate image to obtain the local similarity. The determination module is used to determine the retrieval results from the candidate images based on the global similarity and the local similarity.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image retrieval method based on multimodal feature fusion as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image retrieval method based on multimodal feature fusion as described in any one of claims 1 to 7.

Images