A method and system for cross-retrieval of images and texts based on multi-scale semantic scanning

Abstract

This invention discloses a method and system for image-text cross-retrieval based on multi-scale semantic scanning, relating to the field of cross-modal retrieval technology. First, to generate multi-granularity semantic units from images and text, multi-scale semantic scanning of foreground and background regions is performed using a multi-level overlapping sliding window. Second, to align semantic units between images and text, a stacked cross-attention mechanism is used to calculate the initial image-text similarity. Third, to further adaptively optimize the image-text similarity, a dynamic adaptive weighted loss is designed. Finally, to perform image-text retrieval, the similarities between multi-granularity foreground and background semantic units are fused to obtain the final image-text similarity. This invention truly achieves the complementarity of multi-granularity foreground and background semantic units, effectively mining complete semantic information for image-text retrieval and improving retrieval accuracy.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval technology, and in particular to a method and system for cross-text and image retrieval based on multi-scale semantic scanning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Cross-modal retrieval refers to a retrieval method where the query data and search results belong to different media types. However, the feature encoding methods of different modalities are inconsistent, making it impossible to directly measure the similarity between data; this problem is called the "heterogeneous divide." As a typical example of cross-modal retrieval, image-text retrieval research has important demonstrative significance and a positive driving force for promoting the development of cross-modal retrieval. Early image-text retrieval research used relatively simple strategies to mine semantic information, while recent image-text retrieval research focuses more on the analysis and mining of local detailed semantic information, thereby improving the accuracy of cross-modal retrieval by supplementing global semantic information.

[0004] In image-text retrieval tasks, semantic units typically refer to a local region of an image or a sequence of words in text. The inventors discovered that current object detection techniques can only find salient objects in an image and cannot capture all semantic units in the foreground and background. Simple segmentation of text into sentences or words also fails to locate complete semantic units. Despite the progress made in image-text retrieval, the following specific problems still urgently need to be addressed:

[0005] (1) The identification of foreground and background semantic units in images and text is not comprehensive enough. Existing research on image and text retrieval focuses more on the identification of foreground semantic units. Therefore, existing research has not fully explored the complementary relationship between foreground and background semantic units, which to some extent makes the semantic analysis of image and text data in two modalities incomplete, thereby reducing the performance of image and text retrieval.

[0006] (2) The segmentation of semantic units at different granularities in images and text is not precise enough. After fine-grained segmentation of images and text, the semantic information contained in image blocks and word sequences at different granularities is complementary. That is to say, complete semantic information in images and text is widely distributed in multi-granularity data. However, existing image and text retrieval research cannot achieve accurate segmentation of semantic units for fine-grained segmentation methods of images and text. The main reasons are: 1) Semantic units in images and text exist in image blocks and word sequences at multiple granularities. Segmentation of only a single-granularity local region cannot identify all semantic units; 2) Existing research often uses grid segmentation methods to divide fine-grained data, such as dividing an image into uniform blocks or segmenting a sentence by word. Such simple segmentation strategies may fragment a semantic unit, resulting in incomplete identification of semantic units.

[0007] Therefore, how to accurately identify and segment fine-grained data to locate complete semantic units is an urgent problem to be solved. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for image-text mutual retrieval based on multi-scale semantic scanning. A multi-scale semantic scanning strategy based on a multi-level sliding window is designed to comprehensively and completely identify multi-granular semantic units in the foreground and background regions. Simultaneously, a dual-branch foreground-background fusion network for image-text retrieval tasks is proposed. Features extracted from the multi-granular semantic units of the foreground and background regions are input into this dual-branch foreground-background fusion network for semantic unit alignment, thereby calculating image-text similarity. This method truly achieves complementarity between multi-granular foreground and background semantic units, effectively mining complete semantic information for image-text retrieval and improving retrieval accuracy.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0010] The first aspect of this invention provides a method for mutual image-text retrieval based on multi-scale semantic scanning, comprising the following steps:

[0011] By using a dual-branch foreground-background fusion network and a multi-level sliding window strategy, multi-scale semantic scanning is performed on the image and text data to be retrieved, resulting in multi-granularity foreground and background image semantic units and text semantic units.

[0012] The stacked cross-attention mechanism is used to calculate image-text similarity between image semantic units and text semantic units;

[0013] The image-text similarity is optimized using a dynamic adaptive weighted loss function to obtain the image-text similarity matrix;

[0014] Multiple image-text similarity matrices of the foreground and background are linearly fused to obtain the final image-text similarity matrix, and the retrieval is performed using the final fused image-text similarity matrix.

[0015] A second aspect of the present invention provides a text-image mutual retrieval system based on multi-scale semantic scanning, comprising:

[0016] The semantic scanning module is configured to perform multi-scale semantic scanning on the image and text data to be retrieved using a multi-level sliding window strategy through a dual-branch foreground-background fusion network, thereby obtaining multi-granular foreground and background image semantic units and text semantic units respectively.

[0017] The similarity calculation module is configured to calculate image-text similarity between image semantic units and text semantic units using a stacked cross-attention mechanism;

[0018] The similarity optimization module is configured to optimize the image-text similarity using a dynamic adaptive weighted loss function to obtain the image-text similarity matrix;

[0019] The fusion and retrieval module is configured to linearly fuse the image-text similarity matrix to obtain the final image-text similarity matrix, and then use the final image-text similarity matrix to perform retrieval.

[0020] The above one or more technical solutions have the following beneficial effects:

[0021] This invention discloses a method and system for mutual retrieval of images and text based on multi-scale semantic scanning, which has the following advantages:

[0022] (1) To address the problem of insufficient recognition of foreground and background semantic units in images and text, this invention proposes a dual-branch foreground-background fusion network and applies it to image-text retrieval tasks. This dual-branch network consists of a multi-granularity foreground semantic unit mining sub-network and a multi-granularity background semantic unit mining sub-network. These two sub-networks comprehensively and completely identify multi-granularity semantic units in the foreground and background regions through multi-scale semantic scanning, aligning semantic units of both image and text modalities at the same granularity level. Based on this, the two sub-networks learn foreground image-text similarity and background image-text similarity, respectively. A Dynamic Adaptive Weighted Loss (DSAW) function is also proposed, which optimizes image-text similarity by dynamically assigning appropriate weights to each selected sample. This is then applied to the field of image-text retrieval. In sample pair mining, different weights are dynamically assigned to each sample pair. The adaptive weighting process adapts to sample importance and updates the mining strategy for each sample pair in supervised learning. This ensures the comprehensiveness of semantic unit recognition and achieves in-depth mining of image-text features.

[0023] (2) Addressing the issue of inaccurate segmentation of semantic units of different granularities in images and text, this invention proposes a multi-level sliding window strategy to accurately segment multi-granularity semantic units in images and text through multi-scale semantic scanning. First, multi-scale semantic scanning is performed on both images and text using a multi-level sliding window. To ensure complete semantic unit segmentation, adjacent windows overlap to maintain local continuity during dynamic sliding. Second, a stacked cross-attention mechanism is used for feature weighting to align multi-granularity image and text semantic units, ensuring accurate segmentation of semantic units. This avoids the defect of erroneous segmentation damaging the integrity of semantic units.

[0024] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0026] Figure 1 This is a framework diagram of the dual-branch foreground / background fusion network in Embodiment 1 of the present invention;

[0027] Figure 2 This is a flowchart of the scanning and alignment of semantic units in Embodiment 1 of the present invention. Detailed Implementation

[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0029] It should be noted that in the embodiments of this application, data such as images and text to be retrieved are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] Image-to-text retrieval (ITR) is the process of retrieving relevant samples from another modality given a user's representation in one modality. It typically includes two subtasks: image-to-text retrieval (i2t) and text-to-image retrieval (t2i). Due to the rapid development of deep learning models in computer vision and natural language processing, ITR has also experienced rapid growth. For example, transferring transformer-based pre-trained models from natural language processing to the vision domain has created transformer models for vision, significantly accelerating development in multimodal fields like ITR. The heterogeneity of multimodal data makes the alignment of image and text features crucial. Existing methods fall into two categories: global alignment and local alignment. In global alignment, many studies have explored mapping images and sentences to a common embedding space. Therefore, a global representation is generated for each image-text pair. However, global alignment only considers global-level information from image and text data, modeling the cross-media relevance between the entire image and the complete sentence. Cross-modal retrieval methods based on single-granularity data cannot compensate for this "granularity gap," so applying only global alignment to ITR may result in a loss of fine-grained information. Local alignment refers to learning the correspondence between local regions in an image and words in a sentence. Fine-grained data can provide rich supplementary information to coarse-grained data. Recently, many works have thoroughly explored fine-grained alignment between local image regions and keywords. However, a growing number of researchers have found that it is difficult to further improve performance using only fine-grained data. In fact, different data granularities are complementary when representing specific semantic concepts. Therefore, this invention mainly studies the alignment relationship between multi-granular foreground and background regions in images and text. There is a significant complementary relationship between foreground objects and background regions in images. Simultaneously, for text, there are not only descriptions of foreground words, but also descriptions of state, environment, and other background words. This can also provide additional clues for image-text retrieval.

[0032] Example 1:

[0033] Cross-modal retrieval refers to returning objects from a multimedia database with the highest similarity to other modalities, given input in one modality. As a typical example of cross-modal retrieval, image-text retrieval includes two retrieval tasks: 1) image-to-text retrieval (i2t) and text-to-image retrieval (t2i). This embodiment defines the retrieval tasks as follows:

[0034] Definition 1: Image and text retrieval: It contains n image samples and m text samples A multimedia database, let Q I It is an image query; image-to-text retrieval returns a set of text R. T Furthermore, for text queries Q T Text-to-image retrieval returns a set of images R I .

[0035] To achieve multi-scale semantic scanning, this embodiment defines an n-level sliding window. Specifically, the definitions of the u-th (1≤u≤n) level sliding window for images and text are given respectively.

[0036] Definition 2. The u-th level overlapping sliding window of an image: Assuming the width and height of the image are w and h respectively, and the width and height of the sliding window are... and in, Furthermore, the sliding step sizes in the width and height directions are set to stepsize_w = w / 2x and stepsize_h = h / 2x, respectively. Therefore, after the sliding is complete, the total number of image blocks is M = (2x - 1). 2 =u 2 .

[0037] Definition 3. The u-th level overlapping sliding window of a text: For a text T with K words, assume that the word feature set of T is represented as T q ={w1,…,w k ,…,w K}, st is the sliding step size, win = u is the size of the sliding window, t n level u The text features in the nth window of the u-th level sliding window are merged into: n∈[1, N], where N is the number of merged word regions.

[0038] Embodiment 1 of the present invention provides a method for mutual retrieval of images and text based on multi-scale semantic scanning, comprising the following steps:

[0039] Step 1: Using a dual-branch foreground-background fusion network and a multi-level sliding window strategy, multi-scale semantic scanning is performed on the image and text data to be retrieved, resulting in multi-granularity foreground and background image semantic units and text semantic units.

[0040] Step 2: Calculate image-text similarity for image semantic units and text semantic units using a stacked cross-attention mechanism.

[0041] Step 3: Optimize the image-text similarity using a dynamic adaptive weighted loss function to obtain the image-text similarity matrix.

[0042] Step 4: Linearly fuse the multiple image-text similarity matrices of the foreground and background to obtain the final image-text similarity matrix, and use the final fused image-text similarity matrix to perform retrieval.

[0043] The framework of the Two-Branch Foreground-Background Fusion Network (FB-Net), such as... Figure 1 The diagram shows two branches: 1) B-Net and 2) F-Net. Calculating image-text similarity for each branch involves three steps: first, semantic unit scanning and generation; second, semantic unit alignment; and third, image-text similarity optimization. Then, in the fourth step, foreground-background fusion, the image-text similarities obtained from B-Net and F-Net are fused together to perform image-text retrieval.

[0044] Object detection techniques are used to extract foreground regions using F-Net. Foreground regions of both the image and text are input into a multi-level overlapping sliding window for feature extraction. This embodiment constructs an n-level multi-granularity sub-network by designing sliding windows of different scales. To align semantic units between the image and text, a stacked cross-attention mechanism is used to compute the initial image-text similarity. Finally, to perform image-text retrieval, the similarities between the multi-granularity foreground semantic units are fused, and the final similarity matrix is ​​given.

[0045] In particular, This represents the coarse-grained foreground subnetwork. For the u-th level granularity F-Net(L... u The feature vector set of the foreground image region is as follows:

[0046]

[0047] in, It is the feature vector of the Mth image region at the u-th granularity of F-Net, where M is the number of foreground regions in the image.

[0048] Similarly, the feature vector set of the foreground text region for F-Net(Lu) is as follows:

[0049]

[0050] in is the feature vector of the u-th level text region at the F-Net granularity, and N is the number of foreground regions of the text. The following formula is used to determine the matrix. Similarity scores in the m-th row and n-th column:

[0051]

[0052] The structure of B-Net is similar to that of F-Net, and it calculates similarity scores through attention-weighted feature calculations. In this embodiment, a multi-level sliding window strategy is proposed for both image and text modalities. Sliding windows of different scales are designed, and u-level multi-granularity sub-networks are constructed. The similarity matrix of each sub-network can be written as... in This represents a coarse-grained background subnetwork.

[0053] For a B-Net (Lu) with a granularity of u, the feature vector set of the background image region is as follows:

[0054]

[0055] in, This is the feature vector of the x-th image region at the u-th granularity of B-Net, where X is the number of background regions in the image. Furthermore, the feature vector set of background text regions in B-Net (Lu) is as follows:

[0056]

[0057] in, is the feature vector of the y-th text region at the u-th level of B-Net granularity, where Y is the number of text background regions.

[0058] The following formula is used to determine the matrix. Similarity scores in row x and column y:

[0059]

[0060] Finally, the similarity matrices of each sub-network are linearly fused to obtain the final image-text similarity matrix as shown below.

[0061]

[0062] Where α1,...α u ,β1,...β u λ1 and λ2 are the equilibrium parameters. Matrix S FThis effectively reflects the complementary relationship between multi-granularity foreground and background semantic units. Therefore, using S... F Perform image-text retrieval.

[0063] More specifically, in step 1, the proposed multi-level overlapping sliding window strategy is used to perform semantic scanning on the foreground and background regions respectively. Then, multi-granular features of the image and text are extracted from the local regions segmented by the overlapping sliding windows at different levels, thereby obtaining multi-granular image semantic units and text semantic units for the foreground and background.

[0064] Because images are inherently two-dimensional while text is actually one-dimensional, the multi-scale semantic scanning mechanisms for images and text are very different. Therefore, the multi-level sliding window strategies for images and text are not the same. For example... Figure 2 As shown, the multi-level sliding window strategy includes the image multi-level overlapping sliding window strategy and the text multi-level overlapping sliding window strategy.

[0065] (1) Image multi-level overlapping sliding window strategy

[0066] The image's sliding window uses a two-dimensional data scanning mode. To ensure that semantic units are fully captured, adjacent sliding windows in the image should overlap each other to maintain the continuity of local regions. Therefore, overlapping sliding windows have a greater chance of obtaining semantic units with more complete semantic information. Thus, this embodiment sets a certain overlap between adjacent sliding windows when sliding on the image. The width and height of the sliding window are defined as follows:

[0067]

[0068] in, and Let w and h represent the width and height of the sliding window, respectively, and let w and h be the width and height of the image. To ensure that adjacent sliding windows overlap, the stride of the sliding window on the image is defined as follows:

[0069] stepsize_w = w / 2x

[0070] stepsize_h = h / 2x (9)

[0071] Where stepsize_w and stepsize_h represent the step size in the width and height directions, respectively, and the parameter x controls the degree of overlap between adjacent sliding windows. After scanning the entire image with overlapping sliding windows, the number of image patches generated by the proposed multi-level overlapping sliding window strategy is M = (2x - 1). 2 .

[0072] (2) Multi-level overlapping sliding window strategy for text

[0073] Since adjacent regions in an image often exhibit high similarity, this embodiment employs a multi-level sliding window strategy for images. Similar to this characteristic of images, adjacent words in text also exhibit high correlation; therefore, a sliding window is also used for fine-grained text segmentation. Using a single-scale sliding window for feature extraction captures fewer text features, resulting in looser relationships between characters, syntax, and semantics, and limited connections between adjacent words. This fails to fully utilize the contextual semantic features of sentences, restricting the model's ability to mine deeper semantic features. However, using a multi-level text sliding window strategy for multi-granular feature extraction provides a multi-level strategy for defining the connections between adjacent words and sentences, thereby generating multi-level hierarchical contextual semantic relationships and capturing richer semantic information in the text. Therefore, this embodiment also employs a multi-level sliding window strategy for multi-granular text segmentation.

[0074] Unlike images, the sliding window for text uses a one-dimensional data scanning pattern. For text T, the set of word features is represented as T. q ={w1,…,w k ,…,w K}, where w k T represents the feature of the k-th word, where K is the number of words in the text. This embodiment uses a single-layer bidirectional GRU (Bi-GRU) to extract word features from the text. q The word characteristics in the text are calculated as follows:

[0075]

[0076] in and These represent the hidden states from the forward GRU and backward GRU, respectively.

[0077] Then, the features of the words covered by the sliding window are combined to generate the features of the entire text. For example, the text features in the nth sliding window of level u are calculated as follows:

[0078]

[0079] Where st is the sliding step size and win is the window size. Specifically, the number of sliding windows N is calculated as follows:

[0080]

[0081] The proposed multi-level overlapping sliding window strategy is used to perform semantic scanning on the foreground and background regions separately. Then, multi-granularity features of the image and text are extracted from the local regions segmented by overlapping sliding windows at different levels. Specifically, this includes multi-granularity image feature extraction and multi-granularity text feature extraction. Multi-granularity image feature extraction includes foreground region feature extraction and background region feature extraction, while multi-granularity text feature extraction includes foreground region feature extraction and background region feature extraction.

[0082] (1) Multi-granularity image feature extraction

[0083] Image foreground region object detection techniques create multiple bounding boxes from an image, each containing clear semantic information. Therefore, this embodiment selects bounding boxes with rich semantic information, and the local regions enclosed by these bounding boxes are considered foreground regions. To find more semantic units from the foreground regions, a proposed multi-level overlapping sliding window strategy is used to divide them into many small blocks. Thus, fine-grained, detailed semantic information, such as actions and states, can be effectively captured from the semantic units.

[0084] Specifically, the foreground region of the image is detected and cropped by Faster R-CNN, a two-stage object detection framework. The output of Faster R-CNN includes the coordinates of the bounding box, confidence score, predicted class, and cropped object region. For the image to be retrieved, I is set to the maximum number of objects detected by Faster R-CNN as J. Then, taking the u-th level overlapping sliding window as an example, for the u-th level granularity F-Net(Lu), the feature vector set of the foreground image region is shown in Equation (1).

[0085] The region outside the bounding box can be called the background region, which can usually reflect semantic information such as the state, environment, and location of the entity. The background region of the image differs from the feature extraction of the foreground region, and a multi-level overlapping sliding window strategy is implemented on the entire image. For B-Net (Lu) with a granularity of u, the feature vector set of the background image region is shown in Equation (4).

[0086] (2) Multi-granularity text feature extraction

[0087] The multi-level overlapping sliding window strategy on images can effectively maintain local semantic continuity and further improve the ability of image text retrieval models to understand semantic information in images. Similarly, the multi-level overlapping sliding window strategy proposed in this embodiment can also be used for text processing. This strategy can make full use of word context relationships to improve the accuracy of image text retrieval. Multi-granularity data of text refers to word sequences with different sizes. This embodiment also uses sliding windows of different levels to extract multi-granularity features of text. For the entire text, bidirectional GRU is still used to extract word features. Then, the proposed multi-level overlapping sliding window strategy is used to merge word features within the sliding window. For the text T to be retrieved, the u-th level overlapping sliding window is taken as an example. The feature vector sets of the foreground and background text regions are shown in Equations (2) and (5).

[0088] The feature extraction process involves extracting foreground and background features from a series of images and text, converting the images and text into numerical representations for easy input into FB-Net to calculate their similarity. After feature extraction, the images and text are input into the FB-Net network for attention weighting to calculate the similarity between the images and text.

[0089] In step 2, the alignment process of candidate semantic units between images and text is the process of calculating image-text similarity. To align candidate semantic units between images and text, this embodiment uses a stacked cross-attention mechanism to calculate image-text similarity. The stacked cross-attention mechanism includes stacked cross-attention and attention weights. The data alignment task is divided into two parts: alignment oriented towards the image embedding space and alignment oriented towards the text embedding space.

[0090] Suppose the set of feature vectors of the image region is I = {f1, ..., f2}. x ,…,f X The feature vector set of the text region is T = [t1, ..., t2]. y ,…,t Y To ensure that images and text can be directly computed in the same space, this embodiment uses a fully connected network to reduce image features, that is, using a mapping matrix W to transform the initial features f x Mapping to i x , i.e. i x =Wf x +b, where b is the network parameter. The feature vector of the image region is updated to I = {i1,…,i...} x ,…,i X} and normalize each feature.

[0091] (1) Alignment oriented towards image embedding space

[0092] Fine-grained text features are mapped to the image embedding space through the following steps. First, the similarity between image region features and text region features is calculated; then, the similarity is normalized row-wise. Next, attention weights are calculated using an attention mechanism and weighted; then, the weighted text region features are summed as text context features, and the cosine similarity between each image region and text context feature is further calculated; finally, the cosine similarity between all image region and text context features is merged to obtain the final image-text similarity matrix. The specific process is as follows:

[0093] The similarity matrix sim is normalized column-wise. The similarity formula is: Where, sim xy Represents image region i x and text area t y The similarity.

[0094] Normalization formula:

[0095] Furthermore, according to the formula Calculate attention weights,

[0096] Where δ1 is a parameter of Softmax.

[0097] Next, the weighted text region features are summed to obtain the text context features:

[0098]

[0099] in, The text context features are weighted by the x-th image region.

[0100] Finally, the similarity between I and T is calculated as shown in formula (13):

[0101]

[0102] Where S i (x,y) is the similarity between I and T based on the image embedding space.

[0103] (2) Alignment oriented towards text embedding space

[0104] The weighted features of image regions are mapped to the text embedding space. The process of calculating the similarity between image I and text T is similar to that of image-based embedding spaces. First, the similarity between image region features and text region features is calculated; then, the similarity is normalized row-wise; attention weights are calculated using an attention mechanism, and the weighted features are summed to obtain the image context features. The similarity between each text region and the image context features is then calculated; finally, the cosine similarity between all text regions and the image context features is merged to obtain the final image-text similarity matrix. The specific process is as follows: Image region features i are obtained using the formula... x and text region features t y cosine similarity sim xy ,

[0105] Then normalize by row:

[0106] Using an attention mechanism for weighting, the attention weights are calculated as follows:

[0107] Where δ2 is a parameter of Softmax.

[0108] Furthermore, the weighted image region features are summed to obtain the image context features:

[0109]

[0110] Then, the cosine similarity between I and T is calculated as follows:

[0111]

[0112] Finally, the similarities obtained through formulas (13) and (14) are combined to calculate the image-text similarity between I and T, as shown below:

[0113] S(x,y)=S i (x,y)+S t (x,y) (15)

[0114] In step 3, during the optimization of image-text similarity using a dynamic adaptive weighted loss function, cross-modal similarity should follow the principle that similarity between samples of the same class should be greater than similarity between samples of different classes. That is, the feature vectors of samples within a class should become closer, while samples between classes should be more separated. Clearly, this idea is consistent with the triplet loss function.

[0115] However, the traditional triplet loss function assigns a consistent weight to each sample pair, meaning the threshold set for the triplet is constant. Taking the Wikipedia dataset as an example, the differences between samples of the same modality are relatively small in categories like "literature" and "art," and "history" and "warfare," while the differences are larger in the "sport" and "music" categories. However, the margin in the triplet loss function is always a fixed value. If the margin is set too large, the model may not be able to distinguish "literature" from "art" well, while if the margin is set too small, it may not be able to distinguish "sport" from "music" well. Furthermore, because the triplet loss function generates a large number of sample pairs, it reduces convergence speed and model performance during training. Although previous work has used triplet loss functions to mine hard samples, these methods lose a significant amount of valuable information. Therefore, the ability of using the triplet loss function to distinguish more valuable samples from the selected samples is limited.

[0116] In most cross-modal retrieval datasets, many categories have very similar semantics. If a triplet loss function is used, assigning the same weight to each category during learning makes it difficult to distinguish between semantically similar categories. Therefore, this embodiment proposes a dynamic adaptive weighted loss function. This function optimizes the mining and weighting of each sample pair, dynamically assigning a weight to each pair. The dynamic adaptive weighted loss function is beneficial for optimizing sample pairs that are semantically similar but belong to different categories, as well as sample pairs that have significantly different semantics but belong to different categories, thereby improving the model's convergence speed.

[0117] In image-text retrieval, a given image / text query is used as an anchor. Furthermore, if the target text / image belongs to the same semantic concept, it is considered a positive sample; otherwise, it is considered a negative sample. For anchor x... i The set of exponents of its selected positive and negative pairs is represented as and

[0118] This embodiment provides a redefinition of the cross-modal affinity matrix as follows:

[0119]

[0120] Where σ and θ are hyperparameters.

[0121] Furthermore, each sample pair should be assigned a different weight so that there is a negative sample pair. Its weight can be calculated as follows:

[0122]

[0123] Positive sample pairs weight It can be calculated as:

[0124]

[0125] Where S ij and S ik They represent sample pairs {x} i ,x j} and {x i ,x k The similarity between} is γ, which is a hyperparameter.

[0126] The loss function for negative sample pairs is as follows:

[0127]

[0128] Similarly, the loss function for positive sample pairs is as follows:

[0129]

[0130] Finally, the DSAW loss function was obtained:

[0131]

[0132] Where B is the size of the batch data.

[0133] This embodiment posits that positive samples with low similarity to the anchor point and negative samples with high similarity to the anchor point should be optimized over other samples. Therefore, this embodiment optimizes samples between the most difficult negative samples and the most difficult positive samples in each batch of data, rather than optimizing all samples.

[0134] Information samples can be mined using DSAW loss for discriminative optimization. Therefore, all initial image-text similarity scores are optimized using the proposed DSAW loss.

[0135] In step 4, the image-text similarity matrices are linearly fused to obtain the final image-text similarity matrix, which is then used to perform the retrieval. Specifically, this image-text similarity matrix is ​​a weighted fusion of two similarity matrices, corresponding to the optimized similarities obtained in the two retrieval directions: "image retrieval of text" and "text retrieval of image." Each value in the image-text similarity matrix represents the similarity of a pair of image-text samples, and the retrieval is performed based on the similarity of these image-text sample pairs.

[0136] Example 2:

[0137] Embodiment 2 of the present invention provides a text-image mutual retrieval system based on multi-scale semantic scanning, comprising:

[0138] The feature extraction module is configured to acquire the image and text data to be retrieved, perform feature extraction on the image and text data to be retrieved, and obtain the extracted image feature vector and text feature vector.

[0139] The semantic scanning module is configured to perform multi-scale semantic scanning on the image and text data to be retrieved using a multi-level sliding window strategy through a dual-branch foreground-background fusion network, thereby obtaining image semantic units and text semantic units respectively.

[0140] The similarity calculation module is configured to calculate image-text similarity between image semantic units and text semantic units using a stacked cross-attention mechanism;

[0141] The similarity optimization module is configured to optimize the image-text similarity using a dynamic adaptive weighted loss function to obtain the image-text similarity matrix;

[0142] The fusion and retrieval module is configured to linearly fuse the image-text similarity matrix to obtain the final image-text similarity matrix, and then use the image-text similarity matrix to perform retrieval.

[0143] The steps and methods involved in the above embodiment two correspond to those in embodiment one. For specific implementation details, please refer to the relevant description section of embodiment one.

[0144] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0145] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for mutual retrieval of text and images based on multi-scale semantic scanning, characterized in that, Includes the following steps: The dual-branch foreground-background fusion network utilizes a multi-level sliding window strategy to perform multi-scale semantic scanning on the image and text data to be retrieved, thereby obtaining multi-granular foreground and background image semantic units and text semantic units. The dual-branch foreground-background fusion network includes F-Net and B-Net structures, with the foreground region extracted by the F-Net structure and the background region extracted by the B-Net structure. Multi-level sliding window strategies include image multi-level overlapping sliding window strategies and text multi-level overlapping sliding window strategies. In the image multi-level overlapping sliding window strategy, the sliding window of the image uses a two-dimensional data scanning mode, and adjacent sliding windows in the image overlap each other. In the text multi-level overlapping sliding window strategy, the sliding window of the text uses a one-dimensional data scanning mode. The stacked cross-attention mechanism is used to calculate image-text similarity between image semantic units and text semantic units; In the process of optimizing image-text similarity using a dynamic adaptive weighted loss function, the sample between the most difficult negative sample and the most difficult positive sample in each batch of data is optimized, rather than optimizing all samples, to obtain the image-text similarity matrix; Dynamic adaptive weighted loss function: Where B is the size of the batch data. It is a hyperparameter, γ is a hyperparameter. It is the loss function for negative sample pairs. It is the loss function for positive sample pairs; Multiple image-text similarity matrices of the foreground and background are linearly fused to obtain the final image-text similarity matrix, and the retrieval is performed using the final image-text similarity matrix.

2. The image-text mutual retrieval method based on multi-scale semantic scanning as described in claim 1, characterized in that, The specific steps for obtaining image semantic units and text semantic units are as follows: A dual-branch foreground-background fusion network is used to perform multi-scale semantic scanning of the image and text data to be retrieved using a multi-level sliding window strategy. Using a multi-level overlapping sliding window strategy, semantic scanning is performed on the foreground and background regions respectively. Then, multi-granular features of the image and text are extracted from the local regions segmented by overlapping sliding windows at different levels, thereby obtaining multi-granular image semantic units and text semantic units of foreground and background.

3. The image-text mutual retrieval method based on multi-scale semantic scanning as described in claim 2, characterized in that, Multi-granularity image feature extraction and multi-granularity text feature extraction; wherein, multi-granularity image feature extraction includes foreground region feature extraction and background region feature extraction of an image, and multi-granularity text feature extraction includes foreground region feature extraction and background region feature extraction of a text.

4. The image-text mutual retrieval method based on multi-scale semantic scanning as described in claim 1, characterized in that, The specific steps for calculating image-text similarity using a stacked cross-attention mechanism for image and text semantic units include: using stacked cross-attention and attention weights to calculate image-text similarity and aligning candidate semantic units of the image and text; the alignment of candidate semantic units of the image and text is divided into two parts: alignment oriented towards the image embedding space and alignment oriented towards the text embedding space. The similarity calculated by the alignment oriented towards the image embedding space and the alignment oriented towards the text embedding space is integrated to obtain the image-text similarity.

5. The image-text mutual retrieval method based on multi-scale semantic scanning as described in claim 4, characterized in that, The specific steps for alignment in the image embedding space include: First, calculating the similarity between image region features and text region features; then, normalizing the similarity by row; next, calculating attention weights using an attention mechanism and weighting them; then, summing the weighted text region features as text context features, and further calculating the cosine similarity between each image region feature and text context feature; finally, merging the cosine similarities of all image region features and text context features to obtain the final image-text similarity matrix.

6. The image-text mutual retrieval method based on multi-scale semantic scanning as described in claim 5, characterized in that, The specific steps for alignment in the text embedding space include: First, calculating the similarity between image region features and text region features; then, normalizing the similarity by row; next, calculating attention weights using an attention mechanism, weighting the weighted image region features, summing the weighted image region features to obtain image context features, and further calculating the similarity between each text region feature and the image context features; finally, merging the cosine similarity between all text region features and image context features to obtain the final image-text similarity matrix.

7. A text-image mutual retrieval system based on multi-scale semantic scanning, characterized in that, include: The semantic scanning module is configured to perform multi-scale semantic scanning on the image and text data to be retrieved using a multi-level sliding window strategy through a dual-branch foreground-background fusion network, thereby obtaining multi-granular foreground and background image semantic units and text semantic units respectively; the dual-branch foreground-background fusion network includes F-Net structure and B-Net structure, wherein the foreground region is extracted through the F-Net structure and the background region is extracted through the B-Net structure. Multi-level sliding window strategies include image multi-level overlapping sliding window strategies and text multi-level overlapping sliding window strategies. In the image multi-level overlapping sliding window strategy, the sliding window of the image uses a two-dimensional data scanning mode, and adjacent sliding windows in the image overlap each other. In the text multi-level overlapping sliding window strategy, the sliding window of the text uses a one-dimensional data scanning mode. The similarity calculation module is configured to calculate image-text similarity between image semantic units and text semantic units using a stacked cross-attention mechanism; The similarity optimization module is configured to optimize the samples between the most difficult negative samples and the most difficult positive samples in each batch of data during the process of optimizing image-text similarity using a dynamic adaptive weighted loss function, rather than optimizing all samples, to obtain the image-text similarity matrix; Dynamic adaptive weighted loss function: Where B is the size of the batch data. It is a hyperparameter, γ is a hyperparameter. It is the loss function for negative sample pairs. It is the loss function for positive sample pairs; The fusion and retrieval module is configured to linearly fuse the image-text similarity matrix to obtain the final image-text similarity matrix, and then use the image-text similarity matrix to perform retrieval.

Images