Image-text matching method and system based on BERT and self-attention mechanism

Abstract

The present disclosure provides a method and system for image-text matching based on BERT and self-attention mechanism, which belongs to the field of image-text matching technology. The scheme includes obtaining image and text data to be matched; inputting the image and text data to be matched into a pre-trained image-text matching model to obtain the similarity between the image data and the text data; determining the matching relationship between the image and the text based on the magnitude of the similarity; wherein the specific processing process of the image-text matching model is: the image data is subjected to image region semantic feature extraction module to extract image semantic features, and the text data is subjected to text feature extraction module to extract text features; based on the obtained image semantic features and text features, the image region and the word are aligned and redundant information is removed through the cross-attention and similarity attention filtering modules; finally, the similarity result is obtained through the fully connected layer.

Description

Technical Field

[0001] The present disclosure belongs to the field of image-text matching technology, and in particular relates to an image-text matching method and system based on BERT and self-attention mechanism. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Image and text matching is a fundamental task in vision and language. This cross-modal matching task aims to accurately measure the visual semantic similarity between images and text, and is relevant to many vision-language tasks, including image-text cross-modal retrieval, visual captioning, visual grounding, and visual question answering. This task has attracted significant attention and has been widely used in various applications, such as finding similar text in image annotations and captions given an image query, and retrieving matching images using text queries for image search. Despite significant progress in recent years, it remains a challenging problem because it requires understanding the visual semantics of images and addressing the semantic gap caused by the heterogeneity of underlying data features between modalities.

[0004] like Figure 1 As shown, the first image among the three images matches the text. The inventors found that most current methods either only extract the salient objects in the image or only pay attention to the location of the objects, ignoring the detailed features and background features of the objects, and the extraction of the overall semantic information of the image is not comprehensive enough; due to ignoring the detailed features of the objects themselves and the association between heterogeneous modalities, two other images are often retrieved. For example, when the text describes two children, the images of one child or one adult and a child are located, and their clothing does not match the text description. Summary of the Invention

[0005] In order to solve the above problems, the present disclosure provides a method and system for image-text matching based on Bert and self-attention mechanism. The scheme is based on an independently designed image-text matching model. In view of the fact that previous models tend to ignore the association within the modality and the alignment between modalities, the self-attention mechanism is introduced to extract the relationship between image regions, and Bert is used to bring up the association between words. In addition, the CA-SAF module is introduced when aligning the image region with the text to align the related information between the modalities and remove redundant information, thereby effectively improving the accuracy and efficiency of image-text matching.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for image-text matching based on BERT and a self-attention mechanism is provided, comprising:

[0007] Obtain image and text data to be matched;

[0008] Inputting the image and text data to be matched into a pre-trained image-text matching model to obtain the similarity between the image data and the text data; and determining the matching relationship between the image and the text based on the magnitude of the similarity;

[0009] Among them, the specific processing process of the image-text matching model is as follows: the image data is subjected to the image region semantic feature extraction module to extract image semantic features, and the text data is subjected to the text feature extraction module to extract text features. Based on the obtained image semantic features and text features, the image region and word alignment and redundant information removal are achieved through the cross-attention and similarity attention filtering modules, and finally the similarity result is obtained through the fully connected layer.

[0010] Furthermore, the image data is subjected to image region semantic feature extraction module to extract image semantic features, specifically: feature extraction is performed on the image data based on a pre-trained deep learning model, and at the same time, a self-attention model is introduced into the deep learning model during the feature extraction process to capture the dependency between items in the image feature sequence.

[0011] Furthermore, the deep learning model adopts the Fast-RCNN model.

[0012] Furthermore, the text data is subjected to a text feature extraction module to extract text features, specifically, obtaining a word embedding representation of the text data, and inputting the word embedding representation into a pre-trained Bert model to obtain text features.

[0013] Furthermore, the cross-attention and similarity-attention filtering modules are used to align image regions and words and remove redundant information, specifically:

[0014] Based on the obtained image semantic features and text features, the similarity between all image areas and word pairs is calculated; based on the similarity, the words related to the current image area are determined, and then the text vector map of the current image area is determined; the cosine distance between the text vector map and each image area is calculated; based on the cosine distance between all image areas and the text they focus on, the similarity between the image and the text is determined; based on the similarity between the image and the text, the similarity attention filtering mechanism is used to adjust the image area weight; and based on a preset threshold, image areas with weights lower than the preset threshold are removed.

[0015] Furthermore, the image-text matching model adopts the following objective function:

[0016]

[0017] Among them, c represents the spacing, (I, T) represents the true matching image and text pair, represents a negative sample, and Υ(I,T) represents the similarity between images and text.

[0018] According to a second aspect of an embodiment of the present disclosure, a graph-text matching system based on BERT and self-attention mechanism is provided, comprising:

[0019] A data acquisition unit, which is used to acquire image and text data to be matched;

[0020] A matching unit, configured to input the image and text data to be matched into a pre-trained image-text matching model to obtain a similarity between the image data and the text data; and determine a matching relationship between the image and the text based on the similarity;

[0021] Among them, the specific processing process of the image-text matching model is as follows: the image data is subjected to the image region semantic feature extraction module to extract image semantic features, and the text data is subjected to the text feature extraction module to extract text features. Based on the obtained image semantic features and text features, the image region and word alignment and redundant information removal are achieved through the cross-attention and similarity attention filtering modules, and finally the similarity result is obtained through the fully connected layer.

[0022] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored and running on the memory, wherein when the processor executes the program, the image-text matching method based on Bert and self-attention mechanism is implemented.

[0023] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the image-text matching method based on Bert and self-attention mechanism is implemented.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] (1) The scheme disclosed in the present invention provides a method and system for image-text matching based on Bert and self-attention mechanism. The scheme proposes an independently designed image-text matching model. In view of the fact that previous models tend to ignore the association within the modality and the alignment between modalities, the scheme introduces a self-attention mechanism to extract the relationship between image regions, uses Bert to bring up the association between words, and introduces a CA-SAF module when aligning the image region with the text, so as to align the information related to the modalities and remove redundant information, thereby effectively improving the accuracy and efficiency of image-text matching.

[0026] (2) The image-text matching model proposed in the present disclosure can achieve more accurate matching when performing image-text retrieval by paying more attention to the detailed features of each object in the image.

[0027] Advantages of additional aspects of the present disclosure will be given in part in the following description and in part will become apparent from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation to the present disclosure.

[0029] Figure 1 These are the positive and negative examples of image-text matching described in the background of this disclosure, where the first image is a positive example and the other two images are negative examples;

[0030] Figure 2 Schematic diagram of the image-text matching model structure adopted by the image-text matching method based on Bert and self-attention mechanism described in the embodiments of the present disclosure. DETAILED DESCRIPTION

[0031] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0034] In the absence of conflict, the embodiments of the present disclosure and the features thereof may be combined with each other.

[0035] Example 1:

[0036] The purpose of this embodiment is to provide a graph-text matching method based on BERT and self-attention mechanism.

[0037] A method for image-text matching based on BERT and self-attention mechanism, including:

[0038] Obtain image and text data to be matched;

[0039] Inputting the image and text data to be matched into a pre-trained image-text matching model to obtain the similarity between the image data and the text data; and determining the matching relationship between the image and the text based on the magnitude of the similarity;

[0040] Among them, the specific processing process of the image-text matching model is as follows: the image data is subjected to the image region semantic feature extraction module to extract image semantic features, and the text data is subjected to the text feature extraction module to extract text features. Based on the obtained image semantic features and text features, the image region and word alignment and redundant information removal are achieved through the cross-attention and similarity attention filtering modules, and finally the similarity result is obtained through the fully connected layer.

[0041] Furthermore, the image data is subjected to image region semantic feature extraction module to extract image semantic features, specifically: feature extraction is performed on the image data based on a pre-trained deep learning model, and at the same time, a self-attention model is introduced into the deep learning model during the feature extraction process to capture the dependency between items in the image feature sequence.

[0042] Furthermore, the deep learning model adopts the Fast-RCNN model.

[0043] Furthermore, the text data is subjected to a text feature extraction module to extract text features, specifically, obtaining a word embedding representation of the text data, and inputting the word embedding representation into a pre-trained Bert model to obtain text features.

[0044] Furthermore, the cross-attention and similarity-attention filtering modules are used to align image regions and words and remove redundant information, specifically:

[0045] Based on the obtained image semantic features and text features, the similarity between all image areas and word pairs is calculated; based on the similarity, the words related to the current image area are determined, and then the text vector map of the current image area is determined; the cosine distance between the text vector map and each image area is calculated; based on the cosine distance between all image areas and the text they focus on, the similarity between the image and the text is determined; based on the similarity between the image and the text, the similarity attention filtering mechanism is used to adjust the image area weight; and based on a preset threshold, image areas with weights lower than the preset threshold are removed.

[0046] Furthermore, the image-text matching model adopts the following objective function:

[0047]

[0048] Among them, c represents the spacing, (I, T) represents the true matching image and text pair, represents a negative sample, and Υ(I,T) represents the similarity between images and text.

[0049] Specifically, for ease of understanding, the solution disclosed herein is described in detail below with reference to the accompanying drawings:

[0050] Specifically, such as Figure 2 As shown, for text, if the words "T-shirt" and "white" and "blue" and "shorts" are closely associated, they will have a more accurate representation, which will help obtain the global features of the entire text. For images, if the regions of the two children in the image area and their respective detail regions can be accurately associated, for example, the relationship between the regions of the white T-shirt and blue shorts and the regions of the two children is established, the global information of the entire image can be inferred. Based on the problems existing in the existing methods, this embodiment introduces the self-attention mechanism into the image feature extraction part of the fine-grained alignment of image and text. It can enhance the attention of the local to the whole by increasing the attention weight of each position in the sequence; at the same time, the cross-attention and similarity-attention filtering (CA-SAF: Cross-Attention and Similarity-Attention Filtering) module is introduced to realize the alignment of related image regions and text and increase the contribution of information similarity representation.

[0051] Specifically, this embodiment provides a method for image-text matching based on Bert and self-attention mechanism, which is based on an independently designed image-text matching model (hereinafter referred to as BSAM model: Bert and self-attention mechanism). The model is divided into four parts, namely image area semantic feature extraction, text feature extraction, image area and word alignment, and optimization of the objective function to train the BSAM model. The self-attention mechanism is added to the semantic feature extraction of the image area to enhance the attention to detail features. For text feature extraction, we use a pre-trained Bert model to extract word features with contextual information. For the image area and word alignment part, we introduce a cross-attention and similarity attention filtering (CA-SAF) module to align all relevant image areas and words, and then enhance the weights of detail feature matching pairs and filter out unimportant matching pairs. Finally, the BSAM model is trained to calculate the results of image-text matching. The following is a detailed description:

[0052] (1) Image region semantic feature extraction

[0053] Image representation: Given an image I, our goal is to represent it with a set of image features V = {v1,...,v k},v i ∈R D , each image feature v iEncode a region, k represents the number of regions. Given a region i detected in an image, f i After the average pooling layer, we select 2048-h, and then use a fully connected layer to transform f i Transformed into a one-dimensional embedding space, the transformation is as follows:

[0054] v i =W v f i +b v (1)

[0055] Among them, W v represents the weight matrix, b v Indicates deviation.

[0056] Enhanced image semantic features: In order to capture the input image region V = {v1, ..., v k}, especially to capture the long-range dependencies that are easily lost in image sequence models, we introduce a self-attention model to transform the image sequence V = {v1,...,v k} is mapped to a sequence P = {p1,...,p k}(where image region feature p i including original features and relationships with other regional features), i.e.

[0057]

[0058] Among them, f(v i ,v j ) is a scalar function acting on the i-th and j-th regions of the input sequence, that is, the attention weight coefficient; g(v i ) is the transformation function acting on the j-th element in the input sequence; C(V) is the normalization function.

[0059] g(v i )=W g v i

[0060]

[0061] Among them, W g is a learned weight matrix. g represents 1×1×d v Convolution operation, where d v is the dimension of v; is the similarity between vectors expressed in the form of dot product, corresponding to the normalization function

[0062] (2) Text feature extraction

[0063] Given a text T, our goal is to represent it with a series of text features Q = {q1,...,q n},q i ∈R D ,Each word feature encodes a word in the text, and n represents the number of words in the text. Given the i-th word in a text, we embed the i-th word into a 300-h vector through the embedding matrix, which is randomly initialized as follows:

[0064] B i =W t q i ,i∈[1,n] (3)

[0065] Among them, q i Represents one-hot encoding in text.

[0066] We use a pre-trained BERT model to initialize word embeddings. After initialization, they are fed into the BERT model after two special symbols: [CLS] and [SEP]. Our BERT model contains 12 transforms and a hidden size of 768. The Attention mechanism is essentially derived from human visual attention. Transforms and attention mechanisms can fully reflect the contextual information of text. Therefore, the BERT text encoder can achieve more accurate alignment when faced with complex image-text alignment.

[0067] F={[CLS]q1,q2,...,q n ,[SEP]} (4)

[0068] B i =Bert(F i ),i∈[1,n] (5)

[0069] Extract the contextual information from the word, fuse the vector and its contextual information together into the final word feature, and then map the text to the joint learning space.

[0070] (3) Aligning image regions with words

[0071] We propose a Cross-Attention and Similarity-Attention Filtering (CA-SAF) module to align relevant image regions and words and enhance the weights of important matching pairs while filtering out those with smaller weights.

[0072] The image area is compared with all relevant words in the corresponding text to determine the importance of the image area to the text. The group of areas with a high matching rate of words is given a higher weight.

[0073] First, the cosine distance matrix of all relevant image regions and word pairs is calculated, i.e.

[0074]

[0075] Among them, ij Represents the similarity between the i-th region and the j-th word. The similarity matrix is ​​normalized to

[0076]

[0077] Among them, [X] + ≡max(x,0).

[0078] Focus on each image region p i Related words, the text vector graph of the i-th region is defined as:

[0079]

[0080]

[0081] Among them, μ1 represents the weight of the image region and the attention text word.

[0082] Calculate the cosine distance between the text vector of interest and each image region to determine the image region p i Importance in the text. That is:

[0083]

[0084] Finally, we sum up all regions i and the focus text The similarity distance between them is calculated by average pooling.

[0085]

[0086] If region i is not mentioned in the text, its feature p i Vector with text attention The similarity will be particularly low, i.e. The value is very low, The value of determines the AP Therefore, when Υ AP The higher the value, the more important the image region i is. We use the similarity attention filtering mechanism to assign it a higher weight. Conversely, we assign it a lower weight. Finally, we decide whether to retain the image region based on the weight.

[0087] (4) Training objectives

[0088] Our model can be trained by hinge-based triplet loss, which encourages the similarity score of matched images and texts to be greater than that of unmatched images and texts. The objective function of the optimization model is defined as follows:

[0089]

[0090] Among them, c represents the spacing, (I, T) represents the true matching image and text pair, Representing Negative Samples In practice, we only use a small batch of negative samples instead of summing all negative samples.

[0091] Finally, the image and text data to be matched are input into a pre-trained image-text matching model to obtain the similarity between the image data and the text data; and the matching relationship between the image and the text is determined based on the magnitude of the similarity.

[0092] Furthermore, in order to prove the effectiveness of the solution described in this application, the following comparative tests were conducted to prove:

[0093] To address the problem of previous models neglecting intra-modal connections and inter-modal alignment, BSAM introduces a self-attention mechanism to extract relationships between image regions, leverages Bert to highlight inter-word connections, and incorporates a CA-SAF module when aligning image regions with text to align inter-modal information and remove redundant information. Extensive experiments on two public multimodal datasets, Flickr30K and MSCOCO, demonstrate that our model can better focus on intra-modal connections and inter-modal alignment, offering significant advantages over previous strong baseline models.

[0094] Table 1.Comparison R@k of different existing algorithms for flickr30k

[0095]

[0096] Table 2.Comparison R@k of different existing algorithms for MSCOCO(1K)

[0097]

[0098] For fair comparison, we ensure that the images in the test set are the same as those of other methods on the two datasets. Our model is compared with several state-of-the-art models on the Flickr30k and MS-COCO datasets. The results are shown in (Table 1, Table 2). The experimental results show that the BSAM model achieves the highest "mR", where R@1 is higher than the baseline model in both text retrieval and image retrieval. R@1 represents the information matching accuracy. The improvement of this indicator shows that our proposed method is very effective in learning detailed feature relationships. In addition, the results of various experiments show that our model has a significant improvement in image-text matching methods.

[0099] Example 2:

[0100] The purpose of this embodiment is to provide a graph-text matching system based on BERT and self-attention mechanism.

[0101] A picture-text matching system based on BERT and self-attention mechanism, including:

[0102] A data acquisition unit, which is used to acquire image and text data to be matched;

[0103] A matching unit, configured to input the image and text data to be matched into a pre-trained image-text matching model to obtain a similarity between the image data and the text data; and determine a matching relationship between the image and the text based on the similarity;

[0104] Among them, the specific processing process of the image-text matching model is as follows: the image data is subjected to the image region semantic feature extraction module to extract image semantic features, and the text data is subjected to the text feature extraction module to extract text features. Based on the obtained image semantic features and text features, the image region and word alignment and redundant information removal are achieved through the cross-attention and similarity attention filtering modules, and finally the similarity result is obtained through the fully connected layer.

[0105] Furthermore, the cross-attention and similarity-attention filtering modules are used to align image regions and words and remove redundant information, specifically:

[0106] Based on the obtained image semantic features and text features, the similarity between all image areas and word pairs is calculated; based on the similarity, the words related to the current image area are determined, and then the text vector map of the current image area is determined; the cosine distance between the text vector map and each image area is calculated; based on the cosine distance between all image areas and the text they focus on, the similarity between the image and the text is determined; based on the similarity between the image and the text, the similarity attention filtering mechanism is used to adjust the image area weight; and based on a preset threshold, image areas with weights lower than the preset threshold are removed.

[0107] Furthermore, the system described in this embodiment corresponds to the method described in Example 1, and its technical details have been described in detail in Example 1, so they will not be repeated here.

[0108] In further embodiments, there is also provided:

[0109] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed by the processor. When the computer instructions are executed by the processor, the method described in Example 1 is performed. For the sake of brevity, no further details are given here.

[0110] It should be understood that in this embodiment, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0111] The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0112] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in embodiment 1 is performed.

[0113] The method in Example 1 can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules within the processor. The software module can be located in a storage medium well-established in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not given here.

[0114] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] The above embodiment provides a method and system for image-text matching based on Bert and self-attention mechanism, which can be implemented and has broad application prospects.

[0116] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A graph-text matching method based on BERT and self-attention mechanism, characterized in that: include: Obtain image and text data to be matched; Inputting the image and text data to be matched into a pre-trained image-text matching model to obtain the similarity between the image data and the text data; and determining the matching relationship between the image and the text based on the magnitude of the similarity; The specific processing process of the image-text matching model is as follows: the image data is processed by the image region semantic feature extraction module to extract image semantic features, and the text data is processed by the text feature extraction module to extract text features. Based on the obtained image semantic features and text features, the image regions and words are aligned and redundant information is removed through the cross attention and similar attention filtering modules. Finally, the similarity result is obtained through the fully connected layer. The cross-attention and similarity-attention filtering modules are used to align image regions and words and remove redundant information, specifically: Based on the obtained image semantic features and text features, the similarity between all image areas and word pairs is calculated; based on the similarity, the words related to the current image area are determined, and then the text vector map of the current image area is determined; the cosine distance between the text vector map and each image area is calculated; based on the cosine distance between all image areas and the text they focus on, the similarity between the image and the text is determined; based on the similarity between the image and the text, the similarity attention filtering mechanism is used to adjust the image area weight; and based on a preset threshold, image areas with weights lower than the preset threshold are removed.

2. The image-text matching method based on BERT and self-attention mechanism according to claim 1, characterized in that: The image data is subjected to image semantic feature extraction by an image region semantic feature extraction module, specifically, feature extraction is performed on the image data based on a pre-trained deep learning model. At the same time, a self-attention model is introduced into the deep learning model during the feature extraction process to capture the dependency between items in the image feature sequence.

3. The image-text matching method based on BERT and self-attention mechanism according to claim 2, characterized in that: The deep learning model adopts the Fast-RCNN model.

4. The image-text matching method based on BERT and self-attention mechanism according to claim 1, characterized in that: The text data is subjected to text feature extraction by a text feature extraction module, specifically, obtaining word embedding representations of the text data, and inputting the word embedding representations into a pre-trained Bert model to obtain text features.

5. The image-text matching method based on BERT and self-attention mechanism according to claim 1, characterized in that: The image-text matching model adopts the following objective function: in, Indicates the spacing, Indicates the true matching image-text pair, represents negative samples, Indicates the similarity between images and text.

6. A graph-text matching system based on BERT and self-attention mechanism, characterized by: include: A data acquisition unit, which is used to acquire image and text data to be matched; A matching unit, configured to input the image and text data to be matched into a pre-trained image-text matching model to obtain a similarity between the image data and the text data; and determine a matching relationship between the image and the text based on the similarity; The specific processing process of the image-text matching model is as follows: the image data is processed by the image region semantic feature extraction module to extract image semantic features, and the text data is processed by the text feature extraction module to extract text features. Based on the obtained image semantic features and text features, the image regions and words are aligned and redundant information is removed through the cross attention and similar attention filtering modules. Finally, the similarity result is obtained through the fully connected layer. The cross-attention and similarity-attention filtering modules are used to align image regions and words and remove redundant information, specifically: Based on the obtained image semantic features and text features, the similarity between all image areas and word pairs is calculated; based on the similarity, the words related to the current image area are determined, and then the text vector map of the current image area is determined; the cosine distance between the text vector map and each image area is calculated; based on the cosine distance between all image areas and the text they focus on, the similarity between the image and the text is determined; based on the similarity between the image and the text, the similarity attention filtering mechanism is used to adjust the image area weight; and based on a preset threshold, image areas with weights lower than the preset threshold are removed.

7. An electronic device comprising a memory, a processor, and a computer program stored and running on the memory, characterized in that: When the processor executes the program, it implements the image-text matching method based on Bert and self-attention mechanism as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image-text matching method based on BERT and self-attention mechanism is implemented as described in any one of claims 1 to 5.

Images