A Cross-Modal Image and Text Retrieval Method and Device Based on Pulse Fusion

Abstract

This application discloses a cross-modal image-text retrieval method based on pulse fusion, belonging to the field of image-text retrieval technology. The method includes: inputting a target image set into a target detection network to obtain image floating-point codes, and inputting a target text set into a word segmenter to obtain word floating-point codes; inputting the image floating-point codes and word floating-point codes into a pulse encoder to obtain first image pulse codes and first text pulse codes; inputting the first image pulse codes and first text pulse codes into a pulse cross-attention fusion module to obtain second image pulse codes and second text pulse codes; performing weighted accumulation and average pooling on the second image pulse codes and second text pulse codes respectively to obtain image floating-point feature vector sets and text floating-point feature vector sets, calculating cosine similarity to obtain image-text alignment results; and obtaining text retrieval results for each image in the target image set based on the image-text alignment results, thus improving the accuracy and efficiency of image-text retrieval.

Description

Technical Field

[0001] This application belongs to the field of image and text retrieval technology, and in particular relates to a cross-modal image and text retrieval method and apparatus based on pulse fusion. Background Technology

[0002] With the development of deep learning, image and text retrieval methods have made significant progress. Some methods using pre-trained networks achieve excellent performance metrics. However, due to the complexity of real-world data and practical requirements, these methods often neglect issues such as speed and energy consumption during model inference, which are crucial factors for devices and systems. Image and text retrieval is an important task in the cross-modal domain, aiming to retrieve corresponding text from a dataset using images as queries, and vice versa. With the development of the big data era, how to efficiently, quickly, and accurately retrieve corresponding samples from large-scale image and text databases has become a challenge. This task is of great significance in fields such as search engines, recommendation algorithms, and user privacy. Therefore, maintaining retrieval accuracy while reducing computational energy consumption is a key challenge for the large-scale application of cross-modal retrieval technology.

[0003] While existing image-text retrieval methods have achieved good performance in image and text unimodal tasks, they have not yet been applied to the multimodal image-text domain. The application of spiking neural networks in the multimodal domain faces two major challenges: 1) spiking neural networks with the same structure perform worse than artificial neural networks; 2) the sparsity of spiking networks leads to significant information loss, hindering cross-modal learning. Existing pulse coding schemes struggle to simultaneously preserve the complete information of visual and textual semantics, resulting in difficulties in aligning image and text features and exacerbating the cross-modal semantic gap. Current methods suffer from low retrieval accuracy and efficiency, high retrieval energy consumption, and difficulty in aligning image and text pairs in image-text retrieval. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a cross-modal image and text retrieval method and apparatus based on pulse fusion, which improves the accuracy and efficiency of image and text retrieval.

[0005] Firstly, this application provides a cross-modal image-text retrieval method based on pulse fusion, the method comprising:

[0006] Obtain a target image set and a target text set, wherein the target image set includes n images and the target text set includes n texts;

[0007] The target image set is input into the target detection network, and the regional features of the target image set are extracted offline to obtain the image floating-point code. The target text set is input into the word segmenter to divide it into word tokens to obtain the word floating-point code.

[0008] The image floating-point code and word floating-point code are input into a pulse encoder and converted into pulse code to obtain the first image pulse code and the first text pulse code;

[0009] The first image pulse code is used as a query, and the first text pulse code is used as a key-value pair input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, thereby obtaining the second image pulse code with fused text pulse information;

[0010] The first text pulse code is used as a query, and the first image pulse code is used as a key-value pair. These are input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, resulting in a second text pulse code that integrates the image pulse information.

[0011] The second image pulse code and the second text pulse code are weighted and accumulated and average pooled respectively to obtain the image floating-point feature vector set and the text floating-point feature vector set;

[0012] The cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set is calculated based on the contrastive learning method to obtain the image-text alignment result.

[0013] Based on the image-text alignment results, the text retrieval results for each image in the target image set are obtained.

[0014] According to one embodiment of this application, the step of inputting the target image set into a target detection network, extracting regional features of the target image set offline to obtain image floating-point codes, and inputting the target text set into a word segmenter to divide it into word tokens to obtain word floating-point codes includes:

[0015] The target image set is input into the Faster R-CNN network, and target detection is performed in a bottom-up manner. The floating-point features of the target region of each image in the target image set are obtained and saved offline to obtain the image floating-point code.

[0016] Each text in the target text set is segmented according to a preset word length to obtain multiple text segments;

[0017] The multiple text segments are input into the Tokenizer, divided into individual tokens according to the syntax rules, and index values ​​are added;

[0018] Each token after partitioning is encoded to obtain the word's floating-point code.

[0019] According to one embodiment of this application, the step of inputting the image floating-point code and word floating-point code into a pulse encoder to convert them into pulse codes, thereby obtaining a first image pulse code and a first text pulse code, includes:

[0020] The image floating-point code and the word floating-point code are repeated T times by the Repeat operation to obtain T image floating-point codes and T word floating-point codes;

[0021] The T image floating-point codes are stacked and concatenated to obtain the concatenated image floating-point codes. The T word floating-point codes are stacked and concatenated to obtain the concatenated word floating-point codes.

[0022] The spliced ​​image floating-point code and the spliced ​​word floating-point code are respectively input into the BN layer for normalization, and then activated by neurons with learnable thresholds to obtain the first image pulse code and the first text pulse code.

[0023] According to one embodiment of this application, the step of using the first image pulse code as a query and the first text pulse code as a key-value pair input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion to obtain a second image pulse code with fused text pulse information includes:

[0024] The first image pulse code is passed through the Linear layer, BN layer and LIF layer in sequence to obtain the image query matrix;

[0025] The first text pulse encoding is passed sequentially through a Linear layer, a BN layer, and a LIF layer to obtain a text key matrix and a text value matrix;

[0026] The image query matrix, text key matrix, and text value matrix are input into the pulse cross-attention fusion module, and then intermodal information exchange and pulse fusion are performed sequentially through residual connection and pulse multilayer perceptron to obtain the second image pulse code with fused text pulse information.

[0027] According to one embodiment of this application, the step of using the first text pulse code as a query and the first image pulse code as a key-value pair input into the pulse cross-attention fusion module for intermodal information exchange and pulse fusion to obtain a second text pulse code of fused image pulse information includes:

[0028] The first text pulse encoding is passed sequentially through a Linear layer, a BN layer, and a LIF layer to obtain the text query matrix;

[0029] The first image pulse code is passed through a Linear layer, a BN layer and a LIF layer in sequence to obtain the image key matrix and the image value matrix;

[0030] The text query matrix, image key matrix, and image value matrix are input into the pulse cross-attention fusion module, and then intermodal information exchange and pulse fusion are performed sequentially through residual connection and pulse multilayer perceptron to obtain the second text pulse code of fused image pulse information.

[0031] According to one embodiment of this application, the step of performing weighted accumulation and average pooling on the second image pulse code and the second text pulse code respectively to obtain the image floating-point feature vector set and the text floating-point feature vector set includes:

[0032] The second image pulse code and the second text pulse code are multiplied by the learnable weight coefficients respectively to obtain the weight value at each time step;

[0033] The second image pulse code and the second text pulse code are weighted and accumulated in the time dimension based on the weight value of each time step to obtain the third image pulse code and the third text pulse code;

[0034] Global average pooling is performed on the third image pulse code and the third text pulse code to obtain the image floating-point feature vector set and the text floating-point feature vector set.

[0035] According to one embodiment of this application, the step of calculating the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set based on a contrastive learning method to obtain the image-text alignment result includes:

[0036] The image-text alignment result is obtained by calculating the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set using the following formula:

[0037]

[0038] in, Represents a floating-point feature vector of a single image. This represents a single text floating-point feature vector. For image floating-point feature vector set, For text floating-point feature vector sets, Let be the cosine similarity.

[0039] Secondly, this application provides a cross-modal image and text retrieval device based on pulse fusion, the device comprising:

[0040] An acquisition module is used to acquire a target image set and a target text set, wherein the target image set includes n images and the target text set includes n texts;

[0041] The floating-point encoding module is used to input the target image set into the target detection network, extract the regional features of the target image set offline to obtain the image floating-point encoding, and input the target text set into the word segmenter to divide it into word tokens to obtain the word floating-point encoding;

[0042] The pulse coding module is used to input the image floating-point code and word floating-point code into the pulse encoder to convert them into pulse codes, thereby obtaining the first image pulse code and the first text pulse code;

[0043] The pulse fusion module is used to input the first image pulse code as a query and the first text pulse code as a key-value pair into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, and obtain the second image pulse code with fused text pulse information;

[0044] The first text pulse code is used as a query, and the first image pulse code is used as a key-value pair. These are input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, resulting in a second text pulse code that integrates the image pulse information.

[0045] The time aggregation module is used to perform weighted accumulation and average pooling on the second image pulse code and the second text pulse code, respectively, to obtain the image floating-point feature vector set and the text floating-point feature vector set;

[0046] The image-text alignment module is used to calculate the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set based on the contrastive learning method, so as to obtain the image-text alignment result.

[0047] The retrieval module is used to obtain the text retrieval results for each image in the target image set based on the image-text alignment results.

[0048] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cross-modal image and text retrieval method based on pulse fusion as described in the first aspect above.

[0049] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the cross-modal image and text retrieval method based on pulse fusion as described in the first aspect above.

[0050] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the cross-modal image and text retrieval method based on pulse fusion as described in the first aspect.

[0051] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the cross-modal image and text retrieval method based on pulse fusion as described in the first aspect above.

[0052] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0053] The present invention provides a cross-modal image and text retrieval method based on pulse fusion, which has the following advantages over the prior art:

[0054] (1) This invention inputs the target image set and the target text set into a target detection network and a word segmenter, extracts the regional features of the images offline and divides the text into word tokens, combines a pulse encoder to perform pulse coding conversion between the images and the text, and uses a pulse cross-attention fusion module to perform information exchange and pulse fusion between modalities, effectively achieving deep alignment of image and text information and improving the accuracy of image-text retrieval. By weighted accumulation and average pooling, the floating-point feature vector sets of images and text are obtained, and the cosine similarity is calculated using a contrastive learning method to obtain the alignment results of images and text. This can provide corresponding text retrieval results for each image in the target image set more accurately, improving the efficiency and accuracy of image-text retrieval and reducing retrieval energy consumption and interference from irrelevant information.

[0055] (2) This invention achieves efficient information exchange and pulse fusion between image and text modalities by inputting the first image pulse code as a query and the first text pulse code as a key-value pair into the pulse cross-attention fusion module. By introducing the pulse cross-attention fusion module, pulse information from different modalities is fused and exchanged, enabling the pulse code of one modality to obtain pulse information from another modality while eliminating its own noise information and maintaining its sparsity characteristics. This achieves alignment between the sparse representations of image and text pairs, improving the efficiency and accuracy of image and text retrieval.

[0056] (3) This invention uses the first image pulse code as a query and the first text pulse code as a key-value pair as inputs to the pulse cross-attention fusion module, which enables efficient information exchange and pulse fusion between image and text modalities, effectively improving the feature representation capability between modalities. By sequentially processing the image and text pulse codes through the Linear layer, BN layer and LIF layer, different modal data can exchange and fuse information across modal pulse codes after the pulse-level {activation, maintenance, suppression} process, reducing information loss in the spiking neural network and achieving alignment of image and text codes in the sparse pulse coding space. Attached Figure Description

[0057] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0058] Figure 1 This is one of the flowcharts of the cross-modal image and text retrieval method based on pulse fusion provided in the embodiments of this application;

[0059] Figure 2 This is a schematic diagram of the structure of the pulse cross-attention module provided in the embodiments of this application;

[0060] Figure 3 This is a schematic diagram of the process of image-text retrieval based on a similarity matrix provided in an embodiment of this application;

[0061] Figure 4 This is the second flowchart of the cross-modal image and text retrieval method based on pulse fusion provided in the embodiments of this application;

[0062] Figure 5 This is a schematic diagram of the cross-modal image and text retrieval device based on pulse fusion provided in the embodiments of this application;

[0063] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0064] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0065] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0066] The following description, in conjunction with the accompanying drawings, details the pulse fusion-based cross-modal image and text retrieval method, pulse fusion-based cross-modal image and text retrieval device, electronic device, and readable storage medium provided in this application, through specific embodiments and application scenarios.

[0067] Among them, the cross-modal image and text retrieval method based on pulse fusion can be applied to the terminal, specifically executed by the hardware or software in the terminal.

[0068] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0069] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0070] The pulse fusion-based cross-modal image and text retrieval method provided in this application embodiment can be executed by an electronic device or a functional module or entity within an electronic device that can implement the pulse fusion-based cross-modal image and text retrieval method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The following uses an electronic device as an example to illustrate the pulse fusion-based cross-modal image and text retrieval method provided in this application embodiment.

[0071] Figure 1 This is one of the flowcharts illustrating the cross-modal image and text retrieval method based on pulse fusion provided in this application embodiment, such as... Figure 1As shown, the cross-modal image and text retrieval method based on pulse fusion includes steps 110, 120, 130, 140, 150, 160, 170, and 180.

[0072] Step 110: Obtain the target image set and the target text set, wherein the target image set includes n images and the target text set includes n texts;

[0073] It is easy to understand that the target image set and the target text set are obtained. The target image set includes n images, and the target text set includes n texts. Each image corresponds to only one text.

[0074] Step 120: Input the target image set into the target detection network, extract the regional features of the target image set offline to obtain the image floating-point code, and input the target text set into the word segmenter to divide it into word tokens to obtain the word floating-point code;

[0075] In some embodiments, the step of inputting the target image set into a target detection network, extracting regional features of the target image set offline to obtain image floating-point codes, and inputting the target text set into a word segmenter to divide it into word tokens to obtain word floating-point codes includes:

[0076] The target image set is input into the Faster R-CNN network, and target detection is performed in a bottom-up manner. The floating-point features of the target region of each image in the target image set are obtained and saved offline to obtain the image floating-point code.

[0077] Each text in the target text set is segmented according to a preset word length to obtain multiple text segments;

[0078] The multiple text segments are input into the Tokenizer, divided into individual tokens according to the syntax rules, and index values ​​are added;

[0079] Each token after partitioning is encoded to obtain the word's floating-point code.

[0080] It's easy to understand that a region-based retrieval method is used to align and match image regions and text words. First, region features are extracted from the target image set. Faster R-CNN is used as the backbone network to extract region features offline, in the form of floating-point encoding, represented as... For example, using a pre-trained Faster R-CNN network, a bottom-up approach was adopted to perform object detection tasks on the experimental datasets Flickr30K and MSCOCO, obtaining floating-point feature codes of target regions in the images. 36 regions were extracted from each image, and the dimension of each region encoding was 2048. This data was saved offline locally as direct input to subsequent networks.

[0081] The target text set uses a tokenizer to divide complete sentences into word tokens, which are then encoded into floating-point word codes, represented as follows: Word floating-point encoding involves the following steps:

[0082] (1) Truncate the text sentence to a length of 36 words and add [CLA] and [SEP] markers at the beginning and end of the sentence to indicate the start and end of the sentence. If the length is less than 36 words, add [PAD] marker to indicate a blank.

[0083] (2) Use the Tokenizer provided by the BERT network to divide the text into separate tokens and corresponding indices according to the syntax.

[0084] (3) Use torch.nn.embedding to directly encode the partitioned tokens to obtain the corresponding floating-point format encoding vector with an encoding dimension of 1024.

[0085] In this embodiment, by inputting the target image set into the Faster R-CNN network and performing target detection, the floating-point features of the target region are extracted offline to obtain the floating-point encoding of the image. The target text set is input into the word segmenter to be divided into word tokens and encoded to obtain the floating-point encoding of the words. This facilitates subsequent image-text fusion processing, improves the efficiency and accuracy of image and text data processing, reduces the consumption of computing resources, and provides higher quality feature representations for subsequent cross-modal fusion.

[0086] Step 130: Input the image floating-point code and word floating-point code into the pulse encoder to convert them into pulse codes, and obtain the first image pulse code and the first text pulse code;

[0087] In some embodiments, the step of inputting the image floating-point code and the word floating-point code into a pulse encoder to convert them into pulse codes, thereby obtaining a first image pulse code and a first text pulse code, includes:

[0088] The image floating-point code and the word floating-point code are repeated T times by the Repeat operation to obtain T image floating-point codes and T word floating-point codes;

[0089] The T image floating-point codes are stacked and concatenated to obtain the concatenated image floating-point codes. The T word floating-point codes are stacked and concatenated to obtain the concatenated word floating-point codes.

[0090] The spliced ​​image floating-point code and the spliced ​​word floating-point code are respectively input into the BN layer for normalization, and then activated by neurons with learnable thresholds to obtain the first image pulse code and the first text pulse code.

[0091] To meet the data requirements of spiking neural networks, namely the binary pulse format, it is necessary to convert the data into pulse data format before inputting it into the network. Image floating-point codes and word floating-point codes are input into the pulse encoder to add a time dimension and convert the floating-point codes into pulse codes. The output pulse code is represented as follows: and Specifically, it includes the following steps:

[0092] (1) Use the Repeat operation to encode the image in floating point. and word floating-point encoding Repeat this process T times, then stack and concatenate the results together to add a time dimension. The calculation formula is shown below:

[0093]

[0094]

[0095] in, Floating-point encoding for the stacked image, Floating-point encoding for the stacked words, Image floating-point encoding, Floating-point encoding for words.

[0096] (2) The stacked image floating-point code and word floating-point code are passed through a BN layer to obtain normalized values. After being activated by neurons with learnable thresholds, a binary pulse code is obtained. The calculation formula is as follows:

[0097]

[0098]

[0099] in, The first image pulse is encoded. The first text pulse is encoded, BN is batch normalization, and LIF is the leakage current integral trigger neuron.

[0100] In this embodiment, by repeating, stacking, and concatenating image floating-point codes and word floating-point codes, the expressive power of information can be enhanced while preserving the original data features. By applying BN layer normalization processing to the concatenated image and text data, combined with learnable threshold neuron activation, the accuracy and robustness of pulse coding are further improved.

[0101] Step 140: Use the first image pulse code as a query and the first text pulse code as a key-value pair to input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, and obtain the second image pulse code with fused text pulse information;

[0102] Figure 2 This is a schematic diagram of the structure of the pulse cross-attention module provided in an embodiment of this application, as shown below. Figure 2 As shown, image features extracted from Faster R-CNN and text features obtained directly through the Tokenizer are processed through a linear layer and a batch normalization layer, and then through LIF neurons with learnable thresholds to obtain pulse code features, which are used as Q, K, and V matrices respectively. The cross-modal pulse codes are then fused through matrix dot product. Finally, after passing through a linear layer and being activated, a pulse code that incorporates information from another modality is obtained.

[0103] It is worth noting that, unlike artificial neural networks, spiking neural networks face a significant information loss problem in the binary representation of the pulse information stream. Without pulse fusion, the information obtained solely from the sparse coding of its own modality is severely insufficient. This lack of information makes it difficult to align cross-modal samples. Pulse fusion is an indispensable further step. Therefore, the first image pulse code is used as a query and the first text pulse code is used as a key-value pair to be input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, resulting in a second image pulse code that fuses the text pulse information.

[0104] Step 150: Use the first text pulse code as a query and the first image pulse code as a key-value pair to input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, and obtain the second text pulse code of fused image pulse information;

[0105] Step 160: Perform weighted accumulation and average pooling on the second image pulse code and the second text pulse code respectively to obtain the image floating-point feature vector set and the text floating-point feature vector set;

[0106] In some embodiments, the step of performing weighted accumulation and average pooling on the second image pulse code and the second text pulse code respectively to obtain the image floating-point feature vector set and the text floating-point feature vector set includes:

[0107] The second image pulse code and the second text pulse code are multiplied by the learnable weight coefficients respectively to obtain the weight value at each time step;

[0108] The second image pulse code and the second text pulse code are weighted and accumulated in the time dimension based on the weight value of each time step to obtain the third image pulse code and the third text pulse code;

[0109] Global average pooling is performed on the third image pulse code and the third text pulse code to obtain the image floating-point feature vector set and the text floating-point feature vector set.

[0110] It's worth noting that the time dimension introduced before network input needs to be eliminated after network output to facilitate similarity calculation. Compared to directly taking a single time step and introducing uncertainty, weighted summation at each time step better reflects global information. Both image pulse coding and text pulse coding require weighted summation to eliminate the time dimension, as well as average pooling in the region and token dimensions, to obtain the image floating-point feature vector set. and text floating-point feature vector set Specifically, it includes the following steps:

[0111] (1) The second image pulse coding and the second text pulse coding with time dimension need to first obtain the weight of each time step by multiplying it with the learnable weight coefficients, and then accumulate it in the time dimension to eliminate the time dimension. The calculation formula is as follows:

[0112]

[0113]

[0114] in, For the first Second image pulse coding at each time step For the first Second text pulse encoding at each time step For the third image pulse coding, For third text pulse encoding, , The learnable weight parameters corresponding to each time step.

[0115] (2) By performing global average pooling on the third image pulse code and the third text pulse code in N dimensions, the image floating-point feature vector set and the text floating-point feature vector set are obtained. The calculation formula is as follows:

[0116]

[0117]

[0118] in, For image floating-point feature vector set, This is a set of floating-point feature vectors for text, and AveragePool represents the average pooling operation.

[0119] In this embodiment, by weighted accumulation and global average pooling of the second image pulse code and the second text pulse code, the deep features of the image and text can be effectively extracted and their representational capabilities enhanced. By introducing learnable weight coefficients for weighted accumulation at each time step, the correlation between images and the correlation between texts can be captured more accurately. Global average pooling further reduces the feature dimensionality and improves the comprehensiveness and accuracy of the feature information.

[0120] Step 170: Calculate the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set based on the contrastive learning method to obtain the image-text alignment result;

[0121] Furthermore, by using contrastive learning, the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set is calculated to achieve alignment between them. Contrastive learning is the most commonly used method for cross-modal alignment, with advantages such as robustness and universality. By calculating the similarity, the distance between positive samples of two modalities in the coding space can be effectively shortened, achieving the purpose of matching and alignment.

[0122] Step 180: Based on the image-text alignment results, obtain the text retrieval results for each image in the target image set.

[0123] Finally, based on the image-text alignment results, the text corresponding to each image in the target image set is obtained. Figure 3 This is a schematic diagram of the process for image-text retrieval based on a similarity matrix provided in an embodiment of this application, such as... Figure 3 As shown, the final output is a cosine similarity matrix, with the horizontal axis representing the image index and the vertical axis representing the text index. Values ​​on the diagonal closer to 1 indicate a higher match between the correct image and text, while values ​​closer to 0 indicate a lower match between incorrect image and text. The final text retrieval result is: Figure 1 Graph n matches text 1, and graph n matches text n. Figure 1None of the values ​​in texts 2, 3, 4, and n match. In other words, the ideal state of the similarity matrix is: all values ​​on the diagonal are 1s, and all other values ​​are 0s.

[0124] The cross-modal image-text retrieval method based on pulse fusion provided in this application involves inputting a target image set and a target text set into a target detection network and a word segmenter. Offline, regional features of the images are extracted and the text is segmented into word tokens. A pulse encoder is used for pulse coding conversion between the image and text. A pulse cross-attention fusion module facilitates intermodal information exchange and pulse fusion, effectively achieving deep alignment between image and text information and improving the accuracy of image-text retrieval. By using weighted accumulation and average pooling to obtain floating-point feature vector sets for the image and text, and employing a contrastive learning method to calculate cosine similarity, the alignment result between the image and text is obtained. This method can more accurately provide corresponding text retrieval results for each image in the target image set, improving the efficiency and accuracy of image-text retrieval while reducing retrieval energy consumption and interference from irrelevant information.

[0125] In some embodiments, the step of using the first image pulse code as a query and the first text pulse code as a key-value pair input into the pulse cross-attention fusion module for intermodal information exchange and pulse fusion to obtain a second image pulse code with fused text pulse information includes:

[0126] The first image pulse code is passed through the Linear layer, BN layer and LIF layer in sequence to obtain the image query matrix;

[0127] The first text pulse encoding is passed sequentially through a Linear layer, a BN layer, and a LIF layer to obtain a text key matrix and a text value matrix;

[0128] The image query matrix, text key matrix, and text value matrix are input into the pulse cross-attention fusion module, and then intermodal information exchange and pulse fusion are performed sequentially through residual connection and pulse multilayer perceptron to obtain the second image pulse code with fused text pulse information.

[0129] It is easy to understand that after the first image pulse code and the first text pulse code are processed through {Linear, BN, LIF} layers, the resulting Q, K, and V matrices are all in pulse form. Therefore, the matrix dot product calculation can be reduced to logical summation and addition operations, as shown in the following formula:

[0130]

[0131]

[0132]

[0133] in, For image query matrix, For text key matrix, This is a matrix of text values.

[0134] The first image pulse code and the first text pulse code undergo a {activation, maintenance, and suppression} process through matrix multiplication in the pulse cross-attention, determining the fused pulse state. Important information from the text is activated and reflected in the image code; common information from the image and text is preserved; membrane potential information in the image that is detrimental to modal alignment is treated as noise and suppressed. The image query matrix, text key matrix, and text value matrix are input into the pulse cross-attention fusion module, and intermodal information exchange and pulse fusion are performed sequentially through residual connections and a pulse multilayer perceptron (SMLP) to obtain the second image pulse code that fuses text pulse information. Using residual connections and the SMLP ensures the stability of training. The calculation formula is shown below:

[0135]

[0136]

[0137]

[0138] in, For pulse cross-attention, For query, As key, For value, For the second image pulse coding, This is the intermediate value for the pulse coding of the second image.

[0139] It should be noted that since pulse cross-attention does not involve Softmax and scale operations, the calculation order of Q, K, and V matrix multiplication is swapped, i.e. Then the time complexity can be reduced from Down to To speed up reasoning.

[0140] In this embodiment, by inputting the first image pulse code as a query and the first text pulse code as a key-value pair into the pulse cross-attention fusion module, efficient information exchange and pulse fusion between image and text modalities can be achieved. By introducing the pulse cross-attention fusion module, pulse information from different modalities is fused and exchanged, allowing a pulse code of one modality to obtain pulse information from another modality while removing its own noise information and maintaining its sparsity characteristics. This improves the alignment between the sparse representations of image-text pairs.

[0141] In some embodiments, the step of using the first text pulse code as a query and the first image pulse code as a key-value pair input into the pulse cross-attention fusion module for intermodal information exchange and pulse fusion to obtain a second text pulse code of fused image pulse information includes:

[0142] The first text pulse encoding is passed sequentially through a Linear layer, a BN layer, and a LIF layer to obtain the text query matrix;

[0143] The first image pulse code is passed through a Linear layer, a BN layer and a LIF layer in sequence to obtain the image key matrix and the image value matrix;

[0144] The text query matrix, image key matrix, and image value matrix are input into the pulse cross-attention fusion module, and then intermodal information exchange and pulse fusion are performed sequentially through residual connection and pulse multilayer perceptron to obtain the second text pulse code of fused image pulse information.

[0145] It is easy to understand that after the first image pulse code and the first text pulse code are processed through {Linear, BN, LIF} layers, the Q, K, and V matrices are all in pulse form, and the calculation formulas are as follows:

[0146]

[0147]

[0148]

[0149] in, For text query matrix, For image key matrix, This is the image value matrix.

[0150] The text query matrix, image key matrix, and image value matrix are input into the pulse cross-attention fusion module. Intermodal information exchange and pulse fusion are then performed sequentially through residual connections and a pulse multilayer perceptron to obtain the second text pulse code of the fused image pulse information. The calculation formula is shown below:

[0151]

[0152]

[0153]

[0154] in, Encoding the second text pulse. This is the intermediate value of the second text pulse encoding.

[0155] In this embodiment, by inputting the first image pulse code as a query and the first text pulse code as a key-value pair into the pulse cross-attention fusion module, efficient information exchange and pulse fusion between image and text modalities can be achieved, effectively improving the feature representation capability between modalities. By sequentially processing the image and text pulse codes through a Linear layer, a BN layer, and a LIF layer, information exchange and fusion of different modal data are achieved through pulse-level {activation, maintenance, and suppression} processes, reducing information loss in the spiking neural network and realizing alignment of image and text codes in the sparse pulse coding space.

[0156] In some embodiments, the calculation of the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set based on the contrastive learning method to obtain the image-text alignment result includes:

[0157] The image-text alignment result is obtained by calculating the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set using the following formula:

[0158]

[0159] in, Represents a floating-point feature vector of a single image. This represents a single text floating-point feature vector. For image floating-point feature vector set, For text floating-point feature vector sets, Let be the cosine similarity.

[0160] It's easy to understand that after obtaining the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set, the following steps are performed: The represented positive sample pairs that match each other and Similarity between them and The similarity between mismatched negative sample pairs is represented.

[0161] In some embodiments, this application also provides a cross-modal image and text retrieval model based on pulse fusion, including the following modules;

[0162] The image region feature extraction module uses Faster R-CNN in a bottom-up approach to extract region features from the image, covering the objects appearing in the image. It extracts 36 region features from each image and outputs a region feature code with a dimension of 36×2048, which is in the format of a floating-point value vector.

[0163] The text token segmentation and encoding module uses BERT's Tokenizer to segment text sentences into individual tokens. The text data is truncated to a length of 36 and [CLS] and [SEP] markers are added to the beginning and end. If the length is less than 36, [PAD] marker is added. The tokens are directly encoded into floating-point vectors using torch.nn.embedding, with an encoding dimension of 1024.

[0164] The pulse code conversion module adds a time dimension T to the original data dimension by repeating the process T times and stacking the data. After normalization, the data is fed into neurons with a learnable threshold for activation, resulting in a pulse format, i.e., a binary format feature code.

[0165] The pulse cross-attention fusion module interacts and fuses pulse codes from different modalities. The pulse signals undergo {excitation, hold, inhibition} to achieve alignment. The pulse codes input to the pulse cross-attention fusion module are processed through {Linear, BN, LIF} to obtain Q, K, V matrices, excluding Softmax and ReLU activation functions, which are replaced with LIF neurons. Matrix multiplication operations are all 0-1 matrix multiplications, which can be converted into logical AND and addition operations. By swapping the matrix calculation order, [the module achieves the desired result]. It has linear complexity.

[0166] The time dimension aggregation and pooling module uses T learnable weight coefficients to weight and accumulate the time dimension to eliminate the time dimension T. Then, a global average pooling operation is used to pool the token dimension, finally obtaining a feature vector with a dimension of 1×D.

[0167] The cosine similarity calculation and alignment module calculates the cosine similarity between feature vectors from different modalities, and uses the similarity between negative samples to subtract the similarity between positive samples, thus maintaining the loss decreasing in the downward direction.

[0168] The loss function is set up to train the cross-modal image-text retrieval model with pulse fusion. The formula for calculating the loss function is as follows:

[0169]

[0170] in, This is a temperature coefficient used to adjust the curve of the loss function. Here, B represents the boundary value for hyperparameter control loss, and B represents the batch size. For loss function, The first cosine similarity between mismatched negative sample pairs. The second cosine similarity is between mismatched negative sample pairs. The similarity between mutually matching positive sample pairs is denoted as .

[0171] Figure 4 This is the second flowchart illustrating the cross-modal image and text retrieval method based on pulse fusion provided in this application embodiment, as shown below. Figure 4 As shown, on the image side, Faster R-CNN is used offline to extract region feature codes from the image. On the text side, a Tokenizer is used to convert the text into tokens and directly encode them to obtain word feature codes. The floating-point feature codes of the image and text are sent to a pulse code converter, which adds a time dimension to the floating-point code and converts it into a binary pulse code. The image pulse code is used as the query and the text pulse code is used as the key-value pair. These are then input into the pulse cross-attention fusion module to obtain an image pulse code that incorporates text pulse information. The text pulse code is used as the query and the image pulse code is used as the key-value pair. These are then input into the pulse cross-attention fusion module to obtain a text pulse code that incorporates image pulse information. The pulse codes are weighted and accumulated in the time dimension and averaged in the token number dimension to obtain the final feature vector. A contrastive learning method is used to calculate the similarity between the image feature vector and the text feature vector to achieve cross-modal image-text alignment.

[0172] In this embodiment, by calculating the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set, image-text alignment can be achieved efficiently. By comparing the image floating-point feature vector set with the text floating-point feature vector set to obtain the image-text alignment result, the efficiency and accuracy of image-text retrieval are improved, while reducing retrieval energy consumption and interference from irrelevant information.

[0173] The cross-modal image and text retrieval method based on pulse fusion provided in this application can be executed by a cross-modal image and text retrieval device based on pulse fusion. This application uses the execution of the cross-modal image and text retrieval method based on pulse fusion by a cross-modal image and text retrieval device based on pulse fusion as an example to illustrate the cross-modal image and text retrieval device based on pulse fusion provided in this application.

[0174] This application also provides a cross-modal image and text retrieval device based on pulse fusion, such as... Figure 5 As shown, the cross-modal image and text retrieval device based on pulse fusion includes: an acquisition module 510, a floating-point encoding module 520, a pulse encoding module 530, a pulse fusion module 540, a time aggregation module 550, an image and text alignment module 560, and a retrieval module 570.

[0175] The acquisition module 510 is used to acquire a target image set and a target text set, wherein the target image set includes n images and the target text set includes n texts;

[0176] The floating-point encoding module 520 is used to input the target image set into the target detection network, extract the regional features of the target image set offline to obtain the image floating-point encoding, and input the target text set into the word segmenter to divide it into word tokens to obtain the word floating-point encoding;

[0177] The pulse coding module 530 is used to input the image floating-point code and word floating-point code into the pulse encoder to convert them into pulse codes, so as to obtain the first image pulse code and the first text pulse code;

[0178] The pulse fusion module 540 is used to input the first image pulse code as a query and the first text pulse code as a key-value pair into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion to obtain the second image pulse code with fused text pulse information.

[0179] The first text pulse code is used as a query, and the first image pulse code is used as a key-value pair. These are input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, resulting in a second text pulse code that integrates the image pulse information.

[0180] The time aggregation module 550 is used to perform weighted accumulation and average pooling on the second image pulse code and the second text pulse code respectively to obtain the image floating-point feature vector set and the text floating-point feature vector set;

[0181] Image-text alignment module 560 is used to calculate the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set based on the contrastive learning method, so as to obtain the image-text alignment result.

[0182] The retrieval module 570 is used to obtain the text retrieval result for each image in the target image set based on the image-text alignment result.

[0183] The cross-modal image-text retrieval method based on pulse fusion provided in this application involves inputting a target image set and a target text set into a target detection network and a word segmenter. Offline, regional features of the images are extracted and the text is segmented into word tokens. A pulse encoder is used for pulse coding conversion between the image and text. A pulse cross-attention fusion module facilitates intermodal information exchange and pulse fusion, effectively achieving deep alignment between image and text information and improving the accuracy of image-text retrieval. By using weighted accumulation and average pooling to obtain floating-point feature vector sets for the image and text, and employing a contrastive learning method to calculate cosine similarity, the alignment result between the image and text is obtained. This method can more accurately provide corresponding text retrieval results for each image in the target image set, improving the efficiency and accuracy of image-text retrieval while reducing retrieval energy consumption and interference from irrelevant information.

[0184] The cross-modal image and text retrieval device based on pulse fusion provided in this application embodiment can achieve... Figures 1 to 4 The various processes implemented in the embodiment of the cross-modal image and text retrieval method based on pulse fusion will not be described again here to avoid repetition.

[0185] In some embodiments, such as Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, it implements the various processes of the above-described cross-modal image and text retrieval method embodiment based on pulse fusion and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0186] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0187] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described cross-modal image and text retrieval method embodiment based on pulse fusion and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0188] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0189] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described cross-modal image and text retrieval method based on pulse fusion.

[0190] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0191] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described cross-modal image and text retrieval method embodiment based on pulse fusion, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0192] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a device-level chip, device chip, chip device, or on-chip device chip, etc.

[0193] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the pulse fusion-based cross-modal image and text retrieval method of the various embodiments of this application.

[0195] In the description of this application, "first feature" and "second feature" may include one or more of the features.

[0196] In the description of this application, "multiple" means two or more.

[0197] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0198] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0199] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A cross-modal image and text retrieval method based on pulse fusion, characterized in that, The method includes: Obtain a target image set and a target text set, wherein the target image set includes n images and the target text set includes n texts; The target image set is input into the target detection network, and the regional features of the target image set are extracted offline to obtain the image floating-point code. The target text set is input into the word segmenter to divide it into word tokens to obtain the word floating-point code. The image floating-point code and word floating-point code are input into a pulse encoder and converted into pulse code to obtain the first image pulse code and the first text pulse code; The first image pulse code is used as a query, and the first text pulse code is used as a key-value pair input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, thereby obtaining the second image pulse code with fused text pulse information; The first text pulse code is used as a query, and the first image pulse code is used as a key-value pair. These are input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, resulting in a second text pulse code that integrates the image pulse information. The second image pulse code and the second text pulse code are weighted and accumulated and average pooled respectively to obtain the image floating-point feature vector set and the text floating-point feature vector set; The cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set is calculated based on the contrastive learning method to obtain the image-text alignment result. Based on the image-text alignment results, the text retrieval results for each image in the target image set are obtained.

2. The cross-modal image and text retrieval method based on pulse fusion according to claim 1, characterized in that, The process involves inputting the target image set into a target detection network, extracting regional features of the target image set offline to obtain image floating-point codes, and inputting the target text set into a word segmenter to divide it into word tokens to obtain word floating-point codes, including: The target image set is input into the Faster R-CNN network, and target detection is performed in a bottom-up manner. The floating-point features of the target region of each image in the target image set are obtained and saved offline to obtain the image floating-point code. Each text in the target text set is segmented according to a preset word length to obtain multiple text segments; The multiple text segments are input into the Tokenizer, divided into individual tokens according to the syntax rules, and index values ​​are added; Each token after partitioning is encoded to obtain the word's floating-point code.

3. The cross-modal image and text retrieval method based on pulse fusion according to claim 1, characterized in that, The step of inputting the image floating-point code and word floating-point code into a pulse encoder to convert them into pulse codes, thereby obtaining the first image pulse code and the first text pulse code, includes: The image floating-point code and the word floating-point code are repeated T times by the Repeat operation to obtain T image floating-point codes and T word floating-point codes; The T image floating-point codes are stacked and concatenated to obtain the concatenated image floating-point codes. The T word floating-point codes are stacked and concatenated to obtain the concatenated word floating-point codes. The spliced ​​image floating-point code and the spliced ​​word floating-point code are respectively input into the BN layer for normalization, and then activated by neurons with learnable thresholds to obtain the first image pulse code and the first text pulse code.

4. The cross-modal image and text retrieval method based on pulse fusion according to claim 1, characterized in that, The step of using the first image pulse code as a query and the first text pulse code as a key-value pair input into the pulse cross-attention fusion module for intermodal information exchange and pulse fusion to obtain a second image pulse code with fused text pulse information includes: The first image pulse code is passed through the Linear layer, BN layer and LIF layer in sequence to obtain the image query matrix; The first text pulse encoding is passed sequentially through a Linear layer, a BN layer, and a LIF layer to obtain a text key matrix and a text value matrix; The image query matrix, text key matrix, and text value matrix are input into the pulse cross-attention fusion module, and then intermodal information exchange and pulse fusion are performed sequentially through residual connection and pulse multilayer perceptron to obtain the second image pulse code with fused text pulse information.

5. The cross-modal image and text retrieval method based on pulse fusion according to claim 1, characterized in that, The step of using the first text pulse code as a query and the first image pulse code as a key-value pair input into the pulse cross-attention fusion module for intermodal information exchange and pulse fusion to obtain the second text pulse code of fused image pulse information includes: The first text pulse encoding is passed sequentially through the Linear layer, BN layer and LIF layer to obtain the text query matrix; The first image pulse code is passed through a Linear layer, a BN layer and a LIF layer in sequence to obtain the image key matrix and the image value matrix; The text query matrix, image key matrix, and image value matrix are input into the pulse cross-attention fusion module, and then intermodal information exchange and pulse fusion are performed sequentially through residual connection and pulse multilayer perceptron to obtain the second text pulse code of fused image pulse information.

6. The cross-modal image and text retrieval method based on pulse fusion according to claim 1, characterized in that, The step of performing weighted accumulation and average pooling on the second image pulse code and the second text pulse code respectively to obtain the image floating-point feature vector set and the text floating-point feature vector set includes: The second image pulse code and the second text pulse code are multiplied by the learnable weight coefficients respectively to obtain the weight value at each time step; The second image pulse code and the second text pulse code are weighted and accumulated in the time dimension based on the weight value of each time step to obtain the third image pulse code and the third text pulse code; Global average pooling is performed on the third image pulse code and the third text pulse code to obtain the image floating-point feature vector set and the text floating-point feature vector set.

7. The cross-modal image and text retrieval method based on pulse fusion according to claim 1, characterized in that, The method based on contrastive learning calculates the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set to obtain the image-text alignment result, including: The image-text alignment result is obtained by calculating the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set using the following formula: ； in, Represents a floating-point feature vector of a single image. This represents a single text floating-point feature vector. For image floating-point feature vector set, For text floating-point feature vector sets, Let be the cosine similarity.

8. A cross-modal image and text retrieval device based on pulse fusion, implemented using the cross-modal image and text retrieval method based on pulse fusion as described in any one of claims 1 to 7, characterized in that, The device includes: An acquisition module is used to acquire a target image set and a target text set, wherein the target image set includes n images and the target text set includes n texts; The floating-point encoding module is used to input the target image set into the target detection network, extract the regional features of the target image set offline to obtain the image floating-point encoding, and input the target text set into the word segmenter to divide it into word tokens to obtain the word floating-point encoding; The pulse coding module is used to input the image floating-point code and word floating-point code into the pulse encoder to convert them into pulse codes, thereby obtaining the first image pulse code and the first text pulse code; The pulse fusion module is used to input the first image pulse code as a query and the first text pulse code as a key-value pair into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, and obtain the second image pulse code with fused text pulse information; The first text pulse code is used as a query, and the first image pulse code is used as a key-value pair. These are input into the pulse cross-attention fusion module to perform intermodal information exchange and pulse fusion, resulting in a second text pulse code that integrates the image pulse information. The time aggregation module is used to perform weighted accumulation and average pooling on the second image pulse code and the second text pulse code, respectively, to obtain the image floating-point feature vector set and the text floating-point feature vector set; The image-text alignment module is used to calculate the cosine similarity between the image floating-point feature vector set and the text floating-point feature vector set based on the contrastive learning method, so as to obtain the image-text alignment result. The retrieval module is used to obtain the text retrieval results for each image in the target image set based on the image-text alignment results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cross-modal image and text retrieval method based on pulse fusion as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-modal image and text retrieval method based on pulse fusion as described in any one of claims 1 to 7.

Images