Feature construction method, cross-modal retrieval method, device and computer storage medium

Abstract

The present application proposes a feature construction method, a cross-modal retrieval method, a cross-modal retrieval device and a computer storage medium. The feature construction method includes: segmenting the original modal data to obtain a number of segmented data segments; inputting the original modal data into a modal encoder to extract local modal features and global modal features of the original modal data; using the local modal features to obtain the target granularity features of each segmented data segment; splicing the global modal features and all the target granularity features to obtain the target modal features; establishing an alignment relationship between the target modal features and the preset query features to obtain multi-granularity retrieval features of the original modal data. Through the above-mentioned feature construction method, the modal data is characterized by local features and global features, and multi-granularity retrieval features are constructed to achieve refined feature alignment between different modal data.

Description

Technical Field

[0001] The present application relates to the field of cross-modal retrieval technology, and in particular to a feature construction method, a cross-modal retrieval method, a cross-modal retrieval device, and a computer storage medium. Background Art

[0002] Cross-modal image-text retrieval is a multi-disciplinary research direction that aims to explore and achieve effective matching and retrieval between different media types of data (such as images and text). The rise of this field is closely related to the explosive growth of multimedia data, especially in the Internet and mobile Internet era, where users have an increasing demand for fast and accurate acquisition and understanding of cross-type information.

[0003] With the popularity of applications such as social media, e-commerce platforms, and digital libraries, information retrieval is no longer limited to a single modality. Users may want to find relevant images through a descriptive text, or upload an image to query matching text information. Traditional retrieval methods are often designed for a single modality and are difficult to directly apply to cross-modal scenarios. Therefore, cross-modal image-text retrieval came into being. It uses deep learning, computer vision, natural language processing and other technologies to build a model that can understand and bridge the semantic gap between images and texts, and achieve efficient retrieval and understanding between images and texts.

[0004] However, current multimodal information retrieval methods can complete multi-modal retrieval such as images, text, and audio, but use global granular features of images for retrieval, which is not conducive to target fine-grained retrieval, resulting in poor cross-modal retrieval results. Summary of the invention

[0005] In order to solve the above technical problems, the present application proposes a feature construction method, a cross-modal retrieval method, a cross-modal retrieval device and a computer storage medium.

[0006] In order to solve the above technical problems, the present application proposes a feature construction method, which includes:

[0007] Segment the original modal data to obtain a number of segmented data segments;

[0008] Inputting the original modal data into a modal encoder to extract local modal features and global modal features of the original modal data;

[0009] Using the local modal features, obtaining target granularity features of each segmented data segment;

[0010] Concatenating the global modal feature and all the target granular features to obtain a target modal feature;

[0011] An alignment relationship between the target modal feature and the preset query feature is established to obtain a multi-granularity retrieval feature of the original modal data.

[0012] Wherein, the using the local modal features to obtain the target granularity features of each segmented data segment includes:

[0013] Acquire a feature set of each segmented data segment on the local modal feature;

[0014] Feature pooling is performed on the feature set to obtain target granularity features of each segmented data segment.

[0015] The feature pooling method includes region of interest pooling and / or average pooling.

[0016] Wherein, establishing the alignment relationship between the target modality feature and the preset query feature includes:

[0017] The target modal feature and the preset query feature are input into a cross attention layer, so as to perform a cross attention operation on each preset query feature and the target modal feature respectively, and establish an alignment relationship between the target modal feature and the relevant query feature.

[0018] Wherein, before establishing the alignment relationship between the target modal feature and the preset query feature, the feature construction method further includes:

[0019] Each preset query feature is input into the self-attention layer to adjust the attention weight of each sub-feature in each preset query feature.

[0020] In order to solve the above technical problems, the present application also proposes a cross-modal retrieval method, which includes:

[0021] Acquire a feature vector base library of the first modal data, wherein each multi-granularity retrieval feature in the feature vector base library is constructed by the feature construction method described above;

[0022] Obtaining query data features of the second modal data;

[0023] Calculating the matching similarity between the query data feature and each of the multi-granularity retrieval features;

[0024] The multi-granularity search features of the feature vector base library are sorted according to the matching similarity, and the search results are output.

[0025] The step of calculating the matching similarity between the query data feature and each of the multi-granularity search features includes:

[0026] Obtaining each sub-data feature of the query data feature;

[0027] Obtaining target modality features and related query features in each of the multi-granularity retrieval features;

[0028] Calculating a first matching similarity using the target modality feature and the query data feature;

[0029] Calculating a second matching similarity using the target modal feature and each sub-data feature of the query data feature;

[0030] Calculating a third matching similarity using the relevant query feature and each sub-data feature of the query data feature;

[0031] Calculating a fourth matching similarity using the relevant query feature and the query data feature;

[0032] Based on the first matching similarity, the second matching similarity, the third matching similarity and the fourth matching similarity, a matching similarity between the query data feature and each of the multi-granularity retrieval features is calculated.

[0033] The first modality data is any one of image data, text data, audio data, and video data, and the second modality data is any one of image data, text data, audio data, and video data;

[0034] The first modal data and the second modal data are data of different modalities.

[0035] In order to solve the above-mentioned technical problems, the present application also proposes a cross-modal retrieval device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the feature construction method and / or cross-modal retrieval method as described above.

[0036] In order to solve the above technical problems, the present application also proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above-mentioned feature construction method and / or cross-modal retrieval method.

[0037] Compared with the prior art, the beneficial effects of the present application are as follows: the cross-modal retrieval device segments the original modal data to obtain a number of segmented data segments; the original modal data is input into the modal encoder to extract the local modal features and global modal features of the original modal data; the target granularity features of each segmented data segment are obtained by using the local modal features; the global modal features and all the target granularity features are spliced ​​to obtain the target modal features; the alignment relationship between the target modal features and the preset query features is established to obtain the multi-granularity retrieval features of the original modal data. Through the above-mentioned feature construction method, the modal data is characterized by local features and global features, and multi-granularity retrieval features are constructed to achieve refined feature alignment between different modal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0039] Figure 1 It is a flow chart of an embodiment of a feature construction method provided by the present application;

[0040] Figure 2 It is a schematic diagram of the overall process of extracting multi-granularity retrieval features provided by this application;

[0041] Figure 3 It is a flowchart of an embodiment of a cross-modal retrieval method provided by the present application;

[0042] Figure 4 yes Figure 3 A schematic diagram of a specific process of step S23 of an embodiment of the cross-modal retrieval method shown;

[0043] Figure 5 It is a schematic diagram of calculating the similarity between images and texts of different granularities provided by this application;

[0044] Figure 6 It is a structural schematic diagram of an embodiment of a cross-modal retrieval device provided by the present application;

[0045] Figure 7 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] The technical terms involved and used in this application include but are not limited to:

[0049] Modalities: refers to different forms of data presentation, such as visual modality (image), textual modality (text), etc.

[0050] Cross-modal Retrieval: Search between datasets of different modalities, such as using text to query images or vice versa.

[0051] Deep Learning: An artificial intelligence technology that learns complex data representations through multi-layer neural networks and is widely used in cross-modal matching.

[0052] Feature Learning: Learning meaningful feature representations from raw data is the key to mapping images and text into the same semantic space for cross-modal retrieval.

[0053] Image-Text Matching: The process of evaluating the semantic similarity between images and text is the core task of cross-modal retrieval.

[0054] Adversarial Learning: A training strategy that reduces inter-modal differences by introducing an adversarial network, allowing the model to learn more generalized cross-modal features.

[0055] Scene Graph: A structured graph used to represent objects and their relationships in an image, which helps to establish more precise semantic correspondence between images and text.

[0056] Feature Fusion: Combining features from different modalities to form a unified representation to facilitate the understanding and retrieval of cross-modal information.

[0057] The existing technical solutions for cross-modal image-text retrieval mainly focus on the following core aspects, aiming to solve the problem of how to establish effective mapping and matching between different modal data (images and text):

[0058] Deep learning model construction: Use deep neural networks (such as convolutional neural networks (CNN) for image processing, recurrent neural networks (RNN) or Transformer for text processing) to extract high-level semantic features of images and texts respectively. These models usually share some parameters or use joint training strategies to ensure that the representation of images and texts in high-dimensional space is as close as possible to facilitate comparison and matching.

[0059] Feature embedding and alignment: By designing specific loss functions (such as contrast loss, triple loss, etc.), features extracted from different modalities can be well aligned in a common embedding space. This includes methods that directly map to the same feature space, as well as using cross-modal hashing techniques to convert high-dimensional features into low-dimensional binary codes to reduce storage and retrieval costs.

[0060] Attention mechanism: The attention mechanism is introduced to strengthen the model's attention to key information. Whether it is an area in the image or a word in the text, it can be weighted by attention weights to improve the matching accuracy. The image-text collaborative attention model is an example, which considers the importance of image areas and text words at the same time to improve the efficiency of cross-modal interaction.

[0061] Integration of generative models: Some solutions integrate generative models (such as variational auto-encoders (VAE) and generative adversarial networks (GANs)) to generate or enhance intermediate representations between modalities, further promoting mutual understanding and conversion between modalities. This technology can help the model better capture modality-specific details and improve retrieval accuracy.

[0062] Multimodal pre-trained models: Multimodal pre-trained models such as BERT (Bidirectional Encoder Representations from Transformers) and ViT (Visual Transformer) have become a hot topic in recent years. These models are pre-trained on large-scale multimodal datasets to learn rich cross-modal semantic representations, which are then fine-tuned for specific image and text retrieval tasks, significantly improving performance.

[0063] Graph Neural Networks (GNN) application: Graph neural networks are used to process local and global similarities of cross-modal data. By building a graph structure of image and text data and using GNN to update node features, similarity reasoning can be completed to obtain a more comprehensive image-text association relationship.

[0064] Utilization of local context: Considering the importance of local features of images and texts for matching, some technical solutions pay special attention to the utilization of local context information, such as improving the granularity and accuracy of retrieval by segmenting images to identify key areas or analyzing phrases in texts.

[0065] These technical solutions are constantly evolving, and by continuously optimizing model architecture, loss function, and training strategy, we strive to improve the accuracy and robustness of cross-modal image and text retrieval while maintaining high efficiency. With the continuous advancement of algorithms and the improvement of computing power, the practicality and application scenarios of cross-modal retrieval are also continuously expanding.

[0066] In view of the shortcomings of existing technical solutions, this application proposes a multi-granularity image and text retrieval method. In the retrieval field, it is necessary to integrate fine-grained and global granularity features. For example, when searching for "a person fishing with a fishing rod by the lake", a good retrieval feature should integrate fine-grained information such as the person's posture and position, and global granularity information such as the fishing rod and the environment (lakeside). In addition, this application proposes a multi-granularity similarity matching score calculation method to significantly improve the retrieval effect.

[0067] It should be noted that the feature construction method and cross-modal retrieval method of the present application can be applicable to feature construction and data retrieval of different modal types. Among them, the first modal data introduced in the following embodiment is any one of image data, text data, audio data, and video data, and the second modal data is any one of image data, text data, audio data, and video data. It only needs to be ensured that the first modal data and the second modal data are data of different modalities. For the convenience of description, when introducing the following embodiment, image modal data and text modal data are specifically used as two different modal data for explanation, that is, the first modal data is image data, and the second modal data is text data.

[0068] Specifically, this application constructs a feature vector base library by extracting multi-granularity retrieval features , where N is the number of images in the base library. The user enters the query text and passes through the text encoder Extracted text features , calculate the similarity between each image and text by calculating the cosine similarity, and sort and output the retrieval results.

[0069] Please refer to Figure 1 and Figure 2 , Figure 1 is a flow chart of an embodiment of a feature construction method provided by the present application, Figure 2 It is a schematic diagram of the overall process of extracting multi-granularity retrieval features provided by this application.

[0070] The feature construction method of the present application is applied to a cross-modal retrieval device, wherein the cross-modal retrieval device of the present application can be a server, a terminal device, or a system in which a server and a terminal device cooperate with each other. Accordingly, the various parts included in the cross-modal retrieval device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, can all be set in the terminal device, or can be set in the server and the terminal device respectively.

[0071] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here.

[0072] like Figure 1 As shown, the specific steps are as follows:

[0073] Step S11: segment the original modal data to obtain a number of segmented data segments.

[0074] In an embodiment of the present application, a cross-modal retrieval device segments original modal data to obtain a number of segmented data segments for extracting fine-grained data features of the original modal data.

[0075] like Figure 2 As shown, Figure 2 A technical solution for multi-granularity retrieval features is provided using image data as an example. Specifically, the cross-modal retrieval device detects everything, segments the input image through the SAM (Segment Anything) model, and counts the bounding boxes of the segmentation mask. Among them, SAM is a general model for image segmentation tasks. Unlike previous image segmentation models that can only process a certain type of image, SAM can process all types of images.

[0076] This application uses large detection models such as SAM to perform target detection and segmentation in images, which can cover untrained or undetected target types and extract more target granularity retrieval features in images.

[0077] Step S12: inputting the original modal data into the modal encoder to extract local modal features and global modal features of the original modal data.

[0078] In an embodiment of the present application, the cross-modal retrieval device encodes the local modal features and global modal features of the original modal data through the corresponding modal encoder. For example, text modal data is encoded by a text encoder, image modal data is encoded by an image encoder, and so on.

[0079] like Figure 2 As shown, the cross-modal retrieval device inputs the image into the CLIP (Contrastive Language-Image Pre-Training) image encoder to extract the global features of the image on the one hand and the feature map F of the image on the other hand.

[0080] In a specific implementation, the cross-modal retrieval device uses the patch token output by the second-to-last layer to reshape the i-th image into a feature map F in the shape of (1, C, W, H) through a CLIP image encoder with an input size of 336x336. Alternatively, the cross-modal retrieval device may directly use the encoded features finally output by the CLIP image encoder as the feature map F.

[0081] Step S13: using local modal features to obtain target granularity features for each segmented data segment.

[0082] In an embodiment of the present application, in step S11, the cross-modal retrieval device determines the position of each segmented data segment in the original modal data, and further uses the feature map F obtained in step S12 to analyze the target granularity features of each segmented data segment.

[0083] Specifically, the cross-modal retrieval device locates the position of each segmented data segment on the local module feature, thereby obtaining a feature set of each segmented data segment. Then, the cross-modal retrieval device performs a pooling operation on the feature set to obtain the target granularity feature of each segmented data segment.

[0084] like Figure 2 As shown, the cross-modal retrieval device uses the bounding box of step S11 , on the feature map F, ROI pooling (region of interest pooling) and avg pooling (average pooling) are used to obtain the target granularity features of each Box Specifically, the cross-modal retrieval device presets a certain size of region of interest, such as 7×7, and then extracts the pooling results of each region of interest in the bounding box on the feature map F through ROIpooling. Finally, the pooling results of each region of interest are further pooled through avg pooling to obtain a one-dimensional feature, that is, the target granularity feature .

[0085] In other embodiments, the cross-modal retrieval device may also select one of ROI pooling and avg pooling to perform feature pooling to obtain the target granularity feature. .

[0086] Step S14: concatenate the global modal features and all target granular features to obtain the target modal features.

[0087] In an embodiment of the present application, a cross-modal retrieval device concatenates global modal features and all target granular features to obtain target modal features, thereby achieving a fusion of global features and local features.

[0088] like Figure 2 As shown, the cross-modal retrieval device converts the target granularity feature obtained in step S13 into Combined with global features to form image features .

[0089] Step S15: establishing an alignment relationship between the target modal feature and the preset query feature to obtain multi-granularity retrieval features of the original modal data.

[0090] In an embodiment of the present application, a cross-modal retrieval device uses a cross-modal alignment model to establish an alignment relationship between a target modal feature and a preset query feature, and fuses the target modal feature with the preset query feature and their corresponding relationship to generate a multi-granularity retrieval feature.

[0091] like Figure 2 As shown, the cross-modal retrieval device uses the Q-Former model to establish the alignment relationship between the image modality and the text modality. Among them, Q-Former, namely Querying Transformer, performs visual-language alignment between the frozen visual model and the large language model.

[0092] Specifically, the cross-modal retrieval device combines the learnable query (query text) and image features Input into the Q-Former model to achieve visual-language alignment. The Q-Former model consists of N layers, each of which consists of three sub-layers: self-attention layer, cross-attention layer, and feedforward neural network layer. Q-Former inputs a learnable query, continuously performs self-attention operations, adjusts the attention weights of each word feature of the query, and continuously compares the query with the image features. Do cross attention and build image features The Q-Former model outputs multi-granularity retrieval features after N layers of processing. .

[0093] In an embodiment of the present application, a cross-modal retrieval device segments the original modal data to obtain a number of segmented data segments; the original modal data is input into a modal encoder to extract local modal features and global modal features of the original modal data; the target granularity features of each segmented data segment are obtained using the local modal features; the global modal features and all the target granularity features are concatenated to obtain the target modal features; an alignment relationship between the target modal features and the preset query features is established to obtain multi-granularity retrieval features of the original modal data. Through the above-mentioned feature construction method, modal data is characterized by local features and global features, multi-granularity retrieval features are constructed, and refined feature alignment between different modal data is achieved.

[0094] Feature vector base library established based on the above feature construction method , which can be used for cross-modal retrieval. For details, please refer to Figure 3 , Figure 3 It is a flowchart of an embodiment of a cross-modal retrieval method provided by the present application.

[0095] like Figure 3 As shown, the specific steps are as follows:

[0096] Step S21: Acquire a feature vector base library of the first modal data.

[0097] In an embodiment of the present application, a cross-modal retrieval device obtains a pre-constructed feature vector base library of first modal data, wherein the multi-granularity retrieval features in the feature vector base library can be constructed by the above-mentioned feature construction method, which will not be repeated here.

[0098] Step S22: Acquire query data features of the second modal data.

[0099] In an embodiment of the present application, a cross-modal retrieval device receives second modal data input by a user, and encodes the second modal data through a corresponding modal encoder to obtain corresponding query data features.

[0100] Step S23: Calculate the matching similarity between the query data feature and each multi-granularity retrieval feature.

[0101] In an embodiment of the present application, a cross-modal retrieval device calculates the matching similarity between the query data feature and each multi-granularity retrieval feature, and evaluates the correspondence between the first modal data and the second modal data according to the size of the matching similarity.

[0102] Specifically, this application is aimed at multi-granularity retrieval features The characteristics of the system provide a specific matching similarity calculation method, please continue to refer to Figure 4 and Figure 5 , Figure 4 yes Figure 3 The specific flow chart of step S23 in the cross-modal retrieval method is shown in FIG. Figure 5 This is a schematic diagram of calculating the similarity between images and texts of different granularities provided in this application.

[0103] like Figure 4 As shown, the specific steps are as follows:

[0104] Step S231: Acquire each sub-data feature of the query data feature.

[0105] In the embodiment of the present application, the cross-modal retrieval device obtains each sub-data feature of the query data feature. It can be understood that the query data feature is the global feature of the query data, and each sub-data feature is the local feature of the query data. Taking the query text as an example, the query data feature is the query sentence feature, and each sub-data feature is each word segmentation feature.

[0106] Step S232: Obtain the target modality features and related query features in each multi-granularity retrieval feature.

[0107] In an embodiment of the present application, a cross-modal retrieval device performs avg pooling on multi-granularity retrieval features to obtain image features, namely, target modal features; and extracts queries from the multi-granularity retrieval features, wherein the target modal features have an alignment relationship with the corresponding queries.

[0108] Step S233: Calculate the first matching similarity using the target modal features and the query data features.

[0109] In the embodiments of the present application, Figure 5 As shown, the cross-modal retrieval device calculates the image-sentence matching score s1 by combining the image features with the query text classtoken. The calculation result of the matching score s1 is a single value.

[0110] Step S234: Calculate the second matching similarity using the target modal feature and each sub-data feature of the query data feature.

[0111] In the embodiments of the present application, Figure 5 As shown, the cross-modal retrieval device calculates the image-segmentation matching score s2 by combining the image features with the text segmentation tokens. The calculation results of the image features and the text segmentation tokens are one-dimensional vectors, and the cross-modal retrieval device uses the maximum value thereof as the calculation result of the matching score s2.

[0112] Step S235: Calculate a third matching similarity using the relevant query feature and each sub-data feature of the query data feature.

[0113] In the embodiments of the present application, Figure 5 As shown, the cross-modal retrieval device calculates the query-segmentation matching score s3 for each query and text segmentation tokens. The calculation result of the query and text segmentation tokens is a matrix vector, and the cross-modal retrieval device uses the maximum value as the calculation result of the matching score s3.

[0114] Step S236: Calculate a fourth matching similarity using the relevant query features and the query data features.

[0115] In the embodiments of the present application, Figure 5 As shown, the cross-modal retrieval device calculates the query-sentence matching score s4 for each query and query text class token. The calculation result of the query and the query text class token is a one-dimensional vector, and the cross-modal retrieval device uses the maximum value thereof as the calculation result of the matching score s4.

[0116] Step S237: Based on the first matching similarity, the second matching similarity, the third matching similarity and the fourth matching similarity, the matching similarity between the query data feature and each multi-granularity search feature is calculated.

[0117] In the embodiment of the present application, the cross-modal retrieval device may average all the matching similarities calculated above, and use the average value as the matching similarity between the final query data feature and each multi-granularity retrieval feature.

[0118] In a specific implementation, the cross-modal retrieval device calculates the similarity s=(s1+s2+s3+s4) / 4 between the image i and the text.

[0119] Step S24: sort the multi-granularity search features of the feature vector base according to the matching similarity, and output the search results.

[0120] In the embodiment of the present application, after the cross-modal search device traverses all multi-granularity search features in the feature vector base library, it sorts all the multi-granularity search features according to the matching similarity and outputs the search result accordingly.

[0121] The feature construction method of this application utilizes segmentation to obtain the target bounding box, splicing global and target local features, and using Q-Former to obtain multi-granularity retrieval features that match the text, so as to align the text with the image refined features.

[0122] The cross-modal retrieval method of this application proposes a multi-granularity matching score calculation method to significantly improve the retrieval effect.

[0123] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0124] In order to implement the above-mentioned feature construction method and / or cross-modal retrieval method, the present application also proposes a cross-modal retrieval device, for details, please refer to Figure 6 , Figure 6 It is a structural diagram of an embodiment of a cross-modal retrieval device provided by the present application.

[0125] The cross-modal search apparatus 400 of this embodiment includes a processor 41 , a memory 42 , an input / output device 43 , and a bus 44 .

[0126] The processor 41, memory 42, and input / output device 43 are respectively connected to a bus 44. The memory 42 stores program data. The processor 41 is used to execute the program data to implement the feature construction method and / or cross-modal retrieval method described in the above embodiments.

[0127] In the embodiment of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip having the ability to process signals. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 41 may also be any conventional processor, etc.

[0128] This application also provides a computer storage medium, please continue to refer to Figure 7 , Figure 7 It is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the feature construction method and / or cross-modal retrieval method of the above-mentioned embodiment.

[0129] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0130] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A feature construction method, characterized in that: The feature construction method comprises: Segment the original modal data to obtain a number of segmented data segments; Inputting the original modal data into a modal encoder to extract local modal features and global modal features of the original modal data; Using the local modal features, obtaining target granularity features of each segmented data segment; Concatenating the global modal feature and all the target granular features to obtain a target modal feature; Establishing an alignment relationship between the target modal feature and the preset query feature to obtain a multi-granularity retrieval feature of the original modal data; The establishing of the alignment relationship between the target modal feature and the preset query feature includes: The target modal feature and the preset query feature are input into a cross attention layer, so as to perform a cross attention operation on each preset query feature and the target modal feature respectively, and establish an alignment relationship between the target modal feature and the relevant query feature.

2. The feature construction method according to claim 1, characterized in that: The method of using the local modal features to obtain target granularity features of each segmented data segment includes: Acquire a feature set of each segmented data segment on the local modal feature; Feature pooling is performed on the feature set to obtain target granularity features of each segmented data segment.

3. The feature construction method according to claim 2, characterized in that: The feature pooling method includes region of interest pooling and / or average pooling.

4. The feature construction method according to claim 1, characterized in that: Before establishing the alignment relationship between the target modal feature and the preset query feature, the feature construction method further includes: Each preset query feature is input into the self-attention layer to adjust the attention weight of each sub-feature in each preset query feature.

5. A cross-modal retrieval method, characterized in that: The cross-modal retrieval method comprises: Acquire a feature vector base library of the first modal data, wherein each multi-granularity retrieval feature in the feature vector base library is constructed by the feature construction method according to any one of claims 1 to 4; Obtaining query data features of the second modal data; Calculating the matching similarity between the query data feature and each of the multi-granularity retrieval features; The multi-granularity search features of the feature vector base library are sorted according to the matching similarity, and the search results are output.

6. The cross-modal retrieval method according to claim 5, characterized in that: The calculating the matching similarity between the query data feature and each of the multi-granularity retrieval features includes: Obtaining each sub-data feature of the query data feature; Obtaining target modality features and related query features in each of the multi-granularity retrieval features; Calculating a first matching similarity using the target modality feature and the query data feature; Calculating a second matching similarity using the target modal feature and each sub-data feature of the query data feature; Calculating a third matching similarity using the relevant query feature and each sub-data feature of the query data feature; Calculating a fourth matching similarity using the relevant query feature and the query data feature; Based on the first matching similarity, the second matching similarity, the third matching similarity and the fourth matching similarity, a matching similarity between the query data feature and each of the multi-granularity retrieval features is calculated.

7. The cross-modal retrieval method according to claim 5, characterized in that: The first modality data is any one of image data, text data, audio data, and video data; the second modality data is any one of image data, text data, audio data, and video data; The first modal data and the second modal data are data of different modalities.

8. A cross-modal retrieval device, characterized in that: The cross-modal retrieval device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the feature construction method as described in any one of claims 1 to 4, and / or the cross-modal retrieval method as described in any one of claims 5 to 7.

9. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the feature construction method as described in any one of claims 1 to 4, and / or the cross-modal retrieval method as described in any one of claims 5 to 7.

Images