Cross-modal retrieval method and system and electronic equipment

Abstract

The invention provides a cross-modal retrieval method and system and electronic equipment. The method comprises the steps that a target modal needing to be retrieved by a retrieval request is recognized; performing feature extraction on the first modal data to obtain a first semantic vector; mapping the first semantic vector to a unified semantic space through a pre-trained mapping matrix to obtain a second semantic vector; splitting the second semantic vector into first semantic features of different hierarchies through a hierarchical feature coding model; calculating the similarity of the first semantic feature and the candidate semantic feature in each level in different levels; based on the similarity and the weight of each level, obtaining a comprehensive similarity; and screening out N candidate semantic features from a plurality of candidate semantic features corresponding to the target modality according to the comprehensive similarity. According to the technical scheme, the retrieval accuracy of cross-modal retrieval can be improved.

Description

Technical Field

[0001] This invention relates to the field of rail transit equipment manufacturing technology, and provides a cross-modal retrieval method, system and electronic device. Background Technology

[0002] Current data retrieval methods are mostly limited to single-modal retrieval, such as text retrieval of text and image retrieval of images, failing to meet users' needs for cross-modal data retrieval. Cross-modal retrieval refers to the technology of using one type of data (query modality) to search for another type of data (target modality), breaking the traditional restriction that the "query modality" and "target modality" must be of the same type. For example, using text to search for images or videos, using images to search for text, and using voice to search for videos. However, some current cross-modal retrieval methods sometimes return search results that deviate from the user's actual expectations, resulting in low retrieval accuracy and impacting user experience. This is especially true in the production or maintenance application scenarios of rail transit equipment manufacturing, where current cross-modal retrieval solutions cannot meet the cross-modal retrieval needs of rail transit equipment manufacturing applications. Summary of the Invention

[0003] This invention provides a cross-modal retrieval method, system, and electronic device that can improve retrieval accuracy and meet the cross-modal retrieval needs in the application scenario of rail transit equipment manufacturing.

[0004] This invention proposes a cross-modal retrieval method, which includes: in response to receiving a retrieval request, identifying the target modality to be retrieved in the retrieval request; extracting features from the first modality data included in the retrieval request to obtain a first semantic vector; the first modality data and the target modality have different or the same modality category; mapping the first semantic vector to a unified semantic space through a pre-trained mapping matrix to obtain a second semantic vector; the mapping matrix is ​​used to semantically align the semantic vectors of different modalities; inputting the second semantic vector into a pre-trained hierarchical feature encoding model, and splitting the second semantic vector into first semantic features at different levels through the hierarchical feature encoding model; indexing candidate semantic features corresponding to the target modality in a pre-constructed data center; calculating the similarity between the first semantic features and the candidate semantic features at each level in different levels; obtaining the weights corresponding to each level in different levels; obtaining a comprehensive similarity based on the similarity and weights of each level; and selecting N candidate semantic features from multiple candidate semantic features corresponding to the target modality according to the comprehensive similarity, and using the N original files corresponding to the N candidate semantic features as the retrieval results.

[0005] According to one embodiment of the present invention, different levels include global, local and attribute; the second semantic vector is split into first semantic features of different levels through a hierarchical feature encoding model, including: splitting the second semantic vector into first semantic features corresponding to the three levels of global, local and attribute through a hierarchical feature encoding model.

[0006] According to one embodiment of the present invention, calculating the similarity between the first semantic feature and the candidate semantic feature at different levels includes: calculating the first similarity between the first semantic feature and the candidate semantic feature at the global level; calculating the second similarity between the first semantic feature and the candidate semantic feature at the local level; and calculating the third similarity between the first semantic feature and the candidate semantic feature at the attribute level.

[0007] According to one embodiment of the present invention, a hierarchical feature encoding model is used to decompose a second semantic vector into first semantic features corresponding to three levels: global, local, and attribute. This includes: in response to the identification that the modality category of the first modal data is text, invoking a first-level feature encoding model to decompose the second semantic vector into overall semantic features corresponding to the global level, core viewpoint features corresponding to the local level, and entity features corresponding to the attribute level; or, in response to the identification that the modality category of the first modal data is an image or video frame, invoking a second-level feature encoding model to decompose the second semantic vector into first semantic features corresponding to the global level, core viewpoint features corresponding to the local level, and entity features corresponding to the attribute level; or, in response to the identification that the modality category of the first modal data is an image or video frame, invoking a second-level feature encoding model to decompose the second semantic vector into first semantic features corresponding to the global level, local level, and attribute level. The first modal data is split into global scene features at the global level, device region features at the local level, and first attribute features at the attribute level. The first attribute features are used to represent at least one of the color, shape, or state included in the first modal data. Alternatively, in response to the identification that the modal category of the first modal data is audio, the third-level feature encoding model is invoked. The second semantic vector is split into global sound features at the global level, event features corresponding to the sound type at the local level, and first attribute features at the attribute level. The first attribute features are used to represent at least one of the frequency and intensity corresponding to the first modal data.

[0008] According to one embodiment of the present invention, a first-level feature encoding model includes a first Transformer module, a second Transformer module, and a third Transformer module; the first Transformer module is used to extract overall semantic features from the input second semantic vector; the second Transformer module is used to extract core viewpoint features from the input second semantic vector; and the third Transformer module is used to extract entity features from the input second semantic vector. A second-level feature encoding model includes a first encoding module, a second encoding module, and a third encoding module; the first encoding module is used to extract overall scene features from the input second semantic vector; the second Transformer module is used to extract device region features from the input second semantic vector; and the third Transformer module is used to extract device region features from the input second semantic vector. The rmer module is used to extract at least one of color, shape, or state from the input second semantic vector; wherein, the first encoding module, the second encoding module, or the third encoding module is a convolutional neural network (CNN) or a visual model based on the Transformer architecture, VIT-Transformer; the third-level feature encoding model includes a fourth Transformer module, a fifth Transformer module, and a sixth Transformer module; the fourth Transformer module is used to extract overall sound features from the input second semantic vector; the second Transformer module is used to extract event features corresponding to sound types from the input second semantic vector; and the third Transformer module is used to extract at least one of frequency and intensity from the input second semantic vector.

[0009] According to an embodiment of the present invention, before obtaining the weights corresponding to each level in different levels, the method further includes: identifying the modality category of the first modality data; obtaining the weights corresponding to each level in different levels includes: calling a pre-trained dynamic weight allocation model, inputting the modality category of the first modality data and the modality category of the target modality into the dynamic weight allocation model, and outputting the weights corresponding to each level in different levels through the dynamic weight allocation model; wherein, the dynamic weight allocation model is a model using an attention mechanism, and the dynamic weight allocation model includes an input layer, an attention layer and an output layer.

[0010] According to an embodiment of the present invention, before mapping the first semantic vector to a unified semantic space through a pre-trained mapping matrix to obtain the second semantic vector, the method further includes: obtaining training samples; the training samples include positive sample pairs and negative sample pairs; the positive sample pairs include samples of the first modality and the second modality that are semantically aligned in the same scene; the negative sample pairs include samples of the third modality and the fourth modality that are semantically misaligned in different scenes; using the training samples, with the training objective of minimizing the distance between two samples in the positive sample pair and maximizing the distance between two samples in the negative sample pair, the initial mapping matrix is ​​trained to obtain a pre-trained mapping matrix.

[0011] According to an embodiment of the present invention, the retrieval request is a retrieval request submitted by a user through a basic retrieval entry point; before feature extraction of the first modal data included in the retrieval request, the method further includes: performing lightweight processing on the first modal data; the lightweight processing includes: removing function words and retaining keywords from the first modal data whose modal category is text; performing image compression or frame extraction processing on the first modal data whose modal category is image or video frame; splitting the second semantic vector into first semantic features at different levels through a hierarchical feature encoding model, including: splitting the second semantic vector into first semantic features corresponding to the global level and the local level respectively through a hierarchical feature encoding model; calculating the similarity of the first semantic features and candidate semantic features at each level in different levels, including: calculating the first similarity of the first semantic features and candidate semantic features at the global level and the second similarity at the local level; obtaining the weights corresponding to each level in different levels, including: calling a pre-trained dynamic weight allocation model, inputting the modal category of the first modal data and the modal category of the target modality into the dynamic weight allocation model, and outputting the first weight corresponding to the global level and the second weight corresponding to the local level through the dynamic weight allocation model.

[0012] According to one embodiment of the present invention, the retrieval request is a retrieval request submitted by a user through an advanced retrieval portal; the retrieval request includes retrieval parameters, which include at least one of a confidence threshold, a maximum number of returned results, and a time range; after taking N original files corresponding to N candidate semantic features as retrieval results, the method further includes: according to the retrieval parameters, selecting M retrieval results from the N retrieval results whose confidence is greater than or equal to the confidence threshold and / or whose upload time conforms to the time range; M is less than or equal to the maximum number of returned results.

[0013] According to an embodiment of the present invention, after taking N original files corresponding to N candidate semantic features as retrieval results, the method further includes: exporting metadata corresponding to the retrieval results; the metadata includes at least one of the following: original file ID, comprehensive similarity, dataset name or ID from which the original file originates, upload time, uploader, and source description; the original file ID is used to link to the corresponding original file for users to download or view; and / or, exporting the N original files or the compressed package corresponding to the N original files in the retrieval results.

[0014] According to one embodiment of the present invention, the candidate semantic features corresponding to the target modality and the original files corresponding to the candidate semantic features are stored in a data center; the data center stores datasets of different categories; the datasets of different categories include personal datasets, platform datasets and public datasets.

[0015] According to one embodiment of the present invention, candidate semantic features corresponding to the target modality and the original files corresponding to the candidate semantic features are stored in a data center; the data center stores datasets of different categories; the datasets of different categories include datasets associated with different semantic labels; before indexing the candidate semantic features corresponding to the target modality in the pre-built data center, the method further includes: constructing a multimodal index library using the FAISS tool; before indexing the candidate semantic features corresponding to the target modality in the pre-built data center, the method further includes: Identify the first semantic label corresponding to the first modality data; index the candidate semantic features corresponding to the target modality in the pre-constructed data center, including: indexing the first target dataset associated with the first semantic label in the multimodal index library using the FAISS tool based on the first semantic label; and indexing the candidate semantic features corresponding to the target modality in the first target dataset using the FAISS tool.

[0016] According to one embodiment of the present invention, after taking N original files corresponding to N candidate semantic features as search results, the method further includes: receiving second semantic tags marked by the user on the search results, recording the association between the second semantic tags and the search results, and storing the association in a data center.

[0017] According to an embodiment of the present invention, the method further includes: based on multiple retrieval requests, counting the number of various combinations of index modalities and target modalities requested by the user; identifying combinations whose number exceeds a preset threshold as high-frequency combinations; identifying combinations whose number is less than or equal to the preset threshold as low-frequency combinations; prioritizing the updating and / or compression of corresponding index information for high-frequency combinations; and archiving low-frequency combinations.

[0018] According to an embodiment of the present invention, before performing feature extraction on the first modal data included in the retrieval request, the method further includes: in response to the modality category of the first modal data being text, converting the first modal data into text data in TXT format encoded in UTF-8; in response to the modality category of the first modal data being an image or video frame, performing frame extraction processing on the video frame to obtain an image, and converting the image into a uniform image format with uniform resolution; in response to the modality category of the first modal data being audio, converting the first modal data into a uniform audio format with uniform sampling rate; preprocessing the image data with uniform image format, the audio data with uniform audio format, or the text data in TXT format to obtain preprocessed first modal data; performing basic feature extraction on the preprocessed first modal data to obtain initial features; and performing feature extraction on the first modal data included in the retrieval request, including: performing feature extraction on the initial features corresponding to the first modal data included in the retrieval request.

[0019] According to one embodiment of the present invention, preprocessing of image data with a uniform image format, audio data with a uniform audio format, or text data in TXT format includes: removing Gaussian noise from image data with a uniform image format using OpenCV; or cropping silent segments from audio data with a uniform audio format using Librosa; or removing specified symbols from text data in TXT format using regular expressions.

[0020] According to one embodiment of the present invention, basic feature extraction is performed on the preprocessed first modality data to obtain initial features, including: calculating the color histogram and edge features of the image data after removing Gaussian noise to obtain initial image features; or, extracting Mel-spectrum coefficients from the audio data after cropping the silent segments to obtain initial audio features; or, segmenting the text data after removing specified symbols and generating bag-of-words vectors, using the bag-of-words vectors as initial text features.

[0021] This invention also proposes a cross-modal retrieval system, which includes a data center, a retrieval engine, and a multimodal retrieval module. The multimodal retrieval module is used to identify the target modality to be retrieved in response to a received retrieval request. The retrieval engine includes a feature extraction model, a semantic alignment unit, a hierarchical feature encoding model, and an indexing unit. The feature extraction unit extracts features from the first modality data included in the retrieval request to obtain a first semantic vector. The first modality data may have a different or the same modality category as the target modality. The semantic alignment unit maps the first semantic vector to a unified semantic space using a pre-trained mapping matrix to obtain a second semantic vector. The mapping matrix is ​​used to semantically align the semantic vectors of different modalities. The retrieval engine is used to... The vector is input to a pre-trained hierarchical feature encoding model, which splits the second semantic vector into first semantic features at different levels. An indexing unit is used to index candidate semantic features corresponding to the target modality in a pre-built data center. A multimodal retrieval module includes a similarity calculation unit, which calculates the similarity between the first semantic features and candidate semantic features at each level, and obtains the weights corresponding to each level. Based on the similarity and weights at each level, a comprehensive similarity is obtained. The multimodal retrieval module then selects N candidate semantic features from multiple candidate semantic features corresponding to the target modality based on the comprehensive similarity, and uses the N original files corresponding to the N candidate semantic features as the retrieval results.

[0022] According to one embodiment of the present invention, the different levels include global, local and attribute; the retrieval engine is specifically used to: decompose the second semantic vector into first semantic features corresponding to the three levels of global, local and attribute through a hierarchical feature encoding model.

[0023] According to one embodiment of the present invention, the retrieval engine is specifically configured to: in response to identifying the modality category of the first modal data as text, invoke a first-level feature encoding model to decompose the second semantic vector into a global-level overall semantic feature, a local-level core viewpoint feature, and an attribute-level entity feature; or, in response to identifying the modality category of the first modal data as an image or video frame, invoke a second-level feature encoding model to decompose the second semantic vector into a global-level overall scene feature, a local-level device region feature, and an attribute-level first attribute feature; the first attribute feature is used to represent at least one of color, shape, or state included in the first modal data; or, in response to identifying the modality category of the first modal data as audio, invoke a third-level feature encoding model to decompose the second semantic vector into a global-level overall sound feature, a local-level sound type event feature, and an attribute-level first attribute feature; the first attribute feature is used to represent at least one of frequency and intensity corresponding to the first modal data.

[0024] According to an embodiment of the present invention, the multimodal retrieval module includes a dynamic weight allocation model; the multimodal retrieval module is specifically used for: identifying the modality category of the first modality data; calling the pre-trained dynamic weight allocation model, inputting the modality category of the first modality data and the modality category of the target modality into the dynamic weight allocation model, and outputting the weights corresponding to each level in different levels through the dynamic weight allocation model; wherein, the dynamic weight allocation model is a model using an attention mechanism, and the dynamic weight allocation model includes an input layer, an attention layer and an output layer.

[0025] The present invention also provides an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the cross-modal retrieval method as described above.

[0026] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a cross-modal retrieval method as described above.

[0027] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a cross-modal retrieval method as described above.

[0028] This invention provides a cross-modal retrieval method, system, and electronic device. After receiving a retrieval request submitted by a user, the method first identifies the target modality (target) to be retrieved in the retrieval request, and extracts features from the first modality data (query) included in the retrieval request to obtain a first semantic vector. Then, it maps the first semantic vector to a unified semantic space through a mapping matrix to obtain a second semantic vector. The mapping matrix enables semantic alignment of semantic vectors from different modalities. The process of feature extraction and semantic mapping can convert query data (i.e., first modality data) from different modalities into semantic features (semantic vectors). For example, if an image is used as the query modality, then the image features can be converted into a semantic vector (second semantic vector) through the above-mentioned feature extraction and semantic mapping process. The semantic vector is then used as the query data to retrieve the target modality data. Next, the second semantic vector is input into a pre-trained hierarchical feature encoding model. This model decomposes the second semantic vector into first semantic features at different levels. Candidate semantic features corresponding to the target modality are indexed in a pre-constructed data center. The similarity between the first semantic features and candidate semantic features at each level is calculated. Based on the weights and similarities at each level, a comprehensive similarity is calculated. This comprehensive similarity is a combined value that integrates the similarities between features at different levels. The matching degree between the first semantic features and candidate semantic features in the data center is evaluated at multiple levels. Compared to simply calculating the similarity between features, this improves matching accuracy, resulting in more accurate (i.e., more user-relevant) search results and improving user experience. The feature extraction and semantic mapping processes achieve feature extraction and semantic alignment for different modalities. Further decomposition of the aligned semantic vector into multiple levels of semantic features ensures compatibility with different modalities and accurate feature representation. Multi-level semantic feature matching based on accurate semantic features leads to higher accuracy. Thus, the synergistic effect of these multiple technical features improves search accuracy. The candidate semantic features stored in the data center can be obtained based on various modal data in the application scenario of rail transit equipment manufacturing. Thus, the retrieval scheme can be applied to the application scenario of rail transit equipment manufacturing and meet the cross-modal retrieval needs in this application scenario. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating the cross-modal retrieval method provided by the present invention; Figure 2 This is an example diagram of the software system architecture corresponding to the cross-modal retrieval method provided by the present invention; Figure 3 This is a flowchart illustrating an embodiment of the cross-modal retrieval method provided by the present invention; Figure 4 This is a schematic diagram of the software architecture of the cross-modal retrieval system provided by the present invention; Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0031] Figure label: 401. Data Center; 402. Search Engine; 403. Multimodal Search Module; 510. Processor; 520. Communication interface; 530. Memory; 540. Communication bus. Detailed Implementation

[0032] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0033] Traditional data retrieval methods are mostly limited to single-modal retrieval, such as text retrieval of text and image retrieval of images. Matching single data types cannot achieve cross-modal data retrieval, such as text and images, or images and videos, failing to meet users' needs for cross-type data retrieval. The rapid development of artificial intelligence technology has made cross-modal retrieval more possible. Cross-modal retrieval refers to the technology of using one type of data (query modality) to search for another type of data (target modality), breaking the traditional restriction that the "query" and "target" must be of the same type. Multimodal data can include text, images, audio, and video. Especially in the application scenario of rail transit equipment manufacturing, related cross-modal retrieval methods are difficult to apply to this scenario, and some cross-modal retrieval methods have low retrieval accuracy, failing to meet actual user needs and affecting user experience.

[0034] Furthermore, the retrieval solutions in related technologies lack a refined filtering mechanism for retrieval results, making it difficult to adjust the matching accuracy according to user needs. They generally suffer from problems such as low retrieval accuracy, insufficient flexibility in parameter configuration, and a disconnect between retrieval results and data management processes.

[0035] In view of this, embodiments of the present invention propose a cross-modal retrieval method, system, and electronic device, which can be applied to MaaS (Mobility as a Service Platform) platforms or other designated platforms to meet the diverse retrieval needs in rail transit equipment manufacturing or other application scenarios, optimize retrieval results, and improve retrieval accuracy.

[0036] Specifically, the cross-modal retrieval method proposed in this embodiment of the invention can be applied to data retrieval application scenarios in multiple fields such as rail transit equipment manufacturing, industrial quality inspection, medical imaging, and intelligent security.

[0037] The cross-modal retrieval method proposed in the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0038] The cross-modal retrieval method proposed in this embodiment of the invention, such as... Figure 1 As shown, the method includes: Step 101: In response to receiving a retrieval request, identify the target modality that the retrieval request needs to retrieve.

[0039] For example, when searching for a video using text, the text is the query modality and the image is the target modality; similarly, when searching for a video using an image, the image is the query modality and the video is the target modality. The query modality and the target modality can be different or the same, meaning that the retrieval scheme provided in this embodiment supports both cross-modal and single-modal retrieval.

[0040] A search request typically includes data for the query modality (first modality data) and modality category information for the target modality. For example, if a user needs to search for a video based on an image, and the user uploads an image with the content "train door malfunction" at the search entry point, and specifies in the search request that the target modality to be searched is video, such as the user inputting "Please search for video based on the uploaded image", a search request is generated based on the user input. In this case, the image of "train door malfunction" uploaded by the user is the first modality data, and the video is the target modality to be searched.

[0041] Step 102: Extract features from the first modality data included in the retrieval request to obtain the first semantic vector.

[0042] Feature extraction is performed on the first modality data included in the retrieval request. Specifically, this can be achieved using a model that supports feature extraction from multimodal data, such as one or more combinations of the following models: Contrastive Language-Image Pre-training (CLIP), A Large-scale Image and Noisy-textGrounding (ALIGN), Bootstrapping Language-Image Pre-training (BLIP), and Foundation Language-Vision Alignment (FLAVA).

[0043] Step 103: Map the first semantic vector to a unified semantic space through a pre-trained mapping matrix to obtain the second semantic vector.

[0044] A mapping matrix is ​​used to semantically align semantic vectors of different modalities.

[0045] In this embodiment of the invention, after extracting features from the first modality data of the query modality to obtain the first semantic vector, the first modality data is further semantically aligned. The semantic alignment is achieved by mapping the first semantic vector through a pre-trained mapping matrix to obtain the second semantic vector.

[0046] The size of the mapping matrix is ​​determined by the dimensions of the first semantic vector and the second semantic vector. For example, if the first semantic vector is a 128-dimensional vector and the second semantic vector is a 128-dimensional vector, then the mapping matrix is ​​128×128, where each element is a parameter. The specific mapping method can be to perform a linear operation between the element in the first semantic vector and the parameter at the corresponding position in the mapping matrix, and use the resulting value as the element at the corresponding position in the second semantic vector. For example, the linear operation can be to multiply the element in the first semantic vector by the element at the corresponding position in the mapping matrix (the mapping coefficient), or a bias term can be set. Both the mapping coefficient and the bias term are trainable parameters that can be obtained through training.

[0047] Optionally, in some embodiments, different mapping matrices are trained for different modality combinations. For example, when retrieving images from text, the modality combination is text-image; when retrieving videos from images, the modality combination is image-video, and so on. Combining three query modalities with three target modalities can yield up to nine modality combinations (including single modalities), including three single modalities and six cross-modal combinations. In some embodiments, mapping matrices can be trained for each of these six cross-modal combinations. Specifically, the modality category of the first modality data is identified, and based on the combination of the first modality data and the target modality, a corresponding mapping matrix is ​​selected for semantic mapping to obtain a second semantic vector. Training different mapping matrices for different modality combinations can improve the accuracy of semantic mapping. Configuring dedicated mapping matrices for different cross-modal searches improves semantic alignment accuracy. In other embodiments, the same mapping matrix can be used for different modality combinations.

[0048] Step 104: Input the second semantic vector into the pre-trained hierarchical feature encoding model, and use the hierarchical feature encoding model to split the second semantic vector into first semantic features at different levels.

[0049] Different levels, i.e., at least two levels. A hierarchical feature encoding model can include encoding models of at least two levels. When a second semantic vector is input, the encoding model of one level outputs the first semantic feature of that level. Thus, the first semantic features of at least two levels are output.

[0050] For example, in embodiments of the present invention, different levels may include at least two of the following: global level, local level, and attribute level. The first semantic feature of the global level is used to represent the global features of a modal data (e.g., the first modal data). The first semantic feature of the local level is used to represent the local features of a modal data (e.g., the first modal data), and the first semantic feature of the attribute level is used to represent the attribute features of a modal data (e.g., the first modal data).

[0051] For example, in some embodiments, the different levels include three levels: global, local, and attribute. The second semantic vector is split into first semantic features of different levels through a hierarchical feature encoding model. Specifically, the second semantic vector can be split into first semantic features corresponding to the global, local, and attribute levels respectively through a hierarchical feature encoding model.

[0052] Step 105: Index the candidate semantic features corresponding to the target modality in the pre-built data center.

[0053] In this embodiment of the invention, a data center is pre-constructed, which stores pre-obtained candidate semantic features. The method of obtaining candidate semantic features is consistent with the process of extracting features from the first modality data and obtaining semantic vectors through semantic mapping using a mapping matrix, and then splitting them. That is, the method of processing the first modality data in steps 102-104 can be referred to to perform feature extraction, semantic mapping, and hierarchical feature splitting on the sample data of various pre-collected modalities to obtain semantic candidate features at different levels.

[0054] Step 106: Calculate the similarity between the first semantic feature and the candidate semantic features at each level in different layers.

[0055] For example, if different levels include a first level and a second level, then calculating the similarity between the first semantic feature and the candidate semantic features at each level is equivalent to calculating the similarity between the first semantic feature and the candidate semantic features at the first level, and the similarity between the first semantic feature and the candidate semantic features at the second level. Alternatively, if different levels include a first level, a second level, and a third level, then calculating the similarity between the first semantic feature and the candidate semantic features at each level is equivalent to calculating the similarity between the first semantic feature and the candidate semantic features at the first level, the similarity between the first semantic feature and the candidate semantic features at the second level, and the similarity between the first semantic feature and the candidate semantic features at the third level.

[0056] For example, the first level can be the global level, the second level can be the local level, and the third level can be the attribute level.

[0057] Step 107: Obtain the weights corresponding to each level in different levels.

[0058] In some embodiments of this invention, a dynamic weight allocation mechanism based on an attention mechanism is proposed. A pre-trained dynamic weight allocation model can dynamically predict the weights corresponding to each level for different modality combinations. For example, by inputting the modality category of the query modality (e.g., modality category ID) and the modality category of the target modality (e.g., modality category ID) into the dynamic weight allocation model, the weights of the global level, local level, and attribute level can be dynamically predicted. That is, the weights of each level are dynamically predicted according to different modality combinations, and the weights change dynamically according to the modality combinations in the actual application scenario. For example, in embodiments where different levels include a global level, a local level, and an attribute level, the first weight corresponding to the global level, the second weight corresponding to the local level, and the third weight corresponding to the attribute level are dynamically predicted.

[0059] In other embodiments, the weights of each level can be set as configurable variables, and the weight variables can be manually assigned to obtain the weights of each level. Alternatively, the weights of each level can be set as constants.

[0060] Step 108: Based on the similarity and weight of each level, obtain the comprehensive similarity.

[0061] For example, the similarity scores of different levels can be weighted and summed according to the weight of each level to obtain the comprehensive similarity score. For example, global level × first weight + local level × second weight + attribute level × third weight = comprehensive similarity score.

[0062] Step 109: Based on the comprehensive similarity, select N candidate semantic features from multiple candidate semantic features corresponding to the target modality, and use the N original files corresponding to the N candidate semantic features as the retrieval results.

[0063] For example, candidate semantic features ranked in the top N by comprehensive similarity can be selected, and the N original files of the target modality corresponding to these N candidate semantic features can be used as search results.

[0064] This is used for data (e.g., original files) from which the target modal can be further selected from the search results.

[0065] Figure 1 The illustrated embodiment extracts accurate and multi-level semantic features from different modal data through feature extraction, semantic alignment, and hierarchical feature decomposition. It then performs hierarchical matching between these multi-level semantic features and candidate semantic features of the target modality pre-stored in the data center. Based on the similarity (matching value) and weight of each level, a comprehensive similarity is obtained. This comprehensive similarity accurately reflects the degree of matching between the first semantic feature corresponding to the first modality data (query modality) and the candidate semantic features of the target modality. The retrieval results selected based on the comprehensive similarity can more accurately match user needs and improve retrieval accuracy.

[0066] For example, such as Figure 2 As shown, the cross-modal retrieval method proposed in this embodiment of the invention can be based on Figure 2 The system architecture shown is executed.

[0067] The data center and search engine can be integrated into the MaaS platform, providing the basic capabilities required for cross-modal retrieval based on the platform's underlying architecture. The data center is configured to perform the following functions: storage, preprocessing, dataset classification, access control, and dataset association management for multimodal data such as text, images, audio, and video. For example, datasets can be divided into platform datasets, public datasets, and personal datasets. Dataset association management includes two aspects: batch adding search results to the corresponding datasets and associating semantic tags with datasets of different categories. Specifically, different data can be divided according to their semantics, such as grouping data from the same application scenario or within the same category into the same dataset. For example, a fire safety dataset stores various modalities of data related to fire safety, such as various fire images. Each category of dataset has corresponding semantic tags, and the data center needs to record and maintain the association between semantic tags and datasets.

[0068] The retrieval engine can integrate a feature extraction model, a semantic alignment unit, a hierarchical feature encoding model, and an indexing unit. Optionally, in some embodiments, a heat-aware index update unit can also be integrated.

[0069] Each unit or model in the search engine is specifically configured to perform the following functions: The feature extraction model extracts features from the input data (first modality data) of the query modality to obtain a first semantic vector. For example, the feature extraction model is the CLIP model.

[0070] The semantic alignment unit reads the parameters in the mapping matrix and performs semantic mapping on the first semantic vector to obtain the aligned second semantic vector.

[0071] Hierarchical feature encoding models are used to decompose second semantic vectors to obtain first semantic features at different levels. For example, hierarchical feature encoding models can extract features at different levels for each modality of data. If the first modality of data is text, then document-level (global level) features, paragraph-level (local level) features, and entity-level (attribute level) features are extracted. If the first modality of data is an image or video frame, then global level features, region-level (i.e., local level) features, and attribute-level features are extracted. If the first modality of data is audio, then segment-level (global level) features, event-level (local level) features, and attribute-level features are extracted. The features can be vectors or tensors.

[0072] An index unit is used to index candidate semantic features of a target modality within a data center. The data center may include a multimodal index, and the index unit can also be used to maintain the multimodal index to support fast queries.

[0073] Among them, hierarchical coding models and feature extraction models (such as CLIP) can run in parallel.

[0074] Optionally, a heat-aware index update unit is used to statistically analyze high-frequency search modal combinations, prioritize updating and compressing the index sub-databases involved in high-frequency combinations, and archive low-frequency indexes to save storage resources. For example, statistical analysis shows that text→video and image→text are high-frequency combinations.

[0075] like Figure 2 As shown, the multimodal retrieval module is used to realize cross-retrieval of data of different modalities, breaking the limitation of a single type. It supports text-to-image search, image-to-text search, text-to-video search, text-to-audio search, image-to-video search, etc. Its core function is to perform semantic matching between the input query modality data (first modality data) and the target modality data and return the associated results.

[0076] Specifically, the multimodal retrieval module includes a similarity calculation unit and a dynamic weight allocation model. The dynamic weight allocation model implements dynamic hierarchical attention matching: it introduces an attention mechanism to dynamically predict the weights of each level based on modality combinations. The input to the dynamic weight allocation model is the modality category of the query modality and the modality category of the target modality, and the output is the weights of each level. For example, through training, the dynamic weight allocation model learns that technical document text focuses on entity-level matching, while scene images focus on global or local-level matching. During prediction, it can dynamically allocate weights for each level; for example, in text-image matching, the entity-level and region-level weights are increased to 40%.

[0077] The similarity calculation unit is used to calculate the similarity between the input query modality data (first modality data) and the target modality data in terms of the three levels of "global-local-attribute" features, and to obtain the weights of each level output by the dynamic weight allocation model. After weighting, a comprehensive similarity score is obtained, realizing "hierarchical matching and key focus".

[0078] like Figure 2 As shown, the system architecture may also include a search entry point. In some embodiments, the search entry point specifically includes a basic search entry point and an advanced search entry point.

[0079] The basic search entry point (quick search entry point) is a lightweight search entry point, allowing users to directly input raw data (text keywords, image files, etc.) to quickly initiate searches without requiring parameter configuration. It prioritizes efficiency and returns preliminary matching results (including preview information and source association). It reuses the dynamic hierarchical attention matching logic (dynamic weight allocation) of the multimodal cross-type search module (i.e., the multimodal search module) and simplifies the calculation process, for example, by prioritizing the use of global features and core local features to improve response speed.

[0080] The advanced search entry (precise search entry) allows users to optimize search accuracy through parameter configuration. Users can set confidence thresholds (to filter highly matching results), maximum number of returns (to focus on core information), and time ranges (to limit data upload periods), making it suitable for scenarios with high accuracy requirements. Search parameters are applied to the comprehensive similarity score of dynamic hierarchical attention matching (dynamic weight allocation), further improving result accuracy through secondary filtering. For example, a high confidence threshold can filter results with a comprehensive similarity score below a set value.

[0081] like Figure 2As shown, the system architecture can also include a search result management and reuse module (or result application module). This module is used for subsequent processing and value extraction of search results, including result export (e.g., exporting metadata tables or raw files after packaging), dataset association (batch adding search results to personal datasets, public datasets, or other datasets), and historical record management (e.g., saving search parameters and result snapshots, supporting re-execution or modification of the search).

[0082] The search results management and reuse module can be configured with feedback learning units. These units prompt users to label search results via an interactive interface, such as labeling them as "strongly relevant / weakly relevant / irrelevant." The module detects user labeling behavior, records user labeling data, and records secondary filtering behavior. Furthermore, based on user-feedback labeling data and / or secondary filtering behavior data, online learning algorithms (such as online random forest updates) are used to adjust the weights of each level. For example, if a user repeatedly clicks on "text-image pairs containing specific entities," the weight of entity-level features is increased. Additionally, the threshold can be adaptively adjusted based on the user's preference for "precision-recall." For instance, if a user repeatedly uses a high-confidence threshold, the system automatically increases the default threshold.

[0083] For example, when a user enters content at the search entry point to trigger the flow of information, two search modes are displayed on the interactive interface: basic search and advanced search. Based on the user's selection, the corresponding basic search entry or advanced search entry point is displayed. The multimodal data (text, images, etc.) entered by the user is detected, and a search request is generated.

[0084] Search processing flow: The basic search entry directly transmits the input data to the multimodal cross-type search module (core search layer), triggering the core search process; the advanced search module first receives the search parameters configured by the user (such as confidence threshold), and then transmits the input data and search parameters together to the multimodal cross-type search module.

[0085] The core retrieval layer interacts with the infrastructure (data center and retrieval engine): The multimodal cross-type retrieval module requests the target modality's index library from the retrieval engine. For example, when searching for images with text, it requests the image feature index library and triggers the retrieval engine's "hierarchical feature encoding model" to perform hierarchical feature extraction on the input data and target index library data. For example, it extracts the document-level / paragraph-level / entity-level features of text and the global-level / region-level / attribute-level features of images.

[0086] The retrieval engine completes cross-modal semantic matching: first, it calculates similarity for each of the three levels of features; then, it assigns weights based on the type of modality combination (i.e., the combination of the modality categories of the query modality and the target modality) using a dynamic weight allocation model. For example, for a text-video combination, it increases the weights of the entity level and local level. After generating a comprehensive similarity score, it returns the preliminary retrieval results to the multimodal cross-type retrieval module. If the retrieval request is submitted through the advanced retrieval entry, the multimodal cross-type retrieval module performs a secondary filter on the preliminary retrieval results based on the user-set retrieval parameters. For example, it filters results that meet the comprehensive score criteria based on a confidence threshold before passing them to the retrieval result management and reuse module. Simultaneously, the retrieval engine's popularity-aware index update unit continuously monitors the modality combinations of the current retrieval (e.g., text → video) and incorporates them into the high-frequency retrieval analysis pool.

[0087] The results flow through the application layer: The retrieval results management and reuse module receives the processed retrieval results and displays and applies them. Simultaneously, the feedback learning unit initiates data collection, recording user labeling of results (strongly relevant / weakly relevant / irrelevant) and secondary filtering behaviors (such as prioritizing results containing specific entities). When a user performs the "Add to Dataset" operation, the module sends a request to the "Data Center" supported by the platform architecture, and the Data Center updates the dataset association relationships. All operation logs (such as retrieval time, processed data volume, user feedback data, and high-frequency modality combination statistical results) are synchronized to the platform audit log system, providing data support for feature weight updates in the feedback learning unit and index tuning in the popularity-aware index update unit.

[0088] The following is based on Figure 2 The system architecture shown is illustrated with some specific implementation examples based on real-world application scenarios.

[0089] For example, taking cross-modal retrieval by users in the actual application scenario of rail transit equipment manufacturing as an example, users need to use the advanced retrieval mode to achieve "searching for videos by image". By using images of train door malfunctions, users can find videos related to "train door malfunctions" uploaded within a specified time period (e.g., 2025). The retrieval parameters are confidence level ≥ 0.8 and a maximum number of returned videos of 5. That is, users require high accuracy and can return a maximum of 5 videos.

[0090] The data center has stored a "train operation dataset" containing 3,000 train-related videos uploaded in 2025. The retrieval engine's hierarchical feature coding model (or hierarchical feature coding unit) has extracted global-level (e.g., "train entering the station" scene) vectors, regional-level (e.g., "door area") vectors, and attribute-level (e.g., "door opening / closing status") vectors for these videos and built an index library. Through cross-modal semantic alignment, it is ensured that the comprehensive similarity of the three-level features between the "door malfunction" image and the corresponding video is ≥0.8.

[0091] The multimodal cross-type retrieval module receives a user-input image of "train door malfunction" and identifies it as an image modality. It then calls a hierarchical feature encoding unit to extract the global vector, door region vector, and "switching abnormality" attribute vector of the image. Dynamic hierarchical attention matching is performed using a dynamic weight allocation model, outputting weights of 40% for region-level (local level) features and 40% for attribute-level features, with a 20% weight allocated to the global level. Corresponding candidate semantic features are indexed in the video index library. The similarity between the global vector, door region vector, "switching abnormality" attribute vector, and the corresponding level vectors of the candidate semantic features is calculated. Finally, a comprehensive similarity is calculated, returning 30 preliminary results (comprehensive similarity 0.35-0.96).

[0092] The advanced search configuration module (advanced search entry point) receives the following search parameters: confidence level ≥ 0.8, maximum number of returns = 5, time range from 2025-01-01 to 2025-12-31, and source dataset "Train Operation Dataset". After filtering, 8 results that meet the criteria (overall similarity 0.81-0.96) are retained, and the first 5 are extracted.

[0093] The search results management and reuse module: Users mark three segments as "strongly relevant," and the feedback learning unit records this behavior and increases the weights of the "door area" and "open / closed status" features. Simultaneously, the five results are batch-added to the "Train Fault Analysis Dataset," with the semantic label "door fault" added. Metadata is exported to an Excel spreadsheet, recording the comprehensive similarity (e.g., 0.93, 0.87, etc.) and detailed scores of the three-level features for each video segment.

[0094] The implementation method of the above example is explained in detail below.

[0095] The Maas platform architecture supports standardized formats for multimodal data received by the data center, preprocessing and denoising to extract features, building an index library, and enabling cross-modal semantic alignment in the search engine. This data is synchronized to the platform's audit logs, supporting data interaction and access control for other modules, thus providing a foundation for retrieval. Specifically, this includes: Step 301: Collect sample data and perform standardization, preprocessing, and basic feature extraction on the sample data.

[0096] Collect or receive multimodal sample data (text, images, audio, video, etc.) uploaded by users, and record the original data format (e.g., text is TXT, images are PNG, and videos are MP4) and metadata (upload time, uploader, source description, etc.).

[0097] Next, the sample data is standardized. The data format is converted according to preset rules: text is uniformly converted to UTF-8 encoded TXT; images are uniformly converted to JPG format with a resolution adjusted to 224×224; audio is converted to WAV format with a sampling rate set to 16kHz; video frames are extracted into image sequences (e.g., one frame every 5 seconds or one frame every 10 seconds), and then processed according to image standards.

[0098] Preprocessing: Noise reduction is performed on the standardized data. For example, Gaussian noise is removed from images using OpenCV (Open Source Computer Vision Library), silent segments are cropped from audio using the audio analysis and processing library Librosa, and specified symbols, such as "@" and "#", are removed from text using regular expressions.

[0099] Basic feature extraction. Extracting basic features from the data: For images, calculate color histograms and edge features; for audio, extract Mel-frequency coefficients; for text, perform word segmentation. For example, for Chinese text, use a Chinese word segmentation tool (such as jieba); for English text, use an English word segmentation tool like NLTK (Natural Language Toolkit) to segment words and generate bag-of-words vectors.

[0100] Based on the above exemplary description, it can be seen that before feature extraction is performed on the first modality data included in the retrieval request, in response to the modality category of the first modality data being text, the first modality data is converted into text data in UTF-8 encoded TXT format. Alternatively, in response to the modality category of the first modality data being an image or video frame, the video frame is processed by frame extraction to obtain an image, and the image is converted into a uniform image format with a uniform resolution. Alternatively, in response to the modality category of the first modality data being audio, the first modality data is converted into a uniform audio format with a uniform sampling rate. Next, the image data with a uniform image format, the audio data with a uniform audio format, or the text data in TXT format are preprocessed to obtain preprocessed first modality data. The specific preprocessing method can be to remove Gaussian noise from the image data with a uniform image format using OpenCV; or to crop silent segments from the audio data with a uniform audio format using Librosa; or to remove specified symbols (such as "@", "#", etc.) from the text data in TXT format using regular expressions. Then, basic feature extraction is performed on the preprocessed first modality data to obtain initial features. The specific methods for basic feature extraction can be as follows: calculate the color histogram and edge features of the image data after removing Gaussian noise to obtain the initial image features; or extract the Mel spectrum coefficients of the audio data after cropping the silent segments to obtain the initial audio features; or segment the text data after removing specified symbols and generate bag-of-words vectors, and use the bag-of-words vectors as the initial text features.

[0101] Step 302: Perform deep feature extraction, semantic mapping, and hierarchical feature decomposition on the sample data to obtain candidate semantic features.

[0102] Specifically, the initial features extracted in step 301 (the initial features corresponding to the first modality data, such as initial image features, initial audio features, or initial text features) undergo deep feature extraction, semantic mapping, and hierarchical feature splitting to obtain candidate semantic features. For the specific steps of deep feature extraction, semantic mapping, and hierarchical feature splitting of the sample data, please refer to the steps for feature extraction, semantic mapping, and hierarchical feature splitting of the first modality data; these will not be elaborated upon here.

[0103] Optionally, access control and association management can also be implemented. Based on user settings, sample data, original files, and obtained candidate semantic features are categorized into platform datasets, public datasets, or personal datasets. Alternatively, they can be further subdivided into multiple categories based on platform datasets, public datasets, and personal datasets. Specific subdivisions can be based on modality, resulting in broad dataset categories for different modalities, and then further subdivided based on data content to obtain subcategories. Platform datasets require access permissions, public datasets are visible to the public, and personal datasets are visible only to the user. Users can manually add cross-modal association labels for each dataset, such as associating the semantic label "kitchen fire" with a fire image; these associations are stored in an association database.

[0104] Optionally, an index library can be constructed before or after step 301. For example, a multimodal vector index library can be constructed using the FAISS (Facebook AI Similarity Search, a high-dimensional vector similarity search library): text vectors, image vectors, audio vectors, and video vectors are stored in their respective sub-index libraries (original logic), and the text vectors, image vectors, audio vectors, and video vectors are the candidate semantic features of different modalities.

[0105] Optionally, the heat-aware index update unit, through GPU (Graphics Processing Unit) virtualization technology, achieves parallel updates while simultaneously counting high-frequency retrieval modal combinations (such as "text → video" and "image → text"). It prioritizes updating and compressing the index sub-databases involved in high-frequency combinations (to improve response speed), and archives low-frequency indexes that have not been retrieved for a long time to save storage resources. Specifically, based on multiple retrieval requests, it counts the number of various combinations of the index modal and target modal requested by the user; combinations exceeding a preset threshold are identified as high-frequency combinations; combinations less than or equal to the preset threshold are identified as low-frequency combinations; high-frequency combinations are prioritized for updating and / or compressing their corresponding index information; and low-frequency combinations are archived.

[0106] Step 303: In response to receiving a retrieval request, identify the query modality of the retrieval request and the target modality to be retrieved.

[0107] Step 303 is the input parsing and modality recognition process. The system receives the retrieval data input by the user and identifies the modality type (text / image / audio / video) of the query modality and the target modality by means of file header information (such as “image / jpeg” to identify the image) or content features (such as text containing continuous characters, images containing pixel matrices).

[0108] For example, if a user needs to retrieve an image from a text search, then the text is the query modality and the image is the target modality.

[0109] Step 304: Extract features from the first modality data included in the retrieval request to obtain the first semantic vector.

[0110] It should be noted that before step 304, the first modality data needs to undergo the standardization, preprocessing, and basic feature extraction processes described in step 301 to obtain the initial features corresponding to the first modality data.

[0111] For example, when performing complex modal segmentation, if the first modal data input is video or long audio, segment processing is performed: for video, 1 frame is extracted every 5 seconds, and up to 10 frames are retained (if more are extracted, they are uniformly sampled); for audio, segments are divided into segments every 3 seconds, and up to 5 segments are retained.

[0112] Step 304 involves deep feature extraction from the initial features of the first modality data. A pre-trained feature extraction model (such as CLIP) is invoked to convert the initial features of the first modality data into a high-dimensional semantic vector (the first semantic vector of the original logic). For example, step 304 could specifically involve calling the interface of the retrieval engine's feature extraction model to obtain the semantic vector of the first modality data (the original logic).

[0113] Step 305: Map the first semantic vector to a unified semantic space through a pre-trained mapping matrix to obtain the second semantic vector.

[0114] Step 305 is the process of cross-modal semantic mapping, which transforms the first semantic vector into a unified semantic space through the mapping matrix of the retrieval engine, ensuring semantic alignment and dimensional consistency with the target modal vector.

[0115] The mapping matrix is ​​pre-trained. For example, it can be optimized through contrastive learning as follows: Different modalities of the same scene, such as the text "fire truck" and an image of a fire truck, are treated as positive sample pairs. The vector distance is calculated and minimized within the positive sample pairs. Conversely, different scene data, such as the text "fire truck" and an image of a cat, are treated as negative sample pairs. The vector distance between the negative sample pairs is maximized, ultimately making cross-modal vectors comparable in a unified semantic space (original logic, also supporting cross-modal alignment of hierarchical features). Specifically, a multimodal cross-type retrieval module can parse the input data and identify the modality, extract deep features, and semantically map them to a unified semantic space.

[0116] Based on the above exemplary description, the specific training method for the mapping matrix is ​​as follows: Training samples are obtained; the training samples include positive sample pairs and negative sample pairs; positive sample pairs include samples from the first modality (e.g., the text "fire truck") and the second modality (e.g., images of fire trucks) that are semantically aligned (semantically matched) within the same scene; negative sample pairs include samples from the third modality (e.g., the text "fire truck") and the fourth modality (e.g., images of cats) that are semantically misaligned (semantically mismatched) within different scenes. Using the training samples, with the training objective of minimizing the distance between two samples in a positive sample pair and maximizing the distance between two samples in a negative sample pair, the initial mapping matrix is ​​trained, optimizing the trainable parameters in the mapping matrix until the convergence condition is met, thus obtaining a pre-trained mapping matrix.

[0117] Step 306: Call the hierarchical feature encoding model to extract features from the second semantic vector in layers.

[0118] For example, features are extracted hierarchically from the second semantic vector to obtain a three-level feature vector of "global-local-attribute".

[0119] For example, if the modality category of the first modality data is text: extract "document-level vectors" (overall semantics), "paragraph-level vectors" (core viewpoints), and "entity-level vectors" (such as key entities like "signal" and "switches"); If the modality category of the first modality data is an image / video frame: extract the "global vector" (overall scene), "region vector" (such as a specific device region), and "attribute vector" (such as color, shape, and state); If the modality category of the first modality data is audio: extract "segment-level vectors" (overall sound features), "event vectors" (such as "horn" and "fault alarm"), and "attribute vectors" (such as frequency and intensity).

[0120] Based on the above example, we can obtain the first semantic features corresponding to the global, local, and attribute levels respectively through the hierarchical feature encoding model. Specifically, this can be achieved in the following way: If the modality category of the first modality data is text, that is, in response to the recognition that the modality category of the first modality data is text, the first-level feature encoding model is called. The second semantic vector is split into global-level overall semantic features (e.g., document-level vector), local-level core viewpoint features (e.g., paragraph-level vector), and attribute-level entity features (e.g., key entity features such as "signal" and "switcher").

[0121] In response to the identification that the modality category of the first modality data is an image or video frame, a second-level feature encoding model is invoked. This model decomposes the second semantic vector into global-level overall scene features (e.g., a global vector), local-level device region features (e.g., a region vector corresponding to a specific device region, which can be obtained through image object detection or image segmentation), and attribute-level first attribute features (e.g., color, shape, state, etc.). The first attribute features represent at least one of the color, shape, or state included in the first modality data.

[0122] In response to the identification that the modality category of the first modality data is audio, the third-level feature encoding model is invoked. This model decomposes the second semantic vector into global-level overall sound features (e.g., segment-level vectors), local-level event features corresponding to the sound type (e.g., event features could be "horn," "fault alarm," etc.), and attribute-level first attribute features (e.g., frequency, intensity). The first attribute features represent at least one of the frequency and intensity corresponding to the first modality data.

[0123] Specifically, hierarchical coding models can be implemented based on Transformer and / or CNN (Convolutional Neural Network).

[0124] Specifically, for text-modal data, the Transformer model can be used to implement hierarchical feature extraction. For example, the first-level feature encoding model includes a first Transformer module (or Transformer layer), a second Transformer module, and a third Transformer module. The first Transformer module is used to extract overall semantic features from the input second semantic vector; the second Transformer module is used to extract core viewpoint features from the input second semantic vector; and the third Transformer module is used to extract entity features from the input second semantic vector.

[0125] A Transformer module can include partial layers or the complete Transformer model. For example, a Transformer module can include an input layer, an encoder, a decoder, and an output layer. The encoder can be the encoder in the Transformer model, and the decoder is the decoder in the Transformer model.

[0126] For data whose modality category is image or video frame, hierarchical feature extraction can be achieved using CNN or VIT-Transformer (VisionTransformer). For example, the aforementioned second-level feature encoding model includes a first encoding module, a second encoding module, and a third encoding module; the first encoding module is used to extract overall scene features from the input second semantic vector; the second Transformer module is used to extract device region features from the input second semantic vector; and the third Transformer module is used to extract at least one of color, shape, or state from the input second semantic vector. The first, second, or third encoding module is a convolutional neural network (CNN) or a vision model based on the Transformer architecture, VIT-Transformer.

[0127] For data with an audio modality, a Transformer can be used to implement hierarchical feature extraction. For example, a third-level feature encoding model includes a fourth, fifth, and sixth Transformer module; the fourth Transformer module extracts overall sound features from the input second semantic vector; the second Transformer module extracts event features corresponding to the sound type from the input second semantic vector; and the third Transformer module extracts at least one of frequency and intensity from the input second semantic vector. A Transformer module can include an input layer, an encoder and a decoder, and an output layer.

[0128] The hierarchical feature extraction in step 305 and the deep feature extraction process of the CLIP (Contrastive Language–Image Pre-training) model in step 304 can be run in parallel.

[0129] Step 307: Index the candidate semantic features corresponding to the target modality in the pre-built data center.

[0130] Candidate semantic features corresponding to the target modality and their corresponding original files can be stored in a data center. The data center stores datasets of different categories, such as personal datasets, platform datasets, and public datasets. These datasets are further subdivided into multiple major categories based on modality, and then further subdivided into multiple subcategories based on data content. For example, a major category might be public datasets, which includes datasets corresponding to multiple modalities. Video modality datasets could include specific subcategories such as "train door malfunction" video datasets and "firefighting" video datasets.

[0131] In this embodiment, each dataset has associated semantic tags. Upon receiving a retrieval request, semantic recognition can be performed on the request to determine the semantic tags corresponding to the target modality data to be retrieved. Based on the semantic tags, semantic tags with the same or similar semantics are searched in the data center. According to the association relationship between semantic tags and datasets recorded in the data center's association database, the dataset corresponding to the semantic tag is indexed. Candidate semantic features corresponding to the target modality are indexed in the pre-constructed data center; specifically, this can involve searching for candidate semantic features included in the dataset corresponding to the semantic tag. This indexing process can be implemented using the FAISS tool.

[0132] Based on the above example, it can be seen that the data center stores datasets of different categories; these datasets include datasets associated with different semantic labels. Before indexing candidate semantic features corresponding to the target modality in the pre-built data center, a multimodal index library can be constructed using the FAISS tool. Before indexing candidate semantic features corresponding to the target modality in the pre-built data center, the first semantic label corresponding to the first modality data can be identified. After identifying the semantic label corresponding to the current retrieval request, based on the first semantic label, the first target dataset associated with the first semantic label is indexed in the multimodal index library using the FAISS tool, and candidate semantic features corresponding to the target modality are indexed in the first target dataset using the FAISS tool. For example, if the semantic label is "car door malfunction," the ID of the dataset associated with the "car door malfunction" label can be queried in the association library.

[0133] FAISS, also known as a vector search engine, is an open-source library specifically designed for fast similarity searches of massive vectors.

[0134] Step 308: Calculate the overall similarity.

[0135] Calculating the overall similarity can specifically include: calculating the similarity between the first semantic feature and the candidate semantic feature at each level in different layers; obtaining the weights corresponding to each level in different layers; and obtaining the overall similarity based on the similarity and weights of each level.

[0136] Calculate the similarity between the first semantic feature and the candidate semantic feature at different levels. Specifically, the similarity can be calculated for the three levels of the first semantic feature (global, local, and attribute-level) and the three levels of the candidate semantic feature (global, local, and attribute-level).

[0137] For example, in some embodiments, calculating the similarity between the first semantic feature and the candidate semantic feature at different levels can specifically involve: calculating the first similarity between the first semantic feature and the candidate semantic feature at the global level; calculating the second similarity between the first semantic feature and the candidate semantic feature at the local level; and calculating the third similarity between the first semantic feature and the candidate semantic feature at the attribute level.

[0138] Initiate dynamic hierarchical attention matching: Introduce an attention mechanism to dynamically allocate weights at each level based on the category of modal combination. For example, when matching technical document text with device images, the dynamic weight allocation model automatically increases the weights of text entity level (attribute level) and image region level (local level) to 40%; when matching scene images with descriptive text, the weights of image global level and text chapter level (global level) are increased to 50%, achieving automatic weight allocation.

[0139] For example, in some embodiments, a pre-trained dynamic weight allocation model is invoked. The modality categories of the first modality data and the modality categories of the target modality are input into the dynamic weight allocation model, which then outputs the weights corresponding to each level in different layers. The dynamic weight allocation model is an attention-based model, comprising an input layer, an attention layer, and an output layer. The attention layer includes attention calculation and a query, key, and value matrix. The query, key, and value matrix is ​​the input sequence transformed through three different linear transformation layers (weight matrix W). Q W K W V )get.

[0140] Output the weighted overall similarity score, for example, Overall Similarity = First Similarity at the Global Level × Global Weight (First Weight) + Second Similarity at the Local Level × Local Weight (Second Weight) + Third Similarity at the Attribute Level × Attribute Weight (Third Weight).

[0141] Step 309: Select N candidate semantic features from multiple candidate semantic features corresponding to the target modality according to the comprehensive similarity.

[0142] Step 309 is a process of sorting and filtering the retrieval results. Sort in descending order of the comprehensive similarity, and filter out the invalid results with a comprehensive score < preset threshold (e.g., 0.3). For example, N = 20, and the first 20 preliminary results are retained. Here, N is the default quantity and can be overridden by the advanced retrieval parameters. That is, if the user sets the maximum return number in the retrieval parameters, the user-set maximum return number shall prevail. In addition, the metadata of the associated data center (such as information like data source, upload time, etc.) is associated.

[0143] The embodiments of the present invention provide a basic retrieval mode and an advanced retrieval mode.

[0144] If it is detected that the user selects the basic retrieval mode and submits a retrieval request through the basic retrieval entry, then after the basic retrieval function module (the backend module corresponding to the basic retrieval entry) receives the multi-modal data input by the user (such as text keywords, image files), it directly calls the core logic of the multi-modal cross-type retrieval module to quickly complete feature extraction, similarity calculation, and result sorting, and returns the preliminary results to meet the user's requirement for retrieval speed. The specific implementation is as follows: Receive the original data input by the user. For example, receive the text keyword "gas tank explosion", image file, video clip, etc. input by the user, trigger the retrieval process directly without parameter configuration.

[0145] Next, perform input simplification processing. Lightweight process the input data: for example, retain the core keywords in the text, such as removing function words like "in"; compress the image to retain the core visual features, such as compressing it to 100×100 pixels. Perform frame extraction on the video, such as extracting 3 frames (reducing the processing amount), and at the same time simplify the hierarchical feature extraction, such as preferentially retaining the global level and the core local level features, and omitting some attribute level features to improve the speed.

[0146] After the above simplification processing, call the core retrieval process in the multi-modal retrieval module. Pass the simplified input data to the multi-modal cross-type retrieval module, execute steps 304 - step 309, reuse the dynamic hierarchical attention matching logic, simplify the weight calculation, and obtain the original file information corresponding to the first 20 semantic candidate features as the retrieval results.

[0147] Next, the search results are organized and displayed. For example, lightweight preview information is extracted: if the original file of the target modality is an image / video, a thumbnail of the image / video is displayed; if the original file of the target modality is text, the first 50 words are displayed; if the original file of the target modality is audio, the waveform of the audio is displayed. Furthermore, metadata (source dataset, upload time, etc.) corresponding to the search results in the data center can be associated and displayed in a list format, supporting sorting by "overall similarity" or "upload time".

[0148] Based on the above exemplary description, it can be seen that in some embodiments, when the retrieval request is submitted by a user through the basic retrieval entry point, in order to improve the response speed, the first modality data can be first subjected to lightweight processing. Lightweight processing includes: removing function words and retaining keywords from the first modality data whose modality category is text; and performing image compression or frame extraction processing on the first modality data whose modality category is image or video frame. Next, to further improve the response speed, the second semantic vector can be split into first semantic features corresponding to the global level and local level through a hierarchical feature encoding model, that is, only retaining semantic features at two levels to simplify the computation. The first similarity between the first semantic feature and the candidate semantic features at the global level and the second similarity at the local level are calculated. That is, the similarity calculation is simplified to only two levels. The dynamic weight allocation is also simplified by calling a pre-trained dynamic weight allocation model. The modality category of the first modality data and the modality category of the target modality are input into the dynamic weight allocation model. The dynamic weight allocation model outputs the first weight corresponding to the global level and the second weight corresponding to the local level. In this way, the user's basic search request can be responded to with the fastest response speed through the above lightweight processing and simplified process.

[0149] If a user selects the advanced search mode and submits a search request through the advanced search entry point, the advanced search function module (the backend module corresponding to the advanced search entry point) performs a secondary filter on the original results returned by the core search module based on the user-set confidence threshold, maximum number of returns, time range, etc., removing low-match results and retaining high-precision results that meet the conditions, and generating a report including parameter impact analysis. The specific implementation is as follows: The system receives user-configured search parameters, including: confidence threshold (0.5-1.0), maximum number of returns (1-100), time range (start time - end time), and at least one of the following source datasets: The source dataset is optional; for example, multiple candidate source datasets can be displayed on the interactive interface for the user to choose from, such as when the user selects "security incident dataset".

[0150] Next, the search parameters are validated and transformed. The validity of the search parameters is validated: if the confidence level exceeds the range, it is automatically adjusted to 0.8; if the maximum number of returns exceeds the range, it is adjusted to 20; if the time range is invalid, it is left unrestricted by default. The parameters are then converted into filtering rules (e.g., "Overall similarity ≥ 0.8 and upload time between 2025-01-01 and 2025-06-31, and source is a security incident dataset").

[0151] Next, the core retrieval function is invoked. The user input data and retrieval parameters are passed to the multimodal cross-type retrieval module to obtain the original file information corresponding to the top 50 candidate semantic features as preliminary retrieval results (expanding the scope to reserve filtering space).

[0152] A second filtering process is performed based on the initial search results. The initial search results are filtered according to the transformed rules: results with a comprehensive similarity lower than the confidence threshold are removed; results uploaded within the specified time range are retained; if a source dataset is specified, only data from that dataset is retained. Results are sorted in descending order of comprehensive similarity, and the top M results are selected (M = maximum number of results returned).

[0153] Results optimization and report generation. The generated parameters affect the report, including the number of original results (N), the number of filtered results (M), and the percentage of results with high overall similarity (≥0.8), helping users optimize their search parameters.

[0154] Based on the above example, in advanced search mode, a user submits a search request through the advanced search portal. The search request includes search parameters, which include at least one of the following: confidence threshold, maximum number of results returned, and time range. When filtering search results, based on the search parameters, M search results can be selected from N search results if their confidence level is greater than or equal to the confidence threshold and / or their upload time conforms to the time range; M is less than or equal to the maximum number of results returned.

[0155] After the above steps, the search results are obtained and then further passed to the search result management and reuse module. Upon receiving the results, the search result management and reuse module supports exporting metadata and original files, associating them with the dataset and labeling them, and saving historical records for reuse. The specific implementation is as follows: The search results management and reuse module receives the search results generated by the multimodal search module. The multimodal search module can also integrate a basic search module and / or an advanced search module. Figure 2 (Not shown in the image) The search results management and reuse module receives the final search results returned by the basic search or advanced search module, including search result data, comprehensive similarity, source dataset and other information; at the same time, it starts the feedback learning unit to prepare to collect user feedback data.

[0156] The search result management and reuse module performs export processing, and this embodiment of the invention supports two export methods: Export Method 1: Metadata Export: The search results' ID, overall similarity, source dataset, upload time, and other information are generated into an Excel spreadsheet. The data center's file storage service can then link to the download address of the original file based on the search result's ID.

[0157] Export Method 2: Packaging of Original Files: Compress the original data (images, videos, etc.) corresponding to the results into a ZIP package, and use the high-performance storage (JuiceFS) of the MaaS platform to support breakpoint resume, while verifying user permissions (only allowing the export of datasets that have permission to access).

[0158] Based on the above exemplary description, in some embodiments, after obtaining the search results, the metadata corresponding to the search results can be exported. The metadata includes at least one of the following: original file ID, overall similarity, the dataset name or ID from which the original file originated, upload time, uploader, and source description. The original file ID is used to link to the corresponding original file for users to download or view. And / or, it can support exporting N original files or a compressed package corresponding to N original files from the search results.

[0159] Next, dataset association is performed. Upon receiving a user's "batch add to dataset" request, an association instruction is sent to the data center: specifying the target dataset (e.g., "fire safety dataset"), the data center updates the index of that dataset to include the results; batch addition of semantic tags (e.g., "gas cylinder accident") is supported, and the tag information is synchronized to the data center's tag library. In other words, after obtaining the search results, the system receives the second semantic tags assigned by the user to the search results, records the association between the second semantic tags and the search results, and stores the association in the data center.

[0160] Historical record storage and reuse: Automatically records search history, including input data snapshots, parameter configurations (for advanced searches), result ID list, search time, and overall similarity distribution; stored in a time-series database, supporting retrieval of historical records by time and modality. When a user triggers "re-execution," the historical parameters are directly reused to call the search interface; when "modify search" is triggered, an edit template is generated based on the historical parameters, which the user adjusts before re-execution.

[0161] Feedback learning is performed based on feedback learning units: Feedback data collection: Users are allowed to label search results as "strongly relevant / weakly relevant / irrelevant" through the interface, and secondary filtering behavior of users on the results is recorded (such as prioritizing clicking on a certain type of data from the returned results). Feature importance update: Based on the feedback data, the weights of features at each modality level are adjusted using online learning algorithms (such as online updates of random forests) (e.g., if a user clicks on "text-image pairs containing specific entities" multiple times, the weights of entity-level and region-level features are increased). In addition, a threshold adaptive adjustment mechanism can be adopted: based on the user's preference for "precision-recall" (e.g., if a high confidence threshold is used multiple times, the system automatically increases the default threshold).

[0162] In summary, the MaaS platform, based on the collaboration of multiple parts such as data center, search engine, multimodal cross-type search module (including basic search module and advanced search module), and search result management and reuse, achieves closed-loop management and efficient retrieval of the entire process from multimodal data input to result application through the mechanisms of "multimodal hierarchical feature fusion" and "dynamic evolution of search strategy".

[0163] The standardized processing of data centers and retrieval engines, hierarchical feature extraction, and dynamic index building solve the problems of "single vectors being unable to capture hierarchical semantics" and "low index update efficiency" in traditional cross-modal retrieval. The complex steps are encapsulated into module interfaces, improving the versatility of the solution.

[0164] The dynamic hierarchical attention matching mechanism of the multimodal cross-type retrieval module replaces the single cosine similarity calculation. It solves the problem of "global similarity but key local mismatch" by "hierarchical matching and key focus" based on the attention mechanism. The matching quality is significantly improved compared with the retrieval schemes in related technologies.

[0165] In addition, the basic search module achieves a response time in seconds through "simplified hierarchical features + fast call", while the advanced search module combines dynamic comprehensive similarity for secondary filtering, balancing efficiency and accuracy to adapt to different scenario needs.

[0166] Furthermore, the feedback learning unit of the retrieval results management and reuse module enables adaptive optimization of the retrieval strategy: adjusting feature weights and thresholds based on user behavior to avoid long-term accuracy stagnation; while the popularity-aware index update unit improves system response speed and reduces storage costs through high-frequency optimization.

[0167] The loose coupling design between the above modules ensures compatibility with the Maas platform architecture, supports functional expansion (such as integration with industry knowledge bases), and is suitable for various scenarios such as industrial quality inspection, medical imaging, and intelligent security.

[0168] This invention also provides a cross-modal retrieval system, such as... Figure 4 As shown, the system includes at least a data center 401, a search engine 402, and a multimodal search module 403.

[0169] The multimodal retrieval module 403 is used to identify the target modality to be retrieved in response to a received retrieval request.

[0170] The retrieval engine 402 includes a feature extraction model, a semantic alignment unit, a hierarchical feature encoding model, and an indexing unit. The feature extraction unit extracts features from the first modality data included in the retrieval request to obtain a first semantic vector; the first modality data may or may not be the same as the target modality. The semantic alignment unit maps the first semantic vector to a unified semantic space using a pre-trained mapping matrix to obtain a second semantic vector; the mapping matrix is ​​used for semantic alignment of semantic vectors from different modalities.

[0171] The retrieval engine 402 is used to input the second semantic vector into a pre-trained hierarchical feature encoding model, which then splits the second semantic vector into first semantic features at different levels.

[0172] Indexing units are used to index candidate semantic features corresponding to the target modality in a pre-built data center 401.

[0173] The multimodal retrieval module 403 includes a similarity calculation unit, which is used to calculate the similarity between the first semantic feature and the candidate semantic feature at different levels, and to obtain the weights corresponding to each level in different levels. Based on the similarity and weights of each level, a comprehensive similarity is obtained.

[0174] The multimodal retrieval module 403 is used to select N candidate semantic features from multiple candidate semantic features corresponding to the target modality based on comprehensive similarity, and use the N original files corresponding to the N candidate semantic features as the retrieval results.

[0175] In some embodiments, different levels include global, local, and attribute levels. The retrieval engine 402 is specifically configured to: decompose the second semantic vector into first semantic features corresponding to the global, local, and attribute levels respectively using a hierarchical feature encoding model.

[0176] In some embodiments, the retrieval engine 402 is specifically configured to: in response to identifying the modality category of the first modal data as text, invoke a first-level feature encoding model to decompose the second semantic vector into global semantic features corresponding to the global level, core viewpoint features corresponding to the local level, and entity features corresponding to the attribute level; or, in response to identifying the modality category of the first modal data as an image or video frame, invoke a second-level feature encoding model to decompose the second semantic vector into global scene features corresponding to the global level, device region features corresponding to the local level, and first attribute features corresponding to the attribute level; the first attribute features are used to represent at least one of color, shape, or state included in the first modal data; or, in response to identifying the modality category of the first modal data as audio, invoke a third-level feature encoding model to decompose the second semantic vector into global sound features corresponding to the global level, event features corresponding to sound types corresponding to the local level, and first attribute features corresponding to the attribute level; the first attribute features are used to represent at least one of frequency and intensity corresponding to the first modal data.

[0177] In some embodiments, the multimodal retrieval module 403 includes a dynamic weight allocation model; the multimodal retrieval module is specifically used to: identify the modality category of the first modality data; call the pre-trained dynamic weight allocation model, input the modality category of the first modality data and the modality category of the target modality into the dynamic weight allocation model, and output the weights corresponding to each level in different levels through the dynamic weight allocation model; wherein, the dynamic weight allocation model is a model using an attention mechanism, and the dynamic weight allocation model includes an input layer, an attention layer and an output layer.

[0178] Furthermore, in some embodiments, combined with Figure 2 The system architecture shown in this embodiment of the invention, in the cross-modal retrieval system, the similarity calculation unit is specifically used for: calculating the first similarity between the first semantic feature and the candidate semantic feature at the global level; calculating the second similarity between the first semantic feature and the candidate semantic feature at the local level; and calculating the third similarity between the first semantic feature and the candidate semantic feature at the attribute level.

[0179] In some embodiments, the retrieval engine 402 is specifically configured to: in response to identifying the modality category of the first modal data as text, invoke a first-level feature encoding model to decompose the second semantic vector into global semantic features corresponding to the global level, core viewpoint features corresponding to the local level, and entity features corresponding to the attribute level; or, in response to identifying the modality category of the first modal data as an image or video frame, invoke a second-level feature encoding model to decompose the second semantic vector into global scene features corresponding to the global level, device region features corresponding to the local level, and first attribute features corresponding to the attribute level; the first attribute features are used to represent at least one of color, shape, or state included in the first modal data; or, in response to identifying the modality category of the first modal data as audio, invoke a third-level feature encoding model to decompose the second semantic vector into global sound features corresponding to the global level, event features corresponding to sound types corresponding to the local level, and first attribute features corresponding to the attribute level; the first attribute features are used to represent at least one of frequency and intensity corresponding to the first modal data.

[0180] In some embodiments, the multimodal retrieval module 403 is further configured to: identify the modality category of the first modality data; the similarity calculation unit is further configured to: call a pre-trained dynamic weight allocation model, input the modality category of the first modality data and the modality category of the target modality into the dynamic weight allocation model, and output the weights corresponding to each level in different levels through the dynamic weight allocation model; wherein, the dynamic weight allocation model is a model using an attention mechanism, and the dynamic weight allocation model includes an input layer, an attention layer and an output layer.

[0181] In some embodiments, the system further includes a basic search entry point, where the search request is a search request submitted by the user through the basic search entry point. The multimodal search module also includes a basic search module corresponding to the basic search entry point. Figure 2(Not shown in the image) The basic retrieval module is used to perform lightweight processing on the first modality data. Lightweight processing includes: removing function words and retaining keywords from the first modality data whose modality category is text; and performing image compression or frame extraction on the first modality data whose modality category is image or video frames. The basic retrieval module is also used to trigger the hierarchical encoding model in the retrieval engine to perform hierarchical feature encoding as follows: the second semantic vector is split into first semantic features corresponding to the global level and local level through the hierarchical feature encoding model. The basic retrieval module is also used to trigger the similarity calculation unit to calculate hierarchical similarity as follows: the first similarity between the first semantic feature and the candidate semantic feature at the global level and the second similarity at the local level are calculated. The basic retrieval module is also used to trigger the dynamic weight allocation unit to perform weight allocation as follows: the pre-trained dynamic weight allocation model is called, the modality category of the first modality data and the modality category of the target modality are input into the dynamic weight allocation model, and the dynamic weight allocation model outputs the first weight corresponding to the global level and the second weight corresponding to the local level.

[0182] In some embodiments, the system further includes an advanced search entry point, where the search request is a search request submitted by the user through the advanced search entry point. The multimodal search module also includes an advanced search module corresponding to the advanced search entry point. Figure 2 (not shown in the image), the search request includes search parameters, which include at least one of the following: confidence threshold, maximum number of returns, and time range.

[0183] The system may also include, for example Figure 2 The search result management and reuse module shown is used to filter M search results from N search results based on search parameters, where the confidence level is greater than or equal to the confidence threshold and / or the upload time meets the time range; M is less than or equal to the maximum number of returns.

[0184] The search result management and reuse module is also used to: export the metadata corresponding to the search results; the metadata includes at least one of the following: original file ID, comprehensive similarity, dataset name or ID from which the original file comes, upload time, uploader, and source description; the original file ID is used to link to the corresponding original file for users to download or view; and / or, export N original files or a compressed package corresponding to N original files from the search results.

[0185] In some embodiments, the search result management and reuse module is further configured to: receive second semantic tags marked by users on search results, record the association between the second semantic tags and the search results, and store the association in the data center.

[0186] In some embodiments, the retrieval engine 402 further includes a heat-aware index update unit, which is used to count the number of various combinations of index modalities and target modalities requested by the user based on multiple retrieval requests; determine combinations with a number exceeding a preset threshold as high-frequency combinations; determine combinations with a number less than or equal to the preset threshold as low-frequency combinations; prioritize updating and / or compressing the corresponding index information for high-frequency combinations; and archive low-frequency combinations.

[0187] The data center may include a standardization unit for converting the first modal data into text data in UTF-8 encoded TXT format in response to the modality category of the first modal data being text; for extracting video frames to obtain images in response to the modality category of the first modal data being image or video frames, and converting the images into a uniform image format with uniform resolution; and for converting the first modal data into a uniform audio format with uniform sampling rate in response to the modality category of the first modal data being audio.

[0188] The data center or retrieval engine may also include a preprocessing unit and a basic feature extraction unit. The preprocessing unit handles image data with a uniform image format, audio data with a uniform audio format, or text data in TXT format to obtain preprocessed first modality data. The basic feature extraction unit extracts basic features from the preprocessed first modality data to obtain initial features. Specifically, the feature extraction model in the retrieval engine is used to extract features corresponding to the initial features included in the retrieval request.

[0189] The preprocessing unit is specifically used to: remove Gaussian noise from image data with a uniform image format using OpenCV; or to crop silent segments from audio data with a uniform audio format using Librosa; or to remove specified symbols from TXT format text data using regular expressions.

[0190] The basic feature extraction unit is specifically used for: calculating color histograms and edge features on image data after removing Gaussian noise to obtain initial image features; or extracting Mel-spectrum coefficients from audio data after cropping silent segments to obtain initial audio features; or segmenting text data after removing specified symbols and generating bag-of-words vectors, using the bag-of-words vectors as initial text features.

[0191] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus. The processor 510 can call logical instructions in the memory 530 to execute the following methods: In response to a received retrieval request, the system identifies the target modality to be retrieved; it extracts features from the first modality data included in the retrieval request to obtain a first semantic vector; the first modality data and the target modality have different or the same modality category; it maps the first semantic vector to a unified semantic space through a pre-trained mapping matrix to obtain a second semantic vector; the mapping matrix is ​​used to semantically align the semantic vectors of different modalities; it inputs the second semantic vector into a pre-trained hierarchical feature encoding model, which splits the second semantic vector into first semantic features at different levels; it indexes candidate semantic features corresponding to the target modality in a pre-constructed data center; it calculates the similarity between the first semantic features and candidate semantic features at each level in different layers; it obtains the weights corresponding to each level in different layers; based on the similarity and weights of each level, it obtains a comprehensive similarity; based on the comprehensive similarity, it selects N candidate semantic features from multiple candidate semantic features corresponding to the target modality, and uses the N original files corresponding to the N candidate semantic features as the retrieval results.

[0192] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0193] This invention discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments, such as including: In response to a received retrieval request, the system identifies the target modality to be retrieved; it extracts features from the first modality data included in the retrieval request to obtain a first semantic vector; the first modality data and the target modality have different or the same modality category; it maps the first semantic vector to a unified semantic space through a pre-trained mapping matrix to obtain a second semantic vector; the mapping matrix is ​​used to semantically align the semantic vectors of different modalities; it inputs the second semantic vector into a pre-trained hierarchical feature encoding model, which splits the second semantic vector into first semantic features at different levels; it indexes candidate semantic features corresponding to the target modality in a pre-constructed data center; it calculates the similarity between the first semantic features and candidate semantic features at each level in different layers; it obtains the weights corresponding to each level in different layers; based on the similarity and weights of each level, it obtains a comprehensive similarity; based on the comprehensive similarity, it selects N candidate semantic features from multiple candidate semantic features corresponding to the target modality, and uses the N original files corresponding to the N candidate semantic features as the retrieval results.

[0194] On the other hand, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the transmission methods provided in the above embodiments, including, for example: In response to a received retrieval request, the system identifies the target modality to be retrieved; it extracts features from the first modality data included in the retrieval request to obtain a first semantic vector; the first modality data and the target modality have different or the same modality category; it maps the first semantic vector to a unified semantic space through a pre-trained mapping matrix to obtain a second semantic vector; the mapping matrix is ​​used to semantically align the semantic vectors of different modalities; it inputs the second semantic vector into a pre-trained hierarchical feature encoding model, which splits the second semantic vector into first semantic features at different levels; it indexes candidate semantic features corresponding to the target modality in a pre-constructed data center; it calculates the similarity between the first semantic features and candidate semantic features at each level in different layers; it obtains the weights corresponding to each level in different layers; based on the similarity and weights of each level, it obtains a comprehensive similarity; based on the comprehensive similarity, it selects N candidate semantic features from multiple candidate semantic features corresponding to the target modality, and uses the N original files corresponding to the N candidate semantic features as the retrieval results.

[0195] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0196] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. Such computer software products can be stored in computer-readable storage media, such as ROM / RAM, magnetic disks, optical disks, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal retrieval method, characterized in that, The method includes: In response to receiving a retrieval request, identify the target modality that the retrieval request needs to retrieve; Feature extraction is performed on the first modality data included in the retrieval request to obtain a first semantic vector; the modality category of the first modality data is different from or the same as that of the target modality. The first semantic vector is mapped to a unified semantic space through a pre-trained mapping matrix to obtain the second semantic vector; the mapping matrix is ​​used to perform semantic alignment on semantic vectors of different modalities. The second semantic vector is input into a pre-trained hierarchical feature encoding model, which then splits the second semantic vector into first semantic features at different levels. Index the candidate semantic features corresponding to the target modality in a pre-built data center; Calculate the similarity between the first semantic feature and the candidate semantic feature at each level in different layers; Obtain the weights corresponding to each level in the different levels; Based on the similarity of each level and the weights, a comprehensive similarity is obtained; Based on the comprehensive similarity, N candidate semantic features are selected from multiple candidate semantic features corresponding to the target modality, and the N original files corresponding to the N candidate semantic features are used as the retrieval results.

2. The method according to claim 1, characterized in that, The different levels include global, local, and attribute; The hierarchical feature encoding model decomposes the second semantic vector into first semantic features at different levels, including: The hierarchical feature encoding model is used to split the second semantic vector into first semantic features corresponding to three levels: global, local, and attribute.

3. The method according to claim 2, characterized in that, Calculating the similarity between the first semantic feature and the candidate semantic feature at different levels includes: Calculate the first similarity between the first semantic feature and the candidate semantic feature at the global level; Calculate the second similarity between the first semantic feature and the candidate semantic feature at the local level; Calculate the third similarity between the first semantic feature and the candidate semantic feature at the attribute level.

4. The method according to claim 2, characterized in that, The hierarchical feature encoding model decomposes the second semantic vector into first semantic features corresponding to three levels: global, local, and attribute. In response to the identification that the modality category of the first modality data is text, the first-level feature encoding model is invoked, and the second semantic vector is split into global semantic features corresponding to the global level, core viewpoint features corresponding to the local level, and entity features corresponding to the attribute level through the first-level feature encoding model; or, In response to the identification that the modal category of the first modal data is an image or video frame, a second-level feature encoding model is invoked. The second-level feature encoding model is used to split the second semantic vector into global scene features corresponding to the global level, device region features corresponding to the local level, and first attribute features corresponding to the attribute level. The first attribute features are used to represent at least one of the color, shape, or state included in the first modal data. or, In response to the identification that the modality category of the first modal data is audio, the third-level feature encoding model is invoked. The second semantic vector is split into global sound features corresponding to the global level, event features corresponding to the sound type corresponding to the local level, and first attribute features corresponding to the attribute level. The first attribute features are used to represent at least one of frequency and intensity corresponding to the first modal data.

5. The method according to claim 4, characterized in that, The first-level feature encoding model includes a first Transformer module, a second Transformer module, and a third Transformer module; the first Transformer module is used to extract the overall semantic features from the input second semantic vector; the second Transformer module is used to extract the core viewpoint features from the input second semantic vector; and the third Transformer module is used to extract the entity features from the input second semantic vector. The second-level feature encoding model includes a first encoding module, a second encoding module, and a third encoding module; the first encoding module is used to extract the overall scene features from the input second semantic vector; the second Transformer module is used to extract the device region features from the input second semantic vector; the third Transformer module is used to extract at least one of color, shape, or state from the input second semantic vector; wherein, the first encoding module, the second encoding module, or the third encoding module is a convolutional neural network (CNN) or a visual model based on the Transformer architecture, VIT-Transformer; The third-level feature encoding model includes a fourth Transformer module, a fifth Transformer module, and a sixth Transformer module; the fourth Transformer module is used to extract overall sound features from the input second semantic vector; the second Transformer module is used to extract event features corresponding to sound types from the input second semantic vector; and the third Transformer module is used to extract at least one of frequency and intensity from the input second semantic vector.

6. The method according to any one of claims 1-5, characterized in that, Before obtaining the weights corresponding to each level in the different levels, the method further includes: Identify the modality category of the first modality data; Obtaining the weights corresponding to each level in the different levels includes: The pre-trained dynamic weight allocation model is invoked, and the modality categories of the first modality data and the target modality are input into the dynamic weight allocation model. The dynamic weight allocation model outputs the weights corresponding to each level in the different levels. The dynamic weight allocation model is an attention mechanism model, which includes an input layer, an attention layer, and an output layer.

7. The method according to claim 1, characterized in that, Before mapping the first semantic vector to a unified semantic space using a pre-trained mapping matrix to obtain the second semantic vector, the method further includes: Obtain training samples; the training samples include positive sample pairs and negative sample pairs; the positive sample pairs include samples of the first modality and the second modality that are semantically aligned in the same scene; the negative sample pairs include samples of the third modality and the fourth modality that are semantically misaligned in different scenes; Using the training samples, with the training objectives of minimizing the distance between two samples in the positive sample pair and maximizing the distance between two samples in the negative sample pair, the initial mapping matrix is ​​trained to obtain the pre-trained mapping matrix.

8. The method according to claim 1, characterized in that, The search request is a search request submitted by the user through the basic search entry point; Before performing feature extraction on the first modal data included in the retrieval request, the method further includes: performing lightweight processing on the first modal data; the lightweight processing includes: removing function words and retaining keywords from the first modal data whose modal category is text; and performing image compression or frame extraction processing on the first modal data whose modal category is image or video frame. The hierarchical feature encoding model is used to split the second semantic vector into first semantic features at different levels, including: splitting the second semantic vector into first semantic features corresponding to the global level and the local level, respectively. Calculating the similarity between the first semantic feature and the candidate semantic feature at different levels includes: calculating the first similarity between the first semantic feature and the candidate semantic feature at the global level and the second similarity at the local level; Obtaining the weights corresponding to each level in the different levels includes: The pre-trained dynamic weight allocation model is invoked, and the modality categories of the first modality data and the target modality are input into the dynamic weight allocation model. The dynamic weight allocation model outputs the first weight corresponding to the global level and the second weight corresponding to the local level.

9. The method according to claim 1, characterized in that, The search request is a search request submitted by the user through the advanced search entry; the search request includes search parameters, which include at least one of the following: confidence threshold, maximum number of returns, and time range; After using the N original files corresponding to the N candidate semantic features as the retrieval results, the method further includes: Based on the search parameters, select M search results from the N search results that have a confidence level greater than or equal to the confidence threshold and / or whose upload time conforms to the time range; M is less than or equal to the maximum number of returns.

10. The method according to claim 1, characterized in that, After using the N original files corresponding to the N candidate semantic features as the retrieval results, the method further includes: Export the metadata corresponding to the search results; the metadata includes at least one of the following: original file ID, overall similarity, dataset name or ID from which the original file originated, upload time, uploader, and source description; the original file ID is used to link to the corresponding original file for users to download or view; And / or, Export the N original files or the compressed packages corresponding to the N original files from the search results.

11. The method according to claim 1, characterized in that, The candidate semantic features corresponding to the target modality and the original files corresponding to the candidate semantic features are stored in a data center; the data center stores datasets of different categories; the datasets of different categories include personal datasets, platform datasets and public datasets.

12. The method according to claim 1, characterized in that, The candidate semantic features corresponding to the target modality and the original files corresponding to the candidate semantic features are stored in the data center; the data center stores datasets of different categories; the datasets of different categories include datasets associated with different semantic labels; Before indexing the candidate semantic features corresponding to the target modality in a pre-built data center, the method further includes: Use the FAISS tool to build a multimodal index library; Before indexing the candidate semantic features corresponding to the target modality in a pre-built data center, the method further includes: Identify the first semantic label corresponding to the first modal data; Indexing candidate semantic features corresponding to the target modality in a pre-built data center includes: Based on the first semantic label, the FAISS tool is used to index the first target dataset associated with the first semantic label in the multimodal index library; The FAISS tool is used to index the candidate semantic features corresponding to the target modality in the first target dataset.

13. The method according to claim 12, characterized in that, After using the N original files corresponding to the N candidate semantic features as the retrieval results, the method further includes: The system receives a second semantic tag that the user marks the search results with, records the association between the second semantic tag and the search results, and stores the association in the data center.

14. The method according to claim 1, characterized in that, The method further includes: Based on multiple retrieval requests, count the number of various combinations of index modalities and target modalities requested by users; Combinations with a quantity exceeding a preset threshold are identified as high-frequency combinations; combinations with a quantity less than or equal to the preset threshold are identified as low-frequency combinations; the corresponding index information of the high-frequency combinations is updated and / or compressed first; and the low-frequency combinations are archived.

15. The method according to claim 1, characterized in that, Before performing feature extraction on the first modality data included in the retrieval request, the method further includes: In response to the fact that the modality category of the first modal data is text, the first modal data is converted into text data in TXT format encoded in UTF-8; In response to the modality category of the first modal data being an image or video frame, the video frame is processed by frame extraction to obtain an image, and the image is converted into a unified image format with a uniform resolution. In response to the fact that the modality category of the first modal data is audio, the first modal data is converted into a uniform audio format with a uniform sampling rate; Image data with a uniform image format, audio data with a uniform audio format, or text data in TXT format are preprocessed to obtain preprocessed first modal data. Basic features are extracted from the preprocessed first modality data to obtain initial features; Feature extraction of the first modality data included in the retrieval request includes: Feature extraction is performed on the initial features corresponding to the first modality data included in the retrieval request.

16. The method according to claim 15, characterized in that, Preprocessing of image data with a uniform image format, audio data with a uniform audio format, or text data in TXT format includes: Gaussian noise was removed from image data with a uniform image format using OpenCV. or, Librosa is used to trim silent segments from audio data with a uniform audio format. or, The specified symbols are removed from the TXT format text data using regular expressions.

17. The method according to claim 15, characterized in that, Basic features are extracted from the preprocessed first modality data to obtain initial features, including: Calculate the color histogram and edge features from the image data after removing Gaussian noise to obtain the initial image features; or, Mel spectrum coefficients are extracted from the audio data after the silent segments are removed to obtain the initial audio features; or, The text data with specified symbols removed is segmented into words and a bag-of-words vector is generated. The bag-of-words vector is then used as the initial text features.

18. A cross-modal retrieval system, characterized in that, The system includes a data center, a search engine, and a multimodal search module; The multimodal retrieval module is used to identify the target modality that the retrieval request needs to retrieve in response to receiving a retrieval request; The retrieval engine includes a feature extraction model, a semantic alignment unit, a hierarchical feature encoding model, and an indexing unit. The feature extraction unit is used to extract features from the first modality data included in the retrieval request to obtain a first semantic vector. The first modality data may or may not be the same as the modality category of the target modality. The semantic alignment unit is used to map the first semantic vector to a unified semantic space through a pre-trained mapping matrix to obtain a second semantic vector. The mapping matrix is ​​used to perform semantic alignment on the semantic vectors of different modalities. The retrieval engine is used to input the second semantic vector into a pre-trained hierarchical feature encoding model, and to split the second semantic vector into first semantic features at different levels through the hierarchical feature encoding model; The indexing unit is used to index the candidate semantic features corresponding to the target modality in a pre-built data center; The multimodal retrieval module includes a similarity calculation unit, which is used to calculate the similarity between the first semantic feature and the candidate semantic feature at different levels, and to obtain the weights corresponding to each level in the different levels. Based on the similarity of each level and the weights, a comprehensive similarity is obtained. The multimodal retrieval module is used to select N candidate semantic features from multiple candidate semantic features corresponding to the target modality based on the comprehensive similarity, and use the N original files corresponding to the N candidate semantic features as the retrieval results.

19. The system according to claim 18, characterized in that, The different levels include global, local, and attribute; The retrieval engine is specifically used to: decompose the second semantic vector into first semantic features corresponding to three levels: global, local, and attribute, through the hierarchical feature encoding model.

20. The system according to claim 19, characterized in that, The search engine is specifically used for: In response to the identification that the modality category of the first modality data is text, the first-level feature encoding model is invoked, and the second semantic vector is split into global semantic features corresponding to the global level, core viewpoint features corresponding to the local level, and entity features corresponding to the attribute level through the first-level feature encoding model; or, In response to the identification that the modal category of the first modal data is an image or video frame, a second-level feature encoding model is invoked. The second-level feature encoding model is used to split the second semantic vector into global scene features corresponding to the global level, device region features corresponding to the local level, and first attribute features corresponding to the attribute level. The first attribute features are used to represent at least one of the color, shape, or state included in the first modal data. or, In response to the identification that the modality category of the first modal data is audio, the third-level feature encoding model is invoked. The second semantic vector is split into global sound features corresponding to the global level, event features corresponding to the sound type corresponding to the local level, and first attribute features corresponding to the attribute level. The first attribute features are used to represent at least one of frequency and intensity corresponding to the first modal data.

21. The system according to any one of claims 18-20, characterized in that, The multimodal retrieval module includes a dynamic weight allocation model; the multimodal retrieval module is specifically used for: Identify the modality category of the first modality data; The pre-trained dynamic weight allocation model is invoked, and the modality categories of the first modality data and the target modality are input into the dynamic weight allocation model. The dynamic weight allocation model outputs the weights corresponding to each level in the different levels. The dynamic weight allocation model is an attention mechanism model, which includes an input layer, an attention layer, and an output layer.

22. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cross-modal retrieval method as described in any one of claims 1 to 17.

23. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the cross-modal retrieval method as described in any one of claims 1 to 17.

24. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the cross-modal retrieval method as described in any one of claims 1 to 17.

Images