Media information cross-modal retrieval method, system and medium based on semantic alignment

Abstract

The embodiments of the present application provide a method, system and medium for cross-modal retrieval of media information based on semantic alignment, the method comprising: obtaining a query request, generating user query data based on the query request, preprocessing the user query data, and obtaining semantically aligned request data in different modalities; extracting feature information of different modalities from user request data of different modalities; fusing multimodal features to obtain fused information, and generating a unified feature representation based on the fused information; establishing semantic associations between data of different modalities based on the feature representation to obtain associated information; analyzing the associated information based on natural language processing technology to generate query intent, obtain results of cross-modal retrieval and semantic understanding, and displaying the results of cross-modal retrieval and semantic understanding; by combining multiple modal information such as text, image, audio and video, it is possible to more comprehensively understand and express media content, thereby providing accurate search results.

Description

Technical Field

[0001] The present application relates to the field of intelligent retrieval technology, and in particular to a method, system and medium for cross-modal retrieval of media information based on semantic alignment. Background Art

[0002] With the rapid development of the Internet and information technology, the media industry is undergoing profound changes in digitalization, networking and globalization. The production, dissemination and consumption of media content are undergoing fundamental changes. The emergence of large amounts of text, image, audio and video data has brought unprecedented opportunities and challenges to the media industry. How to efficiently and accurately search and obtain accurate information in massive amounts of media data has become a key issue facing media practitioners.

[0003] Traditional text search technology relies on keyword matching and is mainly used for the retrieval of text data. However, this approach has limitations when dealing with multimedia content because it cannot effectively utilize non-text data such as images, audio, and video. Single-modal image / video search technology achieves search through feature extraction and matching of image or video content, but usually ignores the association with text or other modal data, resulting in insufficient semantic understanding of search results.

[0004] Metadata-based retrieval systems use the metadata of images or videos (such as title, description, tags, etc.) for search, but this approach relies on high-quality metadata and does not work well without sufficient metadata.

[0005] Content-based audio retrieval (CBIR) retrieves similar audio content by analyzing the features of audio signals, but it also suffers from the problem of insufficient understanding of contextual information.

[0006] Most existing technologies only focus on data of a single modality and are unable to comprehensively utilize information from multiple modalities, resulting in limited richness and accuracy of search results.

[0007] Insufficient semantic understanding: Many existing systems lack the deep semantic understanding of media content and find it difficult to accurately capture user query intent, especially in complex query scenarios.

[0008] Existing retrieval methods have technical barriers in the information conversion and association between different modalities, making it difficult to achieve efficient semantic matching. Existing technologies face performance bottlenecks when processing large-scale data, and the search response time is long, which cannot meet the needs of real-time retrieval. Existing annotation work is often time-consuming and labor-intensive, and it is difficult to cover all possible query scenarios. Effective technical solutions are urgently needed to address the above problems. Summary of the invention

[0009] The purpose of the embodiments of the present application is to provide a method, system and medium for cross-modal retrieval of media information based on semantic alignment, which can more comprehensively understand and express media content by combining multiple modal information such as text, images, audio and video, thereby providing richer and more accurate search results.

[0010] The embodiment of the present application also provides a media information cross-modal retrieval method based on semantic alignment, comprising: obtaining a query request, and generating user query data based on the query request;

[0011] Preprocess user query data to obtain semantically aligned request data in different modalities;

[0012] Extract feature information of different modalities from user request data of different modalities;

[0013] The multimodal features are fused to obtain fused information, and a unified feature representation is generated based on the fused information;

[0014] Establish semantic associations between data of different modalities based on feature representation to obtain association information;

[0015] Based on natural language processing technology, related information is analyzed to generate query intent, obtain results of cross-modal retrieval and semantic understanding, and display the results of cross-modal retrieval and semantic understanding.

[0016] Optionally, in the semantically aligned media information cross-modal retrieval method described in the embodiment of the present application, obtaining a query request, generating user query data based on the query request, and preprocessing the user query data to obtain semantically aligned request data under different modalities specifically includes:

[0017] Obtain query requests, obtain user query data, and analyze noise data and abnormal data of user query data;

[0018] Clean the noise data, analyze the abnormal state of the abnormal data, and determine whether the abnormal state is within the set abnormal range;

[0019] If it is in the abnormal range, the abnormal data will be cleaned and adjusted;

[0020] If it is not in the abnormal range, the abnormal data will be eliminated, and the eliminated data will be filled to obtain the user requested data.

[0021] Optionally, in the media information cross-modal retrieval method based on semantic alignment described in the embodiment of the present application, extracting feature information of different modalities from user request data of different modalities specifically includes:

[0022] Obtain user request data of different modes, extract data features, and obtain request features based on the semantics of data features;

[0023] Compare the requested features with the set features to obtain the feature deviation rate;

[0024] Determining whether the characteristic deviation rate is greater than or equal to a set characteristic deviation rate threshold;

[0025] If it is greater than or equal to, it is determined to be an invalid feature and the invalid feature is eliminated;

[0026] If it is less than , it is determined to be a key feature, and the key feature is modally classified based on the classification rules to obtain a multi-modal feature.

[0027] Optionally, in the media information cross-modal retrieval method based on semantic alignment described in the embodiment of the present application, multimodal features are fused to obtain fused information, and a unified feature representation is generated based on the fused information, specifically including:

[0028] Acquire multimodal user request data, wherein the multimodal user request data includes at least one of the following: text data, image data, audio data, and video data;

[0029] Extract the features of different modal data respectively to obtain multimodal features;

[0030] Acquire data features of different modalities, and perform weight fusion on data features of the same modality according to weight values ​​to obtain first fusion information;

[0031] Additively fuse the data features of the same modality to obtain second fusion information;

[0032] Final fusion information is generated based on the first fusion information and the second fusion information, and a unified feature representation is generated based on the final fusion information.

[0033] Optionally, in the media information cross-modal retrieval method based on semantic alignment described in the embodiment of the present application, semantic associations between different modal data are established based on feature representation to obtain association information, specifically including:

[0034] Obtain feature representations of different modalities and analyze the feature representations to generate semantic information of different modal data;

[0035] Analyze the correlation between semantic information of different modalities, and determine whether the correlation satisfies a set correlation interval;

[0036] If satisfied, the semantic association between different modalities is generated to obtain the associated information;

[0037] If not, it is determined that the semantic difference between different modalities is greater than the set difference threshold, and the fusion rule is adjusted.

[0038] Optionally, in the media information cross-modal retrieval method based on semantic alignment described in the embodiment of the present application, the related information is analyzed based on natural language processing technology to generate query intent, obtain the results of cross-modal retrieval and semantic understanding, and display the results of cross-modal retrieval and semantic understanding, specifically including:

[0039] Obtain the correlation information between different modal data and output the query intent based on the natural language processing model;

[0040] Analyze the retrieval results and semantic understanding results of different modes of query data based on query intent;

[0041] Analyze the matching degree between the cross-modal retrieval process and the semantic understanding of the corresponding modality based on the retrieval results of different modalities;

[0042] The jump information of cross-modal retrieval is analyzed based on the semantic understanding matching degree, and the retrieval method of cross-modal retrieval is adjusted based on the jump information.

[0043] In a second aspect, an embodiment of the present application provides a media information cross-modal retrieval system based on semantic alignment, the system comprising: a memory and a processor, the memory comprising a program of a media information cross-modal retrieval method based on semantic alignment, and the program of the media information cross-modal retrieval method based on semantic alignment is executed by the processor to implement the following steps:

[0044] Obtaining a query request, and generating user query data based on the query request;

[0045] Preprocess user query data to obtain semantically aligned request data in different modalities;

[0046] Extract feature information of different modalities from user request data of different modalities;

[0047] The multimodal features are fused to obtain fused information, and a unified feature representation is generated based on the fused information;

[0048] Establish semantic associations between data of different modalities based on feature representation to obtain association information;

[0049] Based on natural language processing technology, related information is analyzed to generate query intent, obtain results of cross-modal retrieval and semantic understanding, and display the results of cross-modal retrieval and semantic understanding.

[0050] Optionally, in the semantically aligned media information cross-modal retrieval system described in the embodiment of the present application, obtaining a query request, generating user query data based on the query request, and preprocessing the user query data to obtain semantically aligned request data under different modalities specifically includes:

[0051] Obtain query requests, obtain user query data, and analyze noise data and abnormal data of user query data;

[0052] Clean the noise data, analyze the abnormal state of the abnormal data, and determine whether the abnormal state is within the set abnormal range;

[0053] If it is in the abnormal range, the abnormal data will be cleaned and adjusted;

[0054] If it is not in the abnormal range, the abnormal data will be eliminated, and the eliminated data will be filled to obtain the user requested data.

[0055] Optionally, in the media information cross-modal retrieval system based on semantic alignment described in the embodiment of the present application, extracting feature information of different modalities from user request data of different modalities specifically includes:

[0056] Obtain user request data of different modes, extract data features, and obtain request features based on the semantics of data features;

[0057] Compare the requested features with the set features to obtain the feature deviation rate;

[0058] Determining whether the characteristic deviation rate is greater than or equal to a set characteristic deviation rate threshold;

[0059] If it is greater than or equal to, it is determined to be an invalid feature and the invalid feature is eliminated;

[0060] If it is less than , it is determined to be a key feature, and the key feature is modally classified based on the classification rules to obtain a multi-modal feature.

[0061] In the third aspect, an embodiment of the present application also provides a computer-readable storage medium, which includes a media information cross-modal retrieval method program based on semantic alignment. When the media information cross-modal retrieval method program based on semantic alignment is executed by a processor, the steps of the media information cross-modal retrieval method based on semantic alignment as described in any one of the above items are implemented.

[0062] As can be seen from the above, the embodiments of the present application provide a method, system and medium for cross-modal retrieval of media information based on semantic alignment, which obtains a query request, generates user query data based on the query request, pre-processes the user query data, and obtains semantically aligned request data in different modalities; extracts feature information of different modalities from user request data of different modalities; fuses multimodal features to obtain fused information, and generates a unified feature representation based on the fused information; establishes semantic associations between different modal data based on the feature representation to obtain associated information; analyzes the associated information based on natural language processing technology, generates query intent, obtains results of cross-modal retrieval and semantic understanding, and displays the results of cross-modal retrieval and semantic understanding; by combining multiple modal information such as text, images, audio and video, it can more comprehensively understand and express media content, thereby providing richer and more accurate search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 A flowchart of a media information cross-modal retrieval method based on semantic alignment provided in an embodiment of the present application;

[0065] Figure 2 A flow chart of a method for preprocessing user query data for a media information cross-modal retrieval method based on semantic alignment provided in an embodiment of the present application;

[0066] Figure 3 A flow chart of a multimodal feature screening method for a media information cross-modal retrieval method based on semantic alignment provided in an embodiment of the present application;

[0067] Figure 4 A logical block diagram of a media information cross-modal retrieval system based on semantic alignment provided in an embodiment of the present application;

[0068] Figure 5 A flowchart of extracting feature information of different modalities in a media information cross-modal retrieval system based on semantic alignment provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0070] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0071] Please refer to Figure 1 , Figure 1 This is a flowchart of a media information cross-modal retrieval method based on semantic alignment in some embodiments of the present application. The media information cross-modal retrieval method based on semantic alignment is used in a terminal device, and the media information cross-modal retrieval method based on semantic alignment includes the following steps:

[0072] S101, obtaining a query request, and generating user query data based on the query request;

[0073] S102, preprocessing the user query data to obtain semantically aligned request data in different modalities;

[0074] S103, extracting feature information of different modalities from user request data of different modalities;

[0075] Specifically, Figure 5As shown in the figure, when extracting feature information of different modalities, multimodal alignment and pre-training based on VIT is adopted. VIT obtains the representation of the whole picture through a trainable CLS token and accesses the fully connected layer to serve the downstream classification task. After pre-training on a large amount of data and migrating to multiple medium or small-scale image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), ViT achieves better results than CNN models, and the computing resources required for training are greatly reduced. VILT is a continuation of VIT's work in the direction of multimodal text and images. VILT directly concatenates the embedding of text and visual patches as the input of the Transformer encoder. The two modalities have their own learnable position encoding and modality type encoding. ALBEF unifies CLIP-like contrastive learning and ViLT-like interactive fusion paradigms into a training framework through multi-task joint training.

[0076] S104, performing feature fusion on the multimodal features to obtain fusion information, and generating a unified feature representation based on the fusion information; wherein the unified feature is a one-dimensional array;

[0077] S105, establishing semantic associations between data of different modalities based on the feature representation to obtain association information;

[0078] Specifically, semantic association information exists in feature representation information, which is obtained by calculating the similarity between feature representation information. The feature representation information is stored in a vector database. The vector database adopts a distributed solution and performs read-write separation. It supports data storage at the billion level and can achieve millisecond-level retrieval delay under hundreds of concurrency.

[0079] S106, analyzing the related information based on natural language processing technology, generating query intent, obtaining the results of cross-modal retrieval and semantic understanding, and displaying the results of cross-modal retrieval and semantic understanding.

[0080] Specifically, the results of cross-modal retrieval are output through intelligent dialogue. In addition to displaying the retrieved contents of different modalities, the output results also display the corresponding semantic similarities with the retrieved contents and provide downloads.

[0081] It should be noted that cross-modal retrieval technology can be used to achieve information conversion and semantic association between different media types, effectively improving the semantic understanding ability of search results and better meeting users' query intentions. By adopting efficient feature fusion strategies and deep learning technologies, the operating efficiency of the algorithm is improved, enabling the system to quickly respond to large-scale data search needs. Automated annotation and semantic understanding technologies are explored, and through natural language processing, image recognition, speech processing and other methods, fast and accurate annotation of media data can be achieved, thereby improving the performance and availability of multimodal search systems.

[0082] Please refer to Figure 2 , Figure 2 This is a flow chart of a user query data preprocessing method of a semantically aligned media information cross-modal retrieval method in some embodiments of the present application. According to an embodiment of the present invention, a query request is obtained, user query data is generated based on the query request, and the user query data is preprocessed to obtain semantically aligned request data under different modalities, specifically including:

[0083] S201, obtaining a query request, obtaining user query data, and analyzing noise data and abnormal data of the user query data;

[0084] S202, cleaning the noise data, analyzing the abnormal state of the abnormal data, and determining whether the abnormal state is within a set abnormal range;

[0085] S203, if it is in the abnormal range, cleaning and adjusting the abnormal data;

[0086] S204: If it is not in the abnormal range, the abnormal data is eliminated, and the eliminated data is filled to obtain the user requested data.

[0087] It should be noted that through multimodal search engines, media managers can integrate and utilize media resources more efficiently, reduce resource waste and duplicate collection, save costs and improve resource utilization.

[0088] Please refer to Figure 3 , Figure 3 The flowchart of a multimodal feature screening method of a media information cross-modal retrieval method based on semantic alignment in some embodiments of the present application is as follows. According to an embodiment of the present invention, extracting feature information of different modalities from user request data of different modalities specifically includes:

[0089] S301, obtaining user request data of different modes, extracting data features, and obtaining request features according to the semantics of the data features;

[0090] S302, comparing the requested feature with the set feature to obtain a feature deviation rate;

[0091] S303, determining whether the characteristic deviation rate is greater than or equal to a set characteristic deviation rate threshold;

[0092] S304, if it is greater than or equal to, it is determined to be an invalid feature and the invalid feature is eliminated;

[0093] S305: If it is less than , it is determined to be a key feature, and the key feature is modally classified based on the classification rules to obtain a multi-modal feature.

[0094] It should be noted that by analyzing the invalid features and valid features in the user request data through the set features, the valid features are determined as key features, thereby improving the accuracy of the intention analysis of the user query data.

[0095] According to an embodiment of the present invention, multimodal features are fused to obtain fused information, and a unified feature representation is generated based on the fused information, specifically including:

[0096] Acquire multimodal user request data, where the multimodal user request data includes at least one of the following: text data, image data, audio data, and video data;

[0097] Extract the features of different modal data respectively to obtain multimodal features;

[0098] Acquire data features of different modalities, and perform weight fusion on data features of the same modality according to weight values ​​to obtain first fusion information;

[0099] Additively fuse the data features of the same modality to obtain second fusion information;

[0100] Final fusion information is generated based on the first fusion information and the second fusion information, and a unified feature representation is generated based on the final fusion information.

[0101] It should be noted that the features of data from different modalities are fused in different ways, and the data fused in different ways are sorted to obtain the final fusion information, improve the fusion effect, and ensure the reflection accuracy of the data.

[0102] According to an embodiment of the present invention, semantic associations between different modal data are established based on feature representations to obtain association information, specifically including:

[0103] Obtain feature representations of different modalities and analyze the feature representations to generate semantic information of different modal data;

[0104] Analyze the correlation between semantic information of different modalities and determine whether the correlation meets the set correlation interval;

[0105] If satisfied, the semantic association between different modalities is generated to obtain the associated information;

[0106] If not, it is determined that the semantic difference between different modalities is greater than the set difference threshold, and the fusion rule is adjusted.

[0107] It should be noted that by analyzing the correlation between semantic information of different modalities and whether the associated information meets the requirements, the cross-modal retrieval module uses the fused feature representation to establish semantic associations between data of different modalities and realize cross-modal information retrieval based on user queries.

[0108] According to an embodiment of the present invention, the related information is analyzed based on natural language processing technology to generate query intent, obtain the results of cross-modal retrieval and semantic understanding, and display the results of cross-modal retrieval and semantic understanding, specifically including:

[0109] Obtain the correlation information between different modal data and output the query intent based on the natural language processing model;

[0110] Analyze the retrieval results and semantic understanding results of different modes of query data based on query intent;

[0111] Analyze the matching degree between the cross-modal retrieval process and the semantic understanding of the corresponding modality based on the retrieval results of different modalities;

[0112] The jump information of cross-modal retrieval is analyzed based on the semantic understanding matching degree, and the retrieval method of cross-modal retrieval is adjusted based on the jump information.

[0113] It should be noted that by analyzing the jump information of cross-modal retrieval and adjusting the retrieval method, the jump information can be understood as the similarity between different modalities to judge the modal difference between the current modal retrieval and the next modal retrieval. The jumpiness is analyzed based on the modal difference to obtain the jump information, thereby ensuring that the modal difference between the retrieval of the next modal data and the retrieval of the current modal data is small and the jump degree is reduced. The user interface module displays relevant multimedia information to the user based on the results of cross-modal retrieval and semantic understanding, such as related images, videos, audio clips or text content.

[0114] According to an embodiment of the present invention, the results of cross-modal retrieval and semantic understanding are displayed, including:

[0115] Obtain the results of cross-modal retrieval and semantic understanding to obtain display data;

[0116] According to the classification of the display data, the display data of the current modality is obtained by dividing the display data into paired data of retrieval and semantic understanding between different modalities;

[0117] Create different data tables for display data in different modes;

[0118] Obtain a display interface, divide the display interface into regions, and obtain different display regions;

[0119] Display different data tables in different display areas.

[0120] It should be noted that the display data is divided into categories according to the modal classification, and the display interface is divided into regions according to the classified display data, so that data of different modalities are displayed in different display areas, thereby improving the display effect.

[0121] Please refer to Figure 4 , Figure 4 It is a logical block diagram of a media information cross-modal retrieval system based on semantic alignment in some embodiments of the present application. In a second aspect, an embodiment of the present application provides a media information cross-modal retrieval system based on semantic alignment, the system comprising: a memory and a processor, the memory comprising a program of a media information cross-modal retrieval method based on semantic alignment, and when the program of the media information cross-modal retrieval method based on semantic alignment is executed by the processor, the following steps are implemented:

[0122] Obtaining a query request, and generating user query data based on the query request;

[0123] Preprocess user query data to obtain semantically aligned request data in different modalities;

[0124] Extract feature information of different modalities from user request data of different modalities;

[0125] The multimodal features are fused to obtain fused information, and a unified feature representation is generated based on the fused information;

[0126] Establish semantic associations between data of different modalities based on feature representation to obtain association information;

[0127] Based on natural language processing technology, related information is analyzed to generate query intent, obtain results of cross-modal retrieval and semantic understanding, and display the results of cross-modal retrieval and semantic understanding.

[0128] According to an embodiment of the present invention, a query request is obtained, user query data is generated based on the query request, and the user query data is preprocessed to obtain semantically aligned request data under different modalities, specifically including:

[0129] Obtain query requests, obtain user query data, and analyze noise data and abnormal data of user query data;

[0130] Clean the noise data, analyze the abnormal state of the abnormal data, and determine whether the abnormal state is within the set abnormal range;

[0131] If it is in the abnormal range, the abnormal data will be cleaned and adjusted;

[0132] If it is not in the abnormal range, the abnormal data will be eliminated, and the eliminated data will be filled to obtain the user requested data.

[0133] According to an embodiment of the present invention, extracting feature information of different modalities from user request data of different modalities specifically includes:

[0134] Obtain user request data of different modes, extract data features, and obtain request features based on the semantics of data features;

[0135] Compare the requested features with the set features to obtain the feature deviation rate;

[0136] Determine whether the feature deviation rate is greater than or equal to a set feature deviation rate threshold;

[0137] If it is greater than or equal to, it is determined to be an invalid feature and the invalid feature is eliminated;

[0138] If it is less than , it is determined to be a key feature, and the key feature is modally classified based on the classification rules to obtain a multi-modal feature.

[0139] It should be noted that by analyzing the invalid features and valid features in the user request data through the set features, the valid features are determined as key features, thereby improving the accuracy of the intention analysis of the user query data.

[0140] According to an embodiment of the present invention, multimodal features are fused to obtain fused information, and a unified feature representation is generated based on the fused information, specifically including:

[0141] Acquire multimodal user request data, where the multimodal user request data includes at least one of the following: text data, image data, audio data, and video data;

[0142] Extract the features of different modal data respectively to obtain multimodal features;

[0143] Acquire data features of different modalities, and perform weight fusion on data features of the same modality according to weight values ​​to obtain first fusion information;

[0144] Additively fuse the data features of the same modality to obtain second fusion information;

[0145] Final fusion information is generated based on the first fusion information and the second fusion information, and a unified feature representation is generated based on the final fusion information.

[0146] It should be noted that the features of data from different modalities are fused in different ways, and the data fused in different ways are sorted to obtain the final fusion information, improve the fusion effect, and ensure the reflection accuracy of the data.

[0147] According to an embodiment of the present invention, semantic associations between different modal data are established based on feature representations to obtain association information, specifically including:

[0148] Obtain feature representations of different modalities and analyze the feature representations to generate semantic information of different modal data;

[0149] Analyze the correlation between semantic information of different modalities and determine whether the correlation meets the set correlation interval;

[0150] If satisfied, the semantic association between different modalities is generated to obtain the associated information;

[0151] If not, it is determined that the semantic difference between different modalities is greater than the set difference threshold, and the fusion rule is adjusted.

[0152] It should be noted that by analyzing the correlation between semantic information of different modalities and whether the associated information meets the requirements, the cross-modal retrieval module uses the fused feature representation to establish semantic associations between data of different modalities and realize cross-modal information retrieval based on user queries.

[0153] According to an embodiment of the present invention, the related information is analyzed based on natural language processing technology to generate query intent, obtain the results of cross-modal retrieval and semantic understanding, and display the results of cross-modal retrieval and semantic understanding, specifically including:

[0154] Obtain the correlation information between different modal data and output the query intent based on the natural language processing model;

[0155] Analyze the retrieval results and semantic understanding results of different modes of query data based on query intent;

[0156] Analyze the matching degree between the cross-modal retrieval process and the semantic understanding of the corresponding modality based on the retrieval results of different modalities;

[0157] The jump information of cross-modal retrieval is analyzed based on the semantic understanding matching degree, and the retrieval method of cross-modal retrieval is adjusted based on the jump information.

[0158] It should be noted that by analyzing the jump information of cross-modal retrieval and adjusting the retrieval method, the jump information can be understood as the similarity between different modalities to judge the modal difference between the current modal retrieval and the next modal retrieval. The jumpiness is analyzed based on the modal difference to obtain the jump information, thereby ensuring that the modal difference between the retrieval of the next modal data and the retrieval of the current modal data is small and the jump degree is reduced. The user interface module displays relevant multimedia information to the user based on the results of cross-modal retrieval and semantic understanding, such as related images, videos, audio clips or text content.

[0159] Specifically, (1) Similarity evaluation:

[0160] First, we need to evaluate the similarity between different modal data. For example, we can use cosine similarity, Jaccard similarity index or other similarity metrics to evaluate the similarity between text and image features.

[0161] (2) Jump information quantification:

[0162] Quantifying the degree of “jumping” in cross-modal retrieval is done by measuring the semantic distance from one modality to another. For example, if the retrieval results from text to image have a large semantic distance from the original query, this may indicate a high degree of jumping.

[0163] (3) Search result feedback loop:

[0164] Collect user feedback based on the semantic matching degree of the search results. If the user is not satisfied with the search results, the system can analyze the reasons to see if the search results deviate from the user's actual needs due to excessive jumps.

[0165] (4) Dynamically adjust search parameters:

[0166] According to the jump information, the parameters of the retrieval algorithm can be dynamically adjusted. For example, if it is found that the semantic match between the image retrieval results and the text query is not high, the weight of the text features in the retrieval algorithm can be increased, or the image feature extraction method can be adjusted.

[0167] (5) Modal conversion strategy:

[0168] Implement modality transfer strategies. For example, if text-to-image retrieval is too fast, you can try converting the text into an intermediate representation of the image (e.g., using a text-to-image generative model) before performing image retrieval.

[0169] (6) Utilization of contextual information:

[0170] Use the context of the query to adjust the search strategy. For example, if the user query has a specific context (such as time, location, etc.), the system can use this context information to optimize the relevance of the search results.

[0171] (7) User behavior analysis:

[0172] Analyze users' historical behaviors and preferences, and adjust retrieval algorithms based on users' clicks, browsing, and feedback behaviors to reduce the jumpiness of retrieval results.

[0173] Example: Assume that a user enters a query about "beach sunset", and the system first retrieves relevant pictures in the image database. If the system finds that the retrieved images do not match the semantics of the user's query well (for example, most images are sunsets but not on the beach), the system can take the following steps to make adjustments:

[0174] Analyze the semantic distance between the two concepts of "beach" and "sunset" to determine whether there is significant jump. Based on the jump information, increase the weight of text features related to "beach" in image retrieval.

[0175] Using modal conversion technology, the text concept of "beach" is converted into image features, and then fused with sunset images for retrieval. Using user feedback, if the user is satisfied with the image of the beach sunset, this retrieval mode is recorded; if not, the retrieval strategy is further adjusted.

[0176] According to an embodiment of the present invention, the results of cross-modal retrieval and semantic understanding are displayed, including:

[0177] Obtain the results of cross-modal retrieval and semantic understanding to obtain display data;

[0178] According to the classification of the display data, the display data of the current modality is obtained by dividing the display data into paired data of retrieval and semantic understanding between different modalities;

[0179] Create different data tables for display data in different modes;

[0180] Obtain a display interface, divide the display interface into regions, and obtain different display regions;

[0181] Display different data tables in different display areas.

[0182] It should be noted that the display data is classified according to the modal classification, and the display interface is divided into areas according to the classified display data, so that data of different modalities are displayed in different display areas, improving the display effect. The user interface module displays relevant multimedia information to the user based on the results of cross-modal retrieval and semantic understanding, such as related images, videos, audio clips or text content.

[0183] A third aspect of the present invention provides a computer-readable storage medium, which includes a program for a media information cross-modal retrieval method based on semantic alignment. When the program for a media information cross-modal retrieval method based on semantic alignment is executed by a processor, the steps of any of the above-mentioned methods for media information cross-modal retrieval based on semantic alignment are implemented.

[0184] The present invention discloses a method, system and medium for cross-modal retrieval of media information based on semantic alignment, which obtains a query request, generates user query data based on the query request, pre-processes the user query data, and obtains semantically aligned request data in different modalities; extracts feature information of different modalities from user request data of different modalities; fuses multimodal features to obtain fused information, and generates a unified feature representation based on the fused information; establishes semantic associations between data of different modalities based on the feature representation to obtain associated information; analyzes the associated information based on natural language processing technology to generate query intent, obtains results of cross-modal retrieval and semantic understanding, and displays the results of cross-modal retrieval and semantic understanding; and by combining multiple modal information such as text, image, audio and video, it is possible to more comprehensively understand and express media content, thereby providing richer and more accurate search results.

[0185] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0186] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0187] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0188] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, the aforementioned program can be stored in a readable storage medium, and when the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0189] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. A media information cross-modal retrieval method based on semantic alignment, characterized in that: include: Obtaining a query request, and generating user query data based on the query request; Preprocess user query data to obtain semantically aligned request data in different modalities; Extract feature information of different modalities from user request data of different modalities; The multimodal features are fused to obtain fused information, and a unified feature representation is generated based on the fused information; Establish semantic associations between data of different modalities based on feature representation to obtain association information; Analyze related information based on natural language processing technology, generate query intent, obtain cross-modal retrieval and semantic understanding results, and display the results of cross-modal retrieval and semantic understanding; Extract feature information of different modalities from user request data of different modalities, including: Obtain user request data of different modes, extract data features, and obtain request features based on the semantics of data features; Compare the requested features with the set features to obtain the feature deviation rate; Determining whether the characteristic deviation rate is greater than or equal to a set characteristic deviation rate threshold; If it is greater than or equal to, it is determined to be an invalid feature and the invalid feature is eliminated; If it is less than , it is determined to be a key feature, and the key feature is modally classified based on the classification rules to obtain a multi-modal feature; The multimodal features are fused to obtain fusion information, and a unified feature representation is generated based on the fusion information, including: Acquire multimodal user request data, wherein the multimodal user request data includes at least one of the following: text data, image data, audio data, and video data; Extract the features of different modal data respectively to obtain multimodal features; Acquire data features of different modalities, and perform weight fusion on data features of the same modality according to weight values ​​to obtain first fusion information; Additively fuse the data features of the same modality to obtain second fusion information; Final fusion information is generated based on the first fusion information and the second fusion information, and a unified feature representation is generated based on the final fusion information.

2. The method for cross-modal media information retrieval based on semantic alignment according to claim 1, characterized in that: Obtain query requests, generate user query data based on the query requests, preprocess the user query data, and obtain semantically aligned request data under different modalities, including: Obtain query requests, obtain user query data, and analyze noise data and abnormal data of user query data; Clean the noise data, analyze the abnormal state of the abnormal data, and determine whether the abnormal state is within the set abnormal range; If it is in the abnormal range, the abnormal data will be cleaned and adjusted; If it is not in the abnormal range, the abnormal data will be eliminated, and the eliminated data will be filled to obtain the user requested data.

3. The method for cross-modal media information retrieval based on semantic alignment according to claim 1, characterized in that: Based on the feature representation, semantic associations between different modal data are established to obtain association information, including: Obtain feature representations of different modalities and analyze the feature representations to generate semantic information of different modal data; Analyze the correlation between semantic information of different modalities, and determine whether the correlation satisfies a set correlation interval; If satisfied, the semantic association between different modalities is generated to obtain the associated information; If not, it is determined that the semantic difference between different modalities is greater than the set difference threshold, and the fusion rule is adjusted.

4. The method for cross-modal media information retrieval based on semantic alignment according to claim 3, characterized in that: Based on natural language processing technology, the related information is analyzed to generate query intent, obtain the results of cross-modal retrieval and semantic understanding, and display the results of cross-modal retrieval and semantic understanding, including: Obtain the correlation information between different modal data and output the query intent based on the natural language processing model; Analyze the retrieval results and semantic understanding results of different modes of query data based on query intent; Analyze the matching degree between the cross-modal retrieval process and the semantic understanding of the corresponding modality based on the retrieval results of different modalities; The jump information of cross-modal retrieval is analyzed based on the semantic understanding matching degree, and the retrieval method of cross-modal retrieval is adjusted based on the jump information.

5. A media information cross-modal retrieval system based on semantic alignment, characterized in that: The system includes: a memory and a processor, wherein the memory includes a program of a media information cross-modal retrieval method based on semantic alignment, and when the program of the media information cross-modal retrieval method based on semantic alignment is executed by the processor, the following steps are implemented: Obtaining a query request, and generating user query data based on the query request; Preprocess user query data to obtain semantically aligned request data in different modalities; Extract feature information of different modalities from user request data of different modalities; The multimodal features are fused to obtain fused information, and a unified feature representation is generated based on the fused information; Establish semantic associations between data of different modalities based on feature representation to obtain association information; Analyze related information based on natural language processing technology, generate query intent, obtain cross-modal retrieval and semantic understanding results, and display the results of cross-modal retrieval and semantic understanding; Extract feature information of different modalities from user request data of different modalities, including: Obtain user request data of different modes, extract data features, and obtain request features based on the semantics of data features; Compare the requested features with the set features to obtain the feature deviation rate; Determining whether the characteristic deviation rate is greater than or equal to a set characteristic deviation rate threshold; If it is greater than or equal to, it is determined to be an invalid feature and the invalid feature is eliminated; If it is less than , it is determined to be a key feature, and the key feature is modally classified based on the classification rules to obtain a multi-modal feature; The multimodal features are fused to obtain fusion information, and a unified feature representation is generated based on the fusion information, including: Acquire multimodal user request data, wherein the multimodal user request data includes at least one of the following: text data, image data, audio data, and video data; Extract the features of different modal data respectively to obtain multimodal features; Acquire data features of different modalities, and perform weight fusion on data features of the same modality according to weight values ​​to obtain first fusion information; Additively fuse the data features of the same modality to obtain second fusion information; Final fusion information is generated based on the first fusion information and the second fusion information, and a unified feature representation is generated based on the final fusion information.

6. The media information cross-modal retrieval system based on semantic alignment according to claim 5, characterized in that: Obtain query requests, generate user query data based on the query requests, preprocess the user query data, and obtain semantically aligned request data under different modalities, including: Obtain query requests, obtain user query data, and analyze noise data and abnormal data of user query data; Clean the noise data, analyze the abnormal state of the abnormal data, and determine whether the abnormal state is within the set abnormal range; If it is in the abnormal range, the abnormal data will be cleaned and adjusted; If it is not in the abnormal range, the abnormal data will be eliminated, and the eliminated data will be filled to obtain the user requested data.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a media information cross-modal retrieval method program based on semantic alignment. When the media information cross-modal retrieval method program based on semantic alignment is executed by a processor, the steps of the media information cross-modal retrieval method based on semantic alignment as described in any one of claims 1 to 4 are implemented.

Images