Semantic comprehension driven cross-modal information fusion and retrieval method and system

Abstract

The invention discloses a cross-modal information fusion and retrieval method and system driven by semantic comprehension, and the method comprises the steps: obtaining text, image and audio original data, and extracting an initial feature set of each modal through a deep neural network; dynamically distributing each modal weight coefficient based on an attention mechanism, and performing weighted fusion on the initial feature set to obtain cross-modal fusion feature representation; through a cross-modal semantic association analysis model, high-dimensional semantic association features are extracted from the fusion feature representation, and semantic enhancement feature vectors are generated; constructing a cross-modal semantic graph network based on the vector, complementing missing modal features, and generating an optimized multi-modal feature set; and inputting the optimized feature set and the query sample into a contrast learning model, calculating a semantic similarity score, and generating a cross-modal retrieval result sorting list according to the score.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence applications, and in particular relates to a semantic understanding-driven cross-modal information fusion and retrieval method and system. Background Art

[0002] With the development of computer technology and artificial intelligence, multimodal information processing technology has emerged. Multimodal information processing aims to comprehensively process multiple different types of data, such as text, images, and audio, to more comprehensively and accurately understand and analyze information. Its hallmark is its ability to fully leverage the complementary information contained in different modalities, improving both the effectiveness of information processing and the intelligence of applications. Traditionally, multimodal information processing often involves processing data from different modalities separately. For example, natural language processing algorithms are used to analyze text data, computer vision techniques are employed to identify image data, and speech recognition techniques are employed to process audio data. These processing results are then integrated through simple concatenation or shallow fusion to enable cross-modal applications. For example, in cross-modal retrieval, results are filtered and sorted based solely on simple feature matching rules. However, this traditional approach to multimodal information processing and retrieval currently presents numerous challenges. For one thing, simple feature concatenation and shallow fusion fail to fully exploit the deep semantic connections between data from different modalities. Consequently, the fused feature representation lacks sufficient semantic information and fails to accurately reflect the overall meaning of the multimodal data. On the other hand, in cross-modal retrieval, simple matching rules cannot adapt well to the complexity and diversity of multimodal data, resulting in low accuracy and relevance of retrieval results, which cannot meet users' needs for efficient and accurate cross-modal information retrieval. Furthermore, traditional methods have limited ability to handle cases with missing modalities and cannot effectively complete the missing information, further affecting the effectiveness of multimodal information processing and retrieval. Summary of the Invention

[0003] Based on this, it is necessary to provide a semantic understanding-driven cross-modal information fusion and retrieval method and system that can solve the above problems.

[0004] In a first aspect, the present application provides a semantic understanding-driven cross-modal information fusion and retrieval method, comprising:

[0005] Obtain raw data of text, images, and audio, and extract the initial feature set of each modality through deep neural networks;

[0006] Based on the attention mechanism, the weight coefficients of each modality are dynamically assigned, and the initial feature set is weightedly fused to obtain a cross-modal fusion feature representation;

[0007] Through the cross-modal semantic association analysis model, high-dimensional semantic association features are extracted from the cross-modal fusion feature representation to generate a semantically enhanced feature vector;

[0008] Construct a cross-modal semantic graph network based on semantically enhanced feature vectors, and complete the missing modal features to generate an optimized multimodal feature set;

[0009] The optimized multimodal feature set is compared with the query sample input into the learning model, the semantic similarity score is calculated, and a ranked list of cross-modal retrieval results is generated based on the score.

[0010] In one embodiment, generating a semantically enhanced feature vector includes:

[0011] Detect the semantic coverage of cross-modal fusion feature representation. When the semantic coverage is lower than the preset threshold, the following semantic enhancement process is triggered:

[0012] The fused feature representation is mapped to a high-dimensional feature space through a feature space mapping function to extract cross-modal semantic associations and generate a semantic association matrix;

[0013] The fused feature representation is orthogonally projected based on the semantic association matrix to generate a semantically enhanced feature vector.

[0014] In one embodiment, the weight coefficients of each modality are dynamically assigned based on the attention mechanism, including determining the weight coefficients of each modality by the following weight calculation function:

[0015]

[0016] Among them, α m Represents the attention weight of modality m, with a value range of 0≤α m ≤1, and∑ m α m = 1, m∈{t,i,a} represents the modality identifier, representing text, image and audio respectively, τ represents the temperature coefficient, represents the activation function, γ represents the leakage coefficient, represents the trainable weight matrix, [Q m′ ||K global ] represents vector concatenation operation, Q m’ represents the query vector, K global Represents the global context key vector.

[0017] In one embodiment, constructing a cross-modal semantic graph network based on semantically enhanced feature vectors includes:

[0018] Perform node space topology mapping based on semantically enhanced feature vectors to obtain a set of graph nodes with modal labels;

[0019] Calculate cross-modal cosine similarity based on the graph node set and generate the inter-modal correlation edge weight matrix;

[0020] The three-hop neighborhood subgraph is divided based on the weight matrix, and the neighborhood features are aggregated through the graph attention network to generate the updated node feature vector;

[0021] Based on the updated node feature vectors, adversarial feature completion is performed on nodes with missing modalities to generate a cross-modal semantic graph network.

[0022] In one embodiment, the method further comprises:

[0023] Use the heat map generation algorithm to generate heat map data corresponding to the semantic association matrix;

[0024] Perform image segmentation and coordinate transformation on the heat map data to generate a binary attention mask;

[0025] Perform Hadamard product operation on the attention mask and the semantic association matrix to obtain the optimized matrix of enhanced association;

[0026] The overlap ratio between the enhanced region and the labeled region in the optimization matrix is calculated. When the overlap ratio is lower than the preset threshold, the feature space mapping function is retrained.

[0027] In one embodiment, raw text, image, and audio data are obtained, and an initial feature set of each modality is extracted using a deep neural network, including:

[0028] Based on the preset multilingual word embedding matrix, the gate unit is used to dynamically select the embedding vector that matches the input language to obtain a multilingual text feature set;

[0029] Based on a multilingual text feature set, a cross-language semantic alignment loss function is used to constrain the latent space distribution and generate a unified cross-language semantic representation.

[0030] Based on the unified cross-language semantic representation and the language weight coefficient output by the language identification branch, a gated residual network is used for feature fusion to obtain an optimized multilingual text feature set.

[0031] In one embodiment, before aggregating neighborhood features through a graph attention network, the method includes:

[0032] Based on the local outlier factor value of node characteristics, the standard deviation threshold is used to detect the abnormal node set;

[0033] Based on the abnormal node set, the dual-channel processing structure is used to generate the original feature stream and the denoised feature stream in parallel;

[0034] Based on the original feature stream and the denoised feature stream, a gated fuser is used to perform dynamic weighting to obtain the preprocessed node feature vector, which is used as the input of the graph attention network to aggregate neighborhood features.

[0035] Secondly, this application also provides a semantic understanding-driven cross-modal information fusion and retrieval system, including:

[0036] Multimodal feature extraction module, which is used to obtain raw data of text, image and audio, and extract the initial feature set of each modality through deep neural network;

[0037] Dynamic attention fusion module, which is used to dynamically assign weight coefficients to each modality based on the attention mechanism and perform weighted fusion on the initial feature set to obtain a cross-modal fusion feature representation;

[0038] The semantic association enhancement module is used to extract high-dimensional semantic association features from the cross-modal fusion feature representation through the cross-modal semantic association analysis model and generate a semantic enhancement feature vector;

[0039] The graph network completion module is used to build a cross-modal semantic graph network based on semantically enhanced feature vectors, and to complete missing modal features to generate an optimized multimodal feature set;

[0040] The comparative retrieval optimization module is used to input the optimized multimodal feature set and query samples into the comparative learning model, calculate the semantic similarity score, and generate a ranked list of cross-modal retrieval results based on the score.

[0041] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned semantic understanding-driven cross-modal information fusion and retrieval method are implemented:

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned semantic understanding-driven cross-modal information fusion and retrieval method:

[0043] The above-mentioned semantic understanding-driven cross-modal information fusion and retrieval method and system uses deep neural networks to extract initial features of text, images and audio, combines the attention mechanism to dynamically allocate modal weights and weighted fusion, changes the defects of traditional simple splicing or shallow fusion, realizes differentiated focus and deep fusion of key information of multimodal data, and explores potential correlations between modalities; extracts high-dimensional semantic correlation features through a cross-modal semantic correlation analysis model to generate enhanced vectors, making up for the problem of insufficient semantic representation in traditional methods; constructs a cross-modal semantic graph network and completes missing modal features, solves the dilemma of limited processing capabilities of traditional methods for missing modalities, and forms a complete and semantically consistent multimodal feature set; calculates semantic similarity through a comparative learning model and generates a retrieval ranking list, which can achieve cross-modal precise matching based on deep semantic associations, improve the accuracy and relevance of retrieval results, and systematically solve the technical problems of insufficient semantic fusion, weak missing modal completion capabilities and poor retrieval effects in traditional multimodal processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a flow chart of a semantic understanding-driven cross-modal information fusion and retrieval method of the present invention;

[0046] Figure 2 This is a structural diagram of a semantic understanding-driven cross-modal information fusion and retrieval system of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0048] The present application discloses a semantic understanding-driven cross-modal information fusion and retrieval method and system, the implementation environment of which includes a data acquisition device, a computing terminal, and a storage device. In scenarios where information fusion and retrieval are required, the data acquisition device transmits the original data to the computing terminal, obtains the pre-trained model and historical data by accessing the storage device, and sequentially completes feature extraction, weighted fusion, semantic enhancement, and graph network completion. The semantic similarity is calculated based on the sample library to be retrieved in the storage device to generate retrieval results.

[0049] In one embodiment, Figure 1 As shown, a semantic understanding-driven cross-modal information fusion and retrieval method is provided. This embodiment uses the method applied to a terminal device as an example for explanation. It should be understood that the method can also be applied to a server, or a distributed system integrating a terminal and a server, to achieve cross-modal information processing through interaction between the two. In this embodiment, the method includes the following steps:

[0050] S101: Obtain the original data of text, image and audio, and extract the initial feature set of each modality through a deep neural network.

[0051] Among them, it can be to obtain three types of heterogeneous data, such as text information such as documents, web page content, image information such as JPEG / PNG format images, and audio information such as voice signals and music clips, and perform basic cleaning such as image denoising, audio denoising, and text segmentation to provide input for deep neural networks. For different modalities, different deep neural networks can be used, such as: for text modalities, RNN (Recurrent Neural Network) is used to capture sequence dependencies, or the Transformer model is used to extract contextual semantic features and output word-level / sentence-level feature vector sets; for image modalities, CNN (Convolutional Neural Network) is used to extract edge / texture features of the image through convolutional layers, reduce the dimensionality of the extracted features through pooling layers, and generate feature vectors based on the reduced dimensionality features in fully connected layers, outputting fixed-dimensional visual features; for audio modalities, the audio signal can be first converted into a Mel-spectrogram and then input into CNN to extract frequency domain features; or RNN can be used directly to process the audio sequence and output acoustic feature vectors. By extracting features such as the grammatical structure of text, semantic entities such as cats, and emotional tendencies; the object contours, color distribution, and spatial layout of images; and the pitch, rhythm, and timbre of audio, we obtain the initial feature set of each modality.

[0052] S102, dynamically assigning weight coefficients to each modality based on the attention mechanism, and performing weighted fusion on the initial feature set to obtain a cross-modal fusion feature representation.

[0053] The information contained in different modalities contributes differently to the task. Text excels at expressing abstract concepts, such as happy emotions; images excel at conveying visual details, such as the facial features of a smile; and audio excels at conveying rhythmic information, such as cheerful intonation. Dynamic weighting allows the model to adaptively focus on key modalities based on the input content. For example, when retrieving images of cats, the image modality is given a higher weight; when retrieving cat meows, the audio modality is given a higher weight. The attention mechanism takes as input an initial feature set from each modality. It then introduces a global context key vector as global reference information for weight calculation to quantify modal importance. It then calculates the attention weight for each modality and performs a weighted fusion. The output is a cross-modal fused feature representation containing the weighted combined information from each modality. The importance of text, image, and audio is dynamically determined based on the input content, enabling on-demand fusion. This dynamic weighting strategy preserves the specific strengths of each modality while strengthening the semantic connections between modalities through global context modeling.

[0054] S103, through the cross-modal semantic association analysis model, extract high-dimensional semantic association features from the cross-modal fusion feature representation to generate a semantically enhanced feature vector.

[0055] While dynamic weighted fusion integrates modal features, it only implements weighted linear superposition and does not explicitly model semantic interactions between modalities. For example, the semantic equivalence between a dog in an image, a dog in text, and a dog bark in audio requires semantic association analysis to establish a unified semantic representation. Low-level fusion features, such as concatenated or weighted vectors, are typically in low- or medium-dimensional spaces and struggle to capture complex cross-modal semantic relationships. High-dimensional spatial mapping and nonlinear transformations can be used to mine high-order interactions between modalities, such as the joint semantics of text sentiment, image color, and audio intonation. This can be achieved by utilizing a cross-modal semantic association analysis model, using a feature space mapping function to generate a semantic association matrix, and then using orthogonal projection to generate a semantically enhanced feature vector. This approach can explicitly mine deep semantic connections between data from different modalities, generating enhanced feature vectors that contain shared semantics and complementary information across modalities.

[0056] S104: construct a cross-modal semantic graph network based on the semantically enhanced feature vector, and complete the missing modal features to generate an optimized multimodal feature set.

[0057] In real-world scenarios, multimodal data often contains missing features, such as video clips without audio or text descriptions without accompanying images. Traditional methods, such as direct ignoring or random filling, can lead to information loss or the introduction of noise. This requires intelligent completion through structured modeling. Graph structures are suitable for modeling the semantic associations of multimodal data. Each node sample in the graph structure can carry the modal label and corresponding features of text / image / audio. The edges of the graph structure can represent the cross-modal semantic similarity between samples, such as the strong correlation between the text "cat" and the image and meow of a cat. Through neighborhood aggregation in the graph network, the complete modal information of similar samples can be used to infer the missing features of the current sample, such as using a dog barking sample with audio to complete the acoustic features of a dog image sample without audio. The nodes of the graph can be constructed through node space topological mapping, generating a weight matrix for the inter-modal association edges. Three-hop neighborhood subgraph partitioning and feature aggregation are then used to complete the missing modal features. Based on semantic similarity rather than simple statistics, the completed features are ensured to conform to cross-modal semantic logic. Through graph aggregation, the completed multimodal features form a unified and coherent feature space in the semantic graph, effectively handling data missing in real scenarios and providing complete and high-quality feature input for subsequent comparative retrieval.

[0058] S105: Input the optimized multimodal feature set and the query sample into the comparison learning model, calculate the semantic similarity score, and generate a ranked list of cross-modal retrieval results based on the score.

[0059] To achieve semantic alignment across modalities, such as matching a text query like "a beach at sunset" with corresponding images and audio, the key challenge lies in bridging the differences in feature spaces between modalities, such as image pixel space versus text word embedding space. The advantage of contrastive learning is that it employs a positive-negative strategy—that is, it pulls semantically related samples closer together and pushes irrelevant samples further apart—to force semantically equivalent samples from different modalities to cluster in the feature space, making it a natural fit for cross-modal alignment. The optimized multimodal feature set and the query sample are input into the contrastive learning model. Cosine similarity is used to calculate the semantic distance between the query embedding and each sample embedding, outputting a similarity score, with higher scores indicating closer semantics. Samples are then sorted from high to low based on the similarity score to generate a cross-modal retrieval result list. Through the semantic alignment capabilities of contrastive learning, multimodal data is transformed into efficiently searchable semantic units, fundamentally addressing the core issues of inaccurate matching and poor modality compatibility in traditional cross-modal retrieval.

[0060] The above-mentioned semantic understanding-driven cross-modal information fusion and retrieval method obtains the original data of text, image and audio and uses deep neural networks to extract the initial features of each modality, changing the defects of traditional simple splicing or shallow fusion, and can effectively extract deep features from multimodal data; based on the attention mechanism, it dynamically allocates the weight coefficients of each modality and weightedly fuses the initial feature set to achieve differentiated focus and deep fusion of key information of multimodal data, and can explore the potential correlation between modalities to make the fusion features more in line with task requirements; through the cross-modal semantic association analysis model, high-dimensional semantic association features are extracted from the fusion features and semantic enhancement feature vectors are generated to make up for the traditional The problem of insufficient semantic representation in the method is enhanced, and the semantic consistency of multimodal data is enhanced; a cross-modal semantic graph network is constructed based on the semantically enhanced feature vector and the missing modal features are supplemented, which solves the dilemma of the limited processing ability of missing modalities of traditional methods and forms a complete and semantically consistent multimodal feature set; the optimized feature set is compared with the query sample input to the learning model to calculate the semantic similarity score and generate a ranked list of retrieval results, which can achieve cross-modal precise matching based on deep semantic associations, improve the accuracy and relevance of retrieval results, and systematically solve the technical problems of insufficient semantic fusion, weak missing modal completion ability and poor retrieval effect in traditional multimodal processing.

[0061] In one embodiment, generating a semantically enhanced feature vector includes:

[0062] S201, detecting the semantic coverage of the cross-modal fusion feature representation. When the semantic coverage is lower than a preset threshold, the following semantic enhancement process is triggered:

[0063] S202, mapping the fused feature representation to a high-dimensional feature space through a feature space mapping function, extracting cross-modal semantic associations and generating a semantic association matrix;

[0064] S203, performing orthogonal projection on the fused feature representation based on the semantic association matrix to generate a semantically enhanced feature vector.

[0065] Specifically, semantic coverage refers to the completeness of effective cross-modal semantic information in the fused features, and a preset threshold can be set as a trigger condition. When it is detected that the semantic coverage of the current fused features is lower than the threshold, it indicates that the cross-modal semantic associations it contains are insufficient and cannot meet the needs of subsequent processing. The semantic enhancement process is started to avoid redundant calculations of semantically sufficient features and achieve efficient resource utilization. High-dimensional space can accommodate more complex nonlinear semantic associations. By calculating the semantic associations between cross-modal features, such as cosine similarity and dot product matching, a semantic association matrix is generated to explicitly characterize the semantic correspondence strength of text, image, and audio features in high-dimensional space. For example, high-value elements in the matrix represent strong semantic associations between a text word and an image area or audio clip. Through matrix decomposition such as singular value decomposition or orthogonal transformation, the linearly independent vector system is converted into an orthogonal system, the feature dimensions corresponding to strong semantic associations are retained, and low-correlation noise is filtered out, so that the generated semantically enhanced feature vector only contains the information with the closest cross-modal semantics. It removes feature redundancy, ensures the semantic independence of each dimension, strengthens cross-modal shared semantics such as the core concepts commonly expressed in multimodal data, and provides high-quality semantic basis vectors for the subsequent construction of a cross-modal semantic graph network.

[0066] In one embodiment, the weight coefficients of each modality are dynamically allocated based on the attention mechanism, including: S301, the weight coefficients of each modality are determined by the following weight calculation function:

[0067]

[0068] Among them, α m Represents the attention weight of modality m, with a value range of 0≤α m ≤1, and∑ m α m = 1, m∈{t,i,a} represents the modality identifier, representing text, image and audio respectively, τ represents the temperature coefficient, represents the activation function, γ represents the leakage coefficient, represents the trainable weight matrix, [Q m′ ‖K global ] represents vector concatenation operation, Q m’ represents the query vector, K global Represents the global context key vector.

[0069] For example, the attention weight α of modality m mRepresents the importance weights of text, image, and audio corresponding to modality m during fusion. The temperature coefficient τ is used to regulate the sharpness of the activation function and the concentration of the weight distribution. It is based on manual preset or cross-validation optimization and does not participate in model training, but affects the sparsity of the weights. The activation function LeakyReLU(x) is used to avoid the problem of traditional ReLU gradient vanishing in the negative region. The leakage coefficient γ is set manually to control the slope of the negative region. By introducing a nonlinear response to negative features, the model's ability to capture negative semantics in modal features is improved. Trainable weight matrix W q It is used to map the concatenated vectors to the attention score space, learn from the training data through back propagation, and automatically adapt to the interaction mode of different modal features, such as strengthening the spatial semantic association between text and images, and the temporal semantic association between audio and text. m’ The global context key vector K is obtained by linear transformation of the initial features. global Generated by the mean, concatenation, or attention mechanism of each modal feature, it dynamically integrates global information from all modalities to represent the overall semantic context of the current sample, such as key words in text, key objects in images, and dominant frequencies in audio. Dynamic weighted fusion of multimodal data is achieved through a combination of training parameters + fixed hyperparameters + dynamic context: trainable parameters learn modal interaction patterns from the data; fixed hyperparameters regulate the mathematical properties of weight distribution; and dynamic context ensures that weight distribution aligns with the global semantics of the specific input, enabling the model to intelligently focus on key modalities based on the input content, suppressing redundant information and providing a more accurate semantic weighting foundation for cross-modal fusion, fundamentally improving the representational power and task adaptability of multimodal features.

[0070] In one embodiment, constructing a cross-modal semantic graph network based on semantically enhanced feature vectors includes:

[0071] S401, performing node space topology mapping based on the semantically enhanced feature vector to obtain a graph node set with modality labels;

[0072] S402, performing cross-modal cosine similarity calculation based on the graph node set to generate an inter-modal correlation edge weight matrix;

[0073] S403, performing three-hop neighborhood subgraph partitioning based on the weight matrix, and aggregating neighborhood features through a graph attention network to generate an updated node feature vector;

[0074] S404: Based on the updated node feature vector, adversarial feature completion is performed on the nodes containing missing modalities to generate a cross-modal semantic graph network.

[0075] Specifically, semantically enhanced feature vectors are mapped into modality-labeled graph nodes, the fundamental building blocks of graph networks. Each node corresponds to a multimodal sample, and its attributes include: a semantically enhanced feature vector carrying cross-modal semantic association information; and a set of modal labels marking the presence of the text, image, or audio modalities, with a label of 1 if present and 0 if missing. Converting high-dimensional semantic features into computable nodes within a graph structure provides structured input for subsequent association analysis. Modal labels explicitly annotate data integrity and provide goal-oriented completion for missing modalities. Cosine similarity quantifies the strength of semantic associations between nodes by measuring the cosine of the angle between feature vectors. The closer the cosine of the angle is to 1, the more semantically similar they are. For example, the text "dog" forms high-weighted edges with images of dogs and audio samples of dog barking in the matrix, while edges with unrelated samples, such as images of cats, form low-weighted edges. This matrix is used to construct the edge structure of the graph network, connecting nodes based on semantic similarity. This provides a weighted basis for subsequent neighborhood aggregation, ensuring that the model prioritizes semantically related neighboring nodes during feature propagation. Taking each node as the center, all adjacent nodes within a three-hop radius (i.e., nodes reachable via at most three edges) are extracted to form a local subgraph. This focuses on semantically closely related clusters of samples, avoiding the high complexity of global graph computation. When aggregating neighborhood features, the graph attention network dynamically assigns importance to neighboring nodes based on an edge weight matrix. The updated node feature vector is generated by combining the node's modality label, such as prioritizing the aggregation of neighboring features containing the target missing modality. This ensures that each node's features not only contain its own information but also incorporate cross-modal semantic information from its neighbors. This enhances the global semantic consistency of features. For example, for image nodes without audio, acoustic correlation information can be indirectly obtained by aggregating features from similar image nodes with audio. For nodes that still contain missing modalities after the update, a generative adversarial network architecture can be used for feature completion. This approach uses the node's non-missing modal features and aggregated neighborhood features as input to generate candidate features for the missing modality. For example, audio features for thunder are generated from textual thunder and image lightning features. The generated features are then evaluated to determine whether they conform to the distribution patterns of the actual modalities, such as the spatial coherence of image features and the frequency plausibility of audio features. Through adversarial training, the generated missing features are forced to conform to neighborhood semantic associations—that is, to be consistent with the existing modal features of the node, closely adhering to the statistical characteristics of the actual modality. This avoids the generation of unreasonable pseudo-features and generates a cross-modal semantic graph network containing complete modal information. Through a closed-loop process of node modeling → associated edge building → neighborhood aggregation → adversarial completion, multimodal data is transformed into a semantically associated graph structure. Leveraging the local connectivity and global semantic propagation capabilities of the graph network, intelligent inference of missing modalities is achieved.Three-hop neighborhood partitioning balances computational efficiency and semantic relevance, graph attention aggregation ensures the targeted propagation of features, and adversarial completion guarantees the authenticity and semantic consistency of missing features. It solves the defect of traditional methods that missing modality completion relies on simple interpolation or neglect processing, provides a complete and semantically consistent feature foundation for subsequent cross-modal retrieval, and improves the robustness and accuracy of multimodal data processing.

[0076] In one embodiment, the method further comprises:

[0077] S501, using a heat map generation algorithm to generate heat map data corresponding to the semantic association matrix;

[0078] S502, performing image segmentation and coordinate transformation on the heat map data to generate a binary attention mask;

[0079] S503, performing a Hadamard product operation on the attention mask and the semantic association matrix to obtain an optimized matrix for strengthening association;

[0080] S504 , calculating the overlap ratio between the enhanced region and the marked region in the optimization matrix, and retraining the feature space mapping function when the overlap ratio is lower than a preset threshold.

[0081] For example, a heatmap generation algorithm, such as a normalized grayscale / color mapping algorithm, can be used to convert a semantic association matrix (a numerical matrix that records the strength of semantic associations between text, image, and audio features) into visual heatmap data. By mapping strongly associated elements in the matrix to warm tones, such as red, and weakly associated elements to cool tones, such as blue, a heatmap can be generated that intuitively reflects the distribution of cross-modal semantic associations. For example, the word vector corresponding to the text "cat" and the visual feature area of the cat in the image will be highlighted red in the heatmap. This conversion of abstract numerical associations into visually perceptible spatial distributions provides visualization input, facilitating the explicit location of key semantically associated regions using image processing techniques. The heatmap can be processed using threshold segmentation or semantic segmentation models to extract regions of interest with high association. The pixel coordinates of the heatmap are mapped back to the dimensional coordinates of the semantic association matrix. For example, the (i, j)th pixel in the heatmap is mapped to the (i, j)th element in the matrix, generating a binary attention mask consistent with the matrix dimensions. By explicitly defining the semantically associated locations that require reinforcement through image processing techniques, this provides structured guidance for subsequent matrix operations and avoids blindly reinforcing irrelevant regions. The Hadamard product of the binary attention mask and the original semantic association matrix is used to obtain an optimized matrix. Through mask guidance, key semantic associations, such as cross-modal correspondences defined by manual annotation or prior knowledge, are targeted and enhanced, suppressing redundant associations. This allows the optimized matrix to focus more on core task-related semantic interactions, such as strengthening the association between object name and visual features in retrieval tasks. The overlap ratio between the enhanced regions in the optimized matrix (i.e., regions with a mask of 1) and manually annotated regions with correct semantic associations, such as cross-modal correspondences defined by domain knowledge, is calculated. If the overlap ratio falls below a preset threshold, indicating that the current feature space mapping function fails to accurately capture the target semantic associations, a retraining mechanism is triggered. This forms a closed loop of visualization guidance → targeted enhancement → effect verification → model calibration. Quantitative indicators are used to ensure the accuracy of the semantic association matrix, avoiding association bias caused by relying solely on model autonomous learning, and improving the interpretability and reliability of cross-modal semantic analysis.

[0082] In one embodiment, raw data of text, image, and audio are obtained, and initial feature sets of each modality are extracted using a deep neural network, including:

[0083] S601, based on a preset multilingual word embedding matrix, using a gating unit to dynamically select an embedding vector that matches the input language to obtain a multilingual text feature set;

[0084] S602, based on the multilingual text feature set, using a cross-language semantic alignment loss function to constrain the latent space distribution and generate a cross-language unified semantic representation;

[0085] S603: Based on the cross-language unified semantic representation and the language weight coefficient output by the language identification branch, a gated residual network is used to perform feature fusion to obtain an optimized multilingual text feature set.

[0086] Specifically, a pre-trained word embedding matrix covering multiple languages, such as Chinese, English, and Japanese, can be used. Each language corresponds to a separate embedding space, for example, the Chinese word "cat" corresponds to the vector "ezh" and the English word "cat" corresponds to "een." The input layer uses a language identification module, such as character encoding and word frequency statistics, to determine the language of the input text and generate a language identifier. Based on the language identifier, the gating unit selects the corresponding language embedding vector from the multilingual word embedding matrix. For example, if the input is the Chinese sentence "cat is running," the gating unit extracts the vectors corresponding to the Chinese words "cat," "in," and "run" from the Chinese word embedding matrix to form a multilingual text feature set. This avoids training a separate model for each language and dynamically selects embedding vectors, enabling rapid expansion to new languages and reducing model deployment costs. Pre-trained language-specific embedding vectors, such as Chinese character embeddings and English word embeddings, can be directly used to capture differences in grammatical structure between languages, such as word segmentation boundaries in Chinese and tense variations in English. Text features from different languages, such as the Chinese word "cat" and the English word "cat," are mapped into a unified latent space. Through contrastive learning or adversarial training, semantically equivalent cross-lingual texts are forced to be close in the latent space, while semantically different texts are forced to be separated. Divergence metrics can be used to constrain the distribution consistency of features in different languages in the latent space, for example, forcing the mean and variance of Chinese and English features to be close, eliminating feature drift between languages. An independent language identification branch is used to classify the input text into a specific language, outputting language weights. For purely Chinese text, the gated vector is biased to retain Chinese features, such as the stroke structure of Chinese characters, while integrating unified semantics, such as the universal concept of "cat." For mixed Chinese and English text, such as "cat is running," the fusion ratio of Chinese and English features is dynamically adjusted based on the language weight coefficient to ensure that the semantics of both languages are correctly captured. A residual connection mechanism allows the model to retain the original feature path for low-quality multilingual text, such as those containing spelling errors or mixed languages, to avoid semantic distortion. Through contrastive loss constraints and residual fusion, multilingual text is ensured to share a unified semantic space while preserving language specificity. This provides semantically consistent, high-quality text input for subsequent cross-modal fusion, such as with image and audio features. This fundamentally addresses the core issues of semantic fragmentation and difficulty in cross-lingual adaptation in traditional multilingual processing.

[0087] In one embodiment, before aggregating neighborhood features through a graph attention network, the method includes:

[0088] S701, based on the local outlier factor value of the node characteristics, the abnormal node set is obtained by using the standard deviation threshold detection;

[0089] S702, based on the abnormal node set, using a dual-channel processing structure to generate the original feature stream and the noise reduction feature stream in parallel;

[0090] In step S703, based on the original feature stream and the denoised feature stream, a gated fuser is used to perform dynamic weighting to obtain a preprocessed node feature vector. The preprocessed node feature vector is used as the input of the graph attention network to aggregate neighborhood features.

[0091] For example, the Local Outlier Factor (LOF) is a metric used to measure the degree of abnormality of a data point relative to its local neighborhood. For each node in the graph, its LOF value reflects the relative density of data points around the node compared to its own density. If a node's LOF value is much greater than 1, it indicates that the node is isolated relative to its neighborhood and may be an outlier. This can be achieved by calculating the mean and standard deviation of the LOF values of all nodes. A threshold based on the standard deviation is set, such as the mean plus a certain multiple of the standard deviation. Nodes with LOF values greater than the threshold are identified as outliers, resulting in a set of outliers. By identifying outliers, these abnormal data, which may affect model performance, can be specially handled in subsequent processing to avoid negatively impacting the aggregation process of the graph attention network. The dual-channel processing structure has two parallel channels for processing node features: for the original feature stream, the original node features can be used directly without any modification. For non-outlier nodes, these features can directly reflect their true information; for outlier nodes, retaining the original features can provide a reference for subsequent fusion. For the denoised feature stream, denoising is performed on the features of anomalous nodes. Denoising techniques such as graph-based smoothing and principal component analysis can be used to remove noise and anomalous components from the features of anomalous nodes, making them more consistent with the distribution of normal data. For non-anomalous nodes, their features can remain unchanged or undergo slight smoothing during the denoising process. By generating the original feature stream and the denoised feature stream in parallel, the negative impact of anomalous nodes can be removed while preserving the original information, providing more options for subsequent feature fusion. The gated fuser is a mechanism that dynamically adjusts the weights of the original and denoised feature streams based on the node's features and state. It learns the appropriate weighting of the original and denoised features for each node. For each node, the gated fuser calculates weights for the original and denoised feature streams based on its features and location within fraction (LOF) value. For example, for normal nodes with smaller LOF values, the original feature stream is given a higher weight; for anomalous nodes with larger LOF values, the denoised feature stream is given a higher weight. The two feature streams are weighted according to the calculated weights to produce the preprocessed node feature vector. This dynamic weighted fusion approach adaptively selects between original and denoised features based on the degree of node anomaly, effectively reducing the interference of abnormal nodes on the subsequent graph attention network aggregation process while preserving useful information. The resulting preprocessed node feature vector serves as input for the graph attention network's aggregated neighborhood features, helping to improve the network's performance and stability. By identifying abnormal nodes, processing features in parallel, and dynamically weighted fusion, graph node features are preprocessed, providing more reliable input for the subsequent graph attention network's neighborhood feature aggregation.

[0092] This semantic understanding-driven cross-modal information fusion and retrieval method systematically addresses three major technical bottlenecks of traditional multimodal processing through the following innovative designs: First, it uses dynamic attention weight allocation and high-dimensional semantic association analysis to achieve deep semantic fusion, overcoming the semantic fragmentation problem of shallow feature splicing. Second, it constructs a cross-modal semantic graph network combined with an adversarial feature completion mechanism and graph node exception handling to effectively repair missing modalities and maintain semantic consistency. Furthermore, it establishes precise cross-modal semantic alignment through a comparative learning framework and heatmap-guided association reinforcement. Furthermore, multilingual gated fusion expands the cross-language adaptability of text modalities, and node preprocessing enhances the robustness of the graph network. These technologies synergistically form a closed-loop optimization loop, enabling deep semantic fusion of cross-modal features, intelligent repair of missing information, and precise matching of retrieval results. This improves the depth of semantic understanding, the accuracy of missing modal completion, and the average precision of cross-modal retrieval, addressing industry pain points such as insufficient semantic fusion, weak missing information handling, and suboptimal retrieval performance.

[0093] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0094] Based on the same inventive concept, the embodiment of the present application also provides a semantic understanding-driven cross-modal information fusion and retrieval system for realizing the above-mentioned semantic understanding-driven cross-modal information fusion and retrieval method. The implementation scheme for solving the problem provided by this system is similar to the implementation scheme recorded in the above-mentioned method. Therefore, the specific limitations in one or more embodiments of a semantic understanding-driven cross-modal information fusion and retrieval system provided below can be found in the above-mentioned limitations on a semantic understanding-driven cross-modal information fusion and retrieval method, which will not be repeated here.

[0095] In an exemplary embodiment, Figure 2 As shown, a semantic understanding-driven cross-modal information fusion and retrieval system is provided, including:

[0096] Multimodal feature extraction module 11, used to obtain original data of text, image and audio, and extract the initial feature set of each modality through deep neural network;

[0097] Dynamic attention fusion module 12, used to dynamically assign weight coefficients to each modality based on the attention mechanism, and perform weighted fusion on the initial feature set to obtain a cross-modal fusion feature representation;

[0098] A semantic association enhancement module 13 is configured to extract high-dimensional semantic association features from the cross-modal fusion feature representation through a cross-modal semantic association analysis model to generate a semantic enhancement feature vector;

[0099] A graph network completion module 14 is used to construct a cross-modal semantic graph network based on the semantic enhancement feature vector, and to complete the missing modal features to generate an optimized multimodal feature set;

[0100] The comparative retrieval optimization module 15 is used to input the optimized multimodal feature set and the query sample into the comparative learning model, calculate the semantic similarity score, and generate a ranked list of cross-modal retrieval results based on the score.

[0101] In one embodiment, the semantic association enhancement module 13 is further configured to:

[0102] Detect the semantic coverage of cross-modal fusion feature representation. When the semantic coverage is lower than the preset threshold, the following semantic enhancement process is triggered:

[0103] The fused feature representation is mapped to a high-dimensional feature space through a feature space mapping function to extract cross-modal semantic associations and generate a semantic association matrix;

[0104] The fused feature representation is orthogonally projected based on the semantic association matrix to generate a semantically enhanced feature vector.

[0105] In one embodiment, the dynamic attention fusion module 12 is further configured to determine the weight coefficient of each modality using the following weight calculation function:

[0106]

[0107] Among them, α m Represents the attention weight of modality m, with a value range of 0≤α m ≤1, and∑ m α m = 1, m∈{t,i,a} represents the modality identifier, representing text, image and audio respectively, τ represents the temperature coefficient, represents the activation function, γ represents the leakage coefficient, represents the trainable weight matrix, [Q m’ ||K global ] represents vector concatenation operation, Q m’ represents the query vector, K global Represents the global context key vector.

[0108] In one embodiment, the graph network completion module 14 is further configured to:

[0109] Perform node space topology mapping based on semantically enhanced feature vectors to obtain a set of graph nodes with modal labels;

[0110] Calculate cross-modal cosine similarity based on the graph node set and generate the inter-modal correlation edge weight matrix;

[0111] The three-hop neighborhood subgraph is divided based on the weight matrix, and the neighborhood features are aggregated through the graph attention network to generate the updated node feature vector;

[0112] Based on the updated node feature vectors, adversarial feature completion is performed on nodes with missing modalities to generate a cross-modal semantic graph network.

[0113] In one embodiment, the system further includes a semantic association enhancement training module for:

[0114] Use the heat map generation algorithm to generate heat map data corresponding to the semantic association matrix;

[0115] Perform image segmentation and coordinate transformation on the heat map data to generate a binary attention mask;

[0116] Perform Hadamard product operation on the attention mask and the semantic association matrix to obtain the optimized matrix of enhanced association;

[0117] The overlap ratio between the enhanced region and the labeled region in the optimization matrix is calculated. When the overlap ratio is lower than the preset threshold, the feature space mapping function is retrained.

[0118] In one embodiment, the multimodal feature extraction module 11 is further configured to:

[0119] Based on the preset multilingual word embedding matrix, the gate unit is used to dynamically select the embedding vector that matches the input language to obtain a multilingual text feature set;

[0120] Based on a multilingual text feature set, a cross-language semantic alignment loss function is used to constrain the latent space distribution and generate a unified cross-language semantic representation.

[0121] Based on the unified cross-language semantic representation and the language weight coefficient output by the language identification branch, a gated residual network is used for feature fusion to obtain an optimized multilingual text feature set.

[0122] In one embodiment, the graph network completion module 14 is further configured to:

[0123] Based on the local outlier factor value of node characteristics, the standard deviation threshold is used to detect the abnormal node set;

[0124] Based on the abnormal node set, the dual-channel processing structure is used to generate the original feature stream and the denoised feature stream in parallel;

[0125] Based on the original feature stream and the denoised feature stream, a gated fuser is used to perform dynamic weighting to obtain the preprocessed node feature vector, which is used as the input of the graph attention network to aggregate neighborhood features.

[0126] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the semantic understanding-driven cross-modal information fusion and retrieval method as described above are implemented.

[0127] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0129] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A semantic understanding driven cross-modal information fusion and retrieval method, characterized by: The method comprises: Obtain raw data of text, images, and audio, and extract the initial feature set of each modality through deep neural networks; Dynamically assigning weight coefficients to the modalities based on the attention mechanism, and performing weighted fusion on the initial feature set to obtain a cross-modal fusion feature representation; extracting high-dimensional semantic association features from the cross-modal fusion feature representation through a cross-modal semantic association analysis model to generate a semantically enhanced feature vector; Constructing a cross-modal semantic graph network based on the semantically enhanced feature vector, and completing the missing modal features to generate an optimized multimodal feature set; The optimized multimodal feature set is compared with the query sample input into the learning model, a semantic similarity score is calculated, and a ranked list of cross-modal retrieval results is generated based on the score.

2. The method according to claim 1, characterized in that Generating a semantic enhancement feature vector includes: Detecting the semantic coverage of the cross-modal fusion feature representation, and triggering the following semantic enhancement process when the semantic coverage is lower than a preset threshold: Mapping the fused feature representation to a high-dimensional feature space through a feature space mapping function, extracting cross-modal semantic associations and generating a semantic association matrix; An orthogonal projection is performed on the fused feature representation based on the semantic association matrix to generate a semantically enhanced feature vector.

3. The method according to claim 2, characterized in that The dynamic allocation of the weight coefficients of the modalities based on the attention mechanism includes determining the weight coefficients of the modalities by the following weight calculation function: Among them, α m Represents the attention weight of modality m, with a value range of 0≤α m ≤1, and∑ m α m = 1, m∈{t,i,a} represents the modality identifier, representing text, image and audio respectively, τ represents the temperature coefficient, represents the activation function, γ represents the leakage coefficient, represents the trainable weight matrix, [Q m′ ||K global ] represents vector concatenation operation, Q m’ represents the query vector, K global Represents the global context key vector.

4. The method according to claim 1, wherein The constructing of a cross-modal semantic graph network based on the semantically enhanced feature vector includes: Performing node space topology mapping based on the semantically enhanced feature vector to obtain a graph node set with modality labels; Performing cross-modal cosine similarity calculation based on the graph node set to generate an inter-modal correlation edge weight matrix; Perform three-hop neighborhood subgraph partitioning based on the weight matrix, aggregate neighborhood features through a graph attention network, and generate an updated node feature vector; Based on the updated node feature vectors, adversarial feature completion is performed on nodes containing missing modalities to generate a cross-modal semantic graph network.

5. The method according to claim 2, characterized in that The method further comprises: Generate heat map data corresponding to the semantic association matrix using a heat map generation algorithm; Performing image segmentation and coordinate transformation on the heat map data to generate a binary attention mask; Performing a Hadamard product operation on the attention mask and the semantic association matrix to obtain an optimized matrix for strengthening association; The overlap rate between the enhanced region and the marked region in the optimization matrix is calculated, and when the overlap rate is lower than a preset threshold, the feature space mapping function is retrained.

6. The method according to claim 1, characterized in that The method of obtaining the original data of text, image and audio, and extracting the initial feature set of each modality through a deep neural network, includes: Based on the preset multilingual word embedding matrix, the gate unit is used to dynamically select the embedding vector that matches the input language to obtain a multilingual text feature set; Based on the multilingual text feature set, a cross-language semantic alignment loss function is used to constrain the latent space distribution and generate a cross-language unified semantic representation; Based on the cross-language unified semantic representation and the language weight coefficient output by the language identification branch, a gated residual network is used to perform feature fusion to obtain an optimized multilingual text feature set.

7. The method according to claim 4, characterized in that Before aggregating neighborhood features through the graph attention network, the following steps are included: Based on the local outlier factor value of node characteristics, the standard deviation threshold is used to detect the abnormal node set; Based on the abnormal node set, a dual-channel processing structure is used to generate an original feature stream and a noise reduction feature stream in parallel; Based on the original feature stream and the denoised feature stream, a gated fuser is used to perform dynamic weighting to obtain a preprocessed node feature vector, which is used as the input of the graph attention network to aggregate neighborhood features.

8. A semantic understanding driven cross-modal information fusion and retrieval system, characterized by: The system comprises: Multimodal feature extraction module, which is used to obtain raw data of text, image and audio, and extract the initial feature set of each modality through deep neural network; A dynamic attention fusion module is used to dynamically assign weight coefficients to the modalities based on the attention mechanism and perform weighted fusion on the initial feature set to obtain a cross-modal fusion feature representation; a semantic association enhancement module, configured to extract high-dimensional semantic association features from the cross-modal fusion feature representation through a cross-modal semantic association analysis model, and generate a semantic enhancement feature vector; A graph network completion module is used to construct a cross-modal semantic graph network based on the semantic enhancement feature vector, and to complete the missing modal features to generate an optimized multimodal feature set; The comparative retrieval optimization module is used to input the optimized multimodal feature set and the query sample into the comparative learning model, calculate the semantic similarity score, and generate a ranked list of cross-modal retrieval results based on the score.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images