Artificial intelligence multi-modal semantic alignment method and system based on knowledge graph enhancement

Abstract

The application provides an AI multi-modal semantic alignment method and system based on knowledge graph enhancement, and relates to the field of multi-modal data processing. The method comprises: acquiring multi-modal original data, inputting multi-modal pre-processed data into a lightweight feature extraction model, extracting multi-modal feature vectors, and performing quality evaluation to obtain multi-modal feature vector quality evaluation results; based on a preset knowledge graph, performing semantic label annotation on the multi-modal feature vectors and constructing semantic association relationships between different modalities; based on the semantic association relationships and the multi-modal feature vector quality evaluation results, determining semantic association weights and performing semantic mapping processing on the modal feature vectors to obtain associated multi-modal feature vectors; based on the associated multi-modal feature vectors, calculating the semantic similarity between different modal feature vectors; performing multi-modal semantic alignment according to the semantic similarity to obtain aligned multi-modal feature vectors.

Description

Technical Field

[0001] This application relates to the field of multimodal data processing, and in particular to an AI multimodal semantic alignment method and system based on knowledge graph enhancement. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, multimodal technology has become one of the core research directions in the AI ​​field, playing a crucial role in achieving effective interaction and semantic association between data from different modalities. Currently, existing AI multimodal semantic alignment technologies largely rely on pure algorithmic mapping or transfer learning based on large pre-trained models, failing to incorporate domain-specific knowledge associations. Furthermore, these technologies suffer from large model parameter scales and high inference complexity, making them difficult to adapt to low-computing-power scenarios such as embedded devices. Therefore, a method is urgently needed to address the technical challenges of low multimodal semantic alignment accuracy and difficulty in deployment under low-computing-power conditions. Summary of the Invention

[0003] This application provides an AI multimodal semantic alignment method and system based on knowledge graph enhancement, which solves the technical problems of low multimodal semantic alignment accuracy and difficulty in deployment under low computing power environment in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution: Firstly, a knowledge graph-based AI multimodal semantic alignment method is provided, comprising: acquiring multimodal raw data and preprocessing the raw data to obtain multimodal preprocessed data; the multimodal raw data includes image data, text data, and speech data; inputting the multimodal preprocessed data into a lightweight feature extraction model to extract multimodal feature vectors, and performing quality evaluation on the multimodal feature vectors to obtain multimodal feature vector quality evaluation results; wherein, the lightweight feature extraction model includes an image feature extraction sub-model, a text feature extraction sub-model, and a speech feature extraction sub-model optimized by model pruning and / or quantization. The audio feature extraction sub-model performs semantic labeling on multimodal feature vectors based on a pre-defined knowledge graph, and retrieves associations in the knowledge graph based on the semantic labels to construct semantic associations between different modalities. Based on the semantic associations and the quality assessment results of the multimodal feature vectors, the semantic association weights are determined, and semantic mapping is performed on each modal feature vector to obtain the associated multimodal feature vectors. Based on the associated multimodal feature vectors, the semantic similarity between different modal feature vectors is calculated. Multimodal semantic alignment is performed based on the semantic similarity to obtain the aligned multimodal feature vectors.

[0005] In conjunction with the first aspect mentioned above, in one possible implementation, after obtaining the semantically aligned multimodal data, the method further includes: determining the corresponding modality combination type based on the modality type and / or the number of modalities in the input original multimodal data; selecting the corresponding lightweight feature extraction sub-model based on the modality combination type, and adjusting the model parameters of the lightweight feature extraction sub-model, including pruning ratio and / or quantization accuracy; adjusting the weights of each modality feature vector in the semantic mapping process based on the multimodal feature vector quality evaluation results; adjusting the semantic association weights and / or attention weights in the semantic mapping process based on the modality combination type and semantic association relationship; and dynamically adjusting the semantic alignment similarity threshold based on the semantic similarity calculation results.

[0006] In conjunction with the first aspect mentioned above, in one possible implementation, the image feature extraction sub-model is a pruned ResNet model, the text feature extraction sub-model is a lightweight BERT model, and the speech feature extraction sub-model is a pruned LSTM model.

[0007] In conjunction with the first aspect mentioned above, in one possible implementation, multimodal preprocessed data is input into a lightweight feature extraction model to extract multimodal feature vectors, including: inputting image preprocessed data into an image feature extraction sub-model to extract image feature vectors; inputting text preprocessed data into a text feature extraction sub-model to extract text feature vectors; inputting speech preprocessed data into a speech feature extraction sub-model to extract speech feature vectors; and performing dimension alignment processing on the image feature vectors, text feature vectors, and speech feature vectors to obtain multimodal feature vectors.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, the multimodal feature vector quality assessment result includes: assessing the image feature vector quality based on the image data's sharpness index, contrast index, and / or feature response intensity to obtain an image feature quality score; assessing the speech feature vector quality based on the speech data's signal-to-noise ratio, speech integrity, and / or speech feature stability to obtain a speech feature quality score; assessing the text feature vector quality based on the text data's semantic integrity, keyword coverage, and / or text length consistency to obtain a text feature quality score; and normalizing the image feature quality score, speech feature quality score, and text feature quality score to obtain the multimodal feature vector quality assessment result.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the construction of the pre-defined knowledge graph includes: extracting semantic information from multimodal data and performing structured processing on the semantic information to obtain multimodal semantic expressions; the multimodal semantic expressions include: semantic expressions for image data, semantic expressions for text data, and semantic expressions for speech data; based on the multimodal semantic expressions, mapping the semantic expressions of different modalities to the same semantic tag system to construct a unified semantic tag set; based on the unified semantic tag set, establishing multimodal semantic associations between semantic tags of different modalities, the multimodal semantic associations include semantic correspondences and semantic mappings; based on the multimodal semantic associations, constructing a knowledge graph oriented towards multimodal semantic alignment; and determining semantic association weights based on the association paths, relationship types, and / or co-occurrence strengths between semantic tags in the knowledge graph.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, based on semantic association relationships and multimodal feature vector quality assessment results, semantic association weights are determined, and semantic mapping processing is performed on each modal feature vector to obtain the associated multimodal feature vector. This includes: obtaining association path information and relationship type information between semantic labels of different modalities based on semantic association relationships; calculating initial semantic association weights based on the path length of the association path, the relationship type weight, and the co-occurrence frequency of semantic labels; normalizing the initial semantic association weights and, in conjunction with the multimodal feature vector quality assessment results, weighting and adjusting the semantic association weights to obtain target semantic association weights; constructing a multimodal attention weight matrix based on the target semantic association weights; and performing weighted mapping and fusion on each modal feature vector based on the multimodal attention weight matrix to obtain the associated multimodal feature vector.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, the semantic similarity between different modal feature vectors is calculated based on the associated multimodal feature vectors, including: calculating the local semantic similarity and global semantic similarity between different modalities based on the associated multimodal feature vectors; wherein, the local semantic similarity is calculated based on the similarity between each dimension or subspace in the feature vector, and the global semantic similarity is calculated based on the similarity of the overall feature vector; the local semantic similarity and global semantic similarity are fused to obtain the semantic similarity between modalities.

[0012] In one possible implementation, local semantic similarity Satisfy the following formula:

[0013] in, , These represent the sub-vectors of different modal feature vectors in the i-th semantic subspace; These are the subspace weight coefficients; This represents the path distance between different modal semantic tags in the knowledge graph within the corresponding semantic subspace. This is the path distance attenuation coefficient; This represents the quality adjustment factor determined based on the multimodal feature vector quality assessment results; n is the number of subspace partitions. In one possible implementation, global semantic similarity Satisfy the following formula:

[0014] Where A and B represent the feature vectors of different modalities after association.

[0015] Secondly, a knowledge graph-based AI multimodal semantic alignment system is provided, characterized in that the system includes: a data preprocessing module, a lightweight feature processing module, a knowledge graph-assisted modeling module, a semantic mapping module, a similarity calculation module, and a semantic alignment module; wherein, the data preprocessing module is used to acquire multimodal raw data and preprocess the raw data to obtain multimodal preprocessed data; the multimodal raw data includes image data, text data, and speech data; the lightweight feature processing module is used to input the multimodal preprocessed data into a lightweight feature extraction model, extract multimodal feature vectors, and perform quality evaluation on the multimodal feature vectors to obtain multimodal feature vector quality evaluation results; wherein, the lightweight feature extraction model includes models pruned and / or quantized optimized... The system includes image feature extraction sub-models, text feature extraction sub-models, and speech feature extraction sub-models; a knowledge graph-assisted modeling module, used to semantically label multimodal feature vectors based on a pre-defined knowledge graph, and to retrieve associations in the knowledge graph based on the semantic labels, constructing semantic associations between different modalities; a semantic mapping module, used to determine semantic association weights based on semantic associations and multimodal feature vector quality assessment results, and to perform semantic mapping processing on each modal feature vector to obtain associated multimodal feature vectors; a similarity calculation module, used to calculate the semantic similarity between different modal feature vectors based on the associated multimodal feature vectors; and a semantic alignment module, used to perform multimodal semantic alignment based on semantic similarity, obtaining aligned multimodal feature vectors.

[0016] This application provides an AI multimodal semantic alignment method and system based on knowledge graph enhancement. By employing a lightweight feature extraction model that has been pruned and / or quantized optimized, the method significantly reduces the model parameter size and computational resource consumption, enabling the solution to be deployed in low-computing-power environments such as embedded devices. Simultaneously, by introducing a pre-defined knowledge graph for semantic tagging and relation retrieval, the method uncovers deep semantic relationships between different modalities and dynamically adjusts semantic relationship weights based on quality assessment results. This addresses the technical problems of low multimodal semantic alignment accuracy and difficulty in deployment in low-computing-power environments in existing technologies.

[0017] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0018] Figure 1 A system architecture diagram of an AI multimodal semantic alignment method system based on knowledge graph enhancement provided for embodiments of this application; Figure 2 A flowchart illustrating a knowledge graph-enhanced AI multimodal semantic alignment method provided in this application embodiment; Figure 3 A flowchart illustrating another AI multimodal semantic alignment method based on knowledge graph enhancement provided in this application embodiment; Figure 4 A flowchart illustrating another AI multimodal semantic alignment method based on knowledge graph enhancement provided in this application embodiment; Figure 5 This is a flowchart illustrating another AI multimodal semantic alignment method based on knowledge graph enhancement provided in this application embodiment. Detailed Implementation

[0019] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0020] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0021] The knowledge graph-enhanced AI multimodal semantic alignment method provided in this application can be applied to, for example... Figure 1 In the knowledge graph-enhanced AI multimodal semantic alignment system shown, such as Figure 1 As shown, the system includes: a data preprocessing module 101, a lightweight feature processing module 102, a knowledge graph-assisted modeling module 103, a semantic mapping module 104, a similarity calculation module 105, and a semantic alignment module 106. The system includes a data preprocessing module 101, which acquires multimodal raw data and preprocesses it to obtain multimodal preprocessed data. The multimodal raw data includes image data, text data, and speech data. A lightweight feature processing module 102 inputs the multimodal preprocessed data into a lightweight feature extraction model to extract multimodal feature vectors and performs quality evaluation on the multimodal feature vectors to obtain the multimodal feature vector quality evaluation results. The lightweight feature extraction model includes image feature extraction sub-models, text feature extraction sub-models, and speech feature extraction sub-models optimized through model pruning and / or quantization. A knowledge graph-assisted modeling module 103 is used to... A pre-defined knowledge graph is used to semantically label multimodal feature vectors and to retrieve associations in the knowledge graph based on the semantic labels, thereby constructing semantic associations between different modalities. A semantic mapping module 104 is used to determine semantic association weights based on semantic associations and multimodal feature vector quality assessment results, and to perform semantic mapping processing on each modal feature vector to obtain associated multimodal feature vectors. A similarity calculation module 105 is used to calculate the semantic similarity between different modal feature vectors based on the associated multimodal feature vectors. A semantic alignment module 106 is used to perform multimodal semantic alignment based on semantic similarity to obtain aligned multimodal feature vectors.

[0022] To address the technical problems of low multimodal semantic alignment accuracy and difficulty in deployment under low computing power in existing technologies, this application provides an AI multimodal semantic alignment method based on knowledge graph enhancement. Figure 2 This is a flowchart illustrating the knowledge graph-based AI multimodal semantic alignment method provided in an embodiment of this application. Figure 2 As shown, the method includes: S201. Obtain the original multimodal data and preprocess the original data to obtain multimodal preprocessed data.

[0023] Multimodal raw data includes image data, text data, and voice data, while multimodal preprocessed data refers to data that has undergone standardization.

[0024] In one possible implementation, the system acquires raw data through a multimodal acquisition device and employs different preprocessing strategies for different modalities. For image data, the system uses a Gaussian filtering algorithm to remove noise and normalizes pixel values ​​to a preset range, while adjusting the image size to meet the model input requirements. For text data, the system performs word segmentation and removes stop words, converting the text into sequence encoding. For speech data, the system uses a Wiener filtering algorithm for noise reduction and extracts Mel-frequency cepstral coefficients as core features. It should be understood that the preprocessing process is not limited to the above methods and may also include data augmentation, missing value imputation, etc., with the aim of eliminating noise and redundancy in the data and improving the accuracy of subsequent feature extraction.

[0025] It should be noted that the sources of multimodal raw data are wide-ranging, including real-time data streams and historical data stored in storage media. This embodiment does not impose any specific limitations on these sources.

[0026] This step, by standardizing the original multimodal data through preprocessing, effectively removes noise interference from the data, unifies the data format, and provides a high-quality data foundation for subsequent feature extraction, thereby improving the overall accuracy of semantic alignment.

[0027] S202. Input the multimodal preprocessed data into the lightweight feature extraction model, extract the multimodal feature vectors, and evaluate the quality of the multimodal feature vectors to obtain the multimodal feature vector quality evaluation results.

[0028] The lightweight feature extraction model includes image feature extraction sub-models, text feature extraction sub-models, and speech feature extraction sub-models that have been optimized through model pruning and / or quantization. The multimodal feature vector quality evaluation results are used to characterize the quality of the extracted feature vectors.

[0029] In one possible implementation, the system inputs the preprocessed data into the corresponding lightweight sub-models. Lightweight processing includes model compression techniques such as model pruning and quantization. These methods significantly reduce the model's parameter size and computational load, making it adaptable to low-computing-power environments such as embedded devices. After extracting feature vectors, the system further evaluates the feature quality. For example, for image feature vectors, it can be evaluated based on sharpness metrics; for speech feature vectors, it can be evaluated based on signal-to-noise ratio. This quality evaluation mechanism can identify features severely affected by noise or with lost information, preventing low-quality features from interfering with subsequent semantic alignment processes.

[0030] Based on the above steps, this step extracts features using a lightweight model and combines it with quality assessment. This not only achieves efficient feature extraction in a low-computing-power environment, but also ensures the effectiveness of the features through a quality screening mechanism, solving the problems of high model resource consumption and poor alignment robustness in existing technologies.

[0031] S203. Based on the preset knowledge graph, semantic labels are applied to the multimodal feature vectors, and the association relationships are retrieved in the knowledge graph according to the semantic labels to construct semantic association relationships between different modalities.

[0032] Knowledge graphs are data structures that store semantic knowledge in the form of graphs. They include entities, attributes, and relationships between entities. Semantic associations are used to represent the corresponding relationships between different modalities of data at the semantic level.

[0033] In one possible implementation, the system pre-constructs a domain-specific knowledge graph that encompasses semantic tags and association rules for multiple modalities, including images, text, and speech. The system first maps the extracted feature vectors to the semantic space of the knowledge graph, labeling them with corresponding semantic tags. Then, using these tags as indexes, the system retrieves relevant nodes and edges in the knowledge graph, uncovering deep associations between tags from different modalities.

[0034] It should be noted that the construction of knowledge graphs can be done offline, or they can be dynamically updated and expanded based on actual application data to adapt to different application scenarios.

[0035] As an example, in intelligent interaction scenarios, knowledge graphs store semantic correspondences between gestures (such as "waving"), voice commands (such as "goodbye"), and text commands (such as "exit the system"). By retrieving the graph, the system can accurately identify that these three different modalities actually express the same semantic intent.

[0036] Based on the above steps, this step introduces a knowledge graph to construct semantic associations and uses prior knowledge to assist feature alignment, effectively solving the problems of large semantic deviations and high misalignment rates in pure algorithm mapping in existing technologies, and significantly improving the accuracy of multimodal semantic alignment.

[0037] S204. Based on the semantic association relationship and the quality assessment results of the multimodal feature vector, determine the semantic association weight, and perform semantic mapping processing on each modal feature vector to obtain the associated multimodal feature vector.

[0038] Among them, the semantic association weight reflects the importance of different modal features in the semantic mapping process, and the associated multimodal feature vector is a vector representation after weight adjustment and feature fusion.

[0039] In one possible implementation, the system comprehensively considers the strength of semantic relationships (such as the length of paths in the knowledge graph) and the quality scores of feature vectors to determine the final weights. For feature vectors with higher quality scores, the system assigns them higher weights; for modal pairs with shorter paths and stronger associations in the knowledge graph, the system also increases their interaction weights accordingly. The system uses the determined weights to construct a mapping matrix and performs weighted mapping and fusion processing on the feature vectors of each modality. This dynamic weight adjustment mechanism allows high-quality, strongly associated features to play a dominant role in the fusion process, suppressing the interference of low-quality or weakly associated features, thereby obtaining fused feature vectors with more accurate semantic expression.

[0040] It should be noted that semantic mapping can be implemented using attention mechanisms, weighted summation, or more complex fusion networks. The specific mapping method depends on the computational resource constraints and accuracy requirements of the application scenario.

[0041] Based on the above steps, this step combines quality assessment and graph association to determine weights and perform mapping, thereby achieving adaptive fusion of multimodal features and further improving the accuracy and robustness of feature representation.

[0042] S205. Based on the associated multimodal feature vectors, calculate the semantic similarity between different modal feature vectors.

[0043] Semantic similarity is a quantitative metric used to measure the closeness of feature vectors of different modalities in the semantic space.

[0044] In one possible implementation, the system maps the associated feature vectors to a unified semantic space and calculates the similarity between vectors using distance or similarity metrics. The system calculates local similarity (focusing on detailed features) and global similarity (focusing on overall distribution) separately, and then weights and fuses them to obtain the final semantic similarity. Local semantic similarity is calculated based on the similarity between different dimensions or subspaces within the feature vectors, while global semantic similarity is calculated based on the similarity of the entire feature vector. This similarity value objectively reflects whether different modalities describe the same semantic content, providing a reliable basis for subsequent alignment judgments.

[0045] Preferred local semantic similarity Satisfy the following formula:

[0046] in, , These represent the sub-vectors of different modal feature vectors in the i-th semantic subspace; These are the subspace weight coefficients; This represents the path distance between different modal semantic tags in the knowledge graph within the corresponding semantic subspace. This is the path distance attenuation coefficient; This represents the quality adjustment factor determined based on the multimodal feature vector quality assessment results; n is the number of subspace partitions. Preferably, global semantic similarity Satisfy the following formula:

[0047] Where A and B represent the feature vectors of different modalities after association.

[0048] Based on the above steps, by calculating similarity in a unified semantic space, the degree of semantic association between modalities is quantified, providing a measurable judgment standard for accurate semantic alignment.

[0049] S206. Perform multimodal semantic alignment based on semantic similarity to obtain the aligned multimodal feature vector.

[0050] The aligned multimodal feature vectors refer to the set of feature vectors that have been matched and confirmed to have consistent semantic labels or be located in the same semantic space.

[0051] In one possible implementation, the system presets a similarity threshold and compares the calculated semantic similarity with the threshold. If the similarity is higher than the threshold, the corresponding modality data is considered to have successfully aligned semantically, and the system associates and stores these feature vectors or outputs them to downstream task modules. If the similarity is lower than the threshold, the alignment is considered to have failed, and the system can discard the data or re-extract and map features. Through this judgment process, the system filters out semantically consistent multimodal data, achieving effective integration of multimodal information.

[0052] It should be noted that the similarity threshold can be dynamically adjusted according to the accuracy requirements of specific application scenarios. For example, a higher threshold can be set in scenarios with high accuracy requirements, while a lower threshold can be set in scenarios where recall is prioritized.

[0053] As an example, with a preset similarity threshold of 0.8, if the similarity between the image and the voice feature vector is 0.85, the system determines that the two are successfully aligned, and stores them together for subsequent intelligent alarms or interactive responses.

[0054] Based on the above steps, semantic alignment is achieved by using a similarity threshold, which effectively filters out semantically inconsistent data and ensures that the final output multimodal data has a high degree of semantic consistency, thus completing the entire process of multimodal semantic alignment.

[0055] Based on the above technical solutions, this embodiment lowers the deployment threshold by using a lightweight model, improves the semantic understanding depth by introducing prior knowledge using knowledge graphs, and optimizes the feature fusion process by combining quality assessment and dynamic weighting mechanisms, ultimately achieving high-precision, real-time multimodal semantic alignment in a low-computing-power environment.

[0056] In one possible implementation, combining Figure 2 ,like Figure 3 As shown, following S206, the AI ​​multimodal semantic alignment method based on knowledge graph enhancement provided in this application embodiment further includes the following S301 to S305: S301. Determine the corresponding mode combination type based on the mode type and / or number of modes in the input multimodal raw data.

[0057] The modality combination type refers to the category identifier of the modality set contained in the current input data, such as the "image, text" bimodal type or the "image, speech, text" trimodal type.

[0058] In one possible implementation, after acquiring the raw multimodal data, the system first parses the data stream to identify the currently active modal channels. Based on the identified modal types (e.g., image, speech) and their quantity (e.g., 2 or 3), the system queries a pre-defined modal combination configuration table to determine the modal combination type corresponding to the current data. This process enables the system to perceive changes in the input data, providing a basis for decision-making in subsequent differentiated processing.

[0059] It should be noted that the classification of modality combination types is not limited to the number of modalities. It can also be subdivided based on the correlation strength between modalities. For example, "image and text" can be classified as a strongly correlated combination, and "sensor data and voice" can be classified as a weakly correlated combination, so as to adopt different processing strategies.

[0060] Based on the above steps, by identifying modal combination types in real time, the system can perceive changes in the input environment, providing accurate input feature information for subsequent adaptive adjustments and avoiding resource waste caused by using a uniform processing mode for all inputs.

[0061] S302. Based on the modality combination type, select the corresponding lightweight feature extraction sub-model and adjust the model parameters of the lightweight feature extraction sub-model, including the pruning ratio and / or quantization accuracy.

[0062] In one possible implementation, the system pre-defines the mapping relationship between different modality combinations and model parameter configurations. When the modality combination type changes, the system automatically loads the corresponding lightweight feature extraction sub-model. If the number of modalities is small (e.g., bimodal), the system's computational load is low, and the pruning ratio can be appropriately reduced or the quantization accuracy increased to retain more model detail information and improve feature extraction accuracy. Conversely, if the number of modalities is large (e.g., trimodal or higher), the system's computational load increases, so the pruning ratio is increased or the quantization accuracy is reduced to ensure real-time performance. This dynamic adjustment mechanism achieves a balance between computational resources and processing accuracy.

[0063] It should be noted that model parameters can be adjusted offline pre-configured or dynamically fine-tuned online, depending on the computing power redundancy of the hardware platform.

[0064] Based on the above steps, by dynamically adjusting the model parameters according to the modal combination type, the system can maximize the use of computing resources to improve the quality of feature extraction while ensuring real-time processing, thus solving the problem that fixed model structures are difficult to adapt to diverse input scenarios.

[0065] S303. Based on the multimodal feature vector quality assessment results, adjust the weights of each modal feature vector in the semantic mapping process.

[0066] In the semantic mapping process, the weight refers to the proportion of different modal features in the fusion process.

[0067] In one possible implementation, the system obtains the quality evaluation results of the multimodal feature vectors obtained in Example 1 and uses them as the basis for weight adjustment. For modal features with higher quality scores, the system considers them to contain more reliable semantic information and therefore assigns them higher weights during semantic mapping; for modal features with lower quality scores, the system reduces their weights to reduce noise interference. This quality-based weight adjustment mechanism makes the fused feature vectors more likely to retain the semantic information of high-quality modalities.

[0068] Based on the above steps, by dynamically adjusting the mapping weights in conjunction with the quality assessment results, the negative impact of low-quality modal data on semantic alignment results is effectively suppressed, and the robustness of the system in complex environments is improved.

[0069] S304. Based on the modality combination type and semantic association relationship, adjust the semantic association weight and / or attention weight in the semantic mapping process.

[0070] Among them, semantic association weight reflects the association strength of semantic labels of different modalities in the knowledge graph, and attention weight is used to control the attention to local information during feature fusion.

[0071] In one possible implementation, the system determines the set of modalities to be computed based on the modality combination type and extracts the semantic relationships between the modalities within that set from the knowledge graph. If the modality combination type changes (e.g., from trimodal to bimodal), the system recalculates the association weights between the remaining modalities and reallocates the attention weights. For example, when a modality is missing, the attention weights originally assigned to that modality are reallocated to the remaining modalities to ensure the integrity of feature fusion.

[0072] Based on the above steps, by dynamically adjusting the association weights and attention weights according to the modality combination, it is ensured that the semantic mapping process can still accurately capture the association features between modalities even when modalities are missing or changed, thus improving the flexibility of the scheme.

[0073] S305. Based on the semantic similarity calculation results, dynamically adjust the semantic alignment similarity threshold.

[0074] The similarity threshold is the critical value for determining whether semantic alignment is successful.

[0075] In one possible implementation, the system monitors the distribution of semantic similarity calculation results and dynamically adjusts the threshold based on the distribution. If the currently calculated similarities are generally high, it indicates significant feature discrimination, and the system can appropriately increase the threshold to filter out edge matches and improve alignment accuracy. If the similarities are generally low, it indicates that the features are ambiguous or noisy, and the system can appropriately decrease the threshold to avoid missed detections and improve recall. This dynamic threshold mechanism allows the system to adapt to data inputs of varying quality.

[0076] Based on the above steps, by dynamically adjusting the similarity threshold, the system can ensure alignment accuracy while taking into account recall, thus avoiding the problem of poor adaptability of fixed thresholds when facing different data distributions.

[0077] Based on the above technical solution, this embodiment dynamically adjusts the model parameters, mapping weights, and alignment thresholds according to the type and number of input modalities, thereby achieving reasonable allocation of computing resources and flexible adaptation of alignment strategies, significantly improving the robustness and practicality of the solution in different modal combination scenarios.

[0078] It should be noted that this embodiment, based on the above embodiments, provides a detailed explanation of the specific architecture of the lightweight feature extraction model and the dimensional alignment processing of feature vectors. This embodiment focuses on describing how to achieve low-computing-power deployment through specific model compression methods, and how to solve the engineering problem of inconsistent feature vector dimensions across different modalities.

[0079] Specifically, the image feature extraction sub-model is a pruned ResNet model, the text feature extraction sub-model is a lightweight BERT model, and the speech feature extraction sub-model is a pruned LSTM model. Pruning refers to removing redundant neurons or connections from the neural network, and quantization refers to converting the model parameters from high-precision floating-point numbers to low-precision fixed-point numbers.

[0080] In one possible implementation, for the image feature extraction sub-model, the system employs a structured pruning strategy to prune channels in the convolutional layers of the ResNet model, removing redundant convolutional kernels and thus reducing the model's parameter size and computational cost. For the text feature extraction sub-model, the system uses weight quantization technology to convert the 32-bit floating-point quantization in the BERT model into 8-bit integers, significantly reducing the model size and accelerating the inference process. For the speech feature extraction sub-model, the system prunes the hidden layers in the LSTM model, removing hidden units that contribute little to the output. Through these lightweight processing steps, the model maintains high feature extraction accuracy while reducing hardware resource requirements, enabling it to run smoothly on low-computing platforms such as embedded devices or mobile terminals.

[0081] It should be noted that the above model structure is only an example. Those skilled in the art can choose other lightweight network structures according to actual application scenarios, such as using MobileNet instead of ResNet, or using DistilBERT instead of BERT. As long as feature extraction can be achieved and the lightweight requirements are met, they are all within the protection scope of this invention.

[0082] As an example, in the smart interactive whiteboard scenario, the image feature extraction sub-model adopts a ResNet18 model with a 50% pruning ratio, reducing the number of parameters from 11.7M to 5.8M; the text feature extraction sub-model adopts an 8-bit quantized BERT model, compressing the model size to 1 / 4 of the original; and the speech feature extraction sub-model adopts a pruned LSTM model, reducing the number of hidden layer units by 40%. This achieves efficient deployment of the model in low-computing-power environments and solves the problem that large models are difficult to run on edge devices in existing technologies.

[0083] It should also be noted that inputting multimodal preprocessed data into a lightweight feature extraction model to extract multimodal feature vectors includes: inputting image preprocessed data into an image feature extraction sub-model to extract image feature vectors; inputting text preprocessed data into a text feature extraction sub-model to extract text feature vectors; inputting speech preprocessed data into a speech feature extraction sub-model to extract speech feature vectors; and performing dimension alignment processing on the image feature vectors, text feature vectors, and speech feature vectors to obtain multimodal feature vectors.

[0084] Dimension alignment refers to mapping feature vectors of different dimensions to a vector space of the same dimension in order to perform subsequent similarity calculations and feature fusion.

[0085] In one possible implementation, due to the different feature extraction model structures for different modalities, the dimensions of their output feature vectors often differ. For example, image feature vectors may have a dimension of 1024, while text feature vectors may have a dimension of 512. To eliminate the impact of dimensionality differences on subsequent calculations, after extracting the feature vectors of each modality, the system maps the feature vectors of each modality to a unified dimensional space through fully connected layers. Specifically, the system constructs three different fully connected layers, corresponding to image, text, and speech modalities respectively, mapping the feature vectors of each modality to feature vectors of a unified dimension, such as 256 dimensions. It should be understood that dimensionality alignment is not limited to fully connected layer mapping; it can also be achieved using pooling operations, principal component analysis (PCA) dimensionality reduction, or zero-padding.

[0086] Based on the above steps, this embodiment eliminates the structural differences between modalities by aligning the feature vectors of different modalities in dimensions. This provides a standardized data foundation for the subsequent construction of a unified semantic space and the calculation of multimodal similarity, and avoids calculation errors or information biases caused by dimension mismatch.

[0087] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S203 can be specifically implemented through the following S401 to S405, which are explained in detail below: S401. Extract semantic information from multimodal data and perform structured processing on the semantic information to obtain multimodal semantic expressions; based on the multimodal semantic expressions, map the semantic expressions of different modalities to the same semantic tag system to construct a unified semantic tag set.

[0088] Among them, multimodal semantic representation refers to structured data with clear semantic meaning extracted from raw data, such as object categories in images, keyword entities in text, and intent labels in speech; a unified semantic label set refers to a standardized label library that covers semantic labels of different modalities.

[0089] In one possible implementation, the system first performs semantic analysis on the large amount of multimodal data collected. For image data, the system uses object detection or image classification techniques to extract key entities (such as "car" and "pedestrian"); for text data, the system uses named entity recognition (NID) to extract keywords (such as "drive" and "stop"); for speech data, the system uses speech recognition and intent recognition techniques to extract speech command tags (such as "start" and "turn off"). Subsequently, the system performs structured processing on this semantic information scattered across different modalities to remove ambiguity. Most importantly, the system constructs a unified semantic tagging system, mapping tags expressing the same meaning in different modalities to the same standard tag ID. For example, the tag for "waving" in images, the tag for "goodbye" in speech, and the tag for "exit" in text are uniformly mapped to the semantic tag ID "User_Exit_Intent". In this way, the system breaks down the semantic barriers between modalities and establishes a standardized semantic index library.

[0090] S402. Based on a unified set of semantic tags, establish multimodal semantic associations between semantic tags of different modalities, and construct a knowledge graph for multimodal semantic alignment based on the multimodal semantic associations; determine the semantic association weights based on the association paths, relationship types and / or co-occurrence strengths between semantic tags in the knowledge graph.

[0091] Among them, multimodal semantic association is used to describe the logical connection between different modal tags, such as correspondence, causal relationship or co-occurrence relationship; knowledge graph is a knowledge base that stores entities and entity relationships in a graph structure, consisting of nodes (semantic tags) and edges (associations).

[0092] In one possible implementation, the system uses tags from a unified semantic tag set as nodes and establishes edges between nodes based on prior knowledge or statistical patterns in the actual application scenario. Specifically, the system analyzes the co-occurrence frequency of different modal tags in historical data. If two tags frequently appear simultaneously (e.g., the image tag "flame" and the speech tag "on fire"), an association edge is established. Simultaneously, the system defines the relationship type (e.g., "strong correlation," "weak correlation") and co-occurrence strength value based on the tightness of the association. Finally, the system integrates these nodes and edges to construct a knowledge graph for multimodal semantic alignment. After the graph is constructed, the system can use graph characteristics to determine semantic association weights. For example, the shorter the association path between two tags in the graph, the tighter their semantic relationship, and the system assigns them a higher initial association weight; conversely, if the path is long or there is no connection, the weight is lower. Furthermore, co-occurrence strength is also an important basis for weight calculation; the higher the co-occurrence strength, the greater the weight.

[0093] Based on the above steps, by constructing a knowledge graph and determining the association weights based on the graph structure, prior knowledge is used to make up for the shortcomings of the pure data-driven method, making the establishment of semantic associations more accurate and interpretable.

[0094] S403. Based on semantic association, obtain the association path information and relationship type information between semantic tags of different modalities; calculate the initial semantic association weight according to the path length of the association path, the relationship type weight and the co-occurrence frequency of semantic tags.

[0095] Among them, the associated path information refers to the path description between two nodes in the graph, and the relationship type information refers to the attribute description of the associated edge.

[0096] In one possible implementation, after the system obtains the multimodal feature vectors to be aligned and their corresponding semantic labels, it first retrieves the association paths between these labels in the knowledge graph. If there is a direct edge between two labels, the path length is 1; if an intermediate node is needed for connection, the path length is greater than 1. The system pre-determines an inverse relationship between path length and weight decay coefficient; the longer the path, the more severe the weight decay. Simultaneously, the system combines relation type weights (e.g., "synonymous relation" has a higher weight than "related relation") and co-occurrence frequency to obtain initial semantic association weights through weighted calculation. This calculation process transforms the topological structure information in the knowledge graph into quantifiable numerical weights.

[0097] Based on the above steps, by quantifying the path and relationship information in the graph, the abstract semantic associations are transformed into specific weight values, providing a calculation basis for subsequent feature fusion.

[0098] S404. Normalize the initial semantic association weights and adjust the weights by combining the multimodal feature vector quality assessment results to obtain the target semantic association weights; construct the multimodal attention weight matrix based on the target semantic association weights.

[0099] Among them, the target semantic association weight refers to the final weight value after quality correction, and the multimodal attention weight matrix refers to the weight parameter matrix used to guide feature fusion.

[0100] In one possible implementation, the system first normalizes the calculated initial semantic association weights, ensuring their values ​​fall within the [0,1] interval to guarantee comparability of weights across different modalities. Subsequently, the system incorporates the multimodal feature vector quality assessment results. If a modality's feature quality score is low, even with a high initial semantic association weight, the system will reduce its final weight through weighted adjustment to prevent low-quality features from misleading the alignment results. The adjustment logic can employ multiplicative adjustment. After obtaining the target semantic association weights, the system constructs a multimodal attention weight matrix, where each element represents the importance of the corresponding modal feature in the fusion process. This matrix not only includes the association strength between modalities but also incorporates feature quality factors, achieving fine-grained control over semantic information.

[0101] Based on the above steps, by dynamically adjusting the weights in conjunction with feature quality, the negative impact of low-quality or disturbed modalities on semantic alignment is effectively suppressed, and the robustness of the system in complex environments is improved.

[0102] S405. Based on the multimodal attention weight matrix, the feature vectors of each modality are weighted, mapped and fused to obtain the associated multimodal feature vectors.

[0103] Weighted mapping and fusion refers to the process of linearly transforming and combining feature vectors based on the weight matrix.

[0104] In one possible implementation, the system utilizes a pre-constructed multimodal attention weight matrix to weight the feature vectors of each modality. Specifically, the system multiplies the weight coefficients in the weight matrix with the corresponding feature vector elements, highlighting the role of high-quality, strongly correlated features and suppressing the influence of low-quality, weakly correlated features. Subsequently, the system concatenates, adds, or fuses the weighted feature vectors of each modality through a fully connected layer to generate a correlated multimodal feature vector. This feature vector not only contains the semantic information of the original data but also incorporates prior knowledge of the knowledge graph and dynamic adjustment information from quality assessment, resulting in stronger semantic expressive power and robustness against interference.

[0105] This embodiment establishes multimodal semantic associations by constructing a knowledge graph and dynamically adjusts the association weights based on feature quality, thereby achieving an organic combination of prior knowledge and data-driven approaches and improving the accuracy and robustness of semantic mapping.

[0106] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 5 As shown, the above S204 can be specifically implemented through the following S501 to S504, which are explained in detail below: S501. Based on the image data's sharpness index, contrast index, and / or feature response intensity, perform a quality assessment on the image feature vector to obtain an image feature quality score.

[0107] Among them, the sharpness index is used to measure the sharpness of image edges, the contrast index is used to measure the sense of hierarchy of image gray levels, and the feature response intensity is used to measure the degree of activation of image content by the feature extraction model.

[0108] In one possible implementation, after extracting the image feature vector, the system simultaneously calculates the quality indicators of the original image or feature map. For the sharpness indicator, the system can use the Laplacian variance algorithm or the gradient function method to calculate the rate of change of the image edge gradient; a larger gradient value indicates a sharper image. For the contrast indicator, the system calculates the distribution range or standard deviation of the image grayscale histogram; a larger standard deviation indicates higher contrast. For the feature response intensity, the system calculates the proportion of non-zero elements in the feature vector or the norm of the eigenvalues. The system performs a weighted average or minimum value operation on the above indicators to obtain the final image feature quality score. It should be understood that quality assessment is not limited to the above indicators and may also include parameters such as signal-to-noise ratio and exposure suitability; this embodiment does not specifically limit these parameters.

[0109] Based on the above steps, by quantifying the quality of image features through multi-dimensional indicators, low-quality image data can be accurately identified, avoiding blurry or low-contrast images from interfering with the subsequent semantic alignment process.

[0110] S502. Based on the signal-to-noise ratio, speech integrity, and / or speech feature stability of the speech data, perform a quality assessment on the speech feature vector to obtain a speech feature quality score.

[0111] Among them, signal-to-noise ratio refers to the ratio of the power of the speech signal to the power of the background noise; speech integrity is used to measure whether there is packet loss or truncation in a speech segment; and speech feature stability is used to measure the smoothness of speech features over time.

[0112] In one possible implementation, the system performs quality analysis on the extracted speech feature vectors. For the signal-to-noise ratio (SNR), the system estimates the background noise energy and speech signal energy using signal processing algorithms and calculates the decibel value. For speech integrity, the system detects whether there are missing or abnormal interruptions in the speech frame sequence. For speech feature stability, the system calculates the cosine or Euclidean distance between the feature vectors of consecutive speech frames; a smaller distance indicates higher stability. The system combines these indicators to generate a speech feature quality score. If the speech contains a large amount of environmental noise (such as wind noise or traffic noise), the SNR decreases, thus lowering the overall quality score.

[0113] S503. Based on the semantic integrity, keyword coverage and / or text length consistency of the text data, perform quality assessment on the text feature vectors to obtain a text feature quality score.

[0114] Among them, semantic integrity is used to measure the completeness of the text's sentence structure, keyword coverage is used to measure the inclusion of core words in the text, and text length consistency is used to measure whether the text length conforms to the preset specifications.

[0115] In one possible implementation, the system performs quality analysis on the input text data. For semantic integrity, the system uses dependency parsing to determine if key components such as subject, verb, and object are missing from the sentence; for keyword coverage, the system calculates the proportion of words from a pre-defined keyword library appearing in the text; for text length consistency, the system calculates the deviation of the text's word count from the average word count. If the input text consists of fragmented phrases or contains numerous typos, its semantic integrity and keyword coverage will significantly decrease, leading to a drop in the quality score.

[0116] It should be noted that although text data is less affected by environmental noise, it is easily affected by factors such as input errors and non-standard expressions. The above-mentioned indicators can effectively filter out high-quality text semantic information.

[0117] Based on the above steps, by conducting structural and semantic quality assessments on the text data, it is ensured that the text features involved in the alignment have clear semantic orientation, thereby improving the contribution of text modalities in the fusion process.

[0118] S504. Normalize the image feature quality score, speech feature quality score, and text feature quality score to obtain the multimodal feature vector quality assessment result.

[0119] In one possible implementation, the calculation methods for quality assessment metrics for image, speech, and text modalities differ, resulting in variations in the numerical range and distribution characteristics of their original scores, making direct comparisons lack objectivity. The system employs Min-Max normalization or Z-Score standardization to map the original scores of each modality to a unified numerical space. For example, the system divides the image sharpness score, speech signal-to-noise ratio score, and text keyword coverage score by their respective theoretical maximum values ​​to obtain standard scores between 0 and 1. After normalization, the quality scores of different modalities become comparable, allowing the system to fairly allocate the weights of each modality in the semantic mapping.

[0120] Based on the above steps, the dimensional differences between different modal evaluation indicators are eliminated through normalization, so that the multimodal feature vector quality evaluation results can objectively and fairly reflect the relative quality level of each modality data. This provides standardized data input for dynamically adjusting semantic association weights based on quality scores in subsequent embodiments, thereby realizing fine control over the multimodal feature fusion process and effectively improving the alignment robustness of the system in complex input environments.

[0121] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0122] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A knowledge graph-based AI multimodal semantic alignment method, characterized in that, include: Acquire multimodal raw data and preprocess the raw data to obtain multimodal preprocessed data; the multimodal raw data includes image data, text data, and voice data; The multimodal preprocessed data is input into a lightweight feature extraction model to extract multimodal feature vectors, and the quality of the multimodal feature vectors is evaluated to obtain the multimodal feature vector quality evaluation results; wherein, the lightweight feature extraction model includes an image feature extraction sub-model, a text feature extraction sub-model, and a speech feature extraction sub-model that have been optimized by model pruning and / or quantization; Based on a pre-defined knowledge graph, semantic labels are used to annotate multimodal feature vectors, and associations are retrieved in the knowledge graph according to the semantic labels to construct semantic associations between different modalities. Based on the semantic association relationship and the quality assessment results of the multimodal feature vectors, the semantic association weights are determined, and semantic mapping processing is performed on each modal feature vector to obtain the associated multimodal feature vectors. Based on the associated multimodal feature vectors, the semantic similarity between different modal feature vectors is calculated; Multimodal semantic alignment is performed based on the semantic similarity to obtain the aligned multimodal feature vector.

2. The method according to claim 1, characterized in that, After obtaining the semantically aligned multimodal data, the method further includes: Based on the modality type and / or number of modalities in the input multimodal raw data, determine the corresponding modality combination type; Based on the modality combination type, a corresponding lightweight feature extraction sub-model is selected, and the model parameters of the lightweight feature extraction sub-model are adjusted, including the pruning ratio and / or quantization accuracy. Based on the multimodal feature vector quality assessment results, adjust the weights of each modality feature vector in the semantic mapping process; Based on the modality combination type and semantic association relationship, adjust the semantic association weight and / or attention weight in the semantic mapping process; Based on the semantic similarity calculation results, the similarity threshold for semantic alignment is dynamically adjusted.

3. The method according to claim 1, characterized in that, The image feature extraction sub-model is a pruned ResNet model, the text feature extraction sub-model is a lightweight BERT model, and the speech feature extraction sub-model is a pruned LSTM model.

4. The method according to claim 1, characterized in that, The step of inputting the multimodal preprocessed data into a lightweight feature extraction model to extract multimodal feature vectors includes: The image preprocessing data is input into the image feature extraction sub-model to extract image feature vectors; The preprocessed text data is input into the text feature extraction sub-model to extract text feature vectors. The preprocessed speech data is input into the speech feature extraction sub-model to extract speech feature vectors. The image feature vector, text feature vector, and speech feature vector are dimensionally aligned to obtain a multimodal feature vector.

5. The method according to claim 1, characterized in that, The multimodal feature vector quality assessment results include: Based on the image data's sharpness index, contrast index, and / or feature response intensity, the quality of the image feature vector is evaluated to obtain an image feature quality score. Based on the signal-to-noise ratio, speech integrity, and / or speech feature stability of speech data, the quality of speech feature vectors is evaluated to obtain a speech feature quality score. Based on the semantic integrity, keyword coverage and / or text length consistency of text data, the quality of text feature vectors is evaluated to obtain a text feature quality score. The image feature quality score, speech feature quality score, and text feature quality score are normalized to obtain the multimodal feature vector quality assessment result.

6. The method according to claim 1, characterized in that, The construction of the preset knowledge graph includes: Semantic information is extracted from multimodal data and the semantic information is structured to obtain multimodal semantic representations; the multimodal semantic representations include: semantic representations of image data, semantic representations of text data, and semantic representations of speech data; Based on the multimodal semantic expression, the semantic expressions of different modalities are mapped to the same semantic tag system to construct a unified semantic tag set; Based on the unified semantic tag set, a multimodal semantic association relationship is established between different modal semantic tags, and the multimodal semantic association relationship includes semantic correspondence relationship and semantic mapping relationship; Based on the aforementioned multimodal semantic associations, a knowledge graph oriented towards multimodal semantic alignment is constructed; The semantic association weights are determined based on the association paths, relationship types, and / or co-occurrence strengths between semantic tags in the knowledge graph.

7. The method according to claim 1, characterized in that, Based on the semantic association relationship and the multimodal feature vector quality assessment results, the semantic association weights are determined, and semantic mapping processing is performed on each modal feature vector to obtain the associated multimodal feature vectors, including: Based on semantic association, obtain the association path information and relationship type information between semantic tags of different modalities; Calculate the initial semantic association weight based on the path length, relation type weight, and semantic tag co-occurrence frequency of the associated path; The initial semantic association weights are normalized, and the semantic association weights are adjusted by weighting based on the multimodal feature vector quality assessment results to obtain the target semantic association weights. Based on the target semantic association weights, a multimodal attention weight matrix is ​​constructed; Based on the multimodal attention weight matrix, the feature vectors of each modality are weighted, mapped and fused to obtain the associated multimodal feature vectors.

8. The method according to claim 1, characterized in that, The step of calculating the semantic similarity between different modal feature vectors based on the associated multimodal feature vectors includes: Based on the associated multimodal feature vectors, local semantic similarity and global semantic similarity between different modalities are calculated respectively; wherein, the local semantic similarity is calculated based on the similarity between each dimension or subspace in the feature vector, and the global semantic similarity is calculated based on the similarity of the overall feature vector; The local semantic similarity and global semantic similarity are fused to obtain the semantic similarity between modalities.

9. The method according to claim 8, characterized in that, Local semantic similarity Satisfy the following formula: in, , These represent the sub-vectors of different modal feature vectors in the i-th semantic subspace; These are the subspace weight coefficients; This represents the path distance between different modal semantic tags in the corresponding semantic subspace within the knowledge graph. This is the path distance attenuation coefficient; This is the quality adjustment factor determined based on the multimodal feature vector quality assessment results; n is the number of subspace partitions. global semantic similarity Satisfy the following formula: Where A and B represent the feature vectors of different modalities after association.

10. A knowledge graph-enhanced AI multimodal semantic alignment system, characterized in that, The system includes: a data preprocessing module, a lightweight feature processing module, a knowledge graph-assisted modeling module, a semantic mapping module, a similarity calculation module, and a semantic alignment module; The data preprocessing module is used to acquire multimodal raw data and preprocess the raw data to obtain multimodal preprocessed data; the multimodal raw data includes image data, text data and voice data. The lightweight feature processing module is used to input the multimodal preprocessed data into the lightweight feature extraction model, extract multimodal feature vectors, and perform quality evaluation on the multimodal feature vectors to obtain multimodal feature vector quality evaluation results; wherein, the lightweight feature extraction model includes an image feature extraction sub-model, a text feature extraction sub-model, and a speech feature extraction sub-model optimized by model pruning and / or quantization; The knowledge graph-assisted modeling module is used to semantically label multimodal feature vectors based on a preset knowledge graph, and to perform relation retrieval in the knowledge graph according to the semantic labels, thereby constructing semantic relational relationships between different modalities. The semantic mapping module is used to determine the semantic association weights based on the semantic association relationship and the multimodal feature vector quality assessment results, and to perform semantic mapping processing on each modal feature vector to obtain the associated multimodal feature vectors. The similarity calculation module is used to calculate the semantic similarity between different modal feature vectors based on the associated multimodal feature vectors; The semantic alignment module is used to perform multimodal semantic alignment based on the semantic similarity to obtain aligned multimodal feature vectors.

Images