Cross-modal semantic alignment method for multi-modal llm-rag local knowledge base

Abstract

The application relates to the field of artificial intelligence, in particular to a cross-modal semantic alignment method of a multi-modal LLM-RAG local knowledge base, which comprises the following steps: S1, constructing a cross-modal semantic cell dynamic recombination base, which is used for disassembling original semantic information of various heterogeneous modes such as text, image and audio into the smallest independently representable semantic cells. The application disassembles various heterogeneous modes such as text, image and audio into semantic cells with mode identification, semantic weight and correlation threshold by constructing the cross-modal semantic cell dynamic recombination base, combines local data dynamic recombination iteration, solves the problem that the prior art ignores the structural difference of mode embedding space and is not suitable for sparse local data, builds a three-layer causal graph of mode-semantic-scene by means of a predictive distribution calibration mechanism with causal perception, locates the root of data distribution change and adjusts the cell cluster rule in combination with time series data prediction, and avoids semantic misalignment caused by data distribution deviation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a cross-modal semantic alignment method for a multimodal LLM-RAG local knowledge base. Background Technology

[0002] As artificial intelligence technology accelerates its evolution towards multimodal fusion and localized deployment, retrieval-enhanced generative architectures based on large-scale language models are gradually becoming the core support for intelligent interactive systems. This architecture significantly improves the accuracy and contextual adaptability of the system in scenarios such as open-domain question answering, personalized recommendations, and decision support by combining the dynamic retrieval capabilities of external knowledge bases with the semantic generation capabilities of language models. Local knowledge bases, as dedicated data containers deployed on terminal devices or in private environments, not only ensure data ownership and information security but also support deep semantic customization for specific users or industry scenarios. However, when the knowledge base content covers heterogeneous modalities such as text, images, and audio, achieving accurate alignment across modal semantic spaces becomes a key bottleneck restricting system performance.

[0003] The core challenge of the multimodal LLM-RAG architecture lies in constructing a cross-modal association mechanism under a unified semantic representation. Its goal is to ensure semantic consistency of data from different modalities within the embedding space, guaranteeing accurate matching of multimodal fragments highly relevant to the query intent during the retrieval phase, and achieving semantically coherent cross-modal output during the generation phase. Ideally, the model should be able to recognize the text description "a golden pheasant standing on a branch," forming a fine-grained semantic mapping with the specific bird's posture, color, and environmental features in the corresponding image, rather than relying solely on coarse-grained category labels or global feature similarity.

[0004] Existing technologies for cross-modal semantic alignment in local knowledge bases generally neglect the structural differences in the embedding spaces between modalities, leading to systematic shifts in semantic associations at the fine-grained entity level. Most methods rely on global alignment loss functions during the pre-training phase or assume that the data distribution is completely consistent between the training and inference phases, failing to adapt to the real-world constraints of data sparsity, distribution shifts, and modal missingness in local scenarios. This embedding bias accumulates layer by layer during retrieval, causing semantic drift: even if the original data is highly relevant, the model may still recall semantically irrelevant fragments due to spatial mapping distortion, ultimately resulting in factual errors, modal conflicts, or contextual breaks in the generated content. Especially in high-precision applications such as medical diagnosis, industrial quality inspection, and legal documents, such semantic inaccuracies directly weaken the reliability of system decisions and user trust, urgently requiring an alignment enhancement method that can dynamically calibrate the cross-modal semantic space and adapt to the heterogeneous distribution of local data. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a cross-modal semantic alignment method for multimodal LLM-RAG local knowledge bases, which solves the problems of inaccurate generated content caused by structural differences in modal embedding space, incompatibility with local data sparsity and distribution shift, modal missingness and semantic drift in existing technologies.

[0006] To achieve the above objectives, this invention provides the following technical solution: a cross-modal semantic alignment method for a multimodal LLM-RAG local knowledge base, the method comprising:

[0007] S1. Construct a cross-modal semantic cell dynamic recombination base to decompose the original semantic information of various heterogeneous modalities such as text, images, and audio into the smallest independently representative semantic cells. Each semantic cell carries a modality identifier, semantic weight, and association threshold. Based on the real-time dynamic requirements of the local deployment scenario, semantic cells are dynamically recombined through the "attraction-repulsion" rule between cells to form a semantic cluster adapted to the current scenario. Combined with the unique data continuously accumulated in the local knowledge base, the dynamic autonomous iterative evolution capability of semantic cells is realized. Iterative evolution includes cell splitting or cell fusion.

[0008] S2. Construct a predictive distribution calibration mechanism for causal perception, which is used to build a causal graph with three layers of association: modality, semantics, and scene, based on the internal data features of the local knowledge base. The causal graph establishes the association links between nodes at each layer. Through deep causal reasoning algorithms, changes in data features are accurately transformed into changes in the weights of causal links to locate the deep root causes of changes in data distribution. Combined with the time-series historical data of the local scene, the accurate dynamic trend of future data distribution is predicted. Based on the prediction results, the internal association rules of the semantic cell cluster are predictively adjusted.

[0009] S3. Construct a cross-modal memory transfer missing completion mechanism to systematically collect various modal missing scenario cases processed in the local knowledge base during actual operation. Transform each specific case into a memory unit containing rich contextual information, thereby forming a dynamically updated and continuously growing memory bank. When the system detects a specific situation of modal missing, it first accurately matches the memory units of the same scenario in the memory pool through scene tags, extracts its core completion logic framework, and then combines the refined features of the semantic cells of the current existing modality to adaptively adjust the completion logic parameters, and finally generates refined semantic features of the missing modality that are adapted to the current needs.

[0010] S4. Construct a pre-aligned guided retrieval-generation closed-loop system. Before multimodal retrieval is initiated, a large-scale language model is used to deeply analyze the user's query intent and comprehensively predict the complete semantic dimensions required for generating content. Based on the prediction results, multiple heterogeneous modal fragments containing corresponding semantic cells are actively screened and recalled during cross-modal retrieval. Before generating content, the predicted semantic dimensions of the generated content are compared and verified with the retrieved semantic cell cluster. If potential semantic conflicts are found, the cell reselection process is automatically triggered. A prediction model is trained based on historical retrieval-generation data to predict alignment accuracy in real time and adjust semantic cell weights, calibration parameters, or completion logic in reverse to form a continuously optimized closed-loop process.

[0011] Furthermore, the process of decomposing the original semantic information of heterogeneous modalities into the smallest independently representative semantic cells in S1 specifically includes:

[0012] The text modality is decomposed into entity attribute cells and relation cells; the image modality is decomposed into visual morphology cells and scene cells; the audio modality is decomposed into acoustic event cells and sentiment cells, and each semantic cell carries a modality identifier, semantic weight, and association threshold.

[0013] Furthermore, the dynamic reorganization of semantic cells in S1 to form a semantic cluster adapted to the current scenario specifically includes:

[0014] Based on the real-time dynamic needs of the local deployment scenario, the preset "attraction-repulsion" rule between semantic cells is triggered, so that semantic cells with semantic similarity higher than the preset association threshold are automatically aggregated to form a semantic cluster adapted to the current scenario, and the semantic weight of semantically conflicting cells is reduced.

[0015] Furthermore, the ability to realize the dynamic autonomous iterative evolution of semantic cells in S1 specifically includes:

[0016] By combining the unique data continuously accumulated in the local knowledge base, when new scenario data enters, it triggers the splitting of semantic cells, refining a cell into multiple sub-cells carrying exclusive features; or it triggers the fusion of semantic cells, merging multiple related cells into a comprehensive cell that integrates different modal semantics.

[0017] Furthermore, the construction of the causal graph with three layers of association between modality, semantics, and scene in S2 specifically includes:

[0018] Modal layer nodes are constructed to represent input modalities such as text, images, and audio; semantic layer nodes are constructed to represent abstract semantic concepts such as entities, attributes, and relationships; and scenario layer nodes are constructed to represent specific application scenarios such as medical diagnosis, industrial quality inspection, and agricultural monitoring. The association links between nodes of each layer are established through causal reasoning algorithms, and changes in data features are accurately transformed into changes in causal link weights, so as to accurately locate the deep root causes of changes in data distribution.

[0019] Furthermore, the step S2, which involves predictively adjusting the internal association rules of the semantic cell cluster based on the prediction results, specifically includes:

[0020] By leveraging the link association characteristics of causal graphs and combining local scene time-series data, we can predict the accurate dynamic trend of future data distribution. Time-series data includes, but is not limited to, the arrival of the rainy season or equipment maintenance cycle. Based on the prediction results, we can adjust the internal association rules of semantic cell clusters in advance to strengthen or weaken the aggregation weight between specific semantic cells.

[0021] Furthermore, the transformation of various modality-deficient scenario cases into memory units containing rich contextual information in S3 specifically includes:

[0022] The memory unit contains the missing modality type, the existing modality feature description, the completion logic framework, and the application scenario label. The memory unit forms a dynamically updated and continuously growing scenario-based completion experience memory bank, which stores scenario-based completion experience.

[0023] Furthermore, the refined semantic features for generating the missing modality adapted to the current requirements in S3 specifically include:

[0024] When the system detects a specific situation of modality missing, it first accurately matches the memory units of the same scene in the memory bank through scene tags and extracts their core completion logic framework; then, it combines the refined features of the semantic cells of the current existing modality, adaptively adjusts the completion logic parameters, and finally generates refined semantic features of the missing modality that are adapted to the current local requirements.

[0025] Furthermore, the pre-aligned guided retrieval in S4 specifically includes:

[0026] Before multimodal retrieval is initiated, a large-scale language model is used to deeply analyze the user's query intent and comprehensively predict the complete semantic dimensions required to generate content. Semantic dimensions include habitat, feather characteristics, call characteristics, or diet. Based on the prediction results, multiple heterogeneous modal fragments containing corresponding semantic cells are actively screened and recalled during cross-modal retrieval.

[0027] Furthermore, the closed-loop process in S4 that pre-avoids semantic conflicts and forms continuous optimization before generating content specifically includes:

[0028] The predicted generated semantic dimension is compared and verified with the retrieved semantic cell cluster. If a potential semantic conflict is found, the cell reselection process is automatically triggered to retrieve the fragment containing the correct semantic cell. The "cell combination-alignment effect" prediction model is trained based on historical retrieval-generation data to predict the alignment evaluation accuracy of subsequent retrievals in real time. If the prediction effect does not meet expectations, the semantic cell weights, calibration parameters, or completion logic are adjusted in reverse to form a closed loop of "prediction-execution-verification-optimization".

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] This invention constructs a dynamic recombination base for cross-modal semantic cells, decomposing heterogeneous modalities such as text, images, and audio into semantic cells with modality-identified semantic weights and association thresholds. Combined with dynamic recombination iteration using local data, it addresses the problems of existing technologies neglecting the structural differences in modality embedding space and being unsuitable for local data sparsity. A predictive distribution calibration mechanism based on causal awareness is used to build a three-layer causal graph of modality, semantics, and scene, locating the root causes of data distribution changes and predictively adjusting cell cluster rules based on time-series data to avoid semantic inaccuracies caused by data distribution shifts. A scene-based memory bank is established through a cross-modal memory transfer missing completion mechanism, matching and completing logic to generate semantic features when a modality is missing, filling the modality gaps. A pre-aligned guided retrieval-generation closed-loop system parses the query to predict semantic dimensions, verifies and avoids conflicts, and reverse-optimizes parameters to reduce semantic drift. Overall, this improves the accuracy of cross-modal semantic alignment, reduces factual errors in generated content, adapts to local scene requirements, and enhances the system's reliability in high-precision scenarios. Attached Figure Description

[0031] Figure 1 This is a flowchart of the cross-modal semantic cell construction and dynamic recombination process of the present invention;

[0032] Figure 2 This is a flowchart of the pre-aligned retrieval-generation closed-loop system of the present invention;

[0033] Figure 3 This is a schematic diagram of the overall architecture of the cross-modal semantic alignment method of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Please see Figure 1-3This invention provides a cross-modal semantic alignment method for a multimodal LLM-RAG local knowledge base, the method comprising:

[0036] S1. Construct a cross-modal semantic cell dynamic recombination base to decompose the original semantic information of various heterogeneous modalities such as text, images, and audio into the smallest independently representative semantic cells. Each semantic cell carries a modality identifier, semantic weight, and association threshold. Based on the real-time dynamic requirements of the local deployment scenario, semantic cells are dynamically recombined through the "attraction-repulsion" rule between cells to form a semantic cluster adapted to the current scenario. Combined with the unique data continuously accumulated in the local knowledge base, the dynamic autonomous iterative evolution capability of semantic cells is realized. Iterative evolution includes cell splitting or cell fusion.

[0037] S2. Construct a predictive distribution calibration mechanism for causal perception, which is used to build a causal graph with three layers of association: modality, semantics, and scene, based on the internal data features of the local knowledge base. The causal graph establishes the association links between nodes at each layer. Through deep causal reasoning algorithms, changes in data features are accurately transformed into changes in the weights of causal links to locate the deep root causes of changes in data distribution. Combined with the time-series historical data of the local scene, the accurate dynamic trend of future data distribution is predicted. Based on the prediction results, the internal association rules of the semantic cell cluster are predictively adjusted.

[0038] S3. Construct a cross-modal memory transfer missing completion mechanism to systematically collect various modal missing scenario cases processed in the local knowledge base during actual operation. Transform each specific case into a memory unit containing rich contextual information, thereby forming a dynamically updated and continuously growing memory bank. When the system detects a specific situation of modal missing, it first accurately matches the memory units of the same scenario in the memory pool through scene tags, extracts its core completion logic framework, and then combines the refined features of the semantic cells of the current existing modality to adaptively adjust the completion logic parameters, and finally generates refined semantic features of the missing modality that are adapted to the current needs.

[0039] S4. Construct a pre-aligned guided retrieval-generation closed-loop system. Before multimodal retrieval is initiated, a large-scale language model is used to deeply analyze the user's query intent and comprehensively predict the complete semantic dimensions required for generating content. Based on the prediction results, multiple heterogeneous modal fragments containing corresponding semantic cells are actively screened and recalled during cross-modal retrieval. Before generating content, the predicted semantic dimensions of the generated content are compared and verified with the retrieved semantic cell cluster. If potential semantic conflicts are found, the cell reselection process is automatically triggered. A prediction model is trained based on historical retrieval-generation data to predict alignment accuracy in real time and adjust semantic cell weights, calibration parameters, or completion logic in reverse to form a continuously optimized closed-loop process.

[0040] Specifically, when constructing a dynamic recombination base for cross-modal semantic cells, the original semantic information of heterogeneous modalities such as text, images, and audio is first decomposed. For the text modality, existing natural language processing techniques such as word segmentation, entity recognition, and relation extraction are used to extract core entities and entity attributes. Individual entities are combined with their corresponding attributes to form entity attribute cells; relationships between entities are analyzed, and each set of relationships is used as a relation cell. For the image modality, existing computer vision algorithms for image segmentation and scene recognition are used to segment images at the pixel level, extracting entity morphological features to construct visual morphological cells, and identifying environmental elements to form scene cells. For the audio modality, existing audio feature extraction and event classification models are used to extract key events to generate acoustic event cells, and audio features are analyzed to determine emotions or states to form emotion cells. Each semantic cell carries a clear modality identifier, semantic weight, and association threshold. The semantic weight is preset according to the importance of the modality in the local scene; for example, in a medical diagnosis scenario, the weight of lesion attribute cells is set to 0.8, and the weight of environmental cells is set to 0.3.

[0041] Next, the "attraction-repulsion" rule is triggered based on the real-time dynamic requirements of the local deployment scenario. This process requires calculating the semantic similarity between semantic cells, using Formula 1:

[0042]

[0043] Where S represents the semantic similarity between two semantic cells, with a value ranging from 0 to 1; w1 and w2 are the semantic weights of the two cells, respectively; and These are the normalized semantic feature vectors of two cells, with a uniform vector dimension of 64 to ensure dimensional consistency; "·" indicates the vector dot product operation; Let w1 represent the L2 norm of the vector. Taking the medical diagnostic scenario of assisting in determining the benign or malignant nature of pulmonary nodules as an example, the w1 of the cell in an image with "rough lesion edges" is 0.8. The text cell for "typical characteristics of malignant nodules" has a w2=0.7. Substituting the values ​​into the formula, we get S≈0.92, which exceeds the preset association threshold of 0.82. The two cells automatically aggregate to form a semantic cluster. However, the cells of the "normal lung texture" image have an S=0.35 calculated with this cluster, which is semantically conflicting. The system then triggers a reduction in its semantic weight from 0.3 to 0.15.

[0044] Finally, by combining the continuously accumulated case data from the local knowledge base, the cells are dynamically iterated. If a special type of "mixed ground-glass nodule" appears in a newly included case, the "lesion morphology cell" is triggered to split into two sub-cells: "pure ground-glass nodule morphology" and "mixed ground-glass nodule morphology," each carrying specific features such as "uniform density" and "non-uniform density containing solid components." If subsequent multimodal data all point to "early lung adenocarcinoma," the corresponding cells are merged into a "comprehensive semantic cell for early lung adenocarcinoma." This implementation addresses the problem in existing technologies where the spatial structural differences in modal embedding are not adapted to the sparsity of local data, reduces semantic drift, improves the adaptability of semantic clusters to local scenarios, and makes semantic representations more aligned with actual needs through iterative evolution.

[0045] In this embodiment, S1, which decomposes the original semantic information of heterogeneous modalities into the smallest independently representative semantic cells, specifically includes:

[0046] The text modality is decomposed into entity attribute cells and relation cells; the image modality is decomposed into visual morphology cells and scene cells; the audio modality is decomposed into acoustic event cells and sentiment cells, and each semantic cell carries a modality identifier, semantic weight, and association threshold.

[0047] Specifically, when performing text modality decomposition, the existing natural language processing technologies of word segmentation entity recognition and relation extraction are adopted. First, the core entities and entity attributes are extracted from the text data, and a single entity and its corresponding attribute are combined to form an entity attribute cell. Then, the relationships such as action attribution between entities are analyzed, and each group of relationships is treated as a relation cell.

[0048] Image modality decomposition is achieved through existing image segmentation and scene recognition algorithms in the field of computer vision. First, the image is segmented at the pixel level to extract the morphological features such as contour, texture, and color of the objects and construct visual morphological cells. Then, environmental elements in the image are identified to form scene cells.

[0049] Audio modality decomposition leverages existing audio feature extraction and event classification models to first extract key events from the audio, generating acoustic event cells. Then, by analyzing features such as frequency, amplitude, and rhythm, it determines the emotion or state conveyed by the sound, forming emotion cells. Each decomposed semantic cell explicitly carries modality identifiers, semantic weights, and association thresholds, ensuring accurate differentiation of semantic information across different modalities. This lays the foundation for subsequent dynamic recombination and improves the independent representation accuracy of semantic cells.

[0050] In this embodiment, the dynamic reorganization of semantic cells in S1 to form a semantic cluster adapted to the current scenario specifically includes:

[0051] Based on the real-time dynamic needs of the local deployment scenario, the preset "attraction-repulsion" rule between semantic cells is triggered, so that semantic cells with semantic similarity higher than the preset association threshold are automatically aggregated to form a semantic cluster adapted to the current scenario, and the semantic weight of semantically conflicting cells is reduced.

[0052] Specifically, during implementation, the real-time requirements of the local deployment scenario should be clearly defined. Taking the industrial parts quality inspection scenario as an example, the requirement is "quickly identify surface scratches and defects on parts". The system calls the text cells, image cells, and audio cells that have been constructed in this scenario. The semantic weights and association thresholds of each cell are preset according to the scenario requirements. For example, the weight of scratch-related text cells is set to 0.75, and the association threshold is set to 0.85.

[0053] The semantic similarity between cells is quantified using a semantic similarity calculation model, and the semantic similarity is calculated using Formula 1. Taking the image cell of "scratch gray value change" and the text cell of "scratch length 2mm" as examples, the former has w1=0.8. The latter, w²=0.75, Substituting into Formula 1, we get S≈0.88, which exceeds the preset association threshold of 0.85, triggering the "attraction" rule, and the two automatically aggregate. If the image cell of "grayscale features of normal surface of part" and the above aggregated cell calculate S=0.32, there is a semantic conflict, and the system automatically triggers the "repulsion" rule, reducing the semantic weight of the normal surface cell from the original 0.25 to 0.1.

[0054] Ultimately, a semantic cluster centered on "scratch recognition on parts surface" is formed. This cluster closely matches the needs of industrial quality inspection scenarios, eliminates irrelevant semantic interference, improves the adaptability of the semantic cluster to the current scenario, and ensures that the subsequent retrieval and generation processes focus on core needs.

[0055] In this embodiment, the dynamic autonomous iterative evolution capability of semantic cells in S1 specifically includes:

[0056] By combining the unique data continuously accumulated in the local knowledge base, when new scenario data enters, it triggers the splitting of semantic cells, refining a cell into multiple sub-cells carrying exclusive features; or it triggers the fusion of semantic cells, merging multiple related cells into a comprehensive cell that integrates different modal semantics.

[0057] Specifically, during implementation, the unique data accumulated by the local knowledge base is relied upon. Taking the monitoring scenario of agricultural crop diseases and pests as an example, the initial semantic cell of "crop leaves turning yellow" covers the basic visual features and text description of yellowing leaves, with a semantic weight of 0.7.

[0058] When new monitoring data enters the system and reveals two different types of "yellowing leaves," the system automatically triggers a cell division mechanism, splitting the original "crop leaf yellowing" cell into two sub-cells: "nitrogen deficiency leaf yellowing" and "disease-related leaf yellowing." These sub-cells carry specific features such as "leaf veins remain green while leaf tissue yellows" and "yellow spots appear on leaves with withered edges," respectively. Their semantic weights are adjusted to 0.72 and 0.75 based on the importance of the scenario. If subsequent monitoring shows that multimodal data all point to the same conclusion, the corresponding cells are merged into a comprehensive semantic cell. The semantic weight of the merged cell is the average of the weights of each sub-cell. For example, after merging the "nitrogen deficiency leaf yellowing" cell (weight 0.72) with the "low soil nitrogen content" text cell (weight 0.68), the comprehensive cell weight is 0.7.

[0059] By splitting and merging cells, semantic cells can continuously adapt to changes in local data, improving the precision and completeness of semantic representation and better meeting scenario requirements.

[0060] In this embodiment, the construction of the causal graph with three layers of association between modality, semantics, and scene in S2 specifically includes:

[0061] Modal layer nodes are constructed to represent input modalities such as text, images, and audio; semantic layer nodes are constructed to represent abstract semantic concepts such as entities, attributes, and relationships; and scenario layer nodes are constructed to represent specific application scenarios such as medical diagnosis, industrial quality inspection, and agricultural monitoring. The association links between nodes of each layer are established through causal reasoning algorithms, and changes in data features are accurately transformed into changes in causal link weights, so as to accurately locate the deep root causes of changes in data distribution.

[0062] Specifically, the implementation involves building nodes in three layers. The modal layer sets up three core nodes: text, image, and audio, corresponding to the three types of input data in the local knowledge base. The semantic layer builds entity nodes, attribute nodes, and relationship nodes, covering the abstract semantic concepts in the data. The scenario layer sets up three nodes: medical diagnosis, industrial quality inspection, and agricultural monitoring, corresponding to common local application scenarios.

[0063] An existing causal Bayesian network is used as the deep causal inference algorithm to establish the correlation links between nodes at each layer. When local data characteristics change, the weight changes of the causal links need to be calculated using Formula 2:

[0064]

[0065] Where Wnew represents the link weight after the data feature changes; Wold represents the original link weight before the data feature changes; ΔF represents the data feature change rate, which ranges from -0.5 to 0.5. It is calculated by dividing the difference between the current data feature and the historical average feature by the historical average feature, and has been normalized to ensure consistent dimensions.

[0066] Taking a medical diagnostic scenario as an example, the original link weight Wold=0.7 for the "image modality node-lesion entity node" link. Due to the upgrade of CT imaging equipment, the clarity of the current image lesion features has increased by 30% compared to the historical average, resulting in a calculated ΔF=0.3. Substituting this into Formula 2, we get Wnew=0.7×(1+0.3)=0.91. This weight change accurately pinpoints the root cause of the data distribution change as "imaging equipment upgrade." Similarly, in an industrial scenario, if the audio acquisition equipment for part inspection malfunctions, the clarity of the current audio features decreases by 20% compared to the historical average, resulting in ΔF=-0.2 and Wold=0.6. Substituting this into Formula 2, we get Wnew=0.6×(1-0.2)=0.48, clearly indicating that the root cause of the data distribution change is "acquisition equipment malfunction." This method achieves precise positioning of the deep-seated root cause of data distribution changes, providing a basis for subsequent semantic cell cluster adjustments.

[0067] In this embodiment, the process of adjusting the internal association rules of the semantic cell cluster based on the prediction results in S2 specifically includes:

[0068] By leveraging the link association characteristics of causal graphs and combining local scene time-series data, we can predict the accurate dynamic trend of future data distribution. Time-series data includes, but is not limited to, the arrival of the rainy season or equipment maintenance cycle. Based on the prediction results, we can adjust the internal association rules of semantic cell clusters in advance to strengthen or weaken the aggregation weight between specific semantic cells.

[0069] Specifically, during implementation, the link correlation characteristics of the causal graph are first utilized, and trend prediction is made by combining the time-series historical data of the local scenario. Taking the agricultural monitoring scenario as an example, the time-series data includes monthly rainfall, crop growth status and pest and disease occurrence over the past three years. Through analysis using existing time series prediction algorithms, it was found that after "monthly rainfall exceeds 100mm for two consecutive weeks", data related to "crop root rot" will increase significantly.

[0070] Based on this prediction, the system anticipates that the future data distribution trend will be "an increase in the proportion of cells related to root rot and a decrease in the proportion of cells related to drought." Subsequently, it adjusts the internal association rules of the semantic cell clusters in advance, adjusting the aggregation weights using Formula 3:

[0071]

[0072] Where Wadjust represents the adjusted aggregate weights; Woriginal represents the original aggregate weights before adjustment; k represents the weight adjustment coefficient, which is set according to the strength of the predicted trend, and the value ranges from 0.4 to 1.2. The stronger the trend, the more the coefficient deviates from 1.

[0073] Taking the adjustment of the aggregation weights of "root rot image cells" and "soil moisture text cells" as an example, Woriginal=0.75, and because the predicted trend of "increasing root rot-related data" is relatively strong, k=1.2 is set. Substituting into Formula 3, we get Wadjust=0.75×1.2=0.9, which strengthens the aggregation weights of the two. For "drought-related cells", Woriginal=0.7, and because its proportion is predicted to decrease, k=0.57 is set. Substituting into Formula 3, we get Wadjust=0.7×0.57≈0.4, which weakens its aggregation weight.

[0074] By making predictive adjustments, when the relevant data actually arrives during the rainy season and enters the system, the semantic cell cluster can quickly adapt to the data changes without having to temporarily adjust the rules, reducing the latency in the retrieval and generation process and improving the system's responsiveness to changes in the scenario.

[0075] In this embodiment, S3, which transforms various modality-deficient scenario cases into memory units containing rich contextual information, specifically includes:

[0076] The memory unit contains the missing modality type, the existing modality feature description, the completion logic framework, and the application scenario label. The memory unit forms a dynamically updated and continuously growing scenario-based completion experience memory bank, which stores scenario-based completion experience.

[0077] Specifically, during implementation, the system uses existing database storage technology to record various modal missing scenario cases that occur during the operation of the local knowledge base in real time. Taking the medical diagnosis scenario as an example, it records cases such as only having patient brain MRI images but missing patient medical history text descriptions, only having electrocardiogram audio data but missing cardiac ultrasound images; taking the industrial quality inspection scenario as an example, it records cases such as only having part appearance images but missing part material text descriptions, only having equipment operation audio but missing equipment temperature detection data.

[0078] Each case is structured and transformed into a memory unit. Taking a case with only brain MRI images and missing medical history text as an example, the memory unit contains the missing modality type as text patient medical history, the existing modality feature description as MRI images showing hippocampal atrophy and mild ventricular dilation, the completion logic framework as matching common medical histories corresponding to similar lesions in the knowledge base based on lesion features in MRI images, and the application scenario label as brain disease diagnosis.

[0079] All memory units are stored in the memory bank, and as the system runs, new modal missing cases are continuously added to update the content of the memory bank, so that the memory bank always keeps growing dynamically. This provides rich scenario-based experience for subsequent modal missing completion and ensures that the completion logic can meet the needs of different scenarios.

[0080] In this embodiment, the generation of refined semantic features for the missing modality adapted to the current requirements in S3 specifically includes:

[0081] When the system detects a specific situation of modality missing, it first accurately matches the memory units of the same scene in the memory bank through scene tags and extracts their core completion logic framework; then, it combines the refined features of the semantic cells of the current existing modality, adaptively adjusts the completion logic parameters, and finally generates refined semantic features of the missing modality that are adapted to the current local requirements.

[0082] Specifically, when the system detects a modal missing condition during implementation, taking the case of an industrial quality inspection scenario where only the image of a part surface scratch is missing the text description of the scratch cause as an example, the system first matches the scene tag "Industrial Part Scratch Detection" in the memory bank to find the memory unit in the same scene where only the image of a part deformation is missing the text description of the deformation cause. The core completion logic framework of the memory unit is then extracted and associated with the causes of defects in common processing or use processes based on the appearance defect features of the part.

[0083] Next, we analyze the semantic cell refinement features of the existing modality, namely, the feature of scratch image cells: the scratches are straight with smooth edges and concentrated at the edge of the part. We combine these features to adaptively adjust the completion logic parameters. When adjusting the parameters, we refer to the weight adjustment idea of ​​Formula 3 to improve the matching weight between the straight scratch edge smoothness feature and the cause of tool scratches during processing. The original matching weight of 0.65 is adjusted to 0.85.

[0084] Finally, semantic features for the missing modalities that meet the current requirements are generated. The scratches on the surface of this part are caused by tool deviation during machining, which is common in high-speed cutting processes. The generated semantic features are highly consistent with the semantics of existing image cells, filling the semantic gaps caused by modal absence and improving the semantic integrity of the system in modal absence scenarios.

[0085] In this embodiment, the pre-aligned guided retrieval in S4 specifically includes:

[0086] Before multimodal retrieval is initiated, a large-scale language model is used to deeply analyze the user's query intent and comprehensively predict the complete semantic dimensions required to generate content. Semantic dimensions include habitat, feather characteristics, call characteristics, or diet. Based on the prediction results, multiple heterogeneous modal fragments containing corresponding semantic cells are actively screened and recalled during cross-modal retrieval.

[0087] Specifically, during implementation, before the multimodal retrieval is initiated, the user initiates a query on how to identify wild sika deer through multimodal features. The system calls an existing large-scale language model to perform in-depth analysis of the query intent and predict the complete semantic dimensions required to generate the answer, including the sika deer's habitat, physical characteristics, behavioral features, and call features.

[0088] Based on these predicted semantic dimensions, the system actively filters semantic cells from the multimodal data in the local knowledge base. During the filtering process, Formula 1 is used to calculate the semantic similarity between the cell and the semantic dimension. For example, the text cell "inhabiting temperate coniferous and broad-leaved mixed forest" has a similarity of S=0.91 with the habitat dimension; the image cell "the antler bifurcation morphology of the sika deer" has a similarity of S=0.87 with the external features dimension; and the audio cell "the low pitch of the call at 5-second intervals" has a similarity of S=0.89 with the call features dimension. All of these exceed the filtering threshold of 0.8 and are included in the recall range.

[0089] Subsequently, multimodal fragments containing these semantic cells are retrieved, such as real-life images of sika deer in mixed coniferous and broadleaf forests, audio clips of sika deer calls, and text paragraphs describing their habits. Pre-alignment guides the retrieval process, ensuring that the retrieved fragments accurately cover the semantic dimensions required by the user's query, reducing the retrieval of irrelevant data, and improving the accuracy and efficiency of multimodal retrieval.

[0090] In this embodiment, the closed-loop process of pre-avoiding semantic conflicts and forming continuous optimization before generating content in S4 specifically includes:

[0091] The predicted generated semantic dimension is compared and verified with the retrieved semantic cell cluster. If a potential semantic conflict is found, the cell reselection process is automatically triggered to retrieve the fragment containing the correct semantic cell. The "cell combination-alignment effect" prediction model is trained based on historical retrieval-generation data to predict the alignment evaluation accuracy of subsequent retrievals in real time. If the prediction effect does not meet expectations, the semantic cell weights, calibration parameters, or completion logic are adjusted in reverse to form a closed loop of "prediction-execution-verification-optimization".

[0092] Specifically, before generating content, the system compares and verifies the predicted semantic dimension of wild sika deer with the retrieved semantic cell clusters. Formula 1 is used to calculate the semantic similarity between the two. If the retrieved audio cell of winter fur color of sika deer and the predicted text cell of winter smoky brown fur color have an S=0.38, indicating a semantic conflict, the system automatically triggers the cell reselection process and re-retrieves audio and text segments containing the semantic cell of winter smoky brown fur color.

[0093] Meanwhile, the cell combination-alignment effect prediction model is trained using historical retrieval-generated data, and the alignment evaluation accuracy is predicted using Formula 4:

[0094]

[0095] Where Ppred represents the alignment prediction accuracy of the current cell combination; Phist represents the average alignment accuracy of historically identical or similar cell combinations; Mcurrent represents the matching degree between the current cell combination and the semantic dimension, with a value ranging from 0 to 1, obtained by calculating the average similarity between the current cell cluster and the semantic dimension using Formula 1, and has been normalized.

[0096] If in a certain search, Phist=0.85, the average similarity Mcurrent between the current cell cluster and the semantic dimension is 0.9, substituting into Formula 4, we get Ppred=0.85×0.9=0.765, which does not reach the preset expected threshold of 0.8. The system then adjusts the semantic weight of the cell in reverse, such as increasing the weight of the sika deer antler-shaped cell from 0.6 to 0.75. If some semantics are missing due to the completion logic, the feature matching parameters in the completion logic are optimized, such as lowering the feature matching threshold from 0.82 to 0.8.

[0097] In summary, this invention addresses the shortcomings of existing technologies by constructing a dynamic recombination base for cross-modal semantic cells. This base decomposes heterogeneous modalities such as text, images, and audio into semantic cells with modality-identified semantic weights and association thresholds. Combined with dynamic recombination iteration using local data, it solves the problems of neglecting the structural differences in modality embedding space and being unsuitable for local data sparsity. Furthermore, it utilizes a causal-aware predictive distribution calibration mechanism to build a three-layer causal graph of modality, semantics, and scene, locating the root causes of data distribution changes and predictively adjusting cell cluster rules based on time-series data to avoid semantic inaccuracies caused by data distribution shifts. A scene-based memory bank is established through a cross-modal memory transfer missing-filling mechanism, generating semantic features by matching and filling in modal gaps when they are missing. Finally, a pre-aligned guided retrieval-generation closed-loop system analyzes and predicts semantic dimensions in queries, verifies and avoids conflicts, and optimizes parameters in reverse to reduce semantic drift. Overall, this improves the accuracy of cross-modal semantic alignment, reduces factual errors in generated content, adapts to local scene requirements, and enhances the system's reliability in high-precision scenarios.

[0098] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0099] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A cross-modal semantic alignment method for a multimodal LLM-RAG local knowledge base, characterized in that the method... include: S1. Construct a cross-modal semantic cell dynamic recombination base to decompose the original semantic information of various heterogeneous modalities such as text, image, and audio into the smallest independently representative semantic cells. Each semantic cell carries a modality identifier, semantic weight, and association threshold. Based on the real-time dynamic requirements of the local deployment scenario, the semantic cells are dynamically recombined through the "attraction-repulsion" rule between cells to form a semantic cluster adapted to the current scenario. Combined with the unique data continuously accumulated in the local knowledge base, the dynamic autonomous iterative evolution capability of semantic cells is realized. The iterative evolution includes cell splitting or cell fusion. S2. Construct a predictive distribution calibration mechanism for causal perception, which is used to build a causal graph with three layers of association: modality, semantics, and scene, based on the internal data features of the local knowledge base. The causal graph establishes the association links between nodes at each layer. Through deep causal reasoning algorithms, changes in data features are accurately transformed into changes in the weights of causal links to locate the deep root causes of changes in data distribution. Combined with the time-series historical data of the local scene, the accurate dynamic trend of future data distribution is predicted. Based on the prediction results, the internal association rules of the semantic cell cluster are predictively adjusted. S3. Construct a cross-modal memory transfer missing completion mechanism to systematically collect various modal missing scenario cases processed in the local knowledge base during actual operation. Transform each specific case into a memory unit containing rich contextual information, thereby forming a dynamically updated and continuously growing memory bank. When the system detects a specific situation of modal missing, it first accurately matches the memory units of the same scenario in the memory pool through scene tags, extracts its core completion logic framework, and then combines the refined features of the semantic cells of the current existing modality to adaptively adjust the completion logic parameters, and finally generates refined semantic features of the missing modality that are adapted to the current needs. S4. Construct a pre-aligned guided retrieval-generation closed-loop system. Before multimodal retrieval is initiated, a large-scale language model is used to deeply analyze the user's query intent and comprehensively predict the complete semantic dimensions required for generating content. Based on the prediction results, multiple heterogeneous modal fragments containing corresponding semantic cells are actively screened and recalled during cross-modal retrieval. Before generating content, the predicted semantic dimensions of the generated content are compared and verified with the retrieved semantic cell cluster. If potential semantic conflicts are found, the cell reselection process is automatically triggered. A prediction model is trained based on historical retrieval-generation data to predict alignment accuracy in real time and adjust semantic cell weights, calibration parameters, or completion logic in reverse to form a continuously optimized closed-loop process.

2. The method for cross-modal semantic alignment of multi-modal LLM-RAG local knowledge base according to claim 1, characterized in that, The process of decomposing the original semantic information of heterogeneous modalities into the smallest independently representative semantic cells in S1 specifically includes: The text modality is decomposed into entity attribute cells and relation cells; the image modality is decomposed into visual morphology cells and scene cells; the audio modality is decomposed into acoustic event cells and sentiment cells, and each semantic cell carries a modality identifier, semantic weight, and association threshold.

3. The cross-modal semantic alignment method for the multimodal LLM-RAG local knowledge base according to claim 1, characterized in that, The dynamic reorganization of semantic cells in S1 to form a semantic cluster adapted to the current scenario specifically includes: Based on the real-time dynamic needs of the local deployment scenario, the preset "attraction-repulsion" rule between semantic cells is triggered, so that semantic cells with semantic similarity higher than the preset association threshold are automatically aggregated to form a semantic cluster adapted to the current scenario, and the semantic weight of semantically conflicting cells is reduced.

4. The cross-modal semantic alignment method for the multimodal LLM-RAG local knowledge base according to claim 1, characterized in that, The dynamic autonomous iterative evolution capability of semantic cells in S1 specifically includes: By combining the unique data continuously accumulated in the local knowledge base, when new scenario data enters, it triggers the splitting of semantic cells, refining a cell into multiple sub-cells carrying exclusive features; or it triggers the fusion of semantic cells, merging multiple related cells into a comprehensive cell that integrates different modal semantics.

5. The cross-modal semantic alignment method for the multimodal LLM-RAG local knowledge base according to claim 1, characterized in that, The construction of the causal graph with three layers of association between modality, semantics, and scene in S2 specifically includes: Modal layer nodes are constructed to represent text, image, and audio input modalities; semantic layer nodes are constructed to represent entities, attributes, and abstract semantic concepts of relationships; and scene layer nodes are constructed to represent specific application scenarios such as medical diagnosis, industrial quality inspection, and agricultural monitoring. The association links between nodes of each layer are established through causal reasoning algorithms, and changes in data features are accurately transformed into changes in causal link weights, so as to accurately locate the deep root causes of changes in data distribution.

6. The cross-modal semantic alignment method for the multimodal LLM-RAG local knowledge base according to claim 5, characterized in that, The S2 step of adjusting the internal association rules of the semantic cell cluster based on the prediction results specifically includes: By leveraging the link association characteristics of causal graphs and combining local scene time-series data, we can predict the accurate dynamic trend of future data distribution. Time-series data includes, but is not limited to, the arrival of the rainy season or equipment maintenance cycle. Based on the prediction results, we can adjust the internal association rules of semantic cell clusters in advance to strengthen or weaken the aggregation weight between specific semantic cells.

7. The cross-modal semantic alignment method for the multimodal LLM-RAG local knowledge base according to claim 1, characterized in that, The transformation of various modality-deficient scenario cases into memory units containing rich contextual information in S3 specifically includes: The memory unit contains the missing modality type, the existing modality feature description, the completion logic framework, and the application scenario label. The memory unit forms a dynamically updated and continuously growing scenario-based completion experience memory bank, which stores scenario-based completion experience.

8. The cross-modal semantic alignment method for the multimodal LLM-RAG local knowledge base according to claim 7, characterized in that, The refined semantic features for generating the missing modalities adapted to the current requirements in S3 specifically include: When the system detects a specific situation of modality missing, it first accurately matches the memory units of the same scene in the memory bank through scene tags and extracts their core completion logic framework; then, it combines the refined features of the semantic cells of the current existing modality, adaptively adjusts the completion logic parameters, and finally generates refined semantic features of the missing modality that are adapted to the current local requirements.

9. The cross-modal semantic alignment method for the multimodal LLM-RAG local knowledge base according to claim 1, characterized in that, The pre-aligned guided retrieval in S4 specifically includes: Before multimodal retrieval is initiated, a large-scale language model is used to deeply analyze the user's query intent and comprehensively predict the complete semantic dimensions required to generate content. The semantic dimensions include habitat, feather characteristics, call characteristics, or diet. Based on the prediction results, multiple heterogeneous modal fragments containing corresponding semantic cells are actively screened and recalled during the cross-modal retrieval process.

10. The cross-modal semantic alignment method for the multimodal LLM-RAG local knowledge base according to claim 9, characterized in that, The S4 process, which involves pre-avoiding semantic conflicts and forming a closed-loop process of continuous optimization before content generation, specifically includes: The predicted generated semantic dimension is compared and verified with the retrieved semantic cell cluster. If a potential semantic conflict is found, the cell reselection process is automatically triggered to retrieve the fragment containing the correct semantic cell. The "cell combination-alignment effect" prediction model is trained based on historical retrieval-generation data to predict the alignment evaluation accuracy of subsequent retrievals in real time. If the prediction effect does not meet expectations, the semantic cell weights, calibration parameters, or completion logic are adjusted in reverse to form a closed loop of "prediction-execution-verification-optimization".

Images