A reference content global feature encoding method and device based on multi-modal fusion

By eliminating cross-modal semantic redundancy and hierarchical partitioning, combined with global master-slave modality determination and benchmark anchoring, and employing recursive fusion and orthogonal coding, the problems of long sequence information decay and semantic drift in multimodal global feature encoding are solved, achieving high-precision and high-consistency global feature encoding.

CN123069253APending Publication Date: 2026-11-10BEIJING GZT NETWORK TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611170760.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-04
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing multimodal global feature encoding techniques suffer from problems such as long sequence information decay, cross-modal semantic drift, and lack of input-output architecture, which cannot meet the high precision and high consistency requirements of high-end business scenarios.

Method used

We employ cross-modal semantic redundancy resolution, hierarchical partitioning, global master-slave modality determination and benchmark anchoring, recursive fusion and orthogonal encoding methods, and achieve complete representation and stability of global features through a recursive closed-loop mechanism of high-level semantic aggregation and reverse context correction.

Benefits of technology

It significantly improves the completeness and contextual consistency of global feature representations for long-sequence reference content such as long texts and long-time-series videos, significantly enhances the global semantic consistency and stability of multimodal fusion features, improves coding accuracy and computational efficiency, and has a clear semantic hierarchy and interpretability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069253A_ABST
    Figure CN123069253A_ABST
Patent Text Reader

Abstract

This application discloses a method and apparatus for global feature encoding of reference content based on multimodal fusion. The method includes: performing global master-slave modality determination and benchmark anchoring on a purified multimodal semantic unit set with semantic hierarchical classification labels to obtain a global semantic benchmark anchoring configuration; performing global initial feature encoding by recursive fusion of the purified multimodal semantic unit set with semantic hierarchical classification labels and the global semantic benchmark anchoring configuration to obtain a global initial feature encoding vector with anchor point dimension mapping relationship; and performing global semantic hierarchical structured mapping encoding on the global initial feature encoding vector with anchor point dimension mapping relationship to obtain a global feature encoding vector. Addressing the shortcomings of existing encoding methods, such as dimensional semantic coupling redundancy, poor interpretability, and insufficient downstream adaptation flexibility, this application significantly improves the scenario adaptability and engineering usability of global encoding.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and apparatus for global feature encoding of reference content based on multimodal fusion. Background Technology

[0002] With the rapid iteration of multimodal artificial intelligence technology, the demand for semantic comparison and constraint guidance based on multimodal reference content is growing exponentially in business scenarios such as cross-modal content retrieval, conditional content generation, digital asset management, and intelligent content compliance verification. Multimodal reference content global feature encoding, as the core supporting technology for these downstream tasks, plays a crucial role in mapping reference content, which includes any combination of modalities such as text, images, audio, and video, into a dense feature vector of a unified dimension. This allows the feature vector to completely and accurately represent the global core semantics of the reference content, while possessing cross-modal semantic consistency, high discriminative power, strong generalization, and low redundancy, directly determining the upper limit of accuracy and usability of downstream tasks.

[0003] It is well known in this field that unimodal global feature encoding is the foundation of cross-modal fusion. Currently, mainstream unimodal global encoding schemes often use the self-attention mechanism of the Transformer architecture to complete semantic modeling, generating global features through CLS tag extraction and global average pooling. Visual modalities such as images and videos often use convolutional neural networks or visual Transformer architectures to complete feature extraction, generating global representations through pyramid feature aggregation and attention-weighted pooling. However, these schemes all employ a forward-oriented unidirectional feature aggregation paradigm. When modeling long-sequence reference content such as long texts and long-time-sequence videos, the dependency weights of distant semantic units decay exponentially with the sequence length, failing to fully preserve the hierarchical semantic relationships between fine-grained semantic units, mid-level semantic clusters, and global topics. This results in a fundamental problem of long-sequence global information loss, and the semantic coverage and contextual consistency of unimodal global features face inherent bottlenecks that are difficult to overcome.

[0004] At the cross-modal fusion coding level, existing technologies have undergone multiple rounds of evolution: early solutions mostly adopted feature concatenation and gated linear fusion, directly performing dimensional concatenation or weighted summation on global features of different modalities. This type of solution has low computational complexity but does not achieve deep alignment at the semantic level, resulting in poor cross-modal semantic consistency of fused features. Current mainstream solutions adopt a technical approach of cross-attention alignment and contrastive learning joint embedding, achieving cross-modal semantic alignment through element-wise cross-attention operations or global contrastive loss constraints, which improves the semantic consistency of fused features to some extent. However, both of the above solutions adopt a modality-equivalent local element-wise alignment mode, without establishing a unified anchoring benchmark at the global semantic level. Error accumulation and semantic drift are prone to occur during multimodal cross-alignment. At the same time, weakly correlated and noisy semantic units can directly participate in the fusion process, resulting in a high proportion of modal noise in the fused features and insufficient stability of the global semantic center. This defect is further amplified in fusion scenarios with three or more modalities, failing to meet the application requirements of high-precision scenarios.

[0005] Furthermore, existing multimodal coding techniques generally suffer from two deficiencies: input preprocessing and output form design. On the input side, existing solutions assume that all modal semantics are effective complementary information and do not address the common problem of cross-modal semantic repetition in multimodal reference content. The same semantic information is repeatedly encoded in multiple modalities, directly leading to semantic bias and dimensional redundancy in fused features, resulting in simultaneous damage to coding efficiency and representation accuracy. On the output side, existing coding schemes all output flattened one-dimensional feature vectors with severe semantic coupling and overlap between feature dimensions. There is no clear semantic hierarchy, which not only results in poor interpretability but also prevents downstream tasks from calling corresponding semantic features in layers according to their own needs, making it difficult to adapt to the differentiated representation requirements of different business scenarios.

[0006] In summary, existing multimodal global feature encoding techniques suffer from multiple inherent limitations, including long-sequence information decay due to single-modal aggregation, semantic drift without a reference point during cross-modal alignment, and architectural deficiencies at both input and output ends. These limitations prevent them from meeting the high-precision, high-consistency, and high-availability requirements of high-end business scenarios for global encoding. Currently, there is no technical solution in this field that can simultaneously address all of the aforementioned shortcomings at the end-to-end architectural level. Therefore, a novel global feature encoding method based on multimodal fusion is urgently needed to overcome the inherent bottlenecks of existing technologies. Summary of the Invention

[0007] The purpose of this invention is to provide a global feature encoding method for reference content based on multimodal fusion to at least solve one of the above-mentioned technical problems.

[0008] One aspect of the present invention provides a global feature encoding method for reference content based on multimodal fusion, the global feature encoding method for reference content based on multimodal fusion comprising:

[0009] Step 1: Obtain the initial semantic unit feature set corresponding to the multimodal reference content to be encoded;

[0010] Step 2: Perform cross-modal semantic redundancy resolution and hierarchical division on the initial semantic unit feature set corresponding to the multimodal reference content to be encoded, thereby obtaining a purified multimodal semantic unit set with semantic hierarchical classification labels;

[0011] Step 3: Perform global master-slave modality determination and baseline anchoring on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the global semantic baseline anchoring configuration;

[0012] Step 4: Perform global feature initial encoding by recursively fusing multimodal semantic units with semantic hierarchical classification labels and global semantic benchmark anchoring configuration to obtain a global feature initial encoding vector with anchor point dimension mapping relationship.

[0013] Step 5: Perform global semantic hierarchical structured mapping encoding on the initial global feature encoding vector with anchor dimension mapping relationship to obtain the global feature encoding vector.

[0014] Optionally, step 2 includes:

[0015] Step 21: Perform cross-modal unit pairwise semantic similarity calculation and redundant grouping on the initial semantic unit feature set to obtain the initial classification result containing cross-modal redundant semantic groups and independent semantic units;

[0016] Step 22: Perform core semantic unit screening and duplicate redundant unit removal on the cross-modal redundant semantic group to obtain a set of effective semantic units after deduplication;

[0017] Step 23: Perform semantic hierarchical attribute mapping and classification labeling on the deduplicated effective semantic unit set to obtain a purified multimodal semantic unit set with semantic hierarchical classification labels.

[0018] Optionally, step 3 includes:

[0019] Step 31: Perform modal-by-modal global semantic density statistical calculation on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the set of global semantic density parameters for each modality;

[0020] Step 32: Sort the global semantic density parameter sets of each modality by semantic carrying capacity and classify modal roles to obtain the modal master-slave role classification results including the dominant modality and subordinate modality;

[0021] Step 33: Based on the modal master-slave role division results, extract and encapsulate the global semantic benchmark anchor of the dominant modality to obtain the global semantic benchmark anchor configuration.

[0022] Optionally, step 4 includes:

[0023] Step 41: Perform initial semantic clustering and recursive global context aggregation on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the global feature set of each modality hierarchy;

[0024] Step 42: Combine the global feature set of each modality hierarchy with the global semantic benchmark anchoring configuration to perform benchmark anchor point matching and cross-modal unidirectional calibration fusion, thereby obtaining cross-modal fusion features with anchor point matching correlation;

[0025] Step 43: Perform orthogonal semantic basis construction and fidelity mapping encoding on the cross-modal fusion features with anchor matching relationships to obtain the initial global feature encoding vector with anchor dimension mapping relationships.

[0026] Optionally, step 5 includes:

[0027] Step 51: Perform semantic hierarchy attribution matching and dimension segmentation on the global feature initial encoding vector with anchor dimension mapping relationship to obtain a three-segment semantic dimension segmentation result;

[0028] Step 52: Perform segmented orthogonal decoupling and inter-segment semantic interference elimination on the three-segment semantic dimension segmentation results to obtain an orthogonal and independent semantic segmentation encoding set;

[0029] Step 53: Perform fixed-order concatenation and normalization on the orthogonal independent semantic segment coding set to obtain the structured global feature coding vector.

[0030] Optionally, step 41 includes:

[0031] Step 411: Perform fixed-scale semantic clustering and cluster center feature extraction on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the initial semantic cluster feature set of each modality;

[0032] Step 412: Perform multi-level forward global aggregation and reverse context correction recursive closed-loop processing on the initial semantic cluster feature sets of each modality to obtain the multi-level recursive cluster feature sets of each modality;

[0033] Step 413: Perform hierarchical semantic regularization and dimensional unification on the feature sets of each modality's multi-level recursive clusters to obtain the hierarchical global feature sets of each modality.

[0034] Optionally, step 412 includes:

[0035] Step 4121: Perform intra-cluster self-attention semantic refinement and inter-cluster global interaction aggregation on the semantic cluster feature set of the current level to obtain the high-level global cluster feature set of the next higher level;

[0036] Step 4122: Map the previous level high-level global cluster feature set in reverse to the current level semantic space, perform low-level feature weighting correction under global semantic constraints, and thus obtain the current level semantic cluster feature set after context calibration;

[0037] Step 4123: Repeat the forward aggregation and backward correction closed-loop operation layer by layer according to the preset fixed recursion layer number until all levels of processing are completed, thereby obtaining the feature set of multi-level recursive clusters of each modality.

[0038] Optionally, step 43 includes:

[0039] Step 431: Based on the cross-modal fusion features with anchor matching relationships, generate initial vectors for each set of matching anchor pairs corresponding to independent semantic dimensions, thereby obtaining the initial semantic basis vector set;

[0040] Step 432: Perform Schmitt orthogonalization and semantic energy fidelity constraint correction on the initial semantic basis vector set to obtain an orthogonal and interference-free semantic basis matrix;

[0041] Step 433: Map the cross-modal fusion features to the orthogonal semantic base space and perform dimensional transformation and normalization to obtain the initial global feature encoding vector with anchor dimensional mapping relationship.

[0042] This application also provides a global feature encoding device for reference content based on multimodal fusion, the global feature encoding device for reference content based on multimodal fusion includes:

[0043] An initial semantic unit feature set acquisition module is used to acquire the initial semantic unit feature set corresponding to the multimodal reference content to be encoded.

[0044] A purified multimodal semantic unit set acquisition module is used to perform cross-modal semantic redundancy resolution and hierarchical division on the initial semantic unit feature set corresponding to the multimodal reference content to be encoded, thereby obtaining a purified multimodal semantic unit set with semantic hierarchical classification labels.

[0045] A global semantic benchmark anchoring configuration acquisition module is used to perform global master-slave modality determination and benchmark anchoring on a set of purified multimodal semantic units with semantic hierarchical classification labels, thereby acquiring the global semantic benchmark anchoring configuration.

[0046] The global feature initial encoding vector acquisition module is used to perform global feature initial encoding by multimodal semantic anchor recursive fusion on the purified multimodal semantic unit set with semantic hierarchical classification labels and global semantic benchmark anchoring configuration, so as to obtain a global feature initial encoding vector with anchor point dimension mapping relationship.

[0047] The global feature encoding vector acquisition module is used to perform global semantic hierarchical structured mapping and encoding on the initial global feature encoding vector with anchor dimension mapping relationship, thereby obtaining the global feature encoding vector.

[0048] The reference content global feature encoding method based on multimodal fusion proposed in this application has the following advantages:

[0049] To address the inherent shortcomings of existing single-modal unidirectional aggregation architectures, such as long-distance dependency decay and loss of hierarchical semantic associations, this invention adopts a recursive closed-loop aggregation mechanism of forward high-level semantic aggregation and reverse global context correction. By reverse weighting and calibrating low-level fine-grained cluster features through high-level global semantic features, this invention fundamentally breaks through the information transmission bottleneck of unidirectional aggregation, fully preserves the complete hierarchical semantic associations of reference content from fine-grained semantic units to global core themes, and significantly improves the integrity of global feature representation and contextual consistency of long-sequence reference content such as long texts and long-time-series videos.

[0050] To address the shortcomings of existing peer-to-peer modality cross-alignment schemes, such as the lack of a unified semantic benchmark, error accumulation and drift, and the susceptibility of noise features to interfering with global semantics, this invention adopts a one-way calibration fusion mechanism based on global master-slave modality determination and benchmark anchoring. The dominant modality with the highest global semantic density is used as the sole semantic benchmark, and all subordinate modalities perform globally optimal anchor matching and one-way semantic calibration towards the dominant modality, thus fundamentally avoiding the error accumulation problem in multimodal cross-alignment. At the same time, by screening anchors with high semantic contribution, weakly correlated noise units are naturally filtered out, significantly improving the global semantic consistency and stability of multimodal fusion features, especially in fusion scenarios with three or more modalities.

[0051] To address the shortcomings of existing encoding schemes that fail to handle cross-modal semantic repetition in multimodal reference content, leading to semantic bias and dimensional redundancy in fusion features, this invention incorporates cross-modal semantic redundancy resolution and hierarchical partitioning steps in the pre-encoding stage. By determining cross-modal semantic similarity and filtering core units to eliminate repetitive and redundant semantics, the effective semantics are divided into two levels: cross-modal common semantics and modal-independent complementary semantics. This purifies the encoding semantic base from the input source, avoiding fusion weight bias caused by repetitive semantics and reducing the amount of ineffective computation in subsequent encoding stages, thus achieving a simultaneous improvement in encoding accuracy and computational efficiency.

[0052] To address the shortcomings of existing flat global encoding methods, such as dimensional semantic coupling redundancy, poor interpretability, and insufficient downstream adaptability, this invention adopts semantic-oriented orthogonal fidelity encoding and a three-segment structured mapping mechanism. Based on matching anchor points, an orthogonal semantic base is constructed to achieve redundancy-free fidelity encoding, completely eliminating semantic overlap between feature dimensions. At the same time, the encoding vector is divided into three orthogonal independent semantic segments according to semantic affiliation: global commonality, dominant core, and multimodal complementarity. This gives the output encoding a clear semantic hierarchy and interpretability. Downstream tasks can call corresponding semantic features segment by segment according to their own needs, greatly improving the scenario adaptability and engineering usability of global encoding. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating a global feature encoding method for reference content based on multimodal fusion according to an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical applications, and advantages of this application clearer, the technical applications in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, of the embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0055] like Figure 1 The reference content global feature encoding method based on multimodal fusion shown includes:

[0056] Step 1: Obtain the initial semantic unit feature set corresponding to the multimodal reference content to be encoded;

[0057] Step 2: Perform cross-modal semantic redundancy resolution and hierarchical division on the initial semantic unit feature set corresponding to the multimodal reference content to be encoded, thereby obtaining a purified multimodal semantic unit set with semantic hierarchical classification labels;

[0058] Step 3: Perform global master-slave modality determination and baseline anchoring on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the global semantic baseline anchoring configuration;

[0059] Step 4: Perform global feature initial encoding by recursively fusing multimodal semantic units with semantic hierarchical classification labels and global semantic benchmark anchoring configuration to obtain a global feature initial encoding vector with anchor point dimension mapping relationship.

[0060] Step 5: Perform global semantic hierarchical structured mapping encoding on the initial global feature encoding vector with anchor dimension mapping relationship to obtain the global feature encoding vector.

[0061] In this embodiment, obtaining the initial semantic unit feature set corresponding to the multimodal reference content to be encoded includes:

[0062] For each modality, a targeted standardization process is performed on the original reference content to eliminate interference from non-semantic differences such as format, size, and duration on subsequent feature extraction. The preprocessing rules for the three modalities in this embodiment are as follows:

[0063] Text Modality: Perform full cleaning on the original product description text, remove redundant format marks, special symbols and meaningless stop words, and unify the full-width / half-width punctuation format; use Chinese word segmentation tools to complete semantic word segmentation, and perform truncation and completion processing on the text according to a fixed maximum sequence length of 512 to ensure that the input sequence length is consistent.

[0064] Image modality: Product structure diagrams of different formats are uniformly converted into RGB three-channel format, and images with transparent channels are filled with pure white transparent areas; bicubic interpolation algorithm is used to normalize the image size to 224×224 pixels, and images with mismatched aspect ratios are processed by proportional scaling + edge grayscale filling to avoid deformation of the main body; linear normalization is performed on pixel values, mapping the original pixel values ​​in the range of 0~255 to the range of [-1,1], matching the input value range of subsequent feature extraction networks.

[0065] Audio modality: Pre-emphasis processing is performed on the original functional explanation audio to enhance high-frequency components. Framing and Hamming windowing are completed with fixed parameters of 25ms frame length and 10ms frame shift. Mel spectrum transformation is performed on each frame of audio signal to generate 64-dimensional Mel spectrum features, which are then uniformly trimmed and completed to a fixed timing length of 300 frames.

[0066] For each modality, a corresponding basic feature extraction network is configured to extract fine-grained semantic unit features for each modality. All network output features are uniformly mapped to a fixed dimension d through a linear projection layer. In this embodiment, d is set to 768. This uniform dimension linear projection layer belongs to the output layer of the basic feature extraction network for each modality. During the training phase, it remains frozen along with the basic feature extraction network as a whole, and its parameters do not participate in the joint optimization of the core encoding network. If domain fine-tuning is to be performed, it can be updated synchronously with the top layer of the basic extraction network.

[0067] The text modality employs a 12-layer Transformer encoder as the basic extraction network. Taking the preprocessed word segmentation sequence as input, it outputs a semantic feature vector corresponding to each word token, forming the initial semantic unit feature sequence of the text, denoted as... , where 512 is the number of text semantic units and 768 is the uniform feature dimension.

[0068] Image modality extraction uses a visual Transformer encoder as the basic extraction network. Taking a normalized RGB image as input, the image is divided into 196 image blocks according to a 16×16 block rule. The output is the semantic feature vector corresponding to each image block, forming the initial semantic unit feature sequence of the image, denoted as . .

[0069] The audio modality employs a three-layer temporal convolutional encoder as the basic extraction network. Taking the Mel-spectral temporal sequence as input, it outputs a temporal semantic feature vector corresponding to each frame of audio, forming the initial semantic unit feature sequence of the audio, denoted as... .

[0070] The initial semantic unit feature sequences of the three modalities are summarized according to modality to form the initial semantic unit feature set corresponding to the multimodal reference content to be encoded.

[0071] In this embodiment, step 2 includes:

[0072] Step 21: Perform cross-modal unit pairwise semantic similarity calculation and redundant grouping on the initial semantic unit feature set to obtain the initial classification result containing cross-modal redundant semantic groups and independent semantic units;

[0073] In this embodiment, pairwise semantic similarity calculation is performed on all semantic units of different modalities, and redundant semantic grouping is completed based on a fixed threshold. The specific implementation process is as follows:

[0074] Iterate through all cross-modal combinations. This embodiment includes three cross-modal combinations: text-image, text-audio, and image-audio. For each combination, pair up the semantic units and calculate the cosine similarity between the two feature vectors. The calculation formula is as follows:

[0075] ;

[0076] in, Let i be the feature vector of the i-th semantic unit in the first modality. Let j be the feature vector of the j-th semantic unit in the second modality. It is the vector 2 norm, and the similarity value ranges from [0,1]. The higher the value, the higher the semantic overlap between the two semantic units.

[0077] Set a fixed redundancy threshold In this embodiment The value is set to 0.85; semantic unit pairs with cosine similarity higher than the threshold are marked as cross-modal redundant association pairs.

[0078] Based on association relationships, connected component grouping is performed. All semantic units with redundant associations are grouped into the same cross-modal redundant semantic group, while semantic units without any cross-modal redundant associations are marked as independent semantic units. The final classification result consists of several cross-modal redundant semantic groups and all independent semantic units. Grouping is implemented using transitive closure logic: if two semantic units have a direct redundant association, or form an indirect association link through other units, then all associated units are assigned to the same cross-modal redundant semantic group, ensuring that all cross-modal units corresponding to the same semantic unit are grouped into the same group.

[0079] Step 22: Perform core semantic unit screening and duplicate redundant unit removal on the cross-modal redundant semantic group to obtain a set of effective semantic units after deduplication;

[0080] In this embodiment, the core unit with the highest information density is selected for each cross-modal redundant semantic group, and the remaining duplicate and redundant units are removed to complete semantic deduplication. The specific implementation process is as follows:

[0081] For each group of cross-modal redundant semantic groups, the semantic information entropy is calculated unit by unit to characterize the semantic information carrying density of that unit. The calculation formula is as follows:

[0082] ;

[0083] in, Let be the feature vector of the semantic unit to be calculated, and d be the feature dimension. The normalized probability value of the k-th dimension of the feature vector is obtained by normalizing the value of this dimension using Softmax. The higher the information entropy value, the richer the semantic information carried by the unit.

[0084] The semantic unit with the highest information entropy in the group is selected as the core semantic unit of the redundant group. All other semantic units in the group are marked as duplicate redundant units and directly removed to avoid duplicate semantics from participating in the subsequent encoding process.

[0085] The core semantic units retained from all redundant groups are combined with all independent semantic units to obtain the deduplicated set of effective semantic units. In this embodiment, after deduplication, the total number of effective semantic units is compressed by approximately 22% compared to the initial set, effectively reducing the computational load in subsequent encoding stages.

[0086] Step 23: Perform semantic hierarchical attribute mapping and classification labeling on the deduplicated effective semantic unit set to obtain a purified multimodal semantic unit set with semantic hierarchical classification labels.

[0087] After deduplication, the effective semantic units are subjected to semantic hierarchy partitioning and labeling to form a purified semantic set with hierarchical attributes. The specific implementation process is as follows:

[0088] Establish semantic hierarchy classification rules: Core units derived from cross-modal redundant semantic groups are marked as cross-modal common semantic units, corresponding to the common core semantics carried by each modality in the multimodal reference content; Independent semantic units without cross-modal redundant associations are marked as modal independent complementary semantic units of the corresponding modality, corresponding to the supplementary semantic information unique to that modality.

[0089] Each valid semantic unit is assigned a classification label, which includes two attributes: the modality to which the unit belongs and the semantic level category. All units are stored in groups according to the modality dimension.

[0090] Finally, a refined set of multimodal semantic units with semantic hierarchical classification labels is generated.

[0091] Semantic hierarchical classification labels are transmitted synchronously along the entire encoding process: In step 41, during semantic clustering, the label of each semantic cluster is determined by the label of the semantic unit with the highest proportion within the cluster; in step 42, during anchor point selection, each anchor point inherits the original label of the corresponding semantic unit; in step 43, during orthogonal encoding, each encoding dimension inherits the label of the corresponding anchor point; finally, in step 5, semantic segment division is completed based on the dimension labels, forming a complete semantic label transmission link from the original input to the final encoding.

[0092] In this embodiment, step 3 includes:

[0093] Step 31: Perform modal-by-modal global semantic density statistical calculation on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the set of global semantic density parameters for each modality;

[0094] In this embodiment, for each modality's effective semantic unit set, the global semantic density of that modality is calculated to quantify the carrying strength of a single modality for the core global information of the reference content. The effective semantic unit set of a single modality includes two categories: cross-modal common semantic units of the corresponding modality and modality-independent complementary semantic units.

[0095] The formula for calculating global semantic density is:

[0096] ;

[0097] in, Let be the global semantic density of the i-th modality; This represents the total number of valid semantic units in this modality. This is the feature vector of the j-th effective semantic unit of this modality; The semantic information entropy calculation function is consistent with the semantic information entropy calculation rule in step 2; the higher the semantic density value, the denser the global core information carried by the modality, and the stronger the semantic representativeness.

[0098] Step 32: Sort the global semantic density parameter sets of each modality by semantic carrying capacity and classify modal roles to obtain the modal master-slave role classification results including the dominant modality and subordinate modality;

[0099] In this embodiment, based on the global semantic density parameters of each modality, modal roles are divided according to fixed rules to determine the baseline modality and subordinate modalities for global fusion.

[0100] The specific partitioning rule is as follows: sort all input modalities in descending order of global semantic density value, select the modality ranked first as the global dominant modality, and uniformly mark all other modalities as subordinate modalities; the dominant modality serves as the sole benchmark for subsequent cross-modal semantic alignment, and the semantic space of all subordinate modalities is aligned unidirectionally to the dominant modality to avoid the error accumulation problem caused by the cross-alignment of multiple modalities.

[0101] After the partitioning is completed, a modality master-slave role partitioning result is generated, which includes the dominant modality identifier and the set of subordinate modalities.

[0102] Step 33: Based on the modal master-slave role division results, extract and encapsulate the global semantic benchmark anchor of the dominant modality to obtain the global semantic benchmark anchor configuration.

[0103] Specifically, based on the modality master-slave role division results, the global semantic benchmark anchor of the dominant modality is extracted and encapsulated into a standardized global semantic benchmark anchor configuration.

[0104] The specific implementation process is as follows: global average pooling is performed on all effective semantic units of the dominant modality to generate an initial global semantic benchmark anchor vector with the same dimension as the basic feature dimension d. This vector represents the core global semantic center of the entire multimodal reference content.

[0105] The modal master-slave role division results are uniformly encapsulated with the initial global semantic benchmark anchor vector to generate a complete global semantic benchmark anchoring configuration.

[0106] Step 4 includes:

[0107] Step 41: Perform initial semantic clustering and recursive global context aggregation on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the global feature set of each modality hierarchy;

[0108] In this embodiment, step 41 addresses the inherent flaws of existing single-modal global feature aggregation architectures, such as long-sequence long-distance semantic dependency decay, hierarchical semantic association breakage, and inability to fully preserve the semantic links from fine-grained units to the global topic. Instead, it employs a hierarchical modeling architecture combining fixed-scale semantic clustering with a bidirectional recursive closed loop to independently complete deep global semantic representation for each modality. The input to this sub-step is the set of effective semantic units for each modality with semantic hierarchical classification labels output from step 2, and the output is the hierarchical global feature set for each modality. All operations are performed based on offline trained and fixed network parameters, with no parameter updates during the inference phase. The specific implementation process is as follows:

[0109] Step 41 includes:

[0110] Step 411: Perform fixed-scale semantic clustering and cluster center feature extraction on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the initial semantic cluster feature set of each modality;

[0111] The core function of this sub-step is to compress a dynamically changing sequence of semantic units into a fixed-size set of semantic clusters, solving the problem of dimensional mismatch in subsequent processing caused by inconsistent lengths of different input samples, and providing a unified input format for standardized recursive operations.

[0112] The specific implementation process is as follows: For each modality's effective semantic unit set, K-means semantic clustering with a fixed number of clusters K is performed. Cosine similarity is used as the cluster distance metric, and the convergence objective is to minimize intra-cluster feature distance and maximize inter-cluster feature distance. After iterative convergence, all semantic units are divided into K semantic clusters. The number of clusters K is a fixed value used for offline calibration. All input samples of the same modality use the same number of clusters to ensure consistency in output dimensions. In this embodiment, the initial number of clusters K is 64. The K-means++ algorithm is used for initial cluster center selection during cluster initialization. The convergence termination condition is that the offset of the cluster center is less than... The clustering process terminates when either condition is met, or when the maximum number of iterations (100) is reached.

[0113] Global mean pooling is performed on all unit features within each semantic cluster to calculate the central feature vector of that cluster. The central features of all clusters together constitute the initial semantic cluster feature set for each modality, with a set dimension of . , where d is the unified dimension of the basic features.

[0114] Step 412: Perform multi-level forward global aggregation and reverse context correction recursive closed-loop processing on the initial semantic cluster feature sets of each modality to obtain the multi-level recursive cluster feature sets of each modality;

[0115] Specifically, unlike existing technologies that only have a hierarchical architecture of bottom-up unidirectional feature aggregation, this step adopts a bidirectional recursive closed-loop architecture of forward high-level semantic aggregation and reverse global context correction. While generating high-level semantic features by aggregating upwards layer by layer, high-level global semantic constraints are injected back into low-level fine-grained features. This breaks the information transmission bottleneck of unidirectional aggregation from a mechanism perspective, solves the inherent problem of long-distance dependency decay, and fully preserves the complete hierarchical semantic association from local fine-grained semantics to global core themes.

[0116] This step pre-determines a fixed number of recursion levels L. The recursion levels increase from low to high, corresponding to a progressively coarser semantic coverage. Closed-loop processing is performed layer by layer, starting from the lowest initial semantic cluster. In this embodiment, the total number of recursion levels L is 4, and the number of clusters per layer is halved; that is, the number of semantic clusters is halved with each upward recursion. Specifically, it includes three sequential execution steps:

[0117] Step 4121: Perform intra-cluster self-attention semantic refinement and inter-cluster global interaction aggregation on the semantic cluster feature set of the current level to obtain the high-level global cluster feature set of the next higher level;

[0118] Specifically, the purpose of this sub-step is to complete the internal semantic condensation of the current level cluster features and aggregate them upwards to generate higher-level global cluster features, thereby achieving hierarchical improvement in semantic granularity. This is divided into two operational stages:

[0119] Intra-cluster self-attention semantic refinement

[0120] The input is the set of all original semantic unit features corresponding to the k-th semantic cluster in the current level, with dimension 1. ,in This represents the number of semantic units contained within the cluster.

[0121] It should be noted that for the first-level (lowest-level) semantic cluster, the input here is the set of features of all original semantic units corresponding to that cluster; for the second-level and higher-level semantic clusters, the input here is the set of central features of all the next-level clusters contained in that higher-level cluster. This represents the number of next-level clusters contained within this higher-level cluster.

[0122] By modeling the internal semantic associations of units within a cluster using standard scaled dot product self-attention operations, the semantic cohesion of features within the cluster is enhanced. The calculation formula is as follows:

[0123] ;

[0124] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively, which are obtained by mapping the intra-cluster unit features through three independent linear projection layers. The three sets of projection weights are fixed parameters that are solidified during network training. The feature dimension of the key vector; This is a scaling factor used to prevent the dot product result from becoming too large, which could cause gradient saturation in the Softmax function. In this embodiment, the attention feature dimension... The value is 384.

[0125] The output of the operation is a semantically refined set of intra-cluster features. Global mean pooling is then performed on this set to obtain the semantically enhanced current cluster center features. After all clusters have undergone the above operation, the semantically refined set of current-level cluster features is obtained, with a dimension of [missing value]. ,in This represents the number of semantic clusters at the current level.

[0126] Inter-cluster global interaction aggregation

[0127] The input is the semantically refined set of cluster features at the current level. Using all current cluster center features as input, the corresponding global query matrix, key matrix, and value matrix are generated. Cross-cluster self-attention operation is performed to model the global semantic association between all clusters and obtain enhanced cluster features that integrate inter-cluster context.

[0128] According to the preset hierarchical cluster number compression rules, the current level of clusters will be compressed. semantic clusters are aggregated into A high-level semantic cluster, satisfying < Each high-level cluster's feature is the weighted sum of the attention of its corresponding lower-level clusters, with the weights obtained through cross-cluster self-attention operations. This ultimately generates a set of global cluster features for the next higher level. The higher the level, the fewer the number of clusters, and the wider the global semantic coverage of a single cluster.

[0129] The correspondence between low-level clusters and high-level clusters is determined by a greedy clustering method: based on the inter-cluster correlation matrix obtained by cross-cluster self-attention operation, starting from the first ungrouped low-level cluster, the remaining low-level clusters with the highest correlation are selected and merged into a cluster group. This process is repeated until all low-level clusters are divided, and then repeated until the number of cluster groups equals the preset number of high-level clusters. After completing the grouping of all low-level clusters, the final result is... There are several cluster groups, each corresponding to a high-level semantic cluster; the feature of each high-level cluster is the weighted sum of the features of all low-level clusters within the cluster group according to their correlation weights.

[0130] Step 4122: Map the previous level high-level global cluster feature set in reverse to the current level semantic space, perform low-level feature weighting correction under global semantic constraints, and thus obtain the current level semantic cluster feature set after context calibration;

[0131] The purpose of this sub-step is to transmit the high-level global semantics back to the current level, calibrate the local semantic deviations of the low-level fine-grained features, and strengthen the semantic association links between levels. This reverse correction link is not available in the existing unidirectional hierarchical aggregation architecture.

[0132] The specific implementation process is as follows: The generated set of global cluster features from the previous level is input into the trained and fixed back linear projection layer. This layer is then upsampled and mapped to a feature dimension that perfectly matches the current level's cluster features, resulting in global semantic constraint features with the same dimension as the current level's cluster features. .

[0133] The inverse linear projection layer uses a fully connected linear transformation to achieve upsampling of the sequence dimension. The specific operation process is as follows: first, the dimension is... The high-level cluster features are flattened by row as the dimension. A one-dimensional vector, after a linear transformation of the trainable weight matrix and the bias, is reshaped into... The two-dimensional feature matrix is ​​used to perform upsampling of the sequence dimension, and the operation satisfies:

[0134] ;

[0135] in, To perform row-by-row flattening operations, Dimensional reshaping operations; The weight matrix is ​​trainable and has dimensions of . ; The bias vector is a trainable bias vector with dimension . .

[0136] Using global semantic constraint features as weight masks, element-wise weighted correction is performed on the original cluster features of the current level. The calculation formula is as follows:

[0137] ;

[0138] in, This is the set of original cluster features at the current level; These are the global semantic constraint features after reverse mapping; is the Sigmoid activation function, used to normalize the values ​​of constrained features to the [0,1] interval and generate a standardized weight mask; ⊙ is the element-wise multiplication operation.

[0139] The output of the operation is a set of semantic cluster features of the current level after context calibration, which embeds the low-level fine-grained features into the global semantic context, avoids the local semantics from deviating from the global core theme of the reference content, and alleviates the problem of semantic weight decay at the end of long sequences from the root.

[0140] Each recursive level corresponds to an independent back linear projection layer. The trainable parameters of each layer are independent of each other and are adapted to the input and output sequence dimensions of the corresponding level. The parameters are not shared.

[0141] Step 4123: Repeat the forward aggregation and backward correction closed-loop operation layer by layer according to the preset fixed recursion layer number until all levels of processing are completed, thereby obtaining the feature set of multi-level recursive clusters of each modality.

[0142] The purpose of this sub-step is to complete bidirectional processing across all recursive levels and construct a multi-level semantic representation system covering the entire granularity.

[0143] The specific iteration rule is as follows: starting from the initial semantic cluster at the bottom layer (level 1), first perform forward aggregation to generate high-level features at level 2, and then calibrate the cluster features at level 1 through reverse correction; then, based on the cluster features at level 2, perform forward aggregation to generate high-level features at level 3, and then calibrate the cluster features at level 2 through reverse correction; repeat the closed-loop operation of forward aggregation and reverse correction layer by layer according to the above rule until the recursive processing of all L levels is completed.

[0144] The current-level calibration cluster features obtained through reverse correction are only used to enrich the semantic representation output of this level and do not participate in the forward aggregation process of higher levels. All forward aggregations at higher levels are always performed based on the original cluster features of the previous level after forward refinement, ensuring the stability of high-level semantic aggregation. The top level (level L) has no reverse correction step and directly outputs the top-level global semantic center features obtained through forward aggregation.

[0145] After iteration, a set of multi-level recursive cluster features for each modality with L semantic levels is finally obtained, covering the complete semantic link from fine-grained local semantics to global topic semantics.

[0146] Step 413: Perform hierarchical semantic regularization and dimensional unification on the feature sets of each modality's multi-level recursive clusters to obtain the hierarchical global feature sets of each modality.

[0147] For each level of feature in the multi-level recursive cluster feature set of each modality, dimensionality calibration is performed through an independent linear projection layer, so that features of the same level in all modalities are uniformly mapped to the same feature dimension. After normalization, a hierarchical global feature set for each modality is generated, where the Lth level (topmost) feature corresponds to the global semantic center feature of the corresponding modality.

[0148] Step 42: Combine the global feature set of each modality hierarchy with the global semantic benchmark anchoring configuration to perform benchmark anchor point matching and cross-modal unidirectional calibration fusion, thereby obtaining cross-modal fusion features with anchor point matching correlation;

[0149] This sub-step addresses the systemic defects of existing technologies' commonly used bidirectional cross-alignment architecture for peer modalities, such as the lack of a unified global semantic benchmark, the accumulation and drift of pairwise alignment errors across multiple modalities, and the interference of weak semantic noise units with the stability of the fusion center. By using a determined dominant modality as the sole global semantic benchmark, all subordinate modalities perform unidirectional anchoring calibration and feature fusion only towards the dominant modality. This avoids the problem of error accumulation in multimodal cross-alignment from the root of the architecture. At the same time, by screening anchors with high semantic contribution, low-value noise units are naturally filtered out, thereby improving the global semantic consistency and stability of the fused features.

[0150] This sub-step takes as input a hierarchical global feature set for each modality and a global semantic benchmark anchoring configuration, and outputs cross-modal fusion features with anchor point matching relationships. All operations are performed based on offline trained and fixed network parameters, with no parameter updates during the inference phase. The dominant / subordinate modality identifier in the global semantic benchmark anchoring configuration is used for modality role determination during the master-slave anchoring calibration fusion phase. The initial global semantic benchmark anchor is only used as a verification reference for anchor point selection; the actual contribution is calculated based on the recursively calculated top-level global semantic center to ensure semantic hierarchy matching. The specific implementation process is as follows:

[0151] Step 421: Filtering anchors with high semantic contribution

[0152] This sub-step selects core units with strong semantic representativeness from the top-level global features of each modality as alignment anchors. Unlike existing full-feature cross-alignment schemes, it only uses units with high semantic contribution as the benchmark for cross-modal alignment. This naturally filters out noisy units with low semantic density and edge-irrelevant features, significantly reducing the computational cost of invalid alignment while avoiding weak semantic units from interfering with the stability of the semantic center of global fusion.

[0153] The specific implementation process is as follows: Based on the global semantic benchmark anchoring configuration, the global semantic contribution of the top-level global features of the dominant modality and each subordinate modality is calculated on a unit-by-unit basis. The calculation formula is as follows:

[0154] ;

[0155] in, The global semantic contribution of the j-th feature unit of the i-th modality; This represents the attention weight of the feature unit relative to the corresponding modality global semantic center. The higher the weight value, the stronger the support of the unit for the modality global semantics. The semantic information entropy of this feature unit is calculated using the same rules as those for calculating semantic information entropy, and is used to characterize the semantic information carrying density of the unit.

[0156] The global semantic baseline anchor used here to calculate the semantic contribution adopts the top-level global semantic center feature of the corresponding modality output in step 41, that is, the top-level global feature after recursive aggregation. It is consistent with the semantic direction of the initial baseline anchor output in step 3, and has a higher hierarchical matching degree, resulting in better semantic accuracy of anchor selection.

[0157] The feature units are sorted in descending order of their global semantic contribution value, and the top M feature units are selected as the semantic anchors for the corresponding modalities, where M is a fixed value for offline calibration. The number of anchors is kept consistent across all modalities. Finally, a set of baseline anchors for the dominant modality and a set of candidate anchors for each subordinate modality are generated, with the anchor feature dimension consistent with the basic feature dimension d.

[0158] In this embodiment, the number of anchor points M is set to 8; all anchor points are selected from the global semantic cluster center of each modality's top layer, therefore the number of anchor points M is not greater than the number of semantic clusters in the recursive top layer. .

[0159] Step 422 Global Optimal Anchor Point Matching

[0160] This sub-step establishes a one-to-one semantic correspondence between candidate anchor points of subordinate modalities and baseline anchor points of dominant modalities. It uses a globally optimal matching algorithm to replace greedy matching, avoids local semantic mismatches, and ensures global consistency of cross-modal alignment.

[0161] The specific implementation process is as follows: For each subordinate mode's candidate anchor point set and the dominant mode's baseline anchor point set, calculate the cosine similarity of the feature vectors pairwise, and generate... The two-way similarity matrix is ​​calculated using the same rules as the cosine similarity rule.

[0162] The Hungarian algorithm is employed to maximize the total global similarity, solving for the globally optimal one-to-one matching scheme and generating a set of matching anchor pairs that correspond one-to-one between master and slave modalities. During the matching process, the unique matching rule for each anchor point is strictly followed, prohibiting many-to-one and one-to-many matching relationships to ensure accurate correspondence in the global semantic dimension and avoid semantic confusion and mismatches. Specifically, the adaptation method involves inverting all similarity values ​​in the M×M bidirectional similarity matrix to obtain the corresponding cost matrix. Specifically, the cosine similarity is subtracted from 1 as the matching cost, resulting in a cost matrix with values ​​ranging from [0,1]. Smaller values ​​indicate higher anchor point similarity, aligning with the optimization objective of minimizing the total cost. Then, the standard Hungarian algorithm is called to solve the problem with the goal of minimizing the total cost. The final matching result is the one-to-one anchor pair with the highest global similarity.

[0163] Step 423 Subordinate modality unidirectional semantic calibration and cross-modal fusion

[0164] This sub-step differs from the bidirectional cross-alignment architecture of modal equivalence in existing technologies. This solution only performs unidirectional semantic calibration from subordinate modalities to dominant modalities, and there is no reverse alignment link. It eliminates the error accumulation and semantic drift problems caused by pairwise cross-alignment of multimodalities from the root, and its advantages are more significant in fusion scenarios of three or more modalities.

[0165] The specific implementation process is divided into two stages:

[0166] Subordinate modality unidirectional semantic space calibration

[0167] Based on the semantic distribution features of the matching anchor pair set, the mean vector and covariance matrix of each subordinate mode anchor pair set, as well as the mean vector and covariance matrix of the dominant mode reference anchor pair set, are calculated respectively. The affine transformation parameters of each subordinate mode relative to the dominant mode are derived through distribution alignment, including translation vectors and scaling transformation matrices.

[0168] To avoid the singularity problem of the covariance matrix caused by small sample estimation, the calculated original covariance matrix needs to be corrected by adding an identity matrix regularization term, i.e. ,in To fix the regularization coefficient, this embodiment uses a value of [value to be filled in]. I is a d-order identity matrix; the corrected covariance matrix satisfies the symmetric positive definite condition, where both the mean vector and the covariance matrix are statistically obtained based on the feature dimensions of the anchor points, with dimensions respectively. and The square root of the covariance matrix and inverse square root All solutions are obtained through Cholesky decomposition, that is, by first performing Cholesky decomposition on the covariance matrix to obtain the lower triangular matrix. Then, by solving for the inverse matrix of the lower triangular matrix, we obtain the corresponding result. .

[0169] Affine transformations are used to map the full hierarchical global features of the corresponding subordinate modalities to the semantic space of the dominant modality, completing unidirectional semantic calibration of the subordinate modalities and eliminating semantic space distribution offsets between modalities. After calibration, the feature spaces of all subordinate modalities are fully aligned with the dominant modality, providing a semantic consistency basis for direct fusion. When multiple subordinate modalities exist, independent affine transformation parameters are calculated for each subordinate modality, and semantic space calibration is performed separately for each modality; the transformation parameters of different subordinate modalities are not shared.

[0170] Multi-source feature weighted fusion

[0171] After calibration, the three types of semantic signals—the original global features of the dominant modality, the global features of all calibrated subordinate modalities, and the cross-modal common semantic features—are fused element-wise with fixed weights according to the offline calibration, generating a unified-dimensional cross-modal fused feature. The feature dimension is consistent with the basic feature dimension d. In this embodiment, the fixed weights of the three types of semantic signals are as follows: the weight of the original global features of the dominant modality is 0.5, the total weight of the global features of all calibrated subordinate modalities is 0.3, and the weight of the cross-modal common semantic features is 0.2. When there are multiple subordinate modalities, the weight of each subordinate modality is allocated according to its proportion of global semantic density, and the sum of the weights of all subordinate modalities is 0.3. The cross-modal common semantic features are generated by global mean pooling of all cross-modal common semantic units output in step 2, and the dimension is consistent with the basic feature dimension d, representing the global core semantics common to all modalities.

[0172] During the fusion process, the association mapping relationship between the matching anchor pairs and the dimensions of the fused features is preserved simultaneously, clarifying the dimensional position of the semantics corresponding to each anchor point in the fused features, and providing a semantic dimension anchoring basis for subsequent orthogonal fidelity coding. The final output is a cross-modal fused feature with anchor matching association relationships.

[0173] The rules for establishing this mapping relationship are as follows: each matching anchor point corresponds to an independent semantic entity. During the fusion process, the dimensional contribution weight distribution of the anchor point's semantics in the cross-modal fusion features is recorded. At the same time, the semantic hierarchical classification label carried by the anchor point itself (cross-modal common semantics / dominant modal independent semantics / subordinate modal independent semantics) is retained. The semantic label of the anchor point is synchronously transmitted to the corresponding encoding dimension during the fusion process, providing a classification basis for subsequent semantic hierarchical segmentation.

[0174] Step 43: Perform orthogonal semantic basis construction and fidelity mapping encoding on the cross-modal fusion features with anchor matching relationships to obtain the initial global feature encoding vector with anchor dimension mapping relationships.

[0175] This sub-step addresses the inherent flaws in existing technologies, such as semantic coupling redundancy in feature dimensions, lack of explicit semantic binding in orthogonal dimensionality reduction, and the tendency to lose core global semantics during orthogonalization. It employs a dedicated architecture based on semantically anchored orthogonal basis construction and semantic energy fidelity constraints. Each encoding dimension strictly corresponds to an independent semantic anchor point. This achieves dimensional decoupling and eliminates semantic overlap while maximizing the preservation of core global semantic information and endowing the encoding dimensions with explicit semantic interpretability. It should be noted that the orthogonal semantic basis matrix in this step is dynamically generated for each single sample. That is, for each input multimodal reference content, a unique semantic basis matrix is ​​independently calculated based on its own matching anchor pair. The semantic basis matrices of different samples are different, and there are no globally shared fixed orthogonal basis parameters. The matrix is ​​calculated in real-time during the inference phase along with the encoding process.

[0176] This sub-step takes cross-modal fusion features with anchor matching relationships as input and outputs a global feature initial encoding vector with anchor dimension mapping relationships. All operations are performed based on offline trained and fixed parameters, and there are no parameter updates during the inference phase. The specific implementation process is as follows:

[0177] Step 431: Based on the cross-modal fusion features with anchor matching relationships, generate initial vectors for each set of matching anchor pairs corresponding to independent semantic dimensions, thereby obtaining the initial semantic basis vector set;

[0178] The specific implementation process is as follows: Each generated pair of matching anchor points corresponds to an independent global semantic dimension. Based on the fusion features of each anchor point pair, an initial semantic basis vector for the corresponding dimension is generated. The initial basis vector for a single anchor point is obtained by weighting the dominant modality's baseline anchor point features and the calibrated subordinate modality's corresponding anchor point features with fixed weights. These weights are consistent with the anchor point weights in the cross-modal fusion stage. In this embodiment, the weighting weight of the dominant modality's baseline anchor point features is 0.6, and the weighting weight of the subordinate modality's corresponding anchor point features is 0.4.

[0179] All initial semantic basis vectors together constitute the initial semantic basis vector set. The total number of basis vectors is exactly the same as the total number of anchors M. The dimension of each basis vector is the same as the basic dimension d of the cross-modal fusion feature, forming a vector set of M d-dimensional initial semantic bases.

[0180] Step 432 Orthogonalization and Semantic Energy Fidelity Constraint Correction

[0181] This sub-step adds semantic energy fidelity constraints to the conventional orthogonalization, which solves the inherent defect of existing pure mathematical orthogonalization schemes that easily lead to the loss of core semantic information. Under the premise of ensuring that each encoding dimension is linearly independent and has no semantic coupling, it maximizes the preservation of the core global semantic energy of the original fusion features.

[0182] The specific implementation process is divided into two sequential stages:

[0183] Schmidt orthogonalization basic processing

[0184] The initial semantic basis vector set is subjected to standard Schmitt orthogonalization. Each basis vector is then orthogonally projected to eliminate linear dependencies between dimensions, generating pairwise linearly independent initial orthogonal semantic basis matrices. After orthogonalization, the inner product of any two basis vectors is zero, ensuring no semantic overlap or coupling interference between encoding dimensions. The initial orthogonal semantic basis matrix has dimensions of [dimensions to be filled in]. The initial semantic basis vectors are the column vectors of the matrix. After Schmitt orthogonalization, all column vectors of the matrix are pairwise orthogonal, meaning the inner product of any two columns is 0. Before Schmitt orthogonalization, all initial semantic basis vectors are sorted from high to low according to their global semantic contribution to the corresponding anchor point. Then, orthogonalization is performed sequentially according to the sorted order to ensure that the basis vectors corresponding to the core semantics retain the main semantic information first.

[0185] Semantic energy fidelity constraint correction

[0186] A semantic energy fidelity constraint is introduced to modify the initial orthogonal semantic basis matrix, thereby improving the energy retention rate of the core semantics without compromising orthogonality. The specific implementation is as follows:

[0187] Using the global semantic energy of the cross-modal fused features as a benchmark, the semantic energy retention rate corresponding to the initial orthogonal basis is calculated, which is the ratio of the total energy of the fused features after projection onto the initial orthogonal basis space to the total energy of the original features. When the retention rate is lower than a preset fixed threshold, the singular value vectors of the orthogonal basis matrix are weighted and adjusted through singular value decomposition, prioritizing the amplification of the singular value proportion corresponding to the core semantics to improve the core semantic energy retention rate. After the adjustment, the reconstructed basis matrix is ​​re-processed with lightweight Schmitt orthogonalization to restore the column orthogonal properties of the matrix until the semantic energy retention rate is not lower than the preset fixed threshold. In this embodiment, lightweight Schmitt orthogonalization means that only standard orthogonal projection is performed on the column vectors of the matrix, without changing the order of the vector magnitudes and semantic order, ensuring that the semantic correspondence of the column vectors remains unchanged after orthogonalization.

[0188] The total energy after projection is the squared L2 norm of the initial encoding vector obtained by orthogonal basis mapping of the fused features, and the initial encoding vector in step 433. The calculation results are completely consistent; the total energy of the original features is the square of the second norm of the cross-modal fused feature vector.

[0189] The specific steps for implementing singular value decomposition correction are as follows:

[0190] The first step is to initialize the orthogonal semantic basis matrix. Perform singular value decomposition to obtain , where U is Left singular orthogonal matrix, for Singular value diagonal matrix, V is Right singular orthogonal matrix;

[0191] The second step is to construct a singular value correction objective function with the semantic energy retention rate as a constraint. While keeping the U and V orthogonal matrices unchanged, the weight of the core singular values ​​is gradually increased from high to low according to the energy proportion, while the proportion of low-energy singular values ​​is simultaneously compressed.

[0192] Singular value correction follows the constraint of conservation of the sum of singular values. The adjustment rule for a single iteration is as follows: the core singular values ​​with the top 30% of energy are uniformly amplified by 1.05 times, and the remaining low-energy singular values ​​are compressed proportionally to ensure that the sum of all singular values ​​remains consistent with that before correction; the semantic energy retention rate is recalculated after each iteration until the target is met or the maximum number of iterations is reached.

[0193] The third step is based on the corrected singular value diagonal matrix. The corrected semantic basis matrix is ​​obtained by reconstruction. And recalculate the semantic energy retention rate;

[0194] The fourth step is to perform lightweight Schmitt orthogonalization on the adjusted semantic basis matrix to restore the column orthogonality of the matrix, thereby obtaining the orthogonal semantic basis matrix after this round of correction, and recalculate the semantic energy retention rate.

[0195] Repeat the iterative correction process from step 2 to step 4 until the semantic energy retention rate is not lower than the preset fixed threshold, or until the maximum number of iterations of 10 is reached, and then output the final orthogonal fidelity semantic basis matrix.

[0196] After correction, the final semantic basis matrix is ​​orthogonal and interference-free, with high fidelity to the core semantics, and its dimensions are maintained at 1. In this embodiment, the preset fixed threshold for semantic energy retention rate is set to 0.9.

[0197] Step 433 Orthogonal space mapping and normalization

[0198] This sub-step maps the original high-dimensional fused features to a low-dimensional orthogonal semantic space, completing the initial encoding of global features. At the same time, it preserves the mapping relationship between dimensions and semantic anchors, providing a semantic dimension basis for the hierarchical structured mapping in the subsequent step 5.

[0199] The specific implementation process is as follows:

[0200] Combined d-dimensional cross-modal fusion feature vector with A linear projection transformation is performed on the orthogonal semantic basis matrix to map it to the feature space corresponding to the orthogonal semantic basis, thus completing the dimensionality transformation to obtain an M-dimensional initial encoding vector. The mapping formula is as follows:

[0201] ;

[0202] in, This is a cross-modal fusion feature vector with dimension . ; It is an orthogonal semantic basis matrix with dimension . ; The initial encoded vector after mapping has a dimension of .

[0203] The initial encoded vector is subjected to L2 normalization, which normalizes the vector's magnitude to 1, eliminating the interference of vector magnitude differences on subsequent similarity calculations. The normalization formula is as follows:

[0204] ;

[0205] in, For vector 2 norm operations; This is the normalized encoding vector.

[0206] During the processing, the mapping relationship between each encoding dimension and the corresponding semantic anchor point is recorded synchronously, clarifying which encoding dimension corresponds to which set of semantic anchor points in the global semantics, and finally generating a global feature initial encoding vector with anchor point dimension mapping relationship.

[0207] In this embodiment, step 5 addresses the inherent defects of existing technologies' flattened global feature vectors, such as overlapping dimensional semantic coupling, lack of clear semantic hierarchy, poor interpretability, and inability of downstream tasks to call semantic features hierarchically. Based on the semantic hierarchy classification rules and anchor point dimension mapping relationships established in the previous step, step 5 performs three-segment semantic dimension splitting, orthogonal decoupling between segments, and structured concatenation to generate a structured global feature encoding vector with clear semantic hierarchy, no interference between segments, and adaptability to downstream calls in multiple scenarios. The input to this step is the initial global feature encoding vector with anchor point dimension mapping relationships output in step 4, and the output is the final global feature encoding vector. All operations are performed based on predefined hierarchical mapping rules and fixed parameters. There are no parameter updates during the inference phase. The specific implementation process is as follows:

[0208] Step 51: Semantic hierarchy attribution matching and dimensional segmentation

[0209] This sub-step divides the initial encoding vector into three independent semantic segments according to semantic affiliation, establishes a one-to-one correspondence between encoding dimensions and semantic levels, and realizes the hierarchical splitting of global features, which is different from the flattened dimension arrangement without differentiation in existing technologies.

[0210] The specific implementation process is as follows:

[0211] Based on the anchor point dimension mapping relationship retained in step 4, and combined with the semantic hierarchy classification rules established in step 2 and the modality master-slave roles determined in step 3, the semantic hierarchy affiliation of each dimension of the initial global feature encoding vector is determined one by one, and the division rules are as follows:

[0212] Global common semantic segment: The encoding dimension corresponding to the cross-modal common semantic anchor point, carrying the common core semantics shared by all modalities, is the most core global semantic foundation of the entire multimodal reference content.

[0213] Dominant core semantic segment: The encoding dimension corresponding to the core semantic anchor point of the dominant modality, carrying the core supplementary semantics unique to the dominant modality, and is the core supporting entity of global semantics.

[0214] Multimodal complementary semantic segments: The encoding dimension corresponding to the independent complementary semantic anchor points of each subordinate modality carries the unique and differentiated supplementary semantics of each subordinate modality, enriching the multimodal information coverage of global features.

[0215] According to the above attribution rules, all dimensions of the initial encoding vector are rearranged and segmented to generate an initial three-segment semantic dimension segmentation result. The dimensions within each segment belong to the same semantic level, and there are clear semantic boundaries between segments.

[0216] The specific judgment and classification rules are as follows:

[0217] Each dimension of the initial global feature encoding vector uniquely corresponds to a set of matching anchor pairs. The anchor pairs inherit the semantic hierarchy classification labels of the original semantic units. Therefore, the semantic hierarchy of the encoding dimension is completely consistent with the label of the corresponding anchor.

[0218] The encoding dimensions corresponding to cross-modal common semantic anchor points are uniformly classified into the global common semantic segment, and the number of dimensions is denoted as [missing information]. ;

[0219] The encoding dimensions of the independent and complementary semantic anchors corresponding to the dominant modality are uniformly classified into the dominant core semantic segment, and the number of dimensions is denoted as . ;

[0220] The encoding dimensions of each subordinate modality's independent complementary semantic anchor point are uniformly classified into a multimodal complementary semantic segment, and the number of dimensions is denoted as . ;

[0221] satisfy The number of dimensions of the three types of semantic segments is naturally determined by the distribution of semantic labels of anchor points, and all samples follow the same partitioning rule.

[0222] Step 52: Segmented orthogonal decoupling and inter-segment semantic interference elimination

[0223] This sub-step further eliminates potential semantic coupling between the three semantic segments, ensuring that the semantics of each segment are completely independent, solving the inherent problem of semantic crosstalk between existing coding dimensions, achieving complete decoupling of different semantic levels, and supporting downstream tasks to independently call a single feature segment without introducing other semantic interference.

[0224] The specific implementation process is as follows:

[0225] For the initial three-segment semantic dimension segmentation results, a progressive orthogonal projection correction scheme is adopted. Taking the segment with higher semantic priority as the benchmark, the remaining semantic segments are calibrated in sequence. The specific process is as follows:

[0226] Using the eigenspace spanned by the semantic basis submatrix of the global common semantic segment as the first benchmark, orthogonal projection correction is performed on the dominant core semantic segment to remove semantic components in the dominant core semantic segment that overlap with the global common semantic segment, ensuring that the corrected dominant core semantic segment is linearly independent of the global common semantic segment.

[0227] The operation object of orthogonal projection correction is the submatrix formed by all semantic basis vectors of the corresponding semantic segment: first, extract the semantic basis submatrix corresponding to the reference semantic segment; then, for each semantic basis vector of the semantic segment to be corrected, perform orthogonal projection onto the space spanned by the reference submatrix; then, subtract the projection component from the original basis vector to obtain the corrected basis vector; finally, recalculate the encoding value of the corresponding semantic segment based on the corrected semantic basis to ensure that the two semantic spaces are completely orthogonal.

[0228] Using the benchmark subspace formed by the corrected global common semantic segment and the dominant core semantic segment as the second reference, orthogonal projection correction is performed on the multimodal complementary semantic segment to remove the components in the multimodal complementary semantic segment that overlap with the first two types of semantics, ensuring that the corrected multimodal complementary semantic segment is linearly independent of the first two semantic segments.

[0229] Intra-segment orthogonal normalization is performed synchronously on the dimensions within each semantic segment to maintain the linear independence of the dimensions within the segment and avoid semantic redundancy within the segment.

[0230] Intra-segment orthogonalization employs the standard Schmidt orthogonalization method consistent with step 432, performing orthogonalization on all semantic basis vectors within a single semantic segment. After processing, the inner product of basis vectors in any two dimensions within the segment is 0, ensuring no semantic coupling within the segment dimensions. Before intra-segment orthogonalization, all basis vectors within the semantic segment are sorted from high to low according to their global semantic contribution to the corresponding anchor point, and then Schmidt orthogonalization is performed, maintaining consistency with the processing rules in step 432.

[0231] The formula for calculating the correction of a single-segment orthogonal projection is:

[0232] ;

[0233] in, The semantic segment feature vector to be corrected; The feature vector of the baseline semantic segment; The projection component of the vector to be corrected onto the reference vector is calculated using the following formula:

[0234] ;

[0235] After processing, an orthogonal and independent semantic segment encoding set is obtained. The inner product of each pair of the three semantic segments is 0, and there is no semantic coupling or interference between segments. Adjustment of any semantic segment will not affect the semantic expression of other semantic segments. The orthogonal correction process only adjusts the feature space distribution of the dimensions, without changing the order of the dimensions or the semantic hierarchy. The semantic anchor point mapping relationship corresponding to each encoding dimension remains unchanged, and the semantic hierarchy label of the original anchor point can still be directly mapped to through the dimension index after correction.

[0236] Step 53: Fixed-order splicing and normalization

[0237] This sub-step involves concatenating three independent semantic segments into a complete structured encoding vector according to fixed rules, while unifying the numerical range of the encoding to ensure the standardization of the encoding and cross-sample comparability.

[0238] The specific implementation process is as follows:

[0239] Fixed-order concatenation: Following a preset fixed order of "global common semantic segment - dominant core semantic segment - multimodal complementary semantic segment," the three orthogonal and independent semantic segment codes are concatenated end-to-end along the dimensional direction to form a complete one-dimensional global feature encoding vector. The concatenation order is fixed throughout, and the dimensional semantics of the encoding vectors of all input samples are completely aligned, providing a basis for direct cross-sample comparison. The total dimension of the concatenated global feature encoding vector remains M, which is completely consistent with the dimension of the initial global feature encoding vector output in step 4. Only the dimensional order is rearranged and segmented according to semantic hierarchy, without increasing or decreasing the number of dimensions.

[0240] Global normalization and regularization: L2 normalization is performed on the concatenated global feature encoding vector to normalize the vector's magnitude to 1, eliminating the interference of vector magnitude differences on downstream similarity calculation, feature matching and other tasks, and ensuring the standardization of the encoding vector.

[0241] The final generated structured global feature encoding vector is the final output of the entire encoding method. This encoding vector has both global semantic integrity and hierarchical structure characteristics. Downstream tasks can directly extract the feature vector of the corresponding semantic segment for independent use according to their own needs, without the need for additional dimension filtering, which greatly improves the scenario adaptability and engineering usability of the encoding.

[0242] This application also provides a global feature encoding device for reference content based on multimodal fusion, which includes:

[0243] An initial semantic unit feature set acquisition module is used to acquire the initial semantic unit feature set corresponding to the multimodal reference content to be encoded.

[0244] A purified multimodal semantic unit set acquisition module is used to perform cross-modal semantic redundancy resolution and hierarchical division on the initial semantic unit feature set corresponding to the multimodal reference content to be encoded, thereby obtaining a purified multimodal semantic unit set with semantic hierarchical classification labels.

[0245] A global semantic benchmark anchoring configuration acquisition module is used to perform global master-slave modality determination and benchmark anchoring on a set of purified multimodal semantic units with semantic hierarchical classification labels, thereby acquiring the global semantic benchmark anchoring configuration.

[0246] A multimodal semantic anchoring recursive fusion coding network is used to perform global feature initial encoding on the purified multimodal semantic unit set with semantic hierarchical classification labels and the global semantic benchmark anchoring configuration, thereby obtaining a global feature initial encoding vector with anchor point dimension mapping relationship.

[0247] The global feature encoding vector acquisition module is used to perform global semantic hierarchical structured mapping and encoding on the initial global feature encoding vector with anchor dimension mapping relationship, thereby obtaining the global feature encoding vector.

[0248] In this embodiment, the specific structure of the multimodal semantic anchoring recursive fusion coding network is as follows:

[0249] The multimodal semantic anchoring recursive fusion coding network is used to complete the core operations of single-modal hierarchical semantic depth modeling, master-slave benchmark unidirectional calibration fusion, and semantic anchoring orthogonal fidelity coding, solving three types of technical problems in existing technologies: long sequence global semantic decay, multimodal alignment error accumulation, and semantic coupling of fusion feature dimensions.

[0250] The network takes as input a set of purified multimodal semantic units with semantic hierarchical classification labels and a global semantic benchmark anchoring configuration containing dominant modality identifiers and global semantic benchmark anchor vectors. The output is a global feature initial encoding vector with anchor dimension mapping relationship. All trainable parameters of the network are optimized through offline end-to-end joint training. All parameters are permanently fixed in the inference encoding stage, and only forward operation is performed without real-time gradient updates.

[0251] The network adopts a three-module serial architecture, consisting of a single-modal recursive aggregation sub-network, a master-slave anchoring calibration fusion sub-network, and a semantic anchoring orthogonal coding sub-network, from front to back. The data link is a unidirectional closed loop without branches.

[0252] The unimodal recursive aggregation subnetwork is used to independently perform hierarchical global semantic modeling for each modality. It overcomes the bottleneck of long-range semantic dependency decay in existing unidirectional hierarchical aggregation through a bidirectional recursive closed-loop architecture. The input is the set of effective semantic units for each modality, and the output is the hierarchical global feature set for each modality. This subnetwork, from front to back, includes a fixed-size clustering layer, an L-layer bidirectional recursive stacking layer, and a hierarchical dimension normalization layer. Each modality corresponds to an independent unimodal recursive aggregation subnetwork. While the network structures are completely identical, the trainable parameters are independent, each adapting to the feature distribution characteristics of different modalities.

[0253] The fixed-size clustering layer serves as the input adaptation layer. It incorporates a K-means clustering operator and a mean pooling operator, using cosine similarity as the clustering distance metric and minimizing intra-cluster distance while maximizing inter-cluster distance as the convergence objective. This layer divides the dynamically varying length of the input single-modal semantic unit set into a fixed number of K semantic clusters, where K is a fixed hyperparameter calibrated offline. Subsequently, it performs dimension-wise mean pooling on all unit features within each cluster to calculate the cluster's central feature vector. The output is an initial semantic cluster feature set with dimension K×d, where d is the uniform basic feature dimension across the entire chain. This layer has no trainable parameters; it ensures consistent output dimensions for inputs of different lengths solely through fixed clustering rules, adapting to subsequent standardized recursive operations.

[0254] Before clustering, L2 normalization is performed on all semantic unit features; the clustering distance uses 1-cosine similarity, with a value range of [0,2]. The smaller the value, the higher the feature similarity, which is aligned with the minimum distance target of K-means.

[0255] It should be noted that this clustering is a single-sample dynamic clustering process, that is, K-means clustering is performed independently for the semantic units of each input sample during the inference stage. There is no globally shared fixed cluster center. The clustering process is completed only based on the feature distribution of the current sample, and the number of clusters for all samples is uniformly K.

[0256] After clustering is completed, all semantic clusters are sorted in descending order of the number of semantic units contained in the cluster. Clusters with the same number of units are sorted in descending order of the L2 norm of the cluster center feature. This ensures that the cluster output order of all samples follows a unified rule, and the semantics of the input positions of subsequent recursive stacked layers are consistent.

[0257] The L-layer bidirectional recursive stacked layer is the core original structure of this sub-network. It has a total of L recursive layers, where L is a fixed hyperparameter. The recursive layers correspond to the semantic coverage range from fine to coarse, and each layer contains a forward aggregation branch and a backward correction branch, which perform a closed-loop operation of "forward generation of high-level features - backward calibration of current layer features".

[0258] The forward aggregation branch consists of a cluster-level self-attention layer and an inter-cluster global interaction attention layer. The cluster-level self-attention layer models the internal semantic relationships between units within a single cluster, enhancing the semantic cohesion of cluster features. Its input is the original set of unit features corresponding to the k-th cluster at the current level, with a dimension of [missing information]. , This represents the number of units in the cluster.

[0259] This layer contains three independent trainable linear projection layers, a scaled dot product attention unit, a dimension mapping linear layer, and a global mean pooling unit. During computation, the input features are first passed through three independent linear projection layers—a query projection layer, a key projection layer, and a value projection layer—to generate layers with all dimensions... The query matrix Q, key matrix K, and value matrix V, With the attention dimension having fixed hyperparameters, the projection operation satisfies , , ,in For trainable weight matrix, For trainable biases; the parameters of the dimension-mapped linear layer belong to the trainable parameters of the intra-cluster self-attention layer, participate in training optimization synchronously with the core network, and are included in the intra-cluster self-attention layer linear projection weights of the total trainable parameters.

[0260] Then, the scaled dot product attention operation is performed, calculated using the following formula: In the formula: It is the set of intra-cluster features after semantic refinement; For row-by-row normalization activation function; This is the transpose of the key matrix K; This is a scaling factor used to prevent the gradient of the Softmax function from becoming too large due to excessively large dot product results.

[0261] The output dimension of the operation is The refined cluster feature set is then mapped back to d dimensions via a linear layer, and finally obtained by global mean pooling to a dimension of d. The single-cluster centrality feature. After performing the above operations on all clusters, the summation yields a dimension of... The set of all cluster center features at the current level. This represents the total number of clusters at the current level.

[0262] The inter-cluster global interaction attention layer is used to model the global semantic associations of all clusters, aggregating and generating higher-level cluster features. The input is the set of cluster center features at the current level, with dimensions of [missing information]. This layer contains three sets of globally trainable linear projection layers, cross-cluster scaling dot product attention units, and hierarchical aggregation units. During computation, the input features are first processed through three independent global linear projections, generating layers with all dimensions being [missing information]. Global query matrix Global key matrix Global value matrix Then, cross-cluster attention operation is performed, calculated using the following formula: ;

[0263] in, To enhance cluster features by incorporating global context; This is the transpose of the global key matrix.

[0264] After obtaining the enhanced cluster features fused with the global context, the data is compressed according to a preset offline compression ratio. The low-level clusters aggregate into A high-level cluster, satisfying Each high-level cluster feature is a weighted sum of the corresponding low-level clusters according to the global attention weights, with the output dimension being... The set of global cluster features at the next higher level.

[0265] The reverse correction branch is a unique link not found in existing unidirectional aggregation architectures. It consists of a reverse linear projection layer and an element-wise weighted gating layer in sequence. The reverse linear projection layer is a single set of trainable linear projection layers, with the input being the feature set of the higher-level clusters from the previous level, and its dimension is [dimensionality missing]. During computation, the sequence dimension of the features is reduced from [previous dimension] by linear projection. Upsampling to The feature dimension remains constant at d, and the calculation formula is as follows: ,in For trainable weights and biases, the output dimension is... Global semantic constraint features.

[0266] The element-wise weighted gating layer incorporates a Sigmoid activation unit and an element-wise multiplication operator. The input consists of the original cluster features of the current layer and global semantic constraint features. During computation, the constraint features are first activated element-wise using Sigmoid activation to normalize the values ​​to [0,1], generating a weight mask. Then, the weight mask is multiplied element-wise with the original cluster features of the current layer. The calculation formula is as follows: ;

[0267] in, This is the current level cluster feature set after context calibration; This is the set of original cluster features at the current level; Use the Sigmoid activation function; This is an element-wise multiplication operation (Hadamard product).

[0268] The operation outputs a set of cluster features at the current level after context calibration, enabling low-level fine-grained features to be embedded into the global semantic context and correcting local semantic biases.

[0269] The layer-by-layer operation rule of this stacked layer is as follows: starting from the initial cluster layer of layer 1, the high-level features of layer 2 are generated first through the forward aggregation branch, and then the features of layer 1 are calibrated through the reverse correction branch; then, with the features of layer 2 as input, the above closed-loop operation is repeated until all L layers of processing are completed, and finally the output is a multi-level recursive cluster feature set containing L semantic levels.

[0270] The hierarchical dimension normalization layer configures one set of independent trainable linear projection layers for each level. It performs linear projection on the features of each level of each modality, unifies the dimension of the features of the same level of different modalities to a fixed value, keeps the feature semantic space unchanged, and outputs a set of hierarchical global features of all modalities with completely consistent dimensions of the same level. The top-level feature is the global semantic center feature of the corresponding modality, which is used for subsequent cross-modal fusion processing.

[0271] The master-slave anchoring calibration fusion subnetwork uses the dominant modality as the sole global semantic benchmark to complete unidirectional anchoring calibration and cross-modal feature fusion of subordinate modalities. This overcomes the error accumulation bottleneck of existing bidirectional cross-alignment between peer modalities. The input consists of hierarchical global feature sets for each modality and global semantic benchmark anchoring configurations. The output is cross-modal fusion features with anchor point dimension mapping relationships. From front to back, this subnetwork comprises a high semantic contribution anchor point selection layer, a globally optimal anchor point matching layer, and a unidirectional semantic calibration fusion layer. The master-slave anchoring calibration fusion subnetwork is a parameter-free, fixed-operation network. All layers do not contain trainable parameters. Anchor point selection, matching, affine calibration, and fusion processes are all completed based on preset rules and input feature statistics, without participating in parameter optimization.

[0272] The high semantic contribution anchor selection layer incorporates a semantic contribution calculation unit and a Top-M selection unit. Combined with the global semantic baseline anchor, it calculates the global semantic contribution of each feature unit by unit. The calculation formula is as follows: ;

[0273] in, The global semantic contribution of the j-th feature unit of the i-th modality; The associated attention weight is obtained by normalizing the cosine similarity between the unit and the corresponding modality's global semantic benchmark anchor using Softmax. This is the semantic information entropy of the unit.

[0274] The specific calculation method for the associative attention weights is as follows: First, calculate the cosine similarity between the j-th feature unit of the i-th modality and the global semantic baseline anchor vector of the corresponding modality. Here, the feature unit for calculating semantic contribution is the center of the top-level global semantic cluster of the corresponding modality. Therefore, the softmax normalization of the associative attention weights is performed within the range of all top-level semantic cluster centers. The calculation formula is as follows:

[0275] ;

[0276] in, Let be the feature vector of the j-th top-level semantic cluster center of the i-th modality. The top-level global semantic center feature of the i-th modality. denoted as the total number of semantic clusters at level L.

[0277] The formula for calculating semantic information entropy is:

[0278] ;

[0279] in, The probability value of the k-th dimension of the feature vector after Softmax normalization.

[0280] After calculation, the features are sorted in descending order of contribution, and the top M feature units are selected as semantic anchors, where M is a fixed hyperparameter for offline calibration. The number of anchors for all modalities is consistent. The final output includes the set of baseline anchors for the dominant modality and the set of candidate anchors for each subordinate modality, both with dimensionality 1. This layer selects only high semantic density core units as alignment benchmarks, naturally filtering out low-value noise units to avoid interfering with global fusion stability, and has no trainable parameters.

[0281] The globally optimal anchor point matching layer incorporates a cosine similarity calculation unit and a Hungarian algorithm solution unit. It first calculates the cosine similarity of master and slave modal anchor points pairwise, generating... A bidirectional similarity matrix is ​​used, and the optimization objective is to maximize the total global matching similarity. The Hungarian algorithm is employed to solve for the set of matching anchor point pairs that correspond one-to-one between master and slave modalities. The matching process strictly follows the single anchor point unique matching rule, prohibiting many-to-one and one-to-many matching to ensure accurate correspondence in the global semantic dimension and avoid local mismatches. This layer has no trainable parameters and follows fixed algorithm rules.

[0282] The unidirectional semantic calibration fusion layer is the core and original structure of this sub-network. It only performs unidirectional semantic calibration from subordinate modes to the dominant mode, without reverse alignment links, thus eliminating the accumulation of errors in multimodal cross-alignment at the source. This layer incorporates an affine transformation solving unit, a unidirectional calibration unit, and a fixed-weight fusion unit. During computation, it first calculates the mean vector of the anchor point set of each subordinate mode. Covariance matrix The mean vector of the dominant modal anchor set Covariance matrix The scaling transformation matrix is ​​derived. Translation vector Subsequently, an affine transformation is used to map all hierarchical features of the subordinate modality to the semantic space of the dominant modality, as shown in the formula: One-way semantic calibration is completed; finally, for the three types of signals—the original features of the dominant modality, the calibrated features of the subordinate modality, and the cross-modal common semantic features—element-wise weighted summation is performed according to the fixed weights of offline calibration. The sum of the three weights is 1, generating a unified dimension. The cross-modal fusion features simultaneously retain the correlation mapping relationship between the matching anchor point and the fusion dimension.

[0283] The semantic anchored orthogonal coding subnetwork is used to construct orthogonal coding dimensions based on semantic anchors, completing the initial global feature encoding. This overcomes the bottleneck of existing pure statistical orthogonal dimensionality reduction methods, which lack semantic binding and are prone to losing core semantics. The input is cross-modal fusion features with anchor dimension mapping relationships, and the output is a global feature initial encoding vector with anchor dimension mapping relationships. From front to back, this subnetwork consists of a semantic anchor base initialization layer, an orthogonal fidelity correction layer, and an orthogonal space mapping layer.

[0284] The semantic anchoring basis initialization layer incorporates a weighted summation operator. Each pair of matched anchor points corresponds to an independent global semantic dimension. The initial semantic basis vectors for each dimension are generated by weighting the dominant modality's baseline anchor point with the calibrated subordinate modality's corresponding anchor point using a fixed weight. A total of M d-dimensional initial semantic basis vectors are generated, forming a structure with dimension M. The initial semantic basis matrix is ​​used to achieve a one-to-one binding between the encoding dimension and semantic entities, which differs from existing non-semantic orthogonal bases based on statistical properties. This layer has no trainable parameters and uses fixed weighting rules.

[0285] The orthogonal-fidelity correction layer is the core and original structure of this sub-network, consisting of a Schmidt orthogonalization unit, a semantic energy calculation unit, and a singular value decomposition correction unit, arranged sequentially. During computation, the initial semantic basis matrix is ​​first subjected to standard Schmidt orthogonalization iteration to generate pairwise linearly independent initial orthogonal basis matrices, ensuring that the inner product of any two basis vectors is zero, thus eliminating semantic coupling between dimensions. Subsequently, the ratio of the total energy of the fused features projected onto the orthogonal basis space to the total energy of the original features is calculated, i.e., the semantic energy retention rate, using the following formula:

[0286] ;

[0287] in, Semantic energy retention rate; Let L2 be the norm of the vector; The projection features of the fused features onto the orthogonal basis space; This represents the original cross-modal fusion feature.

[0288] When the retention rate is lower than a preset fixed threshold, singular value decomposition is performed on the orthogonal basis matrix to adjust the singular value distribution. Then, lightweight Schmitt orthogonalization is used to restore the column orthogonality of the matrix, finally obtaining a semantically high-fidelity orthogonal basis matrix. This layer has no trainable parameters and follows fixed mathematical operation rules.

[0289] The orthogonal spatial mapping layer incorporates a linear projection operator and an L2 normalization unit, first combining d-dimensional fusion features with... A linear transformation is performed on the 3D orthogonal semantic basis matrix, as shown in the formula: Generate an M-dimensional initial encoding vector; then perform modulus normalization on the initial encoding vector, calculated using the following formula:

[0290] ;

[0291] in, This is the normalized initial encoding vector of global features; The L2 norm of the initial encoding vector.

[0292] The operation synchronously preserves the one-to-one mapping between the encoded dimension and the semantic anchor, and the final output dimension is The initial global feature encoding vector with anchored dimensional mapping relationship.

[0293] All trainable parameters in this network include only the linear projection weights of the intra-cluster / inter-cluster self-attention layers, the weights of the inverse linear projection layers, and the weights of the hierarchical dimension normalization layers. End-to-end joint optimization is employed during training, and the loss function consists of a weighted average of the multimodal alignment loss, semantic reconstruction loss, and orthogonal constraint loss. All parameters are permanently fixed after training convergence, and no parameter updates occur during the inference phase.

[0294] In this embodiment, the multimodal semantic anchoring recursive fusion coding network adopts an offline end-to-end joint training method to complete the optimization of all trainable parameters. All trainable parameters are iteratively updated based on the multimodal paired sample set during the training phase. After training convergence, the parameters are permanently fixed. During the inference phase, only forward encoding operations are performed without any parameter adjustments.

[0295] During the training phase, all parameters of the basic feature extraction networks (Transformer encoder, visual Transformer encoder, temporal convolutional encoder) in step 1 are frozen. Only the trainable parameters within the multimodal semantic anchoring recursive fusion encoding network are optimized, including the linear projection weights of the intra-cluster self-attention layer, the linear projection weights of the inter-cluster global interaction attention layer, the weights of the back linear projection layer, and the weights of the hierarchical dimension normalization layer. If further adaptation to specific domain data is required, the top two layers of parameters of the basic feature extraction network can be opened for joint fine-tuning.

[0296] During the fine-tuning phase, the fixed-size clustering layer still only performs forward clustering operations and does not participate in gradient backpropagation. Before each training iteration, clustering is re-executed based on the semantic unit features output by the current base network to generate initial semantic clusters, which are then input into subsequent recursive stacked layers to participate in parameter updates. The clustering results are only used as intermediate feature indices and do not propagate gradients.

[0297] The training objective is to simultaneously improve three core performance aspects: single-modal global semantic representation capability, cross-modal semantic alignment accuracy, and orthogonal coding semantic fidelity. The overall training process includes four core steps: training sample preprocessing, forward propagation encoding, multi-dimensional joint loss calculation, and backpropagation parameter optimization. An adaptive gradient descent optimization algorithm is used to complete the iterative optimization.

[0298] The training sample set is a multimodal pairing reference content sample set. Each sample contains semantic pairing content of at least two modalities, covering various combinations such as image-text pairing, audio-text pairing, and image-text-audio multimodal pairing. All samples have undergone preprocessing operations such as noise reduction and standardization cropping.

[0299] During training, basic feature extraction is first performed on the original content of each modality for each sample to obtain an initial semantic unit feature set with a unified dimension. The extraction rules are completely consistent with the pre-processing rules in the inference stage. This initial semantic unit feature set serves as the input data for network training. The sample set is divided into a training set and a validation set in a fixed ratio. The training set is used for parameter iterative updates, and the validation set is used for training effect verification and early stopping triggering. In this embodiment, the ratio of the number of samples in the training set to the validation set is 8:2.

[0300] The forward propagation process during training is identical to that during inference. Initial semantic unit features from each modality of a single sample are input into the network, sequentially processed by a single-modality recursive aggregation sub-network to complete multi-level bidirectional recursive aggregation, outputting a hierarchical global feature set for each modality. Then, a master-slave anchoring calibration fusion sub-network completes anchor selection, global matching, and unidirectional calibration fusion, outputting cross-modality fused features with anchor mapping relationships. Finally, a semantic anchoring orthogonal encoding sub-network generates the initial global feature encoding vector. The forward propagation process synchronously outputs three types of intermediate results: a set of matched anchor pairs, an orthogonal semantic basis matrix, and top-level global semantic features, used for subsequent calculation of multi-dimensional joint loss.

[0301] Training employs a multi-dimensional joint loss function as the optimization objective. The total loss is composed of a weighted sum of three parts: multimodal alignment loss, semantic reconstruction loss, and orthogonal constraint loss. These correspond to the performance constraints of the three core sub-modules of the network, and the specific calculation method for each loss term is as follows:

[0302] Multimodal alignment loss is used to constrain the semantic alignment accuracy of anchor points between master and slave modalities, promoting semantic space consistency between the calibrated features of the slave modality and the baseline features of the master modality. This ensures the accuracy of cross-modal unidirectional calibration from the training level. The calculation formula is as follows:

[0303] ;in, For multimodal alignment loss; This represents the total number of semantic anchors. This is the function for calculating cosine similarity. The dominant mode Feature vectors of each reference anchor point; For subordinate mode number The feature vectors of candidate anchor points after one-way calibration. The smaller this loss value, the higher the semantic alignment accuracy of the master-slave modality anchor points.

[0304] When there are multiple subordinate modalities, the total multimodal alignment loss is the arithmetic mean of the alignment losses of all subordinate modalities; similarly, the semantic reconstruction loss is the arithmetic mean of the reconstruction losses of all modalities, and then substituted into the total loss formula to participate in joint optimization.

[0305] Semantic reconstruction loss is used to constrain the semantic integrity of the unimodal recursive aggregation process, avoiding the loss of fine-grained semantic information at the lower levels during bidirectional recursive operations and ensuring the coverage of the original semantics by the global features. A temporary reconstruction linear supervision layer is added during the training phase. Based on the top-level global semantic features, it reconstructs the initial semantic unit features at the lower levels through reverse mapping. The output of the reconstruction linear supervision layer is arranged in a fixed order according to the original semantic units at the lower levels, and the MSE loss is calculated positionally corresponding to each position of the original semantic unit features. Clustering is only used to generate the input clusters of the recursive network and does not change the order index of the original semantic units. The information retention capability of the recursive aggregation is constrained by the reconstruction error, and the calculation formula is as follows:

[0306] ;

[0307] in, For semantic reconstruction loss; This represents the total number of single-modal semantic units; For the first Feature vectors of each original semantic unit; The reconstructed feature vector is obtained by mapping the top-level global features through a reconstructed linear layer. represents the L2 norm of the vector. The smaller this loss value, the more complete the semantic information of the recursive aggregation process is preserved. The reconstructed linear layer is only used for supervision constraints during the training phase; the structure of this layer is not retained during the inference phase.

[0308] The reconstructed linear supervision layer uses a fully connected linear transformation to achieve cross-dimensional mapping. The specific operation process is as follows: first, the dimension is... The top-level global cluster features are flattened by rows with dimensions of A one-dimensional vector, after a linear transformation of the trainable weight matrix and the bias, is reshaped into... The two-dimensional feature matrix is ​​used to obtain the reconstructed underlying semantic unit features; this layer only participates in the loss calculation and gradient backpropagation during the training phase, and is completely removed during the inference phase, and does not participate in the encoding operation.

[0309] Each modality corresponds to an independent reconstruction linear supervision layer, which is adapted to the number of underlying semantic units N of the corresponding modality. The reconstruction layer parameters of different modalities are independent of each other and are only used for the reconstruction loss calculation of the corresponding modality.

[0310] The orthogonality constraint loss is used to constrain the dimensional independence and semantic fidelity of orthogonal semantic bases. It consists of two parts: an orthogonality penalty term and a semantic energy fidelity term. This ensures that the encoding dimensions are pairwise linearly independent and semantically uncoupled, while also preventing the loss of core global semantics during the orthogonalization process. The calculation formula is divided into two parts:

[0311] The orthogonality penalty term is used to constrain the orthogonality of the semantic basis matrices, and its calculation formula is as follows:

[0312] ;

[0313] in, This is an orthogonality penalty term; It is an orthogonal semantic basis matrix; This is the transpose of the semantic basis matrix; for An identity matrix of order 1; Let be the Frobenius norm of the matrix. The smaller this loss value, the better the orthogonality of the semantic basis and the lower the coupling of the encoding dimensions.

[0314] Both terms of the orthogonal constraint loss are calculated for the final orthogonal semantic basis matrix after semantic energy fidelity correction. They are used to constrain the orthogonality and semantic fidelity of the final output encoding basis, directly aligning with the final encoding output of the inference stage.

[0315] The semantic energy fidelity term is used to constrain the semantic energy retention rate before and after orthogonal mapping, and its calculation formula is as follows:

[0316] ;

[0317] in, For semantic energy fidelity; The preset minimum semantic energy retention rate threshold, Consistent with the semantic energy retention rate threshold in the reasoning phase, the value is 0.9; The semantic energy retention rate is calculated using the same rules as in formula (6). When the actual retention rate is higher than the preset threshold, the loss for this term is 0; when the retention rate is lower than the threshold, a penalty is applied to drive the network to optimize the orthogonal basis distribution to improve semantic retention capability.

[0318] The total loss is the weighted sum of all loss items, calculated using the following formula:

[0319] ;

[0320] in, Total training loss; , , , These are preset loss weight hyperparameters, all fixed positive values ​​calibrated offline, used to balance the weight proportions of each optimization objective. In this embodiment, the values ​​of each loss weight are as follows: =0.4, =0.3, =0.2, =0.5.

[0321] During batch training, the final optimized total loss is the arithmetic mean of the losses of all individual samples within the batch, and gradient backpropagation and parameter updates are performed based on this average.

[0322] After forward propagation and total loss calculation, the gradient of the total loss with respect to all trainable parameters is calculated backward along the network links using the backpropagation algorithm. The AdamW adaptive optimization algorithm is then used to iteratively update all trainable parameters of the network. These trainable parameters include the linear projection weights of the intra-cluster self-attention layer, the global linear projection weights of the inter-cluster global interactive attention layer, the weights of the backward linear projection layer, and the weights of the hierarchical dimension normalization layer. Training employs a batch iterative approach, processing a fixed number of samples per batch. A cosine annealing learning rate decay strategy is used to adjust the training step size, improving the stability and convergence of the training process. In this embodiment, the training batch size is set to 32, and the initial learning rate is set to... The maximum number of iterations is set to 100 rounds, the early stopping threshold is set when the total loss of the validation set does not decrease for 10 consecutive rounds, and the cosine annealing period is set to 20 rounds.

[0323] During training, the model performance is verified using a validation set at fixed intervals. When the total loss on the validation set no longer decreases after a preset number of iterations, an early stopping mechanism is triggered, and the optimal model parameters on the current validation set are saved. If the preset maximum number of iterations is reached, training is terminated directly. After training, the temporarily added reconstructed linear supervision layer is removed, retaining only the complete structure of the core encoding network. All network parameters are permanently fixed and directly used for multimodal global feature encoding operations during the inference phase.

[0324] This training method corresponds one-to-one with the network's unique three-layer architecture. It simultaneously optimizes three core capabilities—single-modal representation, cross-modal alignment, and orthogonal encoding—through multi-dimensional joint loss, avoiding the error accumulation problem caused by phased training and ensuring that the overall network performance reaches the optimal level.

[0325] In this application, an initial semantic unit feature set corresponding to the multimodal reference content to be encoded is obtained; cross-modal semantic redundancy resolution and hierarchical division are performed on the initial semantic unit feature set corresponding to the multimodal reference content to be encoded, thereby obtaining a purified multimodal semantic unit set with semantic hierarchical classification labels; global master-slave modality determination and benchmark anchoring are performed on the purified multimodal semantic unit set with semantic hierarchical classification labels, thereby obtaining a global semantic benchmark anchoring configuration; global initial feature encoding is performed on the purified multimodal semantic unit set with semantic hierarchical classification labels and the global semantic benchmark anchoring configuration through multimodal semantic anchoring recursive fusion, thereby obtaining a global initial feature encoding vector with anchor point dimension mapping relationship; global semantic hierarchical structured mapping encoding is performed on the global initial feature encoding vector with anchor point dimension mapping relationship, thereby obtaining a global feature encoding vector.

[0326] As can be seen, this application sets up cross-modal semantic redundancy resolution and hierarchical division steps in the pre-encoding stage. Considering that there are common cross-modal repetitive semantics in multimodal reference content, if repetitive semantics directly participate in encoding, it is easy to cause semantic bias and dimensional redundancy in the fusion features. Therefore, this application removes repetitive and redundant semantic units in advance and completes semantic hierarchical classification, purifying the encoding semantic base from the input source. This not only avoids the fusion weight bias caused by repetitive semantics, but also reduces the amount of invalid computation in the subsequent encoding stage.

[0327] Single-modal hierarchical semantic modeling is completed through a recursive closed-loop aggregation mechanism that combines forward global aggregation with reverse context correction. Considering the defects of existing unidirectional hierarchical aggregation architecture, such as long-sequence long-distance semantic dependency decay and hierarchical semantic association breakage, this application uses high-level global semantic reverse calibration of low-level fine-grained features to break the information transmission bottleneck of unidirectional aggregation and fully preserve the full-link semantic association from fine-grained semantic units to global core topics.

[0328] Cross-modal feature fusion is achieved through a one-way calibration fusion architecture based on global master-slave modality determination and benchmark anchoring. Considering that existing peer-to-peer bidirectional cross-alignment schemes lack a unified semantic benchmark and are prone to error accumulation and semantic drift, this application uses the dominant modality with the highest semantic density as the sole benchmark. All subordinate modalities only perform one-way anchoring calibration, thus avoiding the error accumulation problem of multimodal cross-alignment from the root of the architecture. Combined with the selection of anchor points with high semantic contribution to naturally filter out noise units, the global semantic consistency and stability of the fused features are significantly improved.

[0329] The final structured global feature code is generated by semantic anchoring orthogonal fidelity coding and a three-segment structured mapping mechanism. Considering the shortcomings of existing flat coding dimensions such as semantic coupling, lack of clear semantic hierarchy, and insufficient downstream adaptation flexibility, this application constructs an orthogonal semantic base based on semantic anchors to complete fidelity coding, and divides orthogonal independent semantic segments according to semantic affiliation, thus completely eliminating semantic overlap between feature dimensions.

[0330] The global feature encoding generated on this basis not only has complete global semantic representation capabilities and high cross-modal semantic consistency, but also has clear semantic hierarchy division and interpretability. Downstream tasks can call the corresponding semantic features in segments according to their own needs, which greatly improves the scenario adaptability and engineering implementation usability of the global feature encoding.

[0331] Although the present invention has been described in detail above with general description and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention are within the scope of protection claimed by the present invention.

Claims

1. A global feature encoding method for reference content based on multimodal fusion, characterized in that, The reference content global feature encoding method based on multimodal fusion includes: Step 1: Obtain the initial semantic unit feature set corresponding to the multimodal reference content to be encoded; Step 2: Perform cross-modal semantic redundancy resolution and hierarchical division on the initial semantic unit feature set corresponding to the multimodal reference content to be encoded, thereby obtaining a purified multimodal semantic unit set with semantic hierarchical classification labels; Step 3: Perform global master-slave modality determination and baseline anchoring on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the global semantic baseline anchoring configuration; Step 4: Perform global feature initial encoding by recursively fusing multimodal semantic units with semantic hierarchical classification labels and global semantic benchmark anchoring configuration to obtain a global feature initial encoding vector with anchor point dimension mapping relationship. Step 5: Perform global semantic hierarchical structured mapping encoding on the initial global feature encoding vector with anchor dimension mapping relationship to obtain the global feature encoding vector.

2. The global feature encoding method for reference content based on multimodal fusion as described in claim 1, characterized in that, Step 2 includes: Step 21: Perform cross-modal unit pairwise semantic similarity calculation and redundant grouping on the initial semantic unit feature set to obtain the initial classification result containing cross-modal redundant semantic groups and independent semantic units; Step 22: Perform core semantic unit screening and duplicate redundant unit removal on the cross-modal redundant semantic group to obtain a set of effective semantic units after deduplication; Step 23: Perform semantic hierarchical attribute mapping and classification labeling on the deduplicated effective semantic unit set to obtain a purified multimodal semantic unit set with semantic hierarchical classification labels.

3. The global feature encoding method for reference content based on multimodal fusion as described in claim 2, characterized in that, Step 3 includes: Step 31: Perform modal-by-modal global semantic density statistical calculation on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the set of global semantic density parameters for each modality; Step 32: Sort the global semantic density parameter sets of each modality by semantic carrying capacity and classify modal roles to obtain the modal master-slave role classification results including the dominant modality and subordinate modality; Step 33: Based on the modal master-slave role division results, extract and encapsulate the global semantic benchmark anchor of the dominant modality to obtain the global semantic benchmark anchor configuration.

4. The global feature encoding method for reference content based on multimodal fusion as described in claim 3, characterized in that, Step 4 includes: Step 41: Perform initial semantic clustering and recursive global context aggregation on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the global feature set of each modality hierarchy; Step 42: Combine the global feature set of each modality hierarchy with the global semantic benchmark anchoring configuration to perform benchmark anchor point matching and cross-modal unidirectional calibration fusion, thereby obtaining cross-modal fusion features with anchor point matching correlation; Step 43: Perform orthogonal semantic basis construction and fidelity mapping encoding on the cross-modal fusion features with anchor matching relationships to obtain the initial global feature encoding vector with anchor dimension mapping relationships.

5. The global feature encoding method for reference content based on multimodal fusion as described in claim 4, characterized in that, Step 5 includes: Step 51: Perform semantic hierarchy attribution matching and dimension segmentation on the global feature initial encoding vector with anchor dimension mapping relationship to obtain a three-segment semantic dimension segmentation result; Step 52: Perform segmented orthogonal decoupling and inter-segment semantic interference elimination on the three-segment semantic dimension segmentation results to obtain an orthogonal and independent semantic segmentation encoding set; Step 53: Perform fixed-order concatenation and normalization on the orthogonal independent semantic segment coding set to obtain the structured global feature coding vector.

6. The global feature encoding method for reference content based on multimodal fusion as described in claim 4, characterized in that, Step 41 includes: Step 411: Perform fixed-scale semantic clustering and cluster center feature extraction on the purified multimodal semantic unit set with semantic hierarchical classification labels to obtain the initial semantic cluster feature set of each modality; Step 412: Perform multi-level forward global aggregation and reverse context correction recursive closed-loop processing on the initial semantic cluster feature sets of each modality to obtain the multi-level recursive cluster feature sets of each modality; Step 413: Perform hierarchical semantic regularization and dimensional unification on the feature sets of each modality's multi-level recursive clusters to obtain the hierarchical global feature sets of each modality.

7. The global feature encoding method for reference content based on multimodal fusion as described in claim 6, characterized in that, Step 412 includes: Step 4121: Perform intra-cluster self-attention semantic refinement and inter-cluster global interaction aggregation on the semantic cluster feature set of the current level to obtain the high-level global cluster feature set of the next higher level; Step 4122: Map the previous level high-level global cluster feature set in reverse to the current level semantic space, perform low-level feature weighting correction under global semantic constraints, and thus obtain the current level semantic cluster feature set after context calibration; Step 4123: Repeat the forward aggregation and backward correction closed-loop operation layer by layer according to the preset fixed recursion layer number until all levels of processing are completed, thereby obtaining the feature set of multi-level recursive clusters of each modality.

8. The global feature encoding method for reference content based on multimodal fusion as described in claim 7, characterized in that, Step 43 includes: Step 431: Based on the cross-modal fusion features with anchor matching relationships, generate initial vectors for each set of matching anchor pairs corresponding to independent semantic dimensions, thereby obtaining the initial semantic basis vector set; Step 432: Perform Schmitt orthogonalization and semantic energy fidelity constraint correction on the initial semantic basis vector set to obtain an orthogonal and interference-free semantic basis matrix; Step 433: Map the cross-modal fusion features to the orthogonal semantic base space and perform dimensional transformation and normalization to obtain the initial global feature encoding vector with anchor dimensional mapping relationship.

9. A global feature encoding device for reference content based on multimodal fusion, characterized in that, The reference content global feature encoding device based on multimodal fusion includes: An initial semantic unit feature set acquisition module is used to acquire the initial semantic unit feature set corresponding to the multimodal reference content to be encoded. A purified multimodal semantic unit set acquisition module is used to perform cross-modal semantic redundancy resolution and hierarchical division on the initial semantic unit feature set corresponding to the multimodal reference content to be encoded, thereby obtaining a purified multimodal semantic unit set with semantic hierarchical classification labels. A global semantic benchmark anchoring configuration acquisition module is used to perform global master-slave modality determination and benchmark anchoring on a set of purified multimodal semantic units with semantic hierarchical classification labels, thereby acquiring the global semantic benchmark anchoring configuration. A multimodal semantic anchoring recursive fusion coding network is used to perform global feature initial encoding on the purified multimodal semantic unit set with semantic hierarchical classification labels and the global semantic benchmark anchoring configuration, thereby obtaining a global feature initial encoding vector with anchor point dimension mapping relationship. The global feature encoding vector acquisition module is used to perform global semantic hierarchical structured mapping and encoding on the initial global feature encoding vector with anchor dimension mapping relationship, thereby obtaining the global feature encoding vector.