Intelligent digital human training method and system based on multi-modal interaction

Abstract

The invention discloses an intelligent digital human training method and system based on multi-modal interaction, and belongs to the technical field of semantic indexing.The method specifically comprises the steps that voice, vision and text data are analyzed and converted into high-dimensional feature vectors through a modal exclusive encoder, the high-dimensional feature vectors are projected to a unified semantic space through a cross-modal semantic mapping model, and the high-dimensional feature vectors are obtained; generating a semantic primitive containing a modal identifier, a core semantic tag and a feature weight; semantic primitives are used as nodes, directed edges and edge weight table association strength are established based on semantic similarity, typical scene node connection weights are strengthened, and a mesh map containing intra-modal hierarchy and inter-modal cross association is formed; constructing a double-layer index on the basis of the mesh map; semantic primitives are extracted from newly added data, the position of a new node in an association graph is determined through a graph matching algorithm, an association edge with an existing node is automatically established, and a lower-layer modal exclusive index is synchronously updated.

Description

Technical Field

[0001] This invention relates to the field of semantic indexing technology, specifically to a method and system for training intelligent digital humans based on multimodal interaction. Background Technology

[0002] In recent years, intelligent digital human technology has rapidly developed in the field of multimodal interaction, and its training system has integrated multiple data processing technologies such as speech, vision, and text. In the speech modality, end-to-end recognition models based on deep learning have achieved high-accuracy speech-to-text conversion in noisy environments, and emotional speech synthesis technology can simulate subtle changes in human intonation. In the visual modality, 3D pose estimation and facial expression capture technologies can accurately extract facial muscle movements and limb action features. In the text modality, pre-trained language models are used to complete intent recognition and contextual semantic parsing. Existing training methods mostly adopt single-modal feature independent optimization strategies, such as training acoustic models with a large amount of speech data or optimizing action generation networks based on labeled visual samples, and then achieving multimodal fusion through simple feature concatenation to support digital humans in completing basic interaction tasks such as "speech-action" and "text-expression". However, existing systems suffer from significant bottlenecks in multimodal data collaborative training: the lack of a semantic-aware indexing mechanism for heterogeneous data leads to low efficiency in cross-modal feature retrieval. Speech data is stored as Mel-spectrum or MFCC feature sequences, visual data exists as 3D coordinate frames or image feature vectors, and text is represented as word vectors or semantic tensors. The indexing structures for these three modalities are constructed independently based on the data format, without incorporating potential semantic associations such as "voice command - body response" and "text description - scene vision." This necessitates a full scan of irrelevant modal data during model training, severely limiting training efficiency and interaction response speed. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for training intelligent digital humans based on multimodal interaction, addressing the following technical problems: Existing systems lack heterogeneous indexing strategies for multimodal data during training, resulting in low efficiency in cross-modal feature retrieval. Specifically, the database uses a simple parallel storage method and fails to establish semantic mapping relationships between modalities, requiring the traversal of a large amount of irrelevant data during training. The objective of this invention can be achieved through the following technical solutions: A method for training intelligent digital humans based on multimodal interaction includes: S1. Parse speech, visual, and text data, convert them into high-dimensional feature vectors through a modality-specific encoder, and project them into a unified semantic space through a cross-modal semantic mapping model to generate semantic primitives containing modality identifiers, core semantic labels, and feature weights. S2. Using semantic primitives as nodes, directed edges are established based on semantic similarity. The edge weight table represents the association strength, and the connection weight of nodes in typical scenarios is strengthened to form a network graph containing intramodal hierarchy and intermodal cross-associations. S3. Construct a two-layer index based on the network graph. The upper layer is a graph index, which uses a graph database to support cross-modal retrieval. The lower layer is a modality-specific index, which uses an inverted spectrum for speech, a feature tree for vision, and an inverted vocabulary for text. These are linked to the upper layer through unique identifiers. S4. Extract semantic primitives from the new data, determine the position of the new node in the association graph through the graph matching algorithm, automatically establish association edges with existing nodes, and synchronously update the lower-level modality-specific index. When the change in semantic association strength exceeds the threshold, recalculate the weight of the corresponding edge and adjust the index structure. S5. After receiving the retrieval request, parse the request modality type and target semantics, query the semantic path through the graph index, filter the optimal path by accumulating the path weights, call the corresponding modality-specific index to retrieve and return the results.

[0004] As a further aspect of the present invention: in step S1, the method for parsing speech, visual, and text data and converting them into high-dimensional feature vectors via a modality-specific encoder is as follows: The original speech data is pre-emphasized, framed, and windowed. Mel frequency cepstral coefficients and fundamental frequency features are extracted to form a speech feature sequence. This sequence is then input into a pre-trained speech-specific encoder, which outputs a high-dimensional speech feature vector of dimension D. The original visual data is normalized in size and transformed in color space. A multi-scale convolutional neural network is used to extract shallow edge features and deep semantic features to form a visual feature tensor. This tensor is input into a pre-trained visual-specific encoder to output a high-dimensional visual feature vector with dimension D. The original text data is segmented and filtered for stop words to generate a text word sequence. The word sequence is then processed by a pre-trained text-specific encoder to output a high-dimensional text feature vector of dimension D.

[0005] As a further aspect of the present invention: the process of generating semantic primitives containing modality identifiers, core semantic tags, and feature weights is as follows: A cross-modal semantic mapping model based on contrastive learning is constructed, which includes a modality adaptation layer and a shared semantic projection layer. High-dimensional feature vectors of speech, vision, and text are input into the corresponding modality adaptation layers, and after nonlinear transformation, they are fed into the shared semantic projection layer. By minimizing the semantic distance of the same type within the modality and maximizing the semantic distance of different types, the feature vectors are projected into a unified semantic space. The projected vectors are then semantically parsed to generate semantic primitives containing modality identifiers, core semantic labels, and feature weights calculated based on attention weights.

[0006] As a further aspect of the present invention: the construction of the cross-modal semantic association graph includes: in step S2, the process of forming a network graph containing intra-modal hierarchies and inter-modal cross-associations is as follows: Semantic primitives containing modality identifiers, core semantic labels, and feature weights are stored according to a preset data structure, and a node attribute index table is established. The core semantic labels adopt a hierarchical label system, and the feature weights are associated with the corresponding feature dimensions in vector form. Based on feature vectors in a unified semantic space, a weighted Euclidean distance algorithm is used to calculate basic similarity. The word vector matching results of the core semantic tags are combined to calculate tag similarity. The two are then fused in proportion to obtain the final semantic similarity. For node pairs with semantic similarity reaching a set threshold, the subject-predicate relationship of the core semantic tags is analyzed through a semantic flow determination model to determine the direction of the edge. The initial weight of the edge is assigned as the final semantic similarity. Construct a scene weight mapping matrix. When the modal combination of two nodes of a directed edge matches the typical scene type in the matrix, multiply the edge weight by the corresponding association strength correction coefficient and update it. For nodes with the same modality, a hierarchical tag system based on the core semantic tags is used to construct a subordinate relationship tree. A one-way hierarchical edge is established between the parent node and the child node, and the weight is calculated based on the tag subordinate depth. All directed edges are aggregated, and after the integrity of node connections is checked by the graph topology verification algorithm, the node attributes, edge directions, and weight information are imported into the graph database to generate a network graph containing intramodal hierarchical relationships and intermodal cross-associations.

[0007] As a further aspect of the present invention: in step S3, the process of constructing a two-layer index based on the network map is as follows: Extract node and edge attributes from the semantic association graph, construct node attribute mapping table and edge attribute mapping table, store nodes with the unique identifier of semantic primitive as primary key, store edges with the source node unique identifier and the target node unique identifier as composite primary key, and import into the graph database to form a basic graph index. Based on the graph database index engine, a label index is built for the core semantic labels of nodes, a type index is built for the association type of edges, and a semantic path index table is constructed to record node sequences, edge weight accumulation values ​​and modality transformation relationships. Establish an index association mapping table, using the unique identifier of the semantic primitive as the key to associate it with the node ID in the graph index and the position information in the modality-specific index.

[0008] As a further aspect of the present invention: in step S4, the process of automatically establishing associated edges with existing nodes and synchronously updating the lower-level modality-specific index is as follows: Perform a semantic primitive extraction process on the new data to generate incremental semantic primitives containing modality identifiers, core semantic labels and feature weights, and assign them unique identifiers; A hierarchical graph matching algorithm is adopted. First, candidate node sets at the same level are screened by matching core semantic labels. Then, the similarity between the feature vectors of incremental semantic primitives and candidate node sets is calculated. The Top-N matching nodes are determined by sorting the similarity and locating the incremental semantic primitives to the corresponding semantic clusters in the association graph. For the localized incremental semantic primitives and Top-N matching nodes, the semantic flow determination model is invoked to determine the direction of the edges, and initial weights are assigned based on the semantic similarity between the two to establish new associated edges; at the same time, the hierarchical relationship between the incremental semantic primitives and nodes of the same modality is checked, and intramodal hierarchical edges are generated. Based on the modality identifier of the incremental semantic primitive, update the corresponding modality-specific index: for speech data, update the feature block association items of the inverted spectral index; for visual data, insert the corresponding clustering branch of the feature tree; and for text data, supplement the term association records of the inverted vocabulary. Simultaneously update the index position information corresponding to the unique identifier of the incremental semantic primitive in the index association mapping table.

[0009] As a further aspect of the present invention: the process of recalculating the weight of the corresponding edge and adjusting the index structure when a change in semantic association strength is detected to exceed a threshold is as follows: The semantic similarity change of newly added and existing associated edges is calculated in real time. When the change exceeds the preset association strength threshold, the weight of the corresponding edge is recalculated. For the edge with the adjusted weight, the edge attributes in the graph index are updated, and the weight of the associated items in the lower-level modality-specific index is synchronously corrected. For edges with adjusted weights, the topological balance of the associated graph is checked using a graph structure optimization algorithm. If there are locally dense connections or isolated nodes, the subgraph index of that region is reconstructed; the index item grouping of the lower-level modality-specific index is adjusted simultaneously.

[0010] A training system for intelligent digital humans based on multimodal interaction, comprising: The multimodal semantic primitive extraction module is used to parse speech, visual, and text data. The data is converted into high-dimensional feature vectors by a modality-specific encoder and then projected onto a unified semantic space through a cross-modal semantic mapping model, generating semantic primitives containing modality identifiers, core semantic labels, and feature weights. The cross-modal semantic association graph construction module is used to establish directed edges based on semantic similarity with semantic primitives as nodes, and the edge weight table is used to strengthen the connection weight of nodes in typical scenarios, forming a network graph containing intramodal hierarchy and intermodal cross associations. The hierarchical heterogeneous index building module is used to construct a two-layer index based on the network graph. The upper layer is a graph index, stored in a graph database to support cross-modal retrieval; the lower layer is a modality-specific index, using inverted spectrograms for speech, feature trees for vision, and inverted vocabularies for text, linked to the upper layer by unique identifiers. The dynamic index update module is used to extract semantic primitives from new data, determine the position of new nodes in the association graph through graph matching algorithms, automatically establish association edges with existing nodes, and synchronously update the lower-level modality-specific index. When the change in semantic association strength is detected to exceed the threshold, the weight of the corresponding edge is recalculated and the index structure is adjusted. The retrieval path optimization and execution module receives retrieval requests, parses the request modality type and target semantics, queries the semantic path through the graph index, filters the optimal path by accumulating path weights, calls the corresponding modality-specific index to retrieve the results, and returns the results.

[0011] The beneficial effects of this invention are: This invention fundamentally solves the retrieval efficiency problem caused by the heterogeneity of multimodal data by constructing a unified semantic space and a network-based relational graph. The generation of semantic primitives achieves semantic alignment of speech, visual, and text features, making cross-modal data comparable within a unified dimension and breaking down the modal barriers of traditional parallel storage models. The network graph, through semantic similarity-weighted connections, strengthens the intrinsic connections between modalities in typical interaction scenarios. Combined with a two-layer index structure, the upper-layer graph index can quickly locate cross-modal semantic paths, while the lower-layer dedicated index utilizes modal characteristics to achieve accurate retrieval, significantly reducing the amount of irrelevant data traversal and significantly improving data retrieval efficiency during training. Meanwhile, this solution boasts dynamic self-optimization capabilities and superior retrieval accuracy. The automatic graph embedding and index update mechanism for newly added data ensures that the system maintains high retrieval performance even as data volume expands, avoiding the performance degradation that occurs with traditional static indexes as data volume grows. By accumulating semantic path weights to filter for the optimal path, the system prioritizes returning the search results with the strongest associations, directly locating the most semantically related multimodal data combinations and reducing redundant information interference. This not only accelerates the data iteration efficiency in intelligent digital human training but also improves the real-time response speed and semantic matching accuracy of multimodal interactions, providing technical support for natural interaction in complex scenarios. Attached Figure Description

[0012] The invention will now be further described with reference to the accompanying drawings.

[0013] Figure 1 This is a flowchart illustrating an intelligent digital human training method based on multimodal interaction according to the present invention.

[0014] Figure 2This is a schematic diagram of the structure of an intelligent digital human training system based on multimodal interaction according to the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figure 1 As shown, the present invention is a training method for intelligent digital humans based on multimodal interaction.

[0017] S1: Perform feature parsing on speech, vision, and text data respectively. Convert the raw data into high-dimensional feature vectors through a pre-trained modality-specific encoder. Then, use a cross-modal semantic mapping model to project the feature vectors of different modalities into a unified semantic space to generate interoperable semantic primitives. Each semantic primitive contains modality identifier, core semantic label, and feature weights. S2: Using semantic primitives as nodes, directed edges are established between nodes based on semantic similarity calculation results. The weight of the edge represents the semantic association strength between the two nodes. For typical association scenarios such as "voice-gesture" and "text-visual scene", association rules are set to strengthen the connection weight between corresponding nodes, forming a network graph structure that includes intramodal hierarchical relationships and intermodal cross-associations. S3: Construct a two-layer index based on the semantic association graph: The upper layer is the graph index, which uses a graph database to store the relationship information between nodes and edges and supports cross-modal retrieval based on semantic paths; the lower layer is a modality-specific index, which uses an inverted spectral index for speech data, a feature tree index for visual data, and an inverted vocabulary index for text data. Each modality index is associated with the graph index through a unique identifier of a semantic primitive. S4: Monitor newly added training data in real time, perform the semantic primitive extraction process of step S1 on it, determine the position of the new node in the association graph through the graph matching algorithm, automatically establish association edges with existing nodes, and synchronously update the lower-level modality-specific index. When the change in semantic association strength is detected to exceed the threshold, recalculate the weight of the corresponding edge and adjust the index structure. S5: After receiving the retrieval request, it parses the request modality type and target semantics, queries all possible semantic paths through the graph index, uses the path weight accumulation algorithm to select the optimal path, calls the dedicated index of the corresponding modality based on the path to perform accurate retrieval, and returns a result set that meets the semantic association requirements.

[0018] In step S1, the method of performing feature parsing on speech, visual, and text data respectively, and converting the original data into high-dimensional feature vectors through a pre-trained modality-specific encoder is as follows: The raw speech data undergoes pre-emphasis, framing, and windowing processing. Pre-emphasis uses a high-pass filter to boost high-frequency components in the speech signal, compensating for the natural attenuation of high-frequency components during speech transmission and making the signal spectrum more uniform. Framing divides the continuous speech signal into multiple overlapping short frames (typically 20-30 milliseconds long, with a 10-15 millisecond overlap between frames). Because speech signals are approximately stationary over short periods, framing transforms dynamic signals into a series of static frames for processing. Windowing applies a smoothing window to each frame to reduce signal abrupt changes at frame edges and minimize spectral leakage during subsequent spectral analysis. Based on this, Mel-frequency cepstral coefficients and fundamental frequency features are extracted. Mel-frequency cepstral coefficient extraction is based on the nonlinear perception of frequency by the human ear. The spectrum of the speech frame is first mapped to a Mel-scale filter bank that conforms to human perception. Then, the logarithm of the filtering result is taken and a discrete cosine transform is performed to obtain coefficients that effectively characterize the envelope of the speech spectrum. Fundamental frequency features reflect the fundamental frequency of vocal cord vibration and are extracted by detecting periodic components in the speech signal (e.g., using autocorrelation functions to identify signal periods), thus characterizing the pitch information of the speech. Combining these time-sequentially arranged features creates a speech feature sequence that reflects the dynamic changes in speech. This sequence is then input into a pre-trained speech-specific encoder based on a bidirectional Transformer architecture. Its self-attention mechanism calculates the association weights between each frame's features and all other frames in the sequence, capturing the dependencies between speech features at different time points (such as the association between preceding and following syllables). The feedforward neural network performs non-linear transformations on the features at each position to enhance expressive power. Through this deep processing, the encoder can map the original feature sequence to a high-dimensional space, outputting a D-dimensional high-dimensional speech feature vector that integrates the speech's spectrum, fundamental frequency, and temporal information, achieving efficient representation of speech.

[0019] Perform size normalization and color space conversion on the original visual data: Size normalization adjusts images or video frames of different resolutions to a unified size through scaling, avoiding inconsistent network processing during subsequent feature extraction due to size differences and ensuring the unity of the spatial scale of the input data; Color space conversion usually converts the RGB space to spaces such as HSV or YCbCr. Since the RGB components have strong correlations, after conversion, the brightness and chromaticity can be separated, reducing feature redundancy and being more in line with the color perception characteristics of the human eye. Then, a multi-scale convolutional neural network is used to extract features: This network contains convolutional layers with different receptive fields. The shallow convolutional layers use small-size convolutional kernels (such as 3x3), with small receptive fields, mainly capturing local details such as edges and textures in the image, which are the basis for constructing visual content; The deep convolutional layers expand the receptive field through stacked convolutional operations, and can integrate local features to form complex semantic features (such as object parts and overall contours), reflecting high-level semantics. The feature maps output by each layer are concatenated along the channel dimension to form a visual feature tensor containing multi-scale and multi-level information. This tensor is input into a pre-trained visual-specific encoder. The encoder fuses convolutional features with the global correlation features of the Vision Transformer. The convolutional part refines spatial features, and the self-attention mechanism of the Transformer calculates the correlations between features at different spatial positions (such as the relationship between an object and the background), capturing global spatial dependencies. Finally, the three-dimensional tensor is compressed into a high-dimensional visual feature vector of dimension D, integrating local details and global structure information.

[0020] Perform word segmentation and stop word filtering on the original text data: Word segmentation is to split a continuous character sequence into words or subwords with independent semantics (such as splitting "intelligent digital human" into "intelligent" and "digital human"), because text semantics are expressed by word combinations, and word segmentation can break it down into semantic units; Stop word filtering is to remove words that appear frequently but have little semantic contribution, such as "de" (of) and "shi" (is), reducing redundancy and retaining core words. These words are arranged in the original order to form a text word sequence reflecting the core semantics of the text. The word sequence is processed through a pre-trained text-specific encoder. This encoder is trained based on a masked language model and contains multiple Transformer encoder layers. The word sequence is first converted into word embedding vectors. The self-attention mechanism calculates the context correlations between words (such as the correlation between "weather" and "sunny"). The feed-forward neural network enhances feature representation. After multiple layers of processing, the output high-dimensional text feature vector of dimension D fuses the context and overall semantics of the words, achieving a compact representation of the text.

[0021] In the step S1, the process of generating interoperable semantic primitives is as follows: A cross-modal semantic mapping model based on contrastive learning is constructed. This model includes a modality adaptation layer and a shared semantic projection layer. The core logic of contrastive learning is to learn the common semantic representations of different modal data by designing a contrast task of positive and negative sample pairs. The modality adaptation layer is an independent neural network layer designed for the feature distribution characteristics of speech, vision, and text, which is used to eliminate the inherent feature differences between different modalities (such as the temporal nature of speech, the spatial nature of vision, and the sequential nature of text), so that the features of each modality complete the initial distribution alignment before entering the shared space. The shared semantic projection layer is a neural network layer shared by all modalities, which is responsible for further mapping the adapted features to the same high-dimensional semantic space, providing a basis for cross-modal semantic comparison. High-dimensional feature vectors from speech, vision, and text are input into their respective modality adaptation layers. Since the original features of different modalities differ significantly in dimensionality, distribution, and representation (e.g., speech features emphasize spectral and temporal variations, visual features emphasize spatial structure and texture, and text features emphasize inter-word relationships), the modality adaptation layer adjusts the feature distribution through nonlinear transformations involving activation functions (such as nonlinear mappings introduced by ReLU and GELU activation functions). This weakens the interference caused by modality specificity, making the processed features closer to the potential distribution of the shared semantic space. These adapted features are then passed to the shared semantic projection layer. In the shared semantic projection layer, the similarity within the modality is minimized. Semantic distance and maximizing heterogeneous semantic distance enable feature projection. "Same-type semantics" refers to data expressing the same or similar meanings (such as the voice command "point to the left" and the corresponding gesture image), while "heterogeneous semantics" refers to data with significantly different meanings (such as the voice command "point to the left" and the gesture image "point to the right"). This design is because different modal data with the same semantics should have a relatively close distance in a unified space, while heterogeneous semantic data should maintain a relatively large distance. Through this contrast constraint, the model can learn pure semantic features that are not affected by modal form. Finally, the feature vectors of all modalities are projected into a unified semantic space, so that the semantic similarity of different modalities can be directly measured by spatial distance.The projected vectors are semantically parsed to generate semantic primitives containing modality identifiers, core semantic labels, and feature weights calculated based on attention weights. The modality identifier is determined based on the modality type of the original data (speech, vision, or text), because the projection process only changes the representation space of the features without changing their source attributes. The core semantic labels are obtained by comparing the similarity of the projected vectors with a predefined semantic label vector library. Since the projected vectors have highly condensed semantic information, the closer they are to a certain label vector in the label vector library, the more consistent the semantics expressed by the vector is with the corresponding label, and the core semantic label can be determined accordingly. The feature weights are calculated based on attention weights, because during feature encoding, the attention mechanism assigns higher weights to the parts of the input features that contribute more to the semantic expression (such as intonation features that carry emotions in speech, key parts that reflect actions in vision, and keyword features that express the core intent in text). These weights can reflect the importance of different feature dimensions to the overall semantics, and therefore can be directly used as feature weights for semantic primitives. Combining these information forms interoperable semantic primitives.

[0022] In step S2, the process of forming a network graph structure that includes intramodal hierarchical relationships and intermodal cross-correlation is as follows: Semantic primitives containing modality identifiers, core semantic tags, and feature weights are stored according to a pre-defined data structure. This pre-defined data structure is designed to ensure that semantic primitives of different modalities can be organized and accessed in a unified manner, avoiding storage chaos caused by structural differences. A node attribute index table is established to quickly locate the attribute information of each semantic primitive through the index, reducing traversal time during subsequent queries. The core semantic tags adopt a hierarchical tag system because semantics themselves have hierarchical relationships (e.g., "action" includes "gesture," and "gesture" includes "waving"), and the hierarchical system can clearly reflect this hierarchical semantic relationship. Feature weights are associated with corresponding feature dimensions in vector form because the features of each semantic primitive are composed of multiple dimensions, and the vector form can directly correspond to the importance of each dimension, so that the feature weights and the original feature dimensions form a one-to-one correspondence.

[0023] Based on feature vectors in a unified semantic space, a weighted Euclidean distance algorithm is used to calculate basic similarity. The weighting is because different feature dimensions contribute differently to semantic expression (the feature weights already reflect this difference). Euclidean distance can measure the straight-line distance between vectors in space, and the weighting can more accurately reflect the semantic differences caused by differences in key features. The label similarity is calculated by combining the word vector matching results of the core semantic labels, because the core semantic labels are direct textual descriptions of semantics. After the word vectors convert the labels into vectors, their spatial distance can intuitively reflect the semantic similarity of the labels. The two are fused proportionally to obtain the final semantic similarity, because feature vectors focus on the similarity of low-level features, while labels focus on the consistency of high-level semantics. After fusion, the semantic association between the two semantic primitives can be comprehensively measured from multiple dimensions.

[0024] For node pairs whose semantic similarity reaches a set threshold, the threshold is set to filter out node pairs with weak semantic association and avoid meaningless connections in the graph. The direction of the edge is determined by analyzing the subject-predicate relationship of the core semantic labels through the semantic flow determination model, because the subject-predicate relationship can reflect the semantic transmission logic (for example, "voice command" as the subject and "gesture execution" as the predicate, the semantic flow is naturally from voice to gesture). The initial weight of the edge is assigned to the final semantic similarity, because the final semantic similarity directly quantifies the association strength between the two nodes and can be used as the initial benchmark for the edge weight.

[0025] A scene weight mapping matrix is ​​constructed. The role of the matrix is ​​to associate typical interaction scenarios with the rules for adjusting the association strength. When the modal combination of two nodes of a directed edge matches the typical scene type in the matrix, the edge weight is multiplied by the corresponding association strength correction coefficient and updated. This is because in these typical scenarios (such as "voice-gesture" and "text-visual scenarios"), the association between modalities has stronger practical significance and frequency of occurrence. Strengthening its weight by correcting the coefficient can make the graph more in line with actual interaction needs.

[0026] For nodes with the same modality, a hierarchical tagging system based on core semantic tags is used to construct a subordinate relationship tree. This is because semantic primitives within the same modality have a hierarchical relationship between abstract and concrete elements (for example, in a visual modality, "emotional expression" includes "happy expression"). The subordinate relationship tree can clearly define this inherent logic. A unidirectional hierarchical edge is established between parent and child nodes, with the direction pointing from the parent node to the child node. This is because abstract semantics is a higher-level concept than concrete semantics, and the semantic flow should be from abstract to concrete. The weight is calculated based on the tag subordinate depth. The greater the depth (the more concrete the child node), the stronger its subordinate association with the parent node will show a corresponding pattern (e.g., the weight is adjusted according to certain rules as the depth increases), to reflect the degree of semantic dependence between levels.

[0027] All directed edges are aggregated, including cross-modal associative edges and intramodal hierarchical edges. The integrity of node connections is checked using a graph topology verification algorithm. This algorithm checks for completely isolated nodes (i.e., nodes without any edge connections to other nodes), ensuring that all nodes in the graph can form effective associations through edges. Then, node attributes, edge directions, and weight information are imported into a graph database. Graph databases are adept at storing and managing complex relationship networks formed by nodes and edges, and can efficiently support relationship-based queries. The resulting network graph contains intramodal hierarchical relationships and intermodal cross-associations, providing a structured semantic association foundation for subsequent cross-modal retrieval.

[0028] In step S3, the process of constructing a two-layer index based on the semantic association graph is as follows: Extracting node and edge attributes from the semantic association graph is crucial because node attributes (such as modality identifiers, core semantic labels, and feature weights) and edge attributes (such as direction, weight, and association type) are the foundational data for index construction. Only by extracting this information can subsequent organization and storage be performed. Constructing node and edge attribute mapping tables integrates scattered attribute information according to corresponding relationships, creating clear associations between nodes and their attributes, and between edges and their attributes, facilitating rapid subsequent attribute information retrieval. Nodes are stored using the unique identifier of the semantic primitive as the primary key. Since the unique identifier uniquely distinguishes each semantic primitive, using it as the primary key ensures the uniqueness of nodes during storage and retrieval, avoiding confusion of attribute information between different nodes. Edges are stored using a composite primary key of the source node's unique identifier and the target node's unique identifier. Because the existence of an edge depends on two nodes, the composite primary key uniquely identifies an edge, accurately reflecting the connection relationship between nodes and avoiding duplicate or missing edge information. Importing this information into a graph database to form the basic graph index is essential because graph databases are specifically optimized for network relationships formed by nodes and edges, efficiently storing and managing complex association structures, providing fundamental support for subsequent relationship-based retrieval.

[0029] Based on the graph database indexing engine, a tag index is built for the core semantic labels of nodes. The core semantic labels are the direct embodiment of the node's semantics. After the tag index is built, when it is necessary to query a node with a specific semantic label, it can be located directly through the index without traversing all nodes, reducing query time. A type index is built for the association type of edges. The association type of edges (such as cross-modal association and intra-modal hierarchical association) is an important basis for distinguishing edge functions. The type index makes the process of filtering edges by association type more efficient and quickly locates the required type of edge. A semantic path index table is built to record node sequences, edge weight accumulation values ​​and modality transformation relationships. Semantic paths are the key clues for cross-modal retrieval. Pre-storing the key information of the path can avoid recalculating the path during retrieval and can be directly obtained through the index table, thereby improving the efficiency of path traversal in cross-modal retrieval.

[0030] For the semantic primitives corresponding to the speech data, the associated original speech Mel-spectral features are extracted. Mel-spectral features are generated based on the non-linear perception characteristics of human ear for sound frequencies, which can effectively characterize the spectral characteristics of speech and are more in line with human perception of speech. Feature blocks are divided according to frequency ranges because different frequency ranges of speech correspond to different acoustic components (such as fundamental frequency and harmonics). After division, the complex spectral features can be decomposed into more easily processed units, which is convenient for subsequent index construction. An inverted spectral index table is constructed with feature blocks as index items. Each item in the table is associated with a unique identifier of the speech semantic primitive containing the feature block. This inverted structure allows the corresponding primitive to be found directly through the feature block when it is necessary to query the speech semantic primitive containing specific spectral features, without having to check the features of primitives one by one, thus improving retrieval efficiency. The index table synchronously stores the timestamp information of the feature blocks in the original speech. The timestamp can accurately mark the position of the feature blocks in the original speech, thereby associating the feature blocks with the time dimension of the speech semantic primitives, realizing the accurate correspondence between features and primitives, and ensuring that the retrieved features belong to the correct speech segments.

[0031] For the semantic primitives corresponding to visual data, their associated visual feature vectors (including edge and texture features) are extracted. Edge features reflect the contours and boundaries of objects in the image, while texture features reflect the pattern regularity of the object's surface. Together, they constitute the basic structural information of visual content and are key to distinguishing different visual content. The k-means algorithm is used to cluster the feature vectors. The k-means algorithm can cluster vectors with similar features into one class, so that visual semantic primitives with similar features are grouped into the same cluster, reducing the comparison range during retrieval. A feature tree structure is generated. The root node of the feature tree is the global cluster center, representing the overall distribution center of all visual feature vectors. The leaf nodes are the feature vectors of individual visual semantic primitives. Each node stores its child node index and the corresponding unique identifier of the semantic primitive. This tree structure allows retrieval to start from the root node and compare with the child nodes layer by layer, quickly locating the leaf node similar to the target feature, thereby efficiently finding the corresponding visual semantic primitive.

[0032] For the semantic primitives corresponding to text data, extract the keywords included in their core semantic tags. Keywords are the condensation of the core semantics of the text and can directly reflect the main content and intention of the text. After removing stop words, a set of terms is generated. Stop words (such as "of", "is", etc.) appear frequently in the text but have little semantic contribution. Removing them can reduce redundant information and make the set of terms more focused on the core semantics. Using the terms as index terms, an inverted index table is constructed. Each item in the table associates the unique identifier of the text semantic primitive containing the term, and the frequency of occurrence of the term in the core semantic tag. The inverted structure allows direct positioning of the primitive containing the term through the term, and the frequency of occurrence can assist in judging the importance of the term in the semantic expression of the primitive. The terms are grouped and stored according to the similarity of word vectors. Word vectors transform terms into vectors through mapping, and the similarity between vectors can reflect the semantic proximity of terms. After such grouping, when querying a certain term, other terms with similar semantics can be associated simultaneously, thus supporting the associated query of synonymous terms.

[0033] Establish an index association mapping table, using the unique identifier of the semantic primitive as the key to associate its node ID in the graph index and the position information in each modality-specific index. Since the unique identifier can uniquely identify a semantic primitive, using it as the key can integrate the position information of this primitive in different indexes, realizing the bidirectional call between the upper-layer graph index and the lower-layer modality-specific index, ensuring that during the retrieval process, through one index, the corresponding information in other indexes can be quickly located, and ensuring the consistency between indexes.

[0034] In the step S4, the process of automatically establishing the association edge with the existing node and synchronously updating the lower-layer modality-specific index is as follows: Execute the semantic primitive extraction process on the new data. The new data needs to be transformed into a unified semantic representation compatible with the existing system in order to be integrated into the existing association graph and index structure. Generate incremental semantic primitives containing modality identifiers, core semantic tags, and feature weights. Among them, the modality identifier is used to distinguish data types (speech, vision, or text), the core semantic tag condenses the core meaning of the data, and the feature weight reflects the contribution degree of each feature dimension to the semantic expression. The three together constitute the semantic carrier of the incremental data. Assign a unique identifier to it because the unique identifier can ensure that this incremental semantic primitive has a unique identity in the entire system, avoiding confusion with existing primitives, and providing a unique reference for subsequent positioning, association, and index update.

[0035] A hierarchical graph matching algorithm is adopted. First, candidate node sets at the same level are filtered by core semantic label matching. This is because core semantic labels directly reflect the high-level semantics of primitives, and labels at the same level mean that the semantic categories are similar. This filtering can quickly narrow down the matching range and reduce unnecessary calculations. Then, the similarity of feature vectors between incremental semantic primitives and candidate node sets is calculated. Feature vectors contain the low-level feature information of primitives. Vector similarity can quantify the closeness of semantic details and make up for the coarse-grained problem that may exist by relying solely on label matching. The Top-N matching nodes are determined by sorting by similarity. This is to select the node that is most semantically close to the incremental primitive as the association object to ensure the effectiveness of the association. The incremental semantic primitive is located to the corresponding semantic cluster in the association graph. The semantic cluster is a group composed of primitives with similar semantics. Locating it here can integrate the new primitive into the group that is semantically related to it, ensuring the rationality of the graph structure.

[0036] For the located incremental semantic primitives and Top-N matching nodes, the semantic flow determination model is invoked to determine the direction of the edges. This model analyzes the subject-predicate relationship of core semantic tags (such as "voice command" pointing to "gesture action") to clarify the logical order of semantic transmission, ensuring that the direction of the edges conforms to the semantic flow in actual interaction. Initial weights are assigned based on the semantic similarity between the two primitives, as semantic similarity directly quantifies the association strength between the two primitives, and using this as the initial weight can objectively reflect the tightness of the association. New association edges are established to enable the incremental primitives to form semantic connections with existing nodes and integrate them into the association graph. At the same time, the hierarchical relationship between the incremental semantic primitives and nodes of the same modality is checked, and intra-modal hierarchical edges are generated. This is because primitives within the same modality have abstract and concrete hierarchical logic (such as "emotion" containing "joy"), and hierarchical edges can maintain this inherent semantic structure and ensure the integrity of intra-modal relationships in the graph.

[0037] Based on the modality identifier of the incremental semantic primitive, the corresponding modality-specific index is updated: For speech data, the feature block association entries of the inverted spectral index are updated. Since the speech inverted spectral index uses spectral feature blocks as index entries, the spectral features of the newly added speech primitive need to be associated with the corresponding feature blocks in order to locate the primitive through feature retrieval; For visual data, the corresponding clustering branch of the feature tree is inserted. Since the visual feature tree groups similar features into the same branch through clustering, the feature vector of the newly added visual primitive needs to be inserted into a branch similar to its features to maintain the clustering logic of the feature tree; For text data, the term association records of the inverted vocabulary are supplemented. The text inverted vocabulary uses keywords as index entries, and the keywords of the newly added text primitive need to be supplemented into the association records of the corresponding term entries to ensure that the primitive can be retrieved by keywords; The index position information corresponding to the unique identifier of the incremental semantic primitive in the index association mapping table is updated synchronously. The mapping table is the bridge connecting the upper-level spectral index and the lower-level modality-specific index. Synchronous updates can ensure that the corresponding position can be accurately found in the upper and lower-level indexes by using the unique identifier, maintaining the consistency between the indexes.

[0038] In step S4, the process of recalculating the weight of the corresponding edge and adjusting the index structure when the detected change in semantic association strength exceeds the threshold is as follows: The system calculates the changes in semantic similarity between newly added and existing associated edges in real time. As data accumulates and the semantic environment dynamically changes during system operation, the strength of associations between nodes is not constant. Real-time calculation captures these changes promptly. The calculation method compares the semantic similarity of the edge at the current moment with the similarity in historical records; the difference between the two is the change, reflecting fluctuations in association strength. When the change exceeds a preset association strength threshold, the weight of the corresponding edge is recalculated. The preset threshold is set to filter out minor, meaningless fluctuations, preventing instability in the index structure due to frequent adjustments. Only when the change is sufficiently significant can the weight be recalculated based on the latest semantic similarity, ensuring that the weight accurately reflects the current closeness of association between nodes.

[0039] For edges with adjusted weights, the edge attributes in the graph index are updated. The graph index stores key information such as edge weights; updating it ensures that the index information is consistent with the actual association state, providing accurate relational basis for subsequent retrieval. Simultaneously, the weights of association items in the lower-level modality-specific index are synchronously corrected. Since the lower-level modality-specific index and the upper-level graph index are linked through unique identifiers of semantic primitives, changes in the weights of upper-level edges indicate changes in the importance of corresponding association items. Synchronous correction ensures that the weights of association items in the lower-level index are consistent with those in the upper-level index, avoiding information discrepancies between the two levels. For edges with adjusted weights, a graph structure optimization algorithm is used to check the topological balance of the association graph. Topological balance refers to whether the connection distribution of nodes in the graph is reasonable—neither too dense, leading to redundant retrieval paths, nor too sparse, forming information islands. The algorithm analyzes indicators such as the number of connections and connection density of nodes to determine whether there are local areas with excessively dense node connections (e.g., a large number of edges concentrated in a few nodes) or isolated nodes (without effective connections to other nodes). If locally dense connections or isolated nodes exist, the subgraph index for that region is reconstructed. Locally dense connections increase the path traversal complexity during retrieval, while isolated nodes cannot participate in effective cross-modal associations. During reconstruction, edges in dense regions are reasonably simplified or redirected, and necessary connections are established between isolated nodes and related nodes. The structure of the subgraph is optimized to improve retrieval efficiency. At the same time, the grouping of index items in the lower-level modality-specific index is adjusted. Since the grouping method of the lower-level index depends on the association structure of the upper-level graph, the grouping of the corresponding modality-specific index items also needs to be adjusted after the upper-level subgraph is reconstructed to ensure that the organization of the lower-level index matches the association structure of the upper level and maintain the collaborative efficiency of the upper and lower-level indexes during the retrieval process.

[0040] In step S5, after receiving the retrieval request, the request modality type (clarifying the input data type) and target semantics (clarifying the retrieval intent) are parsed, which is the foundation for subsequent retrieval operations. All possible semantic paths are queried through the graph index. A path refers to a series of associated edges from the request's corresponding node to the target semantic node. The structure of the graph index supports efficient traversal of these paths. The optimal path is selected using a path weight accumulation algorithm. The accumulated weight value reflects the overall association strength of the path; a higher accumulated value indicates a tighter semantic association represented by the path, thus serving as a selection criterion. Based on this path, the corresponding modality's dedicated index is invoked for precise retrieval. The path specifies the target modality node to be queried, and invoking its dedicated index leverages the unique retrieval advantages of the modality (such as spectral matching for speech, feature comparison for vision, and keyword matching for text) to improve retrieval efficiency. Finally, a result set that meets the semantic association requirements is returned. These results are associated through the optimal path to ensure semantic consistency with the request. Figure 1 To.

[0041] Please see Figure 2 As shown, the present invention also provides an intelligent digital human training system based on multimodal interaction, comprising: The multimodal semantic primitive extraction module is used to parse speech, visual, and text data. The data is converted into high-dimensional feature vectors by a modality-specific encoder and then projected onto a unified semantic space through a cross-modal semantic mapping model, generating semantic primitives containing modality identifiers, core semantic labels, and feature weights. The cross-modal semantic association graph construction module is used to establish directed edges based on semantic similarity with semantic primitives as nodes, and the edge weight table is used to strengthen the connection weight of nodes in typical scenarios, forming a network graph containing intramodal hierarchy and intermodal cross associations. The hierarchical heterogeneous index building module is used to construct a two-layer index based on the network graph. The upper layer is a graph index, stored in a graph database to support cross-modal retrieval; the lower layer is a modality-specific index, using inverted spectrograms for speech, feature trees for vision, and inverted vocabularies for text, linked to the upper layer by unique identifiers. The dynamic index update module is used to extract semantic primitives from new data, determine the position of new nodes in the association graph through graph matching algorithms, automatically establish association edges with existing nodes, and synchronously update the lower-level modality-specific index. When the change in semantic association strength is detected to exceed the threshold, the weight of the corresponding edge is recalculated and the index structure is adjusted. The retrieval path optimization and execution module receives retrieval requests, parses the request modality type and target semantics, queries the semantic path through the graph index, filters the optimal path by accumulating path weights, calls the corresponding modality-specific index to retrieve the results, and returns the results.

[0042] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A training method for intelligent digital humans based on multimodal interaction, characterized in that, Includes the following steps: S1. Parse speech, visual, and text data, convert them into high-dimensional feature vectors through a modality-specific encoder, and project them into a unified semantic space through a cross-modal semantic mapping model to generate semantic primitives containing modality identifiers, core semantic labels, and feature weights. S2. Using semantic primitives as nodes, directed edges are established based on semantic similarity. The edge weight table represents the association strength, and the connection weight of nodes in typical scenarios is strengthened to form a network graph containing intramodal hierarchy and intermodal cross-associations. S3. Construct a two-layer index based on the network graph. The upper layer is a graph index, which uses a graph database to support cross-modal retrieval. The lower layer is a modality-specific index, which uses an inverted spectrum for speech, a feature tree for vision, and an inverted vocabulary for text. These are linked to the upper layer through unique identifiers. S4. Extract semantic primitives from the new data, determine the position of the new node in the association graph through the graph matching algorithm, automatically establish association edges with existing nodes, and synchronously update the lower-level modality-specific index. When the change in semantic association strength exceeds the threshold, recalculate the weight of the corresponding edge and adjust the index structure. S5. After receiving the retrieval request, parse the request modality type and target semantics, query the semantic path through the graph index, filter the optimal path by accumulating the path weights, call the corresponding modality-specific index to retrieve and return the results.

2. The intelligent digital human training method based on multimodal interaction according to claim 1, characterized in that, In step S1, the method of parsing speech, visual, and text data and converting it into high-dimensional feature vectors via a modality-specific encoder is as follows: The original speech data is pre-emphasized, framed, and windowed. Mel frequency cepstral coefficients and fundamental frequency features are extracted to form a speech feature sequence. This sequence is then input into a pre-trained speech-specific encoder, which outputs a high-dimensional speech feature vector of dimension D. The original visual data is normalized in size and transformed in color space. A multi-scale convolutional neural network is used to extract shallow edge features and deep semantic features to form a visual feature tensor. Input this tensor into a pre-trained vision-specific encoder, and it outputs a high-dimensional visual feature vector with dimension D. The original text data is segmented and filtered for stop words to generate a text word sequence. The word sequence is then processed by a pre-trained text-specific encoder to output a high-dimensional text feature vector of dimension D.

3. The intelligent digital human training method based on multimodal interaction according to claim 2, characterized in that, The process of generating semantic primitives containing modality identifiers, core semantic labels, and feature weights is as follows: A cross-modal semantic mapping model based on contrastive learning is constructed, which includes a modality adaptation layer and a shared semantic projection layer. High-dimensional feature vectors of speech, vision, and text are input into the corresponding modality adaptation layers, and after nonlinear transformation, they are fed into the shared semantic projection layer. By minimizing the semantic distance of the same type within the modality and maximizing the semantic distance of different types, the feature vectors are projected into a unified semantic space. The projected vectors are then semantically parsed to generate semantic primitives containing modality identifiers, core semantic labels, and feature weights calculated based on attention weights.

4. The intelligent digital human training method based on multimodal interaction according to claim 1, characterized in that, The construction of the cross-modal semantic association graph includes: in step S2, the process of forming a network graph containing intra-modal hierarchies and inter-modal cross-associations is as follows: Semantic primitives containing modality identifiers, core semantic labels, and feature weights are stored according to a preset data structure, and a node attribute index table is established. The core semantic labels adopt a hierarchical label system, and the feature weights are associated with the corresponding feature dimensions in vector form. Based on feature vectors in a unified semantic space, a weighted Euclidean distance algorithm is used to calculate basic similarity. The word vector matching results of the core semantic tags are combined to calculate tag similarity. The two are then fused in proportion to obtain the final semantic similarity. For node pairs with semantic similarity reaching a set threshold, the subject-predicate relationship of the core semantic tags is analyzed through a semantic flow determination model to determine the direction of the edge. The initial weight of the edge is assigned as the final semantic similarity. Construct a scene weight mapping matrix. When the modal combination of two nodes of a directed edge matches the typical scene type in the matrix, multiply the edge weight by the corresponding association strength correction coefficient and update it. For nodes with the same modality, a hierarchical tag system based on the core semantic tags is used to construct a subordinate relationship tree. A one-way hierarchical edge is established between the parent node and the child node, and the weight is calculated based on the tag subordinate depth. All directed edges are aggregated, and after the integrity of node connections is checked by the graph topology verification algorithm, the node attributes, edge directions, and weight information are imported into the graph database to generate a network graph containing intramodal hierarchical relationships and intermodal cross-associations.

5. The intelligent digital human training method based on multimodal interaction according to claim 1, characterized in that, In step S3, the process of constructing a two-layer index based on the network graph is as follows: Extract node and edge attributes from the semantic association graph, construct node attribute mapping table and edge attribute mapping table, store nodes with the unique identifier of semantic primitive as the primary key, store edges with the source node unique identifier and the target node unique identifier as the composite primary key, and import them into the graph database to form the basic graph index. Based on the graph database index engine, a label index is built for the core semantic labels of nodes, a type index is built for the association type of edges, and a semantic path index table is constructed to record node sequences, edge weight accumulation values ​​and modality transformation relationships. Establish an index association mapping table, using the unique identifier of the semantic primitive as the key to associate it with the node ID in the graph index and the position information in the modality-specific index.

6. The intelligent digital human training method based on multimodal interaction according to claim 1, characterized in that, In step S4, the process of automatically establishing associated edges with existing nodes and synchronously updating the lower-level modality-specific index is as follows: Perform a semantic primitive extraction process on the new data to generate incremental semantic primitives containing modality identifiers, core semantic labels and feature weights, and assign them unique identifiers; A hierarchical graph matching algorithm is adopted. First, candidate node sets at the same level are screened by matching core semantic labels. Then, the similarity between the feature vectors of incremental semantic primitives and candidate node sets is calculated. The Top-N matching nodes are determined by sorting the similarity and locating the incremental semantic primitives to the corresponding semantic clusters in the association graph. For the localized incremental semantic primitives and Top-N matching nodes, the semantic flow determination model is invoked to determine the direction of the edges, and initial weights are assigned based on the semantic similarity between the two to establish new associated edges; at the same time, the hierarchical relationship between the incremental semantic primitives and nodes of the same modality is checked, and intramodal hierarchical edges are generated. Based on the modality identifier of the incremental semantic primitive, update the corresponding modality-specific index: for speech data, update the feature block association items of the inverted spectral index; for visual data, insert the corresponding clustering branch of the feature tree; and for text data, supplement the term association records of the inverted vocabulary. Simultaneously update the index position information corresponding to the unique identifier of the incremental semantic primitive in the index association mapping table.

7. The intelligent digital human training method based on multimodal interaction according to claim 6, characterized in that, The process of recalculating the weight of the corresponding edge and adjusting the index structure when a change in semantic association strength exceeds a threshold is as follows: The semantic similarity change of newly added and existing associated edges is calculated in real time. When the change exceeds the preset association strength threshold, the weight of the corresponding edge is recalculated. For the edge with the adjusted weight, the edge attributes in the graph index are updated, and the weight of the associated items in the lower-level modality-specific index is synchronously corrected. For edges with adjusted weights, the topological balance of the associated graph is checked using a graph structure optimization algorithm. If there are locally dense connections or isolated nodes, the subgraph index of that region is reconstructed; the index item grouping of the lower-level modality-specific index is adjusted simultaneously.

8. A training system for intelligent digital humans based on multimodal interaction, characterized in that, include: The multimodal semantic primitive extraction module is used to parse speech, visual, and text data. The data is converted into high-dimensional feature vectors by a modality-specific encoder and then projected onto a unified semantic space through a cross-modal semantic mapping model, generating semantic primitives containing modality identifiers, core semantic labels, and feature weights. The cross-modal semantic association graph construction module is used to establish directed edges based on semantic similarity with semantic primitives as nodes, and the edge weight table is used to strengthen the connection weight of nodes in typical scenarios, forming a network graph containing intramodal hierarchy and intermodal cross associations. The hierarchical heterogeneous index building module is used to construct a two-layer index based on the network graph. The upper layer is a graph index, stored in a graph database to support cross-modal retrieval; the lower layer is a modality-specific index, using inverted spectrograms for speech, feature trees for vision, and inverted vocabularies for text, linked to the upper layer by unique identifiers. The dynamic index update module is used to extract semantic primitives from new data, determine the position of new nodes in the association graph through graph matching algorithms, automatically establish association edges with existing nodes, and synchronously update the lower-level modality-specific index. When the change in semantic association strength is detected to exceed the threshold, the weight of the corresponding edge is recalculated and the index structure is adjusted. The retrieval path optimization and execution module receives retrieval requests, parses the request modality type and target semantics, queries the semantic path through the graph index, filters the optimal path by accumulating path weights, calls the corresponding modality-specific index to retrieve the results, and returns the results.

Images