Dynamic graph multi-modal Gaussian adaptive clustering dynamic granularity retrieval method

Abstract

The invention provides a dynamic graph multi-modal Gaussian adaptive clustering dynamic granularity retrieval method, which comprises the following steps of: after receiving multi-modal data, respectively carrying out differential preprocessing, mapping all the multi-modal data into unified semantic feature vectors through a large model system, and then ensuring that core semantic elements are accurate through double-model verification, the method comprises the following steps: calculating a knowledge graph edge weight based on semantic similarity, resource association strength and long-range semantic association degree, introducing a time sequence attenuation factor, automatically recalculating the edge weight during resource updating, constructing a dynamic knowledge graph, clustering semantic feature vectors by adopting a GMM (Gaussian Mixture Model) with customized parameters, and performing EM iterative optimization and feature correction to obtain a dynamic knowledge graph. According to the method, the importance of semantic blocks is evaluated in combination with a graph attention network, a three-level structure is generated, global scene complexity and local semantic complexity are quantified, retrieval granularity is dynamically adapted through a gating function, and finally a retrieval result with'accuracy, efficiency and computing power 'balanced is output in combination with association logic of a dynamic knowledge graph.

Description

Technical Field

[0001] This invention belongs to the fields of computer technology, artificial intelligence and knowledge graph technology, and in particular relates to a dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method. Background Technology

[0002] Driven by both accelerated digital transformation and the widespread adoption of artificial intelligence, multimodal data, including text, images, audio, and tables, has become the mainstream form of information production, dissemination, and storage, widely used in key areas such as academic research, medical diagnosis, business operations, and digital content services. This type of data often contains rich cross-modal correlations—for example, the supporting relationship between academic research paper text and experimental data tables, the complementary information between medical record text and medical images in medical settings, and the correspondence between product descriptions and technical parameter tables in business operations. Maximizing its value relies heavily on efficient cross-modal retrieval technologies.

[0003] Cross-modal retrieval, as a core supporting technology for breaking down barriers between different types of data and realizing the fusion and utilization of multi-source information, has become a key prerequisite for the implementation of upper-level applications such as intelligent retrieval, deep data analysis, and personalized recommendations, directly determining the efficiency of data utilization and the level of service intelligence. Currently, as the scale of data leaps from millions to hundreds of millions, and the data forms become increasingly complex and diverse, with the application scenarios of long-sequence data and multi-document cross-domain data continuing to expand, the industry is placing increasingly higher demands on the accuracy, correlation completeness, real-time performance, and computing power adaptability of cross-modal retrieval.

[0004] However, existing cross-modal retrieval technologies are limited by the following problems:

[0005] 1. Inconsistent semantic representation systems for multimodal data lead to modal fragmentation, and the static and rigid edge weights of traditional knowledge graphs make it difficult to capture long-range implicit relationships across documents and paragraphs, resulting in insufficient timeliness of resource adaptation.

[0006] 2. The retrieval granularity is fixed and lacks quantitative standards for scenario adaptation. Coarse-grained retrieval is prone to missing key details, while fine-grained retrieval consumes too much computing power, making it impossible to balance retrieval accuracy, efficiency and computing power requirements.

[0007] In existing multimodal retrieval technologies, Gaussian Mixture Models (GMMs) mostly employ general parameter clustering, which is not adapted to the characteristics of multimodal data; knowledge graph matching relies on static weights, resulting in weak long-range association capture capabilities; and gating adjustment lacks quantitative basis, leading to blind granularity switching. Therefore, there is an urgent need in this field for a novel multimodal retrieval scheme that can dynamically perceive semantics, adaptively adjust granularity, and effectively capture long-range associations. Summary of the Invention

[0008] In view of this, the present invention aims to overcome the shortcomings of the prior art by proposing a dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method. This method achieves efficient fusion of multimodal data, a dynamic balance between retrieval accuracy and computational power consumption, and ensures the integrity of long-range semantic associations and flexibility in scenario adaptation. To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0009] In a first aspect, the present invention provides a dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method, comprising the following steps:

[0010] Step 1: Collect multimodal data and preprocess it. Map the preprocessed multimodal data into 256-dimensional original feature vectors through a large model cross-modal attention mechanism. Validate the original feature vectors with dual models to obtain a semantically accurate feature vector set and output the effective feature vectors.

[0011] Step 2: Based on the effective feature vectors output in Step 1, calculate the semantic similarity of multimodal data, the domain association strength with temporal decay, and the long-range association strength derived from GAT multi-hop, derive the three-dimensional dynamic edge weight Em, construct a dynamic knowledge graph, and output the association relationship between edge weights and the graph.

[0012] Step 3: The effective feature vectors output from Step 1 and the edge weights and graph associations output from Step 2 are clustered using a Gaussian Mixture Model (GMM) and then iteratively optimized and corrected using EM. The importance of semantic blocks is evaluated using a graph attention network, generating a three-level structure of "coarse-grained semantic clusters - medium-grained sub-semantic blocks - fine-grained feature units".

[0013] Step 4: The outputs of Steps 1, 2, and 3 are quantified using a complex global and local dual-dimensional metric. The granularity of the retrieval is adapted by a gating function, and combined with Em score sorting and timeliness filtering, the final output is a scenario-based and structured retrieval result.

[0014] Furthermore, in step 1, the multimodal data includes text data, image data, audio data, and tabular data. Preprocessing the multimodal data includes:

[0015] Text data: Word embeddings and cross-segment contextual semantic features are extracted using the long-range attention mechanism built into the large model;

[0016] Image data: Text information is extracted by OCR transcription, while object detection boxes, visual feature vectors and gradient change features are preserved;

[0017] Audio data: converted into text through speech recognition, with spectral features and key speech segment markers extracted simultaneously;

[0018] Tabular data: parsed into structured text and numerical features.

[0019] Furthermore, in step 1, the preprocessed multimodal data is mapped to a 256-dimensional original feature vector through a large model cross-modal attention mechanism. The original feature vector is then validated using a dual-model approach to obtain a semantically accurate feature vector set. The output effective feature vectors include:

[0020] Multimodal data is split into independent semantic units based on semantic integrity. Each semantic unit contains several core semantic elements. Through a large model cross-modal attention mechanism, the core semantic elements of different modalities are uniformly mapped to the same feature space and converted into the original feature vector after 256-dimensional unified semantic cross-modal mapping. The overlap rate of core semantic elements and the overlap rate of core semantic element feature vectors are verified by dual models respectively, and the effective feature vectors filtered by the overlap rate are output.

[0021] Furthermore, in step 2, a three-dimensional weighted collaborative edge weight calculation formula is designed, which includes semantic similarity, domain association strength with temporal decay, and long-range association derived from GAT multi-hop.

[0022] Em=α×Sim(Sm,Sn)+β×Rel(Rm,Rn)+(1-α-β)×LongSim(Lm,Ln)

[0023] Where E m As a comprehensive edge weight, it is a weighted fusion result of the similarity of three dimensions. α and β are core adjustment coefficients that satisfy α+β≤1. Sim(Sm, Sn) is the semantic similarity, Rel(Rm, Rn) is the domain association strength, and LongSim(Lm, Ln) is the long-range association.

[0024]

[0025] f m f n These are the 256-dimensional unified semantic feature vectors obtained after preprocessing the data Sm and Sn, respectively.

[0026] ||f m ||、||f n || represents the L2 norm of the two vectors;

[0027] Rel(R m ,R n )=τ×(w1·Corr+w2·Freq+w3·Adapt)

[0028] τ is the time decay factor, w1, w2, and w3 are weighting coefficients that satisfy w1+w2+w3=1, Corr is the domain relevance score, Freq is the usage frequency normalization score, and Adapt is the scene adaptation score.

[0029]

[0030] The final feature vectors of the nodes corresponding to data Sm and Sn after multi-hop propagation by GAT are calculated iteratively based on the propagation rules of the GAT layer:

[0031]

[0032] L is the number of propagation layers, α mn W represents the attention weights for GAT's automatic learning. (l) These are the learnable parameters for the l-th layer.

[0033] Furthermore, step 3 specifically includes:

[0034] Use the effective feature vectors obtained in step 1 as the training input data for the Gaussian Mixture Model (GMM).

[0035] During training, K-Means pre-clustering is first performed to obtain the initial cluster centers. Simultaneously, the initial mixing weight K is set to the number of clusters obtained from pre-clustering, and the calculation method is as follows:

[0036]

[0037] Using the initial covariance matrix As the covariance of samples within each cluster, the EM iterative optimization process is then initiated. The E-step calculates the posterior probability that the i-th feature vector belongs to the k-th cluster based on the multivariate Gaussian distribution probability density function.

[0038]

[0039] The M-step updates the mixed weights π based on the posterior probability. k Cluster center μ k The sum of the covariance matrix ∑ k Repeat the above iterative process until convergence stops when the parameter change is less than 1e-5;

[0040] The feature correction mechanism will be activated after clustering is completed;

[0041] Combining graph attention networks to evaluate the importance of semantic blocks, and using coarse-grained semantic block B output by a Gaussian mixture model (GMM) k For nodes, initial node characteristics:

[0042]

[0043] s i For the traditional importance score at position i, edges are constructed based on semantic similarity or k-nearest neighbor relationships, with edge weights equal to the corresponding semantic similarity. These edges are then propagated through an L-layer graph attention network for multi-hop propagation, with the following propagation rules:

[0044]

[0045] α mn Attention weights are ultimately determined by the final features of the nodes. As the total importance score of semantic blocks S(B) m The top-N core semantic blocks are retained in descending order of their scores.

[0046] Based on this, a three-level structured granularity is generated. The coarse granularity consists of K semantic clusters obtained after GMM convergence. The medium granularity is to split each coarse granular semantic cluster into sub-semantic blocks according to the core semantic element category. The fine granularity consists of the key feature units within the sub-semantic blocks.

[0047] Furthermore, step 4 specifically includes:

[0048] First, the semantic complexity is comprehensively represented using both global and local dimensions. The formula for the global complexity is:

[0049] Complexity = 0.3 × Modal Score + 0.5 × Element Score + 0.2 × Response Score

[0050] Local semantic complexity is achieved by calculating the attention score entropy and the feature gradient magnitude. The formula for the attention score entropy is:

[0051]

[0052] H(C k C is the attention score entropy of the k-th coarse-grained semantic cluster; k The k-th semantic cluster generated by GMM clustering contains feature vectors corresponding to several semantic units; p i For the normalized posterior probability of i:

[0053]

[0054] γ ik Let be the posterior probability of an eigenvector belonging to the k-th cluster, satisfying lnp i For p i The natural logarithm operation;

[0055] Based on the two-dimensional complex quantification results, the formula for calculating the gate function is as follows:

[0056] ρ(x)=σ(W·[Complexity,Entropy,Gradient]+b)

[0057] W and b are learnable references, σ is the Sigmoid function, outputs the fine-grained retrieval ratio ρ(x)∈(0,1), and adapts to different complexity scenarios with a switchable strategy of differentiated granularity, automatically matching the corresponding granularity in the three-level semantic structure.

[0058] Secondly, the present invention provides a dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval device, including...

[0059] The first processing unit is used to collect multimodal data and perform preprocessing. It maps the preprocessed multimodal data into 256-dimensional original feature vectors through a large model cross-modal attention mechanism. The original feature vectors are then validated by a dual model to obtain a semantically accurate feature vector set, and the effective feature vectors are output.

[0060] The second processing unit is used to calculate the semantic similarity of multimodal data, the domain association strength with temporal decay, and the long-range association degree derived from GAT multi-hop based on the effective feature vector output by the first processing unit, derive the three-dimensional dynamic edge weight Em, construct a dynamic knowledge graph, and output the association relationship between edge weight and graph.

[0061] The third processing unit is used to cluster the effective feature vectors output by the first processing unit and the edge weights and graph association relationships output by the second processing unit using a Gaussian mixture model (GMM), and then perform EM iterative optimization and feature correction. It also combines a graph attention network to evaluate the importance of semantic blocks, generating a three-level structure of "coarse-grained semantic clusters - medium-grained sub-semantic blocks - fine-grained feature units".

[0062] The fourth processing unit is used to quantify the outputs of the first, second, and third processing units through a complex global and local dual-dimensional metric. The gating function adapts the retrieval granularity, and combined with Em score sorting and timeliness filtering, finally outputs scenario-based and structured retrieval results.

[0063] Thirdly, the present invention provides an electronic device, including a processor and a memory communicatively connected to the processor and used to store executable instructions of the processor, wherein the processor is used to execute the above-described dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method.

[0064] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method.

[0065] Compared with existing technologies, the dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method described in this invention has the following advantages:

[0066] This invention achieves 256-dimensional unified semantic feature mapping of multimodal data through a large-model cross-modal attention mechanism. It combines three-dimensional dynamic edge weight modeling of semantic similarity, resource association strength, and long-range semantic association with a temporal decay factor. This not only breaks down the semantic barriers of heterogeneous data but also accurately captures long-range implicit associations across documents. At the same time, it adapts to changes in resource timeliness and avoids interference from outdated information, thus solving the problems of modal fragmentation in multimodal data and the static rigidity of edge weights in traditional knowledge graphs.

[0067] This invention generates a three-level semantic structure of "coarse-medium-fine" by Gaussian mixture model (GMM) clustering and EM iterative optimization. Then, based on the dual-dimensional quantization of global scene complexity and local semantic complexity, it dynamically outputs the fine-grained retrieval ratio through a gating function. This effectively balances the contradiction between the tendency of coarse-grained retrieval to miss details and the excessive computational consumption of fine-grained retrieval, improves the flexibility of scene adaptation, and solves the problems of fixed retrieval granularity and lack of quantitative standards for scene adaptation.

[0068] This invention ensures the accuracy of core semantic elements through dual-model + knowledge graph dual-layer verification, adapts GMM customized parameters to multimodal data characteristics, and enhances semantic block importance assessment through graph attention network. This not only improves the accuracy of cross-modal semantic matching, but also ensures the logic and completeness of retrieval results, while significantly reducing the consumption of ineffective computing power. It solves the problems of GMM parameter generalization, blind gating adjustment, and insufficient semantic extraction accuracy in the prior art. Attached Figure Description

[0069] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0070] Figure 1 This is a schematic diagram of the dynamic granularity retrieval method for dynamic graph multimodal Gaussian adaptive clustering according to the present invention. Detailed Implementation

[0071] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0072] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0073] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0074] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0075] Example 1

[0076] This invention provides a dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method, the overall process of which is as follows: Figure 1 As shown, it includes the following steps:

[0077] Step 1: Collect multimodal data and preprocess it. Map the preprocessed multimodal data into 256-dimensional original feature vectors through a large model cross-modal attention mechanism. Validate the original feature vectors with dual models to obtain a semantically accurate feature vector set and output the effective feature vectors.

[0078] This embodiment takes academic literature retrieval as an example. First, academic literature data is input, and multimodal raw data such as text, images, and audio are collected to clarify the retrieval scenario and confirm whether real-time response is required.

[0079] For different types of data, this invention employs a differentiated preprocessing strategy and a unified feature mapping:

[0080] (1) Text data: Word embeddings and cross-segment contextual semantic features are extracted through the long-range attention mechanism built into the large model;

[0081] (2) Image data: Text information is extracted by OCR transcription, while retaining the target detection box, visual feature vector and gradient change features;

[0082] (3) Audio data: The audio data is converted into text by speech recognition, and the spectral features and key speech segment markers are extracted simultaneously.

[0083] (4) Tabular data: parsed into structured text and numerical features.

[0084] Deploy model tools: Deploy any two multimodal large models on different bases (such as CLIP-LLaMA, MiniGPT-4, etc.), configure GMM (Gaussian Mixture Model) clustering tools, and use commercially available network neural network frameworks. Here, we take the GAT graph neural network framework as an example. Taking "graph structure data" as the processing object, we adaptively learn the association weights between nodes and their neighboring nodes through the attention mechanism, and then aggregate the semantic features of multi-hop neighbors to achieve accurate capture of node importance and long-range associations.

[0085] Core semantic elements are the "structured content essence" of multimodal data, fundamentally used to express the core content of concepts. All modal data are first split into independent semantic units based on semantic integrity, with each semantic unit containing several core semantic elements. Then, through a large-scale model cross-modal attention mechanism, the core semantic elements of different modalities are uniformly mapped to the same feature space, transforming them into the original feature vector fi after a 256-dimensional unified semantic cross-modal mapping. 0 The 256-dimensional design balances semantic integrity and computational efficiency, ensuring that semantic elements of different modalities and semantics can be quantitatively compared through vector similarity, making heterogeneous modal data comparable in the same feature space, while also fully preserving long-range semantic association clues.

[0086] Then, dual-model verification is performed. This invention uses CLIP-LLaMA and MiniGPT-4 as examples.

[0087] Calculate the overlap rate for core semantic elements.

[0088] CLIP-LLaMA and MiniGPT-4 were used in parallel to extract core semantic elements from academic documents, resulting in element sets A and B. Taking an academic document retrieval scenario as an example, element A includes text elements, image elements, table elements, and audio elements. Element B has the same structure as A, although there may be differences in expression. Elements with consistent core semantics are included in the intersection element overlap rate. Core semantic elements A and B represent the "core semantic extraction" of multimodal data by two large multimodal models. By extracting structured information such as key entities, attributes, logical relationships, and numerical features from each modality, a set of elements without redundancy and comparable is formed. Consistency is subsequently verified by calculating the overlap rate (MatchRate). Essentially, this ensures that "the core semantics of multimodal data are not affected by the bias of a single model," providing a foundation for the accurate mapping of the subsequent 256-dimensional feature vectors. The formula for calculating core semantic elements is:

[0089] ElementMatchRate=(A∩B) / (A∪B)

[0090] When ElementMatchRate ≥ 0.95, the intersection result is used; when 0.8 ≤ ElementMatchRate < 0.95, the disputed elements are cross-validated by the dual model; if ElementMatchRate < 0.8, manual review is triggered.

[0091] Calculate the overlap rate for semantic feature vectors

[0092] The obtained features are corrected and their effectiveness verified. First, two multimodal large models are fine-tuned in a targeted manner. Based on the multimodal data distribution of the retrieval scenario and the core rules of the domain knowledge graph, the input layer and semantic encoding layer are optimized, the preset modality weights are adapted, and the output target of the 256-dimensional feature vector is focused to reduce the domain semantic bias of the general model. Then, the fine-tuned dual models are respectively based on the initial feature vector f. i 0 The feature sets FA and FB are extracted, and the feature overlap rate is calculated using the following formula:

[0093] FeatureMatchRate=Σ[I(cos(fAi,fBi)≥0.85)] / N

[0094] Where cos(f) Ai f Bi ) is the eigenvector f Ai f Bi The cosine similarity is used to quantify the consistency between two models in extracting feature vectors of the same semantic unit. The calculation formula is:

[0095]

[0096] fAi ·f Bi It is the eigenvector f Ai with f Bi dot product; ||f Ai ||、||f Bi || are the eigenvectors f respectively Ai with f Bi The L2 norm (modulus); output range: [0,1], the closer the value is to 1, the higher the semantic consistency between the two feature vectors.

[0097] When FeatureMatchRate ≥ 0.95, the average feature is used; when 0.8 ≤ FeatureMatchRate < 0.95, the overlap rate is recalculated after the features are extracted twice by the fine-tuned dual model; when FeatureMatchRate < 0.8, the features are extracted again; the feature vector is normalized to the [0,1] interval, and the average vector μ with all cluster centers is calculated. avg The cosine similarity is used as the ValidScore. The ValidScore calculation formula is as follows:

[0098]

[0099] Among them, f i The result is a 256-dimensional unified semantic feature vector after dual-model validation and normalization to the [0,1] interval; μ avg For all cluster centers μ1, μ2, ..., μ after GMM clustering K The average vector, where k is the total number of clusters:

[0100]

[0101] f i ·μ avg For the dot product of two vectors, ||f i ‖、‖μ avg ‖ represents the L2 norm of the two vectors, with an output range of [0,1]. If ValidScore < 0.5, features are re-extracted.

[0102] Finally, the output is an array of feature vectors filtered by overlap rate.

[0103] Step 2: Based on the effective feature vectors output in Step 1, calculate the semantic similarity of multimodal data, the domain association strength with temporal decay, and the long-range association derived from GAT multi-hop, derive the three-dimensional dynamic edge weight Em, construct a dynamic knowledge graph, and output the association relationship between edge weights and the graph.

[0104] Based on the 256-dimensional unified semantic feature vector obtained after dual-model verification in the previous step, in order to accurately capture the direct semantic associations, domain-adaptive associations, and long-range implicit associations of multimodal data, and to adapt to changes in resource timeliness, a three-dimensional weighted collaborative edge weight calculation formula is designed as follows:

[0105] Em=α×Sim(Sm,Sn)+β×Rel(Rm,Rn)+(1-α-β)×LongSim(Lm,Ln)

[0106] Em, as the comprehensive edge weight, is a weighted fusion result of the three dimensions of similarity. Its value directly determines the association priority of two semantic units in the dynamic graph, supporting the association logic judgment and result ranking in the retrieval process. α and β are core adjustment coefficients, satisfying α+β≤1, used to adapt to the needs of different retrieval scenarios. For precise retrieval, α=0.6 and β=0.3, focusing on direct semantic matching; for generalized recommendation, α=0.4 and β=0.4, balancing direct similarity and domain adaptation; for long-range association retrieval, α=0.3 and β=0.2, highlighting the capture of long-range implicit associations. The default initial values ​​for α and β are 0.5 and 0.3, respectively.

[0107] Sim(Sm, Sn) represents the cosine similarity of the feature vectors. The input is a 256-dimensional unified semantic feature vector (f_m) obtained directly from multimodal data after "differential preprocessing + dual-model" processing. m f n This vector directly carries the core semantics of the data's surface layer, capturing the direct semantic relationship between Sm and Sn. The calculation formula is as follows:

[0108]

[0109] f m f n : These are the 256-dimensional unified semantic feature vectors (f) obtained from data Sm and Sn after "multimodal feature extraction and verification". i (Specific examples); f m ·f n The dot product of two vectors; ||f m ||、||f n ||: L2 norm of the two vectors respectively; Output range: [0,1], the closer the value is to 1, the higher the direct semantic similarity.

[0110] Rel(Rm, Rn) is calculated by weighting factors such as domain relevance, usage frequency, and scenario adaptability. The calculation formula is as follows:

[0111] Rel(R m ,R n )=τ×(w1·Corr+w2·Freq+w3·Adapt)

[0112] τ: Time decay factor: Resources not overdue for ≤1 year: τ=1.0; Overdue for 1 year or more: τ decreases by 0.1 each year, with a minimum of τ=0.3, to avoid interference from outdated information;

[0113] w1, w2, w3: Weighting coefficients, satisfying w1+w2+w3=1, which can be fine-tuned according to the domain scenario, and the default is 1 / 3; Corr: Domain relevance score [0,1]: The semantic relevance between Rm and Rn based on the domain knowledge graph query, such as the relevance score between "lung nodules" and "lung cancer diagnosis" is 0.9;

[0114] Freq: Frequency Normalization Score [0,1]: The normalized value is taken after counting the frequency of Rm and Rn appearing together in historical searches and applications.

[0115] Adapt: ​​Scene adaptability score [0,1]: Determine the degree of adaptability between Rm and Rn based on the current search scenario, such as academic search or medical diagnosis;

[0116] Output range: [0.3, 1.0], the closer the value is to 1, the stronger the domain association and the stronger the timeliness. LongSim(Lm, Ln) is the cosine similarity of the feature vectors, and the input is the 256-dimensional effective feature vector f obtained in step one. i The derived feature vector generated after three layers of multi-hop propagation in the GAT graph neural network This vector integrates the semantic information of the target semantic unit and its multi-hop neighbor nodes. It is a feature representation enhanced by long-range association, focusing on capturing long-range implicit associations across documents and paragraphs. The calculation formula is as follows:

[0117]

[0118] The final feature vectors of the nodes (semantic blocks) corresponding to Sm and Sn after GAT multi-hop propagation are calculated iteratively based on the GAT layer propagation rules.

[0119]

[0120] Number of propagation layers L = 3 (preset parameter of the scheme), α mn W represents the attention weights for GAT's automatic learning. (l) These are the learnable parameters for the l-th layer;

[0121] The other symbols have the same meaning as Sim(Sm,Sn), and the output range is [0,1]. The closer the value is to 1, the stronger the long-range hidden association.

[0122] Rel(Rm, Rn) is calculated by weighting domain relevance, usage frequency, and scenario suitability, while LongSim(Lm, Ln) is calculated using the GAT multi-hop propagation results. For precise retrieval, α = 0.6 and β = 0.3; for generalized recommendation, α = 0.4 and β = 0.4; and for long-range association retrieval, α = 0.3 and β = 0.2.

[0123] A time-decrease factor τ is introduced, and the Rel(Rm, Rn) of resources that have not been updated for more than one year decreases by 0.1 annually (minimum 0.3). When a resource undergoes version iteration, content update, or attribute change, the edge weights will be automatically recalculated. Updated semantic features, resource attributes, and long-range association clues will be taken into consideration, thus avoiding outdated information from interfering with the matching results.

[0124] Step 3: The effective feature vectors output from Step 1 and the edge weights and graph associations output from Step 2 are clustered using a Gaussian Mixture Model (GMM) and then iteratively optimized and corrected using EM. The importance of semantic blocks is evaluated using a graph attention network, generating a three-level structure of "coarse-grained semantic clusters - medium-grained sub-semantic blocks - fine-grained feature units".

[0125] To address the pain points of fixed retrieval granularity, poor adaptability to multimodal data clustering, and imbalance between computational power consumption and retrieval accuracy, this invention first processes the 256-dimensional effective feature vector f output from step 1. i Preprocessing is performed to purify the clustering input data, filtering out duplicate features with a cosine similarity ≥ 0.98 to avoid data redundancy leading to cluster duplication, abnormal mode transformation data, and irrelevant semantic information, ensuring that the feature vectors input to the GMM are pure and effective.

[0126] Then, customized training of GMM parameters was initiated to establish the mathematical foundation for clustering: GMM assumes that the set of multimodal feature vectors consists of a mixture of K multivariate Gaussian distributions, and the probability density function is calculated as follows:

[0127]

[0128] in is the abbreviation for multivariate normal distribution, also known as multivariate Gaussian distribution. GMM assumes that the set of multimodal eigenvectors consists of a mixture of K multivariate Gaussian distributions. Therefore, N(f i |μ k ,Σ k ) is used to calculate the i-th eigenvector f i The probability density value belonging to the k-th semantic cluster, i.e., the feature vector f. i With the core features of the k-th cluster, the cluster center μ k Distribution range Σ k The degree of matching. f iThe input is a 256-dimensional unified semantic feature vector after dual-model validation; μ k It is the cluster center of the k-th semantic cluster; Σ k Θ is the covariance matrix of the k-th semantic cluster, describing the feature dispersion within the cluster. Θ is the set of global parameters of the GMM, containing μ. k π k ,∑ k Cluster center μ k Defined as a "multimodal core semantic prototype," it integrates typical feature expressions from different modalities, ensuring that each semantic cluster corresponds to a complete logical unit. The calculation formula is as follows:

[0129]

[0130] Where C k Let γ be the set of feature vectors of the k-th semantic cluster. ik Let be the posterior probability that the i-th feature vector belongs to the k-th cluster.

[0131] Mixed weight π k Binding the importance levels of multimodal information enables data-driven automatic learning; the calculation formula is as follows:

[0132]

[0133] N is the total number of feature vectors, W modality The modal weights are: (core semantic text block w = 0.4, key image feature block w = 0.3, auxiliary audio block w = 0.2, redundant information block w = 0.1). The model can adaptively adjust the weights according to the scenario. For example, in business retrieval, the weight of the "financial table block" will automatically increase to 0.3.

[0134] The covariance matrix Σk is used to describe the feature dispersion within a cluster, focusing on the range of feature fluctuations with the same semantic logic. Its calculation formula is:

[0135]

[0136] This invention provides customized training for GMM. Taking academic literature retrieval as an example, it first collects multimodal raw data of academic literature, including text, images, audio, and tables. After differential preprocessing and unified mapping in the previous step, the data is converted into a 256-dimensional unified semantic feature vector. After dual-model verification, the final pure 256-dimensional semantic feature vector set is used as the training input data for GMM.

[0137] During training, K-Means pre-clustering is first performed to obtain the initial cluster centers. Simultaneously, the initial mixing weight K is set to the number of clusters obtained from pre-clustering, and the calculation method is as follows:

[0138]

[0139] Using the initial covariance matrix As the covariance of samples within each cluster, the EM iterative optimization process is then initiated. The E-step calculates the posterior probability that the i-th feature vector belongs to the k-th cluster based on the multivariate Gaussian distribution probability density function.

[0140]

[0141] The M-step updates the mixed weights π based on the posterior probability. k Cluster center μ k The sum of the covariance matrix ∑ k The above iterative process is repeated 50 to 100 times until convergence stops when the parameter change is less than 1e-5. Before training, duplicate features with cosine similarity ≥ 0.98, abnormal mode transition data, and irrelevant semantic information need to be filtered out to ensure clustering quality.

[0142] μ j This is the core feature vector of the j-th semantic cluster, which integrates typical feature expressions of multimodal data within the cluster, such as text, images, audio, and tables. It is a condensed representation of the "core semantic elements" of the cluster. It ensures that each semantic cluster corresponds to a complete logical unit, used to distinguish the core differences between different semantic clusters.

[0143] Σ j This describes the dispersion of all feature vectors within the j-th semantic cluster, while also characterizing the correlation between different dimensions of the feature vectors. It focuses on the fluctuation range of features under the same semantic logic, avoiding clustering confusion caused by modal differences or subtle fluctuations in features within the cluster.

[0144] π j It represents the "weight proportion" of the j-th semantic cluster in the entire GMM model, which binds the importance level of the multimodal information corresponding to that cluster, and satisfies... K represents the total number of clusters. It reflects the contribution of this semantic cluster to the overall multimodal data distribution, such as the π of the core semantic cluster. j Higher values, π of redundant information clusters j The value is even lower.

[0145] After clustering is complete, a feature correction mechanism will be activated, and F will be extracted from the two finely tuned multimodal large models respectively. A With F B The calculation method is as follows:

[0146]

[0147] N is the total number of feature vectors. For the indicator function, cross-validation is performed, and different schemes are adopted according to different situations:

[0148] When MatchRate ≥ 0.8, the average feature is used; when MatchRate < 0.8, features are re-extracted. Simultaneously, all feature vectors are uniformly mapped to 256 dimensions and normalized to the [0,1] interval, and the average vector μ of all cluster centers is calculated. avg The cosine similarity is used as the ValidScore. If the ValidScore is less than 0.5, the feature extraction needs to be repeated to ensure the validity of the feature.

[0149] Coarse-grained semantic block B output by GMM k For nodes, initial node characteristics:

[0150]

[0151] s i The traditional importance score for position i is given, such as the attention score. Edges are established based on semantic similarity ≥ 0.7 or k-nearest neighbor relationships, with edge weights equal to the corresponding semantic similarity. Then, multi-hop propagation is performed through an L-layer graph attention network, with the propagation rule as follows:

[0152]

[0153] α mn Attention weights are ultimately determined by the final features of the nodes. As the total importance score of semantic blocks S(B) m The top-N core semantic blocks (N = K × 0.6) are retained in descending order of score and rounded down.

[0154] Based on this, a three-level structured granularity is generated. The coarse granularity consists of K semantic clusters obtained after GMM convergence. For small-scale data, K = 5 to 8, and for large-scale data, K = 15 to 20. The medium granularity involves splitting each coarse-grained semantic cluster into sub-semantic blocks according to the core semantic element category. For example, the "project plan cluster" is split into technical parameters and schedule planning sub-blocks. The fine granularity consists of key feature units within the sub-semantic blocks, such as specific performance indicators and numerical ranges, laying the foundation for subsequent adaptive retrieval.

[0155] Step 4: The outputs of Steps 1, 2, and 3 are quantified using a complex global and local dual-dimensional metric. The granularity of the retrieval is adapted by a gating function, and combined with Em score sorting and timeliness filtering, the final output is a scenario-based and structured retrieval result.

[0156] Based on the three-level semantic structure of "coarse-grained semantic cluster - medium-grained sub-semantic block - fine-grained feature unit" generated in step 3, and the three-dimensional dynamic edge weight Em and long-range association information constructed in step 2, this invention achieves dynamic granularity retrieval through two-dimensional semantic complexity quantification. The specific process is as follows:

[0157] First, semantic complexity is comprehensively represented through both global and local dimensions to provide a quantitative basis for granular decision-making:

[0158] Global complexity formula:

[0159] Complexity = 0.3 × Modal Score + 0.5 × Element Score + 0.2 × Response Score

[0160] One point is awarded for each modal type, one point for each core element, and one point for real-time response.

[0161] Local semantic complexity is achieved by calculating the attention score entropy and the feature gradient magnitude. The formula for the attention score entropy is:

[0162]

[0163] H(C k ) represents the attention score entropy of the k-th coarse-grained semantic cluster, indicating the degree of dispersion of local semantic distribution within the cluster (the higher the entropy value, the higher the complexity); C k The k-th semantic cluster generated by GMM clustering contains feature vectors corresponding to several semantic units; p i Normalized posterior probability of i:

[0164]

[0165] γ ik Let be the posterior probability of an eigenvector belonging to the k-th cluster, satisfying lnp i For p i The natural logarithm operation.

[0166] An entropy value ≥ 1.2 indicates high complexity, ≤ 0.8 indicates low complexity, and dramatic changes in feature gradients indicate rich data details.

[0167] Based on the two-dimensional complex quantification results, the formula for calculating the gate function is as follows:

[0168] ρ(x)=σ(W·[Complexity,Entropy,Gradient]+b)

[0169] Entropy is the attention score entropy, which is the k-th coarse-grained semantic cluster C. k Entropy is a quantitative index of the dispersion of semantic distribution within a cluster, reflecting the consistency of semantic units within the cluster. A higher entropy value indicates more dispersed semantics, richer details, and higher local complexity within the cluster; a lower entropy value indicates more concentrated semantics, simpler information, and lower local complexity. The normalized posterior probability p of the feature vectors after GMM clustering is also considered. i The calculation formula is as follows:

[0170]

[0171] C k The k-th coarse-grained semantic cluster generated for GMM; p i For the i-th eigenvector to belong to C k The normalized posterior probability; ln(p i ): For p i The natural logarithm operation.

[0172] Gradient is the feature gradient magnitude, which is a quantitative indicator of the "degree of drastic change in numerical value" of a 256-dimensional unified semantic feature vector. It reflects the richness of detail in the feature vector within a cluster. The higher the gradient magnitude, the more drastic the change in feature value, the more detailed information the data contains (such as the edge features of lesions in medical images, the density of key terms in text), and the higher the local complexity; conversely, the less detail and the lower the complexity.

[0173] The formula is:

[0174]

[0175] For the i-th 256-dimensional semantic feature vector f i The partial derivatives of f are used to characterize f i (Rate of change of the numerical value relative to the cluster center); For the eigenvector f i To its k-th semantic cluster center μ k The "half of the squared Mahalanobis distance" is suitable for the distribution characteristics of GMM clustering; (f i -μ k ) is the deviation vector between the eigenvector and the cluster center; Σ is the covariance matrix of the k-th semantic cluster. k The inverse matrix; ||·||2 is the L2 norm (Euclidean norm), which compresses the 256-dimensional gradient vector obtained from the partial derivatives into a scalar magnitude.

[0176] Outputs the fine-grained retrieval ratio. W and b are learnable references. σ is the sigmoid function, outputting the fine-grained retrieval ratio ρ(x)∈(0,1). It adapts to different complexity scenarios with a switchable strategy for differentiated granularity, automatically matching the corresponding granularity in the three-level semantic structure without manual intervention, but supports manual fine-tuning.

[0177] (1) High complexity scenario (global score ≥ 7 points or local entropy ≥ 1.2 + high gradient magnitude): when ρ(x) ≥ 0.8, match "fine-grained feature unit";

[0178] (2) Medium complexity scenario (4≤global score<7 or local entropy 0.8-1.2): When ρ(x)=0.4-0.7, match “medium granular sub-semantic block + fixed window”;

[0179] (3) Low complexity scenario (global score ≤ 3 points or local entropy ≤ 0.8): When ρ(x) ≤ 0.3, it matches “coarse-grained semantic cluster subset”.

[0180] When the fine-grained retrieval ratio is ρ(x)≥0.8, it targets high-complexity scenario retrieval. The core objective is to accurately mine data details and long-term correlations, which is suitable for scenarios with extremely high retrieval accuracy requirements, such as academic research and medical diagnosis. The core of this scenario relies on the "fine-grained feature unit library" generated in step 3 and the LongSim long-range association information constructed in step 2. In specific implementation, firstly, initial candidate units with semantic similarity Sim≥0.7 and ValidScore≥0.6 with the query are selected from the fine-grained feature unit library. Then, these candidate units are used as the core to associate adjacent units that satisfy "maximum of si+γSim(fi,fj)" and long-range association units with GAT multi-hop propagation, forming a local semantic subgraph. Subsequently, the matching score between the unit and the query is calculated using the dynamic edge weight formula Em, where α=0.3 and β=0.2 to focus on capturing long-range associations. The Top-20 results are retained in descending order of Em score, while outdated units with Rel≤0.3 after time decay are filtered out. Finally, a structured fine-grained result set is output, which includes unit semantic description, modality type, Em score, long-range association path and confidence description.

[0181] When the fine-grained retrieval ratio ρ(x) is between 0.4 and 0.7, it corresponds to medium-complexity scenario retrieval. The core objective is to balance retrieval accuracy and efficiency, suitable for common application scenarios such as product retrieval and routine data analysis. This scenario mainly relies on the "medium-grained sub-semantic blocks" generated in step 3, and the Sim direct similarity and Rel domain fit in step 2. The implementation process is as follows: From the medium-grained sub-semantic blocks, sub-blocks with an ElementMatchRate ≥ 0.8 overlap rate with the core semantic elements of the query are selected. For each sub-block, 1-2 fixed windows containing Top-N core units are extracted, with a window size of 3-5 units. Then, the Em score is calculated on a window-by-window basis, where α = 0.4 and β = 0.4 to balance direct semantic matching and domain fit. At the same time, invalid windows with an average ValidScore < 0.6 are filtered out. The Top-15 results are retained in descending order of Em score, and a semi-structured windowed result set is output, covering sub-block classification, window summary, Em score, key parameter annotation, and efficiency optimization explanation.

[0182] When the fine-grained retrieval ratio ρ(x) ≤ 0.3, low-complexity scenario retrieval is triggered. The core objective is to quickly output core related information, which is suitable for scenarios with higher requirements for retrieval speed, such as generalized recommendation and fast browsing. Its core reliance is the direct similarity between the "coarse-grained semantic cluster subset" generated in step 3 and Sim in step 2. The specific steps are as follows: from the coarse-grained semantic cluster, core subclusters with a semantic similarity Sim ≥ 0.6 with the query, a domain relevance Rel ≥ 0.7, and a core semantic element overlap rate ElementMatchRate ≥ 0.85 verified by the knowledge graph are selected. The Top-8 are retained to avoid redundancy. Then, the Em score is calculated on a subcluster basis. At this time, α = 0.6 and β = 0.3 to focus on direct semantic similarity matching. After sorting in descending order of Em score, outdated subclusters with Rel ≤ 0.4 after time decay are removed. Finally, a simplified core summary result set is output, which includes subcluster semantic summary, Em score, modality coverage, and manual fine-tuning entry.

[0183] In summary, this invention effectively improves the accuracy of cross-modal semantic matching, the accuracy of capturing long-range implicit associations, and the retrieval efficiency. It not only significantly reduces computing power consumption and enhances the flexibility of scenario adaptation, but also ensures the logic and completeness of retrieval results.

[0184] Example 2

[0185] A dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval device, comprising:

[0186] The first processing unit is used to collect multimodal data and perform preprocessing. It maps the preprocessed multimodal data into 256-dimensional original feature vectors through a large model cross-modal attention mechanism. The original feature vectors are then validated by a dual model to obtain a semantically accurate feature vector set, and the effective feature vectors are output.

[0187] The second processing unit is used to calculate the semantic similarity of multimodal data, the domain association strength with temporal decay, and the long-range association degree derived from GAT multi-hop based on the effective feature vector output by the first processing unit, derive the three-dimensional dynamic edge weight Em, construct a dynamic knowledge graph, and output the association relationship between edge weight and graph.

[0188] The third processing unit is used to cluster the effective feature vectors output by the first processing unit and the edge weights and graph association relationships output by the second processing unit using a Gaussian mixture model (GMM), and then perform EM iterative optimization and feature correction. It also combines a graph attention network to evaluate the importance of semantic blocks, generating a three-level structure of "coarse-grained semantic clusters - medium-grained sub-semantic blocks - fine-grained feature units".

[0189] The fourth processing unit is used to quantify the outputs of the first, second, and third processing units through a complex global and local dual-dimensional metric. The gating function adapts the retrieval granularity, and combined with Em score sorting and timeliness filtering, finally outputs scenario-based and structured retrieval results.

[0190] Example 3

[0191] An electronic device includes a processor and a memory communicatively connected to the processor and used to store processor-executable instructions, the processor being used to execute the aforementioned dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method.

[0192] Example 4

[0193] A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, provides the aforementioned dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method.

[0194] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0195] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy and procedures.

[0196] This disclosure is intended to provide implementation schemes for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0197] The acquisition, transmission, storage, use, and processing of data in this disclosed technical solution all comply with the relevant provisions of national laws and regulations.

[0198] It should be noted that in the embodiments disclosed herein, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary and are intended only to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used such solutions.

[0199] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0200] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0201] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0202] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0203] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0204] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0205] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0206] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

[0207] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic spectral multimodal Gaussian adaptive clustering dynamic granularity retrieval method, characterized in that: Includes the following steps: Step 1: Collect multimodal data and preprocess it. Map the preprocessed multimodal data into 256-dimensional original feature vectors through a large model cross-modal attention mechanism. Validate the original feature vectors with dual models to obtain a semantically accurate feature vector set and output the effective feature vectors. Step 2: Based on the effective feature vectors output in Step 1, calculate the semantic similarity of multimodal data, the domain association strength with temporal decay, and the long-range association strength derived from GAT multi-hop, derive the three-dimensional dynamic edge weight Em, construct a dynamic knowledge graph, and output the association relationship between edge weights and the graph. Step 3: The effective feature vectors output from Step 1 and the edge weights and graph associations output from Step 2 are clustered using a Gaussian Mixture Model (GMM) and then iteratively optimized and corrected using EM. The importance of semantic blocks is evaluated using a graph attention network, generating a three-level structure of "coarse-grained semantic clusters - medium-grained sub-semantic blocks - fine-grained feature units". Step 4: The outputs of Steps 1, 2, and 3 are quantified using a complex global and local dual-dimensional metric. The granularity of the retrieval is adapted by a gating function, and combined with Em score sorting and timeliness filtering, the final output is a scenario-based and structured retrieval result.

2. The dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method according to claim 1, characterized in that: In step 1, the multimodal data includes text data, image data, audio data, and tabular data. Preprocessing the multimodal data includes: Text data: Word embeddings and cross-segment contextual semantic features are extracted using the long-range attention mechanism built into the large model; Image data: Text information is extracted by OCR transcription, while object detection boxes, visual feature vectors and gradient change features are preserved; Audio data: converted into text through speech recognition, with spectral features and key speech segment markers extracted simultaneously; Tabular data: parsed into structured text and numerical features.

3. The dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method according to claim 1, characterized in that: In step 1, the preprocessed multimodal data is mapped to a 256-dimensional original feature vector through a large-model cross-modal attention mechanism. The original feature vector is then validated using a dual-model approach to obtain a semantically accurate feature vector set. The output effective feature vectors include: Multimodal data is split into independent semantic units based on semantic integrity. Each semantic unit contains several core semantic elements. Through the cross-modal attention mechanism of the large model, the core semantic elements of different modalities are uniformly mapped to the same feature space and transformed into the original feature vector after 256-dimensional unified semantic cross-modal mapping. The overlap rate of core semantic elements and the overlap rate of feature vectors of core semantic elements are verified by using a dual model, and the effective feature vectors after the overlap rate is filtered are output.

4. The dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method according to claim 1, characterized in that: In step 2, a three-dimensional weighted collaborative edge weight calculation formula is designed, which considers semantic similarity, domain association strength with temporal decay, and long-range association derived from GAT multi-hop. Em=α×Sim(Sm,Sn)+β×Rel(Rm,Rn)+(1-α-β)×LongSim(Lm,Ln) Em is the comprehensive edge weight, which is the weighted fusion result of the three dimensions of similarity. α and β are the core adjustment coefficients, satisfying α+β≤1. Sim(Sm, Sn) is the semantic similarity, Rel(Rm, Rn) is the domain association strength, and LongSim(Lm, Ln) is the long-range association. f m f n These are the 256-dimensional unified semantic feature vectors obtained after preprocessing data Sm and Sn, respectively; ||f m ||、||f n || represents the L2 norm of the two vectors; Rel(R m ,R n )=τ×(w1·Corr+w2·Freq+w3·Adapt) τ is the time decay factor, w1, w2, and w3 are weighting coefficients that satisfy w1+w2+w3=1, Corr is the domain relevance score, Freq is the usage frequency normalization score, and Adapt is the scene adaptation score. The final feature vectors of the nodes corresponding to data Sm and Sn after multi-hop propagation by GAT are calculated iteratively based on the propagation rules of the GAT layer: L is the number of propagation layers, α mn W represents the attention weights for GAT's automatic learning. (l) These are the learnable parameters for the l-th layer.

5. The dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method according to claim 1, characterized in that: Step 3 specifically includes: Use the effective feature vectors obtained in step 1 as the training input data for the Gaussian Mixture Model (GMM). During training, K-Means pre-clustering is first performed to obtain the initial cluster centers. Simultaneously, the initial mixing weight K is set to the number of clusters obtained from pre-clustering, and the calculation method is as follows: Using the initial covariance matrix As the covariance of samples within each cluster, the EM iterative optimization process is then initiated. The E-step calculates the posterior probability that the i-th feature vector belongs to the k-th cluster based on the multivariate Gaussian distribution probability density function. The M-step updates the mixed weights π based on the posterior probability. k Cluster center μ k The sum of the covariance matrix ∑ k Repeat the above iterative process until convergence stops when the parameter change is less than 1e-5; The feature correction mechanism will be activated after clustering is completed; Combining graph attention networks to evaluate the importance of semantic blocks, and using coarse-grained semantic block B output by a Gaussian mixture model (GMM) k For nodes, initial node characteristics: s i For the traditional importance score at position i, edges are constructed based on semantic similarity or k-nearest neighbor relationships, with edge weights equal to the corresponding semantic similarity. These edges are then propagated through an L-layer graph attention network for multi-hop propagation, with the following propagation rules: α mn Attention weights are ultimately determined by the final features of the nodes. As the total importance score of semantic blocks S(B) m The top-N core semantic blocks are retained in descending order of their scores. Based on this, a three-level structured granularity is generated. The coarse granularity consists of K semantic clusters obtained after GMM convergence. The medium granularity is to split each coarse granular semantic cluster into sub-semantic blocks according to the core semantic element category. The fine granularity consists of the key feature units within the sub-semantic blocks.

6. The dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval method according to claim 1, characterized in that: Step 4 specifically includes: First, the semantic complexity is comprehensively represented using both global and local dimensions. The formula for the global complexity is: Complexity = 0.3 × Modal Score + 0.5 × Element Score + 0.2 × Response Score Local semantic complexity is achieved by calculating the attention score entropy and the feature gradient magnitude. The formula for the attention score entropy is: H(C k C is the attention score entropy of the k-th coarse-grained semantic cluster; k The k-th semantic cluster generated by GMM clustering contains feature vectors corresponding to several semantic units; p i For the normalized posterior probability of i: γ ik Let be the posterior probability of an eigenvector belonging to the k-th cluster, satisfying ln p i For p i The natural logarithm operation; Based on the two-dimensional complex quantification results, the formula for calculating the gate function is as follows: ρ(x)=σ(W·[Complexity,Entropy,Gradient]+b) W and b are learnable references, σ is the Sigmoid function, outputs the fine-grained retrieval ratio ρ(x)∈(0,1), and adapts to different complexity scenarios with a switchable strategy of differentiated granularity, automatically matching the corresponding granularity in the three-level semantic structure.

7. A dynamic graph multimodal Gaussian adaptive clustering dynamic granularity retrieval device, characterized in that: include The first processing unit is used to collect multimodal data and perform preprocessing. It maps the preprocessed multimodal data into 256-dimensional original feature vectors through a large model cross-modal attention mechanism. The original feature vectors are then validated by a dual model to obtain a semantically accurate feature vector set, and the effective feature vectors are output. The second processing unit is used to calculate the semantic similarity of multimodal data, the domain association strength with temporal decay, and the long-range association degree derived from GAT multi-hop based on the effective feature vector output by the first processing unit, derive the three-dimensional dynamic edge weight Em, construct a dynamic knowledge graph, and output the association relationship between edge weight and graph. The third processing unit is used to cluster the effective feature vectors output by the first processing unit and the edge weights and graph association relationships output by the second processing unit using a Gaussian mixture model (GMM), and then perform iterative optimization and feature correction using EM. It also combines a graph attention network to evaluate the importance of semantic blocks, generating a three-level structure of "coarse-grained semantic clusters - medium-grained sub-semantic blocks - fine-grained feature units". The fourth processing unit is used to quantify the outputs of the first, second, and third processing units through a complex global and local dual-dimensional metric. The gating function adapts the retrieval granularity, and combined with Em score sorting and timeliness filtering, finally outputs scenario-based and structured retrieval results.

8. An electronic device comprising a processor and a memory communicatively connected to the processor and used for storing processor-executable instructions, characterized in that: The processor is used to execute the method described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

Images