A memory retrieval method for enhancing the multimodal long-context conversation capabilities of large language models

Abstract

The present invention discloses a memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model, including adaptive segmentation of the original multimodal dialogue sequence based on information density and extraction of local features; construction of a three-layer feature pyramid structure to achieve multi-scale feature representation; enhancement of the temporal expression capability of features through feature compression and temporal dependency graph modeling; construction of a memory vector sequence based on feature stability evaluation; structured representation of query data and multi-level interaction with memory vectors; efficient retrieval through hierarchical indexing and multi-dimensional similarity calculation; and regular optimization of memory sequences based on user feedback. The present invention can effectively improve the model's multimodal information understanding and retrieval capabilities in long dialogue scenarios, and has important practical application value.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular to a memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model. Background Art

[0002] In recent years, large language models have made significant progress in the field of multimodal dialogue, demonstrating strong cross-modal understanding and generation capabilities. In actual application scenarios, users often need to have continuous multi-round dialogues with the model. These dialogues may span a long time span and involve information interaction in multiple modalities such as text and images. Improving the model's multimodal information retrieval and understanding capabilities in long dialogue scenarios is of great significance for enhancing the naturalness and coherence of human-computer interaction. Especially in professional fields such as educational counseling, technical support, and medical consultation, users may need to have continuous and in-depth discussions with the model, which requires the model to accurately understand and trace back the key information mentioned above, and organically combine multimodal information to provide coherent and professional responses.

[0003] Current research focuses on several directions: context encoding methods based on attention mechanisms, which capture key information in conversations by calculating attention weights; long-range dependency modeling based on hierarchical memory networks, which stores historical information by building multi-layer memory units; and knowledge integration methods based on retrieval enhancement, which supplement the model's understanding ability through external knowledge retrieval. In terms of multimodal information processing, pre-trained visual-language models are mainly used for feature extraction, and cross-modal information integration is achieved through simple feature splicing or attention fusion. These methods perform well when processing short-term conversations and single-modal information, but they still have many limitations in complex long-conversation scenarios.

[0004] However, existing technologies still face several specific technical challenges: First, when processing long dialogue sequences, simple fixed-length segmentation strategies easily lead to the fragmentation of semantic units, affecting the accuracy of context understanding; second, traditional feature extraction methods often use a single-scale representation, which makes it difficult to capture local details and global semantic information at the same time; third, existing time series modeling methods are mostly based on simple positional encoding or recursive structures, which fail to fully utilize the long-range dependencies in dialogues; in addition, in terms of feature storage, there is a lack of effective memory optimization mechanisms, which easily leads to the accumulation of redundant information and the forgetting of key information; finally, the ranking of retrieval results often only considers the similarity of a single dimension, ignoring the complementarity of multimodal information and contextual coherence. These technical problems seriously restrict the performance of the model in practical applications, especially in professional dialogue scenarios that require long-term memory and deep understanding. Summary of the invention

[0005] The purpose of the invention is to provide a memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model to solve the above-mentioned problems existing in the prior art.

[0006] A technical solution, a memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model, comprises the following steps:

[0007] S1. Receive the original multimodal dialogue sequence and split it according to a preset length to obtain a preprocessed sequence set; extract text and image features based on each subsequence in the preprocessed sequence set, construct a local feature association graph, and obtain a local feature set; perform hierarchical feature fusion on the local feature set to obtain a multi-level feature representation;

[0008] S2. Based on the multi-level feature representation, the importance score of each feature vector is calculated and feature selection is performed to obtain a compressed feature vector; based on the compressed feature vector, a time series dependency graph is constructed and the node representation is updated through a graph neural network to obtain a time series enhancement feature; based on the time series enhancement feature, a stability matrix is ​​calculated; based on the stability matrix, the time series enhancement feature is screened to obtain a memory vector sequence;

[0009] S3, read the multimodal query data input by the user, perform feature extraction, and obtain the original query features; based on the original query features, extract the query intent, build a structure graph, and obtain a structured query representation; interactively calculate the structured query representation and the memory vector sequence to obtain interactive features; perform multi-scale integration and transformation on the interactive features to obtain a fused feature vector;

[0010] S4. Based on the fused feature vector, a multi-layer index structure is constructed and a retrieval candidate set is generated; based on the retrieval candidate set, multi-dimensional similarity is calculated and context consistency is verified to obtain a sorted candidate set; information is filtered and relevance optimized on the sorted candidate set to obtain a retrieval result sequence.

[0011] Beneficial effects: The present invention improves the model's ability to understand the context of long conversations through multi-level feature representation and temporal reinforcement learning; ensures the accuracy and relevance of retrieval results through structured query representation and multi-dimensional similarity calculation; and achieves continuous improvement of system performance through a periodic memory optimization mechanism, with strong scalability and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of the present invention.

[0013] Figure 2 This is a flow chart of step S1 of the present invention.

[0014] Figure 3 This is a flow chart of step S2 of the present invention.

[0015] Figure 4 This is a flow chart of step S3 of the present invention.

[0016] Figure 5 This is a flow chart of step S4 of the present invention. DETAILED DESCRIPTION

[0017] like Figure 1 As shown, the present application proposes a memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model, comprising the following steps:

[0018] S1. Receive the original multimodal dialogue sequence and split it according to a preset length to obtain a preprocessed sequence set; extract text and image features based on each subsequence in the preprocessed sequence set, construct a local feature association graph, and obtain a local feature set; perform hierarchical feature fusion on the local feature set to obtain a multi-level feature representation;

[0019] S2. Based on the multi-level feature representation, the importance score of each feature vector is calculated and feature selection is performed to obtain a compressed feature vector; based on the compressed feature vector, a time series dependency graph is constructed and the node representation is updated through a graph neural network to obtain a time series enhancement feature; based on the time series enhancement feature, a stability matrix is ​​calculated; based on the stability matrix, the time series enhancement feature is screened to obtain a memory vector sequence;

[0020] S3, read the multimodal query data input by the user, perform feature extraction, and obtain the original query features; based on the original query features, extract the query intent, build a structure graph, and obtain a structured query representation; interactively calculate the structured query representation and the memory vector sequence to obtain interactive features; perform multi-scale integration and transformation on the interactive features to obtain a fused feature vector;

[0021] S4. Based on the fused feature vector, a multi-layer index structure is constructed and a retrieval candidate set is generated; based on the retrieval candidate set, multi-dimensional similarity is calculated and context consistency is verified to obtain a sorted candidate set; information is filtered and relevance optimized on the sorted candidate set to obtain a retrieval result sequence.

[0022] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0023] S11, receiving an original multimodal conversation sequence, dividing the original multimodal conversation sequence into a predetermined number of subsequences according to a preset segmentation length threshold, each subsequence containing adjacent text and image data; organizing all the divided subsequences according to time sequence numbers to generate a preprocessing sequence set;

[0024] S12, read each subsequence in the preprocessed sequence set, use a bidirectional encoder to extract word-level feature vectors for the text content in the subsequence; use a visual encoder to extract region-level feature vectors for the image content in the subsequence; calculate the cosine similarity between the word-level feature vectors and the region-level feature vectors as the connection weight between the nodes, and construct a feature association graph; combine the word-level feature vectors and the region-level feature vectors of each subsequence with the corresponding feature association graph to form a local feature set;

[0025] S13. Based on the local feature set, the original feature granularity is maintained as the bottom-level feature; based on the local feature set, the adjacent features are adaptively pooled to obtain the middle-level feature; based on the local feature set, the top-level feature is obtained through the global pooling operation; based on the bottom-level feature, the middle-level feature and the top-level feature, a feature pyramid structure is constructed and a hierarchical multi-level feature representation is generated.

[0026] In one embodiment of the present application, an association graph construction algorithm: association graph weight W(i, j) = δ·S_text(i, j)+(1-δ)·S_img(i, j); wherein S_text(i, j) = BiLSTM(ti)·BiLSTM(tj) / ||BiLSTM(ti)||·||BiLSTM(tj)|| is text feature similarity, and BiLSTM(t) is the output of a bidirectional LSTM encoder; S_img(i, j) = ViT(Ii)·ViT(Ij) / ||ViT(Ii)||·||ViT(Ij)|| is image feature similarity, and ViT(I) is the output of a visual transformer encoder; δ is a modal fusion coefficient, which is dynamically calculated through an attention mechanism; i, j are node indices; a connection is established in the association graph when W(i, j) is greater than a threshold μ.

[0027] This embodiment achieves efficient processing and feature extraction of long dialogue sequences through sequence segmentation of preset lengths and construction of multi-level feature representation. Specifically, adaptive segmentation of the original multimodal dialogue sequence can maintain the semantic integrity of the dialogue fragments and avoid the semantic fragmentation problem that may be caused by fixed-length segmentation; at the same time, the feature vectors of text and image are extracted respectively through the bidirectional encoder and the visual encoder, and the feature association graph is constructed based on cosine similarity, which effectively captures the correlation between cross-modal information; finally, by constructing a feature pyramid structure, while maintaining the fine granularity of the original features, the multi-scale representation of features is achieved through adaptive pooling and global pooling operations, which not only retains local detail information, but also obtains global semantic information, thereby providing a rich feature representation basis for subsequent retrieval tasks. This embodiment improves the model's ability to understand contextual semantics, making the retrieval results more accurate and comprehensive.

[0028] According to one aspect of the present application, step S11 is further:

[0029] S111, reading the original multimodal dialogue sequence, calculating the information density of each position in the sequence; determining the initial segmentation point based on the preset segmentation length and information density, and generating an initial segmentation position set;

[0030] S112, based on the initial segmentation position set, dividing the original multimodal dialogue sequence into subsequences; performing an integrity check on each subsequence, adjusting the segmentation boundary to ensure the integrity of the subsequence, and generating a basic subsequence set;

[0031] S113, assigning an increasing time sequence number to each subsequence based on the position order of the subsequences in the basic subsequence set in the original multimodal dialogue sequence, building a front-to-back association relationship between sequences, and generating a numbered subsequence set;

[0032] S114, constructing a sequence organization structure based on the time sequence numbers and the front-to-back association relationship of the subsequences in the numbered subsequence set; based on the sequence organization structure, storing the subsequences in time sequence and establishing indexes to generate a preprocessing sequence set.

[0033] In one embodiment of the present application, the information density calculation method is as follows: the information density D(p) of sequence position p is D(p)=α·H(p)+β·C(p)+γ·M(p); wherein H(p) =-Σ(P(xi)·log(P(xi))) is the local text entropy, and P(xi) is the word frequency probability; C(p) =Σ(cos_sim(vi, vi+1)) is the local semantic coherence, and vi is the word vector; M(p) = λ1·I(p) +λ2·T(p) is the multimodal information degree, I(p) is the image feature density, and T(p) is the text feature density; α, β, and γ are balance coefficients satisfying α+β+γ=1; λ1 and λ2 are modal weight coefficients satisfying λ1+λ2=1; the information density D(p) is used to determine the optimal split point, and when D(p) is lower than the threshold θ, it is marked as a candidate split point.

[0034] This embodiment achieves high-quality preprocessing of the original multimodal dialogue sequence through an adaptive segmentation strategy based on information density and an integrity assurance mechanism. Specifically, first, by calculating the information density of each position in the sequence and determining the optimal segmentation point in combination with the preset segmentation length, the semantic fragmentation problem that may be caused by simple fixed-length segmentation is avoided; then, the integrity of the initially segmented subsequences is checked, and the boundaries are adjusted to ensure that each subsequence contains complete semantic information units; then, an increasing time sequence number is assigned to each subsequence and a front-and-back association relationship is established to ensure the time sequence continuity between subsequences; finally, by constructing a sequence organization structure and establishing an index, efficient sequence storage and retrieval are achieved. This embodiment not only improves the quality of preprocessing, but also lays a solid foundation for subsequent feature extraction and retrieval tasks.

[0035] According to one aspect of the present application, step S13 is further:

[0036] S131, read the local feature set, keep the original feature granularity and dimension unchanged, and build a feature index table; based on the feature index table, establish the position correspondence between the features and generate the first layer feature sequence;

[0037] S132, based on the first layer feature sequence, calculating the similarity matrix of adjacent features; based on the similarity matrix, determining the size of the feature fusion window; using an adaptive maximum pooling operation to aggregate the features within the determined feature fusion window to generate a second layer feature sequence;

[0038] S133, based on the first layer feature sequence, calculate the global feature distribution and construct an adaptive weight function; based on the adaptive weight function, perform weighted average pooling on all features in the first layer feature sequence to generate a third layer feature sequence;

[0039] S134. Based on the first-layer feature sequence, the second-layer feature sequence and the third-layer feature sequence, an inter-layer mapping matrix is ​​constructed; based on the mapping relationship of the inter-layer mapping matrix, the first-layer feature sequence, the second-layer feature sequence and the third-layer feature sequence are aligned and concatenated, and feature normalization is performed to generate a multi-level feature representation.

[0040] In one embodiment of the present application, an adaptive pooling feature fusion algorithm: fused feature F(k) = Σ(A(i, k)·V(i)), i∈N(k); wherein: A(i, k) = softmax(Q(k)·K(i) / sqrt(d)) is an adaptive attention weight, Q(k) is a query vector, K(i) is a key vector, and d is a feature dimension; V(i) is a value vector; N(k) is a neighborhood set of node k, and the neighborhood size is adaptively determined by feature similarity; the window size w(k) = ceiling(ρ·exp(-σ·D(k))), D(k) is a local feature density, and ρ, σ are adjustment parameters.

[0041] The process of calculating the global feature distribution is as follows: global feature G = Σ(W(fi)·fi) / Σ(W(fi)); where W(fi) = σ·Att(fi) + (1-σ)·Pos(fi) is the adaptive weight; Att(fi) = softmax(Q·fi / sqrt(d)) is the attention weight, Q is the learnable query vector; Pos(fi) = 1 / (1 + exp(ω·(iN / 2))) is the position weight, i is the feature position, N is the sequence length, ω is the position sensitivity parameter; σ is the balance coefficient; feature normalization F'(i)=(F(i)-μ) / sqrt (σ 2 + ε), μ is the mean, σ 2 is the variance, and ε is a small constant.

[0042] This embodiment realizes the efficient acquisition and integration of multi-scale feature representations by constructing a three-layer feature pyramid structure. Specifically, first, the fine granularity of the original features is maintained in the first layer, and a feature index table is established to maintain the position correspondence; then, the optimal fusion window size is determined by calculating the similarity matrix of adjacent features in the second layer, and adaptive maximum pooling is used for feature aggregation to capture medium-scale semantic information; then, in the third layer, weighted average pooling is performed on global features by designing an adaptive weight function to obtain the abstract semantic representation of the highest layer; finally, the three layers of features are aligned and spliced ​​by constructing an inter-layer mapping matrix, thereby realizing the effective fusion of features of different scales. This embodiment not only retains fine-grained local features, but also obtains coarse-grained global semantic information, thereby improving the feature representation capability of the model.

[0043] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0044] S21, reading each feature vector in the multi-level feature representation, calculating the feature importance score through a three-layer feedforward neural network; based on the importance score, filtering and compressing the feature vector to generate a compressed feature vector;

[0045] S22. Based on the compressed feature vector, the temporal correlation between the vectors is calculated and used as the weight of the edge to construct a temporal dependency graph. A gated graph attention network is used to perform message transmission and state update on the nodes in the temporal dependency graph to obtain a temporal enhancement feature containing temporal information.

[0046] S23. Based on the time series enhancement features, the variance of the feature vector in the continuous time window is calculated to obtain a feature stability matrix; based on the feature stability matrix, features whose stability in the time series enhancement features is higher than a preset threshold are selected to form a memory vector sequence.

[0047] In one embodiment of the present application, the feature importance score algorithm is specifically: feature importance score R(f) =η·I(f) + (1-η)·T(f); wherein I(f) = MLP(concat(f, c, h)) is the information score, f is the feature vector, c is the context vector, h is the historical information vector, and MLP is a three-layer perceptron; T(f) = exp(-ω·t) / Σ(exp(-ω·ti)) is the timeliness score, t is the feature timestamp, and ω is the attenuation coefficient; η is a balance parameter; when R(f) is greater than the dynamic threshold τ(t), the feature is retained, τ(t) =τ0·(1 - exp(-φ·t)), τ0 is the baseline threshold, and φ is the time adjustment parameter.

[0048] This embodiment improves the quality and representation efficiency of memory vectors through the triple optimization strategy of feature compression, timing enhancement, and stability screening. First, the feature importance score is calculated through a three-layer feedforward neural network to achieve adaptive compression of features, which not only reduces storage overhead but also retains key information; secondly, a timing dependency graph is constructed based on compressed features, and nodes are updated through a gated graph attention network, effectively encoding timing information into the feature representation, enhancing the model's ability to understand the conversation history; finally, the feature stability is evaluated by calculating the variance of the feature in a continuous time window, and a memory vector sequence is formed based on stability selection, ensuring that the retained features have strong temporal consistency and semantic stability. This embodiment not only improves the quality of memory vectors, but also reduces storage and computational overhead.

[0049] According to one aspect of the present application, step S22 is further:

[0050] S221, constructing a sliding time window of compressed feature vectors, calculating the mutual information coefficient between vector pairs in the sliding time window; constructing a time series correlation matrix based on the mutual information coefficient, and generating a time series association matrix;

[0051] S222, converting the temporal association matrix into an adjacency matrix of a graph; assigning an initial state vector to each node of the adjacency matrix based on the compressed eigenvector; constructing a bidirectional propagation path based on the adjacency matrix and the initial state vector to generate an initial temporal dependency graph;

[0052] S223, based on the initial timing dependency graph, construct a gating unit, calculate the information flow intensity of each edge; based on the information flow intensity, adjust the direction and intensity of message transmission, and iteratively update the node state to generate an updated timing dependency graph;

[0053] S224. Based on the updated timing dependency graph, extract the final state vector of each node; perform weighted combination of the final state vector and the compressed feature vector, retain the original information through residual connection, and generate timing enhanced features.

[0054] In one embodiment of the present application, the temporal correlation calculation and graph construction algorithm: the temporal correlation score TS(i, j) = ψ·MI(i, j) + (1-ψ)·TC(i, j); wherein MI(i, j) = Σ(P(x, y)·log(P(x, y) / P(x)P(y))) is the mutual information coefficient, and P(x, y) is the joint distribution of feature pairs; TC(i, j) = GRU(abs(ti-tj)) is the time interval encoding, GRU is the gated recurrent unit, ti, tj are timestamps; ψ is the weight coefficient; edge weight W(i, j) = sigmoid(TS(i, j))·exp(-κ·|ij|), and κ is the distance attenuation coefficient.

[0055] This embodiment realizes the effective encoding of timing information through the construction of a timing dependency graph and the update mechanism of a gated graph attention network. Specifically, first, a timing association matrix is ​​constructed by calculating the mutual information coefficient of the feature vector in a sliding time window to capture the timing dependency between features; then the association matrix is ​​converted into the adjacency matrix of the graph and an initial state vector is assigned to establish a bidirectional propagation path; then the direction and intensity of message transmission are dynamically adjusted through the gating unit to achieve iterative update of the node state; finally, the updated state vector is fused with the original feature through a residual connection to retain important original information. This embodiment not only effectively encodes timing information into the feature representation, but also can adaptively adjust the importance of different timing dependencies.

[0056] According to one aspect of the present application, step S23 is further:

[0057] S231. Read the time series enhancement features, calculate the mean vector of each feature vector in the continuous time window, calculate the Euclidean distance between the feature vector and the mean vector, construct a fluctuation matrix based on the distance value, and generate a feature fluctuation matrix.

[0058] S232. Read the feature fluctuation matrix and the timing enhancement feature, convert the fluctuation value into a stability score, design an adaptive threshold function to dynamically adjust the stability judgment standard, evaluate the stability of the feature, and generate a feature stability matrix.

[0059] S233, read the feature stability matrix and the timing enhancement feature, sort the features based on the stability score, use the piecewise linear mapping function to calculate the retention probability of the feature, screen the features according to the retention probability, and generate a candidate memory vector.

[0060] S234, read the candidate memory vectors, calculate the redundancy between the vectors, delete the highly repeated vectors, maintain the diversity of the memory sequence, rearrange the retained vectors in time sequence, and generate a memory vector sequence.

[0061] In one embodiment of the present application, a feature stability evaluation algorithm is provided: feature stability score S(f) = θ·V(f)+ (1-θ)·C(f); wherein V(f) = 1 / (1 + exp(ε·var(ft))) is variance stability, and var(ft) is the variance of the feature in the time window; C(f) = Σ(sim(fi, fi+1)) / n is the continuity score, sim is the cosine similarity, and n is the window size; θ is the weight coefficient; stability threshold τs(t) = τs0·(1 + ξ·log(1 + t)), τs0 is the baseline threshold, and ξ is the time adjustment parameter; when S(f) is greater than τs(t), it is determined to be a stable feature.

[0062] Feature retention probability calculation algorithm: retention probability P(f) = η·Stab(f) + (1-η)·Info(f); where Stab(f) = sigmoid(α·(S(f) - τs)) is the stability probability, S(f) is the feature stability score, and τs is the stability threshold; Info(f) = 1 - exp(-β·MI(f, M)) is the information gain probability, and MI(f, M) is the mutual information between the feature and the existing memory; η is the balance coefficient, α and β are scale parameters; feature screening threshold τf(t) = τf0·(1 + γ·t 2 / (t 2 +k 2 )), τf0 is the baseline threshold, γ is the growth coefficient, and k is the saturation parameter.

[0063] This embodiment realizes the construction of a high-quality memory vector sequence through feature stability evaluation and memory vector optimization. Specifically, first, the mean vector and fluctuation degree of the feature vector are calculated in a continuous time window to construct a feature fluctuation matrix; then, the fluctuation value is converted into a stability score through an adaptive threshold function to evaluate the stability of the feature; then, the retention probability of the feature is calculated based on the stability score and the piecewise linear mapping function to achieve preliminary screening of the feature; finally, the redundancy between vectors is calculated and highly repeated vectors are deleted to ensure the diversity and non-redundancy of the memory sequence. This embodiment not only improves the quality of the memory vector, but also optimizes the storage efficiency.

[0064] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0065] S31, reading the multimodal query data input by the user, using a feature extractor to process the text and image content in the multimodal query data respectively, extracting corresponding feature vectors, and combining them to form original query features; based on the original query features, using a semantic analyzer to extract query intent and key information units; based on the query intent and key information units, constructing a query structure diagram, and generating a structured query representation;

[0066] S32. Based on the structured query representation and memory vector sequence, the attention weights between the query and memory nodes are calculated to construct an interaction graph; information is transferred to the interaction graph through a multi-head attention mechanism to generate interaction features;

[0067] S33. Use a feature importance evaluator to calculate the weights of features of different scales in the interactive features; based on the weights, perform weighted aggregation on the interactive features to obtain an aggregation result; based on the aggregation result, process through a nonlinear transformation layer to obtain a fused feature vector.

[0068] This embodiment achieves accurate understanding and efficient retrieval of query intent through query feature extraction, structured representation construction and multi-level interaction. Specifically, first, feature extraction is performed on multimodal query data, and query intent and key information units are extracted through a semantic analyzer to construct a query structure graph, making the query representation more standardized and structured; then, an interaction graph is constructed by calculating the attention weights between query and memory nodes, and a multi-head attention mechanism is used for information transmission, achieving deep interaction between query and memory information; finally, through feature importance evaluation and multi-scale feature integration, a feature vector that integrates query intent and memory information is obtained. This embodiment improves the accuracy and recall of retrieval, especially when processing complex multimodal queries.

[0069] According to one aspect of the present application, step S31 is further:

[0070] S311, reading multimodal query data input by a user, extracting text features using an attention-enhanced text encoder, and extracting image features using a region-aware visual encoder; mapping text features and image features to a unified semantic space through a preconfigured cross-modal alignment module to generate a modal feature set;

[0071] S312, based on the modal feature set, calculating the mutual information matrix between features; based on the mutual information matrix, identifying the cross-modal association pattern; based on the cross-modal association pattern, adaptively fusing the associated features to generate original query features;

[0072] S313, based on the original query features, construct a semantic dependency tree of the features; analyze the semantic dependency tree to extract the query intent; based on the query intent, identify key information nodes and their relationships to generate a query semantic graph;

[0073] S314, performing structural normalization processing on the query semantic graph to obtain a normalized query semantic graph; establishing a hierarchical relationship between nodes in the normalized query semantic graph, and constructing an information transmission channel to generate a structured query representation.

[0074] In one embodiment of the present application, a cross-modal feature fusion algorithm: fusion feature F = Σ(π_k·F_k), k∈{text, img}; wherein text represents text, img represents image, π_k = softmax(MLP(concat(g_k, h))) is the modality weight, g_k is the global feature of each modality, and h is the historical state vector; F_k = CrossAtt(Q_k, K_m, V_m) is the cross-modal attention feature, Q_k is the query matrix of modality k, K_m and V_m are the key value matrices of the memory library; CrossAtt(Q, K, V) = softmax(Q·K T / sqrt(d))·V, d is the feature dimension; the modal weight satisfies Σπ_k = 1.

[0075] This embodiment achieves a deep understanding and representation of multimodal queries by fusing enhanced feature extraction and structured semantic analysis. Specifically, firstly, an attention-enhanced text encoder and a region-aware visual encoder are used to extract features of different modalities respectively, and the features are mapped to a unified semantic space through a cross-modal alignment module; then, cross-modal association patterns are identified based on the mutual information matrix between features, and the associated features are adaptively fused; then, query intent and key information nodes are extracted by constructing a semantic dependency tree of features; finally, the semantic graph is structurally normalized and a hierarchical relationship is established to form a complete structured query representation. This embodiment not only accurately captures the multimodal information of the query, but also achieves a precise understanding of the query intent, laying the foundation for subsequent retrieval matching.

[0076] According to one aspect of the present application, step S32 is further:

[0077] S321. Read the structured query representation and memory vector sequence, calculate the initial attention score between the query node and the memory node, use the adaptive threshold segmentation algorithm to hierarchically process the attention score, and generate a hierarchical attention matrix.

[0078] S322. Read the hierarchical attention matrix, map the attention matrix to the edge weights of the graph, construct the initial interaction graph, use the dynamic connection adjustment algorithm to optimize the topological structure of the graph, and generate the initial interaction graph.

[0079] S323. Read the initial interaction graph, design a decay function based on the path length between nodes, adjust the edge weights, use a graph pruning algorithm to delete low-weight connections, and generate an optimized interaction graph.

[0080] S324. Read and optimize the interaction graph, design a multi-attention message passing path based on the graph structure, update the node representation through an asynchronous message passing mechanism, perform feature aggregation on the updated node representation, and generate interaction features.

[0081] In one embodiment of the present application, the attention score calculation and interaction graph construction algorithm: the interaction graph edge weight E(i, j) = ζ·Att(i, j) + (1-ζ)·Str(i, j); wherein Att(i, j) = MultiHead(Qi, Kj, Vj) is the multi-head attention score, MultiHead(Q, K, V) = concat(head1, ..., headh)·W0 is the multi-head attention mechanism, W0 is the weight matrix, headi = Attention(QWi_Q, KWi_K, VWi_V) is the output of a single attention head; Str(i, j) = cos_sim(G(i), G(j)) is the structural similarity, G(·) is the graph neural network encoding, and cos_sim( ) is the cosine similarity function; ζ is the balance coefficient; the edge weight threshold τe =υ·(1 - exp(-χ·d(i,j))), d(i,j) is the node distance, υ, χ are adjustment parameters, Wi_Q is the query weight matrix of the i-th attention head, Wi_K is the key weight matrix of the i-th attention head, and Wi_V is the value weight matrix of the i-th attention head.

[0082] This embodiment realizes the deep interaction between query and memory information through multi-level attention mechanism and dynamic graph structure optimization. Specifically, the initial attention score between the query node and the memory node is first calculated, and the adaptive threshold segmentation algorithm is used for hierarchical processing; then the attention matrix is ​​mapped to the edge weight of the graph, and the topological structure of the graph is optimized through the dynamic connection adjustment algorithm; then the attenuation function is designed based on the path length between the nodes to adjust the edge weight, and the low-weight connection is deleted through the graph pruning algorithm; finally, the node representation is updated by designing an asynchronous message passing mechanism of multi-head attention. This embodiment not only ensures the computational efficiency of the interaction process, but also realizes the full integration of query and memory information, and improves the accuracy of retrieval.

[0083] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0084] S41, calculating the similarity matrix between the fused feature vectors; constructing a hierarchical index tree based on the similarity matrix; retrieving nodes whose similarity exceeds a preset threshold from the hierarchical index tree to generate a retrieval candidate set;

[0085] S42, based on the retrieval candidate set, respectively calculating the matching scores of the candidate items from the three dimensions of semantic similarity, structural similarity and contextual coherence to form a comprehensive matching score; based on the comprehensive matching score, ranking the candidate items in the retrieval candidate set to generate a ranked candidate set;

[0086] S43. Based on the sorted candidate set, the semantic overlap between the candidate items is calculated; based on the semantic overlap, duplicate or redundant candidate items are deleted to obtain a deduplicated candidate set; based on the deduplicated candidate set, the information entropy of the candidate items is calculated; based on the information entropy, the candidate items in the deduplicated candidate set are reordered to generate a diverse candidate set; based on the diverse candidate set, the correlation between each candidate item and the current conversation context is calculated to construct a context consistency matrix; based on the context consistency matrix, the diverse candidate set is screened to generate a final retrieval result sequence.

[0087] This embodiment achieves efficient and accurate retrieval capabilities through multi-layer index construction, multi-dimensional similarity calculation and result optimization. Specifically, first, a hierarchical index tree is constructed based on the similarity matrix between feature vectors, and nodes with similarities exceeding a preset threshold are retrieved from it, thereby improving retrieval efficiency; then, the matching degree of candidate items is comprehensively evaluated from three dimensions: semantic similarity, structural similarity and contextual coherence, to ensure the comprehensiveness and accuracy of the retrieval results; finally, the quality of the retrieval results is further improved by optimizing operations such as deleting duplicate items, adjusting the sorting to increase diversity, and filtering based on contextual relevance. This embodiment not only ensures retrieval efficiency, but also ensures the relevance and diversity of retrieval results, and is particularly suitable for processing complex retrieval requirements in long conversation scenarios.

[0088] According to one aspect of the present application, step S41 is further:

[0089] S411, read the fused feature vector, construct a normalized representation of the feature vector, calculate the cosine similarity and Euclidean distance between the vectors, obtain the comprehensive similarity by weighted combination of the two distance metrics, and generate a feature similarity matrix.

[0090] S412, read the feature similarity matrix and the fused feature vectors, perform hierarchical clustering on the feature vectors based on the similarity values, build a tree-shaped clustering structure, assign a representative vector to each clustering node, and generate an initial index tree.

[0091] S413, read the initial index tree, calculate the coverage and branching factor of each node, split or merge the nodes, optimize the balance and retrieval efficiency of the tree, and generate an optimized index tree.

[0092] S414, read the optimized index tree, design a multi-path retrieval strategy, start searching for paths whose similarity exceeds a preset threshold from the root node of the tree, collect the leaf nodes at the end of the path, and generate a retrieval candidate set.

[0093] In one embodiment of the present application, the index tree optimization and multi-path retrieval algorithm: node splitting score SP(n) = λ·B(n) + (1-λ)·D(n); wherein B(n) = |L(n)| / maxL is the branch balance, L(n) is the number of leaves of node n, and maxL is the maximum allowed number of leaves; D(n) = avg(dist(ci, cj)) is the distance between clusters, ci, cj are sub-cluster centers; λ is the weight coefficient; retrieval similarity threshold τr(k) = τr0·exp(-φ·k), k is the retrieval depth, τr0 is the initial threshold, and φ is the depth attenuation coefficient; path score P(r) = Π(E(i)), E(i) is the edge weight on the path, and Π is the product operator.

[0094] This embodiment achieves efficient and accurate candidate item retrieval through a multi-level index structure and an adaptive retrieval strategy. Specifically, first, the cosine similarity and Euclidean distance of the feature vectors are calculated, and a comprehensive similarity matrix is ​​obtained through weighted combination; then, the feature vectors are hierarchically clustered based on the similarity values ​​to construct a tree index structure; then, the tree structure is dynamically optimized by calculating the coverage and branching factor of the nodes to improve the retrieval efficiency; finally, a multi-path retrieval strategy is designed to search for paths whose similarity exceeds the threshold starting from the root node. This embodiment not only reduces the computational complexity of the retrieval, but also ensures the quality of the retrieval results, and is particularly suitable for processing large-scale memory retrieval scenarios.

[0095] According to one aspect of the present application, step S43 is further:

[0096] S431, read the sorting candidate set, calculate the semantic overlap between the candidate items, build a similarity graph of the candidate items, identify highly similar candidate clusters based on the graph structure, retain the representative items of each cluster according to the importance score, and generate a deduplicated candidate set.

[0097] S432, read the deduplicated candidate set, calculate the information entropy of the candidate items, construct a diversity scoring function based on the information entropy and the candidate item ranking, reorder the candidate items to maximize the overall diversity, and generate a diverse candidate set.

[0098] S433, read the diversity candidate set, calculate the relevance of each candidate item with the current dialogue context, construct a context consistency matrix, weight the candidate items by importance based on the consistency matrix, and generate a relevance candidate set.

[0099] S434, read the relevance candidate set, integrate the ranking scores, diversity scores and relevance of the candidate items, use a multi-objective optimization method to determine the final ranking, organize the search items in the optimized order, and generate a search result sequence.

[0100] In one embodiment of the present application, the diversity score and context relevance calculation algorithm is: the final ranking score R(c) = α·Rel(c) + β·Div(c) + γ·Con(c); wherein Rel(c) = sim(c, q) is the relevance score, q is the query vector; Div(c) = 1 - max(sim(c, ci)) is the diversity score, ci is the selected candidate; Con(c) = LSTM(concat(h, c)) is the context relevance, h is the history vector; the coefficients satisfy α+β+γ=1; the candidate selection probability P(c) = softmax(R(c) / τ), τ is the temperature parameter.

[0101] This embodiment achieves high-quality screening and organization of search results through multi-dimensional result optimization and sorting strategies. Specifically, first, the semantic overlap between candidate items is calculated to identify highly similar candidate clusters, and the representative items of each cluster are retained to remove redundancy; then, a diversity scoring function is constructed based on information entropy and ranking, and the overall diversity of the results is maximized through re-ranking; then, the relevance of the candidate items to the current conversation context is calculated, and the importance is weighted through the consistency matrix; finally, a multi-objective optimization method is used to integrate the scores to determine the final ranking. This embodiment not only ensures the relevance and diversity of the search results, but also ensures the coherence with the conversation context.

[0102] According to one aspect of the present application, it also includes:

[0103] S5. At predetermined intervals, read the retrieval result sequence and the dialogue feedback information input by the user, and extract the feedback signal; based on the feedback signal, evaluate the retrieval quality and obtain the update strategy; based on the update strategy and the memory vector sequence, perform memory optimization to obtain the updated memory vector sequence.

[0104] S51, reading the search result sequence and the dialogue feedback information input by the user, extracting the user satisfaction signal from the feedback information, calculating the matching degree between the search result and the user intention, formulating a feature weight adjustment plan according to the satisfaction signal and the matching degree, and generating an update strategy;

[0105] S52, read the update strategy and memory vector sequence, recalculate the importance scores of the memory vectors according to the weight adjustment scheme in the update strategy, remove the memory vectors whose importance scores are lower than the threshold, integrate the new retrieval results into the memory sequence, and generate an updated memory vector sequence.

[0106] This embodiment realizes adaptive updating and continuous optimization of the retrieval system through a periodic memory optimization mechanism. Specifically, firstly, based on the retrieval results and user feedback information, the satisfaction signal is extracted, the retrieval quality is evaluated and a feature weight adjustment plan is formulated; then, according to the update strategy, the importance score of the memory vector is recalculated, the low importance vector is removed and the new retrieval results are integrated to realize the dynamic update of the memory sequence. This embodiment enables the system to continuously learn and improve, and continuously improve the retrieval effect with use. Through regular memory optimization, the timeliness of the memory sequence is maintained, and the continuous improvement of system performance is ensured.

[0107] According to one aspect of the present application, step S52 is further:

[0108] S521, read the update strategy and memory vector sequence, extract the weight adjustment parameter in the update strategy, calculate the timeliness attenuation factor for each memory vector, multiply the attenuation factor by the original importance score, and generate a timeliness weighted score.

[0109] S522, read the timeliness weighted score and the memory vector sequence, construct a life cycle curve of the memory vector, determine the activity of the vector according to the slope of the curve, calculate the retention probability based on the activity and timeliness score, and generate an optimized candidate vector.

[0110] S523, read the optimization candidate vector and the current search result sequence, calculate the information gain of the new and old memory vectors, construct a memory update graph, determine the optimal memory replacement scheme through the maximum flow algorithm on the graph, and generate an update candidate sequence.

[0111] S524, read and update the candidate sequence, perform consistency check on the candidate sequence, adjust the timing relationship between vectors, rebuild the index structure of the memory vector, ensure the continuity and integrity of the sequence, and generate an updated memory vector sequence.

[0112] In one embodiment of the present application, the consistency check of the candidate sequence is specifically as follows: consistency score CS(M) = ψ·Temp(M) + (1-ψ)·Coh(M); wherein Temp(M) = Σ(sim(mi, mi+1)) / |M| is temporal consistency, and mi is a memory vector; Coh(M) = min(eig(L)) is spectral consistency, and L is the Laplace matrix of the memory map; ψ is a weight coefficient; update probability P(mi) = softmax((CS(M∪{mi}) - CS(M)) / τ), where τ is a temperature parameter; reconstruction is triggered when CS(M) is lower than a threshold τm, where τm =τm0·(1 - exp(-φ·t)), where τm0 is a reference threshold and φ is a time adjustment parameter.

[0113] This embodiment achieves efficient maintenance and updating of memory vector sequences through dynamic memory management and optimized update strategies. Specifically, first, the weight adjustment parameters in the update strategy are extracted, and the importance of the memory vector is re-evaluated in combination with the timeliness attenuation factor; then, the life cycle curve of the memory vector is constructed, and the activity of the vector is determined based on the slope of the curve; then, the information gain of the new and old memory vectors is calculated, and the optimal memory replacement scheme is determined through the maximum flow algorithm; finally, the update sequence is checked for consistency and the index structure is rebuilt. This embodiment not only ensures the timeliness and quality of the memory sequence, but also achieves efficient use of storage resources, providing strong support for the continuous optimization of the system.

[0114] In another embodiment of the present application, step S11 may also be:

[0115] S11a, read the original multimodal dialogue sequence, perform modal separation on text and image data, construct a temporal association matrix, calculate the temporal correspondence between modalities based on the association matrix, and generate a modal alignment sequence.

[0116] S11b, read the modal alignment sequence, use the dynamic programming algorithm to calculate the optimal segmentation point, perform a preliminary segmentation on the sequence while maintaining the modal integrity, and generate an initial subsequence set.

[0117] S11c, read the initial subsequence set, calculate the semantic coherence scores between subsequences, adjust the segmentation boundaries based on the coherence scores, use an iterative optimization algorithm to balance the information density of the subsequences, and generate a boundary optimized sequence.

[0118] S11d, read the boundary optimization sequence, mark the timing information of each subsequence, build a link relationship graph between sequences, organize the storage structure of the subsequences according to the link relationship, and generate a preprocessing sequence set.

[0119] In one embodiment of the present application, the coherence score C(s1, s2) = μ·Sem(s1, s2) + (1-μ)·Top(s1, s2); wherein Sem(s1, s2) = TransformerEncoder(s1)·TransformerEncoder(s2) is the semantic similarity, s1 and s2 are adjacent subsequences; Top(s1, s2) = Σ(min(TF(wi, s1), TF(wi, s2))) / Σ(max(TF(wi, s1), TF(wi, s2))) is the topic overlap, TF(w, s) is the word frequency; μ is the weight coefficient; the coherence threshold τc=τc0·(1 +ρ·log(len(s))), len(s) is the sequence length, τc0 is the benchmark threshold, and ρ is the length adjustment parameter.

[0120] In another embodiment of the present application, a memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model comprises the following steps:

[0121] Step 1: Divide the multimodal context into blocks to form a memory library. For pictures, each picture is remembered as a block. For text, it can be divided into sentences or segments according to usage requirements. Arrange the divided memories in the original order, and encode each block using the multimodal semantic encoder model, and arrange all the codes in the order of the memory blocks into a sequence, which is called the memory encoding sequence.

[0122] Step 2: When there is a query input, the query is segmented using the same segmentation strategy and encoded using the multimodal semantic encoder to form a query encoding sequence.

[0123] Step 3: Use BERT to encode all query texts and project the encoded embeddings to the same dimension as the query encoding sequence.

[0124] Step 4: Use BERT to encode the CLS vector of the query text and feed it into the feed-forward neural (FFN) network to generate the weighted value g.

[0125] Step 5: For the memory coding sequence, input it into the model of the transfomer structure, and add the output sequence to the original memory coding according to the weight g to obtain a new memory coding sequence.

[0126] Step 6: For the query encoding sequence, the transfomer model with a cross-attention structure is used for processing. The query encoding sequence provides the query matrix Q, and the projected query text embedding provides the key matrix K and the value matrix V. The output sequence is weighted and added to the query encoding sequence according to the g value to obtain a new query encoding sequence.

[0127] Step 7: Perform dot product on the two sequences and take the topN in the output matrix. The memory block at the corresponding position is the retrieval result.

[0128] The large language model in this embodiment is a dual-tower structure model, which consists of a memory encoder and a query encoder. On the block-stored memory bank, the memory encoder uses a multimodal semantic encoder (such as CLIP) to extract the semantic features of each memory block. Each memory block contains only one modality, such as pictures or text. The outputs of the multimodal semantic encoder here are arranged in sequence, which is called a memory semantic sequence. The unnormalized multimodal semantic encoder encoder output is used. In the following text, unless otherwise stated, all multimodal semantic encoder outputs refer to unnormalized outputs. Then, the memory encoder uses a transformer structure to extract the structural features of the memory on the memory semantic sequence using a self-attention mechanism, and the embedded sequence output by the transformer is called a memory non-semantic sequence. The memory encoder in the dual-tower model structure focuses on feature extraction of multimodal content with long contexts, effectively encoding non-semantic information into feature vectors, ensuring that the model is not limited to traditional semantic information when processing long conversation content or knowledge base queries, but has a more comprehensive understanding of contextual information. The query encoder adopts a multi-task head design to further subdivide feature extraction, so that the model can automatically adjust the weight when responding to different types of queries, which is more in line with the user's query intent. Similar to memory, the same block strategy is adopted for each query, and the embedded sequence is obtained through the multimodal semantic encoder, which is called the query semantic sequence. In order to further perform non-semantic retrieval, the query encoder uses the text encoder BERT to encode the query in the text modality (obviously non-semantic retrieval can only appear in text), projects the text embedding to the dimension of the query semantic sequence, and uses the transformer structure again to receive the query semantic sequence and the projected BERT output to extract non-semantic retrieval features. Among them, in the transformer here, the query semantic sequence and the projected BERT output jointly participate in the multi-head attention calculation, and the transformer output of the query encoder is called the query non-semantic sequence.

[0129] This embodiment has achieved significant technical effects by adding non-semantic retrieval capabilities on the basis of traditional retrieval models, making it more accurate and flexible in multimodal long-context tasks. This embodiment has a certain degree of adaptability to the combination of different features, and can be dynamically adjusted in various task scenarios to achieve more flexible scalability. Different types of features (such as semantic features of text and visual features of images) may have different degrees of influence on retrieval results in different scenarios. The strategy based on weight fusion can adaptively adjust the feature weights according to the characteristics of the input data and contextual information, so that the system can provide more accurate retrieval results in various task scenarios. The system learns the importance of each feature in a specific scenario through training, and then optimizes the strategy of feature combination. Through this fusion strategy, this embodiment has extremely high scalability and flexibility. With the addition of new feature types or changes in task requirements, the system can adapt quickly and adjust the feature fusion method according to the new task objectives. This enables this embodiment to not only handle current multimodal retrieval tasks, but also support more complex application scenarios in the future. By introducing the memory and query dual-tower encoder structure and combining the dot product retrieval method, the retrieval model of this embodiment has efficient retrieval speed and accuracy. The dot product retrieval method performs calculations between queries and memories, reducing the complexity in traditional retrieval models and improving retrieval efficiency in long-context tasks.

[0130] Aiming at the semantic fragmentation problem caused by the segmentation of long dialogue sequences, this paper proposes an adaptive segmentation strategy. First, the initial segmentation point is determined by calculating the information density distribution of each position in the sequence, and then the integrity of each subsequence is checked and the segmentation boundary is dynamically adjusted to ensure the semantic integrity of the subsequence. The collaborative segmentation of multimodal information is achieved through the construction of modal alignment sequences, the application of dynamic programming algorithms, and boundary optimization based on semantic coherence. Not only does it maintain the integrity of semantic units, but it also establishes effective associations between subsequences, laying the foundation for subsequent feature extraction.

[0131] In order to solve the problem of insufficient feature representation, the present invention constructs a feature pyramid structure to achieve the acquisition and fusion of multi-scale features. The first layer maintains the fine-grained information of the original features, the second layer captures the medium-scale semantic features through adaptive maximum pooling, and the third layer obtains abstract semantic representation through global weighted average pooling. By constructing an inter-layer mapping matrix, features of different scales are organically integrated, which not only retains local detail information but also obtains global semantic understanding. In particular, an adaptive weight function is used in the feature fusion process, which can dynamically adjust the contribution of different features according to their importance.

[0132] In order to solve the problem of insufficient temporal modeling, this paper proposes a temporal enhancement mechanism based on a gated graph attention network. First, the temporal correlation matrix is ​​constructed by calculating the mutual information coefficient between feature vectors, and then a bidirectional propagation path is constructed to realize message transmission, and the intensity and direction of information flow are dynamically adjusted through the gated unit. It can not only capture long-range dependencies, but also retain the original information through residual connections, which improves the model's ability to understand historical information.

[0133] For the problem of memory storage optimization, the present invention constructs a complete memory management mechanism. By calculating the stability matrix and time-effectiveness decay factor of the feature and combining the life cycle curve analysis, the dynamic optimization of the memory vector is realized. In particular, during the update process, the maximum flow algorithm is used to determine the optimal memory replacement scheme, which not only ensures the retention of important information but also avoids the accumulation of redundant information. The quality and efficiency of the memory sequence are further ensured through regular consistency checks and index reconstruction.

[0134] In order to solve the problem of single ranking of search results, the present invention constructs a multi-dimensional evaluation and optimization strategy. First, the semantic overlap of candidate items is calculated to eliminate redundancy, and then a diversity scoring function is constructed based on information entropy and ranking, and a comprehensive evaluation is performed in combination with context relevance. The final ranking is determined by a multi-objective optimization method, which not only ensures the relevance of the search results, but also ensures diversity and context coherence. In particular, the complementary characteristics of multimodal information are taken into account in the optimization process, which improves the quality of the search results.

[0135] The present invention constructs a complete memory retrieval system for long dialogue scenarios by organically combining technologies such as multimodal feature extraction, multi-level feature representation, temporal enhanced feature learning, multi-dimensional retrieval optimization, and adaptive memory update. The model's ability to understand the context of long dialogues is improved through multi-level feature representation and temporal enhanced learning; the accuracy and relevance of retrieval results are ensured through structured query representation and multi-dimensional similarity calculation; the continuous improvement of system performance is achieved through a periodic memory optimization mechanism; and the modular design makes the system highly scalable and adaptable. The present invention not only solves the challenges faced by traditional retrieval methods when dealing with long dialogue scenarios, but also provides a new solution for improving the multimodal dialogue capabilities of large language models.

[0136] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A memory retrieval method for enhancing the multimodal long-context conversation capability of a large language model, characterized in that: The steps include: S1. Receive the original multimodal dialogue sequence and split it according to a preset length to obtain a preprocessed sequence set; extract text and image features based on each subsequence in the preprocessed sequence set, construct a local feature association graph, and obtain a local feature set; perform hierarchical feature fusion on the local feature set to obtain a multi-level feature representation; S2. Based on multi-level feature representation, calculate the importance score of each feature vector and perform feature selection to obtain a compressed feature vector; Compress the feature vector, construct a temporal dependency graph, and update the node representation through a graph neural network to obtain temporal enhanced features; Based on the time series enhancement features, the stability matrix is ​​calculated; Based on the stability matrix, the time series enhancement features are screened to obtain the memory vector sequence; S3, reading the multimodal query data input by the user, performing feature extraction, and obtaining original query features; Based on the original query features, the query intent is extracted, the structure graph is constructed, and the structured query representation is obtained; Interactively calculate the structured query representation and the memory vector sequence to obtain interactive features; perform multi-scale integration and transformation on the interactive features to obtain a fused feature vector; S4. Based on the fused feature vector, a multi-layer index structure is constructed and a retrieval candidate set is generated; Based on the retrieval candidate set, multi-dimensional similarity is calculated and context consistency is verified to obtain a sorted candidate set; information filtering and relevance optimization are performed on the sorted candidate set to obtain a retrieval result sequence.

2. The memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model according to claim 1, characterized in that: Step S1 is further as follows: S11, receiving an original multimodal conversation sequence, dividing the original multimodal conversation sequence into a predetermined number of subsequences according to a preset segmentation length threshold, each subsequence containing adjacent text and image data; organizing all the divided subsequences according to time sequence numbers to generate a preprocessing sequence set; S12, read each subsequence in the preprocessed sequence set, use a bidirectional encoder to extract a word-level feature vector for the text content in the subsequence; use a visual encoder to extract a region-level feature vector for the image content in the subsequence; calculate the cosine similarity between the word-level feature vector and the region-level feature vector as the connection weight between the nodes, and construct a feature association graph; The word-level feature vector and region-level feature vector of each subsequence are combined with the corresponding feature association graph to form a local feature set; S13. Based on the local feature set, the original feature granularity is maintained as the bottom-level feature; based on the local feature set, the adjacent features are adaptively pooled to obtain the middle-level feature; based on the local feature set, the top-level feature is obtained through the global pooling operation; based on the bottom-level feature, the middle-level feature and the top-level feature, a feature pyramid structure is constructed and a hierarchical multi-level feature representation is generated.

3. The memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model according to claim 2, characterized in that: Step S2 is further as follows: S21, reading each feature vector in the multi-level feature representation, and calculating the feature importance score through a three-layer feedforward neural network; based on the importance score, filtering and compressing the feature vector to generate a compressed feature vector; S22. Based on the compressed feature vector, the temporal correlation between the vectors is calculated and used as the weight of the edge to construct a temporal dependency graph. A gated graph attention network is used to perform message transmission and state update on the nodes in the temporal dependency graph to obtain a temporal enhancement feature containing temporal information. S23, based on the time series enhancement feature, calculating the variance of the feature vector in the continuous time window to obtain a feature stability matrix; Based on the feature stability matrix, features with stability higher than a preset threshold in the time series enhancement features are selected to form a memory vector sequence.

4. The memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model according to claim 3, characterized in that: Step S3 is further as follows: S31, reading multimodal query data input by the user, using a feature extractor to process text and image content in the multimodal query data respectively, extracting corresponding feature vectors, and combining them to form original query features; Based on the original query features, a semantic analyzer is used to extract query intent and key information units; based on the query intent and key information units, a query structure graph is constructed to generate a structured query representation; S32. Based on the structured query representation and memory vector sequence, the attention weights between the query and memory nodes are calculated to construct an interaction graph; information is transferred to the interaction graph through a multi-head attention mechanism to generate interaction features; S33, using a feature importance estimator to calculate the weights of features of different scales in the interactive features; Based on the weights, the interaction features are weighted aggregated to obtain the aggregation results; Based on the aggregation results, the fused feature vector is obtained through nonlinear transformation layer processing.

5. The memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model according to claim 4, characterized in that: Step S4 is further as follows: S41, calculating the similarity matrix between the fused feature vectors; and constructing a hierarchical index tree based on the similarity matrix; Retrieve nodes whose similarity exceeds a preset threshold from the hierarchical index tree to generate a retrieval candidate set; S42, based on the retrieval candidate set, respectively calculating the matching scores of the candidate items from the three dimensions of semantic similarity, structural similarity and contextual coherence to form a comprehensive matching score; based on the comprehensive matching score, ranking the candidate items in the retrieval candidate set to generate a ranked candidate set; S43, based on the ranked candidate set, calculating the semantic overlap between the candidate items; Based on the semantic overlap, duplicate or redundant candidates are deleted to obtain a deduplicated candidate set; based on the deduplicated candidate set, the information entropy of the candidates is calculated; Based on information entropy, re-rank the candidates in the deduplication candidate set to generate a diverse candidate set; Based on the diverse candidate set, the correlation between each candidate and the current conversation context is calculated to construct a context consistency matrix. Based on the context consistency matrix, the diverse candidate set is screened to generate the final retrieval result sequence.

6. The memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model according to claim 5, characterized in that: Step S11 is further as follows: S111, reading the original multimodal dialogue sequence, calculating the information density of each position in the sequence; determining the initial segmentation point based on the preset segmentation length and information density, and generating an initial segmentation position set; S112, based on the initial segmentation position set, dividing the original multimodal dialogue sequence into subsequences; performing an integrity check on each subsequence, adjusting the segmentation boundary to ensure the integrity of the subsequence, and generating a basic subsequence set; S113, assigning an increasing time sequence number to each subsequence based on the position order of the subsequences in the basic subsequence set in the original multimodal dialogue sequence, building a front-to-back association relationship between sequences, and generating a numbered subsequence set; S114, constructing a sequence organization structure based on the time sequence numbers and the front-to-back association relationship of the subsequences in the numbered subsequence set; Based on the sequence organization structure, the subsequences are stored and indexed in chronological order to generate a preprocessed sequence set.

7. The memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model according to claim 5, characterized in that: Step S13 is further as follows: S131, read the local feature set, keep the original feature granularity and dimension unchanged, and build a feature index table; Based on the feature index table, the position correspondence between features is established to generate the first-layer feature sequence; S132, calculating a similarity matrix of adjacent features based on the first-layer feature sequence; Based on the similarity matrix, the size of the feature fusion window is determined; the adaptive maximum pooling operation is used to aggregate the features within the determined feature fusion window to generate the second-layer feature sequence; S133, based on the first layer feature sequence, calculate the global feature distribution and construct an adaptive weight function; based on the adaptive weight function, perform weighted average pooling on all features in the first layer feature sequence to generate a third layer feature sequence; S134, constructing an inter-layer mapping matrix based on the first layer feature sequence, the second layer feature sequence and the third layer feature sequence; Based on the mapping relationship of the inter-layer mapping matrix, the first-layer feature sequence, the second-layer feature sequence and the third-layer feature sequence are aligned and concatenated, and feature normalization is performed to generate a multi-level feature representation.

8. The memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model according to claim 5, characterized in that: Step S22 is further as follows: S221, constructing a sliding time window of compressed feature vectors, calculating the mutual information coefficient between vector pairs in the sliding time window; constructing a time series correlation matrix based on the mutual information coefficient, and generating a time series association matrix; S222, converting the temporal association matrix into an adjacency matrix of a graph; based on Compress the eigenvectors and assign an initial state vector to each node of the adjacency matrix; Based on the adjacency matrix and the initial state vector, a bidirectional propagation path is constructed to generate an initial timing dependency graph; S223, based on the initial timing dependency graph, construct a gating unit and calculate the information flow intensity of each edge; Based on the strength of information flow, adjust the direction and strength of message transmission, iteratively update the node status, and generate an update timing dependency graph; S224. Based on the updated timing dependency graph, extract the final state vector of each node; perform weighted combination of the final state vector and the compressed feature vector, retain the original information through residual connection, and generate timing enhanced features.

9. The memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model according to claim 5, characterized in that: Step S31 is further as follows: S311, reading multimodal query data input by a user, extracting text features using an attention-enhanced text encoder, and extracting image features using a region-aware visual encoder; mapping text features and image features to a unified semantic space through a preconfigured cross-modal alignment module to generate a modal feature set; S312, calculating the mutual information matrix between features based on the modal feature set; Based on the mutual information matrix, the cross-modal correlation pattern is identified; based on the cross-modal correlation pattern, the correlation features are adaptively fused to generate the original query features; S313, constructing a semantic dependency tree of features based on the original query features; Analyze the semantic dependency tree to extract query intent; Based on the query intent, identify key information nodes and their relationships, and generate a query semantic graph; S314, performing structural normalization processing on the query semantic graph to obtain a normalized query semantic graph; establishing a hierarchical relationship between nodes in the normalized query semantic graph, and constructing an information transmission channel to generate a structured query representation.

10. The memory retrieval method for enhancing the multimodal long-context dialogue capability of a large language model according to claim 5, characterized in that: Also includes: S5. Reading the search result sequence and the dialogue feedback information input by the user at every predetermined period, and extracting the feedback signal; Based on the feedback signal, the retrieval quality is evaluated to obtain an update strategy; based on the update strategy and the memory vector sequence, memory optimization is performed to obtain an updated memory vector sequence.

Images