Multimodal large model long sequence information compression retrieval method

Abstract

The application provides a multimodal large model long sequence information compression retrieval method, relates to the technical field of data processing, and comprises the following steps: receiving multimodal long sequence original data of a target patient; mapping each modality data in the multimodal long sequence original data to a same hidden space through a pre-trained multimodal large model to obtain a unified semantic representation sequence with inter-modality semantic association; performing time sequence feature analysis on the unified semantic representation sequence, regarding feature points in the semantic representation as a spatial point set, determining a boundary distribution by constructing a convex hull of each feature cluster, identifying key feature clusters in the semantic representation, and determining a key time sequence interval based on a convex hull distribution mode of the key feature clusters. The application solves the problems of weak cross-modality semantic association, easy loss of key time sequence information, and low compression and retrieval efficiency in existing multimodal long sequence data retrieval.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for compressing and retrieving long sequence information in a multimodal large model. Background Technology

[0002] In scenarios such as medical auxiliary diagnosis, the demand for retrieving multimodal long sequence data is becoming increasingly urgent, but existing methods face problems such as high storage costs, difficulty in cross-modal semantic alignment, and easy loss of key information in long sequences.

[0003] In the retrieval of similar cases of lung cancer patients, it is necessary to process 6 months of chest CT image sequences, daily heart rate and blood oxygenation physiological signals, and weekly follow-up text reports. Traditional methods simply compress and store each modality of data separately, and calculate similarity by metadata matching or simple feature splicing during retrieval. This leads to technical shortcomings: the compression process destroys the cross-modal semantic association between changes in lesions on CT images and the worsening of cough symptoms in text reports, and fails to focus on the key time intervals of patient deterioration. Redundant data dilutes the core diagnostic features, and the cases retrieved ultimately cannot provide effective reference for clinical diagnosis due to the lack of accurate modal association and key time sequence matching. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method for compressing and retrieving long sequence information in multimodal large models, thereby solving the problems of weak cross-modal semantic association, easy loss of key time-series information, and low compression and retrieval efficiency in existing multimodal long sequence data retrieval.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] Firstly, a method for compressing and retrieving long sequence information from a multimodal large model, the method comprising:

[0007] Step 1: Receive the multimodal long sequence raw data of the target patient; through a pre-trained multimodal large model, map the modal data in the multimodal long sequence raw data to the same latent space to obtain a unified semantic representation sequence of semantic association between modalities;

[0008] Step 2: Perform temporal feature analysis on the unified semantic representation sequence. Treat the feature points in the semantic representation as a set of spatial points. Construct the convex hull of each feature cluster to determine the boundary distribution, identify the key feature clusters in the semantic representation, and determine the key temporal intervals based on the convex hull distribution pattern of the key feature clusters.

[0009] Step 3: Based on key time intervals, analyze the feature evolution patterns to construct feature change trajectories, and obtain feature enhancement parameters based on the feature change trajectories;

[0010] Step 4: Apply the feature enhancement parameters to the feature enhancement process of the unified semantic representation sequence. By highlighting key diagnostic features and maintaining cross-modal correlation, the optimized semantic representation is obtained.

[0011] Step 5: Based on the optimized semantic representation, generate a compressed representation that retains key diagnostic information and cross-modal correlations using an adaptive compression algorithm;

[0012] Step 6: Construct a cross-modal index library that integrates multimodal semantics based on compressed representation; when a multimodal retrieval query is received, map the query to the same latent space, perform similarity matching through the cross-modal index library, and retrieve historical cases similar to the target patient from the historical case library.

[0013] Furthermore, the system receives multimodal long-sequence raw data from the target patient; using a pre-trained multimodal large model, it maps each modality of the multimodal long-sequence raw data to the same latent space, obtaining a unified semantic representation sequence of intermodal semantic associations, including:

[0014] Step 1.1: Receive long-sequence raw data of multiple modalities from the target patient; the long-sequence raw data of multiple modalities includes at least text report sequences, medical image sequences, and physiological signal time series.

[0015] Step 1.2: Based on the original multimodal long sequence data, call the pre-trained multimodal large model to parse and align the original data to obtain a set of multimodal data segments with clear temporal correspondence.

[0016] Step 1.3: Based on the set of multimodal data fragments, drive the pre-trained multimodal large model to extract features from each modal data fragment and map the extracted features to a shared latent space to obtain the initial set of modal feature vectors;

[0017] Step 1.4: Based on the initial set of modal feature vectors, the initial modal feature vectors are interacted and integrated through the cross-modal attention fusion module in the pre-trained multimodal large model to strengthen the semantic association between modalities and finally obtain a unified semantic representation sequence.

[0018] Furthermore, temporal feature analysis is performed on the unified semantic representation sequence. The feature points in the semantic representation are treated as a set of spatial points. The convex hulls of each feature cluster are constructed to determine the boundary distribution, identifying key feature clusters in the semantic representation. Based on the convex hull distribution patterns of these key feature clusters, key temporal intervals are determined, including:

[0019] Step 2.1: Based on the unified semantic representation sequence, each semantic representation vector in the sequence is regarded as a feature point in a high-dimensional feature space, and the spatial distribution of feature points is obtained;

[0020] Step 2.2: Based on the spatial distribution of feature points, clustering analysis is performed on the feature points using a clustering algorithm to obtain multiple feature clusters with similar semantic attributes;

[0021] Step 2.3: Based on multiple feature clusters, perform the following for each feature cluster: calculate the convex hull boundary of each feature cluster in the high-dimensional feature space to obtain the boundary distribution representation of each feature cluster.

[0022] Step 2.4: Based on the boundary distribution representation of each feature cluster, identify at least one key feature cluster representing key clinical semantics by analyzing the geometric characteristics and spatial distribution relationship of each convex hull.

[0023] Step 2.5: Based on the key feature clusters, map the feature points in the key feature clusters back to the original time series. Based on the distribution density and continuity of the original time series on the time axis, obtain the key time series intervals.

[0024] Furthermore, based on key time intervals, the evolution patterns of features are analyzed to construct feature change trajectories, and feature enhancement parameters are obtained based on these trajectories, including:

[0025] Step 3.1: Based on the key time intervals, extract the semantic representation subsequences corresponding to the time intervals from the unified semantic representation sequence to obtain the key feature sequence;

[0026] Step 3.2: Based on the key feature sequence, the semantic representation of the continuous time steps in the sequence is modeled for evolutionary trends using a time-series pattern analysis algorithm to construct the feature change trajectory representing the dynamic changes of the features.

[0027] Step 3.3: Based on the feature change trajectory, the importance of features at different time points within the key time series interval is quantified by calculating the trajectory curvature change rate and feature point distribution density, so as to obtain the initial feature weight distribution.

[0028] Step 3.4: Based on the initial feature weight distribution, the weight distribution is corrected and normalized by combining the preset clinical diagnostic prior knowledge, and finally the feature enhancement parameters are obtained.

[0029] Furthermore, feature enhancement parameters are applied to the feature enhancement process of the unified semantic representation sequence. By highlighting key diagnostic features and maintaining cross-modal correlations, optimized semantic representations are obtained, including:

[0030] Step 4.1: Based on the feature enhancement parameters and the unified semantic representation sequence, apply the feature enhancement parameters to the unified semantic representation sequence, and obtain the preliminary enhanced feature mapping through weighted transformation operation;

[0031] Step 4.2: Based on the initial enhanced feature mapping, the enhanced features are regularized using a cross-modal association constraint algorithm to obtain cross-modal consistency enhanced features;

[0032] Step 4.3 involves performing residual connection and layer normalization on the cross-modal consistency enhancement features and the initially enhanced feature maps to obtain the optimized semantic representation.

[0033] Furthermore, based on the optimized semantic representation, an adaptive compression algorithm is used to generate a compressed representation that preserves key diagnostic information and cross-modal correlations, including:

[0034] Step 5.1: Based on the optimized semantic representation, the information entropy and diagnostic relevance index of each feature dimension in the semantic representation are calculated using the feature dimension importance analysis algorithm to obtain the feature importance distribution map;

[0035] Step 5.2: Based on the feature importance distribution map, the high-dimensional semantic representation is projected to the low-dimensional subspace using an adaptive projection dimensionality reduction algorithm to obtain a preliminary compressed representation.

[0036] Step 5.3: Based on the preliminary compressed representation, the preliminary compressed representation is checked and fine-tuned for cross-modal semantic consistency through the cross-modal association preservation mechanism, and finally the compressed representation is obtained.

[0037] Furthermore, a cross-modal index library integrating multimodal semantics is constructed based on compressed representations; when a multimodal retrieval query is received, the query is mapped to the same latent space, and similarity matching is performed through the cross-modal index library to retrieve historical cases similar to the target patient from the historical case database, including:

[0038] Step 6.1: Based on compressed representation, organize the compressed representations of current and historical cases into an index dataset, and construct a cross-modal index library that supports similarity retrieval using a vector indexing algorithm;

[0039] Step 6.2: Based on the cross-modal index library, when a retrieval query containing multimodal data is received, the pre-trained multimodal large model is invoked to map each modal data in the retrieval query to the same latent space as the compressed representation, thereby obtaining the query semantic representation.

[0040] Step 6.3: Based on the query semantic representation, compress the query semantic representation using an adaptive compression algorithm to obtain the query compressed representation;

[0041] Step 6.4: Based on the query compressed representation, perform an approximate nearest neighbor search across the modal index to calculate the similarity between the query compressed representation and the historical case compressed representation in the index, so as to obtain the similarity result;

[0042] Step 6.5: Based on the similarity results, sort the candidate historical cases in descending order of similarity, and use the most similar historical case as the final search result.

[0043] Secondly, a multimodal, large-model, long-sequence information compression and retrieval system includes:

[0044] The acquisition module is used to receive multimodal long sequence raw data of the target patient; through a pre-trained multimodal large model, the modal data in the multimodal long sequence raw data are uniformly mapped to the same latent space to obtain a unified semantic representation sequence of semantic association between modalities;

[0045] The computation module is used to perform temporal feature analysis on the unified semantic representation sequence. It treats the feature points in the semantic representation as a set of spatial points, constructs the convex hull of each feature cluster to determine the boundary distribution, identifies the key feature clusters in the semantic representation, and determines the key temporal intervals based on the convex hull distribution pattern of the key feature clusters.

[0046] The optimization module is used to analyze the feature evolution law based on key time intervals to construct feature change trajectories and obtain feature enhancement parameters based on the feature change trajectories. The feature enhancement parameters are applied to the feature enhancement processing of the unified semantic representation sequence. By highlighting key diagnostic features and maintaining cross-modal correlation, the optimized semantic representation is obtained. Based on the optimized semantic representation, an adaptive compression algorithm is used to generate a compressed representation that retains key diagnostic information and cross-modal correlation.

[0047] The processing module is used to construct a cross-modal index library that integrates multimodal semantics based on compressed representations. When a multimodal retrieval query is received, the query is mapped to the same latent space, and similarity matching is performed through the cross-modal index library to retrieve historical cases similar to the target patient from the historical case library.

[0048] Thirdly, a computing device includes:

[0049] One or more processors;

[0050] A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0051] Fourthly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0052] The above-described solution of the present invention has at least the following beneficial effects:

[0053] This approach overcomes the technical challenges of existing methods by employing a pre-trained multimodal large model to uniformly map multimodal long-sequence data to the same latent space, strengthening modal associations through cross-modal attention fusion, utilizing combined clustering to identify key feature clusters and key time-series intervals, generating feature enhancement parameters based on feature change trajectories and clinical priors, optimizing representations through residual connections, maintaining compressed representations through adaptive compression algorithms combined with cross-modal associations, and constructing a cross-modal index library to achieve efficient similarity retrieval. It overcomes these challenges by addressing the semantic gap in multimodal data leading to weak associations, the susceptibility of key time-series information in long sequences to being obscured by redundant data, the easy loss of key diagnostic information and cross-modal associations during compression, and low retrieval efficiency and insufficient accuracy. Ultimately, it achieves the technical effects of unifying multimodal semantics and strong associations, accurately focusing on key time-series intervals to avoid diluting core information, preserving key diagnostic information and cross-modal associations while efficiently compressing data, and improving the accuracy and efficiency of similar case retrieval. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the multimodal large-model long sequence information compression and retrieval method provided in an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of a multimodal large-model long sequence information compression and retrieval system provided in an embodiment of the present invention. Detailed Implementation

[0056] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0057] like Figure 1 As shown, embodiments of the present invention propose a method for compressing and retrieving long sequence information in a multimodal large model, the method comprising the following steps:

[0058] Step 1: Receive the multimodal long sequence raw data of the target patient; through a pre-trained multimodal large model, map the modal data in the multimodal long sequence raw data to the same latent space to obtain a unified semantic representation sequence of semantic association between modalities;

[0059] Step 2: Perform temporal feature analysis on the unified semantic representation sequence. Treat the feature points in the semantic representation as a set of spatial points. Construct the convex hull of each feature cluster to determine the boundary distribution, identify the key feature clusters in the semantic representation, and determine the key temporal intervals based on the convex hull distribution pattern of the key feature clusters.

[0060] Step 3: Based on key time intervals, analyze the feature evolution patterns to construct feature change trajectories, and obtain feature enhancement parameters based on the feature change trajectories;

[0061] Step 4: Apply the feature enhancement parameters to the feature enhancement process of the unified semantic representation sequence. By highlighting key diagnostic features and maintaining cross-modal correlation, the optimized semantic representation is obtained.

[0062] Step 5: Based on the optimized semantic representation, generate a compressed representation that retains key diagnostic information and cross-modal correlations using an adaptive compression algorithm;

[0063] Step 6: Construct a cross-modal index library that integrates multimodal semantics based on compressed representation; when a multimodal retrieval query is received, map the query to the same latent space, perform similarity matching through the cross-modal index library, and retrieve historical cases similar to the target patient from the historical case library.

[0064] In this embodiment of the invention, because a pre-trained multimodal large model is used to achieve a unified mapping of multimodal long sequence data to the same latent space to construct a representation sequence of semantic associations between modalities, and temporal feature analysis is performed on the unified semantic representation sequence to identify key feature clusters and key temporal intervals, feature change trajectories are constructed based on key temporal intervals and feature enhancement parameters are generated to optimize semantic representations, and an adaptive compression algorithm is combined to generate compressed representations that retain key diagnostic information and cross-modal associations, and similarity matching retrieval for multimodal queries is achieved through a cross-modal index, the technical means of overcoming the existing problems in multimodal long sequence data retrieval, such as weak associations due to the semantic gap between modalities, easy obscuring of key temporal information of long sequences by redundant data, loss of key information and cross-modal associations during compression, and insufficient retrieval efficiency and accuracy, are achieved. Thus, the technical effects of achieving semantic unity and strong association of multimodal data, accurately focusing on key temporal intervals to avoid dilution of core information, retaining key diagnostic value and cross-modal associations while efficiently compressing, and improving the accuracy and efficiency of similar case retrieval are achieved.

[0065] In a preferred embodiment of the present invention, step 1 above may include:

[0066] Step 1.1: Receive multimodal long-sequence raw data of the target patient; the multimodal long-sequence raw data includes at least text report sequences, medical image sequences, and physiological signal time series, specifically including: receiving multimodal long-sequence raw data of the target lung cancer patient, including daily heart rate and blood oxygen physiological signal time series continuously acquired from chest CT image sequences of the patient for 6 consecutive months, as well as text report sequences formed by weekly follow-ups, to ensure complete acquisition of different types of time series data during the patient's diagnosis and treatment.

[0067] Step 1.2: Based on the multimodal long-sequence raw data, a pre-trained multimodal large model is invoked to parse and align the raw data by different modalities to obtain a set of multimodal data segments with clear temporal correspondence. Specifically, based on the received multimodal long-sequence raw data, the pre-trained multimodal large model is invoked to parse each type of raw data. This includes parsing chest CT image sequences to clarify frame structure and image content, parsing physiological signal time series to extract effective sampling points, and parsing text report sequences to organize semantic information. Subsequently, through the model's temporal alignment mechanism, data from different modalities are precisely matched according to acquisition time, so that the physiological signal data and text report content of CT image segments under the same time dimension form a correspondence, ultimately obtaining a set of multimodal data segments with clear temporal correspondence.

[0068] The multimodal large-scale model involves collecting extensive multimodal long-sequence data from the medical field, including chest CT image sequences, time series of physiological signals such as heart rate and blood oxygenation, and follow-up text report sequences, ensuring the data contains complete temporal labels. The data is cleaned, invalid data is removed, and temporal correspondences between modalities are labeled. A two-stage architecture of modal branching and cross-modal fusion is adopted. The CT image branch uses a 3D convolutional network to extract spatiotemporal features, the physiological signal branch uses a temporal convolutional network to capture fluctuation patterns, and the text branch uses a Transformer model to parse semantics. The cross-modal fusion module introduces a temporal attention mechanism to specifically model the temporal dimensional relationships between different modalities. This is applied to general medical data. Multi-task pre-training was performed, including single-modal parsing and cross-modal temporal alignment tasks. Self-supervised learning enabled the model to automatically learn intra-modal features and inter-modal temporal correlation patterns. Fine-tuning was performed using a lung cancer case-specific dataset to optimize the model's sensitivity to lung cancer-related features, with a focus on adjusting the parameters of the temporal alignment mechanism to ensure time matching accuracy for 6-month long-sequence data, adapting to the temporal span and data characteristics of lung cancer diagnosis and treatment scenarios. Gradient pruning and layer normalization techniques were used to improve model stability. Cross-validation was employed to verify the model's parsing accuracy and temporal alignment precision, ensuring that the model can parse the core information of each modality and achieve accurate temporal dimension matching of different modalities in long-sequence scenarios.

[0069] Step 1.3: Based on the multimodal data fragment set, drive the pre-trained multimodal large model to extract features from each modal data fragment and map the extracted features to a shared latent space to obtain an initial set of modal feature vectors. Specifically, this includes: taking the multimodal data fragment set as input, driving the pre-trained multimodal large model to perform feature extraction operations on each modal data fragment. For CT image fragments, extract visual features such as lesion morphology and edge features; for physiological signal fragments, extract temporal features such as fluctuation trend and peak values; for text report fragments, extract semantic features such as symptom description and disease changes. Then, the model maps these extracted features from different modalities to a shared latent space, so that different types of features have a unified semantic expression basis, thereby obtaining the initial set of modal feature vectors.

[0070] Step 1.4: Based on the initial set of modal feature vectors, the cross-modal attention fusion module in the pre-trained multimodal large model is used to interact and integrate the initial modal feature vectors to strengthen the semantic association between modalities and finally obtain a unified semantic representation sequence. Specifically, based on the initial set of modal feature vectors, the cross-modal attention fusion module in the pre-trained multimodal large model is called. This module calculates the semantic correlation between different modal feature vectors and focuses on feature information with related meaning. For example, the feature vector of lesion enlargement in CT images is interacted with the feature vector of cough symptom aggravation in text reports. At the same time, all initial modal feature vectors are fully integrated to strengthen the semantic association between modalities, eliminate the semantic barriers between different modal data, and finally obtain a unified semantic representation sequence that can comprehensively reflect the semantic association of multimodal time series data.

[0071] In this embodiment of the invention, by using multimodal long-sequence raw data such as received text report sequences, medical image sequences, and physiological signal time series, and parsing and temporally aligning the raw data using a pre-trained multimodal large model, extracting features of each modality and mapping them to the same shared latent space, and then using the cross-modal attention fusion module in the model to interactively integrate the initial modal feature vectors, the technical problems of unclear temporal correspondence of multimodal long-sequence data, semantic gaps between modalities, and lack of effective feature association and integration are overcome. Thus, the technical effect of clarifying the temporal correspondence of multimodal data, eliminating semantic barriers between modalities, strengthening semantic association between modalities, and finally obtaining a unified semantic representation sequence rich in cross-modal semantic information is achieved.

[0072] In a preferred embodiment of the present invention, step 2 above may include:

[0073] Step 2.1: Based on the unified semantic representation sequence, each semantic representation vector in the sequence is regarded as a feature point in a high-dimensional feature space to obtain the spatial distribution of feature points. Specifically, this includes: based on the obtained unified semantic representation sequence, which contains the semantic information of the fusion of chest CT images, heart rate, blood oxygenation and physiological signals of lung cancer patients within 6 months, the semantic representation vector corresponding to each time point in the sequence is regarded as a feature point in a high-dimensional feature space. The position of each feature point is determined by the values ​​of each dimension in the vector, thus presenting the distribution of these feature points in the high-dimensional space, forming the spatial distribution of feature points.

[0074] Step 2.2: Based on the spatial distribution of feature points, clustering algorithms are used to perform clustering analysis on the feature points to obtain multiple feature clusters with similar semantic attributes. Specifically, based on the aforementioned spatial distribution of feature points, clustering algorithms are used to group feature points in high-dimensional space. Feature points that are spatially close and represent similar disease semantics are grouped into one category to form multiple feature clusters. For example, feature points corresponding to CT image features representing the stable stage of the disease, stable physiological signal features, and text description features without obvious symptoms are clustered into one cluster, and relevant feature points representing the stage of disease change are clustered into another cluster, ultimately obtaining multiple feature clusters with similar semantic attributes.

[0075] Step 2.3: Based on multiple feature clusters, perform the following for each feature cluster: calculate the convex hull boundary of each feature cluster in the high-dimensional feature space to obtain the boundary distribution representation of each feature cluster. Specifically, for the obtained multiple feature clusters, perform the following for each cluster: calculate the minimum convex polygon boundary that surrounds all feature points in the high-dimensional feature space. The boundary reflects the distribution range and contour of the feature cluster in the high-dimensional space, thereby obtaining the boundary distribution representation of each feature cluster.

[0076] Step 2.4: Based on the boundary distribution representation of each feature cluster, by analyzing the geometric characteristics and spatial distribution relationship of each convex hull, at least one key feature cluster representing key clinical semantics is identified. Specifically, this includes: based on the boundary distribution representation of each feature cluster, analyzing the geometric characteristics of each convex hull, such as the volume of the convex hull, the number of feature points contained, and the feature point distribution density; simultaneously analyzing the spatial distribution relationship between different convex hulls, such as whether they are adjacent and how far apart they are; combining the characteristics of key clinical semantics in lung cancer diagnosis, such as the semantic features corresponding to lesion enlargement, symptom aggravation, and abnormal physiological indicators, identifying feature clusters with larger convex hull volumes, higher feature point density, and close correlation with changes in the disease condition, and determining them as key feature clusters representing key clinical semantics.

[0077] Step 2.5: Based on the key feature clusters, map the feature points in the key feature clusters back to the original time series. According to the distribution density and continuity of the original time series on the time axis, obtain the key time series intervals. Specifically, this includes: for the identified key feature clusters, extract all feature points within the clusters; according to the original time stamp corresponding to each feature point, map these feature points back to the patient's original 6-month time axis and observe the distribution of these feature points on the time axis; statistically analyze the distribution density of feature points in different time periods, and check the continuity of the distribution. Determine the time period in which the feature points are densely and continuously distributed as the key time series interval. For example, during the period from the 3rd to the 4th month, the feature points of the key feature clusters are concentrated and continuously distributed, and the interval is the key time series interval for the deterioration of the patient's condition.

[0078] In this embodiment of the invention, by employing the technical means of treating each vector in the unified semantic representation sequence as a feature point in a high-dimensional feature space to obtain the spatial distribution of feature points, obtaining feature clusters with similar semantic attributes through clustering algorithms, performing boundary distribution representation acquisition on each feature cluster, analyzing the geometric characteristics of the convex hull and the spatial distribution relationship to identify key clinical semantic feature clusters, and then mapping the feature points of the key feature clusters back to the original time series and determining the key time series intervals based on the distribution density and continuity on the time axis, the technical problems of difficulty in accurately screening key clinical semantic features in multimodal long sequence data, inability to effectively locate key time series intervals, and redundant data obscuring core diagnostic information are overcome. Thus, the technical effects of accurately identifying feature clusters representing key clinical significance, accurately locking key time series intervals related to the disease, and eliminating invalid and redundant data to highlight core diagnostic features are achieved.

[0079] In a preferred embodiment of the present invention, step 3 above may include:

[0080] Step 3.1: Based on the key time intervals, extract the semantic representation subsequences corresponding to the time intervals from the unified semantic representation sequence to obtain the key feature sequence. Specifically, this includes: based on the previously determined key time intervals of lung cancer patient disease deterioration, accurately extract the semantic representation vectors corresponding to all time points within the key time interval from the unified semantic representation sequence containing fused semantic information of the patient's 6 months of multimodal data, forming a semantic representation subsequence. This subsequence centrally carries the fused semantic features of symptom descriptions in the follow-up text reports on changes in lesions on chest CT images, heart rate, blood oxygenation, and physiological signal fluctuations within the key interval, ultimately obtaining a key feature sequence focusing on core disease information.

[0081] Step 3.2: Based on the key feature sequence, the evolution trend model of the semantic representation of continuous time steps in the sequence is modeled by the temporal pattern analysis algorithm to construct the feature change trajectory representing the dynamic changes of features. Specifically, this includes: taking the key feature sequence as the analysis object, using the temporal pattern analysis algorithm to analyze the semantic representation vector of continuous time steps in the sequence one by one, capturing the change pattern of semantic features between adjacent time points, such as tracking the gradual trend of lesion morphology features within the key interval, the development trend of abnormal fluctuations in physiological signals, and the change in the semantic intensity of symptom description. By transforming the dynamic change process into a visualized feature change trajectory, the dynamic evolution path of the patient's condition from stable to deteriorating or the symptoms worsening within the key time interval is clearly presented.

[0082] Step 3.3: Based on the feature change trajectory, the importance of features at different time points within the key time series interval is quantified by calculating the rate of change of trajectory curvature and the distribution density of feature points to obtain the initial feature weight distribution. Specifically, this includes: calculating the rate of change of curvature of each node on the constructed feature change trajectory. A higher rate of change of curvature indicates a more drastic change in the disease characteristics at that time point. For example, the rate of change of curvature increases when lesions rapidly enlarge or physiological indicators become drastically abnormal. At the same time, the distribution density of feature points on the trajectory is statistically analyzed. A denser distribution of feature points indicates a more concentrated semantic information related to the disease during that time period. Combining these two indicators, the importance of features at each time point within the key time series interval is quantified and scored. Time points with more drastic changes and more concentrated information receive higher scores, ultimately forming an initial feature weight distribution that reflects the importance of features at different time points.

[0083] Step 3.4: Based on the initial feature weight distribution, and combined with pre-set clinical diagnostic prior knowledge, the weight distribution is corrected and normalized to obtain the feature enhancement parameters. Specifically, this includes: referencing pre-set clinical diagnostic prior knowledge for lung cancer, which includes key diagnostic elements such as changes in lesion size, fluctuation range of key physiological indicators, and worsening of typical symptoms; correcting the initial feature weight distribution based on these key diagnostic elements, for example, increasing the weight corresponding to lesion morphology changes and adjusting the weight of non-key physiological signal fluctuation features to ensure the weight distribution aligns with the focus of clinical diagnosis. Then, the corrected weights are normalized to ensure that the feature weights at all time points are within a uniform quantification range, ultimately yielding feature enhancement parameters that accurately highlight key clinical diagnostic features.

[0084] In this embodiment of the invention, the technical means of extracting corresponding semantic representation subsequences based on key time intervals to obtain key feature sequences, using time-series pattern analysis algorithms to model the evolution trend of semantic representations in continuous time steps to construct feature change trajectories, calculating the trajectory curvature change rate and feature point distribution density to quantify the importance of features at different time points to obtain an initial feature weight distribution, and then combining preset clinical diagnostic prior knowledge to correct and normalize the weight distribution to obtain feature enhancement parameters, overcomes the technical problems of unclear feature evolution patterns in key time intervals, difficulty in accurately quantifying feature importance, and lack of clinical diagnostic adaptability of weight distribution, which leads to the inability to effectively highlight core diagnostic features. Thus, it achieves the technical effects of clearly presenting the dynamic evolution trend of features in key time intervals, accurately distinguishing the importance of features at different time points, and obtaining feature enhancement parameters that fit clinical diagnostic needs.

[0085] In a preferred embodiment of the present invention, step 4 above may include:

[0086] Step 4.1: Based on the feature enhancement parameters and the unified semantic representation sequence, the feature enhancement parameters are applied to the unified semantic representation sequence. A preliminary enhanced feature map is obtained through a weighted transformation operation. Specifically, based on the previously obtained feature enhancement parameters and the unified semantic representation sequence containing 6 months of multimodal fusion semantic information of lung cancer patients, the feature enhancement parameters are assigned to each time point feature of the unified semantic representation sequence. The expression intensity of key diagnostic features is amplified through a weighted transformation operation. For example, the weight of features such as lesion enlargement in chest CT images, abnormal fluctuations in heart rate and blood oxygenation physiological signals, and worsening cough symptoms in follow-up text reports are increased. At the same time, information of other auxiliary features is reasonably retained. Finally, a preliminary enhanced feature map with key diagnostic features highlighted is obtained.

[0087] Step 4.2: Based on the initially enhanced feature map, the enhanced features are regularized using a cross-modal association constraint algorithm to obtain cross-modal consistent enhanced features. Specifically, this includes: based on the initially enhanced feature map, the enhanced features are regularized using a cross-modal association constraint algorithm. The algorithm focuses on constraining the semantic consistency between different modal features to ensure that the association between lesion change features in chest CT images and corresponding symptom description features, physiological signal abnormality features and lesion progression features in the text report are not destroyed, and to avoid cross-modal semantic disconnect caused by excessive enhancement of a single modal feature. Finally, cross-modal consistent enhanced features that maintain stable cross-modal semantic association are obtained.

[0088] Step 4.3 involves performing residual connection and layer normalization on the cross-modal consistency enhancement features and the initially enhanced feature maps to obtain the optimized semantic representation. Specifically, this includes: performing residual connection on the cross-modal consistency enhancement features and the initially enhanced feature maps to retain information in the initially enhanced feature maps that has not been over-adjusted by the cross-modal constraint algorithm, thus avoiding the weakening of key diagnostic features during regularization; then performing layer normalization on the connected features to unify the distribution range of the features and reduce the impact of numerical differences between different feature dimensions on subsequent processing, ultimately obtaining an optimized semantic representation that highlights the core diagnostic features, maintains stable cross-modal semantic association, and has a balanced feature distribution.

[0089] In this embodiment of the invention, by applying feature enhancement parameters to a unified semantic representation sequence and obtaining a preliminary enhanced feature map through weighted transformation, using a cross-modal association constraint algorithm to regularize the enhanced features, and then performing residual connection and layer normalization processing on the cross-modal consistency enhanced features and the preliminary enhanced feature map, the technical problems of easily losing cross-modal semantic associations, modal inconsistencies in enhanced features, and excessive suppression of some useful features are overcome during the enhancement of key diagnostic features. This achieves the technical effect of accurately highlighting key clinical diagnostic features, maintaining the consistency of semantic associations among multimodal data, avoiding information distortion and distribution imbalance caused by feature enhancement, and finally obtaining optimized semantic representations that are rich in core diagnostic value and have stable cross-modal associations.

[0090] In a preferred embodiment of the present invention, step 5 above may include:

[0091] Step 5.1: Based on the optimized semantic representation, the information entropy and diagnostic relevance index of each feature dimension in the semantic representation are calculated using the feature dimension importance analysis algorithm to obtain a feature importance distribution map. Specifically, this includes: based on the optimized semantic representation that highlights core diagnostic features while maintaining cross-modal semantic association, the feature dimension importance analysis algorithm is used to quantitatively analyze each feature dimension in the semantic representation; the information entropy of each feature dimension is calculated separately, with higher information entropy indicating richer disease-related information contained in that dimension; at the same time, the correlation index between each feature dimension and lung cancer diagnosis is calculated, with higher correlation indicating greater reference value for clinical diagnosis; combining these two indicators, key diagnostic feature dimensions such as lesion morphology changes in chest CT images, abnormal fluctuations in heart rate and blood oxygenation physiological signals, symptom aggravation descriptions in follow-up text reports, and redundant feature dimensions are clearly distinguished, ultimately forming a feature importance distribution map that clearly presents the importance of each feature dimension.

[0092] Step 5.2: Based on the feature importance distribution map, an adaptive projection dimensionality reduction algorithm is used to project the high-dimensional semantic representation to a low-dimensional subspace to obtain a preliminary compressed representation. Specifically, this includes: using the feature importance distribution map as a basis, an adaptive projection dimensionality reduction algorithm is used to compress the high-dimensional semantic representation; the algorithm prioritizes retaining the feature dimensions with high importance in the map, focusing on retaining the core information closely related to lung cancer diagnosis, while reasonably deleting or merging redundant feature dimensions with lower importance. Without losing key diagnostic information, the original high-dimensional semantic representation is accurately projected to a low-dimensional subspace, significantly reducing the dimensionality and storage volume of the data, and finally obtaining a preliminary compressed representation with simplified dimensions and complete core information.

[0093] Step 5.3: Based on the preliminary compressed representation, the cross-modal semantic consistency of the preliminary compressed representation is verified and fine-tuned through a cross-modal association preservation mechanism to finally obtain the compressed representation. Specifically, this includes: based on the preliminary compressed representation, initiating the cross-modal association preservation mechanism to verify the cross-modal semantic consistency of the compressed features; checking the correlation between the relevant feature dimensions of the chest CT image and the corresponding symptom description feature dimensions in the follow-up text report, as well as the correlation between the physiological signal feature dimensions and the image lesion change feature dimensions, to ensure that the compression process does not destroy key cross-modal semantic associations such as lesion enlargement, cough aggravation, and abnormal physiological indicators; and making targeted fine-tuning for the association weakening problems found in the verification to strengthen the expression of core association features, ultimately obtaining a compressed representation with significantly reduced storage size, complete retention of key diagnostic information, and stable cross-modal semantic associations.

[0094] In this embodiment of the invention, a feature importance distribution map is obtained by calculating the information entropy and diagnostic relevance index of each feature dimension based on the optimized semantic representation and the feature dimension importance analysis algorithm. Based on this map, the high-dimensional semantic representation is projected to the low-dimensional subspace using an adaptive projection dimensionality reduction algorithm to obtain a preliminary compressed representation. Then, the preliminary compressed representation is checked and fine-tuned for cross-modal semantic consistency through a cross-modal association preservation mechanism. Therefore, the technical problems of easy loss of key diagnostic features, easy destruction of cross-modal semantic association, and high storage costs due to lack of specificity in high-dimensional data compression are overcome in the process of multimodal long sequence data compression. Thus, the technical effect of accurately retaining core diagnostic information and stable cross-modal semantic association while significantly reducing data dimensionality and storage costs is achieved, ultimately obtaining a high-quality compressed representation that combines compression efficiency and diagnostic value.

[0095] In a preferred embodiment of the present invention, step 6 above may include:

[0096] Step 6.1: Based on compressed representations, the compressed representations of current and historical cases are organized into an index dataset. A cross-modal index library supporting similarity retrieval is constructed using a vector indexing algorithm. Specifically, this includes: collecting compressed representations of current lung cancer patients and all patients in the historical lung cancer case database, based on existing compressed representations with low storage volume, complete cross-modal semantic associations, and stable data. These compressed representations are then organized into a standardized index dataset according to a unified data format. Subsequently, the vector indexing algorithm is used to process this dataset, constructing a cross-modal index library capable of supporting rapid similarity retrieval of cross-modal data. The index library efficiently associates compressed representations of different cases.

[0097] Step 6.2: Based on the cross-modal index, when a retrieval query containing multimodal data is received, a pre-trained multimodal large model is invoked to map each modality of data in the retrieval query to the same latent space as the compressed representation, thus obtaining a query semantic representation. Specifically, when the cross-modal index is in standby mode, if a retrieval query containing multimodal data is received, and the query data is the daily heart rate and blood oxygen physiological signals and weekly follow-up text reports of a 6-month chest CT image sequence of a patient to be diagnosed with lung cancer, the pre-set pre-trained multimodal large model is immediately invoked to perform parsing feature extraction and semantic mapping on each modality of data in the retrieval query, accurately mapping it to the same latent space as the compressed representation of historical cases, ensuring that the semantic expression basis of the query data is consistent with that of the data in the index, and finally obtaining a query semantic representation that comprehensively reflects the condition of the queried patient.

[0098] Step 6.3: Based on the query semantic representation, the query semantic representation is compressed using an adaptive compression algorithm to obtain a compressed query representation. Specifically, this includes: using the obtained query semantic representation as the processing object, calling an adaptive compression algorithm consistent with the historical case compression process to perform targeted compression processing on the query semantic representation; during the compression process, the algorithm prioritizes retaining key feature dimensions closely related to lung cancer diagnosis in the query semantic representation, such as lesion features in the patient's CT images, abnormal fluctuation features of physiological signals, and symptom description features in the text report, while reasonably reducing redundant dimensions, reducing data dimensions without destroying cross-modal semantic association, and finally obtaining a compressed query representation with complete core information and a format consistent with the compressed representation of historical cases.

[0099] Step 6.4: Based on the query compressed representation, perform an approximate nearest neighbor search through a cross-modal index to calculate the similarity between the query compressed representation and the compressed representations of historical cases in the index, in order to obtain the similarity result. Specifically, this includes: inputting the query compressed representation into the cross-modal index, starting the approximate nearest neighbor search mechanism, and quickly traversing the compressed representations of all historical lung cancer cases in the index; by calculating the similarity between the query compressed representation and the compressed representation of each historical case, focusing on the degree of fit between the two in terms of key diagnostic features, cross-modal semantic association, key temporal interval features, etc., such as the matching degree between the lesion enlargement feature of the query patient and the corresponding feature of the historical case, the consistency of the association between the description of symptom aggravation and abnormal physiological indicators, etc., and finally obtaining the similarity result between the query compressed representation and the compressed representations of all historical cases.

[0100] Step 6.5: Based on the similarity results, the candidate historical cases are sorted in descending order of similarity. The most similar historical case is selected as the final search result. Specifically, this includes: sorting all candidate historical cases in descending order of similarity based on the calculated similarity results; prioritizing cases that highly match the query patient in terms of cross-modal correlation of key diagnostic features and disease evolution trends, such as historical cases with similar lesion change trajectories and consistent cross-modal correlation of disease deterioration features in key time intervals with the query patient; after sorting, the historical case with the highest similarity is selected as the final search result, providing accurate reference for clinicians to diagnose the query patient's condition.

[0101] In this embodiment of the invention, by employing a cross-modal index library containing current and historical cases based on compressed representation, mapping the multimodal data of the retrieval query to the same latent space and compressing it into a query compressed representation, and using the index library to perform approximate nearest neighbor search to calculate similarity and sort by similarity, the technical problems in existing retrieval methods are overcome, such as the inconsistency between the modal spaces of multimodal queries and historical case data, the lack of cross-modal indexes leading to low retrieval efficiency, and the mismatch between key associations and core information leading to poor retrieval accuracy. This achieves the technical effect of realizing efficient cross-modal similarity matching between multimodal queries and historical cases, ensuring that the retrieval results retain both key diagnostic information and cross-modal associations while having a high degree of matching, and ultimately providing similar case references for clinical diagnosis.

[0102] like Figure 2 As shown, embodiments of the present invention also provide a multimodal large-model long-sequence information compression and retrieval system, including:

[0103] The acquisition module is used to receive multimodal long sequence raw data of the target patient; through a pre-trained multimodal large model, the modal data in the multimodal long sequence raw data are uniformly mapped to the same latent space to obtain a unified semantic representation sequence of semantic association between modalities;

[0104] The computation module is used to perform temporal feature analysis on the unified semantic representation sequence. It treats the feature points in the semantic representation as a set of spatial points, constructs the convex hull of each feature cluster to determine the boundary distribution, identifies the key feature clusters in the semantic representation, and determines the key temporal intervals based on the convex hull distribution pattern of the key feature clusters.

[0105] The optimization module is used to analyze the feature evolution law based on key time intervals to construct feature change trajectories and obtain feature enhancement parameters based on the feature change trajectories. The feature enhancement parameters are applied to the feature enhancement processing of the unified semantic representation sequence. By highlighting key diagnostic features and maintaining cross-modal correlation, the optimized semantic representation is obtained. Based on the optimized semantic representation, an adaptive compression algorithm is used to generate a compressed representation that retains key diagnostic information and cross-modal correlation.

[0106] The processing module is used to construct a cross-modal index library that integrates multimodal semantics based on compressed representations. When a multimodal retrieval query is received, the query is mapped to the same latent space, and similarity matching is performed through the cross-modal index library to retrieve historical cases similar to the target patient from the historical case library.

[0107] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for compressing and retrieving long sequence information in a multimodal large model, characterized in that: The method includes: Receive multimodal long sequence raw data from the target patient; through a pre-trained multimodal large model, map each modality data in the multimodal long sequence raw data to the same latent space to obtain a unified semantic representation sequence of semantic association between modalities; Temporal feature analysis is performed on the unified semantic representation sequence. The feature points in the semantic representation are treated as a set of spatial points. The convex hulls of each feature cluster are constructed to determine the boundary distribution, identifying key feature clusters in the semantic representation. Based on the convex hull distribution patterns of these key feature clusters, key temporal intervals are determined, including: Based on the unified semantic representation sequence, each semantic representation vector in the sequence is regarded as a feature point in a high-dimensional feature space, and the spatial distribution of feature points is obtained. Based on the spatial distribution of feature points, clustering algorithms are used to perform cluster analysis on the feature points to obtain multiple feature clusters with similar semantic attributes; Based on multiple feature clusters, the process is performed for each feature cluster, calculating the convex hull boundary of each feature cluster in the high-dimensional feature space to obtain the boundary distribution representation of each feature cluster. Based on the boundary distribution representation of each feature cluster, by analyzing the geometric characteristics and spatial distribution relationship of each convex hull, at least one key feature cluster representing key clinical semantics is identified. Based on key feature clusters, feature points in the key feature clusters are mapped back to the original time series. Based on the distribution density and continuity of the original time series on the time axis, the key time series intervals are obtained. Based on key time intervals, the evolution law of features is analyzed to construct feature change trajectories, and feature enhancement parameters are obtained based on the feature change trajectories; By applying feature enhancement parameters to the feature enhancement process of the unified semantic representation sequence, the optimized semantic representation is obtained by highlighting key diagnostic features and maintaining cross-modal correlation. Based on the optimized semantic representation, an adaptive compression algorithm is used to generate a compressed representation that retains key diagnostic information and cross-modal correlation. A cross-modal index library integrating multimodal semantics is constructed based on compressed representations. When a multimodal retrieval query is received, the query is mapped to the same latent space, and similarity matching is performed through the cross-modal index library to retrieve historical cases similar to the target patient from the historical case library.

2. The multimodal large-model long sequence information compression and retrieval method according to claim 1, characterized in that, Receive multimodal long-sequence raw data from the target patient; using a pre-trained multimodal large model, map each modality of the multimodal long-sequence raw data to the same latent space to obtain a unified semantic representation sequence of intermodal semantic associations, including: Receive long-sequence raw data of multiple modalities from the target patient; the long-sequence raw data of multiple modalities includes at least text report sequences, medical image sequences, and physiological signal time series; Based on the original multimodal long sequence data, a pre-trained multimodal large model is called to parse and align the original data into different modes to obtain a set of multimodal data segments with clear temporal correspondence. Based on a set of multimodal data fragments, a pre-trained multimodal large model is driven to extract features from each modal data fragment and map the extracted features to a shared latent space to obtain an initial set of modal feature vectors. Based on the initial set of modal feature vectors, the cross-modal attention fusion module in the pre-trained multimodal large model interacts and integrates the initial modal feature vectors to strengthen the semantic association between modalities, and finally obtains a unified semantic representation sequence.

3. The multimodal large-model long sequence information compression and retrieval method according to claim 2, characterized in that, Based on key time intervals, the evolution patterns of features are analyzed to construct feature change trajectories, and feature enhancement parameters are obtained based on these trajectories, including: Based on key time intervals, semantic representation subsequences corresponding to the time intervals are extracted from the unified semantic representation sequence to obtain key feature sequences; Based on key feature sequences, the evolution trend model of semantic representations of consecutive time steps in the sequence is modeled by time-series pattern analysis algorithm in order to construct feature change trajectories that represent dynamic changes in features. Based on the feature change trajectory, the importance of features at different time points within the key time series interval is quantified by calculating the trajectory curvature change rate and feature point distribution density, so as to obtain the initial feature weight distribution. Based on the initial feature weight distribution, the weight distribution is corrected and normalized by combining the preset clinical diagnostic prior knowledge, and finally the feature enhancement parameters are obtained.

4. The multimodal large-model long sequence information compression and retrieval method according to claim 3, characterized in that, By applying feature enhancement parameters to the feature enhancement process of the unified semantic representation sequence, and highlighting key diagnostic features while maintaining cross-modal correlations, the optimized semantic representation is obtained, including: Based on the feature enhancement parameters and the unified semantic representation sequence, the feature enhancement parameters are applied to the unified semantic representation sequence, and a preliminary enhanced feature mapping is obtained through a weighted transformation operation; Based on the initial enhanced feature mapping, the enhanced features are regularized using a cross-modal association constraint algorithm to obtain cross-modal consistency enhanced features; The cross-modal consistency enhancement features and the initially enhanced feature maps are subjected to residual connection and layer normalization to obtain the optimized semantic representation.

5. The multimodal large-model long sequence information compression and retrieval method according to claim 4, characterized in that, Based on the optimized semantic representation, an adaptive compression algorithm is used to generate a compressed representation that preserves key diagnostic information and cross-modal correlations, including: Based on the optimized semantic representation, the information entropy and diagnostic relevance index of each feature dimension in the semantic representation are calculated by the feature dimension importance analysis algorithm, and the feature importance distribution map is obtained. Based on the feature importance distribution map, an adaptive projection dimensionality reduction algorithm is used to project the high-dimensional semantic representation to a low-dimensional subspace to obtain a preliminary compressed representation. Based on the initial compressed representation, the cross-modal association preservation mechanism is used to perform cross-modal semantic consistency verification and fine-tuning on the initial compressed representation, and finally the compressed representation is obtained.

6. The multimodal large-model long sequence information compression and retrieval method according to claim 5, characterized in that, A cross-modal index library integrating multimodal semantics is constructed based on compressed representations. When a multimodal retrieval query is received, the query is mapped to the same latent space, and similarity matching is performed through the cross-modal index library to retrieve historical cases similar to the target patient from the historical case database, including: Based on compressed representation, the compressed representations of current and historical cases are organized into an index dataset, and a cross-modal index library supporting similarity retrieval is constructed using a vector indexing algorithm. Based on the cross-modal index library, when a retrieval query containing multimodal data is received, a pre-trained multimodal large model is invoked to map each modal data in the retrieval query to the same latent space as the compressed representation, thereby obtaining the query semantic representation. Based on the query semantic representation, the query semantic representation is compressed using an adaptive compression algorithm to obtain the query compressed representation; Based on the query compressed representation, an approximate nearest neighbor search is performed across modal indexes to calculate the similarity between the query compressed representation and the historical case compressed representation in the indexes, so as to obtain the similarity results; Based on the similarity results, candidate historical cases are sorted in descending order of similarity, and the most similar historical case is taken as the final search result.

7. A multimodal large-model long-sequence information compression and retrieval system, wherein the system implements the method as described in any one of claims 1 to 6, characterized in that, include: The acquisition module is used to receive multimodal long sequence raw data of the target patient; through a pre-trained multimodal large model, the modal data in the multimodal long sequence raw data are uniformly mapped to the same latent space to obtain a unified semantic representation sequence of semantic association between modalities; The computation module performs temporal feature analysis on the unified semantic representation sequence. It treats feature points in the semantic representation as a set of spatial points, constructs the convex hulls of each feature cluster to determine boundary distribution, identifies key feature clusters in the semantic representation, and determines key temporal intervals based on the convex hull distribution patterns of the key feature clusters. This includes: treating each semantic representation vector in the sequence as a feature point in a high-dimensional feature space to obtain the spatial distribution of feature points; performing clustering analysis on the feature points based on the spatial distribution of feature points to obtain multiple feature clusters with similar semantic attributes; calculating the convex hull boundary of each feature cluster in the high-dimensional feature space for each feature cluster to obtain the boundary distribution representation of each feature cluster; identifying at least one key feature cluster representing key clinical semantics by analyzing the geometric characteristics and spatial distribution relationship of each convex hull based on the boundary distribution representation of each feature cluster; and mapping the feature points in the key feature clusters back to the original temporal sequence, obtaining the key temporal intervals based on the distribution density and continuity of the original temporal sequence on the time axis. The optimization module is used to analyze the feature evolution law based on key time intervals to construct feature change trajectories and obtain feature enhancement parameters based on the feature change trajectories. The feature enhancement parameters are applied to the feature enhancement processing of the unified semantic representation sequence. By highlighting key diagnostic features and maintaining cross-modal correlation, the optimized semantic representation is obtained. Based on the optimized semantic representation, an adaptive compression algorithm is used to generate a compressed representation that retains key diagnostic information and cross-modal correlation. The processing module is used to construct a cross-modal index library that integrates multimodal semantics based on compressed representations. When a multimodal retrieval query is received, the query is mapped to the same latent space, and similarity matching is performed through the cross-modal index library to retrieve historical cases similar to the target patient from the historical case library.

8. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images