Processing method and device of cross-modality medical signals, electronic equipment and storage medium

Abstract

The application provides a processing method and device for cross-modal medical signals, an electronic device and a storage medium, which comprises the following steps: inputting a plurality of original medical physiological signals collected through a plurality of first leads into a medical signal basic large model, performing time sequence feature extraction and semantic feature extraction on each original medical physiological signal based on a mask auto-encoder network layer and a contrastive learning self-supervised network layer to determine a first feature; inputting a new modality medical physiological signal collected through a plurality of second leads for a downstream processing task into the medical signal basic large model to determine a plurality of second features; performing cross-modal similarity matching processing on the plurality of first features and the plurality of second features to determine a cross-modal splicing feature, so that the cross-modal splicing feature is used for automatic processing of the downstream processing task. Through the flexible configuration and precise local adaptation mechanism of multi-path pre-training, the cross-modal analysis efficiency and model generalization ability are significantly improved.

Description

Technical Field

[0001] This application relates to the field of medical signal processing technology, and in particular to methods, apparatus, electronic devices and storage media for processing cross-modal medical signals. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, medical AI and multimodal physiological signal processing have gradually become research hotspots. Physiological signals such as electrocardiograms (ECG), electroencephalograms (EEG), and electromyograms (EMG) are important sources of information reflecting human health status, and the fusion analysis of multimodal signals can more comprehensively reveal the body's physiological dynamics. However, traditional methods are often designed only for a single modality, making it difficult to take into account the differences in sampling frequency, time-frequency characteristics, and noise distribution of different signals, and also failing to achieve efficient semantic alignment across modalities. Meanwhile, existing large models are usually pre-trained on only a limited number of basic modalities, unable to directly transfer or adapt to newly introduced signal types (such as gait and blood oxygenation), which not only increases the cost of secondary development but also severely restricts the system's scalability. Therefore, there is an urgent need for a new general method that can build large models on a limited number of basic modalities and achieve compatibility and efficient fusion of all physiological signal modalities through a flexible adaptation mechanism, in order to overcome the bottlenecks of existing technologies in representation learning, semantic alignment, and small-sample transfer. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method, device, electronic device and storage medium for processing cross-modal medical signals, which can form a general representation capability through self-supervised pre-training of a large model, and can be quickly transferred to multiple physiological signals during surgery. Through the flexible configuration of multi-path pre-training and the precise local adaptation mechanism, the efficiency of cross-modal analysis and the generalization ability of the model can be significantly improved.

[0004] This application provides a method for processing cross-modal medical signals, the method comprising: Multiple raw medical physiological signals acquired through multiple first leads are input into a large medical signal basic model. Based on the mask autoencoder network layer and the contrastive learning self-supervised network layer, temporal and semantic features are extracted for each raw medical physiological signal to determine the first feature of each raw medical physiological signal. The new modal medical physiological signals acquired through multiple second leads in the downstream processing task are input into the basic large model of medical signals to determine multiple second features of the new modal medical physiological signals; wherein, the modality of the new modal medical physiological signals is inconsistent with the modality of the original medical physiological signals; Cross-modal similarity matching is performed on multiple first features and multiple second features to determine cross-modal splicing features, so that downstream processing tasks can be automatically processed using cross-modal splicing features.

[0005] In one possible implementation, the mask-based autoencoder network layer and the contrastive learning self-supervised network layer perform temporal and semantic feature extraction on each of the original medical physiological signals to determine a first feature of each of the original medical physiological signals, including: The masked autoencoder network layer performs slicing, random masking, and reconstruction processing on each of the original medical physiological signals to determine the temporal characteristics of each of the original medical physiological signals. The contrastive learning self-supervised network layer uses a shared-weight encoder to extract features from each of the original medical physiological signals, thereby determining the semantic features of each of the original medical physiological signals. Based on the temporal features and semantic features, the first feature of the original medical physiological signal is determined.

[0006] In one possible implementation, the step of performing slicing, random masking, and reconstruction processing on each of the original medical physiological signals based on the masked autoencoder network layer to determine the temporal characteristics of each of the original medical physiological signals includes: The original medical physiological signals were processed by signal slicing to identify multiple signal segments; Random masking is performed on multiple signal segments to determine the visible signal segments and the masked segments; The visible signal segment is subjected to feature encoding processing to determine the encoded features; Based on the encoded features and the positional information of the mask fragment, reconstruction processing is performed to determine the temporal characteristics of the original medical physiological signal.

[0007] In one possible implementation, the cross-modal similarity matching process performed on a plurality of first features and a plurality of second features to determine cross-modal splicing features includes: Determine the feature similarity between each of the first features and each of the second features; The average similarity matrix formed by the multiple feature similarities is averaged to determine the average similarity matrix between each first lead and each second lead; For any second lead, the optimal lead matching pair for the second lead is determined based on the maximum average similarity matrix of the second lead; wherein the optimal lead matching pair includes the second lead and the optimal first lead that matches the second lead; Based on the optimal lead matching of each second lead, the cosine similarity is averaged in the dimension, and the second feature corresponding to the maximum value after the dimension averaging is taken as the optimal second feature. The optimal second feature and the first feature under the optimal first lead are concatenated to determine the cross-modal concatenation feature.

[0008] In one possible implementation, the basic large model of the medical signal is determined through the following steps: The medical and physiological signals of the sample are processed based on the initial mask autoencoder network layer to determine the reconstructed medical and physiological signals of the sample. The first loss value of the initial mask autoencoder network layer is determined based on the sample medical physiological signal and the reconstructed sample medical physiological signal. Based on the initial contrastive learning self-supervised network layer, feature extraction is performed on two segments of the sample medical physiological signal. The features of the two segments are then processed based on the contrastive loss function to determine the second loss value of the initial contrastive learning self-supervised network layer. Based on the first loss value and the second loss value, the network parameters of the initial mask autoencoder network layer and the initial contrastive learning self-supervised network layer are adjusted to determine the basic large model of the medical signal.

[0009] In one possible implementation, before inputting the multiple novel modal medical physiological signals acquired via the second lead from the downstream processing task into the medical signal basic model and determining the second feature of the novel modal medical physiological signals, the processing method further includes: During the downstream processing task adaptation phase, the network parameters of the basic large model of medical signals are adjusted using any one of three methods: full parameter fine-tuning, lightweight adapter fine-tuning, and freezing feature extractor parameters.

[0010] This application also provides a cross-modal medical signal processing device, the processing device comprising: The first feature extraction module is used to input multiple raw medical physiological signals collected through multiple first leads into the basic medical signal model, and to extract temporal and semantic features for each raw medical physiological signal based on the mask autoencoder network layer and the contrastive learning self-supervised network layer, so as to determine the first feature of each raw medical physiological signal. The second feature extraction module is used to input the new modal medical physiological signals acquired by the downstream processing task through multiple second leads into the basic large model of medical signals, and determine multiple second features of the new modal medical physiological signals; wherein the modality of the new modal medical physiological signals is inconsistent with the modality of the original medical physiological signals; The cross-modal extraction module is used to perform cross-modal similarity matching on multiple first features and multiple second features to determine cross-modal splicing features, so as to automatically process downstream processing tasks using cross-modal splicing features.

[0011] In one possible implementation, when the first feature extraction module performs temporal and semantic feature extraction on each of the original medical physiological signals using the mask-based autoencoder network layer and the contrastive learning self-supervised network layer to determine the first feature of each of the original medical physiological signals, the first feature extraction module is specifically used for: The masked autoencoder network layer performs slicing, random masking, and reconstruction processing on each of the original medical physiological signals to determine the temporal characteristics of each of the original medical physiological signals. The contrastive learning self-supervised network layer uses a shared-weight encoder to extract features from each of the original medical physiological signals, thereby determining the semantic features of each of the original medical physiological signals. Based on the temporal features and semantic features, the first feature of the original medical physiological signal is determined.

[0012] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the cross-modal medical signal processing method described above are performed.

[0013] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the cross-modal medical signal processing method described above.

[0014] This application provides a method, apparatus, electronic device, and storage medium for processing cross-modal medical signals. The processing method includes: inputting multiple raw medical physiological signals acquired through multiple first leads into a large-scale medical signal model; extracting temporal and semantic features from each raw medical physiological signal based on a masked autoencoder network layer and a contrastive learning self-supervised network layer to determine a first feature of each raw medical physiological signal; inputting new modal medical physiological signals acquired through multiple second leads from a downstream processing task into the large-scale medical signal model to determine multiple second features of the new modal medical physiological signals; wherein the modality of the new modal medical physiological signals is inconsistent with the modality of the raw medical physiological signals; performing cross-modal similarity matching processing on the multiple first features and the multiple second features to determine cross-modal splicing features, so as to automatically process the downstream processing task using the cross-modal splicing features. By forming a general representation capability through self-supervised pre-training of a large model, it can be quickly transferred to multiple physiological signals during surgery. Through the flexible configuration of multi-path pre-training and the precise local adaptation mechanism, the efficiency of cross-modal analysis and the model generalization ability are significantly improved.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for processing cross-modal medical signals provided in an embodiment of this application; Figure 2 This is one of the structural schematic diagrams of a cross-modal medical signal processing device provided in an embodiment of this application; Figure 3 This is a second schematic diagram of a cross-modal medical signal processing device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0019] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of medical signal processing technology.

[0020] Research has revealed that with the rapid development of artificial intelligence (AI) technology, medical AI and multimodal physiological signal processing have gradually become research hotspots. Physiological signals such as electrocardiograms (ECG), electroencephalograms (EEG), and electromyograms (EMG) are important sources of information reflecting human health status, and the fusion analysis of multimodal signals can more comprehensively reveal the body's physiological dynamics. However, traditional methods are often designed only for a single modality, making it difficult to consider the differences in sampling frequency, time-frequency characteristics, and noise distribution among different signals, and also failing to achieve efficient semantic alignment across modalities. Meanwhile, existing large models are typically pre-trained on only a limited number of basic modalities, unable to directly transfer or adapt to newly introduced signal types (such as gait and blood oxygenation), which not only increases the cost of secondary development but also severely restricts the system's scalability. Therefore, there is an urgent need for a new general method that can build large models on a limited number of basic modalities and achieve compatibility and efficient fusion of all physiological signal modalities through flexible adaptation mechanisms, in order to overcome the bottlenecks of existing technologies in representation learning, semantic alignment, and small-sample transfer.

[0021] Based on this, the embodiments of this application provide a method for processing cross-modal medical signals. By forming a general representation capability through self-supervised pre-training of a large model, it can be quickly transferred to multiple physiological signals during surgery. Through the flexible configuration of multi-path pre-training and the precise local adaptation mechanism, the efficiency of cross-modal analysis and the generalization ability of the model are significantly improved.

[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for processing cross-modal medical signals provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the processing method includes: S101: Input multiple raw medical physiological signals acquired through multiple first leads into the basic medical signal model. Based on the mask autoencoder network layer and the contrastive learning self-supervised network layer, extract temporal features and semantic features for each raw medical physiological signal to determine the first feature of each raw medical physiological signal.

[0023] In this step, multiple raw medical physiological signals collected through multiple first leads are input into the basic medical signal model. Based on the masked autoencoder network layer and the contrastive learning self-supervised network layer, temporal and semantic features are extracted from each raw medical physiological signal to determine the first feature of each raw medical physiological signal.

[0024] Here, the original medical physiological signals may include electrocardiogram signals, electroencephalogram signals, electromyogram signals, and other physiological signals that reflect the human body's health status.

[0025] In one possible implementation, the mask-based autoencoder network layer and the contrastive learning self-supervised network layer perform temporal and semantic feature extraction on each of the original medical physiological signals to determine a first feature of each of the original medical physiological signals, including: A: The masked autoencoder network layer performs slicing, random masking, and reconstruction processing on each of the original medical physiological signals to determine the temporal characteristics of each of the original medical physiological signals.

[0026] Here, the masked autoencoder network layer performs slicing, random masking, and reconstruction processing on each original medical physiological signal to determine the temporal characteristics of each original medical physiological signal.

[0027] In one possible implementation, the step of performing slicing, random masking, and reconstruction processing on each of the original medical physiological signals based on the masked autoencoder network layer to determine the temporal characteristics of each of the original medical physiological signals includes: (1): The original medical physiological signal was processed by signal slicing to identify multiple signal segments.

[0028] (2): Randomly mask multiple signal segments to determine visible signal segments and masked segments; perform feature encoding on the visible signal segments to determine the encoded features.

[0029] Here, intramodal self-supervised feature modeling is achieved through signal slicing, random masking, and reconstruction learning. Let the original one-dimensional physiological signal be... Divide it into A length of The non-overlapping segments are used to obtain the input sequence. , in The channel dimension at each time point. For which... The proportion of segments is randomly masked to divide them into visible signal segments. With mask fragment Then the encoder Extracting the encoded features of visible segments :

[0030] (3): Based on the coding features and the position information of the mask fragment, the temporal features of the original medical physiological signal are determined.

[0031] Here, the decoder Based on coding features and mask position Reconstructing the temporal characteristics of original medical physiological signals :

[0032] B: The contrastive learning self-supervised network layer uses a shared-weight encoder to extract features from each of the original medical physiological signals, thereby determining the semantic features of each of the original medical physiological signals.

[0033] Here, the contrastive learning self-supervised network layer uses a shared-weight encoder to extract features from each original medical physiological signal, thereby determining the semantic features of each original medical physiological signal.

[0034] C: Based on the temporal features and semantic features, the first feature of the original medical physiological signal is determined.

[0035] Here, based on temporal and semantic features, the first feature of the original medical physiological signal is determined.

[0036] S102: Input the new modal medical physiological signals acquired by the downstream processing task through multiple second leads into the basic large model of medical signals, and determine multiple second features of the new modal medical physiological signals; wherein, the modality of the new modal medical physiological signals is inconsistent with the modality of the original medical physiological signals.

[0037] In this step, the new modal medical physiological signals collected by multiple second leads from the downstream processing task are input into the large medical signal basic model to determine multiple second features of the new modal medical physiological signals.

[0038] Among them, the modalities of the new modal medical physiological signals are different from those of the original medical physiological signals. The original medical physiological signals are signals that serve as a sample of the basic large model of medical signals.

[0039] S103: Perform cross-modal similarity matching on multiple first features and multiple second features to determine cross-modal splicing features, so as to automatically process downstream processing tasks using cross-modal splicing features.

[0040] In this step, cross-modal similarity matching is performed on multiple first features and multiple second features to determine cross-modal splicing features, so that downstream processing tasks can be automatically processed using cross-modal splicing features.

[0041] In one possible implementation, the cross-modal similarity matching process performed on a plurality of first features and a plurality of second features to determine cross-modal splicing features includes: a: Determine the feature similarity between each of the first features and each of the second features.

[0042] Here, each first feature is calculated. With the second feature Feature similarity is calculated using cosine similarity at the average token level:

[0043] in, The first feature The features at time step t, where T is the total number of time steps for the features. The first feature i With the second feature j Cosine similarity between them.

[0044] b: Average the similarity matrices formed by the multiple feature similarities to determine the average similarity matrix between each first lead and each second lead.

[0045] Here, we obtain the similarity matrix. The number of samples in both datasets. Simultaneously, the average is taken to obtain the similarity matrix for cross-sample aggregation. Where M is the set of the first feature and L is the set of the second feature.

[0046] c: For any second lead, determine the optimal lead matching pair for the second lead based on the maximum average similarity matrix of the second lead; wherein the optimal lead matching pair includes the second lead and the optimal first lead that matches the second lead.

[0047] Here, for any second lead, the optimal lead match for that second lead is determined based on the maximum average similarity matrix of that second lead.

[0048] d: Based on the optimal lead matching of each second lead, the average similarity matrix corresponding to it is averaged in terms of dimensions, and the second feature corresponding to the maximum value after the dimensionality averaging is taken as the optimal second feature.

[0049] Here, the average similarity matrix corresponding to the optimal lead matching pair of each second lead is averaged according to its dimensions, and the second feature corresponding to the maximum value after averaging is taken as the optimal second feature.

[0050] Here, for each optimal lead matching pair, its corresponding sub-similarity matrix is ​​extracted. Averaging across all dimensions except the sample dimension:

[0051] in, The cosine similarity is calculated after averaging the dimensions, based on... Sort by size, and select the second feature corresponding to the maximum value as the optimal second feature.

[0052] e: Perform feature concatenation between the optimal second feature and the first feature under the optimal first lead to determine the cross-modal concatenation feature.

[0053] Here, the optimal second feature and the first feature under the optimal first lead are concatenated to determine the cross-modal concatenation feature.

[0054] In one possible implementation, the basic large model of the medical signal is determined through the following steps: I: The medical and physiological signals of the sample are processed based on the initial mask autoencoder network layer to determine the reconstructed medical and physiological signals of the sample.

[0055] II: Determine the first loss value of the initial mask autoencoder network layer based on the sample medical physiological signal and the reconstructed sample medical physiological signal.

[0056] III: Based on the initial contrastive learning self-supervised network layer, feature extraction is performed on two segments of the sample medical physiological signal. The features of the two segments are processed based on the contrastive loss function to determine the second loss value of the initial contrastive learning self-supervised network layer.

[0057] Here, features are extracted from two segments of the sample medical physiological signal based on the initial contrastive learning self-supervised network layer, and the features of the two segments are processed according to the contrastive loss function to determine the second loss value of the initial contrastive learning self-supervised network layer.

[0058] Specifically, a self-supervised training path based on contrastive learning is constructed to enhance the large model's ability to perceive temporal patterns and align individual semantics within physiological signals of the same modality. This path utilizes large-scale unlabeled data, constructing positive and negative sample pairs to guide the model in learning feature convergence between similar signals and distinguishing between dissimilar signals. The input consists of two segments of signals from the same modality. Encoders using shared weights Extract its features , express:

[0059] in Positive sample pairs are formed (e.g., those from the same state or a recent time period), and the remaining samples are negative samples.

[0060] Adopting based on Contrast loss function:

[0061] in Represents cosine similarity. For temperature coefficient, This loss represents the features of other samples in a mini-batch. By minimizing this loss, the model can learn highly discriminative semantic feature representations in unsupervised scenarios, improving the robustness and generalization ability of signal representation across different states or individuals, and providing a stable foundation for subsequent cross-modal matching and dynamic splicing. This is the second loss value.

[0062] IV: Based on the first loss value and the second loss value, adjust the network parameters of the initial mask autoencoder network layer and the initial contrastive learning self-supervised network layer to determine the basic large model of the medical signal.

[0063] Here, the network parameters of the initial mask autoencoder network layer and the initial contrastive learning self-supervised network layer are adjusted based on the first loss value and the second loss value to determine the basic large model of medical signals.

[0064] In one possible implementation, before inputting the multiple novel modal medical physiological signals acquired via the second lead from the downstream processing task into the medical signal basic model and determining the second feature of the novel modal medical physiological signals, the processing method further includes: During the downstream processing task adaptation phase, the network parameters of the basic large model of medical signals are adjusted using any one of three methods: full parameter fine-tuning, lightweight adapter fine-tuning, and freezing feature extractor parameters.

[0065] Here, the full-parameter fine-tuning mode is suitable for scenarios with sufficient labeled data and highly customized task requirements. This method loads a pre-trained MAE or contrastive learning model, opens all network layer parameters for end-to-end optimization, and in this mode, all parameters of the model participate in backpropagation and optimization.

[0066] Lightweight adapter fine-tuning, specifically designed for small-sample or privacy-sensitive scenarios such as primary healthcare institutions, employs a Low-Rank Adaptation (LoRA) fine-tuning strategy. This involves performing alternative low-rank reconstruction training on the linear transformation layers in the large model, freezing the original model parameters, maintaining the original weights, and training only the newly added low-rank matrix. This method significantly reduces training costs and is suitable for scenarios with small sample sizes or high deployment flexibility requirements. It can significantly reduce training costs while preserving the general representational capabilities of the pre-trained model.

[0067] In zero-sample or cross-modal retrieval scenarios, a feature freeze extraction mode is adopted. The encoder parameters of the MAE or contrastive learning model are completely frozen, and pre-trained features are directly extracted for downstream tasks.

[0068] In this application, a general processing system for multimodal physiological signals such as ECG, EEG, EMG, and gait kinematics is constructed by employing a dual-path pre-training framework of MAE (Multimodal Analysis) and contrastive learning, trained independently on finite modalities, a hierarchical downstream task adaptation mechanism, and a sample similarity-driven cross-modal dynamic stitching technique. The system flow is as follows: In the pre-training stage, the MAE path performs random masking and reconstruction training on local key segments of single-modal physiological signals to learn fine-grained feature representations within the modality; the contrastive learning path constructs positive and negative sample pairs on synchronously acquired multimodal physiological signals, reducing semantic differences between heterogeneous data through a feature distribution alignment loss function between modalities; in the downstream task adaptation stage, a hierarchical optimization strategy is designed to support three modes: full parameter fine-tuning, lightweight adapter fine-tuning, and zero-shot inference with frozen feature extractors, meeting the needs of different data scale scenarios; to address the bottleneck of cross-modal interaction, a dynamic stitching technique based on sample similarity matching is proposed, which dynamically selects highly similar regions for local feature stitching input by calculating the sample similarity matrix between modalities. This technology forms a general representation capability through self-supervised pre-training of a large model, which can be quickly transferred to scenarios such as real-time monitoring of multiple physiological signals during surgery and cross-modal health early warning of wearable devices. Through the flexible configuration of multi-path pre-training and the precise local adaptation mechanism, it significantly improves the efficiency of cross-modal analysis and the generalization ability of the model.

[0069] In this application, (1) a general large model is pre-trained in a limited number of basic modalities through self-supervision, which can learn the fine-grained temporal and frequency domain features of each signal and achieve cross-modal distribution alignment. This skeleton greatly reduces the cost of repeated training and maintenance for each physiological signal, ensures that the outputs of all modalities are consistent and comparable, and meets the strict requirements for real-time performance and resource consumption in wearable devices and intraoperative monitoring scenarios. (2) In the downstream task, the similarity matrix between samples of each modality is calculated in real time, and only highly correlated segments are spliced ​​and fused locally. This mechanism not only avoids the information redundancy and loss of key features in traditional static early / late fusion, but also supports the direct use of zero or few samples of new and unseen modalities. When a new physiological signal modality is added to the system, the most matching segment can be automatically selected and spliced ​​according to its similarity distribution with existing modalities, without additional annotation or retraining, ensuring fast access and efficient inference. (3) For diverse application scenarios ranging from high-performance servers to resource-constrained primary hospitals and home wearable devices, three calling modes are provided: full parameter fine-tuning, LoRA low-rank adaptation, and feature freezing extraction. This framework can unleash the full potential of large models in environments with sufficient data, and can also be quickly deployed in environments with few or zero samples, significantly reducing secondary development and maintenance costs.

[0070] This application provides a method for processing cross-modal medical signals. The method includes: inputting multiple raw medical physiological signals acquired through multiple first leads into a large-scale medical signal model; extracting temporal and semantic features from each raw medical physiological signal based on a masked autoencoder network layer and a contrastive learning self-supervised network layer to determine a first feature of each raw medical physiological signal; inputting new modal medical physiological signals acquired through multiple second leads from a downstream processing task into the large-scale medical signal model to determine multiple second features of the new modal medical physiological signals; wherein the modality of the new modal medical physiological signals is inconsistent with the modality of the raw medical physiological signals; performing cross-modal similarity matching on the multiple first features and the multiple second features to determine cross-modal splicing features, so as to automatically process the downstream processing task using the cross-modal splicing features.

[0071] Please see Figure 2 , Figure 3 , Figure 2 This is one of the structural schematic diagrams of a cross-modal medical signal processing device provided in an embodiment of this application; Figure 3 This is a second schematic diagram of a cross-modal medical signal processing device provided in an embodiment of this application. Figure 3 As shown, the processing device 200 includes: The first feature extraction module 210 is used to input multiple raw medical physiological signals collected through multiple first leads into the medical signal basic large model, and to extract temporal features and semantic features for each raw medical physiological signal based on the mask autoencoder network layer and the contrastive learning self-supervised network layer, so as to determine the first feature of each raw medical physiological signal. The second feature extraction module 220 is used to input the new modal medical physiological signals acquired by the downstream processing task through multiple second leads into the basic large model of medical signals, and determine multiple second features of the new modal medical physiological signals; wherein the modality of the new modal medical physiological signals is inconsistent with the modality of the original medical physiological signals; The cross-modal extraction module 230 is used to perform cross-modal similarity matching on multiple first features and multiple second features to determine cross-modal splicing features, so as to automatically process downstream processing tasks using cross-modal splicing features.

[0072] Furthermore, when the first feature extraction module 210 performs temporal and semantic feature extraction on each of the original medical physiological signals using the masked autoencoder network layer and the contrastive learning self-supervised network layer to determine the first feature of each of the original medical physiological signals, the first feature extraction module 210 is specifically used for: The masked autoencoder network layer performs slicing, random masking, and reconstruction processing on each of the original medical physiological signals to determine the temporal characteristics of each of the original medical physiological signals. The contrastive learning self-supervised network layer uses a shared-weight encoder to extract features from each of the original medical physiological signals, thereby determining the semantic features of each of the original medical physiological signals. Based on the temporal features and semantic features, the first feature of the original medical physiological signal is determined.

[0073] Furthermore, when the first feature extraction module 210 performs slicing, random masking, and reconstruction processing on each of the original medical physiological signals based on the masked autoencoder network layer to determine the temporal features of each of the original medical physiological signals, the first feature extraction module 210 is specifically used for: The original medical physiological signals were processed by signal slicing to identify multiple signal segments; Random masking is performed on multiple signal segments to determine the visible signal segments and the masked segments; The visible signal segment is subjected to feature encoding processing to determine the encoded features; Based on the encoded features and the positional information of the mask fragment, reconstruction processing is performed to determine the temporal characteristics of the original medical physiological signal.

[0074] Furthermore, when performing cross-modal similarity matching processing on multiple first features and multiple second features to determine cross-modal splicing features, the cross-modal extraction module 230 is specifically used for: Determine the feature similarity between each of the first features and each of the second features; The average similarity matrix formed by the multiple feature similarities is averaged to determine the average similarity matrix between each first lead and each second lead; For any second lead, the optimal lead matching pair for the second lead is determined based on the maximum average similarity matrix of the second lead; wherein the optimal lead matching pair includes the second lead and the optimal first lead that matches the second lead; Based on the optimal lead matching of each second lead, the cosine similarity is averaged in the dimension, and the second feature corresponding to the maximum value after the dimension averaging is taken as the optimal second feature. The optimal second feature and the first feature under the optimal first lead are concatenated to determine the cross-modal concatenation feature.

[0075] Furthermore, such as Figure 3 As shown, the processing device 200 also includes a model training module 240, which determines the basic large model of the medical signal through the following steps: The medical and physiological signals of the sample are processed based on the initial mask autoencoder network layer to determine the reconstructed medical and physiological signals of the sample. The first loss value of the initial mask autoencoder network layer is determined based on the sample medical physiological signal and the reconstructed sample medical physiological signal. Based on the initial contrastive learning self-supervised network layer, feature extraction is performed on two segments of the sample medical physiological signal. The features of the two segments are then processed based on the contrastive loss function to determine the second loss value of the initial contrastive learning self-supervised network layer. Based on the first loss value and the second loss value, the network parameters of the initial mask autoencoder network layer and the initial contrastive learning self-supervised network layer are adjusted to determine the basic large model of the medical signal.

[0076] Furthermore, such as Figure 3 As shown, the processing device 200 further includes a parameter adjustment module 250, which is used for: During the downstream processing task adaptation phase, the network parameters of the basic large model of medical signals are adjusted using any one of three methods: full parameter fine-tuning, lightweight adapter fine-tuning, and freezing feature extractor parameters.

[0077] This application provides a cross-modal medical signal processing device, comprising: a first feature extraction module, used to input multiple original medical physiological signals acquired through multiple first leads into a large-scale medical signal model, and to extract temporal and semantic features for each original medical physiological signal based on a masked autoencoder network layer and a contrastive learning self-supervised network layer, to determine a first feature for each original medical physiological signal; a second feature extraction module, used to input new modal medical physiological signals acquired through multiple second leads from downstream processing tasks into the large-scale medical signal model, and to determine multiple second features for the new modal medical physiological signals; wherein the modality of the new modal medical physiological signals is inconsistent with the modality of the original medical physiological signals; and a cross-modal extraction module, used to perform cross-modal similarity matching processing on the multiple first features and the multiple second features, to determine cross-modal splicing features, so as to automatically process downstream processing tasks using cross-modal splicing features. By forming a general representation capability through self-supervised pre-training of a large model, it can be quickly transferred to multiple physiological signals during surgery. Through the flexible configuration of multi-path pre-training and the precise local adaptation mechanism, the efficiency of cross-modal analysis and the model generalization ability are significantly improved.

[0078] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0079] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the cross-modal medical signal processing method in the illustrated method embodiment can be found in the method embodiment for specific implementation, and will not be repeated here.

[0080] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the cross-modal medical signal processing method in the illustrated method embodiment can be found in the method embodiment for specific implementation, and will not be repeated here.

[0081] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing cross-modal medical signals, characterized in that, The processing method includes: Multiple raw medical physiological signals acquired through multiple first leads are input into a large medical signal basic model. Based on the mask autoencoder network layer and the contrastive learning self-supervised network layer, temporal and semantic features are extracted for each raw medical physiological signal to determine the first feature of each raw medical physiological signal. The new modal medical physiological signals acquired through multiple second leads in the downstream processing task are input into the basic large model of medical signals to determine multiple second features of the new modal medical physiological signals; wherein, the modality of the new modal medical physiological signals is inconsistent with the modality of the original medical physiological signals; Cross-modal similarity matching is performed on multiple first features and multiple second features to determine cross-modal splicing features, which are then used to automatically process downstream processing tasks. The step of performing cross-modal similarity matching on multiple first features and multiple second features to determine cross-modal splicing features includes: Determine the feature similarity between each of the first features and each of the second features; The average similarity matrix formed by the multiple feature similarities is averaged to determine the average similarity matrix between each first lead and each second lead; For any second lead, the optimal lead matching pair for the second lead is determined based on the maximum average similarity matrix of the second lead; wherein, the optimal lead matching pair includes the second lead and the optimal first lead that matches the second lead; Based on the optimal lead matching of each second lead, the corresponding average similarity matrix is ​​averaged in dimension, and the second feature corresponding to the maximum value after dimensional averaging is taken as the optimal second feature. The optimal second feature and the first feature under the optimal first lead are concatenated to determine the cross-modal concatenation feature.

2. The processing method according to claim 1, characterized in that, The mask-based autoencoder network layer and the contrastive learning self-supervised network layer perform temporal and semantic feature extraction on each of the original medical physiological signals to determine the first feature of each of the original medical physiological signals, including: The masked autoencoder network layer performs slicing, random masking, and reconstruction processing on each of the original medical physiological signals to determine the temporal characteristics of each of the original medical physiological signals. The contrastive learning self-supervised network layer uses a shared-weight encoder to extract features from each of the original medical physiological signals, thereby determining the semantic features of each of the original medical physiological signals. Based on the temporal features and semantic features, the first feature of the original medical physiological signal is determined.

3. The processing method according to claim 2, characterized in that, The process of slicing, randomly masking, and reconstructing each original medical physiological signal based on the masked autoencoder network layer determines the temporal characteristics of each original medical physiological signal, including: The original medical physiological signals were processed by signal slicing to identify multiple signal segments; Random masking is performed on multiple signal segments to determine the visible signal segments and the masked segments; The visible signal segment is subjected to feature encoding processing to determine the encoded features; Based on the encoded features and the positional information of the mask fragment, reconstruction processing is performed to determine the temporal characteristics of the original medical physiological signal.

4. The processing method according to claim 1, characterized in that, The basic large-scale model of the medical signal is determined through the following steps: The medical and physiological signals of the sample are processed based on the initial mask autoencoder network layer to determine the reconstructed medical and physiological signals of the sample. The first loss value of the initial mask autoencoder network layer is determined based on the sample medical physiological signal and the reconstructed sample medical physiological signal. Based on the initial contrastive learning self-supervised network layer, feature extraction is performed on two segments of the sample medical physiological signal. The features of the two segments are then processed based on the contrastive loss function to determine the second loss value of the initial contrastive learning self-supervised network layer. Based on the first loss value and the second loss value, the network parameters of the initial mask autoencoder network layer and the initial contrastive learning self-supervised network layer are adjusted to determine the basic large model of the medical signal.

5. The processing method according to claim 1, characterized in that, Before inputting the multiple novel modal medical physiological signals acquired through the second lead from the downstream processing task into the basic large model of medical signals, and determining the second feature of the novel modal medical physiological signals, the processing method further includes: During the downstream processing task adaptation phase, the network parameters of the basic large model of medical signals are adjusted using any one of three methods: full parameter fine-tuning, lightweight adapter fine-tuning, and freezing feature extractor parameters.

6. A cross-modal medical signal processing device, characterized in that, The processing device includes: The first feature extraction module is used to input multiple raw medical physiological signals collected through multiple first leads into the basic medical signal model, and to extract temporal and semantic features for each raw medical physiological signal based on the mask autoencoder network layer and the contrastive learning self-supervised network layer, so as to determine the first feature of each raw medical physiological signal. The second feature extraction module is used to input the new modal medical physiological signals acquired by the downstream processing task through multiple second leads into the basic large model of medical signals, and determine multiple second features of the new modal medical physiological signals; wherein the modality of the new modal medical physiological signals is inconsistent with the modality of the original medical physiological signals; A cross-modal extraction module is used to perform cross-modal similarity matching on multiple first features and multiple second features to determine cross-modal concatenation features, so as to automatically process downstream processing tasks using cross-modal concatenation features, including: Determine the feature similarity between each of the first features and each of the second features; The average similarity matrix formed by the multiple feature similarities is averaged to determine the average similarity matrix between each first lead and each second lead; For any second lead, the optimal lead matching pair for the second lead is determined based on the maximum average similarity matrix of the second lead; wherein, the optimal lead matching pair includes the second lead and the optimal first lead that matches the second lead; Based on the optimal lead matching of each second lead, the corresponding average similarity matrix is ​​averaged in dimension, and the second feature corresponding to the maximum value after dimensional averaging is taken as the optimal second feature. The optimal second feature and the first feature under the optimal first lead are concatenated to determine the cross-modal concatenation feature.

7. The processing apparatus according to claim 6, characterized in that, When the first feature extraction module is used to perform temporal and semantic feature extraction on each of the original medical physiological signals in the mask-based autoencoder network layer and the contrastive learning self-supervised network layer to determine the first feature of each of the original medical physiological signals, the first feature extraction module is specifically used for: The masked autoencoder network layer performs slicing, random masking, and reconstruction processing on each of the original medical physiological signals to determine the temporal characteristics of each of the original medical physiological signals. The contrastive learning self-supervised network layer uses a shared-weight encoder to extract features from each of the original medical physiological signals, thereby determining the semantic features of each of the original medical physiological signals. Based on the temporal features and semantic features, the first feature of the original medical physiological signal is determined.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the cross-modal medical signal processing method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the cross-modal medical signal processing method as described in any one of claims 1 to 5.

Images