Cross-modal self-supervised learning model training method and device, equipment and medium
By using a cross-modal self-supervised learning model and employing unlabeled EIT-ultrasound data for self-supervised pre-training, the problem of data scarcity in multimodal medical data fusion models is solved, achieving efficient feature extraction and robust feature representation, which is suitable for multimodal medical data fusion tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-01
- Publication Date
- 2026-03-27
AI Technical Summary
Existing multimodal medical data fusion models face challenges in the field of medical AI, including high data acquisition costs, difficulty in pairing data, and data imbalance. Existing self-supervised learning methods cannot effectively utilize the inherent correlations between multimodal medical data, leading to difficulties in model training.
A cross-modal self-supervised learning model is adopted. By constructing a dual-branch encoder-decoder network and a cross-modal consistency constraint module, a large amount of unlabeled EIT-ultrasound data is used for self-supervised pre-training to learn the common semantic features of multimodal data, and then fine-tuning is performed in downstream tasks.
It significantly reduces the reliance on gold standard annotation, improves the generalization and robustness of feature extraction, greatly enhances few-sample learning performance, and is suitable for fusion tasks of multimodal medical data.
Smart Images
Figure CN121745201A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence, deep learning and medical image processing, and specifically to a cross-modal self-supervised learning model training method, device, equipment and medium. Background Technology
[0002] Artificial intelligence-based medical image analysis technologies, particularly in the field of multimodal data fusion, have demonstrated enormous clinical potential. For example, fusing functional information from electrical impedance tomography (EIT) with structural information from ultrasound can create more accurate diagnostic models for lung diseases. However, training these deep learning models heavily relies on large amounts of precisely labeled data. In the medical context, this means collecting a large number of precisely paired samples of "EIT data - ultrasound data - gold standard (such as CT diagnostic results or expert-annotated data)".
[0003] The above requirements lead to huge challenges in practice: (1) The cost of acquisition is extremely high: the gold standard examination (such as CT) is radioactive and cannot be used frequently for follow-up patients. Moreover, expert annotation is time-consuming, laborious, and costly; (2) The matching is difficult: it is very difficult to obtain EIT, ultrasound and CT (Computed Tomography) data of the same patient at the same time in the clinical workflow, resulting in a scarcity of matched data; (3) Data imbalance: there are very few labeled samples for some rare diseases, making it difficult for the model to learn.
[0004] The aforementioned data shortage severely restricts the research and application of medical AI (Artificial Intelligence), especially multimodal fusion models. Existing solutions mainly focus on data augmentation, transfer learning, or using simpler models on limited data, but these methods cannot fundamentally solve the essential problem of the scarcity of high-quality labeled data.
[0005] Self-supervised learning is a technique that has emerged in recent years. Its core idea is to construct supervisory signals from the data itself for pre-training, learn general data representations, and then fine-tune them on downstream tasks using a small amount of labeled data. However, existing self-supervised learning methods (such as contrastive learning and masked image modeling) are mostly designed for single-modal data (such as images or text), and there is no mature solution that can effectively utilize the natural and inherent correlations between multimodal medical data to learn a strong and general feature foundation for cross-modal fusion tasks. Therefore, there is an urgent need for a new learning framework that can utilize massive amounts of unlabeled multimodal medical data. Summary of the Invention
[0006] This application provides a method, apparatus, device, and medium for training a cross-modal self-supervised learning model, which can utilize a large amount of unlabeled paired multimodal data for self-supervised pre-training, thereby significantly reducing the dependence on gold standard labels.
[0007] In a first aspect, embodiments of this application provide a method for training a cross-modal self-supervised learning model, the method comprising: A cross-modal self-supervised learning model was constructed to achieve medical multimodal data fusion, and multiple time-synchronized and unlabeled EIT-ultrasound data pairs were collected. The cross-modal self-supervised learning model is trained and tuned based on time-synchronized and unlabeled EIT-ultrasound data to achieve model convergence, and the pre-training of the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model is completed. The cross-modal self-supervised learning model is trained using supervised learning based on the collected and labeled EIT-ultrasound data pairs.
[0008] In conjunction with the first aspect, in one implementation method, The cross-modal self-supervised learning model includes a data preprocessing module, a dual-branch encoder-decoder network, a cross-modal consistency constraint module, and a pre-training and fine-tuning controller; The data preprocessing module is used to acquire time-synchronized and unlabeled EIT-ultrasound data pairs. The dual-branch encoder-decoder network includes an EIT branch and an ultrasound branch; The cross-modal consistency constraint module is used to calculate the loss function that measures the consistency between EIT data and ultrasound data.
[0009] In conjunction with the first aspect, in one implementation method, The EIT branch consists of an EIT encoder and an optional EIT decoder, wherein the EIT encoder is used to map EIT data to a low-dimensional feature space. The ultrasound branch consists of an ultrasound encoder and an optional ultrasound decoder, wherein the ultrasound encoder is used to map ultrasound data to a low-dimensional feature space.
[0010] In conjunction with the first aspect, in one implementation method, The cross-modal consistency constraint module includes a feature alignment loss unit and an output consistency loss unit. The feature alignment loss unit is used to calculate the distance between the feature vectors output by the EIT branch and the ultrasound branch; The output consistency loss unit is used to achieve consistency of physiological parameters and consistency of image generation. The consistency of the physiological parameters is that when a regression head is connected after the EIT branch to predict global ventilation, and when a regression head is connected to the ultrasound branch to predict local displacement by tracking pleural motion, the temporal trends of global ventilation and local displacement are consistent. The image generation consistency is achieved by designing a cross-modal translator, inputting the feature vector output from the EIT branch into the ultrasound decoder to generate a pseudo ultrasound image, and calculating the reconstruction loss by comparing it with the real local displacement.
[0011] In conjunction with the first aspect, in one implementation method, The pre-training and fine-tuning controller includes a self-supervised pre-training process controller and a downstream task fine-tuning interface; The self-supervised pre-training process controller is used to control the cross-modal self-supervised learning model. It uses time-synchronized and unlabeled EIT-ultrasound data pairs to train the EIT branch and the ultrasound branch with the goal of minimizing cross-modal consistency loss, thereby achieving pre-training. The downstream task fine-tuning interface is used to remove or freeze parts of the decoder after pre-training, connect a task head after the encoder according to a specific downstream task, and fine-tune the task head and encoder using labeled EIT-ultrasound data.
[0012] In conjunction with the first aspect, in one implementation, the training and tuning of the cross-modal self-supervised learning model based on time-synchronized and unlabeled EIT-ultrasound data pairs to achieve model convergence, and the pre-training of the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model, specifically includes: Based on time-synchronized and unlabeled EIT-ultrasound data pairs, the pre-training process is repeated until the cross-modal self-supervised learning model converges, so that the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model learn feature representations that can understand the common semantics of EIT data and ultrasound data, thus completing the pre-training of the EIT encoder and ultrasound encoder. The pre-training process is as follows: Acquire a time-synchronized and unlabeled EIT-ultrasound data pair, input the EIT data from the EIT-ultrasound data pair into the EIT branch to obtain the feature vector and possible output prediction, and input the ultrasound data from the EIT-ultrasound data pair into the ultrasound branch to obtain the feature vector and possible output prediction. Based on the obtained feature vectors and possible output predictions, the total cross-modal consistency loss is calculated, and the parameters of the EIT branch and the ultrasound branch are updated through backpropagation to minimize the total cross-modal consistency loss. The calculation method for the total cross-modal consistency loss is as follows: L_total =λ1×L_feat+λ2×L_param+λ3×L_cycle Where L_total represents the total cross-modal consistency loss, L_feat represents the feature alignment loss, L_param represents the parameter consistency loss, L_cycle represents the cyclic consistency loss, and λ1, λ2, and λ3 represent the weight coefficients.
[0013] In conjunction with the first aspect, in one implementation, training the cross-modal self-supervised learning model using supervised learning based on the acquired labeled EIT-ultrasound data pairs specifically includes: For specific downstream tasks, collect the corresponding labeled EIT-ultrasound data pairs; Load the pre-trained EIT encoder and ultrasonic encoder, and then attach the task head after the EIT encoder and ultrasonic encoder; Based on the collected and labeled EIT-ultrasound data pairs, a cross-modal self-supervised learning model is trained using supervised learning, thereby completing the training of the cross-modal self-supervised learning model.
[0014] Secondly, embodiments of this application provide a cross-modal self-supervised learning model training device, the cross-modal self-supervised learning model training device comprising: The building module is used to construct a cross-modal self-supervised learning model for realizing medical multimodal data fusion and to collect multiple time-synchronized and unlabeled EIT-ultrasound data pairs. The self-supervised pre-training module is used to train and adjust the cross-modal self-supervised learning model based on time-synchronized and unlabeled EIT-ultrasound data pairs to enable model convergence, and to complete the pre-training of the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model. A supervised fine-tuning module is used to train the cross-modal self-supervised learning model in a supervised learning manner based on the acquired labeled EIT-ultrasound data pairs.
[0015] Thirdly, embodiments of this application provide a cross-modal self-supervised learning model training device, the cross-modal self-supervised learning model training device including a processor, a memory, and a cross-modal self-supervised learning model training program stored in the memory and executable by the processor, wherein when the cross-modal self-supervised learning model training program is executed by the processor, the steps of the cross-modal self-supervised learning model training method described above are implemented.
[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a cross-modal self-supervised learning model training program, wherein when the cross-modal self-supervised learning model training program is executed by a processor, it implements the steps of the cross-modal self-supervised learning model training method described above.
[0017] The beneficial effects of the technical solutions provided in this application include: (1) Breaking through the bottleneck of data annotation: Creatively using multimodal data itself as a supervision signal, a powerful feature extractor can be trained with little or no expensive manual annotation, fundamentally solving the biggest obstacle to the development of medical AI; (2) Learning the essential association: Through consistency constraints, the network is forced to learn the common physiological and pathological semantics behind the two modalities, rather than the surface image features, and the resulting feature representations are more generalizable and robust; (3) Significantly improves small sample learning performance: The number of labeled samples required for fine-tuning a model that has been pre-trained by large-scale self-supervised methods can be reduced by an order of magnitude compared to training from scratch, making it possible to develop models for rare diseases or subdivided tasks. (4) Technological foresight and universality: The cross-modal self-supervised learning model of this application is not only applicable to EIT and ultrasound, but can be extended to any multimodal medical data with intrinsic correlation, providing a reusable methodology for the entire medical AI field, and has extremely high commercial and academic value. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating the cross-modal self-supervised learning model training method of this application; Figure 2 This is a schematic diagram of the overall cross-modal self-supervised learning model; Figure 3 This is a schematic diagram of a two-branch network structure and cross-modal consistency constraints. Figure 4 This is a flowchart illustrating the self-supervised pre-training phase. Figure 5 A flowchart illustrating the downstream task fine-tuning phase; Figure 6 This is a schematic diagram of the functional modules of the cross-modal self-supervised learning model training device of this application; Figure 7 This is a schematic diagram of the hardware structure of the cross-modal self-supervised learning model training device of this application. Detailed Implementation
[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0020] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0021] In a first aspect, embodiments of this application provide a cross-modal self-supervised learning model training method, which overcomes the shortcomings of scarce labeled data and high acquisition costs in the training of medical multimodal fusion AI models, and provides a framework and method that can utilize a large amount of unlabeled paired multimodal data for self-supervised pre-training, thereby significantly reducing the dependence on gold standard labeling.
[0022] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the training method for the cross-modal self-supervised learning model in this application. Figure 1 As shown, cross-modal self-supervised learning model training methods include: S1: Construct a cross-modal self-supervised learning model for realizing medical multimodal data fusion, and collect multiple time-synchronized and unlabeled EIT-ultrasound data pairs; that is, collect massive amounts of time-synchronized unlabeled EIT-ultrasound data pairs (D_eit, D_us), where D_eit represents EIT data and D_us represents ultrasound data; S2: Train and adjust the cross-modal self-supervised learning model based on time-synchronized and unlabeled EIT-ultrasound data pairs to make the model converge, and complete the pre-training of the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model; S3: Based on the collected and labeled EIT-ultrasound data pairs, train the cross-modal self-supervised learning model in a supervised learning manner.
[0023] In this application, the cross-modal self-supervised learning model includes a data preprocessing module, a bi-branch encoder-decoder network, a cross-modal consistency constraint module, and a pre-training and fine-tuning controller. The data preprocessing module is used to acquire time-synchronized and unlabeled EIT-ultrasound data pairs; the bi-branch encoder-decoder network includes an EIT branch and an ultrasound branch, thus forming a bi-branch encoder network; the cross-modal consistency constraint module is used to calculate a loss function that measures the consistency between EIT data and ultrasound data. See also... Figure 2 The figure shown is an overall schematic diagram of a cross-modal self-supervised learning model.
[0024] Furthermore, the EIT branch consists of an EIT encoder and an optional EIT decoder, wherein the EIT encoder is used to map EIT data to a low-dimensional feature space; the ultrasound branch consists of an ultrasound encoder and an optional ultrasound decoder, wherein the ultrasound encoder is used to map ultrasound data to a low-dimensional feature space.
[0025] Specifically, the EIT branch consists of an EIT encoder E_E and an optional EIT decoder G_E. The EIT encoder E_E is used to map the EIT data D_eit to a low-dimensional feature space F_eit. For the ultrasound branch, it consists of an ultrasound encoder E_U and an optional ultrasound decoder G_U. E_U is used to map the ultrasound data D_us to a low-dimensional feature space F_us.
[0026] Furthermore, the cross-modal consistency constraint module is the core innovative module of the cross-modal self-supervised learning model in this application, containing multiple loss function calculation units for measuring the consistency between two modalities. Specifically, the cross-modal consistency constraint module includes a feature alignment loss unit and an output consistency loss unit.
[0027] The feature alignment loss unit is used to calculate the distance between the feature vectors output by the EIT branch and the ultrasound branch. Specifically, the feature alignment loss unit is used to calculate the distance (such as cosine similarity, mean square error) between the feature vectors F_eit and F_us output by the two branches, forcing them to align in a high-dimensional semantic space.
[0028] The output consistency loss unit is used to achieve consistency of physiological parameters and consistency of image generation. Physiological parameter consistency is achieved by connecting a regression head after the EIT branch to predict global ventilation and connecting a regression head after the ultrasound branch to predict local displacement by tracking pleural motion, so that the temporal variation trends of global ventilation and local displacement are consistent. Image generation consistency is achieved by designing a cross-modal translator, inputting the feature vector output by the EIT branch into the ultrasound decoder to generate pseudo ultrasound images, and calculating the reconstruction loss with the real local displacement.
[0029] Specifically, for consistency of physiological parameters, a regression head R_E is appended after the EIT branch to predict the global ventilation volume V_eit; a regression head R_U is appended after the ultrasound branch to predict the local displacement D_us by tracking pleural motion. The loss function encourages consistency in the temporal trends of V_eit and D_us (e.g., high correlation coefficient). For consistency in image generation, a cross-modal translator is designed. For example, F_eit is input into G_U to generate a pseudo-ultrasound image G_U(F_eit), and the reconstruction loss is calculated with the real D_us, and vice versa. This cyclic consistency strengthens the semantic correspondence between modalities. See also Figure 3 The diagram shown is a schematic of a two-branch network structure and cross-modal consistency constraints.
[0030] Furthermore, the pre-trained and fine-tuned controller includes a self-supervised pre-training process controller and a downstream task fine-tuning interface.
[0031] The self-supervised pre-training process controller is used to control the cross-modal self-supervised learning model. It uses time-synchronized and unlabeled EIT-ultrasound data pairs to train the EIT and ultrasound branches with the goal of minimizing cross-modal consistency loss, thus achieving pre-training.
[0032] Specifically, a self-supervised pre-training process controller governs the entire framework (cross-modal self-supervised learning model) to train the dual-branch network using unlabeled data (D_eit, D_us) with the goal of minimizing the cross-modal consistency loss. See also Figure 4 The diagram shown is a flowchart of the self-supervised pre-training stage.
[0033] The downstream task fine-tuning interface is used to remove or freeze parts of the decoder after pre-training, attach a task head after the encoder according to the specific downstream task, and fine-tune the task head and encoder using labeled EIT-ultrasound data.
[0034] Specifically, after pre-training, some decoders (including the EIT decoder and ultrasound decoder) are removed or frozen, and a new task head T is appended after the encoder based on the specific downstream task (such as disease classification). The task head and encoder are then fine-tuned using a small amount of labeled data (D_eit, D_us, Label). See also Figure 5 The diagram shown is a flowchart of the downstream task fine-tuning phase.
[0035] Furthermore, in one embodiment, the cross-modal self-supervised learning model is trained and tuned based on time-synchronized and unlabeled EIT-ultrasound data to achieve model convergence, and the pre-training of the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model is completed, specifically including: Based on time-synchronized and unlabeled EIT-ultrasound data pairs, the pre-training process is repeated until the cross-modal self-supervised learning model converges, so that the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model learn feature representations that can understand the common semantics of EIT data and ultrasound data, thus completing the pre-training of the EIT encoder and ultrasound encoder. The pre-training process is as follows: a1: Obtain a time-synchronized and unlabeled EIT-ultrasound data pair, input the EIT data in the EIT-ultrasound data pair into the EIT branch to obtain the feature vector and possible output prediction, and input the ultrasound data in the EIT-ultrasound data pair into the ultrasound branch to obtain the feature vector and possible output prediction. a2: Based on the obtained feature vectors and possible output predictions, calculate the total cross-modal consistency loss, and update the parameters of the EIT branch and the ultrasound branch through backpropagation to minimize the total cross-modal consistency loss. The calculation method for the total cross-modal consistency loss is as follows: L_total =λ1×L_feat+λ2×L_param+λ3×L_cycle Where L_total represents the total cross-modal consistency loss, L_feat represents the feature alignment loss, L_param represents the parameter consistency loss, L_cycle represents the cyclic consistency loss, and λ1, λ2, and λ3 represent the weight coefficients.
[0036] Specifically, after collecting time-synchronized and unlabeled EIT-ultrasound data pairs (D_eit, D_us), a pre-training process is performed. (D_eit, D_us) is input into a dual-branch network to obtain feature vectors F_eit and F_us, as well as possible output predictions (such as V_eit, D_us, or generated pseudo-images). Then, the cross-modal consistency total loss is calculated, and all parameters of the EIT and ultrasound branches are updated through backpropagation. The goal is to minimize L_total. The pre-training process is repeated until the cross-modal self-supervised learning model converges. At this point, the EIT encoder E_E and ultrasound encoder E_U in the cross-modal self-supervised learning model have learned powerful feature representations that can understand the common semantics of the two modalities.
[0037] Furthermore, in one embodiment, the cross-modal self-supervised learning model is trained using supervised learning based on the collected and labeled EIT-ultrasound data pairs, specifically including: S301: For specific downstream tasks, collect the corresponding labeled EIT-ultrasound data pairs; S302: Load the pre-trained EIT encoder and ultrasonic encoder, and connect the task head after the EIT encoder and ultrasonic encoder; S303: Based on the collected and labeled EIT-ultrasound data pairs, a cross-modal self-supervised learning model is trained in a supervised learning manner, thereby completing the training of the cross-modal self-supervised learning model.
[0038] Specifically, for a specific downstream task (such as pneumothorax detection), a small labeled dataset is prepared, and the pre-trained EIT encoder and ultrasound encoder are loaded. Then, a task-specific task head T is connected. The encoder can be frozen or fine-tuned as a whole. Using the small labeled dataset, a cross-modal self-supervised learning model is trained in a supervised learning manner (minimizing cross-entropy loss) to obtain a high-performance cross-modal self-supervised learning model that can realize the fusion of medical multimodal data.
[0039] The following section provides a detailed explanation of the cross-modal self-supervised learning model training method proposed in this application.
[0040] For the hardware and data environment, the computing platform uses a high-performance server equipped with multiple GPUs for model pre-training; the data source consists of tens of thousands of time-synchronized but disease-unlabeled EIT data frames and ultrasound video frames collected from the clinical monitoring system.
[0041] Regarding the details of algorithm implementation: First, the network structure: E_E and E_U use 3D CNN or VisionTransformer as the backbone network; for R_E, it is a global average pooling layer + fully connected layer, outputting a scalar (ventilation); for G_E and G_U, U-Net or generative adversarial network is used as the decoder.
[0042] Secondly, the loss function is used. For the feature alignment loss L_feat, cosine embedding loss is used: L_feat = 1-cos_sim(F_eit, F_us); for the parameter consistency loss L_param, the negative value of the Pearson correlation coefficient of the V_eit and D_us time series is calculated: L_param=-corr(V_eit, D_us); for the cycle consistency loss L_cycle, L1 loss is used: L_cycle=||G_U(F_eit) -D_us||_1 + ||G_E(F_us) -D_eit ||_1.
[0043] For the training strategy, during pre-training, the Adam optimizer was used with a learning rate of 1e-4, a batch size of 32, and 100 epochs were trained. During fine-tuning, the learning rate was reduced to 1e-5, and only 100 cases with pneumothorax annotations were used for 10 epochs.
[0044] In practical applications, for pneumothorax detection tasks, the method of this application is compared with two other methods: Method A: Train a fusion model from scratch on 500 labeled data points, achieving an accuracy of 85%; Method B: Trained on 500 labeled data points, but the encoder uses weights pre-trained on ImageNet, achieving 87% accuracy; Method C (this application): First, self-supervised pre-training was performed on 50,000 pairs of unlabeled EIT-ultrasound data, and then fine-tuned on only 50 labeled data, with an accuracy of 91%.
[0045] The results show that the method in this application achieves significantly better performance than traditional methods with only one-tenth of the labeled data, which fully demonstrates the great power of the cross-modal self-supervised learning model in solving the problem of data scarcity.
[0046] It should be further noted that traditional supervised learning paradigms rely on a large amount of accurately labeled data, which results in high data acquisition costs and long cycles. Model performance is severely limited by the quantity and quality of labeled data, and data scarcity hinders development. In contrast, the supervised learning paradigm proposed in this application utilizes massive amounts of unlabeled data for pre-training and requires only a small amount of labeled data for fine-tuning. It achieves or even surpasses the performance of traditional methods at a low cost, breaking through the data bottleneck and accelerating the implementation of AI.
[0047] For visualization in practical applications (t-SNE dimensionality reduction), in the feature space visualization, the features of existing zero-trained models are mixed and difficult to distinguish, with pneumothorax, normal, and effusion distributed in a mixed point-like pattern; the features of the pre-trained model in this application are clustered in the same category and separated in the opposite category, with pneumothorax, normal, and effusion forming obvious clusters.
[0048] The cross-modal self-supervised learning model training method in this application embodiment is as follows: (1) Breaking through the bottleneck of data annotation: Creatively using multimodal data itself as a supervision signal, a powerful feature extractor can be trained with little or no expensive manual annotation, fundamentally solving the biggest obstacle to the development of medical AI; (2) Learning the essential association: Through consistency constraints, the network is forced to learn the common physiological and pathological semantics behind the two modalities (such as the association between "ventilation" and "motor"), rather than the surface image features. The resulting feature representations are more generalizable and robust. (3) Significantly improves small sample learning performance: The number of labeled samples required for fine-tuning a model that has been pre-trained by large-scale self-supervised methods can be reduced by an order of magnitude compared to training from scratch, making it possible to develop models for rare diseases or subdivided tasks. (4) Technological foresight and universality: The cross-modal self-supervised learning model of this application is not only applicable to EIT and ultrasound, but can also be extended to any multimodal medical data with intrinsic correlation (such as CT-MRI, electrocardiogram-phonocardiogram), providing a reusable methodology for the entire medical AI field, and has extremely high commercial and academic value.
[0049] Secondly, embodiments of this application also provide a cross-modal self-supervised learning model training device.
[0050] In one embodiment, reference is made to Figure 6 , Figure 6 This is a schematic diagram of the functional modules of the cross-modal self-supervised learning model training device of this application. Figure 6 As shown, the cross-modal self-supervised learning model training device includes: a construction module, a self-supervised pre-training module, and a supervised fine-tuning module.
[0051] The construction module is used to build a cross-modal self-supervised learning model for realizing medical multimodal data fusion, and to collect multiple time-synchronized and unlabeled EIT-ultrasound data pairs; the self-supervised pre-training module is used to train and adjust the cross-modal self-supervised learning model based on the time-synchronized and unlabeled EIT-ultrasound data pairs to make the model converge, and to complete the pre-training of the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model; the supervised fine-tuning module is used to train the cross-modal self-supervised learning model in a supervised learning manner based on the collected labeled EIT-ultrasound data pairs.
[0052] Thirdly, embodiments of this application provide a cross-modal self-supervised learning model training device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.
[0053] Reference Figure 7 , Figure 7 This is a schematic diagram of the hardware structure of the cross-modal self-supervised learning model training device involved in the embodiments of this application. In the embodiments of this application, the cross-modal self-supervised learning model training device may include a processor, a memory, a communication interface, and a communication bus.
[0054] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.
[0055] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the cross-modal self-supervised learning model training device, as well as interfaces used for interconnecting the cross-modal self-supervised learning model training device with other devices (such as other computing devices or user devices). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user devices can be displays, keyboards, etc.
[0056] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.
[0057] The processor can be a general-purpose processor, which can call the cross-modal self-supervised learning model training program stored in memory and execute the cross-modal self-supervised learning model training method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the cross-modal self-supervised learning model training program is called can be referred to the various embodiments of the cross-modal self-supervised learning model training method of this application, and will not be repeated here.
[0058] Those skilled in the art will understand that Figure 7 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0059] Fourthly, embodiments of this application also provide a computer-readable storage medium.
[0060] The present application stores a cross-modal self-supervised learning model training program on a computer-readable storage medium, wherein when the cross-modal self-supervised learning model training program is executed by a processor, it implements the steps of the cross-modal self-supervised learning model training method as described above.
[0061] The method implemented when the cross-modal self-supervised learning model training procedure is executed can be referred to in the various embodiments of the cross-modal self-supervised learning model training method of this application, and will not be repeated here.
[0062] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.
[0063] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.
[0064] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.
[0065] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.
[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.
[0067] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for training a cross-modal self-supervised learning model, characterized in that, The cross-modal self-supervised learning model training method includes: A cross-modal self-supervised learning model was constructed to achieve medical multimodal data fusion, and multiple time-synchronized and unlabeled EIT-ultrasound data pairs were collected. The cross-modal self-supervised learning model is trained and tuned based on time-synchronized and unlabeled EIT-ultrasound data to achieve model convergence, and the pre-training of the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model is completed. The cross-modal self-supervised learning model is trained using supervised learning based on the collected and labeled EIT-ultrasound data pairs.
2. The cross-modal self-supervised learning model training method as described in claim 1, characterized in that: The cross-modal self-supervised learning model includes a data preprocessing module, a dual-branch encoder-decoder network, a cross-modal consistency constraint module, and a pre-training and fine-tuning controller; The data preprocessing module is used to acquire time-synchronized and unlabeled EIT-ultrasound data pairs. The dual-branch encoder-decoder network includes an EIT branch and an ultrasound branch; The cross-modal consistency constraint module is used to calculate the loss function that measures the consistency between EIT data and ultrasound data.
3. The cross-modal self-supervised learning model training method as described in claim 2, characterized in that: The EIT branch consists of an EIT encoder and an optional EIT decoder, wherein the EIT encoder is used to map EIT data to a low-dimensional feature space. The ultrasound branch consists of an ultrasound encoder and an optional ultrasound decoder, wherein the ultrasound encoder is used to map ultrasound data to a low-dimensional feature space.
4. The cross-modal self-supervised learning model training method as described in claim 3, characterized in that: The cross-modal consistency constraint module includes a feature alignment loss unit and an output consistency loss unit. The feature alignment loss unit is used to calculate the distance between the feature vectors output by the EIT branch and the ultrasound branch; The output consistency loss unit is used to achieve consistency of physiological parameters and consistency of image generation. The consistency of the physiological parameters is that when a regression head is connected after the EIT branch to predict global ventilation, and when a regression head is connected to the ultrasound branch to predict local displacement by tracking pleural motion, the temporal trends of global ventilation and local displacement are consistent. The image generation consistency is achieved by designing a cross-modal translator, inputting the feature vector output from the EIT branch into the ultrasound decoder to generate a pseudo ultrasound image, and calculating the reconstruction loss by comparing it with the real local displacement.
5. The cross-modal self-supervised learning model training method as described in claim 4, characterized in that: The pre-training and fine-tuning controller includes a self-supervised pre-training process controller and a downstream task fine-tuning interface; The self-supervised pre-training process controller is used to control the cross-modal self-supervised learning model. It uses time-synchronized and unlabeled EIT-ultrasound data pairs to train the EIT branch and the ultrasound branch with the goal of minimizing cross-modal consistency loss, thereby achieving pre-training. The downstream task fine-tuning interface is used to remove or freeze parts of the decoder after pre-training, connect a task head after the encoder according to a specific downstream task, and fine-tune the task head and encoder using labeled EIT-ultrasound data.
6. The cross-modal self-supervised learning model training method as described in claim 2, characterized in that, The training and tuning of the cross-modal self-supervised learning model based on time-synchronized and unlabeled EIT-ultrasound data to achieve model convergence, and the pre-training of the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model, specifically includes: Based on time-synchronized and unlabeled EIT-ultrasound data pairs, the pre-training process is repeated until the cross-modal self-supervised learning model converges, so that the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model learn feature representations that can understand the common semantics of EIT data and ultrasound data, thus completing the pre-training of the EIT encoder and ultrasound encoder. The pre-training process is as follows: Acquire a time-synchronized and unlabeled EIT-ultrasound data pair, input the EIT data from the EIT-ultrasound data pair into the EIT branch to obtain the feature vector and possible output prediction, and input the ultrasound data from the EIT-ultrasound data pair into the ultrasound branch to obtain the feature vector and possible output prediction. Based on the obtained feature vectors and possible output predictions, the total cross-modal consistency loss is calculated, and the parameters of the EIT branch and the ultrasound branch are updated through backpropagation to minimize the total cross-modal consistency loss. The calculation method for the total cross-modal consistency loss is as follows: L_total =λ1×L_feat+λ2×L_param+λ3×L_cycle Where L_total represents the total cross-modal consistency loss, L_feat represents the feature alignment loss, L_param represents the parameter consistency loss, L_cycle represents the cyclic consistency loss, and λ1, λ2, and λ3 represent the weight coefficients.
7. The cross-modal self-supervised learning model training method as described in claim 2, characterized in that, The cross-modal self-supervised learning model, trained using supervised learning based on the collected and labeled EIT-ultrasound data pairs, specifically includes: For specific downstream tasks, collect the corresponding labeled EIT-ultrasound data pairs; Load the pre-trained EIT encoder and ultrasonic encoder, and then attach the task head after the EIT encoder and ultrasonic encoder; Based on the collected and labeled EIT-ultrasound data pairs, a cross-modal self-supervised learning model is trained using supervised learning, thereby completing the training of the cross-modal self-supervised learning model.
8. A cross-modal self-supervised learning model training device, characterized in that, The cross-modal self-supervised learning model training device includes: The building module is used to construct a cross-modal self-supervised learning model for realizing medical multimodal data fusion and to collect multiple time-synchronized and unlabeled EIT-ultrasound data pairs. The self-supervised pre-training module is used to train and adjust the cross-modal self-supervised learning model based on time-synchronized and unlabeled EIT-ultrasound data pairs to enable model convergence, and to complete the pre-training of the EIT encoder and ultrasound encoder in the cross-modal self-supervised learning model. A supervised fine-tuning module is used to train the cross-modal self-supervised learning model in a supervised learning manner based on the acquired labeled EIT-ultrasound data pairs.
9. A cross-modal self-supervised learning model training device, characterized in that, The cross-modal self-supervised learning model training device includes a processor, a memory, and a cross-modal self-supervised learning model training program stored in the memory and executable by the processor, wherein when the cross-modal self-supervised learning model training program is executed by the processor, it implements the steps of the cross-modal self-supervised learning model training method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a cross-modal self-supervised learning model training program, wherein when the cross-modal self-supervised learning model training program is executed by a processor, it implements the steps of the cross-modal self-supervised learning model training method as described in any one of claims 1 to 7.