Multi-modal sarcopenia diagnosis system, method and device and medium

Abstract

The invention provides a multi-modal sarcopenia diagnosis system, method and device and a medium, a technical framework of the multi-modal sarcopenia diagnosis system, method and device integrates multi-modal data integration, a complex neural network technology, a personalized meta-learning algorithm and an interpretable artificial intelligence technology, and aims to create an accurate sarcopenia diagnosis tool with high adaptability. The method not only makes a breakthrough on the basis of a traditional bioelectrical impedance technology, but also shows a huge potential of realizing efficient and personalized diagnosis and monitoring in diversified medical environments through application of multi-modal data fusion and an intelligent model.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal data medical diagnosis, and in particular to a multimodal sarcopenia diagnosis system, method, device and medium. Background Art

[0002] Sarcopenia, a degenerative disease of muscle mass and function that occurs with aging, is particularly prevalent among the elderly. Prompt identification of sarcopenia is crucial for reducing frailty, reducing the risk of falls, and alleviating the burden of healthcare. Bioimpedance technology is currently an important non-invasive method for measuring muscle mass. It uses electrical current to measure the impedance of human tissue and infer the distribution of muscle, fat, and water. Because muscle tissue has a higher water content and lower impedance, while fat tissue has higher impedance, analyzing frequency domain characteristics can effectively assess muscle mass.

[0003] However, the diagnostic method using bioelectrical impedance alone is easily affected by the following factors, which may limit the diagnostic accuracy:

[0004] (1) Noise interference: The bioimpedance measurement process will be affected by various external noises, such as electromagnetic interference, power supply fluctuations, etc. These noises will lead to inaccurate measurement data.

[0005] (2) Patient activity: Minor movements of the patient during the measurement process, such as slight muscle contraction or relaxation, may affect the path and speed of current passing through the tissue, thereby affecting the impedance measurement results.

[0006] (3) Different physiological states: Individual physiological states, such as water intake, body temperature changes, metabolic state, etc., may affect the electrical impedance of tissues. For example, the impedance of the human body will increase in a dehydrated state, while excessive water intake may lead to a decrease in impedance. These physiological changes will mask the true changes in muscle mass.

[0007] (4) Individual differences: Different individuals have different body shapes, muscle distribution, and fat content. These differences can affect the accuracy of bioimpedance measurements, especially without individualized correction.

[0008] (5) Disease status: Certain diseases, such as edema or inflammation, may change the electrical impedance of tissues, thereby affecting the diagnosis of sarcopenia.

[0009] Therefore, this field urgently needs an adaptive learning method that can be dynamically adjusted to improve the accuracy of sarcopenia diagnosis. Summary of the Invention

[0010] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a multimodal sarcopenia diagnosis system, method, device and medium to solve the technical problem that the prior art sarcopenia diagnosis is not accurate enough.

[0011] To achieve the above-mentioned objectives and other related objectives, the first aspect of the present application provides a multimodal sarcopenia diagnostic system, comprising: a multimodal data preprocessing module for collecting and preprocessing bioimpedance data, ultrasound image data, gait and strength data related to muscle mass; a model construction module, connected to the multimodal data preprocessing module, for building a hybrid architecture by combining convolutional neural networks, graph neural networks and Transformer models to process the input bioimpedance data, ultrasound image data, gait and strength data to capture different aspects of muscle mass, function and quality, and output sarcopenia judgment results based on them; a diagnosis module, connected to the model construction module, comprising a multilayer perceptron, and the fusion features output by the hybrid architecture are passed to the multilayer perceptron for classification to predict whether sarcopenia exists and the stage of sarcopenia; a model interpretation module, connected to the diagnosis module, for performing corresponding model interpretation on the sarcopenia risk score output by the diagnosis module.

[0012] In some embodiments of the first aspect of the present application, the multimodal data preprocessing module is also used to: generate rare sarcopenia case data based on a generative adversarial network to expand the training set, and the process includes: collecting a certain amount of rare sarcopenia case data as training data for the generative adversarial network; inputting the collected sarcopenia case data into the generative adversarial network for training; during the training process, the generator in the generative adversarial network learns how to imitate the distribution of real data to generate new rare sarcopenia case data; the discriminator in the generative adversarial network is used to identify generated cases and real cases; the generated rare sarcopenia case data is evaluated based on several preset indicators, and the generated data that passes the evaluation is included in the real data to synthesize the original training set.

[0013] In some embodiments of the first aspect of the present application, the multimodal data preprocessing module is further used to: perform time warping and frequency adjustment on gait and strength data to expand the training set; wherein, the gait and strength data are time warped by stretching or compressing the time series data, and / or the increase or decrease in muscle strength is simulated by increasing or decreasing the frequency of the data to perform frequency adjustment.

[0014] In some embodiments of the first aspect of the present application, the multimodal data preprocessing module preprocesses bioimpedance data, ultrasound image data, gait and strength data related to muscle mass, which includes: the preprocessing method of bioimpedance data includes: normalizing the impedance signal and extracting frequency domain features related to muscle mass; the preprocessing method of ultrasound image data includes: performing standardized grayscale processing on the ultrasound image data, and separating muscle tissue from the ultrasound image after standardized grayscale processing through an image segmentation model, and extracting structural features of muscle thickness and mass; the preprocessing method of gait and strength data includes: converting the original gait and strength data into a time series format and performing standard deviation processing.

[0015] In some embodiments of the first aspect of the present application, in the model building module: a convolutional neural network is used for bioimpedance frequency domain feature extraction and ultrasound image tissue feature recognition, which uses a one-dimensional convolutional neural network to extract the frequency domain features of bioimpedance for analyzing data related to muscle composition; and uses a two-dimensional convolutional neural network to identify tissue features in ultrasound images, including muscle thickness and muscle mass.

[0016] In some embodiments of the first aspect of the present application, the one-dimensional convolutional neural network and the two-dimensional convolutional neural network both include three convolutional layers, and each convolutional layer is followed by an activation function for performing nonlinear processing and a pooling layer for reducing the spatial size of the features.

[0017] In some embodiments of the first aspect of the present application, in the model building module: the Transformer model uses a visual Transformer layer to process state and force data to capture temporal dependencies and detect temporal changes in muscle strength and gait consistency; wherein the visual Transformer layer includes two Transformer encoding blocks, each encoding block includes a self-attention layer and a dense layer; the self-attention layer is used to dynamically focus on the key points in the time series that are most relevant to muscle strength and gait changes, and the dense layer is connected to the self-attention layer for further nonlinear transformation of the information of these key points.

[0018] In some embodiments of the first aspect of the present application, in the model construction module: a graph neural network is used to use bioelectrical impedance data, ultrasound image data, gait and strength data as nodes, and the correlation between these modal data as edges to construct a graph structure; a graph attention network is used and edge weights between different nodes are assigned through an attention mechanism.

[0019] In some embodiments of the first aspect of the present application, the multilayer perceptron in the diagnostic module uses a binary cross-entropy loss function to predict the presence of sarcopenia, and / or uses a categorical cross-entropy loss function to predict different stages of sarcopenia; the sarcopenia stages include early sarcopenia, moderate sarcopenia, and severe sarcopenia.

[0020] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a multimodal sarcopenia diagnosis method, which is applied to a computer device; the method includes: collecting and preprocessing bioimpedance data, ultrasound image data, gait and strength data related to muscle mass; combining convolutional neural networks, graph neural networks and Transformer models to build a hybrid architecture, and processing the input bioimpedance data, ultrasound image data, gait and strength data to capture different aspects of muscle mass, function and quality, and output sarcopenia judgment results based on them; using a multi-layer perceptron to receive the fusion features output by the hybrid architecture and classify them to predict whether sarcopenia exists and the stage of sarcopenia; and performing corresponding model interpretation on the sarcopenia risk score output by the diagnostic module.

[0021] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the method when executed by a processor.

[0022] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes computer program code. When the computer program code is run on a computer, the computer implements the method.

[0023] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides a computer device / equipment / system / electronic terminal, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the method.

[0024] As described above, the multimodal sarcopenia diagnostic system, method, device, and medium of this application have the following beneficial effects: The technical framework of the present invention integrates multimodal data integration, complex neural network technology, personalized meta-learning algorithms, and explainable artificial intelligence technology to create an accurate and highly adaptable sarcopenia diagnostic tool. This method not only achieves breakthroughs based on traditional bioelectrical impedance technology, but also demonstrates great potential for efficient and personalized diagnosis and monitoring in diverse medical environments through the application of multimodal data fusion and intelligent models. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1Shown is a structural schematic diagram of a multimodal sarcopenia diagnostic system in one embodiment of the present application.

[0026] Figure 2 Shown is a schematic diagram of the process of generating rare sarcopenia case data based on a generative adversarial network (GAN) in one embodiment of the present application.

[0027] Figure 3 Shown is a schematic diagram of the structure in the middle figure in one embodiment of the present application.

[0028] Figure 4 Shown is a flowchart of a multimodal sarcopenia diagnosis method in one embodiment of the present application.

[0029] Figure 5 Shown is a structural diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0031] Before further explaining the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations:

[0032] <1> Multimodality: Multimodality refers to the integration and processing of information from multiple different types of data sources, such as text, images, sound, and video. In the field of artificial intelligence, multimodal learning aims to improve the performance and generalization ability of models by combining these different types of data, enabling them to more comprehensively understand and interpret complex scenarios.

[0033] <2> Convolutional Neural Networks (CNNs): A type of deep learning model particularly well-suited for processing data with grid-like topologies, such as images. CNNs extract features through convolutional layers that capture both local features and spatial hierarchical structures, making them highly effective in tasks such as image recognition, classification, and segmentation.

[0034] <3> The Transformer model is a deep learning model based on the self-attention mechanism. It was originally designed for processing sequential data, particularly in the field of natural language processing (NLP). It effectively captures long-range dependencies by processing all elements in a sequence in parallel. It has been widely used in tasks such as machine translation and text generation.

[0035] <4> Graph Neural Networks (GNNs): A deep learning model for processing graph-structured data. It updates node feature representations by aggregating node neighbor information, thereby capturing complex relationships between nodes and the structural information of the graph.

[0036] <5> U-Net: A convolutional neural network architecture particularly well-suited for image segmentation tasks. It has an encoder-decoder structure, where the encoder progressively reduces the spatial dimensionality of the image, while the decoder progressively restores it. The U-Net design allows it to efficiently learn precise location information from images and is commonly used in medical image analysis, such as cell segmentation.

[0037] <6> Multilayer Perceptron (MLP): A basic feedforward neural network consisting of multiple layers, including an input layer, hidden layers, and an output layer. Each layer consists of multiple neurons, connected by weights. MLPs use these layers to learn nonlinear representations of input data and are suitable for a variety of tasks, including classification and regression.

[0038] The application scenarios of the technical solution of the present invention include but are not limited to: (1) Clinical application: This tool is designed to provide clinicians with multi-dimensional sarcopenia risk scores and interpretable diagnostic reports, which are particularly suitable for early screening and continuous monitoring of sarcopenia. (2) Elderly care and home monitoring: It is suitable for elderly care centers and home health monitoring environments. Through integration with wearable devices, it can achieve real-time risk feedback and monitoring updates to optimize the health management of the elderly. (3) Insurance and medical services: Insurance companies can use AI diagnostic results for early intervention to reduce patients' long-term health risks and medical expenses, while expanding market demand and bringing new growth points to the medical service industry.

[0039] Figure 1 A schematic diagram of the structure of a multimodal sarcopenia diagnostic system in an embodiment of the present invention is shown. This multimodal sarcopenia diagnostic system effectively suppresses interference from noise, patient activity, and physiological status in the bioelectrical impedance signal. By integrating bioelectrical impedance, ultrasound images, gait, and strength data, and through complex data fusion and an attention-based weighting mechanism, it dynamically focuses on the key features of sarcopenia that best reflect each modality, thereby achieving personalized and accurate diagnosis.

[0040] The multimodal sarcopenia diagnosis system includes the following modules: a multimodal data preprocessing module 101 , a model building module 102 , a diagnosis module 103 , and a model interpretation module 104 .

[0041] The multimodal data preprocessing module 101 is used to collect and preprocess bioimpedance data, ultrasound image data, gait and strength data related to muscle mass.

[0042] In some examples, in addition to collecting bioimpedance data, ultrasound image data, gait and strength data related to muscle mass from historical medical records, the multimodal data preprocessing module 101 also expands the diversity of the training set in the following ways to ensure the robustness of the model.

[0043] Dataset expansion method 1) Generate rare sarcopenia case data based on generative adversarial network (GAN).

[0044] A Generative Adversarial Network (GAN) is a deep learning model composed of two competing neural networks: a generator and a discriminator. The generator aims to create realistic data (such as images and audio), while the discriminator attempts to distinguish generated data from real data. During training, these two networks compete against each other, with the generator continuously learning how to produce more realistic data and the discriminator continuously improving its recognition capabilities.

[0045] In the application scenario of sarcopenia in the embodiment of the present application, the process of generating rare sarcopenia case data based on the generative adversarial network (GAN) is as follows: Figure 2 As shown, the following steps are included:

[0046] Step S2a: Collect a certain amount of rare sarcopenia case data to serve as training data for the generative adversarial network. The collected rare sarcopenia case data includes but is not limited to the patient's medical images, clinical records, etc.

[0047] Step S2b: The collected sarcopenia case data is input into a generative adversarial network for training. During the training process, the generator in the generative adversarial network learns how to imitate the distribution of real data to generate new rare sarcopenia case data. The discriminator in the generative adversarial network is used to distinguish generated cases from real cases.

[0048] Step S2c: The generated rare sarcopenia case data are evaluated based on several preset indicators, and the generated data that pass the evaluation are incorporated into the real data to synthesize the original training set.

[0049] In this way, GANs can help generate data on rare cases, improving the model's robustness when dealing with rare or unseen cases. This is crucial for improving the accuracy of sarcopenia diagnosis and the generalization ability of the model. This approach also helps protect patient privacy because it does not require the use of real patient data. Pre-defined metrics used include, but are not limited to, accuracy, precision, recall, ROC curves, F1 scores, and AUC values.

[0050] Dataset expansion method 2) Time warping and frequency adjustment of gait and force data to simulate various muscle states to improve the adaptability of the model.

[0051] Time warping can alter the temporal characteristics of gait data to simulate different gait rhythms and patterns. For example, time series data can be stretched or compressed to simulate changes in walking speed. Frequency adjustment can alter the frequency characteristics of data to simulate different muscle activity states. For example, increasing or decreasing the frequency of the data can simulate increases or decreases in muscle strength. By simulating various muscle states, the model can more accurately diagnose and predict a patient's health status. Furthermore, data augmentation can help protect patient privacy by reducing reliance on real patient data.

[0052] In some examples, preprocessing of the bioimpedance data includes normalizing the impedance signal and extracting frequency domain features related to muscle mass.

[0053] In bioelectrical impedance analysis, normalization of impedance signals is used to ensure data consistency and comparability. Normalization generally refers to converting impedance data into a ratio that is independent of an individual's height, weight, or other physiological parameters. For example, the axillary skeletal muscle mass index (aSMI) is the skeletal muscle mass of the axillary limbs (kg) divided by the square of the height (m), which is used to assess muscle mass. The extraction of frequency domain features involves Fourier transforming the bioelectrical impedance signal to analyze impedance changes at different frequencies. These features can reflect the physiological state and function of the muscle. Among them, the frequency domain features extracted that are related to muscle mass include median frequency (MF) and mean power frequency (MPF). The median frequency (MF) represents the median of the power spectral density function and reflects the center frequency of the signal. In muscle fatigue detection, a decrease in MF is usually associated with an increase in the degree of muscle fatigue. The mean power frequency (MPF) represents the average frequency of the power spectral density function and reflects the frequency distribution of the signal. Changes in MPF can also be used to assess the degree of muscle fatigue.

[0054] In some examples, the preprocessing method of the ultrasound image data includes: performing grayscale normalization processing on the ultrasound image data, separating muscle tissue from the grayscale normalized ultrasound image through an image segmentation model, and extracting structural features of muscle thickness and mass.

[0055] Grayscale normalization of ultrasound images involves subtracting the mean from the image pixel values and dividing them by the standard deviation, so that the image pixel distribution conforms to a normal distribution, with a pixel mean of 0 and a standard deviation of 1. This processing helps eliminate overall brightness offsets in the image, accelerates model convergence, and improves the model's stability to the input data.

[0056] Image segmentation models (such as the U-Net model) can segment ultrasound images to isolate muscle tissue. The U-Net model is a deep learning model suitable for segmentation tasks in medical image analysis. It has an encoder-decoder structure. The encoder is used to capture high-level semantic features and gradually reduce the spatial dimensions of the feature map, while the decoder is used to restore spatial details and dimensions. The ultrasound image, which has been processed by normalized grayscale, is input into the U-Net model. Each pixel is assigned to the muscle tissue or background category. The U-Net model extracts features and downsamples through the encoder, while the decoder gradually increases the size of the feature map, ultimately generating a segmentation result image. Muscle tissue features, such as muscle area and circumference, are extracted from the segmented image. These features are used to estimate muscle thickness and mass. The extracted features are quantitatively analyzed, such as calculating the average muscle thickness and muscle fiber distribution, to assess muscle health.

[0057] In some examples, preprocessing of the gait and force data includes converting the original gait and force data into a time series format and performing standard deviation processing.

[0058] Raw gait and force data include, but are not limited to, ground reaction force data, body and joint force data, and dynamic electromyographic data. These data are converted to a time series format, that is, sorted in time series, for easier analysis and processing. This conversion to a time series format can be achieved by aggregating gait information into a single image using statistical functions or by using sequence-based methods to learn temporal relationships within gait sequences. To reduce the impact of individual differences, the data needs to be standardized, including comparing the data with a reference population (divided by age and gender) to eliminate the influence of these variables. For example, spline curves and Poisson regression in the R language can be used to standardize gait and force data. Standardized data can be used for clinical analysis, such as assessing a patient's rehabilitation progress or diagnosing specific movement disorders. By comparing with standard values, reports containing joint motion angles, stride lengths, and stance and swing phase times can be generated, providing rapid feedback to physicians and therapists.

[0059] The model building module 102 is used to build a hybrid architecture by combining convolutional neural networks, graph neural networks and Transformer models to process the input bioimpedance data, ultrasound image data, gait and strength data to capture different aspects of muscle mass, function and quality, and output sarcopenia judgment results based on this.

[0060] Convolutional Neural Networks (CNNs) are deep learning models primarily designed for processing data with grid-like topological structures, such as images. CNNs utilize convolutional layers to automatically and adaptively learn spatial hierarchical features, enabling them to excel in visual tasks such as image recognition, video analysis, and image classification. The key components of a CNN include convolutional layers, activation layers, pooling layers, fully connected layers, normalization layers, loss functions, and optimizers.

[0061] In an embodiment of the present application, a convolutional neural network is used for bioimpedance frequency domain feature extraction and ultrasound image tissue feature recognition, and a one-dimensional convolutional neural network is used to extract the frequency domain features of bioimpedance for analyzing data related to muscle composition; and a two-dimensional convolutional neural network is used to identify tissue features in ultrasound images, including muscle thickness and muscle mass. One-dimensional convolutional neural network (1D-CNN) and two-dimensional convolutional neural network (2D-CNN) are two types of convolutional neural networks, which are suitable for processing one-dimensional and two-dimensional data, respectively. A one-dimensional convolutional neural network slides a convolution kernel on a one-dimensional signal to extract features, thereby being able to capture correlations in time series. A two-dimensional convolutional neural network extracts features by sliding a convolution kernel on a two-dimensional image, thereby capturing spatial correlations (such as objects, textures, and shapes in an image).

[0062] Both one-dimensional and two-dimensional convolutional neural networks consist of three convolutional layers, each followed by an activation function for nonlinear processing and a pooling layer for reducing the spatial size of features. For example, the first convolutional layer contains 32 filters, the second convolutional layer contains 64 filters, and the third convolutional layer contains 128 filters. This architecture allows the network to gradually extract features from simple to complex through multiple convolutional layers, ultimately outputting a planar feature representation.

[0063] In an embodiment of the present application, a Transformer model uses a visual Transformer (ViT) layer to process gait and force data to capture temporal dependencies and detect temporal changes in muscle force and gait consistency. The visual Transformer (ViT) layer includes two Transformer encoding blocks, each of which includes a self-attention layer and a dense layer; the self-attention layer is used to dynamically focus on the key points in the time series that are most relevant to muscle force and gait changes, and the dense layer is connected after the self-attention layer to perform further nonlinear transformations on the information of these key points.

[0064] The Visual Transformer (ViT) is a model that applies the Transformer architecture to vision tasks such as image classification. ViT works by dividing an image into small patches, treating these patches as elements in a sequence, and applying an encoding block to process them. ViT is able to capture spatial relationships and temporal dependencies in images, which are important for tasks such as understanding image content and performing classification.

[0065] The self-attention layer is the core of the Transformer model, which allows the model to consider all elements in the sequence simultaneously when processing each element in the sequence. This mechanism enables the model to capture the dependencies between different positions in the sequence, which is crucial for understanding complex relationships in sequential data such as text or time series. The self-attention layer works by computing internal representations of the query (Q), key (K), and value (V), and uses these representations to determine the importance of different elements in the sequence.

[0066] A dense layer, also known as a fully connected layer, connects each input element to each output element through weights. In the Transformer model, a dense layer typically follows the self-attention layer as part of a feed-forward neural network (FFN). An FFN typically consists of two dense layers with a nonlinear activation function (such as ReLU) sandwiched between them. The dense layer performs further nonlinear transformations on the output of the self-attention layer, enhancing the model's expressive power.

[0067] Graph neural networks are used to construct graph structures by taking bioelectrical impedance data, ultrasound image data, gait and strength data as nodes and the associations between these modal data as edges, for example Figure 3 As shown in the figure, various data themselves are used as nodes, and the connections are used as edges. A graph attention network is used to assign edge weights between different nodes through the attention mechanism.

[0068] Specifically, bioelectrical impedance data, ultrasound image data, gait, and force data are used as nodes in the graph. Each node contains its corresponding feature vector, which can be the node's attributes, such as the resistance value of bioelectrical impedance, image features of ultrasound imaging, and time series data of gait. Each edge represents the association between different modalities. These associations can be based on prior knowledge (such as the physical connection between different modalities) or determined through data-driven methods (such as correlation analysis).

[0069] The Graph Attention Network (GAT) dynamically assigns edge weights between different nodes through the attention mechanism, thereby achieving the priority combination of the most relevant features. In GAT, the interaction between nodes is learned through the corresponding attention coefficients. The feature update of each node depends not only on its own features, but also on the features of its neighboring nodes, which are obtained by weighted summation of the attention coefficients. In this way, the model can adaptively learn the importance of different neighboring nodes, thereby prioritizing the combination of the most relevant features. The node feature representations learned by the GAT model can be used to predict the risk of sarcopenia. These feature representations can be input into a classifier or regressor to generate a sarcopenia risk score. The score calculation formula is as follows:

[0070]

[0071] Among them, a i Representation characteristics i The attention weights are obtained by multiplying each feature with its corresponding attention weight and then summing these products.

[0072] The diagnosis module 103 is connected to the hybrid architecture built by the model building module 102 and includes a multi-layer perceptron. The fusion features output by the hybrid architecture are passed to the multi-layer perceptron for classification to predict the presence and stage of sarcopenia.

[0073] A multilayer perceptron (MLP) is a feedforward artificial neural network consisting of an input layer, one or more hidden layers, and an output layer. MLPs are trained using optimization algorithms such as gradient descent to minimize a loss function, thereby learning the mapping between input data and output targets.

[0074] In some examples, the MLP uses a binary cross entropy loss function to predict the presence of sarcopenia. The binary cross entropy loss function is constructed as follows:

[0075] L = -(y·log(p)+(1-y)·log(1-p)); Formula (2)

[0076] Where y is the true label and p is the probability that the model predicts the sample to be positive. The smaller the value of the binary cross entropy loss function, the higher the prediction accuracy of the model.

[0077] In some examples, the MLP uses a categorical cross entropy loss function to predict different stages of sarcopenia, including early sarcopenia, moderate sarcopenia, and severe sarcopenia. The categorical cross entropy loss function is constructed as follows:

[0078]

[0079] Where C is the total number of categories, y i is the one-hot encoding of the true label, p i is the probability that the model predicts that the sample belongs to the i-th class. The categorical cross entropy loss function optimizes the model by minimizing the difference between the predicted probability distribution and the true distribution.

[0080] In a preferred embodiment of the present application, the multimodal sarcopenia diagnostic system adapts to new tasks and / or new data through personalized adaptive meta-learning.

[0081] Meta-learning with personalized adaptation is a machine learning method that aims to improve a model's generalization ability by quickly adapting to new tasks or data. This approach is particularly valuable in the healthcare field because it can be used for personalized diagnosis, where a model is adjusted based on the specific data of an individual patient to provide more accurate medical advice or diagnosis.

[0082] In some examples, a multimodal sarcopenia diagnostic system adapts to new tasks and / or new data through small-sample meta-learning, which includes: using the MAML meta-learning framework to fine-tune a model that has been pre-trained on a large data set using a small amount of new patient data. In this way, personalized diagnosis can be achieved, and parameters are adjusted through an outer loop and individual data is optimized through an inner loop, reducing the need for large-scale retraining. It should be understood that MAML (Model-Agnostic Meta-Learning) is a meta-learning framework that allows any machine learning model to quickly adapt to new tasks through a small amount of gradient updates. The MAML meta-learning framework is implemented through an outer loop and an inner loop. In the outer loop, the parameters of the model are adjusted so that the model can better adapt to new tasks. In the inner loop, a small amount of data is used on each new task for rapid fine-tuning to optimize the model's performance on the task. In this way, the MAML meta-learning architecture can quickly adapt to new patient data and achieve personalized diagnosis without large-scale retraining.

[0083] In some examples, a modality-based sarcopenia diagnosis system uses domain adaptation to adjust for differences between different data distributions. Domain adaptation aims to adapt the model to new data distributions, addressing differences between different data distributions to ensure stable performance across different populations or medical settings. It also adapts to baseline data from different populations (e.g., bioelectrical impedance, ultrasound, and gait data).

[0084] The model interpretation module 104 is connected to the diagnosis module 103 and is used to perform corresponding model interpretation on the sarcopenia risk score output by the diagnosis module 103 .

[0085] In some examples, the model interpretation module 104 performs a corresponding model interpretation on the sarcopenia risk score based on the Shapley value feature attribution method. The Shapley value feature attribution method provides a method for quantifying and explaining model predictions, which helps improve the interpretability and trustworthiness of the model. The process includes:

[0086] First, for each feature in the model, its contribution to the model's predictions is calculated. This is done by considering the contribution of each feature across all possible feature combinations. The Shapley value determines feature importance by considering all possible feature combinations and averaging the marginal contributions of each feature across these combinations.

[0087] Next, the marginal contributions of each feature are summed up to obtain the Shapley value of each feature. This value represents the average contribution of the feature to the model prediction, that is, the average change in the model prediction result if the feature is removed or added.

[0088] Finally, the calculated Shapley values are used to interpret the model's predictions, allowing clinicians to understand which features have the greatest impact on the final risk score through these values, thereby better understanding the source and basis of the diagnosis.

[0089] In some examples, the model explanation module 104 provides an analysis of the impact of input changes on prediction results based on the LIME method, thereby enhancing the transparency of sarcopenia diagnosis. LIME (Local Interpretable Model-Agnostic Explanations) is a locally interpretable, model-agnostic explanation method that can explain any decision of any model. Its core idea is to explain the decision-making process of the original model by constructing a simplified model. The specific process is as follows:

[0090] First, neighborhood data are randomly sampled from the original data set, including generating some perturbation samples around the sample to be explained. These samples are used to simulate the local neighborhood environment of the original sample.

[0091] Secondly, a simplified model is trained on the neighborhood data, including using these perturbation samples. LIME trains a simplified model (usually a linear model or decision tree) that can approximate the behavior of the original model in the local neighborhood.

[0092] Finally, the simplified model is used to explain the original model’s decisions. By analyzing the weights or decision paths of the simplified model, LIME provides an explanation of the original model’s predictions, allowing users to understand which features have the greatest impact on the model’s predictions.

[0093] In some examples, the model interpretation module 104 performs model interpretation by generating an attention heat map and a saliency map.

[0094] An attention heatmap is a visualization technique used to show the data points or feature areas that a model focuses on when making a prediction. In medical image analysis, this can help highlight areas that the model considers important, such as tumors, lesions, or other pathological features. Attention heatmaps can be generated using a variety of methods, including techniques such as Class Activation Mapping (CAM) and Grad-CAM. These techniques introduce an attention mechanism in the last few layers of the model to generate a heatmap, which is then superimposed on the original image to intuitively demonstrate the basis for the model's decision. A saliency map is an image that highlights the areas in an image that our eyes first focus on. In medical image analysis, saliency maps can be used to identify significant features in an image that are very important to the human visual system. The goal of a saliency map is to reflect the importance of pixels to the human visual system.

[0095] The combined use of attention heatmaps and saliency maps not only enhances the transparency of AI diagnosis but also provides clinicians with an intuitive tool to better understand and utilize the predictions of AI models. This visualization approach helps build physicians' trust in AI-assisted diagnosis and may improve patient treatment outcomes. In the diagnosis of sarcopenia, attention heatmaps and saliency maps can help clinicians quickly identify features that are critical to diagnosis by highlighting key areas that the model considers important. They also help explain the predictions of AI models, allowing physicians to better understand the model's decision-making process. By identifying key areas, physicians can develop more targeted treatment plans, such as exercises and nutritional supplements for specific muscle groups.

[0096] It should be understood that the division of modules in the embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present application may be integrated into a single processor, or may exist physically as separate modules, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0097] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" represent examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0098] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0099] Figure 4 A schematic flow chart of a multimodal sarcopenia diagnosis method according to an embodiment of the present application is shown, including:

[0100] Step S41: Collect and pre-process bioimpedance data, ultrasound image data, gait and strength data related to muscle mass.

[0101] Step S42: A hybrid architecture is constructed by combining convolutional neural networks, graph neural networks, and Transformer models to process the input bioimpedance data, ultrasound image data, gait, and strength data to capture different aspects of muscle mass, function, and quality, and output sarcopenia judgment results based on them.

[0102] Step S43: Use a multi-layer perceptron to receive the fusion features output by the hybrid architecture and perform classification to predict whether sarcopenia exists and the stage of sarcopenia.

[0103] Step S44: Perform corresponding model interpretation on the sarcopenia risk score output by the diagnosis module.

[0104] It should be noted that the implementation process and principle of the modal sarcopenia diagnostic method in the embodiment of the present application are similar to those of the modal sarcopenia diagnostic system mentioned above, and will not be repeated here.

[0105] Figure 5 : is a schematic block diagram of a computer device provided in an embodiment of the present application. Figure 5 As shown, the computer device includes: at least one processor 501, a memory 502, at least one network interface 503 and a user interface 505. The various components in the device are coupled together via a bus system 504. It is understood that the bus system 504 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 504 is not described in detail. Figure 5 Various buses are labeled as bus systems.

[0106] The user interface 505 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0107] It will be appreciated that the memory 502 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM) or a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0108] The memory 502 in the embodiment of the present invention is used to store various categories of data to support the operation of the computer device 500. Examples of these data include: any executable program for operating on the computer device 500, such as an operating system 5021 and an application 5022; the operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 5022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The sarcopenia diagnostic method provided by the embodiment of the present invention can be included in the application 5022.

[0109] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 501 or by software instructions. The above processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 501 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor 501 may be a microprocessor or any conventional processor. The steps of the accessory optimization method provided in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium located in a memory. The processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0110] In an exemplary embodiment, the computer device 500 may be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), and complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0111] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: a computer program code, which, when executed on a computer, causes the computer to execute Figures 1 to 4 The multimodal sarcopenia diagnostic method of any one of the illustrated embodiments.

[0112] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code. When the program code is run on a computer, the computer executes the above method.

[0113] As used in this specification, the terms "component," "module," "system," and the like are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a computing device and a computing device can be a component. One or more components can reside in a process and / or an execution thread, and a component can be located on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures stored thereon. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component on a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0114] Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0117] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0119] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (program) are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0120] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store program codes.

[0121] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0122] In summary, the present application provides a multimodal sarcopenia diagnostic system, method, device and medium. The technical framework of the present invention integrates multimodal data integration, complex neural network technology, personalized meta-learning algorithm and explainable artificial intelligence technology, aiming to create an accurate and highly adaptable sarcopenia diagnostic tool. This method not only makes a breakthrough based on traditional bioelectrical impedance technology, but also demonstrates great potential for efficient, personalized diagnosis and monitoring in a diversified medical environment through the application of multimodal data fusion and intelligent models. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0123] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A multimodal sarcopenia diagnostic system, characterized in that: include: A multimodal data preprocessing module for collecting and preprocessing bioimpedance data, ultrasound image data, gait and force data related to muscle mass; A model building module, connected to the multimodal data preprocessing module, is used to build a hybrid architecture combining convolutional neural networks, graph neural networks, and Transformer models to process input bioimpedance data, ultrasound image data, gait, and strength data to capture different aspects of muscle mass, function, and quality, and output sarcopenia diagnosis results based on them; a diagnosis module, connected to the model building module, comprising a multi-layer perceptron, wherein the fusion features output by the hybrid architecture are passed to the multi-layer perceptron for classification to predict the presence and stage of sarcopenia; The model interpretation module is connected to the diagnosis module and is used to perform corresponding model interpretation on the sarcopenia risk score output by the diagnosis module.

2. The multimodal sarcopenia diagnostic system according to claim 1, characterized in that: The multimodal data preprocessing module is further used to generate rare sarcopenia case data based on a generative adversarial network to expand the training set, the process of which includes: Collect a certain amount of rare sarcopenia case data as training data for the generative adversarial network; The collected sarcopenia case data is fed into a generative adversarial network for training. During the training process, the generator in the generative adversarial network learns how to mimic the distribution of real data to generate new rare sarcopenia case data. The discriminator in the generative adversarial network is used to distinguish between generated cases and real cases. The generated rare sarcopenia case data were evaluated based on several preset indicators, and the generated data that passed the evaluation were incorporated into the real data to synthesize the original training set.

3. The multimodal sarcopenia diagnostic system according to claim 1, characterized in that: The multimodal data preprocessing module preprocesses bioimpedance data related to muscle mass, ultrasound image data, gait and strength data, which includes: The preprocessing methods of bioimpedance data include: normalizing the impedance signal and extracting frequency domain features related to muscle mass; The preprocessing method of the ultrasound image data includes: performing grayscale normalization processing on the ultrasound image data, separating muscle tissue from the grayscale normalized ultrasound image using an image segmentation model, and extracting structural features of muscle thickness and quality; The preprocessing of gait and force data includes converting the original gait and force data into time series format and performing standard deviation processing.

4. The multimodal sarcopenia diagnostic system according to claim 1, characterized in that: In the model building module: Convolutional neural networks are used for bioimpedance frequency domain feature extraction and ultrasound image tissue feature recognition. A one-dimensional convolutional neural network is used to extract the frequency domain features of bioimpedance to analyze data related to muscle composition; and a two-dimensional convolutional neural network is used to identify tissue features in ultrasound images, including muscle thickness and muscle mass.

5. The multimodal sarcopenia diagnostic system according to claim 1, characterized in that: In the model building module: The Transformer model uses visual Transformer layers to process posture and force data to capture temporal dependencies and detect temporal changes in muscle force and gait consistency; The visual Transformer layer includes two Transformer encoding blocks, each of which includes a self-attention layer and a dense layer; the self-attention layer is used to dynamically focus on the key points in the time series that are most relevant to muscle strength and gait changes, and the dense layer is connected to the self-attention layer to perform further nonlinear transformations on the information of these key points.

6. The multimodal sarcopenia diagnostic system according to claim 1, characterized in that: In the model building module: Graph neural networks are used to construct graph structures using bioelectrical impedance data, ultrasound image data, gait and strength data as nodes and the correlation between these modal data as edges; a graph attention network is used to assign edge weights between different nodes through the attention mechanism.

7. The multimodal sarcopenia diagnostic system according to claim 1, characterized in that: The multilayer perceptron in the diagnostic module uses a binary cross entropy loss function to predict the presence of sarcopenia, and / or uses a categorical cross entropy loss function to predict different stages of sarcopenia; the sarcopenia stages include early sarcopenia, moderate sarcopenia, and severe sarcopenia.

8. A multimodal sarcopenia diagnostic method, characterized in that: Applied to a computer device, the method comprises: Acquire and preprocess bioimpedance data related to muscle mass, ultrasound image data, gait and force data; A hybrid architecture combining convolutional neural networks, graph neural networks, and Transformer models processes input bioimpedance data, ultrasound image data, gait, and strength data to capture different aspects of muscle mass, function, and quality, and outputs sarcopenia diagnosis results. Using a multilayer perceptron to receive and classify the fusion features output by the hybrid architecture to predict the presence and stage of sarcopenia; Provide corresponding model interpretation for the sarcopenia risk score output by the diagnostic module.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal sarcopenia diagnosis method according to claim 8 is implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the multimodal sarcopenia diagnosis method according to claim 8.

Images