Chain prompt and multi-modal large model-based sarcopenia detection method and device

Abstract

The application discloses a sarcopenia detection method and device based on a chain prompt and a multi-modal large model, and belongs to the technical field of video processing. The method comprises the following steps: collecting a patient action video, and dividing each frame into a plurality of small blocks to be mapped into a feature vector sequence; information fusion is performed between the same frame and different time points, and a global visual feature vector with spatial and temporal context is output; a continuous feature sequence is cut into a plurality of semantic coherent action stages; corresponding segmented prompt text is retrieved from a pre-defined mapping table based on the test type; a natural language description containing quantitative indicators and action features is generated at one time; a sarcopenia diagnosis and its detailed reasons are output; a probability value is taken as the confidence of the diagnosis; and a complete diagnosis report containing quantitative and qualitative analysis is output by one key. The application can greatly improve the universality and robustness across scenes and devices, and significantly improve the explainability of model decision.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method and device for sarcopenia detection based on chained prompts and a multimodal large model. Background Technology

[0002] As video analytics technology is increasingly applied in the field of medical rehabilitation, existing video-based sarcopenia detection methods mainly rely on traditional visual modules or shallow learning algorithms, such as human skeletal keypoint detection and manually designed gait feature extraction. These methods have the following main shortcomings:

[0003] 1. Poor versatility and robustness

[0004] Traditional skeletal key point detection algorithms are often optimized for specific shooting angles, lighting conditions, or human postures. Once the shooting environment, camera equipment, or patient posture changes, the data needs to be re-labeled and the parameters readjusted, making it difficult to adapt to clinical applications in multiple scenarios and among multiple populations.

[0005] 2. Insufficient interpretability

[0006] Existing methods typically output only a binary classification result of "probability of sarcopenia" or "health / risk," lacking quantitative explanation and logical description of the model's decision-making process. They cannot produce quantitative exercise indicators and inference chain reports, making it difficult for clinicians and patients to have sufficient trust in the results.

[0007] 3. Dependence on large-scale annotation

[0008] Traditional algorithms are prone to overfitting with a small number of labeled samples, while fine-tuning for large-scale pre-trained models can improve performance, but it has extremely high requirements for labeling, computing power and deployment costs, making it impossible to extend to primary healthcare or home rehabilitation monitoring.

[0009] 4. Lack of a complete diagnostic process

[0010] Existing technologies cannot connect "action time sequence segmentation → feature quantification → multi-step logical reasoning → visualization report", and cannot generate a comprehensive diagnostic report containing time sequence segment analysis, key indicators and chain reasoning with one click, which limits the practical application of scenarios such as telemedicine and intelligent monitoring.

[0011] Chinese Patent CN113488163B discloses a machine vision-based method, device, equipment, and medium for sarcopenia identification. This technology employs a machine vision approach for sarcopenia identification, but its technical process has significant limitations. The scheme requires multiple complex steps, including acquiring video streams, obtaining raw frame image sequences, processing images using human pose recognition algorithms to extract human key points and gait contour maps, performing affine transformations on human key points to obtain multi-angle data, and calculating gait data accordingly. Finally, a gait recognition network needs to be established and trained to output the recognition result. Each step requires close coordination; errors in any step can affect the final recognition result. This places extremely high demands on the operator's skill and system stability, making debugging and maintenance difficult, thus hindering the overall implementation of the process.

[0012] Chinese patent application CN117333932A discloses a method, equipment, device, and medium for recognizing sarcopenia based on machine vision. This technology has certain advantages in solving existing diagnostic problems, but it also has the following disadvantages:

[0013] First, the diagnostic criteria of this scheme are relatively singular, primarily relying on chair standing time to identify sarcopenia, which limits its effectiveness. Although studies have shown that standard chair standing time can serve as a sensitive marker of sarcopenia, and experiments have established a relationship between chair standing time and gait speed, validating its effectiveness as a gait speed surrogate, the pathogenesis of sarcopenia is complex, involving changes in muscle mass, strength, and function. A single chair standing time indicator is insufficient to encompass these complex factors.

[0014] Secondly, this solution requires explicit modeling of the human skeleton, which increases the complexity and cost of technical implementation. Obtaining key points of the entire human body from different angles and calculating skeletal feature maps involves complex operations such as affine transformations and spatial coordinate calculations, placing high demands on the computing power of the equipment and the accuracy of the algorithm. It not only requires high-performance hardware to support the computation but also complex algorithm design and optimization, which increases development costs and technical difficulty. Summary of the Invention

[0015] The technical problem to be solved by the present invention is to provide a method and device for sarcopenia detection based on chained prompts and a multimodal large model that is universal, robust, and interpretable.

[0016] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0017] This invention provides a method for sarcopenia detection based on chained prompts and a multimodal large model, comprising:

[0018] Step S1: Acquire patient action video under fixed angle and shooting parameters, decode the video into frames, crop each frame image, unify the size and normalize the pixels, and divide each frame image into several small blocks and map them into feature vector sequences.

[0019] Step S2: Input the feature vector sequence into a multilayer alternating coding network, perform information fusion between the same frame and different time points, and output a global visual feature vector with both spatial and temporal context for subsequent action segmentation;

[0020] Step S3: Automatically detect action transition points based on the differences in global visual feature vectors of adjacent frames, divide the continuous feature sequence into multiple semantically coherent action stages, obtain the index of each sub-stage, and provide accurate temporal segmentation for subsequent type discrimination and description generation;

[0021] Step S4: Determine the global cue template set according to the test category to which the current video belongs, and then index each sub-stage, retrieve the corresponding segmented cue text from the global cue template set based on the test type, and map the segmented cue text into a cue embedding with the same dimension as the global visual feature vector through a text encoder, so as to provide differentiated semantic input for subsequent description generation;

[0022] Step S5: For each action segment, the mean of the global visual feature vector of the action segment is concatenated with the corresponding prompt embedding and then input into the visual-text large model to generate a natural language description containing quantitative indicators and action features in one go, thus obtaining the description of each segment, providing a complete and interpretable semantic context for subsequent few-shot reasoning.

[0023] Step S6: Connect a small number of labeled examples with the descriptions in each paragraph to form a thought chain prompt, input it into the visual-text large model, complete multi-step logical reasoning under the guidance of the examples, and directly output the sarcopenia diagnosis and its detailed reasons. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the sarcopenia detection method based on chained prompts and a multimodal large model of the present invention.

[0025] Figure 2 These are partial frame images of video clips collected in an actual operation case of the present invention;

[0026] Figure 3 This is a schematic diagram of the preprocessed video frames in an actual operation example of the present invention;

[0027] Figure 4 This is an example of reporting key information in a real-world application of the present invention;

[0028] Figure 5This is a schematic diagram of the sarcopenia detection device based on chained prompts and a multimodal large model according to the present invention. Detailed Implementation

[0029] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0030] To address the technical bottlenecks in the background art, this invention proposes a video sarcopenia detection method and device "based on chained prompts and multimodal large model". Its main objective is to:

[0031] 1. By adaptively detecting visual features between video frames, implicit segmentation of action stages such as sitting-standing and walking can be achieved without any external skeleton or manual feature modules, greatly improving the versatility and robustness across scenes and devices.

[0032] 2. By using a small number of high-quality examples and contextual thinking chain prompts, the black-box probabilistic prediction is transformed into a multi-step logical reasoning process, and the output is natural language analysis rich in quantitative motion indicators, which significantly improves the interpretability of model decision-making.

[0033] 3. With minimal need for labeled data, the system can quickly adapt to new scenarios through prompt mapping and contextual learning, reducing reliance on large-scale fine-tuning, balancing computing power and labeling costs, and enhancing the system's deployability in primary healthcare and home environments.

[0034] 4. Finally, a complete integrated diagnostic report is generated (including action segment descriptions, quantitative indicators, and analytical reasoning chains), meeting the multiple needs of clinical practice and patent applications for readability, interpretability, and applicability.

[0035] On the one hand, this invention provides a (video) sarcopenia detection method based on chained prompts and a multimodal large model, such as... Figure 1 As shown, it includes:

[0036] Step S1: Acquire patient action video under fixed angle and shooting parameters, decode the video into frames, crop each frame image, unify the size and normalize the pixels, and divide each frame image into several small blocks and map them into feature vector sequences.

[0037] In this step, to ensure consistent and high-quality input across different acquisition devices and environments, fixed-angle (front and side) cameras were first used to capture video of the patient performing three actions on a flat surface: the "sit-to-stand test," the "single-leg standing test," and the "walking test," encoded in H.264 at 30 fps and with a resolution of 1920×1080. Each test lasted approximately 30 seconds. After acquisition, the video stream was decoded, and frames were uniformly extracted. , ,in This provides the basic data for subsequent frame-level processing.

[0038] Each frame of the image is cropped based on a fixed shooting angle, retaining only the main human body area, and the cropped result is scaled to a preset size (specifically, it can be...). The resolution is adjusted to standardize the input size. Then, pixel values ​​from 0-255 are linearly mapped to... :

[0039] ;

[0040] This is to eliminate numerical offsets caused by differences in the acquisition equipment.

[0041] As an optional embodiment, step S1 includes:

[0042] After pixel normalization, the image is rearranged from BGR (blue-green-red) channels to RGB (red-green-blue) channels. Then, based on the channel mean and standard deviation used by the ImageNet pre-trained model, the pixel values ​​of each channel are standardized to adjust the data to a zero-mean, unit-variance distribution.

[0043] In practice, to maintain consistency with the channel order during the pre-training of the large model, the BGR channels are then rearranged into RGB channels:

[0044] ;

[0045] Subsequently, based on the channel mean used in the ImageNet pre-trained model with standard deviation Perform normalization on the pixel values ​​of each channel:

[0046] ;

[0047] Adjust the data to a zero-mean, unit-variance distribution to improve the model's generalization ability.

[0048] After completing pixel-level preprocessing, the standardized image will be... According to a fixed size (like Evenly cut into Each local patch (small patch / plot / subgraph) is denoted as . To capture fine-grained spatial information.

[0049] As another optional embodiment, step S1 includes:

[0050] Each patch is flattened and projected onto a learnable linear layer. The dimensional embedding space is used to obtain the patch embedding of each patch;

[0051] A trainable classification token is inserted before all patch embeddings in each frame to obtain the sequence of patch tokens corresponding to the action video, which is the feature vector sequence.

[0052] In practice, for each Flattening to obtain vectors And project it onto a learnable linear layer. 3D embedding space, to obtain the 1st Frame number Embedded patch:

[0053]

[0054] in, and These are the projection matrix and the bias vector, respectively.

[0055] To extract global context information for each frame, a trainable classification token is inserted before all patch embeddings in each frame. The final result is the sequence of patch tokens (i.e., feature vector sequence) corresponding to the action video:

[0056] ;

[0057] This will be used as input for the subsequent step S2 (visual feature encoding).

[0058] Step S2: Input the feature vector sequence into a multilayer alternating coding network, perform information fusion between the same frame and different time points, and output a global visual feature vector with both spatial and temporal context for subsequent action segmentation;

[0059] In this step, a multi-layered, alternately stacked attention and feedforward structure is used, so that each patch token from step S1 can not only perceive neighborhood information within the same frame, but also capture motion changes in the video across frames, thereby generating a high-quality spatiotemporal feature representation to support the accurate localization of subsequent action segments.

[0060] As an optional embodiment, step S2 includes:

[0061] Step S21: Concatenate the patch embeddings of all frames with the classification tokens in chronological order to form the initial sequence (i.e., the first layer output).

[0062] The initial sequence formed in this step is as follows:

[0063]

[0064] in, Used to aggregate global information from each frame. Indicates the first Frame number Embed a patch. It is a matrix that contains all the patch embeddings for the T-frame.

[0065] Step S22: In the first During the layer encoding process, layer normalization is first performed on the upper layer output to eliminate the magnitude difference between different dimensions. Then, the association weights between each category token are calculated through multi-head self-attention, and the original input information is retained to obtain the intermediate vector.

[0066] In this step, at the... During the layer encoding process, the upper layer output is first processed. The execution layer is normalized to eliminate the magnitude difference between different dimensions. Then, the association weights between each token are calculated through multi-head self-attention, while retaining the original input information, to obtain the intermediate vector. :

[0067]

[0068] in, Representation layer normalization. This refers to multi-head self-attention, where residual connections allow for stable training and preservation of original features.

[0069] Step S23: Perform layer normalization on the intermediate vector again to adapt to the input distribution of the subsequent feedforward network. Then, use a position-aware feedforward network composed of two fully connected layers and nonlinear activation layers, and perform residual fusion with the input to generate the output of this layer.

[0070] In this step, for Layer normalization is performed again to adapt to the input distribution of subsequent feedforward networks. Then, a position-aware feedforward network consisting of two fully connected layers and nonlinear activation layers is used, and residual fusion is performed with the input to generate the first... Layer output:

[0071]

[0072] in, This refers to a feed-forward network, which enhances feature representation through position-by-position nonlinear transformations.

[0073] Step S24: Repeat steps S22-S23 to complete the calculation of all coding layers and obtain the final output. Extract the classification token corresponding to each frame as the global visual feature vector.

[0074] In this step, formulas (2a)-(2b) are executed repeatedly. After this, the calculations for all coding layers are completed, and the final output is obtained:

[0075]

[0076] From these outputs, extract the category token corresponding to each frame, i.e. These features serve as frame-level global visual feature vectors. They contain both spatial information within the frame and motion context between different moments, providing a solid feature foundation for implicit action segmentation in step S3.

[0077] Step S3: Automatically detect action transition points based on the differences in global visual feature vectors of adjacent frames, divide the continuous feature sequence into multiple semantically coherent action stages, obtain the index of each sub-stage, and provide accurate temporal segmentation for subsequent type discrimination and description generation;

[0078] This step aims to automatically identify action switching points by measuring global visual feature changes in adjacent frames of a video without any manual annotation, and to divide the entire video sequence into several semantically coherent sub-stages in order to perform more refined analysis and reasoning for different types of experiments.

[0079] Specifically, this system will segment the videos of the three different experiments as follows:

[0080] (1) Sitting-standing test

[0081] In this experiment, the patient repeatedly performed the "sit to stand" action several times. The system first calculated the global feature difference value between two adjacent frames. When the feature difference value exceeded an adaptive threshold, it was determined as the boundary of a single standing or sitting action. This automatically divided the entire 30-second video segment into several complete "sit → stand" and "stand → sit" cycles. Each cycle was identified as a sub-segment for subsequent analysis of indicators such as "time to stand up" and "smoothness of sitting down."

[0082] (2) Single-leg standing test

[0083] In this experiment, patients were required to stand on one leg and maintain balance until the end. The system also used inter-frame feature differences to identify two key moments: "establishing a single-leg standing posture" and "losing balance or putting down the leg." Therefore, a single-leg standing video is generally divided into two sub-segments: the first sub-segment corresponds to the patient's transition from two legs to one leg and the initial stability achieved, while the second sub-segment corresponds to the slight swaying that may occur during the single-leg standing process and the final stage of putting down the leg; if the patient maintains balance throughout, the entire video is considered as a single complete "stable standing" sub-segment.

[0084] (3) Walking test

[0085] For the gait test, the patient walks in a continuous gait on screen. The system detects the start and end boundaries of each complete gait cycle, dividing the entire gait video into several "single gait cycle" segments. Each segment covers a complete left-foot step and right-foot step cycle, facilitating the subsequent extraction of gait features such as "step frequency," "stride consistency," and "acceleration changes" and the generation of segmented descriptions.

[0086] As an optional embodiment, step S3 includes:

[0087] Step S31: Calculate the Euclidean distance between the global visual feature vectors of two adjacent frames. ;

[0088] In this step, the classification token feature sequence for each frame is extracted from the L layer output of the Vision Transformer (ViT). This sequence contains both spatial information from all local patches within each frame and motion context from preceding and following frames. The intensity of motion changes can be quantified by calculating the Euclidean distance between the global visual feature vectors of adjacent frames.

[0089]

[0090] when When the value increases significantly, it indicates that in the first... Frame to the There are clear transitions in action between frames.

[0091] Step S32: For Applying moving average smoothing, we obtain ;

[0092] In this step, to suppress short-term peaks caused by characteristic noise or minute jitter, the following steps can be taken: Apply moving average smoothing:

[0093] ;

[0094] Among them, window width The validation set experiment was set to 5 frames. Smoothed. It can more accurately reflect the actual trend of movement changes.

[0095] Step S33: Statistically analyze the difference distribution of the entire video and calculate the mean. with standard deviation and with adaptive threshold As a criterion for action switching, continuous frame features are divided into several action stages, among which... These are preset coefficients.

[0096] In this step, the system statistically analyzes the difference distribution across the entire video and calculates the mean and standard deviation: ;

[0097] and with adaptive threshold ( A value of 1.0 to 1.5 is used as the criterion for action switching. In specific implementation, all actions that meet the criteria... (or directly) The frame index of the segment boundary is considered as the segment boundary, and together with the first and last frames (of the entire video), they form the complete set of segment boundary indices. .

[0098] To avoid generating excessively short segments that may contain noise, a minimum segment length constraint is introduced: if two adjacent stage boundaries... and The number of frames between them is less than the preset lower limit Then the two corresponding segments will be merged to ensure that each stage has enough time for subsequent analysis.

[0099] Finally, the continuous frame features are divided into several action stages:

[0100] ;

[0101] Each Each represents a complete and semantically consistent action segment, providing a precise temporal basis for subsequent segment type identification and segment description generation.

[0102] Step S4: Determine the global prompt template set according to the test category to which the current video belongs, and then index each sub-stage, retrieve the corresponding segmented prompt text from the global prompt template set based on the test type, and map the segmented prompt text into a prompt embedding with the same dimension as the global visual feature vector through a text encoder, so as to provide differentiated semantic input for subsequent description generation;

[0103] This step first determines the global cue template set based on the trial category of the current video ("sitting-standing trial", "standing on one leg trial", or "walking trial"). Then, for each sub-stage index divided in step S3, the corresponding segmented cue text is retrieved from the template set corresponding to that trial. The segmented cue text is then vectorized into a cue embedding with the same dimension as the visual features. This allows the multimodal large model to focus on the quantitative indicators required for each trial when generating descriptions of each segment, achieving differentiated and accurate Chain-of-Thought (CoT) analysis.

[0104] The system pre-builds a mapping table from tags to prompt templates (sets) during the initialization phase. The three tests (i.e., labels) – “sitting-standing test,” “single-leg standing test,” and “walking test” – correspond one-to-one with their respective template sets: the “sitting-standing test” template set contains the key points of analysis for the standing / sitting cycle; the “single-leg standing test” template set contains the key points of analysis for the single-leg balance phase; and the “walking test” template set contains the key points of analysis for the gait cycle.

[0105] Regarding the first In the joke stage, the semantic meaning corresponding to the joke within this experiment has already been determined by step S3. The corresponding prompt template is extracted from the mapping table using the following formula:

[0106]

[0107] For example, Then the first The segment is a complete "sit → stand" or "stand → sit" cycle. For template examples, please refer to the Prompt template corresponding to the "sit-stand test" in Table 1.

[0108] To seamlessly integrate the prompt text with visual features, the system invokes the text encoder. The extracted Prompt (cue word / cue text) is vectorized to obtain a cue embedding with the same dimension as the visual features:

[0109] ;

[0110] in, By using the built-in text branches of the large model, we can ensure that the semantics of the prompts are accurately expressed in the high-dimensional space.

[0111] As an optional embodiment, in step S4, for the sit-to-stand test, the corresponding segmented prompt text in the global prompt template set includes descriptions related to "speed from sitting to standing," "postural stability," and "upper limb assistance"; for the single-leg standing test, the corresponding segmented prompt text in the global prompt template set includes descriptions related to "preparatory movement speed," "upper limb stability," "balance time," "dynamic changes in center of gravity," and "body sway amplitude"; for the walking test, the corresponding segmented prompt text in the global prompt template set includes descriptions related to "gait characteristics," "trunk stability," "upper limb swing," and "step frequency and stride length." In specific implementation, Table 1 provides an example of the complete mapping relationship:

[0112] Table 1

[0113] Segment type tags Prompt Template Sit-stand test Please analyze a complete sit-to-stand cycle from the "sit-to-stand test" using the following steps, returning the results in JSON format first, followed by a natural language description of at least 50 words: 1. Sit-to-stand speed: Observe and calculate the time (in seconds) for standing up, determining if there is any slowness of movement. If it exceeds a preset threshold, mark it as "slow"; otherwise, mark it as "normal". 2. Postural stability: Assess whether there is any forward / backward or left / right tilting during the standing process. If there is a significant tilt exceeding the threshold, mark it as "unstable"; otherwise, mark it as "stable". 3. Upper limb assistance: Check whether the subject uses their arms for support (e.g., holding onto a chair) when standing up. If assistance is detected, mark it as "using assistance"; otherwise, mark it as "no assistance". Example JSON output: { "Phase": "Sit-to-Stand Test_Cycle 1", "Time to Stand_s": 2.5, "Slowness of Movement": "Normal", "Postural Stability": "Stable", "Upper Limb Assistance": "No Assistance"} Example Natural Language Description: This sit-to-stand cycle took approximately 2.5 seconds, with no significant slowness of movement. The center of gravity did not shift forward, backward, or sideways during the standing process, indicating a stable state. The subject did not use their arms for support, and the entire movement was fluid. This demonstrates good muscle strength and balance. Single-leg standing test For the "Single-Leg Standing Test" phase, please analyze the following steps and return the results in JSON format first, followed by a natural language description of no less than 50 words: 1. Preparatory Movement Speed: Record and calculate the time (in seconds) for transitioning from standing on two feet to standing on one foot. Determine if there is any slowness of movement. If it exceeds a preset threshold, mark it as "slow"; otherwise, mark it as "normal". 2. Upper Limb Stability: Observe whether the subject can keep their arms hanging naturally and detect if there is any arm swing. If there is swing, mark it as "unstable"; otherwise, mark it as "stable". 3. Balance Time: Record the duration (in seconds) the subject can maintain single-leg standing with eyes closed. If it is less than the preset target, mark it as "unsatisfactory"; otherwise, mark it as "satisfactory". 4. Dynamic Changes: Analyze the dynamic changes in the body's center of gravity during this single-leg standing process. If significant swaying or tilting exceeding the threshold is detected, mark it as "unstable"; otherwise, mark it as "stable". 5. Body Sway Amplitude: Assess the sway amplitude (in centimeters) of the head, shoulders, and hips during single-leg standing. If it exceeds the threshold, record it as "excessive"; otherwise, record it as "normal". JSON Output Example: { "Phase": "Single-Leg Standing Test", "Preparation Time_s": 1.2, "Slowness of Movement": "Normal", "Upper Limb Stability": "Stable", "Balance Time with Eyes Closed_s": 8.5, "Balance Achieved": "Achieved", "Dynamic Changes": "Stable", "Head Sway_cm": 0.5, "Shoulder Sway_cm": 0.8, "Hip Sway_cm": 1.0, "Sway Amplitude Evaluation": "Normal"} Natural Language Description Example: In the single-leg standing test, the subject transitioned from two legs to one leg in approximately 1.2 seconds with no significant slowness of movement. The arms hung naturally and steadily without any arm swing. The subject maintained a single-leg standing position for approximately 8.5 seconds with eyes closed, achieving the expected goal. The body's center of gravity remained relatively stable throughout the process, with the head, shoulders, and hips swaying by approximately 0.5, 0.8, and 1.0 centimeters respectively, all within the normal range, demonstrating good overall balance control. Walking test Please analyze a complete gait cycle from the "Walking Test" according to the following steps, returning the results in JSON format first, followed by a natural language description of no less than 50 words: 1. Gait Characteristics: Observe and determine if the subject exhibits dragging, limping, or difficulty lifting the heel. If abnormal, mark the specific type; otherwise, mark as "normal." 2. Trunk Stability: Assess the subject's trunk sway amplitude (cm) during walking. If it exceeds the threshold, mark as "unstable"; otherwise, mark as "stable." 3. Upper Limb Swing: Check the coordination of the subject's left and right arm swings while walking. If the swing amplitude or symmetry is poor, mark as "uncoordinated"; otherwise, mark as "coordinated." 4. Cadence and Stride Length: Calculate the average cadence (steps / minute) and average stride length (meters) for this gait cycle for quantitative analysis. Example JSON output: { "Phase": "Walking Trial_Cycle 1", "Gait Characteristics": "Normal", "Trunk Stability_cm": 1.2, "Stability Evaluation": "Stable", "Upper Limb Swing": "Coordination", "Average Cadence_spm": 105, "Average Stride_m": 0.60} Example Natural Language Description: In this walking cycle, the subject's gait was normal, with no dragging or limping, and smooth heel lift. The trunk sway was approximately 1.2 cm, which can be considered stable. Upper and lower limb coordination was good, and arm swing was coordinated. The average cadence was approximately 105 steps / minute, and the average stride length was approximately 0.60 meters. The overall gait was balanced and rhythmic.

[0114] Through the above process, each action can obtain a targeted cue vector. This, along with the mean visual features of this segment, is input into a multimodal large model to generate the natural language description for the next stage.

[0115] Step S5: For each action segment, the mean of the global visual feature vector of the action segment is concatenated with the corresponding cue embedding and then input into the Large Language Model (LLM) of the visual-text model. This generates a natural language description containing quantitative indicators and action features in one go, thus obtaining the description of each segment. This provides a complete and interpretable semantic context for subsequent few-shot reasoning.

[0116] This step aims to deeply integrate the visual features of each action with the semantic cues for that action phase, and automatically generate fluent, accurate and interpretable natural language descriptions through a large model, providing a complete semantic context for subsequent few-sample Chain-of-Thought inference.

[0117] At the implementation level, the system first targets the first... The set of global category tokens representing all frames within a segment. Calculate their mean to obtain the global visual representation of this stage:

[0118]

[0119] in, This averaging operation not only smooths out sporadic noise within segments and local inter-frame jitter, but also condenses the core spatial and motion information during the action, giving the subsequently generated description a more stable and representative feature base.

[0120] After completing visual aggregation, the system will convert the visual vectors... Embedded with hints retrieved and encoded from the mapping table By directly concatenating the features along the feature dimension, a joint input for a multimodal large model (i.e., the aforementioned visual-text large language model) is constructed:

[0121]

[0122] This approach ensures that the model can simultaneously "see" quantitative visual data and "hear" analytical instructions defined by experts when generating text, thereby enabling differentiated attention to different stages of the action.

[0123] Finally, the vector The loaded multimodal large model is fed in, and the model will output a natural language description for this stage based on the composite vector in one go:

[0124]

[0125] Video-LLM stands for Visual-Text Large Model, which is also known as Multimodal Large Model.

[0126] For example, for the second phase of the "walking experiment," the model might generate the following description:

[0127] {

[0128] Phase: "Walking Trial - Cycle 2"

[0129] Gait characteristics: "Slight limping"

[0130] "Trunk stability_cm": 2.3,

[0131] Stability Assessment: Slightly Unstable

[0132] "Upper limb swing": "Incoordination"

[0133] Average cadence (spm): 90,

[0134] Average stride length (m): 0.52

[0135] }

[0136] During this walking cycle, the subject exhibited slight limping: the right leg had a slightly shorter stride and difficulty lifting the heel. The torso swayed approximately 2.3 cm while walking, indicating slight instability. The arm swings were not symmetrical, indicating incoordination. The average cadence was approximately 90 steps per minute, and the average stride length was approximately 0.52 meters, indicating a relatively slow overall pace and slightly poor balance.

[0137] The above generated It includes key quantitative indicators and describes action characteristics in a coherent textual way, providing intuitive and readable semantic input for subsequent few-shot inference in multimodal large models, ensuring that the diagnostic process is more interpretable and reliable.

[0138] Step S6: Connect a small number of labeled examples with the descriptions in each paragraph to form a thought chain prompt, input it into the visual-text large model, complete multi-step logical reasoning under the guidance of the examples, and directly output the sarcopenia diagnosis and its detailed reasons.

[0139] This step leverages the in-context learning capabilities of large language models to guide the model through multiple logical reasoning steps with very few labeled samples, thereby arriving at a diagnosis of sarcopenia and its detailed rationale.

[0140] First, prepare before system deployment. High-quality small sample examples Each example is described by a piece of natural language. and corresponding diagnostic labels Composition. And the following quantitative judgment criteria are explicitly cited in the example:

[0141] 1. 30-second sit-to-stand test: <12 repetitions for men and <11 repetitions for women indicate decreased muscle strength;

[0142] A walking speed of <1 m / s over a 6-meter distance suggests decreased muscle strength.

[0143] 3. Standing on one leg for less than 5 seconds indicates a decline in balance.

[0144] These examples not only provide textual descriptions of two typical cases, "normal" and "sarcopenia," but also include a complete chain-of-thought reasoning process as a template for subsequent in-context prompts.

[0145] In actual reasoning, the system will use this The examples are concatenated in sequence, with descriptions generated at each stage of the current video appended to the end. The description should include quantitative results for each sub-segment (such as time and number of times for sitting up; walking speed; duration of standing on one leg) and qualitative observations (such as whether the movement is stable, whether the upper limbs are used, etc.). Finally, add the instruction: "Based on the above quantitative criteria (number of times for sitting up in 30 seconds, walking speed of 6 meters, duration of standing on one leg) and qualitative characteristics, provide a diagnosis of sarcopenia and detailed reasoning." The complete Chain-of-Thought Prompt constructed in this way is as follows:

[0146]

[0147] in," The quotation mark (") indicates string-level concatenation. This prompt contains both the complete inference chain in the example and a description and unified instructions for the paragraph to be tested, which can maximize the inference capabilities of large models.

[0148] Then The data is directly input into the already loaded multimodal large model. The model first "reviews" the quantitative threshold application and multi-step inference logic in the example, and then performs step-by-step analysis of each segment based on the example pattern: First, it checks whether the number of times you sit / walk speed / stand time on one leg is lower than the threshold. Then, combined with qualitative features such as "upper limb assistance", "center of gravity sway", and "rhythm stability", it gradually outputs the Chain-of-Thought logic and finally gives a diagnostic conclusion. ("Normal" or "Sarcopenia") and the corresponding confidence level. This process requires no additional fine-tuning and enables efficient and interpretable diagnostic inference even with scarce annotations.

[0149] As an optional embodiment, the method further includes:

[0150] Step S7 (Classification and Confidence Calculation): The original scores of the model for the "normal" and "sarcopenia" categories are first processed into a probability distribution through exponential normalization. Then, the category with the higher probability is selected as the final diagnosis, and the probability value is directly used as the confidence level of the diagnosis to provide a quantitative level of confidence for the report.

[0151] This step probabilistically processes the raw scores (logits) output at the end of the Chain-of-Thought inference of the multimodal large model, quantifies the model's confidence in the "normal" and "sarcopenia" categories, and uses the category with the highest probability as the final diagnosis result, thereby providing an interpretable confidence index while providing a clear judgment.

[0152] After completing the Chain-of-Thought inference, the model outputs a binary logits vector:

[0153] ;

[0154] in, This represents the model's unnormalized score for the "normal" category. This represents the model's unnormalized score for the "sarcopenia" category.

[0155] To convert these scores into statistically significant probabilities, the Softmax function is needed:

[0156]

[0157] The above formula ensures that the calculated probabilities are non-negative and sum to 1, resulting in a clear and interpretable confidence distribution in the model output. Specifically, This is the probability that the model considers the patient to be "normal". This represents the probability that the model considers the patient to have "sarcopenia".

[0158] After obtaining the category probabilities, the system will select the category with the highest probability as the final diagnosis result:

[0159]

[0160] in, Corresponding to "normal", This corresponds to "sarcopenia". This optimal choice maximizes the model's decision confidence under in-context inference.

[0161] To more intuitively present the model's confidence level, the highest probability value can also be used. This serves as the confidence level output for the diagnosis. The closer the index is to 1, the more certain the model is about the result; if it is close to 0.5, it indicates that the model is hesitant between the two classes and should be interpreted with caution or confirmed in conjunction with other clinical examination results.

[0162] Through the above classification and confidence calculation steps, the system can not only provide a clear binary diagnosis, but also attach a quantifiable confidence assessment to each diagnosis result, thereby enhancing the interpretability and clinical reference value of the diagnostic report.

[0163] As another optional embodiment, the method further includes:

[0164] Step S8: Integrate the descriptions in Step S5, the reasoning steps and diagnostic results in Step S6, and the confidence level in Step S7 into a structured report, and finally output a complete diagnostic report containing quantitative and qualitative analysis with one click.

[0165] In this step, the system first processes the natural language descriptions generated in step S5. The Chain-of-Thought reasoning steps output in step S6, and the diagnostic results obtained in step S7. With corresponding confidence level The data is integrated and automatically assembled into a structured diagnostic report. The diagnostic results are presented in...

[0166]

[0167] Indicate; confidence level is defined as

[0168] ;

[0169] To measure the model's confidence in the diagnosis. Simultaneously, it can be based on... Add relevant medical advice, such as when In cases of sarcopenia, it is recommended to perform muscle strength testing, bone density testing, or develop a personalized rehabilitation training plan; when (Under normal circumstances) Regular physical fitness monitoring and preventive health guidance can be recommended.

[0170] The report generation supports two output formats: one is automatically formatted into a PDF or HTML document, presented according to the structure of "Patient Basic Information → Segmented Analysis → Chain-of-Thought Reasoning → Diagnostic Results and Medical Recommendations"; the other is stored as standardized JSON for inter-system calls and secondary processing, with the format defined as follows:

[0171]

[0172] Through the above steps, the system can output a complete diagnostic report with one click. It includes quantitative and qualitative analysis, fully meeting the dual requirements of clinical needs and patent applications for readability and interpretability.

[0173] Actual operation cases

[0174] In actual operation, each participant undergoes a 30-second "sit-to-stand test," "single-leg standing test," and "walking test." Video footage is taken from both frontal and side angles. For example, the video clip from the side-view "sit-to-stand test" is shown below. Figure 2 As shown in each frame.

[0175] Step S1: Video Acquisition and Patch Token Preprocessing

[0176] The system captured a 30-second "sit-stand test" video at a fixed side angle, encoded in H.264, at 30fps, and with a resolution of 1920×1080. After recording, the video was decoded and 900 frames were evenly extracted. Each frame underwent the following operations: first, it was cropped to retain only the human body area, then scaled to 224×224 pixels, as shown in the example below. Figure 3 As shown;

[0177] Pixel values ​​are linearly mapped from 0-255 to [0,1] and standardized according to the channel mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225]. Each frame is divided into 16×16 blocks (14×14=196 patches). Each patch is flattened and projected onto a 768-dimensional embedding using a learnable linear mapping. A global classification token is inserted before each frame. This results in a 900×197 patch token sequence for subsequent visual feature encoding.

[0178] Step S2: Visual Feature Encoding

[0179] The system concatenates the patch token sequence obtained in step S1 in chronological order to form the initial sequence. On each floor During the encoding process, the upper layer output is first processed. Perform layer normalization to eliminate dimensionality differences, then calculate the correlation weights between all tokens using a multi-head self-attention mechanism and add them back to the residuals to generate an intermediate vector. Then on Perform layer normalization again, feed it into a feedforward network consisting of two fully connected layers and nonlinear activation layers, and finally add back the residuals to obtain the result. After repeating the above process for a total of 12 layers, the classification token features of each frame are extracted. These features contain both spatial information of the frame and motion context from previous and subsequent frames, laying the foundation for subsequent implicit segmentation.

[0180] Step S3: Implicit Action Segmentation

[0181] The system first analyzes the global classification token features for each frame. Calculate the Euclidean distance between adjacent frames:

[0182]

[0183] In the video of this "sit-stand test," the subjects actually completed multiple cycles of "sit → stand" and "stand → sit" within 30 seconds. Therefore, the timing of "standing up" and "sitting down" was crucial. A distinct peak will appear; during the intervals between "sitting still" and "preparing to stand up," The value remains low. To suppress spurious peaks caused by inter-frame jitter, the system... A moving average with a width of 5 frames was applied to obtain the smoothed result. Subsequently, statistics were compiled for all... mean and standard deviation And set an adaptive threshold:

[0184]

[0185] when At that time, the system determines the first A frame represents the boundary of one action transition. After this discrimination step, the system detected 24 frame indices in 900 frames (corresponding to 12 "stand up" boundaries and 12 "sit down" boundaries respectively). To avoid excessively short segments caused by noise, the system further specifies a minimum segment length of approximately 60 frames (about 2 seconds), merging boundaries where adjacent peaks are less than 60 frames apart into a single segment. Ultimately, the entire video is divided into 24 sub-stages. Odd-numbered sub-stages Corresponding to one "sit → stand" cycle, even-numbered sub-stages are designated. This corresponds to one cycle of "standing → sitting". For example:

[0186] Includes frames 1-37 (first "sitting → standing"),

[0187] Includes frames 38-74 (first "standing → sitting"),

[0188] …

[0189] Includes frames 829-865 (the 12th "sitting → standing"),

[0190] Includes frames 866-900 (the 12th "standing → sitting" sequence).

[0191] At this point, the system has automatically inferred that the subject completed 12 complete cycles within 30 seconds, which was implicitly inferred from the video through feature difference peak identification and minimum segment length rules.

[0192] Step S4: Mapping Trial Types and Segmentation Hints

[0193] The system clearly identifies the current video's test type as a "sitting-standing test (side view)" and has prepared a set of prompt templates specifically for the "sitting-standing test" scenario. Regarding the first... Sub-stages The system retrieves the corresponding text prompts from this template set. For example, the prompt text retrieved for the first segment (frames 1-37, corresponding to the first "sit → stand" cycle) is the Prompt template / prompt input corresponding to "sit-stand test" in Table 1 above.

[0194] The system inputs this prompt into the pre-trained text encoder. Generate cue embedding vectors with dimensions consistent with visual features. .

[0195] And so on, the system is as follows: Generate prompt embedding This is so that subsequent descriptions can generate the necessary differentiated semantic inputs.

[0196] Step S5: Segmented Description Generation

[0197] For the Sub-stages The system first calculates the mean of the global features of all frames in the segment:

[0198]

[0199] in, For the first The number of frames in a segment. For example, if segment 1 contains 37 frames, the calculated vector... The system will then... Embed with corresponding prompts Concatenate along the feature dimensions to obtain the joint input vector:

[0200]

[0201] Will The data is fed into a large-scale visual-text model (Video-LLM), which outputs both the JSON result and a natural language description for that segment in one go. Taking segment 1 as an example, the generated JSON is:

[0202] {

[0203] Phase: "Sitting / Standing Test - First Cycle"

[0204] "Time taken to stand up_s": 2.1,

[0205] "Slowed movement": "Normal"

[0206] Postural stability: "Stable"

[0207] "Upper limb assistance": "No assistance"

[0208] }

[0209] The corresponding natural language description example is:

[0210] The patient took approximately 2.1 seconds to stand during the first sitting-to-standing cycle, without any slowness of movement; the standing process was smooth, without any forward, backward, or left-right tilting; the patient did not use their arms for support throughout the process, and the movements were natural and fluid.

[0211] The system sequentially generates 24 sets of JSON results (segments 2 through 24) and 24 text descriptions. Each sub-stage can obtain similar detailed quantitative indicators and coherent textual descriptions, ensuring the complete semantic context required for subsequent reasoning.

[0212] Step S6: Chain-of-Thought In-context Reasoning

[0213] To facilitate chain-of-thought reasoning, the preferred Chain-of-Thought reasoning template is as follows:

[0214] (1) Compare the number of cycles the subject completes within a preset time (e.g., 30 seconds) with a preset threshold;

[0215] (2) Compare the average time taken for the subjects to stand up / sit down with the preset threshold;

[0216] (3) Assess the subject's postural stability against the preset threshold;

[0217] (4) Assess the subject's upper limb assistive function;

[0218] The diagnostic conclusion is then output based on the results of the above multiple steps.

[0219] Three small-sample examples were prepared during system deployment, demonstrating in detail how to combine quantitative thresholds with qualitative observations for multi-step inference. For example:

[0220] 1. Example 1 (Normal)

[0221] "A 65-year-old male completed 14 cycles of the 30-second sit-to-stand test, with an average standing time of 1.8 seconds, a maximum forward tilt of 10°, and a steady and uniform performance without upper limb assistance."

[0222] Reasoning logic: Number of loops 14 > 12 (male threshold), time to stand up < 2.5 seconds, smooth and without assistance → judge as "normal".

[0223] 2. Example 2 (Sarcopenia)

[0224] "A 70-year-old male completed 8 cycles, with an average standing time of 3.2 seconds, a maximum forward tilt of 15°, and slight shaking."

[0225] Reasoning logic: Number of cycles 8 < 12, time taken to stand up > 2.5 seconds, tremors present → diagnosis of "sarcopenia".

[0226] 3. Example 3 (Normal)

[0227] "A 68-year-old woman completed 12 cycles (equal to the threshold), with an average standing time of 2.6 seconds, a maximum forward tilt of 12°, without assistance, and occasional slight instability."

[0228] Reasoning logic: Number of loops = 12 (boundary), the time to stand up is slightly slower but the overall process is stable, no assistance → judge "normal".

[0229] In actual reasoning, the system compares these three examples with the 24 natural language descriptions of the current subject. Concatenate sequentially, and append instructions at the end:

[0230] Prompt_CoT = (Example 1) "Normal", (Example 2) "Sarcopenia", (Example 3) "Normal", D_1 "D_2" ... "D_{24}" "Please combine the number of sit-stand cycles in 30 seconds, the speed of standing up, the angle of forward leaning and the stability to give a diagnosis of sarcopenia and a detailed reasoning."

[0231] The prompt is then input into Text-LLM (it should be noted that this article only involves one model, namely the visual-text large model; the Video-LLM mentioned earlier mainly emphasizes the use of this model's visual encoding capabilities, while the prompt inference here only involves the text portion of the large model, i.e., Text-LLM). The model first "reviews" the multi-step inference process and quantitative threshold application in the example, and then performs step-by-step analogy and analysis on the 24 descriptions in this case:

[0232] 1. The number of cycles completed in this case was 12 (male threshold) → threshold boundary, indicating that muscle strength is acceptable;

[0233] 2. The standing time for each cycle was mostly between 2.0 and 2.2 seconds, less than 2.5 seconds, which is within the normal range.

[0234] 3. Check if there is any forward or backward or left or right tilting in each cycle → The vast majority are marked as "stable", and the occasional slight instability is still within the acceptable range;

[0235] 4. Verify the presence of upper limb assistance → Mark all cycles as "no assistance";

[0236] 5. Based on the boundary conditions in Example 3, the model ultimately judges it as "normal" and provides the corresponding inference chain, with explanations such as "the number of cycles = 12, the standing speed and stability indicators both meet the normal standards, and no auxiliary was used".

[0237] The final model outputs the inference chain and conclusion: "The subject's cycle count and quantitative indicators both meet the health standards, therefore the subject is judged to be normal," with a confidence level of ≈ 0.94.

[0238] Step S7: Classification and Confidence Calculation

[0239] Text-LLM outputs the raw scores of "Normal" (label 0) and "Sarcopenia" (label 1) at the end of the Chain-of-Thought inference. The system converts it into a probability using Softmax:

[0240] ;

[0241] because The largest, therefore the final diagnosis (Normal), confidence level 0.94.

[0242] Step S8: Reporting and Visualization

[0243] The system integrates 24 JSON outputs with natural language descriptions, a chain-of-thought reasoning process (example review, 24 quantitative and qualitative analyses, comprehensive judgment), and a final diagnosis of "normal" with a confidence level of 0.94, automatically generating a structured diagnostic report. Examples of key report information are shown below. Figure 4 As shown. The report can be exported as PDF / HTML, or saved as JSON for later use.

[0244] In this practical case study, the large model was specifically optimized and adjusted for sarcopenia:

[0245] 1. Domain-adaptive segmentation threshold

[0246] In implicit motion segmentation, this case does not simply use the general "mean + 1.2σ" threshold. Instead, it statistically analyzed the optimal sensitivity parameters for four specific movements—"sitting → standing," "standing → sitting," "standing on one leg," and "walking"—based on a large number of sarcopenic and healthy control videos. For example, the peak threshold α for standing up was set to 1.0 in sarcopenic individuals and 1.4 in healthy elderly individuals; the lower limit for the single-leg standing segment length was set to 70 frames to avoid misjudging brief balance jitters. These specific data calibrations significantly improved the accuracy of motion segmentation for sarcopenic individuals.

[0247] 2. Dedicated Clinical Indicator Extraction Module

[0248] This project developed a customized logic for extracting core quantitative indicators of sarcopenia, including "number of times sitting and standing," "time to stand up / sit down," "maximum forward / backward lean angle," "walking speed," and "duration of standing on one leg," and encapsulated it as a preprocessing layer before a multimodal large model. This layer automatically calculates and outputs the above five key indicators after visual encoding, rather than relying on general optical flow or keypoint algorithms, thereby achieving accurate capture of sarcopenia characteristics.

[0249] 3. Customized Chain-of-Thought prompt templates

[0250] To address the clinical diagnostic logic of sarcopenia, this case study designed several detailed CoT prompts—such as a four-step chain reasoning process: "the ratio of the number of sit-to-stand cycles to the 12-cycle threshold; the ratio of the time taken to stand to the 2.5-second threshold; assessment of postural stability; and assessment of upper limb assistance"; a three-step reasoning process: "the ratio of the duration of standing on one leg to the 5-second threshold; and assessment of the amplitude of center of gravity shift"; and a three-step reasoning process: "the ratio of walking speed to the 1 m / s threshold; and assessment of trunk / upper limb stability." These templates were directly incorporated into the Prompt, rather than using generic video descriptions, to guide LLMs in conducting specialized analyses in the field of sarcopenia.

[0251] 4. Cross-experiment multimodal fusion strategy

[0252] This case does not rely solely on a single sitting / standing video, but rather combines features from three independent tests (six video feeds from the front and side views) for a comprehensive diagnosis. During inference, the model first generates intra-segment descriptions for each of the three tests, then concatenates the outputs of the three tests into the final CoT prompt, providing LLM with more comprehensive muscle strength / balance / gait information, thus achieving cross-dimensional sarcopenia diagnosis.

[0253] Therefore, by organically integrating implicit action segmentation, multi-stage differentiated cues, and few-sample chain reasoning, this invention significantly improves the practicality and interpretability of video-based sarcopenia detection technology, specifically in the following aspects:

[0254] 1. High robustness across scenarios and devices

[0255] By automatically segmenting based on the feature differences between adjacent frames, without the need for any external skeletal models or manual feature extraction, the algorithm can accurately identify key action stages such as "sitting to standing" and "walking" under different shooting angles, lighting conditions, and resolutions, greatly improving the versatility and stability of the algorithm in various scenarios such as clinical settings, home settings, and remote monitoring.

[0256] 2. High efficiency and low annotation cost

[0257] By utilizing in-context chain-of-thought hints with few samples, the model can quickly adapt to new environments and new populations without large-scale fine-tuning. The amount of annotations is reduced from thousands of frames in traditional methods to tens of frames, effectively reducing the cost of expert annotation and the system iteration cycle.

[0258] 3. Abundant quantitative indicators and transparent reasoning process

[0259] Differentiated prompt templates are designed for different stages of movement, and structured indicators (such as cadence, stride length, lift speed, stability assessment, etc.) and complete multi-step reasoning chains are automatically output. This not only meets the clinical requirements for the precision of key movement parameters, but also provides auditable logical explanations for the basis of judgment, thereby enhancing the interpretability and traceability of the results.

[0260] 4. Integrated and visualized diagnostic reports

[0261] It generates reports with one click, including segmented descriptions, quantitative analysis, and chain-of-thought reasoning. Reports can be exported as PDF / HTML or JSON for inter-system use, meeting the diverse needs of clinicians, rehabilitation therapists, and patients, and improving the efficiency of technology implementation.

[0262] 5. Simple end-to-end system deployment

[0263] The method of this invention does not require the introduction of complex external algorithm libraries or hardware devices. The end-to-end design based on the visual-text large model greatly simplifies the system architecture, makes it easy to integrate into existing medical imaging or monitoring platforms, and supports cloud or edge deployment, thereby reducing operation and maintenance costs.

[0264] In summary, this invention relates to the application of computer vision in the medical field. Specifically, it employs a multimodal large-scale model to analyze visual modal information and combines this with the physician's professional knowledge to diagnose sarcopenia. This invention not only breaks through the reliance of traditional methods on skeletal key points and large-scale annotation, but also brings significant comprehensive benefits in terms of clinical applicability, interpretability, and ease of deployment.

[0265] In addition, the present invention includes the following key points:

[0266] 1. Implicit Action Segmentation Mechanism

[0267] Adaptive difference detection based on video frame-level global features automatically divides action stages such as "sitting to standing", "walking", and "stopping", without the need for manual annotation or external skeleton models.

[0268] 2. Large model segment type identification

[0269] Each brief feature summary is input into a large language model to perform efficient and accurate stage classification within a predefined set of five action labels.

[0270] 3. Differentiated prompt mapping

[0271] Fine-grained, multi-step cue templates are designed for each action stage, and mapped to cue embeddings of the same dimension as visual features through a text encoder, thereby achieving joint input of "visual + semantic".

[0272] 4. Multimodal description generation

[0273] The mean visual features within a segment are concatenated with the embedded prompts and fed into a large visual-text model to generate a natural language description containing quantitative indicators and feature analysis in one go.

[0274] 5. Few-sample chain reasoning

[0275] By constructing a Chain-of-Thought in-context Prompt using a small number of labeled examples, a large language model can be triggered to output diagnostic conclusions and reasons through multi-step logical reasoning, significantly reducing the need for annotation.

[0276] 6. Reporting and Visualization

[0277] A structured report can be generated by integrating segmented descriptions, reasoning processes, diagnostic results, and confidence levels with a single click.

[0278] On the other hand, the present invention provides a sarcopenia detection device based on chained prompts and a multimodal large model, such as Figure 5 As shown, it includes:

[0279] The video acquisition and preprocessing module 11 is used to acquire patient action videos under fixed angle and shooting parameters, decode the video into frames, crop each frame image, unify the size and normalize the pixels, and divide each frame image into several small blocks and map them into feature vector sequences.

[0280] The visual feature encoding module 12 is used to input the feature vector sequence into a multilayer alternating encoding network, perform information fusion between the same frame and different times, and output a global visual feature vector with both spatial and temporal context for subsequent action segmentation.

[0281] The implicit action segmentation module 13 is used to automatically detect action transition points based on the difference in global visual feature vectors between adjacent frames, divide the continuous feature sequence into multiple semantically coherent action stages, obtain the index of each sub-stage, and provide accurate temporal segmentation for subsequent type discrimination and description generation.

[0282] The test type and segmented prompt mapping module 14 is used to determine the global prompt template set according to the test category to which the current video belongs, and then index each sub-stage, retrieve the corresponding segmented prompt text from the global prompt template set based on the test type, and map the segmented prompt text into a prompt embedding with the same dimension as the global visual feature vector through a text encoder, so as to provide differentiated semantic input for subsequent description generation;

[0283] The segmented description generation module 15 is used to concatenate the mean of the global visual feature vector of each action segment with the corresponding prompt embedding and input it into the visual-text large model to generate a natural language description containing quantitative indicators and action features at one time, thus obtaining the description of each segment, providing a complete and interpretable semantic context for subsequent few-shot reasoning.

[0284] The few-sample reasoning module 16 is used to link a small number of labeled examples with the descriptions in each paragraph to form a thought chain prompt, input the visual-text large model, complete multi-step logical reasoning under the guidance of examples, and directly output the sarcopenia diagnosis and its detailed reasons.

[0285] The apparatus of this embodiment can be used to perform Figure 1 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0286] Preferably, the device further includes:

[0287] The classification and confidence calculation module is used to first process the original scores of the model for the "normal" and "sarcopenia" categories into a probability distribution through exponential normalization, then select the category with the higher probability as the final diagnosis, and use the probability value directly as the confidence level of the diagnosis to provide a quantitative level of confidence for the report.

[0288] Preferably, the device further includes:

[0289] The reporting and visualization module is used to integrate the segment descriptions in the segmented description generation module 15, the reasoning steps and diagnostic results in the few-sample reasoning module 16, and the confidence in the classification and confidence calculation module into a structured report, and finally outputs a complete diagnostic report containing quantitative and qualitative analysis with one click.

[0290] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting sarcopenia based on chained cueing and a multimodal large model, characterized in that, include: Step S1: Acquire patient action video under fixed angle and shooting parameters, decode the video into frames, crop each frame image, unify the size and normalize the pixels, and divide each frame image into several small blocks and map them into feature vector sequences. Step S2: Input the feature vector sequence into a multilayer alternating coding network, perform information fusion between the same frame and different time points, and output a global visual feature vector with both spatial and temporal context for subsequent action segmentation; Step S3: Automatically detect action transition points based on the differences in global visual feature vectors of adjacent frames, divide the continuous feature sequence into multiple semantically coherent action stages, obtain the index of each sub-stage, and provide accurate temporal segmentation for subsequent type discrimination and description generation; Step S4: Determine the global cue template set according to the test category to which the current video belongs, and then index each sub-stage, retrieve the corresponding segmented cue text from the global cue template set based on the test type, and map the segmented cue text into a cue embedding with the same dimension as the global visual feature vector through a text encoder, so as to provide differentiated semantic input for subsequent description generation; Step S5: For each action segment, the mean of the global visual feature vector of the action segment is concatenated with the corresponding prompt embedding and then input into the visual-text large model to generate a natural language description containing quantitative indicators and action features in one go, thus obtaining the description of each segment, providing a complete and interpretable semantic context for subsequent few-shot reasoning. Step S6: Connect a small number of labeled examples with the descriptions in each paragraph to form a thought chain prompt, input it into the visual-text large model, complete multi-step logical reasoning under the guidance of the examples, and directly output the sarcopenia diagnosis and its detailed reasons; Step S3 includes: Step S31: Calculate the Euclidean distance between the global visual feature vectors of two adjacent frames. ; Step S32: For Applying moving average smoothing, we obtain ; Step S33: Statistically analyze the difference distribution of the entire video and calculate the mean. with standard deviation and with adaptive threshold As a criterion for action switching, continuous frame features are divided into several action stages, among which... These are preset coefficients.

2. The method for sarcopenia detection based on chained prompts and a multimodal large model according to claim 1, characterized in that, In step S1, the motion video includes motion videos corresponding to the sit-to-stand test, the single-leg standing test, or the walking test. And / or, step S1 includes: After pixel normalization, the image is rearranged from BGR channels to RGB channels. Then, based on the channel mean and standard deviation used by the ImageNet pre-trained model, the pixel values ​​of each channel are standardized to adjust the data to a zero-mean, unit-variance distribution. And / or, step S1 includes: Each small patch is flattened and projected onto a learnable linear layer. The dimensional embedding space is used to obtain the patch embedding of each patch; A trainable classification token is inserted before all patch embeddings in each frame to obtain the sequence of patch tokens corresponding to the action video, which is the feature vector sequence.

3. The method for sarcopenia detection based on chained prompts and a multimodal large model according to claim 2, characterized in that, Step S2 includes: Step S21: Concatenate the patch embeddings of all frames with the classification tokens in chronological order to form the initial sequence; Step S22: In the first During the layer encoding process, layer normalization is first performed on the upper layer output to eliminate the magnitude difference between different dimensions. Then, the association weights between each category token are calculated through multi-head self-attention, and the original input information is retained to obtain the intermediate vector. Step S23: Perform layer normalization on the intermediate vector again to adapt to the input distribution of the subsequent feedforward network. Then, use a position-aware feedforward network composed of two fully connected layers and nonlinear activation layers, and perform residual fusion with the input to generate the output of this layer. Step S24: Repeat steps S22-S23 to complete the calculation of all coding layers and obtain the final output. Extract the classification token corresponding to each frame as the global visual feature vector.

4. The method for sarcopenia detection based on chained prompts and a multimodal large model according to claim 1, characterized in that, In step S33, all satisfying or The frame index is considered as the stage boundary, and together with the first and last frames, it forms a complete set of segment boundary indices.

5. The method for sarcopenia detection based on chained prompts and a multimodal large model according to claim 4, characterized in that, In step S33, if the number of frames between two adjacent stage boundaries is less than a preset lower limit, the corresponding two segments are merged.

6. The method for sarcopenia detection based on chained prompts and a multimodal large model according to claim 2, characterized in that, In step S4 For the sit-to-stand test, the corresponding segmented prompt text in the global prompt template set includes relevant descriptions of "speed from sitting to standing", "postural stability" and "upper limb assistance status"; For the single-leg standing test, the corresponding segmented prompt text in the global prompt template set includes descriptions of "preparatory movement speed", "upper limb stability", "balance time", "dynamic changes in center of gravity" and "body sway amplitude". For the walking test, the corresponding segmented prompt text in the global prompt template set includes descriptions of "gait characteristics", "trunk stability", "upper limb swing" and "step frequency and stride length".

7. The method for sarcopenia detection based on chained prompts and a multimodal large model according to any one of claims 1-6, characterized in that, The method further includes: Step S7: First, the original scores of the model for the "normal" and "sarcopenia" categories are processed into a probability distribution through exponential normalization. Then, the category with the higher probability is selected as the final diagnosis, and the probability value is directly used as the confidence level of the diagnosis to provide a quantitative level of confidence for the report.

8. The method for sarcopenia detection based on chained prompts and a multimodal large model according to claim 7, characterized in that, The method further includes: Step S8: Integrate the descriptions in Step S5, the reasoning steps and diagnostic results in Step S6, and the confidence level in Step S7 into a structured report, and finally output a complete diagnostic report containing quantitative and qualitative analysis with one click.

9. A sarcopenia detection device based on chained prompts and a multimodal large model, characterized in that, include: The video acquisition and preprocessing module is used to acquire patient action videos under fixed angle and shooting parameters. After decoding the video into frames, each frame image is cropped, uniformly sized and pixel normalized, and each frame image is divided into several small blocks and mapped into a feature vector sequence. The visual feature encoding module is used to input the feature vector sequence into a multilayer alternating encoding network, perform information fusion between the same frame and different times, and output a global visual feature vector with both spatial and temporal context for subsequent action segmentation. The implicit action segmentation module is used to automatically detect action transition points based on the differences in global visual feature vectors between adjacent frames. It divides continuous feature sequences into multiple semantically coherent action stages and obtains the index of each sub-stage, providing accurate temporal segmentation for subsequent type discrimination and description generation. The test type and segmented prompt mapping module is used to determine the global prompt template set according to the test category to which the current video belongs, and then index each sub-stage, retrieve the corresponding segmented prompt text from the global prompt template set based on the test type, and map the segmented prompt text into a prompt embedding with the same dimension as the global visual feature vector through a text encoder, so as to provide differentiated semantic input for subsequent description generation; The segmented description generation module is used to concatenate the mean of the global visual feature vector of each action segment with the corresponding prompt embedding and input it into the visual-text large model. This generates a natural language description containing quantitative indicators and action features in one go, thus obtaining the description of each segment. This provides a complete and interpretable semantic context for subsequent few-shot reasoning. The few-sample reasoning module is used to connect a small number of labeled examples with the descriptions in each paragraph to form a thought chain prompt, which is input into the visual-text large model. Under the guidance of the examples, it completes multi-step logical reasoning and directly outputs the sarcopenia diagnosis and its detailed reasons. The implicit action segmentation module is further used to calculate the Euclidean distance between the global visual feature vectors of two adjacent frames. ;right Applying moving average smoothing, we obtain Analyze the distribution of differences across the entire video and calculate the mean. with standard deviation and with adaptive threshold As a criterion for action switching, continuous frame features are divided into several action stages, among which... These are preset coefficients.

Images