Rehabilitation report generation method, electronic equipment and storage medium
By using deep alignment training of a multimodal fusion model, the problem of poor fusion results of multimodal medical data was solved, generating more accurate rehabilitation reports that meet professional and personalized needs.
Patent Information
- Application Number
- CN202511432948.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-30
- Publication Date
- 2026-01-16
AI Technical Summary
Existing rehabilitation report generation technologies cannot effectively integrate multimodal medical data, resulting in a low degree of matching between rehabilitation reports and patients' actual physiological states. They also fail to meet the needs of accurately covering professional terminology, adapting to individual patient differences, and integrating multimodal medical data.
A multimodal fusion model is adopted, which obtains multimodal feature vectors through feature encoding, and uses a medical knowledge-enhanced pre-trained model to add a feature projection layer, a cross-modal cross-attention layer and a gating fusion layer for deep alignment training to generate a rehabilitation report.
It significantly improves the accuracy of rehabilitation reports, enabling them to more accurately reflect the patient's actual physiological state and rehabilitation progress, and integrates multi-dimensional patient information to avoid the limitations of single-modal data.
Smart Images
Figure CN121354784A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically providing a method for generating rehabilitation reports, an electronic device, and a storage medium. Background Technology
[0002] In clinical settings such as stroke rehabilitation management, rehabilitation reports serve as core medical documents recording patient condition assessments, functional recovery progress, and personalized rehabilitation recommendations. The quality of these reports directly impacts the accuracy of clinical decision-making and the effectiveness of rehabilitation interventions. An ideal rehabilitation report must simultaneously meet three requirements: first, it must accurately cover stroke-related terminology, matching the professional standards of clinical documentation; second, it must fully adapt to individual patient differences, such as comorbidities and the degree of cognitive impairment; and third, it must integrate multimodal medical data to ensure that rehabilitation recommendations are highly consistent with the patient's actual physiological state. However, current mainstream technological solutions have failed to effectively address these needs, exhibiting significant limitations.
[0003] Existing rehabilitation report generation technologies mainly fall into two categories. One category consists of template-based rule engines or traditional machine learning models. For example, efficiency can be improved by filling content with predefined templates. However, the fixed logic of the templates cannot dynamically adapt to individual patient differences, and the coverage of professional terminology is insufficient, resulting in an accuracy rate of only 68.2%. The other category consists of improved solutions based on single-modal deep learning. For example, fine-tuning the BERT-based model can improve the terminology accuracy rate to 79.5%. However, this type of solution can only process text data and cannot integrate key non-textual modal information such as medical images, leading to a low degree of matching between rehabilitation recommendations and the patient's actual physiological state.
[0004] The core issue hindering the quality of rehabilitation reports lies in the inadequacy of multimodal medical data fusion technology. Current multimodal fusion methods often employ simple feature-level stitching or preliminary decision-level integration. For example, using a two-stream neural network architecture to directly stitch together medical images and electronic medical record features results in a cross-modal feature alignment error of 23.7%. Although recent research has introduced cross-modal attention mechanisms, the lack of a unified temporal alignment module leads to an 18.4% increase in prediction errors for time-sensitive indicators such as NIHSS score evolution when processing heterogeneous time-series data (such as voice follow-ups and static medical records), making it difficult to support dynamic adjustments to personalized rehabilitation plans. Therefore, a rehabilitation report generation technology that can achieve deep alignment and fusion of multimodal data while balancing professionalism and personalization is urgently needed. Summary of the Invention
[0005] To overcome the aforementioned shortcomings, this application is proposed to provide a solution, or at least a partial solution, to the technical problem that existing rehabilitation report generation methods suffer from poor multimodal medical data fusion, resulting in low matching between rehabilitation reports and patients' actual physiological states. This application provides a rehabilitation report generation method, an electronic device, and a storage medium.
[0006] In a first aspect, this application provides a method for generating a rehabilitation report, the method comprising:
[0007] Collect multimodal data related to patients;
[0008] The multimodal data is feature-encoded to obtain a multimodal feature vector;
[0009] The multimodal fusion model is trained by deep alignment using the multimodal feature vectors to obtain a pre-trained multimodal fusion model. The multimodal fusion model is based on the pre-trained model with medical knowledge enhancement and adds a feature projection layer, a cross-modal attention layer and a gated fusion layer.
[0010] A rehabilitation report generation model is obtained based on the pre-trained multimodal fusion model;
[0011] A rehabilitation report is generated based on the rehabilitation report generation model.
[0012] In one embodiment of the rehabilitation report generation method of this application, the multimodal data includes case text, medical images, and follow-up text;
[0013] The feature encoding of the multimodal data includes: inserting preset markers into the case text for text segmentation; performing word segmentation on the segmented case text to obtain a case text feature vector; and / or
[0014] The medical image is encoded using a visual encoder, and the output vector corresponding to the feature aggregation marker at the start position of the sequence is extracted as the medical image feature vector based on the encoding result; and / or
[0015] The follow-up text is subjected to structured information extraction according to a preset template to obtain structured data pairs; feature extraction is performed on the structured data pairs to obtain follow-up text feature vectors.
[0016] In one embodiment of the rehabilitation report generation method of this application, the feature projection layer includes a visual feature projection layer and a follow-up feature projection layer; the multimodal feature vector includes a case text feature vector, a medical image feature vector, and a follow-up text feature vector;
[0017] The multimodal fusion model includes:
[0018] The visual feature projection layer is used to map the medical image feature vectors to a multimodal shared text space to obtain medical image projection features.
[0019] The follow-up feature projection layer is used to map the follow-up text feature vector to a multimodal shared text space to obtain the follow-up text projection features.
[0020] The splicing layer is used to splice the case text feature vector with the medical image projection feature to generate the first fusion feature;
[0021] The pre-trained model is used to perform cross-modal fusion processing based on the follow-up text projection features and the first fusion feature to obtain multimodal depth-aligned semantic fusion features.
[0022] In one embodiment of the rehabilitation report generation method of this application, the pre-trained model includes:
[0023] The first cross-attention layer is used to obtain the second fusion feature by using the first fusion feature as the query vector and the follow-up text projection feature as the key vector and value vector, through information interaction via a group attention mechanism.
[0024] The second cross-attention layer is used to obtain the third fusion feature by using the follow-up text projection features as the query vector and the second fusion feature as the key vector and value vector, and by interacting with information through a group attention mechanism.
[0025] The intermediate layer is used to semantically integrate the third fusion feature to obtain high-level semantic features;
[0026] The gated fusion layer is used to perform high-level fusion based on the second fusion feature, the third fusion feature and the high-level semantic feature to obtain high-dimensional fusion features.
[0027] In one embodiment of the rehabilitation report generation method of this application, the gated fusion layer includes:
[0028] The splicing module is used to splice the second fusion feature, the third fusion feature, and the high-level semantic feature to obtain the spliced feature;
[0029] A linear module is used to perform dimensionality transformation on the concatenated features and generate a weight vector after passing them through an activation function;
[0030] The weighted fusion module is used to perform weighted fusion on the second fusion feature, the third fusion feature and the high-level semantic feature to obtain a weighted fusion feature;
[0031] The standardization layer is used to standardize the weighted fusion features to obtain the high-dimensional fusion features.
[0032] In one embodiment of the rehabilitation report generation method of this application, the step of using the multimodal feature vector to perform deep alignment training on the multimodal fusion model includes:
[0033] Activate the visual feature projection layer and the follow-up feature projection layer, and freeze all other layers except the visual feature projection layer and the follow-up feature projection layer;
[0034] The multimodal fusion model is trained for alignment using a loss function that is a weighted combination of contrast alignment loss and modality reconstruction loss.
[0035] Activate the first cross-attention layer, the second cross-attention layer, and the gated fusion layer, and freeze all other layers except the first cross-attention layer, the second cross-attention layer, and the gated fusion layer;
[0036] The multimodal fusion model is trained using a contrastive alignment loss function.
[0037] In one embodiment of the rehabilitation report generation method of this application, obtaining the rehabilitation report generation model based on the pre-trained multimodal fusion model includes:
[0038] Using the pre-trained multimodal fusion model as the base model, the model parameters of the multimodal fusion model are frozen;
[0039] A fine-tuning module is introduced and fine-tuning training is performed on the fine-tuning module;
[0040] The rehabilitation report generation model is obtained based on the pre-trained multimodal fusion model and the fine-tuned model.
[0041] In one embodiment of the rehabilitation report generation method of this application, the pre-trained model is obtained through the following steps:
[0042] Acquire medical knowledge text, which includes a first marker identifying key paragraphs and a second marker identifying technical terms;
[0043] The medical knowledge text is dynamically masked based on the first and second markers;
[0044] The medical knowledge text of the mask is used to predict terms using a pre-trained base model;
[0045] The difference between the predicted results and the true terms is calculated using the cross-entropy loss function;
[0046] The training parameters of the pre-trained base model are adjusted according to the difference until the difference is less than a first preset threshold or the number of training iterations reaches a second preset threshold, thereby obtaining the trained pre-trained model.
[0047] In a second aspect, an electronic device is provided, comprising:
[0048] At least one processor;
[0049] And, a memory communicatively connected to the at least one processor;
[0050] The memory stores a computer program, which, when executed by the at least one processor, is the aforementioned method for generating rehabilitation reports.
[0051] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the rehabilitation report generation method described in any of the preceding claims.
[0052] The above-described technical solutions of this application have at least one or more of the following beneficial effects:
[0053] The rehabilitation report generation method in this application includes: collecting multimodal data related to patients; performing feature encoding on the multimodal data to obtain multimodal feature vectors; using the multimodal feature vectors to perform deep alignment training on a multimodal fusion model to obtain a pre-trained multimodal fusion model. The multimodal fusion model is based on a medical knowledge-enhanced pre-trained model, with the addition of a feature projection layer, a cross-modal cross-attention layer, and a gated fusion layer. This effectively solves the problems of strong heterogeneity and poor alignment fusion effect of multimodal data in traditional technologies, and can fully explore the semantic associations between different modal data. A rehabilitation report generation model is obtained based on the pre-trained multimodal fusion model. A rehabilitation report is generated based on the rehabilitation report generation model, which can integrate multi-dimensional patient information, avoid the limitations of single-modal data, and thus the generated rehabilitation report can more accurately reflect the patient's actual physiological state and rehabilitation status, significantly improving the accuracy of the rehabilitation report. Attached Figure Description
[0054] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Furthermore, similar numbers in the drawings are used to denote similar components, wherein:
[0055] Figure 1 This is a schematic diagram of the main process of a rehabilitation report generation method in one embodiment of this application;
[0056] Figure 2 This is a schematic diagram of the structure of a VIT encoder in one embodiment of this application;
[0057] Figure 3 This is a schematic diagram of the structure of a multimodal fusion model in one embodiment of this application;
[0058] Figure 4 This is a schematic diagram of a stroke rehabilitation report in one embodiment of this application;
[0059] Figure 5 This is a schematic diagram of the main structure of the rehabilitation report generation device in one embodiment of this application;
[0060] Figure 6 This is a schematic diagram of the structure of an electronic device in one embodiment of this application. Detailed Implementation
[0061] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.
[0062] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and can also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.
[0063] Current traditional methods for generating rehabilitation reports are ineffective at integrating multimodal medical data, resulting in low accuracy in matching rehabilitation reports with patients' actual physiological states. Therefore, this application provides a method for generating rehabilitation reports, an electronic device, and a storage medium.
[0064] See appendix Figure 1 , Figure 1 This is a schematic flowchart of the main steps of a rehabilitation report generation method according to an embodiment of this application.
[0065] like Figure 1 As shown, the rehabilitation report generation method in this application embodiment mainly includes the following steps S10-S50.
[0066] Step S10: Collect multimodal data related to the patient.
[0067] Multimodal data refers to a collection of various types of medical data related to patients collected from different sources and in different forms, covering a variety of heterogeneous data types such as text data (e.g., electronic medical records, follow-up records), image data (e.g., CT and MRI images), and voice data (e.g., voice follow-up recordings or text).
[0068] Step S20: Perform feature encoding on the multimodal data to obtain multimodal feature vectors.
[0069] Feature encoding refers to the process of processing multimodal data and converting the raw data into a high-dimensional numerical vector that can be recognized and calculated by a computer, with the aim of extracting key semantic and feature information from the data.
[0070] Step S30: Use multimodal feature vectors to perform deep alignment training on the multimodal fusion model to obtain a pre-trained multimodal fusion model. The multimodal fusion model is based on the pre-trained model enhanced with medical knowledge, and adds a feature projection layer, a cross-modal attention layer, and a gated fusion layer.
[0071] A multimodal fusion model is a deep learning model used to integrate multimodal feature vectors. For example, based on a pre-trained model enhanced with medical knowledge, additional feature projection layers, cross-modal attention layers, and gated fusion layers are added.
[0072] Step S40: Obtain the rehabilitation report generation model based on the pre-trained multimodal fusion model.
[0073] The rehabilitation report generation model is a specialized model built on a pre-trained multimodal fusion model. It has the ability to generate rehabilitation reports that conform to clinical standards by utilizing the aligned and fused multimodal features. It is the core carrier connecting multimodal data and the final rehabilitation report output.
[0074] Step S50: Generate a rehabilitation report based on the rehabilitation report generation model.
[0075] Based on steps S10-S50 above, firstly, multimodal data related to patients is collected, providing a comprehensive and rich foundation of patient information for the model; feature encoding is performed on the multimodal data to obtain multimodal feature vectors; deep alignment training is performed on the multimodal fusion model using the multimodal feature vectors to obtain a pre-trained multimodal fusion model. The multimodal fusion model is based on the pre-trained model enhanced with medical knowledge, and adds a feature projection layer, a cross-modal cross-attention layer, and a gated fusion layer, effectively solving the problems of strong heterogeneity and poor alignment fusion effect of multimodal data in traditional technologies, and can fully explore the semantic associations between different modal data; a rehabilitation report generation model is obtained based on the pre-trained multimodal fusion model; a rehabilitation report is generated based on the rehabilitation report generation model, which can integrate multi-dimensional patient information, avoid the limitations of single-modal data, and thus the generated rehabilitation report can more accurately reflect the patient's actual physiological state and rehabilitation status, significantly improving the accuracy of the rehabilitation report.
[0076] In one specific embodiment of the rehabilitation report generation method of this application, the pre-trained model is obtained through the following steps: acquiring medical knowledge text, the medical knowledge text including a first marker identifying key paragraphs and a second marker identifying professional terms; dynamically masking the medical knowledge text according to the first and second markers; using the pre-trained base model to predict terms in the masked medical knowledge text; calculating the difference between the prediction result and the actual terms using the cross-entropy loss function; adjusting the training parameters of the pre-trained base model according to the difference until the difference is less than a first preset threshold or the number of training iterations reaches a second preset threshold, thereby obtaining the trained pre-trained model.
[0077] Medical knowledge texts are professional documents in the medical field, such as medical literature and clinical guidelines (including terms such as Brunnstrom staging and Fugl-Meyer score).
[0078] Specifically, the first step is to collect texts containing medical expertise (such as medical literature and clinical guidelines) and structure them by using a first tag (e.g., [DOMAIN]) to identify key domain paragraphs and a second tag (e.g., [TERM]) to wrap technical terms (e.g., Brunnstrom staging, upper limb flexor synergistic movements). Next, a dynamic masking operation is performed on the tagged medical knowledge text. Specifically, a higher masking rate (e.g., 30%) is applied to technical terms identified by the second tag, while a lower masking rate (e.g., 15%) is applied to common words. Simultaneously, key paragraphs are assigned a higher attention weight (e.g., 1.5 times) through the first tag to enhance the model's learning of core domain knowledge. Subsequently, a large model based on the Transformer-decoder architecture (e.g., Qwen2.5-14B) is used to predict technical terms in the masked text. The difference between the predicted results and the actual terms is calculated using the cross-entropy loss function, and the model training parameters are adjusted based on this difference (e.g., learning rate of 3e-5, batch size of 6, etc.). Training continues until the loss difference is less than a preset threshold or a preset number of training epochs is reached (e.g., 3 epochs), and finally, a pre-trained model is obtained.
[0079] By introducing a labeling mechanism to distinguish key paragraphs from technical terms, and employing dynamic masking and attention weighting strategies, the pre-trained model can specifically enhance its learning of medical domain knowledge, significantly improving its understanding and prediction capabilities of technical terms. Simultaneously, the training process and parameter optimization based on the cross-entropy loss function ensure that the model can efficiently absorb professional information from medical knowledge texts, solving the problems of low coverage of medical terminology and insufficient understanding of domain knowledge in general pre-trained models. This provides a model foundation with strong medical expertise for subsequent tasks such as multimodal data fusion and rehabilitation report generation.
[0080] The following provides further explanation of steps S10 to S50.
[0081] Regarding step S10, this application uses case text, medical images, and follow-up text as multimodal data. The case text is collected from the de-identified electronic medical record text of the patient and typically includes eight modules: basic patient information, chief complaint, present illness history, physical examination, auxiliary examination conclusions (image description, laboratory indicators), diagnostic conclusion, treatment plan, and key points of the progress notes.
[0082] The medical images may be MRI (magnetic resonance imaging) images of the patient's brain after privacy information has been removed. For example, a stroke patient can be used as an example; subsequent embodiments will use a stroke patient as an example to explain in detail the generation of the rehabilitation report in this application.
[0083] The follow-up records are collected from the recorded data with the patient's permission, which can be further transcoded into single-channel waveform audio file format (wav) data with 16 kHz and 16 bits, and then the asr-api is called to complete the text transcription. Finally, it can be used only after passing the manual verification.
[0084] The above is a further explanation of step S10. Next, a further explanation of step S20 will be continued.
[0085] Before executing step S20, the multimodal data can be preprocessed first to reduce the noise in the multimodal data and improve the accuracy of the multimodal data.
[0086] Exemplarily, for case texts, the preprocessing process may include:
[0087] Text cleaning and standardization, specifically including clearing non-text symbols (such as garbled characters and special characters), and retaining necessary punctuation marks (periods, commas); unifying the date format, digital expression (Arabic numerals) and units; standardizing language expressions (such as replacing the non-standard expression "high blood pressure" with "hypertensive disease");
[0088] Structured field sub-module processing, specifically including: splitting the text according to eight major modules (chief complaint, current medical history, etc.), and extracting structured fields (such as drug names and dosages in the allergy history); completing the missing field labels (for example, no allergy history is marked as <no allergy history>);
[0089] Word segmentation and stop word filtering, specifically including: using a medical field dictionary (such as the ICD-11 disease coding table) to enhance the recognition of professional terms; filtering general stop words (such as "de", "shi", "le") and medical irrelevant words (hospital names, equipment models);
[0090] Stem extraction and synonym mapping, specifically including: merging the same word roots (such as "resection" and "post-resection" unified as "resection"); establishing a synonym table (such as mapping "myocardial infarction" to "heart attack").
[0091] Exemplarily, for medical images, the preprocessing process may include:
[0092] Denoising, such as frequency domain filtering, using wavelet transform to perform soft threshold processing on the high-frequency noise coefficients to reduce the noise in the high-frequency part (such as small miscellaneous points); spatial domain filtering, using the non-local means filtering (NLM) algorithm to find similar small block regions in the image (such as the normal brain tissue region), and taking the average value of these regions to denoise, avoiding damaging the edges of the lesions. Through denoising, the image can be made clearer while retaining the boundaries of cerebral infarction (such as the white high signal in the acute infarction area).
[0093] Resampling, such as interpolation adjustments, "stretches" or "compresses" images to the same size, unifying the resolution of all sequences (e.g., adjusting all to 1mm). 3 (Small cubic pixels); orientation alignment forces all images to be adjusted to a uniform "right-front-up" orientation, avoiding misalignment of brain structures. Resampling can align the layers of different images, making it easier for doctors to observe lesions from multiple angles.
[0094] Image enhancement, such as contrast stretching, uses histogram equalization to "enlarge" the difference in brightness between hazy areas (similar to contrast adjustment in mobile phone photos); edge sharpening uses methods similar to "sharpening filters" (such as Gaussian-Laplacian filtering) to make the boundaries of infarcted areas clearer (e.g., sharpening the edges of acute infarction). Image enhancement can make infarcted areas (such as highlighted areas in DWI sequences) more prominent, facilitating lesion location identification.
[0095] Normalization, such as global brightness adjustment using healthy brain white matter (e.g., the corpus callosum) as a benchmark, brings the brightness of all images to the same level; and matching color distribution, such as through histogram matching, ensures that the grayscale distribution of all images approximates a standard template. Normalization can unify the tonal views of images from different MRI devices, facilitating direct comparative analysis and reducing misjudgments.
[0096] In one specific embodiment of the rehabilitation report generation method of this application, the multimodal data includes case text, medical images, and follow-up text; the feature encoding of the multimodal data includes: inserting preset tags into the case text for text segmentation; performing word segmentation on the segmented case text to obtain a case text feature vector; and / or
[0097] The medical image is encoded using a visual encoder, and the output vector corresponding to the feature aggregation marker at the start position of the sequence is extracted as the medical image feature vector based on the encoding result; and / or
[0098] The follow-up text is subjected to structured information extraction according to a preset template to obtain structured data pairs; feature extraction is performed on the structured data pairs to obtain follow-up text feature vectors.
[0099] Specifically, when encoding multimodal data, for case texts, special markers can first be inserted to segment the eight modules in the case to make the boundaries of feature information clear, for example: "[MAIN] Chief complaint: weakness in the left limbs...[MAIN]"; then a pre-trained model enhanced with medical knowledge can be used, which is processed into word vectors by a tokenizer and used to participate in the training of the subsequent alignment model.
[0100] For medical images, a visual encoder can be used to encode the images; specifically, a VIT-structured visual encoder can be used to encode the features of the medical images. For example... Figure 2 As shown, the visual encoder with the VIT structure includes an image segmentation and embedding layer, a position encoding layer, a transformer encoder, and a classification head.
[0101] Image segmentation and embedding layers are used to divide the input image into fixed-size blocks (e.g., 16×16 pixels). Each block is flattened and then converted into an embedding vector through linear projection. For example, a 224×224 RGB image is divided into 14×14=196 blocks, each block is flattened into a 16×16×3=768-dimensional vector, and then mapped to a 768-dimensional vector space through a linear projection layer.
[0102] The positional encoding layer is used to add learnable positional embeddings to each image patch to preserve spatial information. For example, there are 197 positional encoding vectors for 196 image patches and 1 classification token ([CLS] token), each with the same 768 dimensions as the embedding vector.
[0103] The Transformer encoder consists of a multi-layer structure, with each layer containing a multi-head self-attention mechanism (capturing global dependencies) and a feedforward neural network (non-linear transformation). For example, the structure of each encoder layer may include: a normalization layer (LayerNorm), multi-head attention, residual connections, a multilayer perceptron (MLP), and residual connections.
[0104] The classification head (MLP Head) is used to extract the output vector corresponding to the [CLS] token. It performs classification prediction through a multilayer perceptron (MLP) and maps the 768-dimensional [CLS] vector to a preset target category, such as 1000 target categories.
[0105] In the above process, the output vector corresponding to the extracted [CLS] token is the feature vector encoding of the patient's medical image, with dimensions [1, 768].
[0106] Follow-up texts can be dedicated to follow-up questionnaires. For example, a set of 50 follow-up assessment questions designed with reference to official stroke prevention and control guidelines and literature. The question types include multiple choice and fill-in-the-blank questions, and the content covers a comprehensive and multi-dimensional range of information, including records of new events during the follow-up period, descriptions of current symptoms, details of physiological indicators, behavioral habits, dietary management, medication use, rehabilitation progress, chronic disease management, and psychological status.
[0107] The preset template can be a pre-built structured template, such as a Prompt template. The structure of a Prompt template could be: "You are a text information extraction expert with extensive knowledge of stroke medicine. Below, I will give you a question and a doctor-patient dialogue. Here is the dialogue content: [Enter follow-up call] Question: [What is the patient's fasting blood glucose level?] Please help me extract the answer to the question from the dialogue and return it in the format [Analysis]...[Answer]...."
[0108] For follow-up texts, large model APIs (such as GPT-4) can be called to extract question-answer pairs from the follow-up texts to obtain structured data pairs. For example, the prompt project can first convert the texts into a formatted questionnaire, represented in JSON format. Then, the questionnaire containing 50 follow-up questions and user answers can be grouped into 50 question-answer pairs, such as, "What was the patient's blood pressure in the past 7 days? 70 / 120 mmHg".
[0109] For example, the format template can be as follows: [{"question_id":1,"question_name":"What is the patient's fasting blood glucose level in the past 7 days?","answer":"4.2mmol / L"},{"question_id":2,"question_name":"What is the patient's blood pressure in the past 7 days?","answer":"70 / 120mmHg"},...].
[0110] The text vectorization pre-trained model (BERT) is further called to extract features from 50 texts, generating a [50*128]-dimensional matrix, which is then flattened into a vector of dimension [6400, 1], and used as the vector representation of the random access volume.
[0111] The above is a further explanation of step S20. Step S30 will be further explained below.
[0112] In one specific embodiment of the rehabilitation report generation method of this application, the feature projection layer includes a visual feature projection layer and a follow-up feature projection layer; the multimodal feature vector includes case text feature vectors, medical image feature vectors, and follow-up text feature vectors. For example... Figure 3 As shown, the multimodal fusion model includes:
[0113] The visual feature projection layer is used to map the medical image feature vectors to a multimodal shared text space to obtain medical image projection features. Specifically, the visual feature projection layer adopts an MLP structure and superimposes absolute position encoding (ROPE) to project the features (medical image feature vectors) encoded by the ViT structure visual encoder onto the multimodal shared text space to obtain medical image projection features.
[0114] The follow-up feature projection layer is used to map the follow-up text feature vectors to a multimodal shared text space to obtain follow-up text projection features. Specifically, the follow-up feature projection layer may include a learnable self-attention module and an MLP, used to learn deep information of the follow-up text feature vectors and map them to a multimodal shared text space to obtain follow-up text projection features.
[0115] The concatenation layer is used to concatenate the case text feature vector with the medical image projection feature to generate a first fused feature. Specifically, the token concatenation layer is used to concatenate the case text feature vector (e.g., dimension B, T_len, 5120) with the medical image projection feature (e.g., dimension B, 768, 5120) along a second dimension to generate a vector of dimension (B, T_len + 768, 5120), where B (Batch Size) is the batch size, T_len (Text Length) is the text length, and 5120 is the feature dimension.
[0116] The pre-trained model is used to perform cross-modal fusion processing based on the follow-up text projection features and the first fusion feature to obtain multimodal depth-aligned semantic fusion features. Specifically, as follows... Figure 3 As shown, the pre-trained model is based on the medical knowledge-enhanced pre-trained model, with the addition of a feature projection layer, a cross-modal cross-attention layer, and a gated fusion layer. The specific positions of the cross-modal cross-attention layer and the gated fusion layer in the pre-trained model can be adaptively adjusted according to the number of layers in the actual base model. The medical knowledge-enhanced pre-trained model fully retains the 48-layer Transformer structure of Qwen2.5-14B, inheriting its core components such as RoPE positional encoding and SwiGLU activation. The specific training process of the medical knowledge-enhanced pre-trained model has been detailed in the preceding steps and will not be repeated here.
[0117] The pre-trained model includes:
[0118] The first cross-attention layer is used to obtain the second fusion feature by using the first fusion feature as the query vector and the follow-up text projection feature as the key vector and value vector, through information interaction via a group attention mechanism. Specifically, as follows... Figure 3As shown, the first cross-attention layer can be connected to either the 16th or 32nd layer of the pre-trained model. The first cross-attention layer uses the first fusion feature as the query vector and the follow-up text projection feature as the key vector and value vector. It splits the 5120-dimensional vector into 40 heads, each head being 128-dimensional. At the same time, it adopts a group attention mechanism, with 8 heads for key and value computation. Each key-value head is shared by multiple query heads (40 / 8 = 5), which reduces memory usage while ensuring expressive power.
[0119] The second cross-attention layer uses the follow-up text projection features as the query vector and the second fusion feature as the key and value vectors, respectively, to interact with information through a group attention mechanism to obtain the third fusion feature. Specifically, as follows... Figure 3 As shown, the second cross-attention layer can be connected to either the 32nd or 45th layer of the pre-trained model. The second cross-attention layer uses the follow-up text projection features as the query vector and the second fusion features as the key vector and value vector. It also uses a group attention mechanism for information interaction, and the grouping method is the same as that of the first cross-attention layer.
[0120] The intermediate layer is used to semantically integrate the third fusion feature to obtain high-level semantic features. For example... Figure 3 As shown, the intermediate layer refers to the network layer located between the second cross-attention layer and the gated fusion layer. Specifically, it performs semantic integration on the third fusion feature output by the second cross-attention layer to obtain high-level semantic features.
[0121] A gated fusion layer is used to perform high-level fusion based on the second fusion feature, the third fusion feature, and the high-level semantic features to obtain high-dimensional fusion features. For example... Figure 3 As shown, the gated fusion layer can be connected to either layer 45 of the pre-trained model or the output layer, wherein the gated fusion layer includes:
[0122] The splicing module is used to splice the second fusion feature, the third fusion feature, and the high-level semantic feature to obtain a spliced feature. Specifically, the splicing module can splice the second fusion feature output by the first cross-attention layer, the third fusion feature output by the second cross-attention layer, and the high-level semantic feature output by the intermediate layer in the second dimension. The final output spliced feature dimension is [B,T_len+768,5120*3]=[B,T_len+768,15360].
[0123] The linear module is used to perform dimensionality transformation on the concatenated features and generate a weight vector after passing them through an activation function. Specifically, the linear module includes a linear layer and a sigmoid activation function, which can transform the dimension [B, T_len+768, 15360] of the concatenated features into [B, T_len+768, 3], and generate a weight vector after passing it through the activation function, corresponding to the weights of h_self, h_cross1, and h_cross2, respectively, with values between [0, 1].
[0124] The weighted fusion module is used to perform weighted fusion on the second fusion feature, the third fusion feature, and the high-level semantic feature to obtain a weighted fusion feature. Specifically, the weighted fusion module is used to perform weighted fusion on the second fusion feature, the third fusion feature, and the high-level semantic feature according to the three weight vectors output by the linear module to obtain a weighted fusion feature with dimensions [B, T_len+768, 5120].
[0125] A standardization layer is used to standardize the weighted fusion features to obtain the high-dimensional fusion features. Specifically, the standardization layer may include a normalization layer, residual connections, and a feedforward neural network (FFN), wherein the normalization layer is used to normalize the weighted fusion features in the feature dimension, and by adjusting the mean and variance of the features, the numerical distribution of each feature is made more stable.
[0126] Residual connections alleviate the gradient vanishing problem in deep networks while preserving original feature information.
[0127] Feedforward neural networks (FFNs) typically consist of two linear transformations and an intermediate nonlinear activation function (such as ReLU or GELU). They can perform complex nonlinear mappings on input features and capture higher-order correlations between features. Although the input and output dimensions ([B, T_len+768, 5120]) remain unchanged, FFNs, through internal dimensionality expansion (e.g., first increasing the dimensionality to a higher dimension and then reducing it back to 5120), enable the model to learn richer feature details and improve its ability to recognize complex patterns.
[0128] The multimodal model constructed in this application innovatively designs grouped cross-attention and dynamic gating modules (sigmoid weights + residual connections) to achieve cross-modal semantic alignment. This overcomes the semantic gap problem between multi-source heterogeneous medical data (text, images, and speech) in traditional single-modal or shallow fusion methods, and improves the robustness of the model and its ability to fuse multimodal data.
[0129] It is understood that the encoder used to encode data in this application can be in the form of other network structures, and the structure of the pre-trained model in this application can also be a large model with any transformer-decoder architecture.
[0130] Specifically, step S30 can be implemented through the following steps S301 to S304.
[0131] Step S301: Activate the visual feature projection layer and the follow-up feature projection layer, and freeze all other layers except the visual feature projection layer and the follow-up feature projection layer.
[0132] Specifically, the visual feature projection layer and follow-up feature projection layer in the multimodal fusion model are activated first, while the parameters of other layers in the model are frozen to avoid interference from irrelevant layer parameters in the basic alignment process.
[0133] Step S302: The multimodal fusion model is trained by using a loss function that is a weighted combination of contrast alignment loss and modal reconstruction loss.
[0134] Specifically, the alignment training aims to establish a unified semantic space across modalities and align the underlying features of different modalities such as questionnaires, medical records, and images. The training loss function adopts a weighted combination of symmetric contrastive loss and modality reconstruction loss. The contrastive alignment loss is constructed based on the InfoNCE loss function, taking the follow-up text projection feature qa and the concatenated vector (first fusion feature) med after projecting medical records and images as input. It achieves cross-modal feature alignment by maximizing the similarity of matching (Q,V) pairs and suppressing the similarity of non-matching pairs. The modality reconstruction loss adopts L2 loss (mean squared error), using the follow-up text feature vector obtained by the BERT encoder and the medical image feature vector obtained by the VIT structure encoder as the original vector X. raw Let X be the vector after the projection module. projected By constraining the projection function f proj Acting on X raw The result after that approximates X projected To ensure information fidelity during feature projection, two loss mechanisms are used, with weights adjusted via hyperparameters λ1 and λ2 (which require tuning), to jointly support accurate cross-modal feature alignment training. The contrastive alignment loss function can be expressed as:
[0135] Where qa represents the follow-up text projection feature, med represents the first fusion feature, and the mathematical definition of the InfoNCE loss function is:
[0136] Where B is the batch size; i and k are both indices of samples in the batch; V i t represents the first fusion feature of the i-th sample; k Let be the follow-up text feature vector of the k-th sample; Represents the eigenvector V i With tk The similarity score between them; τ is the temperature parameter.
[0137] The modal reconstruction loss can be expressed as:
[0138] in, These are the follow-up text feature vectors obtained by the BERT encoder and the medical image feature vectors obtained by the VIT structure encoder, respectively. These are the projection vectors obtained after passing through the visual feature projection layer and the follow-up feature projection layer, respectively. These are the projection functions for the visual feature projection layer and the follow-up feature projection layer, respectively, used to map the raw data to the target modality space.
[0139] L2 loss (mean squared error) is used to constrain the projected feature f proj (X raw Approaching target feature X projected The L2 loss can be expressed as:
[0140] After weighting the alignment loss and mode reconstruction loss: L total =λ1L align +λ2L recon
[0141] The hyperparameters λ1 and λ2 are used to control the weights of contrast alignment and reconstruction.
[0142] Step S303: Activate the first cross-attention layer, the second cross-attention layer and the gated fusion layer, and freeze the other layers except for the first cross-attention layer, the second cross-attention layer and the gated fusion layer.
[0143] Specifically, after completing the alignment training of the multimodal fusion model through the aforementioned steps, the first cross-attention layer, the second cross-attention layer, and the gated fusion layer are activated, while the other layers are frozen.
[0144] Step S304: Use the contrast alignment loss function to perform fusion training on the multimodal fusion model.
[0145] Specifically, the contrastive alignment loss function is continued to be used to train the multimodal fusion model, so as to realize the dynamic interaction and semantic fusion of multimodal features through the attention mechanism, improve the understanding of complex scenes, and thus effectively solve the problem of attention fragmentation in multimodal medical data.
[0146] The pre-trained multimodal fusion model can be obtained through the above training. By adopting a phased parameter freezing strategy, combined with contrastive loss and modality reconstruction loss, efficient model convergence and consistency of multimodal representations are achieved, alleviating parameter redundancy and modality conflict problems in end-to-end multimodal training, thus saving training resources while ensuring model performance. Simultaneously, by fully utilizing patients' pathological information and health status, personalized rehabilitation plans can be provided, thereby improving patients' medical experience and accelerating the rehabilitation process. Furthermore, automatic report generation can shorten doctors' review time and improve the utilization rate of medical resources.
[0147] The above is a further explanation of step S30. Step S40 will be further explained below.
[0148] Specifically, step S40 can be implemented through the following steps S401 to S403.
[0149] Step S401: Using the pre-trained multimodal fusion model as the base model, freeze the model parameters of the multimodal fusion model.
[0150] Specifically, it can be frozen Figure 3 Model parameters of the multimodal fusion model in [the context].
[0151] Step S402: Introduce the fine-tuning module and perform fine-tuning training on the fine-tuning module.
[0152] LoRA (Low-Rank Adaptation) is a parameter-efficient model fine-tuning technique. Its core principle is to capture subtle changes in model weights through low-rank matrix factorization, rather than updating all parameters. Essentially, it inserts trainable low-rank perturbation matrices into key layers of the model, learning a small number of low-rank parameters to adapt the model to a specific task while maintaining the core capabilities of the pre-trained model.
[0153] The fine-tuning module in this step is the LoRA module. The LoRA module is embedded into the key layers of the base model in the form of low-rank matrix pairs (usually the attention weight layer in the Transformer architecture, such as the query / value weight matrix in Qwen2.5-14B-instruct). By modifying the low-rank perturbation terms of these layers, the LoRA module indirectly affects the feature interaction and output logic of the model.
[0154] The fine-tuning of the LoRA module is specifically based on the XTuner framework and the Qwen2.5-14B-instruction code. This includes using over 2000 paired samples of trimodal data and rehabilitation reports verified by professional doctors. These samples must contain the correspondence between inputs (fusion features of questionnaires, medical records, and images) and outputs (structured rehabilitation report text). Some samples can be structured using rehabilitation report templates and content maps. Within the XTuner framework, the insertion position of the LoRA module (e.g., the attention layer of the base model), the rank of the low-rank matrix (usually 8-32, controlling parameter size), and the fine-tuning range (training only the LoRA module parameters, freezing all parameters of the base model) are specified. A low learning rate adapted to LoRA (usually 1e-5 to 5e-5, such as 3e-5) is used to avoid disrupting the pre-trained knowledge of the base model due to excessive parameter updates.
[0155] Cross-entropy loss, commonly used in data generation tasks, can be employed. The optimization objective is to address the difference between the model-generated rehabilitation report text and the actual report text, ensuring the accuracy and professionalism of the output. The number of epochs should be set to 3-10 based on the amount of data, dynamically adjusted using validation set performance to avoid overfitting. Backpropagation is used to update only the low-rank matrix parameters of the LoRA module, enabling the model to learn the mapping logic from three-modal features to rehabilitation reports while maintaining its multimodal fusion capabilities. Ultimately, this results in a rehabilitation report generation model adapted to clinical needs.
[0156] By freezing the base model and training the LoRA module with lightweight methods, the model is efficiently adapted to the rehabilitation report generation task while significantly reducing computational resource consumption, thus balancing the preservation of pre-trained knowledge with the optimization of task performance.
[0157] Step S403: Obtain the rehabilitation report generation model based on the pre-trained multimodal fusion model and the fine-tuned model.
[0158] Specifically, the final rehabilitation report generation model is the result of fine-tuning the model after training, inheriting the cross-modal processing capabilities of the multimodal fusion model and the task adaptation capabilities learned by the fine-tuned model. It takes trimodal data as input, and through a complete process of first fusing the trimodal data and then adapting the generation logic through the LoRA module, it directly outputs a rehabilitation report that meets clinical requirements.
[0159] The above is a further explanation of step S40. Step S50 will be further explained below.
[0160] Specifically, in step S50, the collected multimodal data (including case text, medical images, and follow-up text) can be input into the rehabilitation report generation model to output a rehabilitation report.
[0161] For example, Figure 4 This can serve as an example of the aforementioned stroke rehabilitation report. Specifically, as follows: Figure 4 As shown, a stroke rehabilitation report is a personalized rehabilitation report for patients. The primary sub-topics of this report include a core health overview, key data tracking, risk warning, rehabilitation plan, medical basis, and next-stage goals. These primary sub-topics can be further subdivided into secondary sub-topics. For example, secondary sub-topics for the core health overview might include stroke recurrence risk, motor function, and psychological state; secondary sub-topics for key data tracking might include new events and symptoms, physiological indicator monitoring, and rehabilitation progress. Of course, these secondary sub-topics can be further refined to obtain tertiary sub-topics. For instance, the tertiary sub-topic corresponding to stroke recurrence risk could be status (medium risk) and basis (no recurrence within three months, but a history of transient ischemic attack).
[0162] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of this application.
[0163] Furthermore, this application also provides a rehabilitation report generation device.
[0164] See appendix Figure 5 , Figure 5 This is a main structural block diagram of a rehabilitation report generation device according to an embodiment of this application.
[0165] like Figure 5 As shown, the rehabilitation report generation device in this embodiment mainly includes a data acquisition module 11, an encoding module 12, a training module 13, an acquisition module 14, and a generation module 15. In some embodiments, one or more of the data acquisition module 11, encoding module 12, training module 13, acquisition module 14, and generation module 15 can be combined into a single module.
[0166] In some embodiments, the acquisition module 11 can be configured to acquire multimodal data related to the patient.
[0167] The encoding module 12 can be configured to perform feature encoding on the multimodal data to obtain a multimodal feature vector.
[0168] The training module 13 can be configured to perform deep alignment training on the multimodal fusion model using the multimodal feature vectors to obtain a pre-trained multimodal fusion model. The multimodal fusion model is based on the medical knowledge-enhanced pre-trained model and adds a feature projection layer, a cross-modal attention layer, and a gated fusion layer.
[0169] The acquisition module 14 can be configured to acquire a rehabilitation report generation model based on the pre-trained multimodal fusion model.
[0170] The generation module 15 can be configured to generate a rehabilitation report based on the rehabilitation report generation model.
[0171] In one implementation, a description of the specific functions can be found in steps S10-S50.
[0172] The aforementioned rehabilitation report generation device is used for execution Figure 1 The embodiments of the rehabilitation report generation method shown are similar in technical principle, technical problem solved and technical effect produced. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the rehabilitation report generation device can be referred to the contents described in the embodiments of the rehabilitation report generation method, and will not be repeated here.
[0173] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device described in this application, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of both. Therefore, the number of modules shown in the figures is merely illustrative.
[0174] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of this application; therefore, the technical solutions after splitting or combining will fall within the protection scope of this application.
[0175] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0176] Furthermore, this application also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the rehabilitation report generation method described in any of the above embodiments. See also Figure 6 As shown, Figure 6 The structure of an electronic device, including a processor 100 and a memory 200, is illustrated by way of example.
[0177] Furthermore, this application also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the rehabilitation report generation method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described rehabilitation report generation method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a memory device formed by various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.
[0178] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.
[0179] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.
[0180] The technical solution of this application has been described in conjunction with the specific embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.
Claims
1. A rehabilitation report generation method characterized by, The method comprises: collecting multi-modal data related to a patient; feature encoding the multi-modal data to obtain a multi-modal feature vector; training a multi-modal fusion model using the multi-modal feature vector to obtain a pre-trained multi-modal fusion model, the multi-modal fusion model being based on a pre-trained model enhanced with medical knowledge, and adding a feature projection layer, a cross-modal cross-attention layer and a gating fusion layer; obtaining a rehabilitation report generation model based on the pre-trained multi-modal fusion model; generating a rehabilitation report based on the rehabilitation report generation model.
2. The rehabilitation report generation method according to claim 1, characterized by, The multi-modal data includes case text, medical images and follow-up text; The feature encoding of the multi-modal data comprises: inserting a preset mark into the case text for text segmentation; performing word segmentation processing on the case text after text segmentation to obtain a case text feature vector; and / or using a visual encoder to encode the medical images, and extracting an output vector corresponding to a feature aggregation mark at a sequence start position as a medical image feature vector; and / or extracting structured information from the follow-up text according to a preset template to obtain a structured data pair; and extracting features from the structured data pair to obtain a follow-up text feature vector.
3. The rehabilitation report generation method of claim 1, wherein, The feature projection layer comprises a visual feature projection layer and a follow-up feature projection layer; and the multi-modal feature vector comprises a case text feature vector, a medical image feature vector and a follow-up text feature vector; The multi-modal fusion model comprises: The visual feature projection layer is configured to map the medical image feature vector to a multi-modal shared text space to obtain a medical image projection feature; The follow-up feature projection layer is configured to map the follow-up text feature vector to the multi-modal shared text space to obtain a follow-up text projection feature; The splicing layer is configured to splice the case text feature vector and the medical image projection feature to generate a first fusion feature; The pre-trained model is configured to perform cross-modal fusion processing based on the follow-up text projection feature and the first fusion feature to obtain a multi-modal depth-aligned semantic fusion feature.
4. The rehabilitation report generation method according to claim 3, characterized by, The pre-trained model comprises: The first cross-attention layer is configured to take the first fusion feature as a query vector, take the follow-up text projection feature as a key vector and a value vector, and perform information interaction through a grouping attention mechanism to obtain a second fusion feature; The second cross-attention layer is configured to take the follow-up text projection feature as a query vector, take the second fusion feature as a key vector and a value vector, and perform information interaction through a grouping attention mechanism to obtain a third fusion feature; The intermediate layer is configured to perform semantic integration on the third fusion feature to obtain a high-level semantic feature; The gating fusion layer is configured to perform high-level fusion based on the second fusion feature, the third fusion feature and the high-level semantic feature to obtain a high-dimensional fusion feature.
5. The rehabilitation report generation method according to claim 4, characterized by, The gating fusion layer comprises: The splicing module is configured to splice the second fusion feature, the third fusion feature and the high-level semantic feature to obtain a spliced feature; and A linear module is configured to convert dimensions of the splicing feature and generate a weight vector after an activation function; A weighted fusion module is configured to perform weighted fusion on the second fusion feature, the third fusion feature and the high-level semantic feature based on the weight vector to obtain a weighted fusion feature. A standardization layer is configured to standardize the weighted fusion feature to obtain the high-dimensional fusion feature.
6. The rehabilitation report generation method of claim 4, wherein, The depth alignment training of the multi-modal fusion model by using the multi-modal feature vector comprises: activating the visual feature projection layer and the follow-up feature projection layer, and freezing other layers except the visual feature projection layer and the follow-up feature projection layer; performing alignment training on the multi-modal fusion model by using a loss function that is a weighted combination of a contrast alignment loss and a modal reconstruction loss; activating the first cross-attention layer, the second cross-attention layer and the gating fusion layer, and freezing other layers except the first cross-attention layer, the second cross-attention layer and the gating fusion layer; performing fusion training on the multi-modal fusion model by using a contrast alignment loss function.
7. The rehabilitation report generation method of claim 1, wherein, The rehabilitation report generation model based on the pre-trained multi-modal fusion model comprises: taking the pre-trained multi-modal fusion model as a base model, freezing model parameters of the multi-modal fusion model; introducing a fine-tuning module and fine-tuning the fine-tuning module; obtaining the rehabilitation report generation model based on the pre-trained multi-modal fusion model and the fine-tuning model.
8. The rehabilitation report generation method of claim 1, wherein, The medical knowledge enhanced pre-training model is obtained by the following steps: obtaining medical knowledge text, the medical knowledge text containing first marks identifying key paragraphs and second marks identifying professional terms; performing dynamic masking on the medical knowledge text according to the first marks and the second marks; performing term prediction on the masked medical knowledge text by using a pre-training base model; calculating the difference between the prediction result and the true term by using a cross-entropy loss function; adjusting the training parameters of the pre-training base model according to the difference until the difference is less than a first preset threshold or the number of training reaches a second preset threshold, and obtaining the trained pre-training model.
9. An electronic device, comprising: comprise: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program, and the computer program is executed by the at least one processor to implement the rehabilitation report generation method in any one of claims 1 to 8.
10. A computer readable storage medium having stored therein a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by the processor to execute the rehabilitation report generation method in any one of claims 1 to 8. The program code is adapted to be loaded and run by the processor to execute the rehabilitation report generation method in any one of claims 1 to 8.