Method and system for monitoring patient dietary compliance after discharge based on multimodal model

Abstract

The present invention discloses a method and system for monitoring the dietary compliance of patients discharged from hospital based on a multimodal model. The method comprises: inputting a binary group including a dietary image and a desensitization discharge medical order text and a labeling result obtained by labeling the binary group whether the diet complies with the medical order dietary recommendation into a pre-trained Transformer-based multimodal model for training the dietary compliance recognition task; extracting the harmful dietary name in the desensitization discharge medical order text to generate a harmful dietary image, calculating the image similarity between the dietary image and the harmful dietary image, correcting the labeling result according to the image similarity and assigning a weight to the loss function of the corresponding binary group, and fine-tuning the model using the weight-corrected loss function; and using the fine-tuned Transformer-based multimodal model for dietary compliance monitoring. The present invention can effectively improve the accuracy, comprehensiveness and reliability of monitoring the compliance of patients discharged from hospital medical orders.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent health management, and in particular relates to a method and system for monitoring dietary compliance of discharged patients based on a multimodal model. Background Art

[0002] With the continuous development of medical technology and the increase in patient management needs, how to effectively monitor and evaluate patients' compliance with medical orders after discharge, especially dietary compliance, has become an important research topic in the medical field. Good dietary compliance is one of the key factors to ensure patient recovery. However, monitoring of patients' dietary compliance with medical orders after discharge often faces many challenges such as strong subjectivity, difficulty in data acquisition, and high complexity of analysis, making it difficult to ensure the accuracy, comprehensiveness and reliability of the monitoring results.

[0003] Traditional monitoring methods mostly rely on self-reports from patients or direct observations from doctors. These methods have limitations such as strong subjectivity and difficulty in continuous tracking. With the development of computer vision and neural networks, traditional applications usually use convolutional neural networks (CNNs) as image recognition models. For example, the Chinese patent application with publication number CN114360690A provides a method and system for dietary nutrition management for patients with chronic diseases. It identifies and locates multiple dishes in dietary images and segments the ingredients, and then performs perspective restoration of the ingredient segmentation map, calculates the nutritional elements of the ingredients, and provides personalized dietary recommendations. However, this type of method can only recognize image content and output category labels for image objects, and cannot simultaneously understand the input text content of medical records. This method has obvious limitations, because evaluating a patient's dietary compliance requires comprehensive consideration of multiple information including images and text.

[0004] In addition, traditional similar applications often use simple rules to determine whether the diet is harmful to the patient's physical condition after using CNN to identify the category labels of food names. For example, the Chinese patent application with publication number CN113948188A provides a diet and exercise intelligent monitoring system, which obtains diet evaluation opinions based on diet photos and conducts compliance analysis in combination with diet recipes. There are two main problems with this type of method: first, the recall rate of simple rules is low and cannot handle complex situations; second, mechanical matching methods are prone to errors, especially when patients suffer from multiple diseases at the same time, and the dietary requirements of different diseases may conflict, and simple rules are difficult to take into account.

[0005] In recent years, with the rise of multimodal technology, comprehensive analysis combining multiple data sources such as images, physiological sensor data, and text has provided new ideas for further improving the accuracy, comprehensiveness, and reliability of patient discharge medical order compliance monitoring. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a method and system for monitoring the dietary compliance of patients discharged from hospital based on a multimodal model. Through the training of the multimodal model based on Transformer and the fine-tuning based on the error correction mechanism, the accuracy, comprehensiveness and reliability of the identification of the patient's compliance with medical orders after discharge can be improved, which is helpful to improve the quality of medical care, optimize the continuous monitoring and management of patient health, and promote the efficient use of medical resources.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0008] The embodiment of the present invention provides a method for monitoring the dietary compliance of patients discharged from hospital based on a multimodal model, comprising the following steps:

[0009] Input the labeled results of the two-tuples including the dietary image and the desensitization discharge instructions text and whether the two-tuples follow the dietary recommendations of the doctor into the pre-trained Transformer-based multimodal model for training the dietary compliance recognition task;

[0010] Extract harmful dietary names from desensitization discharge instructions to generate harmful dietary images, calculate the image similarity between dietary images and harmful dietary images, correct the annotation results based on the image similarity, assign weights to the loss function of the corresponding binary groups based on the correction results, and use the weighted loss function to fine-tune the trained Transformer-based multimodal model.

[0011] The patient's diet photos and the text of the patient's discharge instructions are input into the fine-tuned Transformer-based multimodal model for dietary compliance recognition to obtain a dietary compliance report.

[0012] Preferably, the step of inputting the labeling result obtained by labeling the two-tuple including the diet image and the desensitization discharge doctor's order text and whether the diet complies with the doctor's order and dietary recommendations into the pre-trained Transformer-based multimodal model for training the dietary compliance recognition task includes:

[0013] Pre-training a Transformer-based multimodal model that can process both images and text on a large-scale medical data corpus;

[0014] Constructing several bigrams including diet images and desensitization discharge instructions texts, and labeling the bigrams to determine whether the diet complies with the doctor's dietary recommendations to obtain labeling results;

[0015] The bigram and the first cue word are input into the pre-trained Transformer-based multimodal model for dietary compliance recognition to determine whether the dietary image complies with the dietary recommendations in the desensitization discharge instructions text. The annotation results are used as labels for supervised training to finally generate a dietary compliance report.

[0016] Preferably, the Transformer-based multimodal model includes using the LLaVA multimodal model as a base model.

[0017] Preferably, the labeling of the binary group to determine whether the diet complies with the doctor's dietary recommendations to obtain a labeling result includes:

[0018] Each group of dietary images and desensitization discharge instructions were input into the QWEN-VL-MAX model. By designing the second prompt word, the QWEN-VL-MAX model was enabled to label whether the diet followed the doctor's instructions. Based on the labeling results, a label of whether the diet followed the doctor's instructions was generated for each binary group.

[0019] Preferably, the step of extracting harmful diet names from the desensitization discharge instructions to generate harmful diet images includes:

[0020] The names of harmful diets that may affect patients' health were extracted from the desensitization discharge instructions, and the names of harmful diets were input into the Vincent graph diffusion model, and the corresponding harmful diet images were generated using the third prompt word.

[0021] Preferably, the calculating of the image similarity between the dietary image and the harmful dietary image comprises:

[0022] The cosine similarity of the dietary images and harmful dietary images was calculated as the image similarity using the CLIP model.

[0023] Preferably, the step of correcting the labeling results according to the image similarity and assigning weights to the loss functions of the corresponding binary groups according to the correction results includes:

[0024] For the annotation results that are judged to be in compliance with the doctor's advice, if the corresponding image similarity is s If the value is higher than the threshold, the result of following the doctor's advice is judged to be wrong, and the weight of the loss function of the corresponding binary group is reduced to 1- s ; For the annotated results that are judged as not following the doctor's advice, if the corresponding image similarity s If the value is higher than the threshold, the result of not following the doctor's advice is considered correct, and the weight of the loss function of the corresponding binary group is increased to 1+ s .

[0025] Preferably, the step of fine-tuning the trained Transformer-based multimodal model using the weight-corrected loss function comprises:

[0026] During the model fine-tuning process, the first The loss function for a binary pair Multiply by the corresponding weight, the calculation formula is as follows:

[0027] ,

[0028] in, For the The weighted loss function of the binary group;

[0029] The model is fine-tuned based on the weight-corrected loss function of each binary group, and finally a fine-tuned Transformer-based multimodal model is obtained.

[0030] Preferably, the content of the dietary compliance report includes:

[0031] Whether the diet complies with the doctor's instructions, the basis for judgment, dietary health analysis and improvement suggestions.

[0032] To achieve the above-mentioned purpose of the invention, the embodiment of the present invention also provides a patient discharge diet compliance monitoring system based on a multimodal model, comprising: a multimodal model training module, a multimodal model fine-tuning module and a diet compliance recognition module;

[0033] The multimodal model training module is used to input the binary group including the diet image and the desensitization discharge doctor's order text and the labeling result obtained by labeling the binary group whether the diet complies with the doctor's order and dietary recommendations into the pre-trained Transformer-based multimodal model to perform training on the dietary compliance recognition task;

[0034] The multimodal model fine-tuning module is used to extract the names of harmful diets in the desensitization discharge instructions text to generate harmful diet images, calculate the image similarity between the diet images and the harmful diet images, correct the annotation results according to the image similarity and assign weights to the loss function of the corresponding binary according to the correction results, and use the weighted loss function to fine-tune the trained Transformer-based multimodal model;

[0035] The dietary compliance recognition module is used to input the patient's dietary photos and the patient's discharge instructions text into the fine-tuned Transformer-based multimodal model to perform dietary compliance recognition to obtain a dietary compliance report.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) The present invention uses a Transformer-based multimodal model to realize automated and intelligent dietary compliance monitoring, which can greatly improve the efficiency of patient management. Medical institutions can remotely monitor patients' dietary compliance, thereby reducing unnecessary follow-up visits and re-hospitalizations, effectively saving medical resources and reducing medical costs.

[0038] (2) The present invention uses a Transformer-based multimodal model to obtain dietary compliance reports, which can help medical institutions promptly identify patients' dietary compliance problems and provide specific cause analysis, so as to take targeted intervention measures, improve patients' prognosis quality, and reduce the risk of complications.

[0039] (3) The present invention designs a correction mechanism based on image similarity and annotation results, assigns weights to the loss function of the corresponding binary according to the correction results, and uses the weighted loss function to fine-tune the trained Transformer-based multimodal model, which can effectively improve the accuracy, comprehensiveness and reliability of patient discharge medical order compliance recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 It is a flow chart of a method for monitoring dietary compliance of patients discharged from hospital based on a multimodal model according to an embodiment of the present invention;

[0042] Figure 2 It is a flowchart of an error correction mechanism in a model fine-tuning process provided by an embodiment of the present invention;

[0043] Figure 3 It is a flow chart of monitoring the dietary compliance of patients discharged from hospital by using a fine-tuned Transformer-based multimodal model provided by an embodiment of the present invention;

[0044] Figure 4 It is a structural schematic diagram of a patient discharge diet compliance monitoring system based on a multimodal model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0046] The inventive concept of the present invention is: in response to the problems of insufficient accuracy, comprehensiveness and reliability of patient discharge compliance monitoring in the prior art, the embodiments of the present invention provide a patient discharge diet compliance monitoring method and system based on a multimodal model. By adopting a Transformer-based multimodal model to monitor patient discharge diet compliance, it has significant advantages such as cross-modal information fusion, complex situation processing, personalized analysis and utilization of medical prior knowledge. Furthermore, a model fine-tuning based on an error correction mechanism is designed, which can further improve the accuracy, comprehensiveness and reliability of patient discharge compliance identification, help improve medical quality, optimize continuous monitoring and management of patient health, and promote efficient use of medical resources.

[0047] Figure 1 FIG. 1 is a flow chart of a method for monitoring dietary compliance of discharged patients based on a multimodal model provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a method for monitoring patient diet compliance after discharge based on a multimodal model, comprising the following steps:

[0048] S1, input the annotation results of the binary group including the dietary image and the desensitization discharge instruction text and whether the dietary compliance with the doctor's dietary recommendations is followed into the pre-trained Transformer-based multimodal model for training the dietary compliance recognition task.

[0049] S1.1, the Transformer-based multimodal model can process images and texts at the same time, making full use of information from both modalities. The model can not only understand the image content, but also understand the text semantics, and output the corresponding dietary compliance report according to the input instructions. This cross-modal information fusion capability greatly improves the accuracy and comprehensiveness of dietary compliance judgments. At the same time, the model adopts attention mechanism and deep learning methods, which can automatically learn key information in images and texts, and make comprehensive judgments based on the patient's individualized medical records, greatly improving the accuracy and personalization of dietary compliance judgments, and can well deal with complex situations and coexisting diseases.

[0050] In the embodiment, the Transformer-based multimodal model uses the LLaVA multimodal model as the basic model and is pre-trained on a large-scale medical data corpus. The model learns a large amount of medical field knowledge during the training process and has rich medical prior knowledge. This prior knowledge is crucial to improving the accuracy of dietary compliance judgments, because the judgment requires comprehensive consideration of the patient's disease type, symptom manifestations, treatment plans and other medical factors. Due to the lack of such medical prior knowledge, traditional models are difficult to make comprehensive and accurate judgments. The Transformer-based multimodal model provided in the embodiment of the present invention can make full use of this prior information through pre-training on massive medical data, making the judgment results more reliable and professional.

[0051] S1.2, a key challenge in training a multimodal model is to obtain high-quality training data. An innovative data acquisition method is used in the embodiment to effectively solve the problem of lack of ready-made medical compliance data. In the embodiment, about 10,000 Chinese dishes and 2,000 Western food images were collected from food-related websites as image inputs for the model. At the same time, 1,000 real discharge orders after desensitization were collected as text inputs for the model. In order to generate the correspondence between medical orders and dietary photos, all dietary images and desensitized discharge order texts are randomly combined into 1,000 pairs of tuples including dietary images and medical orders, and all tuples are constructed as training data sets.

[0052] S1.3, input the bigrams in the training dataset into the pre-trained Transformer-based multimodal model for dietary compliance recognition, and design the first prompt word input model. The model determines whether the dietary image complies with the dietary recommendations in the desensitization discharge instructions text, and uses the annotation results as labels for supervised training, and finally generates a dietary compliance report.

[0053] The example of designing the first prompt word is as follows:

[0054] Please analyze the following patient's medical records and dietary photos to determine whether the patient's diet complies with the doctor's instructions and explain your reasons in detail:

[0055] 1. Carefully read the medical records provided, paying special attention to the doctor's dietary recommendations and restrictions for the patient;

[0056] 2. Carefully observe the patient's diet photos and identify the food and drinks in the photos;

[0057] 3. Compare the doctor's instructions with the actual diet to determine whether the patient complies with the doctor's dietary recommendations;

[0058] 4. If you find any situation that is inconsistent with the doctor's advice, please specify which food or eating behavior is inconsistent with the doctor's advice;

[0059] 5. If the patient's diet is in accordance with the doctor's instructions, please explain what went well;

[0060] 6. Analyze the possible impact of the patient’s eating habits on his or her health status;

[0061] 7. Give a comprehensive assessment of whether the patient has generally complied with the doctor's orders and provide suggestions for improvement.

[0062] Please provide detailed and accurate analysis and judgment based on the information in the medical record text and dietary photos. The answer should include specific observations and reasoning process.

[0063] Finally, the dietary compliance report output by the model includes the following aspects:

[0064] 1. Whether the patient complies with the doctor's dietary recommendations;

[0065] 2. If there is any discrepancy, indicate which foods or dietary behaviors are inconsistent with the doctor's instructions;

[0066] 3. If yes, describe what went well;

[0067] 4. Analyze the possible impact of the patient’s eating habits on his or her health status.

[0068] 5. Provide a comprehensive assessment of whether the patient has generally complied with the doctor’s orders and provide suggestions for improvement.”

[0069] When constructing the label, each set of diet images and desensitization discharge instructions are input into the QWEN-VL-MAX model, and the second prompt word is designed to enable the QWEN-VL-MAX model to label whether the diet follows the doctor's instructions. Based on the labeling results, a label of whether the diet follows the doctor's instructions is generated for each set of diet images and desensitization discharge instructions. This method can generate training data quickly and on a large scale, saving a lot of time and cost for manual labeling.

[0070] Among them, the example of designing the second prompt word is as follows:

[0071] “Please analyze the following patient’s medical records and dietary photos to determine whether the patient’s diet complies with the doctor’s instructions:

[0072] 1. Carefully read the medical records provided, paying special attention to the doctor's dietary recommendations and restrictions for the patient;

[0073] 2. Carefully observe the patient's diet photos and identify the food and drinks in the photos;

[0074] 3. Compare the doctor’s instructions with the actual diet to determine whether the patient complies with the doctor’s dietary recommendations.”

[0075] S2, extract the harmful diet names in the desensitization discharge instructions to generate harmful diet images, calculate the image similarity between the diet image and the harmful diet image, correct the annotation results according to the image similarity and assign weights to the loss function of the corresponding binary group according to the correction results, and use the weighted loss function to fine-tune the trained Transformer-based multimodal model. Figure 2 shown.

[0076] S2.1, extract the names of harmful diets that doctors believe may have an impact on the patient's health from the desensitization discharge instructions, then input these harmful diet names into the text-generated graph diffusion model, and use the third prompt word to generate corresponding harmful diet images. The text-generated graph diffusion model in the embodiment adopts a Stable-diffusion model.

[0077] Among them, the example of designing the third prompt word is as follows:

[0078] "[Specific dish name], clear ingredient details, bright colors, high-resolution food photography, overhead angle, authentic presentation, natural light, extreme details, photo-realistic."

[0079] S2.2, use the CLIP model to calculate the cosine similarity of diet images and harmful diet images as image similarity .

[0080] S2.3, for the annotation results that are judged by the QWEN-VL-MAX model as following the doctor's advice, the corresponding bigram is a high compliance bigram (i.e., the patient follows the doctor's advice and avoids eating harmful foods). If the corresponding image similarity is s If the value of the binary group is higher than a threshold value (in the embodiment, the threshold value is set to 0.6), the labeling result of following the doctor's advice for the binary group is determined to be wrong, and the weight of the loss function of the corresponding binary group is reduced to 1- s On the contrary, for the annotation results that are judged as not following the doctor's advice, the corresponding bigram is a low compliance bigram (that is, the patient did not follow the doctor's advice and ate harmful food). s If the value is higher than the threshold, the result of not following the doctor's advice is considered correct, and the weight of the loss function of the corresponding binary group is increased to 1+ s .

[0081] S2.4, during the model fine-tuning process, the first The loss function for a binary pair Multiply by the corresponding weight to adjust the contribution of the bigram to model training. The calculation formula is as follows:

[0082] ,

[0083] in, For the The weight-corrected loss function of each binary is used to fine-tune the model, and finally a fine-tuned Transformer-based multimodal model is obtained. The above-mentioned data error correction method based on annotation results and image similarity calculation can effectively reduce the impact of errors and improve the performance of the fine-tuned Transformer-based multimodal model.

[0084] S3, input the patient's diet photos and the patient's discharge instructions text into the fine-tuned Transformer-based multimodal model for dietary compliance recognition to obtain a dietary compliance report.

[0085] like Figure 3 As shown in the figure, the fine-tuned Transformer-based multimodal model is used to detect the patient's dietary compliance with the doctor's orders before discharge. The patient first uses the APP to take photos of his daily diet, and the APP uploads the photos. At the same time, the APP obtains relevant dietary orders from the medical database through doctor's order query, and then inputs the dietary photos and relevant dietary orders into the fine-tuned Transformer-based multimodal model, and finally outputs a dietary compliance report.

[0086] In summary, the embodiment of the present invention provides a method for monitoring the dietary compliance of patients discharged from the hospital based on a multimodal model. The Transformer-based multimodal model trained by the data acquisition and error correction method provided in the above embodiment has an accuracy 8% higher than the model trained directly using the large model annotation results. This shows that the method provided by the embodiment of the present invention can effectively improve the quality of training data, thereby improving the performance of the model, and effectively improve the accuracy, comprehensiveness and reliability of patient discharge medical order compliance monitoring. It is an important innovation and progress in the application of artificial intelligence technology in the medical field, and provides a new solution for intelligent and precise dietary compliance monitoring.

[0087] Based on the same inventive concept, Figure 4 As shown, an embodiment of the present invention further provides a patient discharge dietary compliance monitoring system 400 based on a multimodal model, comprising: a multimodal model training module 410, a multimodal model fine-tuning module 420 and a dietary compliance identification module 430.

[0088] The multimodal model training module 410 is used to input the binary group including the dietary image and the desensitization discharge instruction text, as well as the labeling results of the binary group to indicate whether the diet complies with the doctor's dietary recommendations, into the pre-trained Transformer-based multimodal model for training the dietary compliance recognition task.

[0089] The multimodal model fine-tuning module 420 is used to extract the names of harmful diets in the desensitization discharge order text to generate harmful diet images, calculate the image similarity between the diet images and the harmful diet images, correct the annotation results according to the image similarity and assign weights to the loss function of the corresponding binary according to the correction results, and use the weighted corrected loss function to fine-tune the trained Transformer-based multimodal model.

[0090] The dietary compliance recognition module 430 is used to input the patient's dietary photos and the patient's discharge instructions text into the fine-tuned Transformer-based multimodal model to perform dietary compliance recognition to obtain a dietary compliance report.

[0091] It should be noted that the patient discharge dietary compliance monitoring system based on a multimodal model provided in the above-mentioned embodiment belongs to the same inventive concept as the patient discharge dietary compliance monitoring method based on a multimodal model. The specific implementation process is detailed in the embodiment of the patient discharge dietary compliance monitoring method based on a multimodal model, which will not be repeated here.

[0092] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for monitoring dietary compliance of discharged patients based on a multimodal model, characterized in that: The following steps are involved: Input the binary group including the dietary image and the desensitization discharge doctor's order text and the labeling result of labeling the binary group whether the diet complies with the doctor's order and dietary recommendations into the pre-trained Transformer-based multimodal model for training the dietary compliance recognition task. The Transformer-based multimodal model includes the LLaVA multimodal model as the basic model; Extract the harmful diet names in the desensitization discharge instructions to generate harmful diet images, and calculate the image similarity between the diet image and the harmful diet image; correct the annotation results according to the image similarity and assign weights to the loss function of the corresponding binary group according to the correction results, including: for the annotation results that are judged to be in compliance with the doctor's instructions, if the corresponding image similarity is s If the value is higher than the threshold, the result of following the doctor's advice is judged to be wrong, and the weight of the loss function of the corresponding binary group is reduced to 1- s ; For the annotated results that are judged as not following the doctor's advice, if the corresponding image similarity s If the value is higher than the threshold, the result of not following the doctor's advice is considered correct, and the weight of the loss function of the corresponding binary group is increased to 1+ s ; Use the weight-corrected loss function to fine-tune the trained Transformer-based multimodal model, including: in the process of model fine-tuning, the first The loss function for a binary pair Multiply by the corresponding weight, the calculation formula is as follows: ， in, For the The weight-corrected loss function of each bigram is used to fine-tune the model, and finally a fine-tuned Transformer-based multimodal model is obtained; The patient's diet photos and the text of the patient's discharge instructions are input into the fine-tuned Transformer-based multimodal model for dietary compliance recognition to obtain a dietary compliance report.

2. The method for monitoring dietary compliance of patients discharged from hospital based on a multimodal model according to claim 1, characterized in that: The step of inputting the labeled results of the binary group including the diet image and the desensitization discharge doctor's order text and whether the diet complies with the doctor's order and dietary recommendations into the pre-trained Transformer-based multimodal model for training the dietary compliance recognition task includes: Pre-training a Transformer-based multimodal model that can process both images and text on a large-scale medical data corpus; Constructing several bigrams including diet images and desensitization discharge instructions texts, and labeling the bigrams to determine whether the diet complies with the doctor's dietary recommendations to obtain labeling results; The bigram and the first cue word are input into the pre-trained Transformer-based multimodal model for dietary compliance recognition to determine whether the dietary image complies with the dietary recommendations in the desensitization discharge instructions text. The annotation results are used as labels for supervised training to finally generate a dietary compliance report.

3. The method for monitoring dietary compliance of patients discharged from hospital based on a multimodal model according to claim 1 or 2, characterized in that: The labeling result of labeling whether the diet of the two-tuple group complies with the doctor's dietary advice is obtained, including: Each group of dietary images and desensitization discharge instructions were input into the QWEN-VL-MAX model. By designing the second prompt word, the QWEN-VL-MAX model was enabled to label whether the diet followed the doctor's instructions. Based on the labeling results, a label of whether the diet followed the doctor's instructions was generated for each binary group.

4. The method for monitoring dietary compliance of patients discharged from hospital based on a multimodal model according to claim 1, characterized in that: The step of extracting harmful diet names from desensitization discharge instructions to generate harmful diet images includes: The names of harmful diets that may affect patients' health were extracted from the desensitization discharge instructions, and the names of harmful diets were input into the Vincent graph diffusion model, and the corresponding harmful diet images were generated using the third prompt word.

5. The method for monitoring dietary compliance of patients discharged from hospital based on a multimodal model according to claim 1, characterized in that: The calculating the image similarity between the dietary image and the harmful dietary image comprises: The cosine similarity of the dietary images and harmful dietary images was calculated as the image similarity using the CLIP model.

6. The method for monitoring dietary compliance of discharged patients based on a multimodal model according to claim 1 or 2, characterized in that: Contents of the dietary compliance report include: Whether the diet complies with the doctor's instructions, the basis for judgment, dietary health analysis and improvement suggestions.

7. A patient diet compliance monitoring system based on a multimodal model, characterized in that: include: Multimodal model training module, multimodal model fine-tuning module and dietary compliance identification module; The multimodal model training module is used to input the binary group including the diet image and the desensitization discharge doctor's order text and the labeling result obtained by labeling the binary group whether the diet complies with the doctor's order and dietary recommendations into the pre-trained Transformer-based multimodal model for training the dietary compliance recognition task, and the Transformer-based multimodal model includes using the LLaVA multimodal model as the basic model; Extract the harmful diet names in the desensitization discharge instructions to generate harmful diet images, calculate the image similarity between the diet image and the harmful diet image, correct the annotation results according to the image similarity and assign weights to the loss function of the corresponding binary group according to the correction results, including: for the annotation results that are judged to be in compliance with the doctor's instructions, if the corresponding image similarity is s If the value is higher than the threshold, the result of following the doctor's advice is judged to be wrong, and the weight of the loss function of the corresponding binary group is reduced to 1- s ; For the annotated results that are judged as not following the doctor's advice, if the corresponding image similarity s If the value is higher than the threshold, the result of not following the doctor's advice is considered correct, and the weight of the loss function of the corresponding binary group is increased to 1+ s ; Use the weight-corrected loss function to fine-tune the trained Transformer-based multimodal model, including: in the process of model fine-tuning, the first The loss function for a binary pair Multiply by the corresponding weight, the calculation formula is as follows: ， in, For the The weight-corrected loss function of each bigram is used to fine-tune the model, and finally a fine-tuned Transformer-based multimodal model is obtained; The dietary compliance recognition module is used to input the patient's dietary photos and the patient's discharge instructions text into the fine-tuned Transformer-based multimodal model to perform dietary compliance recognition to obtain a dietary compliance report.

Images