Ward monitoring method and system based on multi-modal large model and edge computing, terminal and storage medium

Abstract

The application relates to the technical field of data processing, and discloses a ward monitoring method and system based on a multi-modal large model and edge computing, a terminal and a storage medium.The method comprises the following steps: inputting user query voice into a target large language model to output a response result; inputting preprocessed video stream data into a fall detection model to output a fall detection result; constructing an abnormal behavior classification model, inputting the preprocessed video stream data into the abnormal behavior classification model, and outputting an abnormal behavior classification result; performing face key point recognition and emotion state classification processing on the preprocessed video stream data to obtain an emotion state evaluation result; and performing standardization processing and risk early warning processing on the response result, the fall detection result, the abnormal behavior classification result and the emotion state evaluation result to obtain a ward monitoring early warning result.The application can effectively improve the timeliness and accuracy of ward monitoring by constructing a multi-modal large model to perform real-time detection on the state of a patient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a ward monitoring method and system based on a multi-modal large model and edge computing, a terminal and a computer readable storage medium. BACKGROUND

[0002] The traditional nursing method for patients in a hospital mainly relies on periodic manual patrol and manual review of medical staff, but the patient data obtained by the patrol is in a discrete state, and all-weather and multi-dimensional continuous monitoring of the patient state cannot be realized.

[0003] The above-mentioned ward monitoring method is difficult to capture the subtle dynamic changes of the patient state, resulting in broken health data and missing individual characteristics of the patient, and precise and rapid monitoring and nursing of the patient cannot be realized.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a ward monitoring method and system based on a multi-modal large model and edge computing, a terminal and a computer readable storage medium, which aims to solve the problem that the traditional nursing method in the prior art mainly relies on periodic manual patrol and manual review of medical staff, and it is difficult to capture the subtle dynamic changes of the patient state, resulting in broken health data and missing individual characteristics of the patient, and precise and rapid monitoring and nursing of the patient cannot be realized.

[0006] To achieve the above-mentioned purpose, the present application provides a ward monitoring method based on a multi-modal large model and edge computing, which comprises the following steps:

[0007] Obtain a user query voice, and construct a target large language model, input the user query voice into the target large language model, and output a response result;

[0008] Obtain pre-processed video stream data, and construct a fall detection model, input the pre-processed video stream data into the fall detection model, and output a fall detection result;

[0009] Construct an abnormal behavior classification model, and input the pre-processed video stream data into the abnormal behavior classification model, and output an abnormal behavior classification result;

[0010] Perform face key point recognition and emotion state classification processing on the pre-processed video stream data to obtain an emotion state evaluation result;

[0011] The response result, the fall detection result, the abnormal behavior classification result and the emotional state evaluation result are standardized and risk early warning processed to obtain a ward monitoring early warning result.

[0012] Optionally, the ward monitoring method based on the multi-modal large model and edge calculation, wherein the user query voice is obtained, and a target large language model is constructed, the user query voice is input into the target large language model, and a response result is output, specifically including:

[0013] The preset medical corpus data is obtained, and the initial preset language model is trained according to the preset medical corpus data to obtain a pre-training model;

[0014] The multi-modal data is obtained, and the pre-training model is subjected to self-supervised learning processing according to the multi-modal data to obtain a multi-modal base model;

[0015] The preset demonstration data is obtained, and the multi-modal base model is subjected to supervised fine-tuning processing according to the preset demonstration data to obtain a supervised fine-tuning model;

[0016] The supervised fine-tuning model is subjected to first classification processing and reinforcement learning processing by using knowledge enhancement and RAG mechanism to obtain a target large language model;

[0017] When the user query voice is received, the user query voice is input into the target large language model, and the target large language model performs semantic understanding on the user query voice to output a response result.

[0018] Optionally, the ward monitoring method based on the multi-modal large model and edge calculation, wherein the pre-processing video stream data is obtained, and a fall detection model is constructed, the pre-processing video stream data is input into the fall detection model, and a fall detection result is output, specifically including:

[0019] The real-time video stream data of the patient is obtained, and the real-time video stream data is subjected to first preprocessing to obtain pre-processing video stream data, wherein the first preprocessing includes image denoising processing, target region cropping processing and target region compression processing;

[0020] The first preset data set is obtained, and the first preset data set is subjected to behavior annotation to obtain a first annotation data set;

[0021] The initial fall detection model is determined, and the initial fall detection model is subjected to model training and model optimization according to the first annotation data set to obtain a fall detection model;

[0022] The preprocessed video stream data is input into the fall detection model, and the fall detection model performs fall detection processing on the preprocessed video stream data to obtain the fall detection result.

[0023] Optionally, in the ward monitoring method based on multimodal large model and edge computing, the initial fall detection model includes a dual-branch parallel encoder, a multi-level feature fusion module, and a behavior state classifier. The dual-branch parallel encoder includes a skeletal point temporal encoder and a three-dimensional convolutional feature encoder.

[0024] The process of determining an initial fall detection model and training and optimizing the initial fall detection model based on the first labeled dataset to obtain a fall detection model specifically includes:

[0025] The first labeled dataset is input into the skeletal point temporal encoder and the three-dimensional convolutional feature encoder in the initial fall detection model, respectively.

[0026] The first labeled dataset is encoded using the skeleton point temporal encoder to obtain the first encoded feature;

[0027] The second encoded feature is obtained by performing motion information capture and texture feature extraction on the first labeled dataset using the three-dimensional convolutional feature encoder.

[0028] The first encoded feature and the second encoded feature are input into the multi-level feature fusion module, and the multi-level feature fusion module performs feature fusion on the first encoded feature and the second encoded feature to obtain the fused feature;

[0029] The fused features are input into the behavior state classifier, and the behavior state classifier performs a second classification process on the fused features to obtain the behavior analysis results. The initial fall detection model training is then completed.

[0030] A reinforcement learning mechanism is used to optimize the initial fall detection model after training, resulting in a new fall detection model.

[0031] Optionally, the ward monitoring method based on multimodal large model and edge computing, wherein the step of constructing an abnormal behavior classification model, inputting the preprocessed video stream data into the abnormal behavior classification model, and outputting the abnormal behavior classification result specifically includes:

[0032] Perform behavioral annotation and decomposition annotation on the first preset dataset to obtain the second labeled dataset;

[0033] An initial abnormal behavior classification model is acquired, and model training is performed on the initial abnormal behavior classification model according to the second labeled data set to obtain a first abnormal behavior classification model;

[0034] The first abnormal behavior classification model is fine-tuned by using an incremental learning mechanism to obtain an abnormal behavior classification model;

[0035] The preprocessed video stream data is input into the abnormal behavior classification model, and an abnormal behavior classification result is output.

[0036] Optionally, the ward monitoring method based on the multi-modal large model and edge computing, wherein the preprocessing video stream data is subjected to face key point recognition and emotion state classification processing to obtain an emotion state evaluation result, specifically comprising:

[0037] A face detection model is determined, and the preprocessed video stream data is subjected to second preprocessing and region positioning processing by using the face detection model to obtain a target face region, wherein the second preprocessing includes cropping processing and normalization processing;

[0038] The target face region is subjected to face key point detection and key point position change tracking processing by using a face key point detection algorithm and a multi-target tracking algorithm to obtain a face key point motion sequence;

[0039] The emotional features and behavioral features of the patient are acquired, and the face key point motion sequence, the emotional features and the behavioral features are input into an emotion recognition model, and the emotion recognition model is used to perform emotion state classification processing on the face key point motion sequence, the emotional features and the behavioral features to obtain an emotion state evaluation result.

[0040] Optionally, the ward monitoring method based on the multi-modal large model and edge computing, wherein the response result, the fall detection result, the abnormal behavior classification result and the emotion state evaluation result are subjected to standardization processing and risk early warning processing to obtain a ward monitoring early warning result, specifically comprising:

[0041] The response result, the fall detection result, the abnormal behavior classification result and the emotion state evaluation result are subjected to standardization processing and third preprocessing to obtain structured data; wherein the standardization processing includes data cleaning processing, format conversion processing and timestamp alignment processing, and the third preprocessing includes localized data cleaning processing and abnormality detection processing;

[0042] A multi-modal time series prediction model is constructed, and the structured data is input into the multi-modal time series prediction model;

[0043] Adopting a time series analysis algorithm, the multi-modal time series prediction model is used to perform risk prediction processing on the structured data, and a risk prediction result is obtained.

[0044] A preset rule engine is determined, and the risk prediction result is processed for early warning according to the preset rule engine, and a ward monitoring early warning result is obtained.

[0045] In addition, to achieve the above-mentioned purpose, the application also provides a ward monitoring system based on a multi-modal large model and edge computing, wherein the ward monitoring system based on the multi-modal large model and edge computing comprises:

[0046] A response result output module is configured to obtain user query voice, construct a target large language model, input the user query voice into the target large language model, and output a response result.

[0047] A fall result output module is configured to obtain preprocessed video stream data, construct a fall detection model, input the preprocessed video stream data into the fall detection model, and output a fall detection result.

[0048] A behavior result output module is configured to construct an abnormal behavior classification model, input the preprocessed video stream data into the abnormal behavior classification model, and output an abnormal behavior classification result.

[0049] An emotion result output module is configured to perform face key point recognition and emotion state classification processing on the preprocessed video stream data, and obtain an emotion state evaluation result.

[0050] An early warning result output module is configured to perform standardization processing and risk early warning processing on the response result, the fall detection result, the abnormal behavior classification result, and the emotion state evaluation result, and obtain a ward monitoring early warning result.

[0051] In the application, the user query voice is obtained, the target large language model is constructed, the user query voice is input into the target large language model, and the response result is output. The preprocessed video stream data is obtained, the fall detection model is constructed, the preprocessed video stream data is input into the fall detection model, and the fall detection result is output. The abnormal behavior classification model is constructed, the preprocessed video stream data is input into the abnormal behavior classification model, and the abnormal behavior classification result is output. The face key point recognition and emotion state classification processing are performed on the preprocessed video stream data, and the emotion state evaluation result is obtained. The response result, the fall detection result, the abnormal behavior classification result, and the emotion state evaluation result are standardized and processed for risk early warning, and the ward monitoring early warning result is obtained. The application can effectively improve the timeliness and accuracy of ward monitoring by constructing a multi-modal large model to perform real-time detection on the state of a patient. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a flowchart of a preferred embodiment of the ward monitoring method based on the multimodal large model and edge computing of the present application;

[0053] Figure 2 is a schematic diagram of the overall structure of a preferred embodiment of the ward monitoring method based on the multimodal large model and edge computing of the present application;

[0054] Figure 3 is a schematic diagram of the intelligent ward active nursing system of multimodal cognitive computing of a preferred embodiment of the ward monitoring method based on the multimodal large model and edge computing of the present application;

[0055] Figure 4 is a schematic diagram of the construction process of the multimodal large language model training data of a preferred embodiment of the ward monitoring method based on the multimodal large model and edge computing of the present application;

[0056] Figure 5 is a schematic diagram of the training process of the multimodal large language model of a preferred embodiment of the ward monitoring method based on the multimodal large model and edge computing of the present application;

[0057] Figure 6 is a schematic diagram of the workflow of the intelligent sensing and computer vision system of a preferred embodiment of the ward monitoring method based on the multimodal large model and edge computing of the present application;

[0058] Figure 7 is a schematic diagram of the processing flow of the medical Internet of Things data fusion hub of a preferred embodiment of the ward monitoring method based on the multimodal large model and edge computing of the present application;

[0059] Figure 8 is a schematic diagram of the hardware and software architecture of a preferred embodiment of the ward monitoring method based on the multimodal large model and edge computing of the present application;

[0060] Figure 9 is a structural diagram of a preferred embodiment of the ward monitoring system based on the multimodal large model and edge computing of the present application;

[0061] Figure 10 is a structural diagram of a preferred embodiment of the terminal of the present application. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0063] Traditional hospital ward monitoring has the following main shortcomings: 1. Lack of continuity and individualization in health assessment: Traditional care mainly relies on periodic manual patrol of medical staff, and the assessment process is discrete, which cannot realize all-weather and multi-dimensional continuous monitoring. This mode is difficult to capture the subtle dynamic changes of the patient's state, leading to broken health data and missing individual characteristics, so it cannot support the accurate formulation and dynamic optimization of individualized nursing plan. 2. Low efficiency of doctor-patient communication and insufficient information transmission: Traditional communication mainly relies on oral communication, and the channel is single. Patients often cannot accurately express their needs due to memory loss, emotional tension or speech disorders, and nurses also have difficulty in responding to the situation of each patient in real time. This not only increases the workload of nurses, but also easily causes unclear transmission of medical orders and insufficient understanding of patients, thereby affecting treatment compliance and psychological state. 3. Complicated inpatient process, poor patient experience: Especially for the elderly or patients with difficulty in movement, the traditional inpatient process involves multiple manual handling, queuing and waiting links, such as admission registration, examination appointment and payment for medicine, etc. The process is complex and inefficient, significantly increasing the operation and time cost of patients and medical staff. 4. Uneven distribution of nursing resources and lack of emergency response mechanism: In the traditional ward, the distribution of nursing manpower is often uneven. When an emergency occurs, limited resources are occupied, and regular nursing is easily delayed. At the same time, due to the lack of automatic risk grading and early warning mechanism, nursing staff cannot quickly identify and prioritize high-risk patients, resulting in delayed response and low resource utilization efficiency. 5. Serious fragmentation of medical data, restricting the improvement of nursing quality: Patient data is scattered in different information systems or paper documents, such as electronic medical records, test reports, medication records, etc., lacking real-time integration and correlation analysis capabilities. The data island phenomenon not only increases the difficulty of information retrieval, but also hinders medical staff from making scientific decisions based on global data, affecting the optimization of nursing process and the continuous improvement of service quality.

[0064] To solve the above problems, the present application provides an intelligent ward active nursing system based on multi-modal artificial intelligence, which aims to solve the key problems in traditional ward nursing through multi-modal perception, intelligent analysis and active intervention. The system integrates multi-modal large models, computer vision, medical Internet of Things and big data analysis technology, and builds a comprehensive, real-time and intelligent ward nursing and monitoring platform.

[0065] In view of the systematic technical defects mentioned in the technical background, the present application realizes the construction of the active nursing system through the following mechanisms: 1. Establish an intelligent driving mechanism for nursing process to solve the problem of missed manual patrol. 2. Develop a patient self-service guide diagnosis system to improve the efficiency of doctor-patient communication.

[0066] 3. Deploy an intelligent ward safety monitoring system to replace the lagging manual patrol mode. 4. Build a medical Internet of Things data fusion hub to solve the problem of data fragmentation.

[0067] The technical field involved in the present application includes the following:

[0068] 1. Multi-modal artificial intelligence technology: The present application takes multi-modal large models as the core, realizes deep understanding and efficient processing across multiple data modalities, and builds an advanced human-computer interaction interface, which specifically includes: cross-modal cognition and semantic understanding: The present application has the ability of joint analysis and semantic understanding of structured medical data and unstructured multi-modal data. Specifically, it includes the fusion processing of structured data such as electronic medical records and physiological parameters, and unstructured data such as visual behavior and speech emotion, so as to realize comprehensive perception and accurate judgment of patient status, and provide technical foundation for personalized nursing and active service. Virtual digital human interaction: The present application belongs to the cross field of artificial intelligence and virtual digital human technology, and particularly relates to a virtual digital human generation and multi-modal interaction method for medical scenarios. The system realizes natural human-computer interaction based on voice, expression and body movement by constructing a virtual medical assistant with emotional expression ability, which is used for medical order analysis, medical guidance and emotional accompaniment, and significantly improves the intelligence and humanization level of medical service.

[0069] 2. Intelligent perception and computer vision: The present application relates to the integrated application of intelligent perception and computer vision technology in medical monitoring system, which specifically includes realizing automatic identification and risk assessment of patient behavior through visual sensor and deep learning algorithm, upgrading traditional manual patrol mechanism to unmanned and real-time intelligent monitoring system, which specifically includes: high-risk behavior real-time monitoring: The present application realizes real-time monitoring and identification of high-risk behaviors such as falling, violent struggle and long-time stillness through high-definition visual sensing equipment deployed in wards and public areas combined with deep learning behavior recognition algorithm, and quickly triggers alarm mechanism when abnormality is detected, so as to improve emergency response efficiency. State and emotion assessment: The present application realizes automatic identification and assessment of emotional states such as pain and anxiety by analyzing patient facial micro-expression and voice signal. The multi-modal analysis method improves the accuracy of state judgment and provides more comprehensive decision basis for clinical nursing. Identity security management and tracking: The system integrates face recognition and voiceprint verification technology to realize real-time verification and permission management of the identity of personnel entering and leaving the ward, and prevent unauthorized access. At the same time, it has visual-based target detection and tracking function, effectively preventing patients from getting lost, and protecting the safety and privacy of medical areas.

[0070] 3. Medical Internet of Things and multi-source data fusion technology: The present application relates to the collection, standardization and fusion processing method of multi-source heterogeneous data in medical Internet of Things system, and builds a unified data processing hub to provide real-time and reliable data support for upper-layer intelligent application, which specifically includes:

[0071] Multi-protocol data acquisition and standardization: The present application accesses various medical devices, sensors and environmental monitoring devices through a multi-protocol gateway, realizes real-time acquisition and preliminary processing of multi-source data, and performs data cleaning, format standardization and time synchronization to form a complete and consistent patient health record. Intelligent terminal and data processing: The intelligent terminal, as a data access and edge computing node, has local data processing and interaction capabilities, can preprocess physiological parameters and behavior data, and upload data to the central data fusion platform through the communication module. Device data fusion: The present application realizes the comprehensive analysis and joint modeling of patient status by correlating and integrating multi-source heterogeneous data, and provides a data basis for risk prediction and task automation.

[0072] 4. Medical big data analysis and decision support: The present application relates to the analysis and mining technology of medical big data, in particular to an intelligent prediction and decision support method based on multi-modal time series data, which realizes the transformation of nursing mode from passive response to active prevention, specifically including: intelligent risk prediction model: based on the fused multi-modal time series data, the present application constructs a deep learning model, which can analyze the change trend of patient physiological indicators and behavior data, thereby predicting potential health risks (such as falls and cardiovascular accidents). Once the prediction probability exceeds the threshold, the system will trigger an early warning. Time series data analysis: the present application supports time series analysis of long-term patient data (such as medication records and sleep cycles), and mines individual behavior patterns to provide in-depth state insight for medical staff. Personalized nursing optimization: based on individual data and historical analysis results, the present application can generate and recommend personalized nursing plans, including medication time adjustment, rehabilitation exercise suggestion and diet optimization strategy, to improve the accuracy and effectiveness of nursing services.

[0073] The ward monitoring method based on multi-modal large model and edge computing according to the preferred embodiment of the present application, as shown in Figure 1 and Figure 2 , includes the following steps:

[0074] Step S10, acquiring user query voice, and constructing a target large language model, inputting the user query voice into the target large language model, and outputting a response result.

[0075] The present application provides an intelligent ward active nursing system based on multi-modal artificial intelligence, which aims to solve the key problems in traditional ward nursing through multi-modal perception, intelligent analysis and active intervention. The system integrates multi-modal large models, computer vision, medical Internet of Things and big data analysis technology, and constructs a comprehensive, real-time and intelligent ward nursing and monitoring platform.

[0076] As shown in Figure 3As shown, the intelligent ward accompanying system of the application is composed of the following core modules, which work together to provide patients with comprehensive intelligent nursing services: 1. Personified medical assistants (nurse stations or bed terminals): virtual medical assistants based on multi-modal large models, providing 24-hour consultation services, medical order analysis, payment and drug collection navigation, etc. 2. Intelligent sensing and computer vision system (public area or bed terminal): through the camera, the patient's behavior, emotion and identity verification are monitored in real time. 3. Medical Internet of Things data fusion hub (base support platform): responsible for data collection, fusion, real-time analysis and risk warning. 4. Bed intelligent terminal and multi-modal interaction system: embedded interaction equipment provides patients with remote calls, medication reminders, emotional companionship and other services. 5. Base support platform: including domestic AI computing platform, edge computing device, multi-modal sensing algorithm and medical big data analysis engine.

[0077] Among them, the application sets up a research and development patient self-service guide diagnosis system, which includes:

[0078] Technical goal: Build a one-stop intelligent guide diagnosis service to empower patients to complete the medical process independently and reduce the transactional burden of nursing staff.

[0079] Function description: Based on multi-modal large models, a virtual medical assistant is built to provide patients with convenient and humanized interactive services. Medical order analysis and medication instructions under natural language interaction: patients can directly ask about the content of the medical order through voice, and the assistant will explain it in simple and easy-to-understand language. Using Bluetooth beacons, Wi-Fi fingerprint recognition or ultra-wideband technology, an accurate indoor positioning network is built in the hospital. The patient's intelligent terminal (such as a mobile phone app or a bed terminal) can receive these signals and display the patient's current location and recommended path on the preloaded hospital map. This greatly facilitates patients who are unable to move or are not familiar with the hospital environment.

[0080] Intelligent verification of medical insurance settlement documents: through computer vision technology, patients can quickly check and process medical expenses, simplifying the medical process.

[0081] The application builds a virtual medical assistant (personified medical assistant) based on multi-modal large models, which can provide patients with 24-hour uninterrupted and humanized intelligent interactive services. The assistant not only understands the patient's natural language instructions, but also responds with emotional voice, expressions and movements, thus providing professional medical services while considering emotional companionship and psychological support.

[0082] Among them, the personification medical assistant technical process is as follows: 1, multi-modal perception and interaction interface: the application integrates multiple interaction methods, including but not limited to voice interaction, touch screen operation and multi-modal perception. Through high-precision voice recognition service, accurate capture and understanding of patient natural language are realized, and 7x24h voice wake-up service is supported. At the same time, high-definition touch display screen interaction is used as an auxiliary, especially suitable for scenes where voice communication is inconvenient or interface operation (such as viewing reports, selecting services) is needed. 2, style image driving: according to the nursing needs of different departments and the characteristics of patient groups, multiple styles of digital human images are designed and generated, for example, the department can upload cartoon, hand-drawn, real person pictures, generate 3d digital human image through calling generation model, and use it for image construction. The characteristics of the virtual image can be dynamically adjusted according to the type of the ward. Combined with expression generation algorithm and body language driving technology, it is ensured that the digital human not only has static image, but also can generate rich and natural expressions and body movements in real time according to the content of the conversation, emotional state and context. 3, large language model: as the "brain" of the system, it is responsible for semantic understanding and intention recognition of the patient's voice instruction, and generates accurate and professional text answers combined with its professional knowledge in the medical field. 4, multi-modal response linkage: the application realizes the cooperative response of multi-modal. When the large language model generates an answer, it will output voice content, expression instruction (mouth shape synchronization) and body action instruction at the same time. These instructions will drive the voice synthesis engine and image driving system at the same time, ensuring that the voice, expression and action of the digital human are completed at the same time point, providing a coherent and smooth interactive experience.

[0083] Specifically, preset medical corpus data is acquired, and an initial preset language model is trained according to the preset medical corpus data to obtain a pre-training model; multi-modal data is acquired, and the pre-training model is subjected to self-supervised learning processing according to the multi-modal data to obtain a multi-modal base model; preset demonstration data is acquired, and the multi-modal base model is subjected to supervised fine-tuning processing according to the preset demonstration data to obtain a supervised fine-tuning model; the supervised fine-tuning model is subjected to first classification processing and reinforcement learning processing by using knowledge enhancement and RAG mechanism (Retrieval-Augmented Generation), to obtain a target large language model; when a user query voice is received, the user query voice is input to the target large language model, and the target large language model performs semantic understanding on the user query voice and outputs a response result.

[0084] As shown in Figure 4 The construction process of the multi-modal large language model training data is as follows: in order to construct the data set required by the large model, the application will widely collect open source data and medical professional book materials, and on this basis, a private data set (i.e. the preset medical corpus data in the application) is constructed for model training.

[0085] Foundation model and pre-training: Based on self-developed multi-modal large model, the training process is divided into four stages: 1. Large-scale unsupervised pre-training: using massive general text and medical professional corpus for pre-training, constructing the basic language ability and medical field macro-cognition of the model, and improving the understanding ability of medical entities and their relationships; 2. Multi-modal self-supervised fusion and correlation learning: through cross-modal data alignment task, promote the model to understand the internal correlation between text, image and other multi-modal data. 3. Instruction fine-tuning and human feedback reinforcement learning: optimize the instruction following ability and professional dialogue style of the model, so that its output is more in line with the needs of medical scene and human preference; 4. Knowledge enhancement and retrieval enhancement generation mechanism integration: introduce external knowledge base and real-time retrieval mechanism, enhance the accuracy and timeliness of the model's answer.

[0086] As Figure 5As shown, for the base model and pre-training, the present application is based on a self-developed multi-modal large language model (i.e., the initial preset language model in the present application), and the training process is divided into four stages: 1. The first stage is large-scale unsupervised pre-training (basic cognitive construction), the goal of which is to let the model learn the basic grammar, semantics and world knowledge of language from massive general and medical texts, and preliminarily establish the "macro-cognition" of the medical field. The data sources include massive texts, books, encyclopedias, news, etc. on the Internet, general data, and large-scale medical text corpus such as authoritative medical textbooks, academic journal papers, medical encyclopedias, unstructured electronic medical records, de-sensitization processed diagnosis and treatment records and operation reports, as well as the latest published disease diagnosis and treatment guidelines, expert consensus, supplemented by scraping doctor forum big data and patient community content. Through language modeling and knowledge extraction in this stage, the model can preliminarily identify diseases, symptoms, drugs and other entities and understand their basic relationships, and at the same time, through the customization of special vocabulary in the medical field (such as professional terms, drug names, gene sequences), a more efficient vocabulary table is constructed, significantly improving the coding efficiency and accuracy of the model. 2. The second stage carries out multi-modal self-supervised fusion and correlation learning, the goal of which is to let the model understand the internal relationship between different modal data such as text and image. Through cross-modal data alignment to construct large-scale paired data (such as text-image pairs composed of CT images (Computed Tomography, i.e. electronic computed tomography) and their corresponding imaging reports), and design innovative self-supervised learning tasks on the aligned data, including modal completion (predict the corresponding imaging report given the CT image) and cross-modal matching (determine whether the image and text match given the image and text). 3. The third stage implements instruction fine-tuning, RLHF (Reinforcement Learning with Human Feedback, reinforcement learning with human feedback) and knowledge enhancement, the goal of which is to let the model become a personified assistant that can "accurately, safely and responsibly" interact with users. First, through instruction fine-tuning, the model learns to follow instructions and generate answers that conform to specific medical roles and scenarios, then through human preference alignment to collect feedback from professional doctors to train the reward model, and finally through reinforcement learning to make the output of the large model more consistent with the medical professional preference. 4. The fourth stage integrates knowledge enhancement and RAG mechanism, by constructing a vector knowledge base and a structured knowledge graph and integrating them into the model inference process, when the patient asks a question, the model first retrieves related facts in the knowledge base, and then generates answers combined with its own knowledge, so as to ensure the accuracy and timeliness of the answers.

[0087] Step S20, obtaining pre-processed video stream data, and constructing a fall detection model, inputting the pre-processed video stream data into the fall detection model, and outputting a fall detection result. Wherein, the initial fall detection model comprises a double-branch parallel encoder, a multi-level feature fusion module and a behavior state classifier, and the double-branch parallel encoder comprises a skeleton point time series encoder and a three-dimensional convolution feature encoder.

[0088] The present application also provides an intelligent ward safety monitoring system.

[0089] Technical goal: Establish a comprehensive intelligent monitoring and early warning capability, realize real-time and automatic response to patient safety risks, and replace the lagging manual patrol mode.

[0090] Function description: 1. High-risk behavior identification and immediate alarm: It can identify dangerous conditions such as falls, violent struggles, sudden syncope, etc. The system can respond extremely quickly within 0.5 seconds, automatically trigger sound and light alarms, and push emergency notifications with video screenshots to the mobile terminals of medical staff, avoiding rescue delays. 2. Abnormal behavior warning: Identify high-risk behaviors such as long-term bedridden and abnormal wandering to prevent health risks caused by long-term bedridden or patient loss.

[0091] The present application also provides an intelligent sensing system based on computer vision and deep learning algorithm, which aims to comprehensively, real-time and non-contact monitor the public area and bed area of the ward. Through accurate identification and analysis of patient behavior, state and environmental information, the system upgrades the traditional passive monitoring mode relying on manual patrol to an automatic and active intelligent risk warning mechanism, thereby significantly improving the safety and nursing efficiency of the ward.

[0092] As Figure 6 shown, the capabilities and achievements of the intelligent sensing system based on computer vision and deep learning algorithm include:

[0093] Real-time dynamic monitoring and behavior identification: High-definition cameras and edge computing devices are used to continuously and uninterruptedly monitor the patient's daily behavior. Through deep learning algorithms, it can accurately identify and distinguish various behavior patterns such as normal walking, sitting, resting, etc., and potential risk behaviors such as falls, abnormal struggles, long-term stillness, etc. When high-risk behaviors are detected, the system can trigger an alarm in a very short time, converting the traditional lagging monitoring relying on manual patrol into immediate response and active intervention, thereby greatly shortening the response time and reducing the risk of patient accidents.

[0094] Specifically, real-time video stream data of a patient is acquired, and first preprocessing is performed on the real-time video stream data to obtain preprocessed video stream data, wherein the first preprocessing includes image denoising processing, target region cropping processing, and target region compression processing; a first preset data set is acquired, and behavior labeling is performed on the first preset data set to obtain a first labeled data set; the first labeled data set is input into the skeletal point time sequence encoder and the three-dimensional convolution feature encoder in the initial fall detection model; the first labeled data set is encoded by the skeletal point time sequence encoder to obtain first encoded features; the first labeled data set is subjected to motion information capturing and texture feature extraction processing by the three-dimensional convolution feature encoder to obtain second encoded features; the first encoded features and the second encoded features are input into the multi-level feature fusion module, and the first encoded features and the second encoded features are subjected to feature fusion by the multi-level feature fusion module to obtain fused features; the fused features are input into the behavior state classifier, and the fused features are subjected to second classification processing by the behavior state classifier to obtain a behavior analysis result, and the initial fall detection model is trained; the trained initial fall detection model is subjected to model optimization processing by using a reinforcement learning mechanism to obtain a fall detection model; and the preprocessed video stream data is input into the fall detection model, and the preprocessed video stream data is subjected to fall detection processing by the fall detection model to obtain a fall detection result.

[0095] For fall detection, the application adopts a behavior recognition algorithm based on skeletal point detection or a three-dimensional convolutional neural network. This model can learn the time sequence features of human posture and motion patterns. When the posture of a patient is quickly changed from standing or sitting to lying on the ground and there is no movement for a long time, the system will immediately trigger a fall event alarm.

[0096] As shown in Figure 6 , the specific implementation scheme of the intelligent perception system technology provided in the application is as follows:

[0097] Data acquisition and preprocessing: 1. High-definition video stream acquisition: high-definition cameras are deployed in public areas such as wards and corridors to acquire and transmit high-resolution video stream data (i.e., real-time video stream data of a patient in the application) in real time. 2. Edge computing preprocessing: preliminary processing (i.e., first preprocessing in the application) is performed on the video stream on the camera side or the nearest edge computing device, including image denoising, target region cropping and compression, to reduce the bandwidth and delay of data transmission and protect the privacy of patients.

[0098] Core algorithm and intelligent analysis: the application adopts a behavior recognition algorithm to construct a fall detection model (a deep learning model based on fusion of skeleton points and time sequence characteristics), and the fall detection model adopts a skeleton point and time sequence characteristic fusion architecture, which is specially used for processing complex light, shielding and multi-body position change problems in a ward monitoring scene.

[0099] The innovation points of the fall detection model structure include: 1. Double-branch parallel encoder: the model contains two parallel working encoder branches. Branch one (skeleton point time sequence encoder): a lightweight graph convolutional network is used to encode human skeleton point data. This branch can efficiently capture the spatial topological relationship between human joints, and use a gated recurrent unit or a memory network to model the skeleton point sequence in time sequence, so as to understand the dynamic changes of actions. Branch two (three-dimensional convolution feature encoder): a lightweight three-dimensional convolutional neural network is used, which directly processes the continuous frame sequence of the video stream, and is used to capture motion information and texture features, making up for the lack of skeleton point data in detail information. The advantage of this double-branch design is that it can comprehensively analyze the behavior of the patient from two different dimensions. 2. A multi-level feature fusion module is designed to dynamically fuse the features extracted by the two branches. This module can intelligently adjust the importance of the features of the two branches according to the quality of the input data and the environmental conditions (such as light changes or body shielding), ensuring that the model can make accurate judgments in various complex situations. 3. The fused features are sent to a behavior state classifier, which can output more detailed behavior states, not just “fall” or “non-fall”. This fine classification provides more accurate signals for subsequent millisecond-level alarms, ensuring the effectiveness and timeliness of the warning.

[0100] The training process of the fall detection model is as follows: 1. Large-scale, diversified private dataset construction: the application collects and annotates ward-specific video data containing normal behavior and abnormal fall, and the data source covers daytime, nighttime, different light conditions, different angles, and fall scenes simulated by actors with different body shapes (such as old people, children, patients). This solves the problem of lack of medical scene specificity in general datasets, ensuring the adaptability of the model to real ward environments. 2. Multi-task joint training: the fall detection model not only performs binary classification of fall events (judges whether to fall), but also simultaneously performs joint training of auxiliary tasks such as skeleton key point prediction and time sequence feature regression. This multi-task learning method can improve the understanding of human posture and motion patterns, further improving the accuracy of fall detection. 3. Reinforcement learning assisted optimization: the application uses a reinforcement learning mechanism to dynamically optimize the warning threshold of the model. The fall detection model will receive rewards or penalties according to the actual warning results (false positive rate, false negative rate), and adjust the internal parameters through a reinforcement learning algorithm, so that it can minimize false negatives while ensuring a low false positive rate, ultimately achieving the best warning performance.

[0101] Step S30, an abnormal behavior classification model is constructed, and the preprocessed video stream data is input into the abnormal behavior classification model to output an abnormal behavior classification result.

[0102] The application sets an intelligent driving mechanism for the nursing process, including:

[0103] Technical goal: change the traditional nursing mode relying on artificial memory into a programmed and proactive intelligent intervention to ensure the timely and accurate execution of key nursing tasks.

[0104] Function description: the system realizes real-time perception of the patient's state through embedded multi-modal sensors (such as vital sign acquisition sensors, bed terminal cameras) deployed in the ward and patient behavior data analysis. In the key period of medication, the risk period caused by physiological discomfort due to long-term maintenance of the same body position, and other key nodes requiring nursing tasks, the system can automatically trigger multi-modal reminders. The reminder method will be intelligently selected according to the patient's state, such as through voice broadcast and visual prompts for medication or examination reminders, or through tactile feedback for disturbance-free turning over reminders, thereby effectively eliminating human errors and improving nursing quality.

[0105] Specifically, the first preset data set is behavior labeled and decomposed labeled to obtain a second labeled data set; an initial abnormal behavior classification model is obtained, and the initial abnormal behavior classification model is trained according to the second labeled data set to obtain a first abnormal behavior classification model; an incremental learning mechanism is used to fine-tune the first abnormal behavior classification model to obtain an abnormal behavior classification model; and the preprocessed video stream data is input into the abnormal behavior classification model to output an abnormal behavior classification result.

[0106] As shown in Figure 6 For the classification of abnormal behaviors: the application deploys a behavior classification model for real-time analysis of the patient's behavior sequence. The model can distinguish between normal behaviors (such as walking, sitting, sleeping) and abnormal behaviors (such as violent struggle, abnormal wandering, and long-term stillness), and classify abnormal behaviors to provide a basis for subsequent risk assessment.

[0107] The abnormal behavior classification model in the application is an innovation based on a spatio-temporal graph convolution network for real-time analysis of the patient's fine-grained behavior sequence. Unlike traditional general behavior recognition models, the model of the application is optimized in network architecture and training strategy.

[0108] The structural innovation points of the behavior classification model are as follows: 1. Multi-granularity spatiotemporal feature extraction: the behavior classification model can capture the mutual relationship (such as the relative position between joints) of the human skeleton points in the spatial dimension and the continuous change (such as the action sequence) in the time dimension through multiple layers of spatiotemporal graph convolution layers. The present application designs a multi-granularity feature extraction module, which not only focuses on macroscopic whole body movement, but also focuses on microscopic local movement, such as hand movement and head rotation. 2. Abnormal behavior pattern library and feature matching: the present application constructs a feature library containing pre-defined abnormal behavior patterns such as "violent struggle", "long time stillness" and "abnormal wandering". When the behavior classification model analyzes the skeleton point time series data in real time, it will match the extracted features with the pattern library in real time. When the matching degree reaches the preset threshold, the system will trigger the corresponding abnormal warning, which ensures that the model can accurately identify and classify various subdivided abnormal behaviors. 3. Multimodal information fusion interface: the behavior classification model reserves a data fusion interface with the voice emotion analysis module. When the behavior classification model determines that the patient has an abnormal behavior (such as violent struggle), it will cross-verify the emotional information such as "pain" or "anxiety" provided by the voice module, so as to improve the accuracy of abnormal behavior determination and reduce false positives.

[0109] The training process of the behavior classification model is as follows: 1. Customized label and behavior decomposition: the present application performs fine labeling on the collected video data, not only labeling the behavior category, but also decomposing the behavior process (such as "start struggling" and "struggle peak period"). This fine labeling helps the model learn the internal time sequence structure of the behavior. 2. Long time sequence memory and prediction training: the behavior classification model is trained by an adversarial generative network or an autoregressive model, so that it can learn and predict the long-term behavior pattern of the patient. For example, the model can identify the abnormal behavior of "long time stillness", which is not based on instantaneous stillness, but based on the comprehensive analysis of the patient's behavior pattern in the past period of time. 3. Continuous learning and model iteration: considering the differences in behavior patterns of different patients and different diseases, the present application adopts an incremental learning mechanism. When the system finds new abnormal behavior patterns in actual application, it will add the data to the training set and fine-tune the model, so as to realize the continuous optimization and iteration of the model, and maintain its intelligence and adaptability.

[0110] Step S40, performing face key point recognition and emotion state classification processing on the preprocessed video stream data to obtain an emotion state evaluation result.

[0111] The present application also provides emotion state evaluation: through facial expression analysis and voice tone analysis (such as moaning and rapid breathing), the pain degree, anxiety, depression and other emotional states of the patient are automatically evaluated, so as to provide more comprehensive nursing basis for medical staff.

[0112] The application also provides identity security management: combined with face recognition and voiceprint verification technology, only authorized personnel such as medical staff and family members can enter the ward, effectively preventing unauthorized personnel from entering and exiting, and protecting patient privacy and ward safety.

[0113] As Figure 6 shown, for emotional and physiological state assessment: the system not only focuses on the macro behavior of the patient, but also analyzes the state in depth at the micro level. By capturing the key point changes of the patient's facial micro expressions, the system can automatically assess the patient's emotional state, such as anxiety, pain, depression, etc. This cross-modal emotional analysis method significantly improves the accuracy of the determination. In addition, by monitoring the patient's long-term bedridden or abnormal wandering behavior patterns, the system can also indirectly assess the patient's physiological state, providing medical staff with more comprehensive nursing basis.

[0114] As Figure 6 shown, for identity security verification and area management: in order to protect the safety of the ward and the privacy of the patient, the system integrates face recognition and voiceprint verification technology. By pre-recording the face and voiceprint data of authorized personnel (such as medical staff and family members), the system can monitor the entrance and exit of the ward in real time. When someone enters or tries to enter the ward, the system will conduct an instant identity verification to ensure that only authorized personnel can pass. For unauthorized strangers who break in, the system will immediately issue an alarm and notify the medical staff, effectively preventing unauthorized personnel from entering and exiting, and creating a safer and more private rehabilitation environment for patients.

[0115] For multi-level early warning and information linkage: the system establishes an intelligent early warning mechanism with hierarchical response. For high-risk conditions such as falls and violent struggles, the system can respond quickly within 0.5 seconds, automatically trigger sound and light alarms, and push emergency notifications with video screenshots and location information to the mobile terminals of medical staff. For emotional fluctuations and abnormal wandering, the system will generate real-time reminders and display them on the nurse station or bed terminal, prompting medical staff and family members to pay attention. In addition, all visual analysis results will be synchronized in real time to the medical Internet of Things data fusion hub as core data, together with other physiological parameters to build a complete health record for the patient, providing strong data support for medical big data analysis and decision-making.

[0116] Specifically, a face detection model is determined, and the pre-processed video stream data is subjected to second preprocessing and region positioning processing through the face detection model, to obtain a target face region, wherein the second preprocessing includes cropping processing and normalization processing; the target face region is subjected to face key point detection and key point position change tracking processing through a face key point detection algorithm and a multi-target tracking algorithm, to obtain a face key point motion sequence.

[0117] For expression and emotional state analysis: the application captures the key point changes of the patient's face through a deep learning algorithm, inputs into an emotion recognition model, uses the facial key point motion sequence as the main input, and introduces the emotional features (such as pitch, speech rate, and spectrum) and behavioral features (such as limb movement patterns) of the voice signal for joint modeling. The model uses an attention mechanism to dynamically weight the contribution of each modal feature, and realizes end-to-end emotional state classification (such as pain, anxiety, sadness, or depression) through a deep neural network structure. To further improve the reliability of the judgment, the model integrates the speech emotion analysis results output by the cross-modal semantic understanding module, cross- validates and calibrates the preliminary emotional judgment, and significantly improves the accuracy and robustness of the emotional state evaluation.

[0118] The emotional features and behavioral features of the patient are obtained, and the facial key point motion sequence, the emotional features, and the behavioral features are input into an emotion recognition model, and the facial key point motion sequence, the emotional features, and the behavioral features are processed by the emotion recognition model to obtain an emotional state evaluation result.

[0119] Among them, the acquisition process of the facial key point change sequence includes: 1, preprocessing and region positioning: through a lightweight face detection model, the region of the patient's face is accurately positioned in the continuous video frames, and cutting and normalization processing is performed, which can effectively avoid background interference and improve the efficiency of subsequent analysis. 2, through dynamic face key point detection, multiple key points of the face are detected, once the key points are detected, a multi-target tracking algorithm is used to continuously track the position changes of the key points in the continuous video frames. These key point coordinates are connected in the time dimension to form a high-dimensional time sequence feature sequence, which contains all the dynamic information of the patient's facial muscle activity in a period of time, and is the core input of emotional analysis.

[0120] For risk early warning and information output, the application sets up a multi-level early warning mechanism, including:

[0121] 1, immediate alarm: for high-risk behaviors such as falling and violent struggle, the system can respond extremely fast within 0.5 seconds, automatically trigger sound and light alarm and push emergency notification to the mobile terminal of medical staff, with video screenshot and alarm location, to avoid rescue delay. 2, real-time monitoring and reminding: for abnormal wandering and emotional fluctuation, the system will generate real-time reminders and display them on the nurse station digital human interface or the bed intelligent terminal, reminding medical staff and family members to pay attention to the patient's condition. 3, data integration and application: the visual analysis results (such as behavior type, emotional score, and alarm event) output by the system will be synchronized in real time to the medical Internet of Things data fusion hub as one of the core data sources, providing a basis for subsequent medical big data analysis and decision-making.

[0122] These data will be used to build a patient's behavior profile, combined with physiological parameters, electronic medical records, etc. Data analysis to achieve a more comprehensive patient status insight.

[0123] Step S50, the response result, the fall detection result, the abnormal behavior classification result and the emotional state evaluation result are standardized and risk warning processing is carried out, and the ward monitoring warning result is obtained.

[0124] As shown in Figure 7 The medical Internet of Things data fusion hub is provided in the application, which includes:

[0125] Technical objectives: Build a multi-source, real-time, secure data processing platform, and provide strong data support for upper intelligent applications.

[0126] Functional description, including: 1, multi-source data real-time synchronization and integration: the system can collect and integrate multi-element data from physiological data acquisition devices (such as heart rate, blood pressure, blood oxygen), multi-modal sensing system (behavior, emotion) and ward environment parameters in real time, and form a unified and complete patient health record. 2, intelligent risk warning and task reminder: based on the fused data, the system uses preset medical rules and AI models for continuous analysis. Once the data fluctuation exceeds the safe range or there is a potential health risk, the system will automatically push the warning. At the same time, the system can also send automatic task reminders to patients or medical staff at key time points according to the nursing plan, to ensure the accurate execution of the nursing process. 3, data support and application empowerment: the fused data can be used as the basis for upper multi-modal AI models and various applications, providing strong data support for patient health assessment, nursing plan optimization, virtual medical assistant service, etc. Form a complete intelligent closed loop.

[0127] The medical Internet of Things data fusion hub provided in the application is the core data processing platform of the intelligent ward system, which aims to collect various data from multi-modal sensing systems, bedside medical devices and patient intelligent terminals in real time and safely, and provide decision support for upper layer applications through intelligent analysis.

[0128] Specifically, the response result, the fall detection result, the abnormal behavior classification result and the emotional state evaluation result are standardized and thirdly preprocessed to obtain structured data; wherein the standardization processing includes data cleaning processing, format conversion processing and timestamp alignment processing, and the third preprocessing includes localized data cleaning processing and anomaly detection processing; a multi-modal time series prediction model is constructed, and the structured data is input into the multi-modal time series prediction model; a time series analysis algorithm is used to perform risk prediction processing on the structured data through the multi-modal time series prediction model to obtain a risk prediction result; a preset rule engine is determined, and the risk prediction result is prewarned according to the preset rule engine to obtain a ward monitoring early warning result.

[0129] For the unified management process of multi-source heterogeneous data, the following processes are set in the application in order to standardize and integrate data from different sources and formats: 1. Data acquisition and standardization layer: access the data stream of various medical devices (such as electrocardiogram monitors, ventilators, X-ray machines, CT scanners) and AI vision systems through a multi-protocol gateway, supporting multiple protocols and formats (such as numerical, event, and time series data). This layer is responsible for data cleaning, format conversion, and timestamp alignment, converting heterogeneous data into a unified standardized format to ensure data consistency and integrability. 2. Edge computing preprocessing layer: deploy edge computing nodes at the data source (such as AI cameras, intelligent bedside terminals) for localized data cleaning and anomaly detection. For example, after the AI camera recognizes a fall event, it directly generates a structured event data packet at the edge, rather than transmitting raw video streams, significantly reducing data transmission volume and system latency.

[0130] For real-time data synchronization and low-latency processing: to meet the needs of emergency warning and immediate decision-making, the system uses a streaming processing architecture, including: 1. Streaming data synchronization: structured data processed by edge devices is transmitted in real time to the cloud data fusion hub through a high-throughput message queue. This ensures that data can be quickly synchronized to the backend with seconds of delay, laying the foundation for subsequent real-time analysis. 2. Computing and event processing: the data hub processes data in real time after receiving the data stream. It can aggregate, correlate and pattern match data at an extremely fast speed.

[0131] For intelligent risk warning and task automation, the application uses fused data, combined with AI models and rule engines, to achieve intelligent risk warning and task automation, specifically including:

[0132] 1、AI risk prediction model: The invention constructs a multi-modal time series prediction model, which takes the patient's physiological parameters, behavior data, medication records, ward environment data, etc. as input features, and predicts the probability of the patient's risk of falling, cardiovascular accident, etc. in the future through deep learning or time series analysis algorithm. Once the prediction probability exceeds the preset threshold, the system will trigger an early warning. The model takes the patient's physiological parameters, behavior data, medication records, and ward environment monitoring data as input features, and through the fusion of time series convolution, recurrent neural network, and self-attention mechanism, etc. Deep learning methods extract multi-modal features and model their long-term dependencies, realizing the probability prediction of the patient's risk of falling, cardiovascular accident, etc. in the future. The model collects and synchronizes multiple data streams in real time, and after feature alignment and standardization, it inputs the trained prediction network for forward inference to calculate the real-time risk probability. When the output probability exceeds the decision threshold based on clinical experience, the system automatically triggers a hierarchical early warning mechanism.

[0133] 2、Intelligent rule engine: The system has a built-in configurable rule engine to execute preset care logic. For example, the rule can be defined as: "If the patient's blood oxygen saturation is less than 90% and the heart rate exceeds 120 times / min, send an "emergency warning" immediately." At the same time, the rule engine is also responsible for task automation, such as: "Send a "turn over reminder" task to the bed terminal at 9 am, 3 pm, and 9 pm every day."

[0134] 3、Multi-level early warning linkage: The system will push the warning information to different terminals according to the risk level (such as critical, high risk, medium risk) and data type. For example, for critical situations, emergency notifications will be sent to the head nurse and on-duty doctor through SMS and mobile terminal App; for general reminders, they will be displayed on the nurse station large screen or bed intelligent terminal.

[0135] For data security and privacy protection, data security is the core principle of the system, from data collection to storage, to use, strict protection measures are taken, including: 1, data desensitization and encrypted transmission: in data collection, the key information related to patient privacy (such as face data, name) will be desensitized locally in the edge device, or only the feature vector will be transmitted. All data in the transmission process, end-to-end encryption technology is adopted to ensure that data cannot be stolen and tampered in the transmission link. 2, access control and permission management: the system adopts role-based access control mechanism, and assigns different data access permissions to users with different roles (such as doctors, nurses, administrators, family members). For example, family members can only view the patient's routine health data, while doctors can access the complete electronic medical record and physiological data. 3, security audit and log tracking: the system will record all access and operation behaviors of patient data, and generate detailed audit logs. These logs can be used to track any abnormal data access or operation, ensuring data integrity and compliance.

[0136] Further, the present application also sets up a sickbed intelligent terminal and a multi-modal interaction system, aiming to provide a integrated and humanized intelligent interaction interface for patients, realizing real-time and personalized interaction with patients through embedded multi-modal perception and processing system, so as to directly deliver intelligent nursing service to the bedside, including: 1, one-stop information and service hub: integrating nursing reminders, medical order queries, remote calls and other functions in one terminal, simplifying patient operation and improving medical experience. 2, active and personalized reminders: based on nursing plans and physiological data, dynamically generating and pushing reminders for medication, turning over, examination, etc., transforming traditional passive nurse patrol into active intervention. 3, multi-modal interaction experience: supporting voice, touch, vision and other multi-modal interaction, especially providing convenient operation approach for elderly patients with limited mobility. 4, emotional accompaniment and psychological support: realizing emotional accompaniment through virtual digital people, alleviating patients' anxiety and loneliness, and providing more comprehensive humanistic care.

[0137] As shown in Figure 8 For hardware and software architecture, the present application sets up an embedded hardware platform, which adopts an embedded processor as the core computing unit, equipped with a high-definition touch display screen, not only as the main information display window, but also provides intuitive touch screen operation function. To support multi-modal interaction, the terminal integrates a high-fidelity microphone array for accurate pickup of patient voice instructions; at the same time, it is equipped with stereo speakers to provide clear voice broadcast and music playback.

[0138] In terms of visual perception, the terminal is equipped with a high-resolution camera for functions such as remote video calls, facial recognition, and micro-expression capture. In addition, to connect with medical Internet of Things devices, the terminal integrates various wireless communication modules, including Bluetooth, Wi-Fi, and Zigbee, to ensure real-time and stable connection with physiological data sensors such as smart mattresses and heart rate oximeters.

[0139] As shown in Figure 8 The application sets up a multi-modal perception and processing module, which specifically includes:

[0140] 1. Automatic Speech Recognition Engine: An ASR (high-precision speech recognition) engine optimized for medical scenarios is integrated. This engine can efficiently process patients' natural language, support all-weather voice wake-up functions (such as "Assistant, hello"), and convert voice commands into text. The engine performs partial processing on the embedded side to reduce latency and protect privacy.

[0141] 2. Text-to-Speech Engine: A highly anthropomorphic TTS (highly realistic speech synthesis) engine is used to convert system-generated text information (such as medical orders, nursing reminders, and chat content) into natural and smooth voice broadcasts. The TTS engine can automatically adjust the tone and speed according to the context, providing a more humanized interactive experience.

[0142] 3. Visual Interaction Module: Using embedded cameras and communication modules, the terminal supports high-definition, low-latency video calls between patients and medical staff or family members. Lightweight face recognition algorithms are deployed to verify the identity of medical staff or family members in specific scenarios, ensuring the safety of the ward. By capturing changes in key facial points, the terminal can preliminarily identify emotions, providing a reference for emotional chat and nursing staff.

[0143] 4. Sensor Data Fusion Module: This module serves as the data access hub, collecting physiological parameters (such as heart rate, blood oxygen saturation, and blood pressure) and patient status data from various sensors around the bed through Bluetooth and transmission protocols. These status data may include but are not limited to patient body position, posture, and body pressure distribution information, providing support for in-depth analysis and linkage. After preliminary cleaning and preprocessing locally, these data are uploaded to the medical Internet of Things data fusion hub for in-depth analysis and linkage.

[0144] 5. Software System: The software system of the terminal is based on an embedded operating system and designed with a simple and easy-to-use graphical user interface (UI).

[0145] 6. Nursing reminder application: This application can receive instructions from the data fusion hub and push reminders such as medication, turning over, examination, etc. to the patient in various forms such as voice, text or vibration. The patient can confirm the completion of the task through touch screen or voice.

[0146] 7. Service list application: Provides a one-stop service portal where patients can query medical orders, medical records, hospital bills, or initiate remote consultations, apply for nurse assistance, etc.

[0147] 8. Entertainment and companion application: Integrates music playback, audiobooks, and emotional chat functions driven by virtual digital people. Through natural language interaction, virtual digital people can provide psychological comfort and entertainment for patients, alleviating the sense of loneliness during hospitalization.

[0148] Technical effects of the present application:

[0149] 1. The present application provides a "hospital monitoring and active nursing system based on multi-modal artificial intelligence", which aims to fundamentally solve the key problems existing in traditional hospital nursing by integrating advanced multi-modal artificial intelligence, computer vision, medical Internet of Things and big data analysis technology, and building a comprehensive, real-time and intelligent hospital nursing and monitoring platform.

[0150] 2. The present application proposes four core technical solutions to address traditional pain points, supported by multiple collaborative technical modules. The four core technical solutions include:

[0151] Nursing process intelligent driving mechanism: This mechanism aims to transform the traditional nursing mode relying on human memory and experience into a programmed and proactive intelligent intervention process. It realizes real-time perception of patient status through embedded multi-modal sensors deployed in the ward and patient behavior data analysis. In critical periods of medication, physiological discomfort caused by long-term maintenance of the same body position, and other key nodes requiring nursing tasks (such as blood glucose monitoring, wound dressing), the system can automatically trigger multi-modal reminders (such as voice broadcast, visual cues, tactile feedback), effectively eliminating human errors and improving nursing quality.

[0152] Developing a patient self-service guide system: This solution is committed to building a one-stop intelligent guide service to enable patients to independently complete transactional links in the medical process, thereby significantly reducing the transactional burden of nursing staff. Based on multi-modal large models, the system builds a virtual medical assistant that provides medical order analysis, medication instructions, in-hospital precise positioning and navigation, and intelligent verification of medical insurance settlement documents, etc. functions under natural language interaction.

[0153] Intelligent Ward Safety Monitoring System: This system aims to establish comprehensive intelligent monitoring and early warning capabilities, enabling real-time and automated responses to patient safety risks, thereby replacing the traditional lagging manual patrol mode. It can instantly identify high-risk conditions such as falls, violent struggles, and sudden syncope, and trigger sound and light alarms and push emergency notifications to medical staff within a very short time (0.5 seconds). At the same time, the system can also provide abnormal behavior early warning, emotional state assessment, and identity security management, providing comprehensive protection for ward safety and patient privacy.

[0154] Medical Internet of Things Data Fusion Hub: As the core data processing platform of the entire system, this hub aims to build a multi-source, real-time, and secure data processing platform to provide strong data support for all intelligent applications at the upper level. It can collect and integrate multi-dimensional data from physiological data acquisition devices, multi-modal sensing systems, and ward environment parameters in real time, forming a unified and complete patient health record. Based on the fused data, the system uses pre-set medical rules and AI models for continuous analysis, enabling intelligent risk early warning and task reminders, and providing data support for patient health assessment, nursing plan optimization, and other services.

[0155] The strength of the present invention lies in the synergistic effect of its core technical modules, rather than simply stacking independent technologies. Through the "multi-modal large model", it achieves "cross-modal cognitive and semantic understanding", which means that the system can jointly analyze information from different data modalities (such as structured medical data, visual behavior, and voice emotion), thereby obtaining more comprehensive and accurate insights into the patient's state. This multi-modal deep understanding capability is the cornerstone of proactive nursing and personalized services. For example, when the computer vision system detects that a patient has a risk of falling, combined with the physiological data provided by the Internet of Things hub and the emotional changes captured by the virtual assistant, the system can make more accurate judgments and trigger more precise interventions. The interaction and verification between different modalities of information significantly improve the intelligence and reliability of the system.

[0156] In addition, the present invention has achieved a fundamental shift from "passive response" to "active prevention" in design philosophy. Traditional nursing modes often intervene after problems occur, while the present invention uses "intelligent risk prediction models" and "real-time monitoring of high-risk behaviors" to predict potential risks in advance and trigger early warnings or interventions at the problem germination stage. This forward-looking nursing mode can effectively reduce the incidence of medical accidents and improve the overall safety of patients.

[0157] The launch of "Ward monitoring and active nursing system based on multi-modal artificial intelligence" marks a fundamental shift from traditional passive response mode to intelligent and proactive prevention mode in the field of medical care. Through the deep integration of multi-modal artificial intelligence, computer vision, medical Internet of Things and big data analysis, the invention systematically solves the five core pain points in traditional ward nursing: lack of personalized assessment, inefficient doctor-patient communication, cumbersome hospitalization process, uneven allocation of nursing resources and data fragmentation.

[0158] The invention realizes all-weather, non-missing and accurate insight into the patient's condition. Through intelligent risk prediction model, it improves nursing from "after-treatment" to "early warning" and "active intervention", significantly improving patient safety. The introduction of anthropomorphic medical assistants and bed intelligent terminals not only greatly facilitates patients to obtain information and independently complete medical procedures, but also provides humanistic care and psychological support that traditional nursing cannot achieve through emotional companionship, and improves the patient's medical experience.

[0159] At the same time, the intelligent monitoring and task-driven mechanism effectively optimizes the allocation of nursing resources, reduces the transactional burden of medical staff, and enables them to devote more energy to professional nursing and personalized services, thereby improving the overall nursing efficiency and quality. More importantly, the medical Internet of Things data fusion hub built by the invention completely solves the problem of data fragmentation, converts massive heterogeneous data into actionable intelligent insights, and provides a solid data foundation for scientific decision-making and continuous optimization of nursing quality.

[0160] From a broader perspective, the invention is not only an independent intelligent nursing solution, but also a technology platform for future smart medical care. Its core multi-modal large model, powerful data fusion capability and advanced AI analysis engine make it highly scalable and adaptable. This means that the system can be continuously upgraded as medical technology and artificial intelligence continue to develop, and can easily integrate new medical devices, sensors and AI algorithms to address new challenges and demands in the future medical and health field. The invention lays a solid foundation for building a safer, more efficient and more humanized future hospital, and has important strategic significance and broad market prospects in promoting the development of smart medical care.

[0161] The innovation points of the invention are as follows:

[0162] I. Multi-modal cognitive intelligence engine:

[0163] 1、The application takes a multi-modal large model as the core and constructs a cross-modal semantic understanding framework dedicated to medical treatment. Through a unique four-stage training mechanism, including large-scale unsupervised pre-training (basic cognitive construction), multi-modal self-supervised fusion and correlation learning, instruction fine-tuning and RLHF (human feedback reinforcement learning), and integrated knowledge enhancement and RAG (retrieval augmented generation) mechanism, deep joint reasoning and efficient processing of structured medical record data (such as physiological parameters, medication records) and unstructured data (such as facial micro-expression, speech emotion tone) are realized. This ability enables it to comprehensively and accurately understand the patient's state, laying the foundation for realizing proactive care and personalized services.

[0164] 2、Cross-modal correlation method: The application focuses on protecting its self-supervised learning tasks optimized in medical scenarios, such as constructing large-scale paired data through CT images and their corresponding imaging reports, and designing modal completion (predicting the corresponding imaging report given the CT image) and cross-modal matching (judging whether the image and text match given the image and text) innovative tasks for training.

[0165] 3、Dynamic knowledge enhancement mechanism: The application protects the construction of medical knowledge graph and vector knowledge base in the RAG architecture, real-time linkage update strategy, and how the model retrieves the knowledge base and generates answers combining its own knowledge in the reasoning process to ensure the accuracy and timeliness of the answers.

[0166] II. Scene-based interaction system of anthropomorphic medical assistants

[0167] 1、Innovation essence: The application creates a sentiment-based virtual digital human system customized for medical scenarios. By integrating high-precision speech recognition (ASR), high-realistic speech synthesis (TTS), and three-dimensional image driving technologies, and combining style-driven image driving engines (such as using cartoonized images for pediatric wards, and anthropomorphic or realistic images for surgical or general wards) and multi-modal response linkage technology (ensuring synchronization of speech synthesis, expression generation, and body movements), it realizes natural and humanized interaction for medical order analysis, visit navigation, and emotional companionship services.

[0168] 2、Dynamic image adaptation method: Protects the real-time digital human image generation rule base and driving algorithm based on department type, patient age, and even patient emotional state (perceived by other modules).

[0169] 3、Medical dialogue control mechanism: Protects the medical order explanation dialogue optimization strategy combined with RLHF (human feedback reinforcement learning), enabling it to generate answers that conform to specific medical roles and scenarios, and more in line with medical professional preferences.

[0170] The application deeply integrates "emotional computing" into medical interaction, has a natural conversation with patients through emotional voice, expression and body language, and provides emotional accompaniment. This solves the problem that the low patient compliance and the ineffective alleviation of psychological pressure caused by the cold response of traditional robots, and significantly improves the humanization of interaction and the quality of service.

[0171] III. Millisecond-level safety monitoring system (hardware-algorithm fusion):

[0172] 1. The application provides a high-risk behavior recognition architecture based on edge computing optimization, which realizes 0.5 second level fast response (full link from recognition to automatic triggering of sound and light alarm and pushing of emergency notification) to dangerous conditions such as falling, violent struggle and sudden syncope by deploying high-definition cameras and deep learning algorithms (such as models based on skeleton point detection or three-dimensional convolutional neural network). At the same time, the system also has a hierarchical warning mechanism, which can identify long-term bed rest, abnormal wandering and other high-risk behaviors and give warnings.

[0173] 2. Localized preprocessing flow: a method for protecting the camera side or the nearest edge computing device to preliminarily process the video stream, including image denoising, target area cropping and compression, and generating structured "fall event" data packets locally, instead of transmitting raw video stream, to reduce data transmission volume and delay.

[0174] 3. Multi-level alarm triggering logic: a method for the protection system to automatically allocate alarm channels according to risk levels (such as critical, high risk and medium risk), for example, sending emergency notifications to medical staff through SMS and mobile terminal App for critical situations, and displaying on the nurse station large screen or bed intelligent terminal for general reminders.

[0175] The application realizes millisecond-level response speed (0.5 seconds), which effectively improves the efficiency compared with the average response time of traditional manual patrol. This ability of immediate active intervention can significantly reduce the incidence of medical accidents, and builds a high safety and efficiency barrier.

[0176] IV. Medical Internet of Things data fusion hub:

[0177] 1. Innovation essence: the medical Internet of Things data fusion hub is the core data processing platform of the intelligent ward system, which builds a real-time data lake architecture supporting multi-protocol heterogeneous device access (such as electrocardiogram monitor, breathing system, X-ray machine, CT scanner, AI vision system, environmental sensor, etc.). It cleanses, standardizes and timestamps aligns the data from different sources and formats through time alignment and unified data model (using graph database or multi-dimensional table structure), thereby building a dynamically updated patient health record.

[0178] 2. Streaming data synchronization technology: Protects structured data processed by edge devices by transmitting it to the cloud data fusion hub in real time through a high-throughput message queue. This ensures that data can be quickly synchronized to the backend with a latency of only seconds and can be aggregated, correlated, and matched at extremely fast speeds. For example, upon detecting "abnormal heart rate", it can immediately query whether there are any "abnormal behavior" events in the last 5 minutes and quickly trigger a linkage warning.

[0179] 3. Joint Analysis Model: Protects deep learning or time series analysis algorithms built on fused multimodal time series data (physiological parameters, behavioral data, medication records, ward environment data, etc.) to create intelligent risk prediction models for real-time prediction of potential health risks to patients (such as falls, cardiovascular accidents).

[0180] This invention solves the problem of traditional fragmented data by transforming massive amounts of heterogeneous data into a unified data source. Through a unified data model and real-time processing capabilities, the accuracy of risk prediction is effectively improved compared to single-source data models.

[0181] V. Embedded Multimodal System for Intelligent Terminals in Hospital Beds:

[0182] 1. Core Innovation: The embedded multimodal system provides patients with an integrated and user-friendly intelligent interactive interface. Its innovation lies in integrating medical-grade multimodal interaction hardware (such as high-fidelity microphone arrays and medical cameras) with local lightweight AI models (such as embedded ASR / TTS engines), enabling screenless interactive functions such as medication reminders and tactile feedback for turning over. It can proactively perceive patient needs, provide personalized services, and even engage in emotional interaction.

[0183] 2. Privacy-first architecture: Protects the method of local desensitization of critical information involving patient privacy (such as facial data and names) at the edge, or only transmitting their feature vectors, and measures to use end-to-end encryption technology (such as TLS / SSL) for all data during transmission.

[0184] 3. Multi-source sensor fusion interface: Protects the various wireless communication modules (including Bluetooth, Wi-Fi, Zigbee) integrated in the terminal and their plug-and-play protocols for medical devices, used to collect physiological parameters and patient status data in real time, and upload them to the medical IoT data fusion center after local preprocessing.

[0185] This invention provides a pure voice operation path for patients with limited mobility or disability, significantly reducing reliance on touch operation.

[0186] Furthermore, such as Figure 9As shown, based on the above-mentioned ward monitoring method based on multimodal large model and edge computing, the present invention also provides a ward monitoring system based on multimodal large model and edge computing, wherein the ward monitoring system based on multimodal large model and edge computing includes:

[0187] The response result output module 51 is used to acquire the user's query voice, construct a target large language model, input the user's query voice into the target large language model, and output the response result;

[0188] The fall result output module 52 is used to acquire preprocessed video stream data, construct a fall detection model, input the preprocessed video stream data into the fall detection model, and output the fall detection result;

[0189] The behavior result output module 53 is used to construct an abnormal behavior classification model, input the preprocessed video stream data into the abnormal behavior classification model, and output the abnormal behavior classification result.

[0190] The emotion result output module 54 is used to perform facial key point recognition and emotion state classification on the preprocessed video stream data to obtain the emotion state evaluation result.

[0191] The early warning result output module 55 is used to standardize and perform risk warning processing on the response result, the fall detection result, the abnormal behavior classification result, and the emotional state assessment result to obtain the ward monitoring early warning result.

[0192] Furthermore, such as Figure 10 As shown, based on the above-mentioned ward monitoring method and system based on multimodal large model and edge computing, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 10 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0193] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Further, the memory 20 can include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software installed on the terminal and various types of data, such as program codes of the installed terminal, etc. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores the ward monitoring program based on a multi-modal large model and edge computing 40, which can be executed by the processor 10 to implement the ward monitoring method based on a multi-modal large model and edge computing in the present application.

[0194] The processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes or process data stored in the memory 20, such as to execute the ward monitoring method based on a multi-modal large model and edge computing, etc.

[0195] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 30 is used to display information on the terminal and to display a visualized user interface.

[0196] In an embodiment, the steps of the ward monitoring method based on a multi-modal large model and edge computing are implemented when the processor 10 executes the ward monitoring program based on a multi-modal large model and edge computing 40 in the memory 20.

[0197] In summary, the application provides a ward monitoring method, system and terminal based on a multi-modal large model and edge computing, the method comprising: acquiring user query voice, and constructing a target large language model, inputting the user query voice into the target large language model, and outputting a response result; acquiring pre-processed video stream data, and constructing a fall detection model, inputting the pre-processed video stream data into the fall detection model, and outputting a fall detection result; constructing an abnormal behavior classification model, and inputting the pre-processed video stream data into the abnormal behavior classification model, and outputting an abnormal behavior classification result; performing face key point recognition and emotion state classification processing on the pre-processed video stream data to obtain an emotion state evaluation result; and performing standardization processing and risk early warning processing on the response result, the fall detection result, the abnormal behavior classification result and the emotion state evaluation result to obtain a ward monitoring early warning result. The application can effectively improve the timeliness and accuracy of ward monitoring by constructing a multi-modal large model to detect the state of a patient in real time.

[0198] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, such that processes, methods, articles, or terminals including a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such processes, methods, articles, or terminals. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or terminal including the element.

[0199] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware (such as a processor, a controller, etc.) to complete, and the program can be stored in a computer-readable computer-readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a magnetic disc, an optical disc, etc.

[0200] It should be understood that the application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all such improvements and changes should be within the scope of protection of the claims of the application.

Claims

1. A ward monitoring method based on a multi-modal large model and edge computing, characterized in that, The ward monitoring method based on the multimodal large model and edge calculation comprises: Obtaining a user query voice, constructing a target large language model, inputting the user query voice into the target large language model, and outputting a response result, comprising: Obtaining preset medical corpus data, and training an initial preset language model according to the preset medical corpus data to obtain a pre-training model; Obtaining multimodal data, and performing self-supervised learning processing on the pre-training model according to the multimodal data to obtain a multimodal base model; Obtaining preset demonstration data, and performing supervised fine-tuning processing on the multimodal base model according to the preset demonstration data to obtain a supervised fine-tuning model; Using knowledge enhancement and RAG mechanism to perform first classification processing and reinforcement learning processing on the supervised fine-tuning model to obtain a target large language model; When receiving a user query voice, inputting the user query voice into the target large language model, the target large language model performing semantic understanding on the user query voice, and outputting a response result; Obtaining preprocessed video stream data, and constructing a fall detection model, inputting the preprocessed video stream data into the fall detection model, and outputting a fall detection result, comprising: Obtaining real-time video stream data of a patient, and performing first preprocessing on the real-time video stream data to obtain preprocessed video stream data, wherein the first preprocessing comprises image denoising processing, target region cropping processing, and target region compression processing; Obtaining a first preset data set, and performing behavior annotation on the first preset data set to obtain a first annotated data set; Determining an initial fall detection model, and performing model training and model optimization on the initial fall detection model according to the first annotated data set to obtain a fall detection model; Inputting the preprocessed video stream data into the fall detection model, and performing fall detection processing on the preprocessed video stream data by the fall detection model to obtain a fall detection result; Constructing an abnormal behavior classification model, and inputting the preprocessed video stream data into the abnormal behavior classification model to output an abnormal behavior classification result; Performing face key point recognition and emotion state classification processing on the preprocessed video stream data to obtain an emotion state evaluation result; Performing standardization processing and risk warning processing on the response result, the fall detection result, the abnormal behavior classification result, and the emotion state evaluation result to obtain a ward monitoring warning result.

2. The ward monitoring method based on a multi-modal large model and edge calculation according to claim 1, characterized in that, The initial fall detection model comprises a double-branch parallel encoder, a multi-level feature fusion module, and a behavior state classifier, and the double-branch parallel encoder comprises a skeleton point time series encoder and a three-dimensional convolution feature encoder; The determination of the initial fall detection model and the model training and model optimization of the initial fall detection model according to the first annotated data set to obtain a fall detection model specifically comprises: Inputting the first annotated data set into the skeleton point time series encoder and the three-dimensional convolution feature encoder in the initial fall detection model respectively; Performing encoding processing on the first annotated data set by the skeleton point time series encoder to obtain first encoding features; The first encoding feature and the second encoding feature are input into the multi-level feature fusion module, and the first encoding feature and the second encoding feature are fused by the multi-level feature fusion module to obtain a fusion feature; The fusion feature is input into the behavior state classifier, and the fusion feature is subjected to second classification processing by the behavior state classifier to obtain a behavior analysis result, and the initial fall detection model training is completed; The trained initial fall detection model is subjected to model optimization processing by using a reinforcement learning mechanism to obtain a fall detection model. The abnormal behavior classification model is constructed, and the preprocessed video stream data is input into the abnormal behavior classification model to output an abnormal behavior classification result, specifically including: 3.The ward monitoring method based on multi-modal large model and edge computing according to claim 1, characterized in that, The first preset data set is subjected to behavior annotation and decomposition annotation to obtain a second annotation data set; An initial abnormal behavior classification model is obtained, and the initial abnormal behavior classification model is subjected to model training according to the second annotation data set to obtain a first abnormal behavior classification model; The first abnormal behavior classification model is subjected to model fine-tuning by using an incremental learning mechanism to obtain an abnormal behavior classification model; The preprocessed video stream data is input into the abnormal behavior classification model to output an abnormal behavior classification result. The preprocessed video stream data is subjected to face key point recognition and emotion state classification processing to obtain an emotion state evaluation result, specifically including: 4.The ward monitoring method based on multi-modal large model and edge computing according to claim 1, wherein, A face detection model is determined, and the preprocessed video stream data is subjected to second preprocessing and region positioning processing by the face detection model to obtain a target face region, wherein the second preprocessing includes cropping processing and normalization processing; The target face region is subjected to face key point detection and key point position change tracking processing by a face key point detection algorithm and a multi-target tracking algorithm to obtain a face key point motion sequence; Emotional features and behavioral features of a patient are obtained, and the face key point motion sequence, the emotional features and the behavioral features are input into an emotion recognition model, and the face key point motion sequence, the emotional features and the behavioral features are subjected to emotion state classification processing by the emotion recognition model to obtain an emotion state evaluation result. The response result, the fall detection result, the abnormal behavior classification result and the emotion state evaluation result are subjected to standardization processing and risk early warning processing to obtain a ward monitoring early warning result, specifically including: 5.The ward monitoring method based on multi-modal large model and edge computing according to claim 1, characterized in that, The response result, the fall detection result, the abnormal behavior classification result and the emotion state evaluation result are subjected to standardization processing and third preprocessing to obtain structured data; wherein the standardization processing includes data cleaning processing, format conversion processing and timestamp alignment processing, and the third preprocessing includes local data cleaning processing and abnormality detection processing; A multi-modal time series prediction model is constructed, and the structured data is input into the multi-modal time series prediction model; ​ Adopting a time series analysis algorithm, the multi-modal time series prediction model is used to perform risk prediction processing on the structured data, and a risk prediction result is obtained. A preset rule engine is determined, and the risk prediction result is processed for early warning according to the preset rule engine, and a ward monitoring early warning result is obtained.

6. A ward monitoring system based on a multi-modal large model and edge computing, characterized in that, The ward monitoring system based on the multi-modal large model and edge computing is applied to the ward monitoring method based on the multi-modal large model and edge computing in any one of claims 1-5, and the ward monitoring system based on the multi-modal large model and edge computing comprises: The response result output module is configured to obtain a user query voice, construct a target large language model, input the user query voice into the target large language model, and output a response result. The fall result output module is configured to obtain preprocessed video stream data, construct a fall detection model, input the preprocessed video stream data into the fall detection model, and output a fall detection result. The behavior result output module is configured to construct an abnormal behavior classification model, input the preprocessed video stream data into the abnormal behavior classification model, and output an abnormal behavior classification result. The emotion result output module is configured to perform face key point recognition and emotion state classification processing on the preprocessed video stream data, and obtain an emotion state evaluation result. The early warning result output module is configured to perform standardization processing and risk early warning processing on the response result, the fall detection result, the abnormal behavior classification result, and the emotion state evaluation result, and obtain a ward monitoring early warning result.

7. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a ward monitoring program based on a multi-modal large model and edge computing stored on the memory and executable on the processor, and the ward monitoring program based on the multi-modal large model and edge computing implements the steps of the ward monitoring method based on the multi-modal large model and edge computing in any one of claims 1-5 when executed by the processor.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a ward monitoring program based on a multi-modal large model and edge computing, and the ward monitoring program based on the multi-modal large model and edge computing implements the steps of the ward monitoring method based on the multi-modal large model and edge computing in any one of claims 1-5 when executed by the processor.

Images