A multi-modal large model-based nursing quality control method and system

Abstract

The application discloses a kind of based on multimodal big model nursing quality control method and system, for existing nursing quality control can only basic rule check, cannot adapt complex clinical scene, AI scheme has not solved the problem of multi-source data fusion, quality control dimension single, lack of logical rectification, the application is collected multimodal data by HIS system, electronic medical record, vital signs equipment etc., after standardization fusion, feature extraction, utilize the multimodal big model containing multiple encoder, cross-modal alignment module and clinical knowledge graph, complete coding, alignment and clinical reasoning;Rely on quality control decision matrix to output quality control problem, generate score by weight dynamic calculation model, execute real-time early warning, automatic scoring and rectification push, form logical quality control.The application realizes multimodal data fusion and intelligent quality control, improves nursing quality control accuracy, comprehensiveness and rectification landing.

Description

Technical Field

[0001] This invention relates to the fields of smart healthcare and nursing quality control technology, specifically to a nursing quality control method and system based on a multimodal large model. Background Technology

[0002] Nursing quality is a primary guarantee of medical safety and service level. Precise and efficient nursing quality control is crucial for reducing medical risks and standardizing nursing practices. Traditional nursing quality control relies on manual sampling and review, which has shortcomings such as low efficiency, strong subjectivity, incomplete coverage, and easy omission of hidden problems. It is difficult to adapt to the refined and real-time quality control needs of modern clinical nursing.

[0003] Existing intelligent nursing quality control technologies have significant shortcomings. First, based on fixed basic rules, they can only identify simple format-related issues, with limited rule coverage. This makes them unable to handle complex clinical nursing scenarios and struggles to accurately identify non-standardized, hidden quality control problems such as missing nursing procedures and omitted risk assessments, resulting in poor adaptability. Second, while AI technology has been introduced to upgrade quality control, it is limited to processing single-source text data and fails to address the challenge of multi-source data fusion. It cannot integrate multimodal information such as nursing records, vital sign time-series data, and nursing operation images / videos, resulting in a single quality control dimension and an inability to comprehensively reconstruct the entire nursing process, leading to biased and inaccurate quality control results. Furthermore, existing technologies lack deep clinical semantic understanding capabilities, failing to accurately parse semantic issues in nursing records such as errors in professional terminology, temporal logic conflicts, and inconsistent medical orders. Moreover, they lack a logical rectification mechanism, only able to identify quality control defects but unable to generate targeted rectification suggestions, resulting in poor implementation of quality control rectification measures.

[0004] In summary, existing nursing quality control technologies suffer from problems such as weak scenario adaptability, insufficient integration of multi-source data, low level of intelligence, and illogical rectification. There is an urgent need for a nursing quality control solution that integrates a multimodal large model to address the above-mentioned technical pain points and improve the comprehensiveness, accuracy, and efficiency of nursing quality control. Summary of the Invention

[0005] To address this, the present invention provides a nursing quality control method and system based on a multimodal large model, which solves the problems that existing nursing quality control can only verify basic rules and cannot adapt to complex clinical scenarios; the introduction of AI has not solved the problem of multi-source data fusion, the quality control dimension is single, lacks logical rectification, and has insufficient intelligence and accuracy.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a nursing quality control method based on a multimodal large model, comprising:

[0007] S1. Collect multimodal data related to patient care through HIS system, electronic medical records, nursing records, vital signs equipment, mobile nursing PDA and monitoring video;

[0008] S2. Extract data from the multimodal data, generate JSON data in a structured manner, perform standardization processing and fusion;

[0009] S3. Extract features from the fused multimodal data to obtain text semantic features, temporal change features, and image behavior features;

[0010] S4. Encode the extracted features using a multimodal large model, perform cross-modal alignment, and apply clinical knowledge reasoning.

[0011] S5. Output quality control questions based on the multimodal quality control decision matrix, calculate nursing quality scores using a weighted dynamic calculation model, and generate quality control decisions;

[0012] S6. Based on the quality control decision, perform real-time early warning, automatic scoring, and push rectification suggestions; the rectification suggestions are generated by matching standard problem codes with the built-in knowledge base to generate fixed rectification solutions, and rectification suggestions for non-standard problems are generated by the multimodal large model.

[0013] As a preferred embodiment of the nursing quality control method based on a multimodal large model, in S1, the multimodal data includes text data, time-series data, image data, and structured medical data.

[0014] As a preferred embodiment of the nursing quality control method based on a multimodal large model, in S4, the multimodal large model includes a text encoder, a time encoder, an image encoder, a cross-modal alignment module, and a clinical knowledge graph.

[0015] The text encoder processes text data, the time encoder processes time-series data, the image encoder processes video / image data, and the cross-modal alignment module realizes multimodal feature alignment and performs quality control analysis in conjunction with the clinical knowledge graph.

[0016] The cross-modal alignment module employs a multi-head cross-modal attention mechanism to achieve multi-modal feature alignment, and its mathematical expression is as follows:

[0017]

[0018] In the formula, Q is the text encoding vector output by the text encoder, with dimension 1. , 768 is the text encoding dimension. The maximum length of the text sequence;

[0019] For key vectors, The value vectors are obtained by dimensionality transformation of the temporal encoded vector output by the temporal encoder and the image encoded vector output by the image encoder, respectively. The transformed dimensions are all... , 64 represents the single attention head dimension. The number of features;

[0020] The time-series coding vector is of dimension 1. , , The time step is [time step]; the image encoding vector is [dimension] . , , The feature map size;

[0021] This is a scaling factor used to prevent the gradient of the Softmax function from vanishing due to an excessively large dot product result.

[0022] Softmax is a normalization function that outputs the attention weight distribution of each modality feature, with the weights summing to 1.

[0023] As a preferred option for a multimodal large-scale nursing quality control method, in S4, the quality control issues include errors in professional terminology, missing nursing procedures, omissions in risk assessment, conflicts in time logic, and discrepancies in medical orders.

[0024] As a preferred embodiment of the multimodal large-scale nursing quality control method, in S5, the quality control analysis process of the multimodal quality control decision matrix includes:

[0025] A preliminary verification of the timeliness and frequency of nursing procedures was conducted.

[0026] Deep semantic analysis of the semantic integrity and logical consistency of nursing records;

[0027] Perform behavioral recognition on nursing operation videos to verify the standardization of operations;

[0028] By combining the clinical knowledge graph, deviations between the nursing procedures and the diagnostic and treatment guidelines can be identified;

[0029] The multimodal quality control decision matrix adopts a four-level weighted decision model, and the final decision score calculation formula for a single candidate quality control problem is as follows:

[0030]

[0031] In the formula, The final decision score is determined by the following criteria: a score ≥ 0.7 indicates a genuine quality control issue; a score ≤ 0.5 and < 0.7 indicates a suspected issue requiring manual review; and a score < 0.5 indicates a false alarm.

[0032] , , , The weight coefficients for each layer of verification are 0.2, 0.3, 0.25, and 0.25 respectively, with the weight sum being 1.

[0033] The basic verification score is calculated using the following formula:

[0034]

[0035] In the formula, This represents the deviation between the actual operating time and the standard time. This is the standard duration of the operation. For missing operation counts, This represents the total number of operations to be performed.

[0036] For semantic analysis scores, the semantic similarity between nursing records and standard records is calculated using the medical BERT similarity model, with values ​​ranging from [0,1].

[0037] The behavior recognition score is the operational standardization score output by the SlowFast video behavior recognition model, with a value range of [0,1].

[0038] The knowledge graph matching score is calculated from the matching degree between nursing operations and rules, and the value ranges from [0,1].

[0039] As a preferred embodiment of the multimodal large-scale model-based nursing quality control method, the nursing quality score calculation formula is as follows:

[0040] Nursing quality score = 100 - Σ(defect item weight × severity coefficient)

[0041] In the formula, the weight of timeliness defect is 1.2 and the severity coefficient is 1.0–1.5; the weight of terminology defect is 0.8 and the severity coefficient is 1.0; the weight of single veto item is infinite, and it is automatically judged as Class C.

[0042] As a preferred option for a multimodal large-scale nursing quality control method, in S6, when a nursing defect is identified, a real-time early warning instruction is generated and pushed to the nurse's terminal; when the nursing defect is not identified, the quality control is marked as passed and a quality control report is generated.

[0043] As a preferred scheme of the nursing quality control method based on multimodal large model, in S6, the nurse terminal receives the early warning information and performs rectification operation. After submitting the rectification record, the quality control engine automatically verifies the rectification record. If the verification is successful, the quality control is marked as successful and a quality control report is generated. If the verification fails, the real-time early warning instruction is generated again.

[0044] As a preferred option for nursing quality control methods based on multimodal large models, in S6, the rectification suggestions are generated based on the type of quality control problem and combined with the clinical knowledge graph, including targeted suggestions such as operating procedures and rectification steps, and are pushed to the nurse terminal simultaneously with the early warning information.

[0045] This invention also provides a nursing quality control system based on a multimodal large model, employing the aforementioned nursing quality control method based on a multimodal large model, comprising:

[0046] Data source unit: used to collect multimodal data related to patient care through HIS system, electronic medical record, nursing record, vital signs device, mobile nursing PDA and monitoring video;

[0047] Multimodal fusion unit: used to extract data from the multimodal data, generate JSON data in a structured manner, perform standardization processing and fusion;

[0048] Feature extraction engine unit: used to extract features from the fused multimodal data to obtain text semantic features, temporal variation features and image behavior features;

[0049] Multimodal large model unit: used for encoding extracted features, cross-modal alignment, and clinical knowledge reasoning through multimodal large models;

[0050] Quality Control Decision Center Unit: Used to output quality control questions based on the multimodal quality control decision matrix, calculate nursing quality scores using a weighted dynamic calculation model, and generate quality control decisions.

[0051] Application Unit: Used to perform real-time early warning, automatic scoring, and rectification suggestion push operations based on the quality control decision; the rectification suggestions are generated by matching standard problem codes with the built-in knowledge base to generate fixed rectification solutions, and rectification suggestions for non-standard problems are generated by the multimodal large model.

[0052] The present invention has the following advantages:

[0053] First, this invention achieves multi-source, multi-modal data fusion, integrating text, time series, image, and structured medical data, breaking through the limitations of single data processing in existing technologies, adapting to complex clinical nursing scenarios, and providing more comprehensive quality control coverage.

[0054] Secondly, this invention relies on a multimodal large model and clinical knowledge graph, possessing deep semantic understanding and intelligent reasoning capabilities, accurately identifying five types of hidden quality control problems: errors in professional terminology, missing procedures, omissions in risk assessment, time conflicts, and discrepancies in medical orders, thus significantly improving the accuracy of quality control.

[0055] Third, this invention uses a weighted dynamic calculation model to quantify nursing quality scores, distinguish the severity of defect types and set single-item veto rules, and the scoring standards are objective and unified, so as to achieve standardized and quantifiable evaluation of quality control results.

[0056] Fourth, this invention constructs a real-time early warning and logical rectification mechanism, quickly pushes early warning of quality control defects, generates standardized and intelligent rectification suggestions and automatically verifies the rectification effect, forming a complete quality control logic, and the rectification is highly implementable.

[0057] Fifth, this invention replaces the traditional manual quality control model, realizing automated and real-time processing of the entire nursing quality control process, solving the problems of low efficiency, strong subjectivity, and easy omissions in manual review, and greatly improving the efficiency and standardization of nursing quality control. Attached Figure Description

[0058] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0059] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0060] Figure 1 This is a schematic diagram of a nursing quality control method based on a multimodal large model provided in Embodiment 1 of the present invention;

[0061] Figure 2 This is a technical roadmap for a nursing quality control method based on a multimodal large model provided in Embodiment 1 of the present invention;

[0062] Figure 3 This is a data source architecture diagram of a nursing quality control method based on a multimodal large model provided in Embodiment 1 of the present invention;

[0063] Figure 4 This is a diagram illustrating the internal architecture of a multimodal large-scale nursing quality control method provided in Embodiment 1 of the present invention.

[0064] Figure 5 This is a flowchart of a quality control process for a nursing quality control method based on a multimodal large model, as provided in Embodiment 1 of the present invention.

[0065] Figure 6 This is a schematic diagram of a nursing quality control system architecture based on a multimodal large model provided in Embodiment 1 of the present invention. Detailed Implementation

[0066] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Example 1

[0068] See Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 This invention provides a nursing quality control method based on a multimodal large model, comprising:

[0069] S1. Collect multimodal data related to patient care through HIS system, electronic medical records, nursing records, vital signs equipment, mobile nursing PDA and monitoring video;

[0070] Specifically, nursing data is distributed across hospital information systems, medical records, nursing operation terminals, and on-site monitoring equipment. The HIS system provides structured diagnostic and treatment data, electronic medical records and nursing records carry text-based operation records, vital sign devices output time-series physiological data, mobile nursing PDAs record bedside operation trajectories, and monitoring videos preserve nursing operation footage. The joint collection of multiple data sources can cover data from all nursing scenarios and all stages, avoiding information gaps caused by a single data source and laying a data foundation for subsequent comprehensive quality control.

[0071] S2. Extract data from the multimodal data, generate JSON data in a structured manner, perform standardization processing and fusion;

[0072] Specifically, raw multimodal data suffers from problems such as heterogeneous formats, inconsistent fields, data redundancy, and noise interference. Standardization processing generates JSON data through data cleaning, field mapping, format unification, and structure transformation, eliminating data differences. Fusion maps different modal data to a unified feature space, realizing the association and integration of text, time series, and image data, breaking down modal barriers, and forming a complete nursing data representation.

[0073] S3. Extract features from the fused multimodal data to obtain text semantic features, temporal change features, and image behavior features;

[0074] Specifically, the original data after fusion has high dimensionality and a lot of redundant information, making direct processing inefficient and susceptible to interference. Feature extraction uses a feature extraction engine to filter and extract textual semantic features, temporal change features, and image behavioral features from the fused data, eliminating invalid and redundant information, retaining features strongly related to nursing quality control, reducing data processing complexity, and improving the efficiency of subsequent model analysis.

[0075] S4. Encode the extracted features using a multimodal large model, perform cross-modal alignment, and apply clinical knowledge reasoning.

[0076] Specifically, the extracted multimodal features have significant differences in dimensions and representation methods. The encoding operation converts each modality feature into a unified low-dimensional vector through the corresponding encoder, achieving feature dimensionality reduction and standardization. Cross-modal alignment eliminates semantic deviations and dimensional differences between features of different modalities, achieving deep feature fusion. Clinical knowledge reasoning relies on the built-in clinical knowledge graph to associate the fused features with nursing standards, treatment guidelines, and medical order rules, giving the model clinical professional judgment capabilities and achieving intelligent quality control analysis.

[0077] S5. Output quality control questions based on the multimodal quality control decision matrix, calculate nursing quality scores using a weighted dynamic calculation model, and generate quality control decisions;

[0078] Specifically, the multimodal quality control decision matrix pre-sets multi-level quality control verification rules, integrates basic rules, semantic rules, behavioral rules and knowledge graph rules, performs hierarchical verification of model inference results, accurately matches and outputs standardized quality control issues; the weight dynamic calculation model quantifies the severity of different defects, quantifies the impact of quality control defects through weighted summation, transforms abstract quality control results into quantifiable scores, and finally combines the problem type and score results to form accurate quality control decisions.

[0079] S6. Based on the quality control decision, perform real-time early warning, automatic scoring, and push rectification suggestions; the rectification suggestions are generated by matching standard problem codes with the built-in knowledge base to generate fixed rectification solutions, and rectification suggestions for non-standard problems are generated by the multimodal large model.

[0080] Specifically, quality control decisions include defect identification results, quantitative scoring, and risk levels. Real-time alerts quickly reach nursing terminals for high-risk defects, preventing the risk from escalating. Automatic scoring simultaneously archives quantitative results, forming a quantitative nursing quality file. Corrective action suggestions are pushed out to match the defect type and output targeted solutions. These three operations work together to achieve full-process quality control of "problem identification, risk warning, result quantification, and corrective action guidance," adapting to real-time clinical quality control needs. Corrective action suggestions adopt a hybrid generation mode of "knowledge base and large model": for standard problems, such as terminology errors, standardized corrective action plans are generated through knowledge base matching to ensure compliance; for non-standard problems, such as missing complex processes, personalized corrective action suggestions are generated through the large model to improve the targeting and operability of the corrective action.

[0081] In one possible embodiment, in S1, the multimodal data includes text data, time-series data, image data, and structured medical data.

[0082] Specifically, text data corresponds to unstructured textual information such as nursing records, medical orders, and medical record descriptions, carrying details of nursing operations; time-series data corresponds to continuous monitoring data of vital signs such as body temperature, blood pressure, and heart rate, reflecting changes in the patient's physiological state; image data corresponds to visual information such as nursing operation monitoring video frames and wound images, reconstructing the nursing operation scene; structured medical data corresponds to standardized data such as patient basic information, diagnosis and treatment items, and medication records in the HIS system. These four types of data cover all dimensions of nursing information and are indispensable, ensuring the comprehensiveness and completeness of quality control analysis.

[0083] The data is extracted and structured as follows:

[0084] Table 1: Structured Data

[0085] First-level node name Second-level node name Frequency of occurrence definition General situation Patient Name 1..1 The patient's surname and given name officially registered with the public security household registration department. gender 1..1 The patient's biological sex age 1..1 Patient survival time Ward Name 0..1 Name of the patient's current ward Department Name 0..1 The name of the department where the patient sought medical treatment. Ward number 0..1 The ward number corresponding to the patient's hospitalization Bed number 0..1 The bed number corresponding to the patient's hospitalization Hospital number 0..1 A sequence number assigned to hospitalized patients according to a specific coding rule. Medical record number 0..1 Inpatient medical records are assigned a sequence number according to a specific coding rule. Recording time 1..1 A complete description of the Gregorian calendar date recorded on the nursing system record sheet. Nursing report code 1..1 Nursing report code of nursing system record sheet Nursing report name 1..1 Nursing report name in the nursing system record sheet Nursing report content 1..n Specific information in the nursing report Nursing item number 0..n Nursing entry number in the nursing report Nursing item coding 0..n Nursing item codes in nursing reports Nursing Project Name 1..n Nursing item names in the nursing report Nursing item value 1..n Nursing item values ​​in nursing reports Measurement time 0..n Nursing report nursing item values ​​measurement time Operation time 0..n Nursing report nursing item values ​​and operation time

[0086] In one possible embodiment, in S4, the multimodal large model includes a text encoder, a time encoder, an image encoder, a cross-modal alignment module, and a clinical knowledge graph; the text encoder processes text data, the time encoder processes time-series data, the image encoder processes video / image data, and the cross-modal alignment module realizes multimodal feature alignment and performs quality control analysis in conjunction with the clinical knowledge graph;

[0087] The cross-modal alignment module employs a multi-head cross-modal attention mechanism to achieve multi-modal feature alignment, and its mathematical expression is as follows:

[0088]

[0089] In the formula, Q is the text encoding vector output by the text encoder, with dimension 1. , 768 is the text encoding dimension. The maximum length of the text sequence;

[0090] For key vectors, The value vectors are obtained by dimensionality transformation of the temporal encoded vector output by the temporal encoder and the image encoded vector output by the image encoder, respectively. The transformed dimensions are all... , 64 represents the single attention head dimension. The number of features;

[0091] The time-series coding vector is of dimension 1. , , The time step is [time step]; the image encoding vector is [dimension] . , , The feature map size;

[0092] This is a scaling factor used to prevent the gradient of the Softmax function from vanishing due to an excessively large dot product result.

[0093] Softmax is a normalization function that outputs the attention weight distribution of each modality feature, with the weights summing to 1.

[0094] Specifically, the multimodal large model adopts an architecture design of "modal encoding, cross-modal fusion, and knowledge enhancement." The text encoder uses a medical pre-trained BERT-base model, which has been pre-trained on massive amounts of medical text and possesses professional medical semantic understanding capabilities. The time encoder uses a Temporal-BERT model, specifically adapted to the continuous and trend features of time-series data. The image encoder uses a CLIP-ViT-B / 32 model, which has powerful visual-language alignment capabilities and can establish associations between operational actions and text descriptions. The multi-head cross-modal attention mechanism calculates the dot product similarity between text features and temporal and image features to obtain attention weights between features of different modalities. The higher the weight, the stronger the semantic association between the two features. Through attention-weighted fusion, semantic alignment of features of different modalities is achieved, solving the problem of the same nursing event being represented differently in different modalities. Authoritative norms, as well as professional knowledge such as disease nursing routines, risk assessment standards, and medical orders and contraindications, are stored to provide clinical rule constraints for model inference and avoid the model generating quality control conclusions that do not conform to clinical reality.

[0095] In one possible embodiment, in S4, the quality control issues include errors in technical terminology, missing nursing procedures, omissions in risk assessment, time-related logical conflicts, and inconsistent medical orders.

[0096] Specifically, the system addresses five key areas of nursing quality control: 1) **Professional terminology errors:** Focusing on the standardization of nursing record writing to avoid misunderstandings caused by non-standard expressions; 2) **Missing nursing procedures:** Comparing nursing procedures with clinical nursing guidelines to identify omissions in operational steps and ensure compliance; 3) **Risk assessment omissions:** Addressing common nursing risks such as pressure ulcers, falls, and bleeding by identifying any failures to conduct risk screenings as required; 4) **Time logic conflicts:** Verifying the consistency of record time, operation time, and doctor's order execution time to eliminate temporal inconsistencies; 5) **Doctor's order discrepancies:** Identifying deviations between nursing procedures and doctor's orders to ensure compliance in treatment execution. These five categories of issues cover the main pain points in nursing quality control, accurately matching actual clinical quality control needs and resolving issues of missed detections and incompleteness in existing technologies.

[0097] See the following case: Analyze the quality control issues in the following nursing records, including:

[0098] 1. Incorrect use of technical terms

[0099] 2. Nursing procedures are missing (as required by [XX Guidelines]).

[0100] 3. Omissions in risk assessment (pressure sores / falls, etc.)

[0101] 4. Time logic conflict (recording time vs. operation time)

[0102] 5. Inconsistent with doctor's orders

[0103] Output only critical issues; leave blank if no issue type is specified.

[0104] Output in JSON format:

[0105] {

[0106] "Terminology Error": "Error Location and Correction Suggestions",

[0107] "Missing Process": "Explanation of Missing Steps"

[0108] "Risk assessment omissions": "Items that should have been assessed but were not".

[0109] "Time Conflict": "Explanation of Conflicting Time Points",

[0110] "Inconsistent medical orders": "Contradictory medical order entries"

[0111] }

[0112] example:

[0113] Input data:

[0114] {

[0115] Request ID:"NUR20250717001",

[0116] Nursing Record: "The patient complained of abdominal pain and was given ibuprofen 0.3g orally. The pain did not subside after 30 minutes; no further treatment was given."

[0117] "Nursing type": "Symptom management"

[0118] Patient Information:{"Gender":"Female","Age":"65","Diagnosis":"Peptic Ulcer"}

[0119] "Medical Order Association": "Disable NSAIDs",

[0120] "Key Time Points": ["Recorded at 14:00 on 2025-07-17", "Administered at 14:05"]

[0121] }

[0122] Large model output:

[0123] {

[0124] "Terminology error": "Ibuprofen should be labeled with its chemical name ibuprofen".

[0125] "Incomplete procedures": "Pain assessment not conducted; vital signs not recorded."

[0126] "Risk assessment omission": "The risk of gastrointestinal bleeding was not assessed".

[0127] "Time conflict": "Administration time was earlier than recording time".

[0128] "Inconsistent Medical Order": "Violation of the medical order to 'disable NSAIDs'"

[0129] }

[0130] In one possible embodiment, in S5, the quality control analysis process of the multimodal quality control decision matrix includes: preliminary verification of the timeliness and frequency of nursing operations; in-depth semantic analysis of the semantic integrity and logical consistency of nursing records; behavioral recognition of nursing operation videos to verify the standardization of operations; and identification of deviations between the nursing operations and the treatment guidelines by combining the clinical knowledge graph.

[0131] The multimodal quality control decision matrix adopts a four-level weighted decision model, and the final decision score calculation formula for a single candidate quality control problem is as follows:

[0132]

[0133] In the formula, The final decision score is determined by the following criteria: a score ≥ 0.7 indicates a genuine quality control issue; a score ≤ 0.5 and < 0.7 indicates a suspected issue requiring manual review; and a score < 0.5 indicates a false alarm.

[0134] , , , The weight coefficients for each layer of verification are 0.2, 0.3, 0.25, and 0.25 respectively, with the weight sum being 1.

[0135] The basic verification score is calculated using the following formula:

[0136]

[0137] In the formula, This represents the deviation between the actual operating time and the standard time. This is the standard duration of the operation. For missing operation counts, This represents the total number of operations to be performed.

[0138] For semantic analysis scores, the semantic similarity between nursing records and standard records is calculated using the medical BERT similarity model, with values ​​ranging from [0,1].

[0139] The behavior recognition score is the operational standardization score output by the SlowFast video behavior recognition model, with a value range of [0,1].

[0140] The knowledge graph matching score is calculated from the matching degree between nursing operations and rules, and the value ranges from [0,1].

[0141] Specifically, the first layer of basic verification uses rules to quickly handle explicit issues such as timeliness and frequency, with low computational complexity, achieving a response time in seconds and filtering out over 80% of simple problems. The second layer, semantic analysis, uses a pre-trained language model to parse the deep semantics of nursing records, identifying implicit problems such as ambiguous expressions, logical contradictions, and missing content. The third layer, behavioral recognition, uses a computer vision model to analyze operation videos, verifying whether the operation actions conform to standards, addressing the shortcomings of traditional quality control that "only look at the record and not the operation." The fourth layer, knowledge verification, uses a clinical knowledge graph to align the operation with authoritative standards, ensuring the professionalism and compliance of quality control conclusions. The four-layer weighted decision model balances the impact of each layer's verification results by setting different weights. Semantic analysis has the highest weight because the semantic accuracy of nursing records directly affects the transmission of medical information; behavioral recognition and knowledge verification have the next highest weights, corresponding to operational standardization and professional compliance; basic verification has the lowest weight because timeliness issues are relatively easy to rectify.

[0142] Specifically, the quality control analysis adopts a "from shallow to deep, layered and progressive" logic. Initial verification focuses on the compliance of basic operations, quickly screening for explicit issues such as exceeding time limits and insufficient frequency, thus improving the efficiency of basic quality control. Deep semantic analysis addresses implicit issues in text records, analyzing the operational logic behind the text and identifying problems such as ambiguous expressions and logical contradictions. Behavioral recognition relies on image features to reconstruct the nursing operation scene and verify whether the operational actions conform to standards. Clinical knowledge graph comparison enables professional rule verification, aligning operational behaviors with authoritative guidelines and standards to accurately identify professional deviations. This four-layer analysis balances efficiency and accuracy, adapting to complex clinical nursing scenarios. See Table 2 for details.

[0143] Table 2: Multimodal Quality Control Decision Matrix

[0144] Quality control type Data modal Analysis Model Output Time-sensitive control Time series + text Temporal-BERT Timeout events and deviation values Frequency quality control Time series + numerical LSTM frequency detection Missing records and time points Consistency check Text + Image + Number CLIP+ Regression Model Consistency score Integrity check Text + Form Entity recognition model List of missing fields Terminology check text Medical term KG List of non-standard terms

[0145] In one possible embodiment, the formula for calculating the quality of care score is:

[0146] Nursing quality score = 100 - Σ(defect item weight × severity coefficient)

[0147] In the formula, the weight of timeliness defect is 1.2 and the severity coefficient is 1.0–1.5; the weight of terminology defect is 0.8 and the severity coefficient is 1.0; the weight of single veto item is infinite, and it is automatically judged as Class C.

[0148] Specifically, the formula uses a maximum score of 100 as a basis, and achieves quality quantification by deducting the impact score of defects, which conforms to the medical quality control scoring convention. The weight of defect items distinguishes the basic impact degree of different problems. Timeliness defects are directly related to nursing safety and have a higher weight, while terminology defects affect the standardization of records and have a lower weight. The severity coefficient dynamically matches the actual harm of the defect. The harm of timeliness defects increases with the degree of time delay, so a floating coefficient is set. The single veto item corresponds to major defects that endanger patient safety and has an infinite weight, directly determining the lowest level and ensuring major medical safety. The overall model realizes the objective and standardized quantification of quality control results, replacing traditional manual subjective scoring.

[0149] In one possible embodiment, in S6, when a nursing defect is identified, a real-time warning instruction is generated and pushed to the nurse's terminal; when the nursing defect is not identified, the quality control is marked as passed and a quality control report is generated.

[0150] Specifically, nursing defects pose immediate medical risks, and real-time alerts can quickly push defect information to the responsible nurse's terminal, enabling early detection and intervention of problems and preventing the risk from escalating; when there are no defects, the system automatically marks the condition as passed and generates a report, eliminating the need for manual review and significantly improving quality control efficiency; this logic is adapted to the clinical needs of "real-time quality control and rapid response," taking into account both risk prevention and work efficiency.

[0151] In one possible embodiment, in S6, the nurse terminal receives the warning information and performs rectification operations. After submitting the rectification record, the quality control engine automatically verifies and rectifys the rectification record. If the verification is successful, the quality control is marked as successful and a quality control report is generated. If the verification fails, the real-time warning instruction is generated again.

[0152] Specifically, the quality control logic consists of rectification, verification, and re-early warning. After receiving an early warning, the nurse terminal performs targeted rectification and submits a record to retain the rectification trajectory. The quality control engine automatically verifies the rectification effect based on models and knowledge graphs, avoiding the subjectivity of manual verification. If the verification passes, the logic is archived; if the verification fails, a second early warning is issued, forcing the defect to be rectified. This mechanism solves the problem of existing technologies "emphasizing identification but neglecting rectification" and ensures that quality control measures are implemented effectively.

[0153] In one possible embodiment, in S6, the rectification suggestions are generated based on the type of quality control problem and the clinical knowledge graph, including targeted suggestions such as operating procedures and rectification steps, and are pushed to the nurse terminal simultaneously with the warning information.

[0154] Specifically, different quality control issues correspond to different rectification needs, and combining the issue type can accurately pinpoint the direction of rectification; the clinical knowledge graph provides authoritative nursing guidelines and standard procedures to ensure that rectification suggestions are professional and compliant; the suggestions include operational guidelines and rectification steps, clarifying rectification standards and implementation paths, reducing the difficulty of rectification for nurses; simultaneous push notifications with alerts can reduce information transmission links, improve rectification response speed, and adapt to the efficient pace of clinical work. See the following case:

[0155] Intelligent suggestion generation:

[0156] defgenerate_correction(violation):

[0157] knowledge_base={

[0158] "BP1001": "Establish an instant reminder mechanism for separate beds and set up an 8-hour countdown dashboard".

[0159] "CP3002": "Use standardized skin assessment tools to increase the frequency of check-ups for patients at high risk of pressure ulcers."

[0160] }

[0161] returnknowledge_base.get(violation,

[0162] llm_generate(f"Suggestions for improvement regarding {violation}:"))

[0163] The application scenarios of this invention are as follows:

[0164] The real-time quality control scenario for daily nursing care of inpatients is applicable to inpatients in general wards and specialized wards of hospitals at all levels. It covers the entire nursing cycle and is compatible with multimodal materials such as nursing record texts, vital sign time-series data, mobile nursing PDA operation logs, ward monitoring videos, structured data from electronic medical records, and medical orders. After nurses complete nursing operations and upload relevant data, the system standardizes and integrates the multimodal data and extracts features. Through a multimodal big data model, it completes encoding, cross-modal alignment, and clinical knowledge reasoning. Relying on a multimodal quality control decision matrix, it identifies five types of problems: terminology errors, missing procedures, omissions in risk assessment, time conflicts, and inconsistent medical orders. It uses a weighted dynamic calculation model to generate nursing quality scores, pushes early warnings and targeted rectification suggestions in real time, and forms logical rectification verification. This meets the needs of real-time quality control of the entire inpatient nursing process, accurate identification of hidden defects, standardization of nursing behavior, and reduction of the incidence of adverse nursing events.

[0165] Intelligent quality control scenario for compliance of outpatient nursing operations: Applicable to the entire process of review of outpatient nursing operations in hospital outpatient infusion rooms, dressing rooms, injection rooms, etc. It is compatible with multimodal materials such as outpatient nursing operation record text, skin test / dressing operation monitoring video, patient vital sign time series data, outpatient medical order text, and outpatient electronic medical record structured data. After the outpatient nurse completes the operation, she uploads the relevant materials. The system automatically completes multimodal data fusion and feature extraction, analyzes semantics through a multimodal big data model, verifies the time series, and identifies operation behavior. It conducts quality control verification against clinical nursing guidelines and medical order specifications, outputs quality control defects and quantitative scores, and simultaneously pushes rectification suggestions and supports automatic verification of rectification records. It meets the needs of rapid compliance review of outpatient nursing operations, prevention of operation risks, unification of quality control standards, and improvement of the standardization level of outpatient nursing services.

[0166] Operating Room Nursing Coordination Full-Process Quality Control Scenario: Applicable to the entire process of operating room nursing quality control, including preoperative preparation, intraoperative coordination, and postoperative care. It is compatible with multimodal materials such as operating room nursing records, surgical monitoring videos, instrument count records, patient intraoperative vital signs timeline data, surgical orders, and anesthesia nursing records. After data from each stage of the surgery is uploaded in real time, the system completes multimodal data standardization and feature extraction. Through a multimodal large-scale model, it performs behavior recognition, semantic analysis, and temporal verification to identify issues such as non-standard aseptic operation, missing instrument count procedures, time conflicts in nursing records, and deviations in order execution. It generates quality control scores and provides real-time warnings of high-risk defects, pushes rectification suggestions, and performs logical verification, meeting the needs of precise quality control in operating room nursing, ensuring surgical safety, tracing nursing operations, and standardizing intraoperative nursing coordination processes.

[0167] Risk management scenario for high-risk geriatric patients: Applicable to the nursing management of high-risk inpatients in geriatric departments, such as those with pressure ulcers, falls, and medication allergies. It is compatible with multimodal materials including geriatric patient nursing records, skin wound assessment images, fall prevention nursing operation videos, vital sign time-series data, pressure ulcer / fall risk assessment forms, and medication orders. The system collects and integrates multimodal data in real time, using a multimodal big data model combined with a geriatric nursing knowledge graph to focus on verifying issues such as omissions in risk assessment, missing nursing procedures, inconsistent medication orders, and logical conflicts in record timelines. It dynamically calculates nursing quality scores and pushes risk warnings in real time, generating targeted risk prevention and rectification suggestions to meet the needs of early warning of nursing risks for high-risk elderly patients, accurate identification of potential hazards, standardization of risk prevention and control procedures, and reduction of the incidence of nursing complications in elderly patients.

[0168] The rehabilitation nursing guidance quality review scenario is applicable to the quality review of rehabilitation nursing guidance in rehabilitation departments, including limb rehabilitation, speech rehabilitation, and functional training. It is compatible with multimodal materials such as rehabilitation nursing guidance record texts, rehabilitation training operation videos, patient rehabilitation assessment reports, rehabilitation medical orders, and vital sign time series data during rehabilitation training. After rehabilitation nurses complete the guidance and upload relevant data, the system performs feature extraction and cross-modal alignment on the multimodal data. Through a multimodal large model, it analyzes the semantics of the guidance content, identifies operational standardization, verifies the completeness of the process, identifies quality control defects by comparing with rehabilitation nursing guidelines, outputs quantitative scores and pushes rectification suggestions, and automatically verifies the rectification effect. This meets the needs of standardized review of rehabilitation nursing guidance, improving the professionalism of rehabilitation guidance, tracking nursing quality improvement, and helping to improve patient rehabilitation outcomes.

[0169] It should be noted that the method of this embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this embodiment, and the multiple devices will interact with each other to complete the described multimodal large-scale nursing quality control method.

[0170] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0171] Example 2

[0172] See Figure 6 Embodiment 2 of the present invention also provides a nursing quality control system based on a multimodal large model, which adopts the above-mentioned nursing quality control method based on a multimodal large model, including:

[0173] Data source 100: Used to collect multimodal data related to patient care through HIS system, electronic medical records, nursing records, vital signs devices, mobile nursing PDA and monitoring video;

[0174] Multimodal fusion unit 200: used to extract data from the multimodal data, generate JSON data in a structured manner, perform standardization processing and fusion;

[0175] Feature extraction engine unit 300: used to extract features from the fused multimodal data to obtain text semantic features, temporal variation features and image behavior features;

[0176] Multimodal Large Model Unit 400: Used for encoding extracted features, cross-modal alignment, and clinical knowledge reasoning through a multimodal large model;

[0177] Quality Control Decision Center Unit 500: Used to output quality control questions based on the multimodal quality control decision matrix, calculate nursing quality scores using a weighted dynamic calculation model, and generate quality control decisions.

[0178] Application unit 600: is used to perform real-time early warning, automatic scoring and rectification suggestion push operations based on the quality control decision; the rectification suggestions are generated by matching standard problem codes with the built-in knowledge base to generate fixed rectification solutions, and rectification suggestions are generated by the multimodal big model for non-standard problems.

[0179] It should be noted that the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0180] Example 3

[0181] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code for a multimodal large-scale nursing quality control method. The program code includes instructions for executing the multimodal large-scale nursing quality control method of Embodiment 1 or any possible implementation thereof.

[0182] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0183] Example 4

[0184] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0185] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute a multimodal large-scale nursing quality control method based on Embodiment 1 or any possible implementation thereof by calling the program instructions.

[0186] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0187] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0188] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0189] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A nursing quality control method based on a multimodal large model, characterized in that, include: S1. Collect multimodal data related to patient care through HIS system, electronic medical records, nursing records, vital signs equipment, mobile nursing PDA and monitoring video; S2. Extract data from the multimodal data, generate JSON data in a structured manner, perform standardization processing and fusion; S3. Extract features from the fused multimodal data to obtain text semantic features, temporal change features, and image behavior features; S4. Encode the extracted features using a multimodal large model, perform cross-modal alignment, and apply clinical knowledge reasoning. S5. Output quality control questions based on the multimodal quality control decision matrix, calculate nursing quality scores using a weighted dynamic calculation model, and generate quality control decisions; S6. Based on the quality control decision, perform real-time early warning, automatic scoring, and push rectification suggestions; the rectification suggestions are generated by matching standard problem codes with the built-in knowledge base to generate fixed rectification solutions, and rectification suggestions for non-standard problems are generated by the multimodal large model.

2. The nursing quality control method based on a multimodal large model according to claim 1, characterized in that, In S1, the multimodal data includes text data, time-series data, image data, and structured medical data.

3. The nursing quality control method based on a multimodal large model according to claim 1, characterized in that, In S4, the multimodal large model includes a text encoder, a time encoder, an image encoder, a cross-modal alignment module, and a clinical knowledge graph; The text encoder processes text data, the time encoder processes time-series data, the image encoder processes video / image data, and the cross-modal alignment module realizes multimodal feature alignment and performs quality control analysis in conjunction with the clinical knowledge graph. The cross-modal alignment module employs a multi-head cross-modal attention mechanism to achieve multi-modal feature alignment, and its mathematical expression is as follows: In the formula, Q is the text encoding vector output by the text encoder, with dimension 1. , 768 is the text encoding dimension. The maximum length of the text sequence; For key vectors, The value vectors are obtained by dimensionality transformation of the temporal encoded vector output by the temporal encoder and the image encoded vector output by the image encoder, respectively. The transformed dimensions are all... , 64 represents the single attention head dimension. The number of features; The time-series coding vector is of dimension 1. , , The time step is [time step]; the image encoding vector is [dimension] . , , The feature map size; This is a scaling factor used to prevent the gradient of the Softmax function from vanishing due to an excessively large dot product result. Softmax is a normalization function that outputs the attention weight distribution of each modality feature, with the weights summing to 1.

4. The nursing quality control method based on a multimodal large model according to claim 1, characterized in that, In S4, the quality control issues include errors in professional terminology, missing nursing procedures, omissions in risk assessment, conflicts in time logic, and discrepancies in medical orders.

5. The nursing quality control method based on a multimodal large model according to claim 1, characterized in that, In S5, the quality control analysis process of the multimodal quality control decision matrix includes: A preliminary verification of the timeliness and frequency of nursing procedures was conducted. Deep semantic analysis of the semantic integrity and logical consistency of nursing records; Perform behavioral recognition on nursing operation videos to verify the standardization of operations; By combining the clinical knowledge graph, deviations between the nursing procedures and the diagnostic and treatment guidelines can be identified; The multimodal quality control decision matrix adopts a four-level weighted decision model, and the final decision score calculation formula for a single candidate quality control problem is as follows: In the formula, The final decision score is determined by the following criteria: a score ≥ 0.7 indicates a genuine quality control issue; a score ≤ 0.5 and < 0.7 indicates a suspected issue requiring manual review; and a score < 0.5 indicates a false alarm. , , , The weight coefficients for each layer of verification are 0.2, 0.3, 0.25, and 0.25 respectively, with the weight sum being 1. The basic verification score is calculated using the following formula: In the formula, This represents the deviation between the actual operating time and the standard time. This is the standard duration of the operation. For missing operation counts, This represents the total number of operations to be performed. For semantic analysis scores, the semantic similarity between nursing records and standard records is calculated using the medical BERT similarity model, with values ​​ranging from [0,1]. The behavior recognition score is the operational standardization score output by the SlowFast video behavior recognition model, with a value range of [0,1]. The knowledge graph matching score is calculated from the matching degree between nursing operations and rules, and the value ranges from [0,1].

6. The nursing quality control method based on a multimodal large model according to claim 1, characterized in that, The formula for calculating the nursing quality score is as follows: Nursing quality score = 100 - Σ(defect item weight × severity coefficient) In the formula, the weight of timeliness defect is 1.2 and the severity coefficient is 1.0–1.5; the weight of terminology defect is 0.8 and the severity coefficient is 1.0; the weight of single veto item is infinite, and it is automatically judged as Class C.

7. The nursing quality control method based on a multimodal large model according to claim 1, characterized in that, In S6, when a nursing defect is identified, a real-time warning instruction is generated and pushed to the nurse's terminal; when the nursing defect is not identified, the quality control is marked as passed and a quality control report is generated.

8. The nursing quality control method based on a multimodal large model according to claim 7, characterized in that, In S6, the nurse terminal receives the warning information and performs rectification operations. After submitting the rectification record, the quality control engine automatically verifies and rectifys the rectification record. If the verification is successful, the quality control is marked as successful and a quality control report is generated. If the verification fails, the real-time warning instruction is generated again.

9. The nursing quality control method based on a multimodal large model according to claim 1, characterized in that, In S6, the rectification suggestions are generated based on the type of quality control problem and combined with the clinical knowledge graph to provide targeted suggestions including operating procedures and rectification steps, which are then pushed to the nurse's terminal along with the early warning information.

10. A nursing quality control system based on a multimodal large-scale model, employing the nursing quality control method based on a multimodal large-scale model as described in any one of claims 1-9, characterized in that, include: Data source unit: used to collect multimodal data related to patient care through HIS system, electronic medical record, nursing record, vital signs device, mobile nursing PDA and monitoring video; Multimodal fusion unit: used to extract data from the multimodal data, generate JSON data in a structured manner, perform standardization processing and fusion; Feature extraction engine unit: used to extract features from the fused multimodal data to obtain text semantic features, temporal variation features and image behavior features; Multimodal large model unit: used for encoding extracted features, cross-modal alignment, and clinical knowledge reasoning through multimodal large models; Quality Control Decision Center Unit: Used to output quality control questions based on the multimodal quality control decision matrix, calculate nursing quality scores using a weighted dynamic calculation model, and generate quality control decisions. Application Unit: Used to perform real-time early warning, automatic scoring, and rectification suggestion push operations based on the quality control decision; the rectification suggestions are generated by matching standard problem codes with the built-in knowledge base to generate fixed rectification solutions, and rectification suggestions for non-standard problems are generated by the multimodal large model.

Images