Automatic medical record writing system and method based on multi-modal input and multi-agent driving

Abstract

The invention relates to the technical field of medical information, in particular to a method and a system for processing medical record documents by utilizing artificial intelligence, and particularly relates to a method and a system which can receive and process multi-modal input information including images, videos and voices, can work cooperatively through a multi-agent system and can process the medical record documents. The invention discloses a system for automatically generating, controlling quality and safely inputting medical records in combination with a medical knowledge base and an implementation method thereof. The invention relates to an automatic medical record writing system based on multi-modal input and multi-agent driving. The automatic medical record writing system comprises a multi-modal input module, a multi-agent processing platform and a safety intranet input module. According to the method, establishment of the data channel between the medical intranet and the external AI system is proposed for the first time, the problem of medical intranet isolation is solved, industrial pain points are solved in a breakthrough mode, and a new medical AI landing path is developed.

Description

Technical Field

[0001] The present invention relates to the field of medical information technology, and more specifically to a method and system for processing medical records using artificial intelligence, particularly a system that can receive and process multimodal input information including images, videos, and voice, automatically generate, quality-control, and securely input medical records through the collaborative work of a multi-agent system and in combination with a medical knowledge base, and a method for implementing the system. Background Art

[0002] Medical records are the core records of medical activities, and their writing quality is directly related to the effectiveness of diagnosis and treatment, medical safety, and scientific research and teaching. Traditional medical record writing mainly relies on manual data entry by medical staff, which has problems such as long time consumption, low efficiency, prone to errors, and strong subjectivity. With the development of information technology, electronic medical record systems (EMR) have become popular, but most systems still rely on structured or semi-structured text input. They have limited processing capabilities for the large amount of unstructured and multimodal information generated during the diagnosis and treatment process (such as medical images, examination report photos, surgical videos, doctor-patient communication audio, scanned copies of handwritten notes, etc.), and cannot fully utilize this information to assist in medical record writing.

[0003] In recent years, the application of artificial intelligence (AI) technology, especially natural language processing (NLP) and computer vision (CV) technology in the medical field has gradually deepened, and some AI-assisted medical record writing or medical record quality control tools have emerged. However, most of the existing technical solutions have the following limitations: (1) Insufficient multimodal information fusion capabilities: It is difficult to effectively integrate data from different sources and different formats, and it is impossible to fully understand the patient's condition. (2) Limited intelligence: Most of them stay at the stage of information extraction or simple template filling, lacking deep understanding, reasoning and complex logical judgment capabilities, and it is difficult to generate high-quality, personalized medical records. (3) Intranet deployment and data security challenges: In many medical institutions, the core medical record system is deployed in a strictly controlled intranet environment, and it is prohibited to connect to the external network and install unauthorized software at will. This makes it difficult for external advanced AI models and computing resources to be directly applied to intranet medical record writing. At the same time, how to safely and conveniently import externally processed medical records into the intranet system is also a problem. (4) Lack of systematic collaboration: Medical record writing involves multiple links such as information collection, understanding, integration, generation, and review. Existing single-point technologies are difficult to cover the entire process and lack an effective collaboration mechanism.

[0004] However, the prior art has the following disadvantages.

[0005] (1) Efficiency and accuracy bottlenecks: Manual writing or traditional electronic medical record input methods are inefficient and prone to typos, omissions, or logical errors. Existing AI-assisted tools are ineffective in processing complex, non-standardized, multimodal information, making it difficult to ensure the accuracy and completeness of the generated content.

[0006] (2) Insufficient utilization of multimodal information: A large amount of valuable medical information such as pictures, videos, and voices has not been effectively integrated into medical records, resulting in information silos and waste of resources.

[0007] (3) Limited intranet applications: Advanced AI models typically require powerful computing resources and are difficult to deploy directly on the intranet. Furthermore, the intranet’s strict security policies restrict access to external devices and data, making it difficult to utilize external AI capabilities to assist in intranet medical record writing.

[0008] (4) The quality of medical records varies: The lack of unified and intelligent medical record generation and quality control standards has led to different styles of medical records written by different doctors, making it difficult to ensure quality and increasing the difficulty of subsequent use (such as clinical research and medical supervision).

[0009] (5) Fragmented workflow: Information collection, processing, medical record generation, and intranet entry are independent of each other, failing to form a smooth and automated workflow, which increases the operational burden on medical staff. Summary of the Invention

[0010] This application aims to solve the problems in the existing technology of heavy burden on medical personnel in writing medical records, low efficiency, insufficient use of multimodal information, difficulty in ensuring the quality of medical records, and difficulty in external intelligent systems to safely and conveniently serve the intranet medical record system. Specifically, it includes:

[0011] First, how to build a system that can efficiently receive, process and integrate medical information in multiple modalities such as images, videos, and voice.

[0012] Second, how to use the multi-agent collaborative mechanism to simulate the working methods of human expert teams and realize the automation of the entire process from information understanding to automatic generation of medical records and intelligent quality control.

[0013] Third, how to combine authoritative medical knowledge bases and standard medical record templates to ensure that the generated medical records are professional, standardized, accurate and meet quality control requirements.

[0014] Fourth, how to design a safe, reliable, and driver-free technical solution to conveniently and stream the final medical records generated by the multi-agent system running on an external server into a securely isolated intranet computer medical record system.

[0015] The purpose of the present invention is to provide a system and its implementation method that can receive and process multimodal input information including images, videos, and voice, and automatically generate, quality control, and securely input medical records through the collaborative work of a multi-agent system in combination with a medical knowledge base.

[0016] The first aspect of the present invention provides an automatic medical record writing system based on multimodal input and multi-agent drive, including: a multimodal input module, a multi-agent processing platform, and a secure intranet input module.

[0017] Preferably, the multimodal input module includes images, videos, voice and text, and is used to receive and pre-process multimodal data from medical scenarios.

[0018] Preferably, the multi-agent processing platform is deployed on an external server and includes the following agents working in collaboration:

[0019] Task scheduling agent, used to decompose tasks and assign them to various processing agents;

[0020] Data preprocessing agent, which performs feature extraction and normalization of multimodal data;

[0021] Information fusion agent, achieving semantic alignment and association reasoning through cross-modal attention mechanism

[0022] Knowledge base interactive agent, dynamically calling standard templates, terminology and clinical guidelines in the medical knowledge base;

[0023] Medical record generation agent, which generates structured medical record drafts based on generative AI technology;

[0024] Quality control intelligent agent, which performs multi-dimensional automated review and correction based on the medical rule base;

[0025] Comprehensive medical knowledge base, including standard medical record templates, medical ontologies, clinical knowledge graphs and quality control rules.

[0026] Preferably, the secure intranet input module includes: an encrypted WIFI transmission unit, which streams the final medical record text to a portable USB receiving device; and a USB HID simulated keyboard hardware, which converts the encrypted data into a keyboard scan code and inputs it into the intranet system without the need for driver installation.

[0027] Preferably, the information fusion agent implements cross-modal association reasoning through the following steps:

[0028] Calculate the semantic association weight σ of images, videos, voice and text based on the multi-head attention mechanism

[0029] Fuse the multimodal feature vector into the text vector to construct the initial knowledge graph;

[0030] Combined with the causal inference model in the medical knowledge base, the knowledge graph is optimized to generate medical record content that is consistent with clinical logic.

[0031] Preferably, the secure intranet input module uses the AES-256-GCM encryption algorithm to encrypt the transmitted data, and ensures data integrity and tamper resistance through a physical trusted execution environment (TEE).

[0032] Preferably, the quality control agent performs the following operations based on the dynamically updated medical knowledge base: detecting the terminology standardization, diagnosis and treatment compliance and logical consistency in the medical records σ triggering the multi-agent collaborative optimization process according to the quality control feedback to form a self-evolving quality control closed loop.

[0033] A second aspect of the present invention provides an automatic medical record writing method based on multimodal input and multi-agent drive, characterized by comprising the following steps:

[0034] Receive multimodal medical data and perform preliminary formatting and metadata tagging

[0035] Through multi-agent collaboration, data preprocessing, cross-modal fusion, knowledge base query and medical record generation

[0036] Combined with the medical knowledge base, intelligent quality control and automatic correction of the draft medical records are carried out;

[0037] The final medical record is transmitted to the USB HID device via encrypted WIFI and simulated keyboard input into the intranet system.

[0038] Preferably, the cross-modal fusion step includes: using OCR to recognize text in the image and ASR to convert speech into text; and generating a semantically consistent patient information view by associating image findings, pathology reports and patient symptoms through knowledge graph reasoning.

[0039] The following effects are achieved through the present invention:

[0040] 1. Multimodal Information Fusion

[0041] Cross-modal processing: Integrate image (OCR recognition), speech (ASR conversion), text (NER entity recognition) and other data to build a unified patient information view.

[0042] Deep semantic association: Through knowledge graph reasoning, image findings are logically associated with text descriptions to explore potential clinical clues.

[0043] 2. Multi-agent collaboration mechanism

[0044] Emergent intelligence: Heterogeneous intelligent agents (data preprocessing, knowledge base interaction, quality control) dynamically collaborate to generate comprehensive capabilities that go beyond a single model.

[0045] Self-learning optimization: Continuously optimize generation and reasoning strategies based on quality control feedback and new knowledge input.

[0046] 3. Secure intranet input module

[0047] WIFI+USB HID technology: Encrypted transmission of medical record text to a simulated keyboard device, enabling driver-free, cross-platform intranet data input and circumventing security restrictions.

[0048] 4. Knowledge base empowerment

[0049] Dynamic medical knowledge base: integrates standard templates, terminology systems, and clinical guidelines to support a closed loop of generation and quality control.

[0050] This invention proposes for the first time the establishment of a data channel between the medical intranet and the external AI system, which solves the problem of medical intranet isolation, solves the industry's pain points in a breakthrough way, and opens up a new path for the implementation of medical AI. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Shows the overall system architecture diagram

[0052] Figure 2 Showing the internal structure and interaction flow chart of the multi-agent platform

[0053] Figure 3 Showing the multimodal information processing flow chart

[0054] Figure 4 Shows the working principle diagram of the secure intranet input module (WIFI+USB HID) DETAILED DESCRIPTION

[0055] The present invention is described in detail below through examples. It is necessary to point out that the following examples are not to be construed as limiting the scope of protection of the present invention. If a person skilled in the art makes some non-essential improvements and adjustments to the present invention based on the above-mentioned invention content, they still fall within the scope of the present invention.

[0056] The present invention provides an automatic medical record writing system based on multimodal input and multi-agent drive, characterized by comprising:

[0057] -Multimodal input module, used to receive and pre-process multimodal data from medical scenarios, including images, videos, voice and text;

[0058] -Multi-agent processing platform, deployed on an external server, including the following collaborative agents:

[0059] Task scheduling agent, used to decompose tasks and assign them to various processing agents;

[0060] Data preprocessing agent, which performs feature extraction and normalization of multimodal data;

[0061] Information fusion agent, which achieves semantic alignment and associative reasoning through cross-modal attention mechanism;

[0062] Knowledge base interactive agent, dynamically calling standard templates, terminology and clinical guidelines in the medical knowledge base;

[0063] Medical record generation agent, which generates structured medical record drafts based on generative AI technology;

[0064] Quality control intelligent agent, which performs multi-dimensional automated review and correction based on the medical rule base;

[0065] - Comprehensive medical knowledge base, including standard medical record templates, medical ontologies, clinical knowledge graphs and quality control rules;

[0066] -Secure intranet input module, including:

[0067] Encrypted WIFI transmission unit, streaming the final medical record text to a portable USB receiving device;

[0068] USB HID simulates keyboard hardware and converts encrypted data into keyboard scan codes for input into the intranet system without driver installation.

[0069] This invention achieves synergistic and safe application effects that are difficult to foresee with existing technologies through the ingenious combination of cross-domain technologies and innovative design of system architecture:

[0070] 1. Emergent Intelligence Multi-Agent Collaborative Architecture: This groundbreaking technology builds a distributed collaborative system comprised of multiple heterogeneous, specialized agents. Each agent not only handles specific tasks (multimodal recognition, fusion, reasoning, generation, quality control, etc.), but also, through dynamic negotiation and adaptive learning mechanisms, generates "emergent intelligence" in complex scenarios that surpasses the sum of individual capabilities. This enables a deep understanding of difficult cases and the precise generation of high-quality, personalized medical records. This performance far surpasses traditional AI systems based on single models or simple process chains, achieving an unexpected level of professionalism.

[0071] 2. Cross-modal Deep Understanding and Associative Reasoning: This technology not only enables unified processing of multimodal inputs such as images, video, voice, and text, but more importantly, through advanced multimodal fusion models and knowledge graph reasoning, it achieves deep semantic alignment and associative analysis of cross-modal information. For example, it can logically associate imaging findings with pathology report descriptions and patients' verbal symptoms, exploring potential connections and generating highly consistent and inherently logical medical records. This addresses the challenges of superficial information fusion and insufficient associations encountered by existing technologies.

[0072] 3. Knowledge-Base-Enabled Self-Evolving Generation and Quality Control Closed Loop: A dynamically updated comprehensive medical knowledge base has been constructed and deeply integrated into the entire process of medical record generation and quality control. Medical record generation is no longer simply template filling, but rather a process of reasoning based on the knowledge base. Intelligent quality control goes beyond rule verification and can also uncover underlying logical errors and potential risks based on the knowledge base. The system possesses self-learning and self-optimization capabilities based on quality control feedback and new knowledge input, forming a continuously evolving intelligent closed loop.

[0073] 4. Multi-agent Adaptive Collaboration and Dynamic Evolution Mechanism: This system innovatively incorporates a multi-agent adaptive collaboration mechanism. Each agent possesses an independent knowledge graph, reasoning engine, and feedback learning module, dynamically adjusting its division of labor and collaboration strategies based on task complexity, changes in data distribution, and quality control feedback. Through reinforcement learning and meta-learning algorithms, these agents continuously optimize their capabilities in real-world applications, enabling cross-modal reasoning, knowledge transfer, and adaptive handling of abnormal scenarios, significantly enhancing the overall robustness and innovation of the system.

[0074] 5. Cross-modal deep reasoning and causal analysis capabilities: The system integrates a multi-level cross-modal reasoning network, supports multi-level information fusion from low-level features to high-level semantics, and combines it with a causal inference model to automatically identify causal relationships between medical events, assisting doctors in discovering potential causes, complication risks, and treatment path optimization recommendations, significantly improving the scientific nature and foresight of medical records.

[0075] 6. Innovative implementation of a secure intranet input module: High-strength encryption algorithms (such as AES-256-GCM) ensure secure Wi-Fi data transmission. A USB HID keyboard emulation module supports multi-OS compatibility and high-speed streaming input, and features anti-replay, anti-tampering, and real-time anomaly detection mechanisms. A trusted execution environment (TEE) and physical protection design at the hardware level ensure data integrity and non-repudiation during transmission and input.

[0076] 7. System openness and scalability: The platform supports plug-in intelligent agent expansion and hot updates of the knowledge base, facilitating the subsequent integration of new AI models, medical standards, and third-party services, and has the ability to continuously evolve for the future development of medical information technology.

[0077] 8. Cross-domain integrated WIFI+USB HID secure intranet "ferrying" technology: Creatively combines mature WIFI wireless communication technology with the universal USB HID (Human Interface Device) protocol to design a new, non-invasive intranet data security input solution. This solution cleverly utilizes the operating system's "trust" in standard keyboards to "ferry" high-value medical record text processed by an external AI system via encrypted WIFI to a hardware device that simulates keyboard behavior, which is then securely and quickly "typed" into the intranet system. This method completely avoids the security risks and driver compatibility issues brought about by traditional data import methods (such as USB flash drives, network interfaces, and dedicated software), providing an unexpected and extremely effective cross-domain solution for resolving data interaction in highly isolated environments.

[0078] In the present invention, Figure 1 The overall system architecture diagram is shown in FIG.

[0079] 1. Multimodal input module: Deployed on the user interaction end (such as mobile apps and web applications), it is responsible for receiving medical data in various modalities uploaded by users, including but not limited to:

[0080] -Image files: photos of examination reports, screenshots of medical images, photos of patients’ vital signs, etc.

[0081] -Video files: surgical video clips, patient activity records, endoscopy videos, etc.

[0082] - Voice files: recordings of doctor-patient conversations, doctor's oral records, etc.

[0083] -Text files: medical records from other sources, descriptions of chief complaints, etc. This module performs preliminary formatting and metadata tagging of input data.

[0084] 2. External multi-agent processing platform (deployed in the cloud or on a designated server): This is the system's core intelligent processing unit. It utilizes a multi-agent collaborative architecture, with each agent clearly defined and able to communicate and collaborate efficiently through internal message queues or APIs. The platform possesses self-reflection-based self-evolution capabilities. When system performance deteriorates, errors occur, or significant changes in data distribution are detected, it automatically triggers optimization processes (such as adjusting model parameters, optimizing processing flows, and updating the knowledge base).

[0085] In the present invention, Figure 2 The internal structure and interaction flow chart of the multi-agent platform. The main agents are as follows:

[0086] Task Scheduling and Management Agent (Orchestrator): As the central coordinator, it receives task requests from the input module, decomposes tasks, intelligently distributes subtasks to the most appropriate processing agent, and monitors task execution status and resource allocation in real time.

[0087] -Data Preprocessing and Feature Engineering Agent: Responsible for cleaning, format conversion, and standardization of received multimodal data. Leveraging advanced CV, ASR, and NLP models, it performs preliminary information extraction (e.g., OCR for report text, ASR for speech-to-text conversion, and NER for identifying medical entities within text). It also performs feature engineering, extracts key features, and handles missing and outlier values, preparing for subsequent integration and analysis.

[0088] -Multimodal Information Fusion and Understanding Agent: The core task is to integrate processing results from different modalities, perform cross-modal information alignment (such as associating imaging findings with text descriptions), semantic association analysis, and potential conflict resolution, to build a unified, comprehensive, and structured patient information view and deeply understand the patient's condition.

[0089] -Knowledge Base Interaction and Reasoning Agent: Serving as a bridge between the system and the medical knowledge base, it proactively queries the knowledge base based on the integrated information to obtain relevant standard medical record templates, medical terminology, clinical diagnosis and treatment guidelines, disease knowledge graphs, similar cases, drug information, quality control rules, etc. It performs logical reasoning based on the knowledge base to provide a basis for medical record generation and quality control.

[0090] Medical Record Generation and Optimization Agent: Based on fused information and knowledge base support, it follows selected medical record templates and writing standards, and utilizes generative AI technologies such as Large Language Models (LLMs) to automatically write structured and personalized draft medical records. It can iteratively optimize based on quality control feedback to improve generation quality.

[0091] - Intelligent Medical Record Quality Control Agent: Based on the quality control rule set in the knowledge base (covering completeness, consistency, timeliness, logic, terminology standardization, and treatment compliance), it automatically reviews and scores the generated medical record drafts from multiple dimensions. It can identify potential errors, omissions, or irregularities, and provide specific modification suggestions or automatically make corrections when the confidence level is high.

[0092] - Output Formatting and Secure Transmission Agent: This agent encodes the medical record text, which has passed quality control, into the format (e.g., UTF-8) required by the target intranet system (e.g., plain text stream). It collaborates with the sender of the secure intranet input module to securely and stably transmit the text stream to a portable USB receiving device via an encrypted Wi-Fi channel (e.g., WPA2 / WPA3).

[0093] 3. Comprehensive Medical Knowledge Base: Serving as the "brain" of the system, it stores massive, multi-dimensional, structured and unstructured medical knowledge. Its core components include:

[0094] -Standard medical record template library: contains standard and custom medical record templates for different departments and types (such as admission records, medical records, surgical records, discharge summaries, etc.).

[0095] -Medical ontologies and terminology systems: such as SNOMED CT, ICD codes (ICD-10 / 11), LOINC, ATC drug classification, and authoritative medical dictionaries to ensure terminology standardization.

[0096] -Clinical Knowledge Graph: Builds a complex relationship network between entities such as diseases, symptoms, signs, examinations, tests, treatments, and drugs, supporting deep reasoning.

[0097] -Clinical diagnosis and treatment guidelines and pathways: Integrate clinical practice guidelines and standardized diagnosis and treatment pathways published by domestic and foreign authorities.

[0098] -Medical record writing standards and quality control rule library: Contains the basic medical record writing standards issued by national and local health commissions, as well as a refined and executable quality control rule set based on expert experience and historical data mining.

[0099] Anonymized historical case database: Used for model training, similar case retrieval, and pattern discovery. The knowledge base must establish a continuous update mechanism, connect to authoritative medical databases and literature repositories, and allow for maintenance and expansion by internal experts. The knowledge base interactive agent interacts with the knowledge base through efficient retrieval and reasoning engines (e.g., graph database query and logical reasoning engine).

[0100] 4. Secure intranet input module (based on WiFi encrypted transmission + USB HID simulated keyboard): Aims to solve the data transmission problem between external intelligent systems and secure isolated intranet systems, and realize safe, driver-free, high-speed, cross-platform text input.

[0101] In the present invention, Figure 3 A multimodal information processing flow chart is shown.

[0102] The doctor uses the mobile app to take a picture of the patient's examination report, record a conversation with the patient, select the corresponding medical record template type, and submit the task.

[0103] 1. The multimodal input module receives data and sends it to the external multi-agent platform.

[0104] 2. The task scheduling agent starts the corresponding processing flow.

[0105] 3. The visual agent performs OCR recognition on the image; the speech agent converts the speech into text and extracts key information.

[0106] 4. The text agent processes the recognized text and other text information entered by the doctor.

[0107] 5. Information fusion agents integrate information from all sources.

[0108] 6. The knowledge base interactive agent queries related templates and knowledge.

[0109] 7. The medical record generation agent writes the first draft of the medical record.

[0110] 8. The medical record quality control agent reviews the medical records and may automatically make corrections or mark items for confirmation.

[0111] 9. After the final medical record is confirmed, the output transmission agent sends the text stream to a portable USB receiving device connected to the intranet computer via encrypted WIFI.

[0112] 10. The USB receiving device decrypts the data and simulates the keyboard to input the medical record content into the medical record editor opened on the intranet computer.

[0113] In the present invention, Figure 4 The working principle diagram of the secure intranet input module.

[0114] The sender (integrated into an external multi-agent platform or user-side application) is controlled by the output formatting and secure transmission agent, which encodes the finalized medical record text into character encoding (e.g., UTF-8). A secure Wi-Fi connection (either point-to-point ad-hoc or to a dedicated secure hotspot) is established using a strong encryption protocol (e.g., WPA2 / WPA3 Personal / Enterprise) to stream the encoded text data to a portable USB receiving device. Streaming allows medical record content to be transmitted as it is generated, significantly reducing the overall waiting time for long text input.

[0115] -Portable WIFIUSB receiving device (hardware):

[0116] Core components: Typically include a module that supports Wi-Fi connectivity (such as the ESP8266 / ESP32 series chips) and a microcontroller (MCU, such as the ESP32-S2 / S3, RP2040, or ATmega32U4 with native USB support) with USB HID (Human Interface Device) host emulation capabilities.

[0117] WIFI reception and decryption: The built-in WIFI module is responsible for receiving the encrypted data stream from the sender and performing the decryption operation.

[0118] Data parsing and HID conversion: The MCU parses the decrypted text data stream and converts character sequences (such as UTF-8 encoded characters) into standard USB keyboard scan code sequences compatible with the target operating system in real time.

[0119] USB interface: Connect to the USB port of the intranet computer via a standard USB-A or USB-C interface.

[0120] Power supply: Can be powered by USB port.

[0121] -Receiving end (intranet computer):

[0122] Driver-free identification: The operating system of the intranet computer (such as Windows) automatically identifies the USB device as a standard USB keyboard (compliant with HID Class specifications), without the need to install any additional drivers or software, meeting the strict security policies of the intranet.

[0123] Simulated keyboard input: When a user activates the input focus (places the cursor within the input box) in an intranet medical record editing system, text editor, or other application that allows text input, the portable USB receiving device begins simulating the keystrokes of a physical keyboard, "tapping" the corresponding characters, numbers, symbols, and necessary control keys (such as Shift and Enter) one by one in the received sequence, quickly and accurately inputting the medical record text into the target location. This principle is based on the Windows system's built-in ALT Code, which converts Chinese characters and special characters through a combination of the Alt key and the numeric keys on the numeric keypad.

[0124] Compatibility: Compatible with any Windows software that accepts standard keyboard input, such as Word, WPS, WordPad, various EMR / HIS systems, etc.

[0125] Beneficial effects of the present invention

[0126] This application not only solves the pain points of the existing technology through the above technical solutions, but also brings unexpected technical advantages and application value:

[0127] 1. A double leap in the efficiency and quality of medical record writing: The generation mechanism driven by multi-agent collaboration and knowledge base has not only greatly improved writing efficiency, but also improved the depth, accuracy, and logic of medical records to an unexpected extent. The output is close to or even exceeds the level of experienced doctors, achieving nonlinear growth in efficiency and quality.

[0128] 2. Breakthrough Intranet Security Interaction: The innovative combination of Wi-Fi and USB HID provides an unprecedented, secure and convenient method for intranet data input. It cleverly bypasses traditional security restrictions, enabling "zero trust" empowerment of intranet services with external intelligent capabilities. This highly secure and universally applicable solution offers valuable cross-domain insights for other scenarios requiring the introduction of external data into isolated networks.

[0129] 3. Activating the potential value of multimodal information: Through deep fusion and correlation analysis, the system can extract clinical clues from seemingly fragmented multimodal information that are difficult to discover using traditional methods, providing unexpected insights for precise diagnosis and personalized treatment, and significantly improving the depth and breadth of medical information utilization.

[0130] 4. System robustness and adaptability driven by emergent intelligence: The multi-agent architecture empowers the system with high flexibility and adaptability. Even if some agents encounter difficulties or data patterns change, the system can maintain overall stability through coordinated adjustments and self-learning, demonstrating unexpected robustness even in complex and rare cases.

[0131] 5. Reshaping the medical workforce: Freeing doctors from the arduous paperwork of document processing, allowing them to focus more on clinical diagnosis and treatment. This automated, intelligent, closed-loop process significantly improves the work experience and enhances professional satisfaction, profoundly impacting the healthcare system.

Claims

1. An automatic medical record writing system based on multimodal input and multi-agent drive, characterized by: include: Multimodal input module, multi-agent processing platform, and secure intranet input module.

2. The system according to claim 1, wherein: The multimodal input module, including images, videos, voice and text, is used to receive and pre-process multimodal data from medical scenarios.

3. The system according to claim 1, wherein: The multi-agent processing platform is deployed on an external server and includes the following collaborative agents: Task scheduling agent, used to decompose tasks and assign them to various processing agents; Data preprocessing agent, which performs feature extraction and normalization of multimodal data; Information fusion agent, achieving semantic alignment and association reasoning through cross-modal attention mechanism Knowledge base interactive agent, dynamically calling standard templates, terminology and clinical guidelines in the medical knowledge base; Medical record generation agent, which generates a structured draft of medical records based on generative AI technology Quality control intelligent agent, which performs multi-dimensional automated review and correction based on the medical rule base; Comprehensive medical knowledge base, including standard medical record templates, medical ontologies, clinical knowledge graphs and quality control rules.

4. The system according to claim 1, wherein: The secure intranet input module includes: Encrypted WIFI transmission unit, streaming the final medical record text to a portable USB receiving device USB HID simulates keyboard hardware and converts encrypted data into keyboard scan codes for input into the intranet system without driver installation.

5. The system according to claim 1, wherein: The information fusion agent implements cross-modal association reasoning through the following steps: Calculate the semantic association weight σ of images, videos, voice and text based on the multi-head attention mechanism Fuse the multimodal feature vector into the text vector to construct the initial knowledge graph; Combined with the causal inference model in the medical knowledge base, the knowledge graph is optimized to generate medical record content that is consistent with clinical logic.

6. The system according to claim 1, wherein: The secure intranet input module uses the AES-256-GCM encryption algorithm to encrypt the transmitted data and ensures data integrity and tamper resistance through a physical trusted execution environment (TEE).

7. The system according to claim 1, wherein: The quality control agent performs the following operations based on the dynamically updated medical knowledge base: Check the terminology standardization, diagnosis and treatment compliance and logical consistency in medical records The multi-agent collaborative optimization process is triggered based on quality control feedback to form a self-evolving quality control closed loop.

8. An automatic medical record writing method based on multimodal input and multi-agent drive, characterized in that: The following steps are involved: Receive multimodal medical data and perform preliminary formatting and metadata tagging; Through multi-agent collaboration, data preprocessing, cross-modal fusion, knowledge base query and medical record generation Combined with the medical knowledge base, intelligent quality control and automatic correction of the draft medical records are carried out; The final medical record is transmitted to the USB HID device via encrypted WIFI and simulated keyboard input into the intranet system.

9. The method according to claim 8, characterized in that The cross-modal fusion step includes: Use OCR to recognize text in images and ASR to convert speech into text Through knowledge graph reasoning, image findings, pathology reports and patient symptoms are associated to generate a semantically consistent patient information view.

Images