An AI artificial intelligence-based auxiliary communication system

Abstract

The application discloses an AI artificial intelligence auxiliary communication system and belongs to the field of artificial intelligence multi-agent calculation. The system comprises a multi-modal pathological perception hardware layer, a medical privacy edge calculation layer, a multi-agent AI algorithm layer, a medical knowledge enhanced RAG layer, a bidirectional adaptive interactive service layer and a clinical closed loop application layer. The system is equipped with five collaborative work AI agents, a vertical medical communication exclusive RAG knowledge base, a doctor-patient double-end digital twin adaptation model and a pathological defect semantic analysis algorithm. The whole system is locally edge offline running, and completes the doctor-patient intelligent communication according to a six-step closed loop process of "perception collection-intelligent analysis-bidirectional adaptation-interactive output-data iteration-rehabilitation intervention". The pathological defect expression recognition precision of the application is improved. The pathological semantic reconstruction algorithm is used to optimize the fragmented expression of aphasia and cognitive impairment patients, the recognition accuracy is improved compared with a general AI model, and the problem that a traditional device cannot recognize pathological non-standard expression is solved.

Description

Technical Field

[0001] This invention relates to the field of multi-agent computing in artificial intelligence, and in particular to an AI-assisted communication system. Background Technology

[0002] There is a large number of special patient groups with communication barriers in clinical practice, including stroke patients with aphasia, dysarthria, Alzheimer's cognitive impairment, patients who cannot speak after ICU intubation, hearing impairment, and elderly people with declining expressive abilities. These patients are unable to complete standardized language expression, resulting in information asymmetry between doctors and patients, low efficiency of diagnosis and treatment communication, and critically ill patients lack effective interaction channels. There are huge communication barriers in clinical diagnosis, treatment and rehabilitation work.

[0003] Currently, mainstream medical auxiliary communication devices and general-purpose AI speech-to-text conversion systems all suffer from inherent flaws such as excessive versatility and insufficient adaptability to medical specialties. The existing technical pain points are concentrated in five aspects: General AI models have poor adaptability: Traditional speech and text recognition models can only parse standard complete sentences and cannot recognize fragmented speech, blurred gestures, disordered expressions, and discontinuous and incomplete expressions of aphasic patients. The accuracy of pathological expression recognition is extremely low. The interaction mode is limited to one-way communication: existing devices only support one-way translation of information from patients to medical staff, and cannot adapt medical professional medical terminology and medical orders into content that patients can understand. The barrier to two-way communication between doctors and patients has not been broken down. Lack of clinical condition linkage capability: The model cannot dynamically adjust the interaction mode according to the patient's disease type, disease stage, and recovery status, and its intelligent adaptation capability is insufficient. Data security deployment has flaws: most systems rely on cloud-based online inference, and patient pathology and communication privacy data need to be uploaded to the server, which poses a risk of leakage. This cannot meet the compliance requirements of high-confidential medical scenarios such as ICU, operating room, and infectious disease ward, and cloud network latency can easily cause communication delays. Functional fragmentation and lack of a medical closed loop: It only has basic text and voice translation functions and cannot be linked to electronic medical records, medical records, rehabilitation training, and prognosis assessment. It is only used as an independent tool and cannot be integrated into the complete medical service process.

[0004] Existing publicly available accessible communication devices and general speech-to-text AI products have not been optimized for pathological disability expressions in a vertical medical context, nor have they established offline privacy computing architectures and two-way adaptation mechanisms between doctors and patients, making it difficult to address the deficiencies in real-world clinical communication barriers. Summary of the Invention

[0005] Purpose of the invention: The invention provides an AI-based artificial intelligence-assisted communication system that addresses the technical shortcomings of existing medical auxiliary communication systems, such as generalization, one-way interaction, high privacy risks, rigid models, and fragmented functions.

[0006] Technical Solution: To address the aforementioned technical problems, according to one aspect of the present invention, more specifically, it is an AI-assisted communication system. The system adopts a six-layer, decoupled architecture, consisting of a multimodal pathology perception hardware layer, a medical privacy edge computing layer, a multi-agent AI algorithm layer, a medical knowledge enhancement RAG layer, a bidirectional adaptive interactive service layer, and a clinical closed-loop application layer. The system is equipped with five collaborative AI agents, a dedicated RAG knowledge base for vertical medical communication, a digital twin adaptation model for both doctors and patients, and a pathological incompleteness semantic parsing algorithm. The entire system operates offline at the local edge, completing intelligent doctor-patient communication through a six-step closed-loop process: "perception and acquisition - intelligent parsing - bidirectional adaptation - interactive output - data iteration - rehabilitation intervention."

[0007] First layer: Multimodal pathology sensing hardware layer A comprehensive signal acquisition hardware module adapted to patients with various communication impairments was built, abandoning the single voice acquisition mode. The integrated modules include: a high-sensitivity noise-reducing voice acquisition module, a high-definition facial expression recognition camera, a flexible gesture sensing terminal, an eye-tracking module, and a limb micro-motion sensor. Dedicated acquisition combinations were configured for different groups of people with disabilities: a fragmented voice waveform enhancement acquisition module was used for aphasic patients, facial expression and limb micro-motion capture modules were optimized for cognitive impairment patients, eye-tracking + touch perception modules were used only for ICU intubated patients, and visual signal acquisition modules were enhanced for hearing-impaired patients, thus completing the complete recording of all incomplete and non-standardized expressive signals of patients.

[0008] Second layer: Medical privacy edge computing layer It establishes a local offline edge inference node for medical applications, where all perception data, pathology data, and AI inference operations are completed on the local terminal without uploading to the cloud. It has three built-in security control modules: a medical data desensitization module, a hierarchical access control module, and a data timed dynamic destruction module. It strictly follows the "Medical Institution Data Security Specification" and is suitable for scenarios without network access, high-confidential ICUs, operating rooms, and infectious disease wards. At the same time, local inference achieves millisecond-level interactive response, eliminating cloud network latency and lag issues.

[0009] Third layer: Multi-agent AI algorithm layer The system's core inference layer adopts a multi-agent collaborative architecture, comprising five independently functioning and dynamically scheduled agents: (1) Pathological perception intelligent agent: completes the preprocessing of multi-source hardware acquisition data, filters environmental noise, repairs fragmented speech and blurred gesture data, and realizes multimodal perception data fusion; (2) Incomplete Semantic Analysis Intelligent Agent: Equipped with a self-developed pathological incomplete semantic reconstruction algorithm, it performs semantic reconstruction on disordered word order, missing words, and fragmented expressions to restore the patient's true needs; Core algorithm formula:

[0010] In the formula: To standardize the semantic results of reconstruction; The patient's original incomplete semantic vector; Pathological matching weights for disease type and disorder type; A factor for modifying the diagnostic and treatment context; For dynamic weighting coefficients, satisfying The intelligent agent, linked to the patient's condition, adjusts in real time according to the recovery stage. (3) Doctor-patient twin adaptive intelligent agent: Build a dual-end digital twin model of patient and medical staff, learn the patient's pathological expression habits and the standardized diagnosis and treatment language of medical staff respectively, and complete the two-way adaptive conversion of language; (4) Disease-linked iterative intelligent agent: Connects with hospital EMR electronic medical records, HIS system and rehabilitation management platform, synchronously collects disease course, treatment plan and rehabilitation data, and dynamically updates model recognition parameters; (5) Risk control and compliance intelligent agent: Real-time verification of medical compliance of communication content and privacy security of data transmission, and interception of illegal medical statements and privacy leakage behaviors.

[0011] Fourth layer: Medical knowledge enhancement RAG layer We build a dedicated knowledge base for vertical medical communication, integrating pathological characteristics, common clinical needs, standard treatment scripts, and rehabilitation communication norms for communication disorders such as stroke, cognitive impairment, post-critical care, and hearing impairment. We adopt RAG retrieval enhancement generation technology, which allows AI to retrieve authoritative medical knowledge in real time to anchor semantics during inference, eliminating the illusion problem of general large models. We support hospitals and departments to customize and enter their own treatment guidelines and departmental communication standards, ensuring the medical accuracy of translated content.

[0012] Fifth layer: Two-way adaptive interactive service layer Breaking the limitations of one-way interaction, establishing a two-way intelligent conversion channel: Patient-side conversion: Automatically simplifies medical terminology and complex medical orders into short voice sentences, graphic prompts, and simple text to match the patient's cognitive level; Healthcare professionals can automatically organize fragmented and ambiguous patient expressions into structured and standardized texts of their medical needs and simultaneously push communication reference tips. It supports multi-form interactive output including voice, text, static images and text, dynamic gestures, and eye-tracking commands.

[0013] Sixth layer: Clinical closed-loop application layer It covers all medical scenarios and implements functional modules to achieve a closed loop of communication, diagnosis, rehabilitation, and assessment: real-time doctor-patient auxiliary communication module, intelligent patient request reporting module, intelligent medical order translation module, automatic communication data archiving module, personalized rehabilitation communication training module, communication ability adaptation assessment module, and doctor-patient communication quality review module; it is suitable for all scenarios including outpatient clinics, inpatient wards, ICUs, rehabilitation departments, community healthcare, and home-based elderly care and rehabilitation.

[0014] Furthermore, the operational process is as follows: Step 1: Multimodal pathological perception acquisition: The hardware layer acquires multi-dimensional expressive signals of the patient's voice, facial expressions, gestures, and eye movements in real time, and completes noise reduction and incomplete data preprocessing; Step 2: Offline intelligent agent collaborative analysis: The edge node starts parallel reasoning of five intelligent agents, and restores the patient's true needs by using the incomplete semantic reconstruction algorithm, combined with the medical RAG knowledge base and patient medical record data; Step 3: Two-way adaptive matching between doctors and patients: The doctor-patient twin matching intelligent agent calls the dual-end digital twin model to bidirectionally convert the content expressed by both doctors and patients; Step 4: Multi-form intelligent interactive output: Complete two-way information transmission between doctors and patients through voice, images, and text; Step 5: Data archiving and model iteration: Automatically store the communication record, synchronously update the patient communication behavior dataset, and automatically adjust the AI ​​model weight parameters in conjunction with the patient's condition data; Step 6: Rehabilitation closed-loop intervention: Based on the patient's degree of recovery from communication barriers, generate a personalized communication rehabilitation training plan and simultaneously complete a quantitative assessment of communication ability.

[0015] Beneficial effects: (1) The present invention improves the accuracy of pathological incomplete expression recognition. It optimizes the fragmented expression of patients with aphasia and cognitive impairment through pathological semantic reconstruction algorithm, and the recognition accuracy is higher than that of general AI models, solving the industry pain point that traditional equipment cannot recognize pathological non-standard expressions. Through a digital twin adaptation mechanism between doctors and patients, professional medical terminology can be converted in both directions to be expressed in simple terms by patients, completely breaking down the barrier of one-way translation and reducing the cost of communication and understanding between doctors and patients. The edge-local offline inference architecture ensures that data is not uploaded to the cloud. Combined with data anonymization, access control, and automatic data destruction mechanisms, it meets the compliance requirements of highly confidential departments in hospitals, while eliminating network latency and achieving millisecond-level interactive response. (2) The present invention realizes the dynamic adaptive optimization of the AI ​​model according to the condition. By linking with the hospital's electronic medical record system, it automatically adjusts the recognition accuracy and interaction mode according to the patient's recovery process, adapts to the patient's communication needs throughout the entire treatment cycle, and solves the defects of the traditional system model being fixed and unable to adapt to changes in the condition. No longer limited to a simple information translation tool, it integrates functions such as diagnosis and treatment assistance, rehabilitation training, prognosis assessment, and communication review, deeply embedding itself into the hospital's existing diagnosis and rehabilitation business processes to improve the overall diagnosis and treatment experience for patients with special disabilities; The modular, layered architecture can be flexibly tailored to adapt to various scenarios such as ICU, neurology, rehabilitation, geriatrics, home-based elderly care, and community healthcare. It can also be expanded to include VR virtual communication, vital sign linkage sensing, and other extension modules. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the system framework of the present invention. Detailed Implementation

[0017] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Example 1

[0019] Reference Figure 1 An AI-assisted communication system, designed for ICU intubated aphasic patients, is implemented as follows: This embodiment is applied to ICU patients who have undergone endotracheal intubation after surgery or are completely aphasic upon recovery from severe illness. These patients are unable to speak and have limited limb movement. Furthermore, ICU wards are characterized by high confidentiality, no external network access, and high priority of treatment. Therefore, the system operates in a minimalist, fully offline, and privacy-first mode. The specific implementation steps are as follows: The first step is for the system to automatically adapt to the patient's physical condition and the ward scene, turn off redundant modules such as voice acquisition, facial expression, and gesture sensing, and only enable the eye tracking module and high-precision touch sensing terminal to continuously and silently collect weak expressive signals such as the patient's eye movement trajectory and touch operation 24 hours a day. The second step involves directly transmitting all raw data collected by the hardware to the local edge computing node. There is no network connection or cloud data upload throughout the process. The data desensitization module built into the node performs desensitization processing on the collected signals in real time. At the same time, the pathological perception intelligent agent completes noise reduction of weak signals and preprocessing for defect repair. The third step involves the incomplete semantic parsing agent calling upon the ICU-specific medical RAG knowledge base to focus on high-frequency demands of critically ill patients, such as pain feedback, body position adjustment, oxygen and fluid inhalation, and coughing discomfort, and to accurately interpret the patient's intentions in conjunction with the context of critical care diagnosis and treatment. The fourth step involves the doctor-patient twin-adaptive intelligent agent transforming professional medical terms such as monitoring inquiries, nursing orders, and treatment guidance from the medical staff into simplified icons and short text pop-ups that are pushed to the patient's terminal, thus enabling the easy-to-understand delivery of medical orders and accurate feedback on patient needs. The fifth step involves the intelligent agent, which synchronizes with the ICU vital signs monitoring system to obtain real-time postoperative recovery data and dynamically adjusts the sensitivity of eye movement recognition to adapt to the dynamic changes in the patient's eye and limb movement abilities. The sixth step involves the system automatically archiving the communication records of this diagnosis and treatment, encrypting and storing them on the local terminal, and automatically destroying the data daily. These records are used solely for medical staff to assess the patient's level of consciousness and changes in their condition, providing data support for critical care.

[0020] Example 2

[0021] Reference Figure 1 An AI-assisted communication system, specifically designed for neurological stroke patients with aphasia, is implemented as follows: This embodiment is applied to aphasia patients with stroke in the neurology department. These patients exhibit pathological speech impairments such as disordered speech, missing vocabulary, fragmented speech, and unclear gestures. They possess basic expressive abilities but their expression is not standardized. The system needs to simultaneously meet the needs of in-hospital communication and phased professional rehabilitation. Therefore, the system adopts a full-modal acquisition, fully intelligent agent collaboration, and dynamic algorithm iteration operation mode. The specific implementation steps are as follows: The first step is to activate the full-dimensional hardware acquisition module, which uses a high-sensitivity noise-reducing voice acquisition module, a facial expression recognition camera, a gesture sensing terminal, and a limb micro-motion sensor to capture all non-standardized pathological expressive signals such as fragmented speech, disordered facial expressions, and blurred gestures from the patient. The second step involves the pathological perception agent performing noise reduction, fusion, and incomplete repair preprocessing on multi-source collected data to remove environmental interference noise and improve incomplete semantic data. The third step involves the five intelligent agents to perform fully collaborative parallel reasoning. The incomplete semantic parsing intelligent agent fully utilizes the pathological incomplete semantic reconstruction algorithm, and combines the stroke clinical exclusive dataset with the diagnosis and treatment context to dynamically adjust the weight coefficients of α, β, and γ to accurately reconstruct the patient's fragmented expression of real diagnosis and treatment needs. The fourth step involves the doctor-patient twin-adapted intelligent agent relying on a dual-end digital twin model. On the one hand, it organizes the patient's pathological and disordered expressions into standardized and structured diagnostic and treatment texts and pushes them to the medical staff terminal. On the other hand, it transforms medical and nursing professional consultations, medication guidance, and rehabilitation orders into simple short sentences, voice prompts, and graphic prompts to adapt to the patient's cognitive level. The fifth step involves the system automatically archiving daily doctor-patient communication data and intelligently generating personalized language rehabilitation training tasks such as short sentence pronunciation, vocabulary practice, and gesture coordination based on the patient's communication deficiencies and expression weaknesses. The sixth step involves the intelligent agent, which iterates weekly by connecting with the hospital's EMR system and rehabilitation assessment data. Based on the patient's progress in language function recovery and rehabilitation score, the agent adaptively optimizes the algorithm's recognition accuracy and model parameters to achieve dynamic adaptation and synchronous iteration between diagnosis and treatment communication and professional rehabilitation within the hospital.

[0022] Example 3

[0023] Reference Figure 1 An AI-assisted communication system, specifically designed for home-based rehabilitation of elderly individuals with cognitive impairment, is implemented as follows: This embodiment is applied to patients with Alzheimer's disease and cognitive decline requiring home care. These patients have weak cognitive comprehension and disordered logical expression, and lack professional medical care for real-time support. The home setting emphasizes simple communication assistance and cognitive maintenance; therefore, the system adopts a lightweight hardware, minimalist interaction, and low-frequency iteration operation mode. The specific implementation steps are as follows: The first step is to adapt the system to a lightweight home use scenario by simplifying the hardware configuration, retaining only the noise-canceling voice acquisition module and the simple touch display terminal, and removing the high-precision professional medical sensing module to meet the needs of simple home operation. The second step involves the hardware acquiring simple expressive signals from the patient's daily speech and touch operations in real time to complete basic noise reduction and signal preprocessing. The third step involves the incomplete semantic parsing agent calling upon a knowledge base of common language corpus specific to elderly people with cognitive impairment to lower the semantic parsing accuracy threshold, adapt to the confused and ambiguous daily expression characteristics of patients, and accurately identify the daily needs of patients. The fourth step involves a two-way adaptive interaction module that transforms information into simplified language, converting everyday family communication scripts and remote follow-up medical orders into minimalist content with large fonts, short sentences, and a combination of text and images, thus reducing the difficulty for patients to understand. The fifth step involves the system automatically recording home communication data daily and generating simple cognitive communication training tasks based on the patient's cognitive decline status to help the patient maintain language and cognitive abilities in the long term. The sixth step involves the system iterating and optimizing the model at a low rate each month, and periodically synchronizing the locally encrypted communication data to the community medical terminal for medical staff to remotely assess the patient's cognitive and communication status, thereby achieving low-cost, routine home-based communication assistance and cognitive rehabilitation care.

[0024] In summary, the ICU critical care scenario prioritizes privacy and security, and silent emergency medical communication, with minimal hardware configuration, real-time data destruction, and no rehabilitation training functions. The in-hospital rehabilitation scenario in the Department of Neurology focuses on the accurate identification of pathological disability expressions and the integration of diagnosis, treatment and rehabilitation. It utilizes all hardware and algorithms, iterates models frequently, and has strong medical professionalism. The home care scenarios for the elderly are centered on easy operation, cognitive care, and routine use. The hardware is lightweight, the interaction is simple, the model iteration frequency is low, and it is precisely adapted to the core needs of different scenarios.

[0025] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An AI-assisted communication system, comprising a multimodal pathology perception hardware layer, a medical privacy edge computing layer, a multi-agent AI algorithm layer, a medical knowledge enhancement RAG layer, a bidirectional adaptive interaction service layer, and a clinical closed-loop application layer, characterized in that: The multi-agent AI algorithm layer has five built-in collaborative scheduling agents, namely, pathological perception agent, incomplete semantic parsing agent, doctor-patient twin adaptation agent, disease linkage and iteration agent, and risk control and compliance agent. The system completes the entire AI inference process offline at the local edge node, following a six-step closed-loop operation process of "perception and collection - intelligent analysis - two-way adaptation - interactive output - data iteration - rehabilitation intervention", to achieve two-way adaptive intelligent communication between patients with communication barriers and medical staff.

2. The AI-assisted communication system according to claim 1, characterized in that: The multimodal pathological perception hardware layer integrates a high-sensitivity noise-reducing voice acquisition module, a facial expression recognition camera, a flexible gesture sensing terminal, an eye-tracking module, and a limb micro-motion sensor; it enables differentiated hardware acquisition combinations for different types of patients, such as those with aphasia, cognitive impairment, ICU intubation, and hearing impairment, to complete the acquisition of fragmented and non-standardized pathological expression signals from patients.

3. The AI-assisted communication system according to claim 1, characterized in that: The medical privacy edge computing layer establishes a local offline edge inference node, with built-in medical data desensitization module, hierarchical permission control module, and data dynamic destruction module; all patient perception data, pathology data, and AI inference operations are processed locally without uploading to the cloud, and local inference achieves millisecond-level interactive response, adapting to medical scenarios without network and with high confidentiality.

4. The AI-assisted communication system according to claim 1, characterized in that: The incomplete semantic parsing agent is equipped with a pathological incomplete semantic reconstruction algorithm, the algorithm formula of which is: In the formula, To standardize the semantic results of reconstruction; The patient's original incomplete semantic vector; Pathological matching weights for disease type and disorder type; A factor for modifying the diagnostic and treatment context; For dynamic weighting coefficients, satisfying The weighting coefficients are adaptively adjusted in real time by the disease-linking iterative intelligent agent based on the patient's recovery stage.

5. The AI-assisted communication system according to claim 1, characterized in that: The doctor-patient twin-adaptive intelligent agent has a built-in dual-track digital twin model, namely a patient communication behavior twin model and a medical staff diagnosis and treatment script twin model. The patient-side twin model learns the patient's pathological expression habits and cognitive level, while the medical staff-side twin model matches clinical consultation and medical order standards. The two models work together in real time to complete the bidirectional adaptive conversion of doctor-patient dialogue.

6. The AI-assisted communication system according to claim 1, characterized in that: The medical knowledge enhancement RAG layer builds a vertical medical communication knowledge base that integrates pathological characteristics, clinical needs, and standard diagnosis and treatment communication norms for stroke, cognitive impairment, intensive care intubation, and hearing impairment. It adopts retrieval enhancement generation technology, which retrieves medical knowledge to anchor semantics during AI reasoning, eliminates the illusion of large models, and supports hospitals and departments to customize and enter their own diagnosis and treatment norms.

7. The AI-assisted communication system according to claim 1, characterized in that: The bidirectional adaptive interactive service layer enables bidirectional content conversion; for patients, it transforms medical terminology and complex medical orders into simple voice and text / image sentences; for medical staff, it organizes patients' fragmented expressions into standardized and structured text of their medical needs; and it supports multi-form interactive output including voice, text, images / images, gestures, and eye movements.

8. The AI-assisted communication system according to claim 1, characterized in that: The clinical closed-loop application layer integrates seven functional modules: real-time doctor-patient communication, patient request reporting, medical order translation, communication data archiving, personalized rehabilitation training, communication ability assessment, and communication quality review. It is adaptable to all medical scenarios, including outpatient, inpatient, ICU, rehabilitation, community healthcare, and home rehabilitation, forming an integrated service closed loop of communication, diagnosis and treatment, and rehabilitation.

9. The AI-assisted communication system according to claim 1, characterized in that: The disease-linking iterative intelligent agent connects with the hospital's HIS, EMR electronic medical records, and rehabilitation management system to synchronize patient disease type, disease course, treatment plan, and rehabilitation effect data in real time. It automatically adjusts the weight, interaction mode, and interaction complexity of the incomplete semantic parsing algorithm to achieve dynamic iterative optimization of the AI ​​model according to the patient's condition.

10. The AI-assisted communication system according to claim 1, characterized in that: All AI agents and large models in the system are lightweighted through model distillation and medical knowledge trimming. Relying on edge local nodes, multiple agents can perform parallel collaborative offline inference without relying on cloud networks. At the same time, the risk control and compliance agent verifies the compliance of communication content and the privacy and security of data transmission in real time.

Images