A visual guidance-based mechanical arm mouth care, breath detection and holographic diagnosis and treatment auxiliary system and method

Abstract

The application discloses a kind of based on visual guidance's mechanical arm mouthwash nursing, breath detection and holographic diagnosis and treatment auxiliary system and method, it is related to stomatology field, including: main control module, auxiliary mechanical arm module, gas detection module, visual sensing module, diagnosis and treatment intelligent body interaction module, audio and video acquisition communication module with main control module electricity connection, and multi-modal mobile nursing terminal.The application realizes the pose control of mechanical arm and the imaging visual angle adjustment of diagnosis and treatment intelligent body by visual sensing system, integrates supine position automatic mouthwash nursing, exhalation component disease screening, doctor-patient / holographic visualization communication, full-process audio and video diagnosis and treatment support, non-standard oral speech driven standard medical record automatic generation, automatically collects non-standard oral speech content in diagnosis and treatment full process, generates standardization medical record meeting medical quality control requirement, greatly improves diagnosis and treatment efficiency, patient comfort and diagnosis and treatment standardization, and can be widely integrated in dental treatment chair.

Description

Technical Field

[0001] This invention relates to the field of oral medicine technology, and more specifically to a vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic and treatment auxiliary system and method. Background Technology

[0002] In modern oral medicine, the dental chair is the core piece of equipment, typically consisting of a dental chair, instrument tray, surgical light, and side-mounted spittoon. However, existing dental chairs still have the following problems: On the one hand, during procedures such as ultrasonic scaling, tooth preparation, and root canal treatment, cooling water, saliva, blood, and tissue debris continuously accumulate in the patient's mouth. Current techniques for managing oral fluid accumulation primarily involve two methods: one is passive suction using a weak / strong suction tube held by a healthcare professional in the patient's mouth. However, negative pressure suction is insufficient to completely remove fine debris and can easily attract soft tissue, causing discomfort and pain. The other method involves rinsing the mouth with a fixed spittoon. This requires the patient to frequently switch from a supine to a sitting position to spit water into the spittoon, severely disrupting the continuity of treatment. After repositioning, the doctor needs to readjust the chair, operating light, microscope, and other equipment, significantly reducing treatment efficiency. Furthermore, patients with lumbar spine disorders, the elderly, pregnant women, and patients under sedation cannot independently coordinate position changes, posing risks of accidental swallowing, choking, and spillage.

[0003] On the other hand, volatile sulfur compounds, acetone, aldehydes, and other components in oral exhalation are important diagnostic indicators for oral diseases and systemic metabolic diseases. However, the rinsing systems of existing dental chairs only have cleaning and waste removal functions and cannot complete the standardized collection and analysis of exhaled breath components during the natural treatment process of rinsing and exhaling. Early disease screening requires an additional independent testing process, increasing the number of treatment steps and the burden on patients.

[0004] In addition, the existing dental treatment process has significant shortcomings in doctor-patient communication, medical technology collaboration, treatment operation assistance, and medical record writing: doctors can only explain treatment plans to patients through language and hand-drawn diagrams, resulting in low patient understanding of dental lesions and treatment processes, and poor treatment compliance; communication between doctors and technicians regarding restoration plans can only be completed through offline data and text descriptions, leading to errors in information transmission and easy rework of restorations; during the treatment process, doctors cannot easily access technical support content such as operating procedures and complication warnings.

[0005] Meanwhile, the pace of dental clinic treatment is fast, with a single dental chair seeing an average of 15-20 patients per day. Doctors spend over 30% of their daily work time writing medical records, significantly encroaching on clinical treatment time and increasing their workload. Existing voice-based medical record systems can only recognize standardized medical record terminology spoken by doctors and cannot adapt to the natural expression habits of dentists during treatment. Doctors often use colloquial, fragmented, and non-standardized expressions during consultations and procedures, including daily conversations with patients, self-talk during procedures, and collaborative communication with assistants. Current technology cannot accurately understand, summarize, and structure this non-standardized content. Doctors still need to manually organize and complete medical records after treatment, failing to achieve true automated medical record generation and making it difficult to guarantee the consistency and quality control of medical record writing.

[0006] In summary, current dental diagnostic aids have not yet achieved the organic integration of automated mouthwash care, breath composition analysis, intelligent interactive diagnosis and treatment, full-process audio and video support, and automatic generation of non-standardized spoken language-driven medical records. Therefore, how to provide comprehensive auxiliary services throughout the entire dental diagnostic and treatment process through a single system to meet the demands of modern oral healthcare for efficiency, intelligence, and standardization is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic assistance system and method, which integrates supine automated mouthwash care, breath component disease screening, holographic visual communication between doctors and patients / medical technicians, full-process audio-visual diagnostic assistance, and automatic generation of standard medical records driven by non-standardized spoken language into a single system. Through a single set of visual guidance, the system achieves synergy between robotic arm control and intelligent interaction. It can automatically collect non-standardized spoken language content from the entire diagnostic and treatment process without requiring doctors to change their existing treatment habits, generating standardized medical records that meet quality control requirements. This solves the core technical problems of fragmented diagnostic and treatment processes, low patient comfort, inefficient doctor-patient communication, heavy burden of medical record documentation for doctors, and reliance on standardized spoken language in existing voice medical records, thus achieving intelligent, efficient, and standardized dental treatment processes.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: This invention discloses a vision-guided robotic arm mouthwash care, breath detection, and holographic diagnosis and treatment auxiliary system, comprising: a main control module, an auxiliary robotic arm module, a gas detection module, a vision sensing module, a diagnosis and treatment intelligent agent interaction module, an audio and video acquisition and communication module, and a multimodal mobile nursing terminal, all electrically connected to the main control module. The fixed end of the auxiliary robotic arm module is fixedly installed on the column or lamp arm base of the dental chair. The multimodal mobile nursing terminal is fixedly installed on the end joint of the auxiliary robotic arm module. It reaches the target position through the multi-degree-of-freedom movement of the auxiliary robotic arm to perform waste collection, mouthwash water supply, and breath sampling. The gas detection module performs quantitative analysis on the components of the exhaled breath samples collected by the multimodal mobile nursing terminal; The visual sensing module acquires images of the patient's facial and oral cavity postures, recognizes user gesture commands, and transmits them to the main control module. The intelligent diagnostic and treatment interaction module visualizes the diagnostic and treatment data to the patient and enables human-computer interaction. The audio and video acquisition and communication module collects audio, video, and voice interaction data throughout the entire diagnosis and treatment process and transmits it to the main control module. The main control module calculates the target motion trajectory of the auxiliary robotic arm based on the data output by the visual sensing module, and adjusts the imaging angle and display content of the diagnostic intelligent agent interaction module; based on the audio and video and voice interaction data collected by the audio and video acquisition and communication module, it performs real-time content processing and generates electronic medical records.

[0009] Furthermore, the multimodal mobile nursing terminal is fixedly connected to the end joint of the auxiliary robotic arm module via a tray; the multimodal mobile nursing terminal integrates a sputum cup, a rinsing nozzle, an air blowing nozzle, a water hose, and an air hose, the sputum cup, rinsing nozzle, and air blowing nozzle are arranged circumferentially along the rotation center of the tray, the water hose is connected to the water supply and sewage pipes of the dental chair, and the air hose is connected to the gas detection module; the working positions of the sputum cup, rinsing nozzle, and air blowing nozzle are switched by rotating the end joint.

[0010] Furthermore, the visual sensing module is a depth camera that updates the three-dimensional pose data of the patient's oral cavity in real time at a frequency of 30Hz. During the exhalation sampling process, the gas detection module monitors the carbon dioxide concentration of the sampled gas in real time using a carbon dioxide sensor. When the concentration is lower than a preset threshold, it is determined that the sampled gas has been diluted, and a voice prompt for resampling is issued through a speaker.

[0011] Furthermore, the intelligent diagnostic interaction module employs at least one of a medium-free aerial imaging module, a holographic projection module, or a touch screen. When using a medium-free aerial imaging module, the module's light output direction is directly facing the patient's face and the doctor's operating position, respectively, presenting a medium-free 3D dental model, treatment plan animation, real-time detection data, and medical record generation preview in the air. When using a holographic projection module, a miniature holographic chassis and a holographic imaging film are provided to output interactive 3D holographic images. When using a touch screen, it is a foldable, rotatable, sterile, waterproof medical touch screen fixed to the movable support arm of the dental chair.

[0012] Furthermore, the main control module incorporates a robotic arm control unit, which performs coordinate system transformation, posture constraint control, end-effector mode switching control, visual servo trajectory planning, and safety impedance control. The posture constraint control ensures that the normal vector of the spittoon remains parallel to the opposite direction of gravity during the movement of the robotic arm. The end-effector mode switching control switches the tool center point TCP between the center of the spittoon, the tip of the gargling nozzle, and the tip of the blowing nozzle, according to the working mode.

[0013] Furthermore, the main control module has a built-in AI diagnosis and treatment intelligent agent unit, which includes a diagnosis and treatment process guidance subunit, a doctor-patient communication visualization subunit, a medical technology collaboration subunit, a diagnosis and treatment operation assistance subunit, an audio and video archiving subunit, and an intelligent medical record automatic generation subunit. The diagnosis and treatment process guidance subunit automatically pushes corresponding process nodes, key operation points and time node prompts according to the diagnosis and treatment type and progress; The patient-doctor communication visualization sub-unit presents the patient's oral scan data, CBCT images, 3D model of the teeth and jaws, treatment plan animation, prognosis comparison data and cost details through the intelligent treatment agent interaction module. The medical technology collaboration subunit is used for real-time screen sharing, 3D annotation and audio-visual communication between the doctor's end and the technician's end, and synchronously pushes oral scan data, restoration design plan, implant guide parameters and occlusal relationship data. The diagnostic and treatment operation assistance subunit pushes corresponding prompts based on the current diagnostic and treatment step. The audio and video archiving subunit automatically records, edits, generates summaries, encrypts and archives, and retrieves audio and video recordings of the diagnosis and treatment process. The intelligent medical record automatic generation subunit processes the non-standardized spoken language content acquired by the audio and video acquisition and communication module in real time, and completes speech transcription, spoken language semantic disambiguation, diagnosis and treatment entity recognition, fragmented information integration and summarization, standardized structure conversion, and automatically fills it into the corresponding standard medical record template to generate an electronic medical record.

[0014] Furthermore, the medical semantic processing engine includes: a spoken language preprocessing module, a medical semantic disambiguation module, a diagnosis and treatment entity normalization module, a fragmented information context integration module, and a medical record structure generation module; The spoken language preprocessing module performs filtering of interjections, removal of duplicate content, sentence segmentation correction, and homophony correction on the transcribed spoken language text to obtain the diagnosis and treatment text content. The medical semantic disambiguation module, based on a dental corpus, maps colloquial expressions to medical terms, eliminating semantic discrepancies between everyday speech and medical terminology. The diagnostic entity normalization module extracts key diagnostic entities from the text and completes entity classification and standardized mapping. The fragmented information context integration module performs timeline association and contextual semantic association on fragmented information at different stages of the entire diagnosis and treatment process, merges duplicate information, completes logical chains, and forms a complete diagnosis and treatment information chain. The medical record structure generation module automatically fills the integrated diagnosis and treatment information into the standard medical record template corresponding to the diagnosis and treatment category to generate electronic medical records.

[0015] Furthermore, the standard medical record templates include: templates for dental endodontics, periodontitis, prosthodontics, dental implants, orthodontics, oral surgery, and pediatric dentistry; each template sets required fields, optional fields, and quality control rules to adapt to the medical record writing requirements of different treatment scenarios.

[0016] This invention also discloses a vision-guided robotic arm method for mouthwash care, breath detection, and holographic diagnostic assistance, comprising: S0: System standby; S1: Pre-treatment breath test, doctor-patient communication and pre-extraction of medical record information process; S2: The process of mouthwash care during diagnosis and treatment, intelligent agent assistance in diagnosis and treatment, and real-time generation of medical records; S3: Post-treatment medical record final review, education and archiving process.

[0017] As can be seen from the above technical solution, compared with the prior art, the present invention provides a vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic auxiliary system and method, which has the following beneficial effects: This invention helps improve the continuity of the treatment process and the efficiency of chairside operations. Through a vision-guided robotic arm, it enables automatic mouth rinsing care in the supine position, eliminating the need for patients to frequently change positions and doctors to repeatedly adjust treatment equipment. Single cleaning and tooth preparation operations reduce the number of patient position changes and doctors' repeated equipment adjustments, significantly increasing chair turnover. At the same time, the integrated functions of breath detection, doctor-patient communication, and medical technology collaboration eliminate the need for additional treatment steps, further reducing the overall treatment time.

[0018] This invention helps improve the patient's experience of rinsing their mouth while lying supine and reduces the risks of liquid spillage and choking. For patients with lumbar spine diseases, the elderly, pregnant women, and special patients under sedation, it completely eliminates the physiological burden of getting up to rinse the mouth and the inconvenience of changing positions, reducing the risk of accidental swallowing, choking, and liquid splashing. At the same time, the safe impedance control and compliant motion planning of the robotic arm avoid hard collision injuries and greatly improve the patient's treatment experience.

[0019] This invention significantly improves the efficiency of doctor-patient communication and treatment compliance. Through 3D visualization using aerial imaging / holographic projection, abstract dental lesions and treatment plans are presented intuitively to patients, which helps improve patients' understanding of the treatment plan, greatly enhances the efficiency of informed consent communication, and significantly improves patients' treatment compliance and satisfaction.

[0020] This invention significantly reduces doctors' paperwork burden and comprehensively improves the quality control of medical records. Through the intelligent automatic medical record generation function, it can automatically collect non-standard spoken content from the entire treatment process without doctors needing to change their treatment habits or verbally recite standardized medical record terminology. This generates standardized electronic medical records that comply with national standards, reducing the workload of doctors manually organizing and entering medical records after treatment. It completely solves the industry pain point of heavy medical record writing burden for dental clinic doctors. At the same time, the three-level quality control verification mechanism helps to improve the structuring degree of medical records and the consistency of quality control.

[0021] This invention enhances the standardization and capability of diagnosis and treatment. The AI-powered intelligent diagnostic and treatment unit provides doctors with prompts for standardized operation throughout the entire process, early warnings of complications, and technical support, reducing operational risks and improving the standardization of diagnosis and treatment. At the same time, the remote consultation and real-time medical technology collaboration functions enable efficient linkage of diagnostic and treatment resources, reduce the rework rate of prostheses, and improve the quality of diagnosis and treatment.

[0022] This invention features highly integrated equipment that improves the utilization rate of treatment space. By integrating functions such as mouthwash care, breath detection, doctor-patient communication, audio and video recording, treatment assistance, and automatic medical record generation into the same system, it replaces multiple independent devices, significantly reduces the occupation of treatment space, avoids interference of scattered devices with the aseptic operation flow, and is suitable for dental clinics of all sizes.

[0023] This invention upgrades the management and teaching capabilities of dental clinics, with automatic recording and encrypted archiving of audio and video throughout the entire process. It can be used for quality control of diagnosis and treatment, tracing medical disputes, clinical teaching and popular science promotion, providing a complete solution for the digital management and talent training of dental clinics. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the overall structure provided for an embodiment of the present invention.

[0026] Figure 2This is a schematic diagram of the structure of the multimodal mobile nursing terminal provided in an embodiment of the present invention; Figure 3 This is a block diagram of the gas detection module provided in an embodiment of the present invention; Figure 4 A block diagram of the module composition of the intelligent medical record automatic generation subunit provided in the embodiments of the present invention; Figure 5 This is a flowchart illustrating the steps of a pre-diagnosis breath test and doctor-patient communication process provided in an embodiment of the present invention. Figure 6 This is a flowchart illustrating the steps of the mouthwash care and real-time medical record generation process provided in an embodiment of the present invention. Figure 7 This is a schematic diagram of coordinate system transformation for robotic arm control provided in an embodiment of the present invention.

[0027] Figure 8 This is a schematic diagram of the working position switching of the multimodal mobile nursing terminal provided in an embodiment of the present invention.

[0028] In the diagram, 1. Diagnostic and treatment intelligent interaction module; 2. Visual sensing module; 3. Main control module; 4. Auxiliary robotic arm module; 5. Multimodal mobile nursing terminal; 6. Sputum cup; 7. Tray; 8. Mouthwash nozzle; 9. Water hose; 10. Air hose; 11. Air blowing nozzle. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] This invention discloses a vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic assistance system, such as... Figure 1 As shown, it includes: a main control module, an auxiliary robotic arm module, a gas detection module, a vision sensing module, a diagnostic and treatment intelligent agent interaction module, an audio and video acquisition and communication module, and a multimodal mobile nursing terminal, all electrically connected to the main control module; The fixed end of the auxiliary robotic arm module is fixedly installed on the column or lamp arm base of the dental chair; the load capacity is ≥1kg and the repeatability is ≤±0.1mm. It is used to drive the multimodal mobile nursing terminal at the end to complete multi-degree-of-freedom spatial movement to meet the service needs of different body positions and different tooth positions.

[0031] Specifically, the auxiliary robotic arm module uses a six-degree-of-freedom collaborative robotic arm with a rated load of 1kg, a repeatability of ±0.05mm, and an IP54 protection rating. It is fixedly installed on the left column of the dental chair via an adapter flange. The robotic arm's range of motion completely covers the oral cavity area of ​​the patient in a supine position and will not interfere with other parts of the dental chair.

[0032] The multimodal mobile nursing terminal is fixedly installed on the end joint of the auxiliary robotic arm module. It reaches the target position through the multi-degree-of-freedom movement of the auxiliary robotic arm to perform waste collection, mouthwash water supply, and breath sampling. The gas detection module, integrated into the main control module, includes a volatile sulfide sensor, an acetone sensor, an aldehyde sensor, and a carbon dioxide sensor, which performs quantitative analysis on the components of exhaled breath samples collected by the multimodal mobile nursing terminal. The visual sensing module is fixedly installed next to the dental chair, covering the patient-bearing area of ​​the dental chair. It collects images of the patient's face and oral cavity posture, recognizes user gesture commands (including breath detection trigger, contamination trigger, water supply trigger, stop, model rotation, model scaling, previous page, next page), and transmits them to the main control module. The intelligent diagnostic agent interaction module visualizes diagnostic data for patients and facilitates human-computer interaction. The audio and video acquisition and communication module includes a high-definition camera for the surgical area, a noise-canceling microphone array, a speaker, and a remote communication unit. It collects audio, video, and voice interaction data throughout the entire diagnosis and treatment process and transmits it to the main control module. The noise-canceling microphone array is used to collect all non-standardized spoken voice content during the diagnosis and treatment process, including doctor-patient consultations, doctors' self-talk during operations, and medical staff collaborative communication.

[0033] Specifically, the high-definition surgical area camera can be integrated into the end effector of the surgical lamp or auxiliary robotic arm to collect real-time high-definition images of the intraoral surgical area; the noise-reducing microphone array uses an array of 2 or more microphones, with a pickup distance of 0.5-2m and a noise reduction depth of ≥30dB, to collect all non-standardized spoken language content of doctor-patient consultations, doctors' self-talk during operations, and medical staff collaborative communication throughout the entire diagnosis and treatment process, and is also used for voice interaction, prompt tone broadcasting, and real-time calls; the remote communication unit supports 5G / Ethernet / WiFi communication for real-time data and audio / video transmission for medical technology collaboration and remote consultation.

[0034] The main control module, based on the data (pose data and commands) output by the vision sensing module, calculates the target motion trajectory of the auxiliary robotic arm, drives the multimodal mobile nursing terminal to reach the target pose, adjusts the imaging angle and display content of the intelligent medical agent interaction module to match the observation perspective of the patient and doctor; and performs real-time content processing and generates electronic medical records based on the audio and video and voice interaction data collected by the audio and video acquisition and communication module.

[0035] Specifically, the main control module adopts an NXP i.MX8MP industrial-grade embedded controller, which has a built-in quad-core Cortex-A53 processor and an 8-TOPS computing power NPU unit, runs a Linux real-time operating system, and has a built-in robotic arm control unit and AI diagnosis and treatment intelligent agent unit. It communicates with the robotic arm, vision sensing module and various functional modules via Ethernet, with a control cycle of 1ms. Among them, the intelligent medical record automatic generation sub-unit has a built-in oral specialty large language model based on Llama3 fine-tuning. The training corpus includes 1 million standardized medical records from oral clinics, oral specialty textbooks, guidelines and terminology sets, which can achieve accurate semantic conversion of non-standard spoken content.

[0036] Specifically, the main control module adopts an industrial-grade embedded controller with a built-in NPU computing unit. It is electrically connected to the auxiliary robotic arm module, vision sensing module, gas detection module, diagnostic intelligent agent interaction module, and audio / video acquisition and communication module, respectively, and is used to perform equipment control, data processing, algorithm calculation, and full-process scheduling. The main control module integrates an AI diagnostic intelligent agent unit and a robotic arm control unit.

[0037] In one specific embodiment, the multimodal mobile nursing terminal is fixedly connected to the end joint of the auxiliary robotic arm module via a tray; such as Figure 2 As shown, the multimodal mobile nursing terminal integrates a sputum cup, a rinsing nozzle, an air blowing nozzle, a water hose, and an air hose. The sputum cup, rinsing nozzle, and air blowing nozzle are arranged circumferentially along the rotation center of the tray. The water hose is connected to the water supply and drainage pipes of the dental chair, and the air hose is connected to the gas detection module. Figure 8 As shown, the working positions of the sputum cup, gargling nozzle, and blowing nozzle can be switched by rotating the end joint, eliminating the need for a complex multi-arm structure, greatly reducing the size of the equipment, and avoiding interference with the treatment space.

[0038] Specifically, the water inlet of the water hose is connected to the clean water supply line of the dental chair, the outlet is connected to the mouthwash nozzle, the waste outlet is connected to the bottom of the sputum cup, and the return water is connected to the negative pressure waste outlet line of the dental chair; the air inlet of the air hose is connected to the blowing nozzle, the outlet is connected to the sampling port of the gas detection module, and the air hose has a built-in micro diaphragm pump for quantitative collection of exhaled breath samples.

[0039] In one specific embodiment, the visual sensing module is a depth camera that updates the three-dimensional pose data of the patient's oral cavity in real time at a frequency of 30Hz. The depth camera has a frame rate ≥30Hz and a depth measurement accuracy ≤±2mm. It is used to acquire three-dimensional pose images of the patient's face and oral cavity, recognize user-preset gesture trigger commands, and provide spatial coordinates for robotic arm control and imaging perspective adjustment.

[0040] Gas detection module, such as Figure 3 As shown, the enclosed chassis integrated within the main control module contains a volatile sulfur compound sensor for oral diseases, an acetone sensor and an aldehyde sensor for systemic metabolic diseases, and a carbon dioxide sensor for sampling quality monitoring. The outputs of each sensor are electrically connected to the AD acquisition unit of the main control module for quantitative analysis of the collected patient exhaled breath components, outputting corresponding concentration data and health assessment results. During the exhaled breath sampling process, the gas detection module monitors the carbon dioxide concentration of the sampled gas in real time using the carbon dioxide sensor. When the concentration falls below a preset threshold, it is determined that the sampled gas has been diluted, and a voice prompt for resampling is issued via a speaker.

[0041] In one specific embodiment, the diagnostic and treatment intelligent interaction module employs at least one of a medium-free aerial imaging module, a holographic projection module, or a touch screen. When using a medium-free aerial imaging module, the module's light output direction is directly facing the patient's face and the doctor's operating position, respectively, presenting a medium-free 3D dental model, treatment plan animation, real-time detection data, and medical record generation preview in the air. When using a holographic projection module, a miniature holographic chassis and a holographic imaging film are provided to output interactive 3D holographic images. When using a touch screen, it is a foldable, rotatable, sterile, waterproof medical touch screen fixed to the movable support arm of the dental chair.

[0042] In one specific embodiment, the main control module integrates a robotic arm control unit, which performs coordinate system transformation, attitude constraint control, end-effector mode switching control, visual servo trajectory planning, and safety impedance control; wherein: like Figure 7 As shown, coordinate system transformation is used to transform between the world coordinate system, camera coordinate system, robot arm base coordinate system, end flange coordinate system, and tool center point coordinate system. The extrinsic parameter matrices of the camera and robot arm base are obtained through pre-calibration, and the target pose of the patient's oral cavity in the robot arm base coordinate system is calculated. The coordinate system transformation is pre-calibrated using hand-eye alignment to obtain the extrinsic parameter matrix between the camera coordinate system and the robot arm base coordinate system. After the visual sensing module identifies the three-dimensional coordinates of the patient's oral cavity, its target pose in the robot arm base coordinate system is calculated using the following formula: ;in, The extrinsic parameter matrix for camera and base calibration. The pose of the oral cavity in the camera coordinate system. A safety distance compensation transformation matrix is ​​set along a preset safety direction to prevent the device from colliding with the patient's face.

[0043] Attitude constraint control is used in the path planning of a robotic arm's motion to introduce attitude constraints on a spittoon. It stipulates that the normal vector of the spittoon's opening, in the base coordinate system, must always remain parallel to the opposite direction of gravity. This ensures the spittoon remains horizontal during movement, preventing liquid spillage. In the path planning of the robotic arm's motion, attitude constraint control introduces attitude constraints on the spittoon, stipulating that the normal vector of the spittoon's opening, in the base coordinate system, must always remain parallel to the opposite direction of gravity, satisfying the constraint condition: ;in, Let be the vector of the spittoon normal in the base coordinate system. The gravitational acceleration vector ensures that the spittoon remains horizontal during movement, preventing liquid from spilling out.

[0044] The end-effector mode switching control switches the tool center point TCP between the center of the sputum cup, the tip of the rinsing nozzle, and the tip of the blowing nozzle, depending on the working mode, corresponding to precise positioning in the soil collection mode, water supply mode, and breath detection mode. In the soil collection mode, the TCP is defined at the center of the sputum cup opening, and the robotic arm's movement target is to move the TCP to the soil collection position 1cm below the patient's mouth. In the water supply / breath detection mode, the TCP switches to the tip of the rinsing / blowing nozzle. The robotic arm keeps joints 1-5 stationary, while independently driving the 6th joint at the end to rotate, switching the nozzle to the working position. Simultaneously, coordinate transformation compensates for the eccentric displacement caused by the rotation, ensuring the nozzle is accurately aligned with the patient's mouth.

[0045] Visual servo trajectory planning is used to dynamically correct the movement trajectory of the robotic arm based on the patient's oral cavity pose updated in real time by the visual sensing module, so as to achieve dynamic tracking of the patient's head micro-movements. At the same time, an S-shaped velocity curve is used for acceleration and deceleration planning to limit the maximum acceleration and ensure smooth and stable movement.

[0046] The visual sensing module updates the patient's oral cavity posture in real time at a frequency of 30Hz. When the patient's head makes a slight movement, the robotic arm corrects the target point in real time to achieve dynamic following. The acceleration and deceleration of the robotic arm movement adopts an S-shaped velocity curve planning, with the maximum acceleration limited to 500mm / s³, ensuring a smooth and stable start-stop process. At the same time, Cartesian space impedance control is activated. When the external force on the end of the robotic arm exceeds the safety threshold of 5N, it will yield in the direction of the external force to avoid causing harm to the patient.

[0047] Safety impedance control is used to control the force at the end of the robotic arm in Cartesian space. When the external force on the end of the robotic arm exceeds a preset safety threshold, it will yield in the direction of the external force to avoid causing hard collision injury to the patient.

[0048] In one specific embodiment, the main control module has a built-in AI diagnosis and treatment intelligent agent unit, which includes a diagnosis and treatment process guidance subunit, a doctor-patient communication visualization subunit, a medical technology collaboration subunit, a diagnosis and treatment operation assistance subunit, an audio and video archiving subunit, and an intelligent medical record automatic generation subunit. The diagnosis and treatment process guidance sub-unit automatically pushes corresponding process nodes, key operation points and time node prompts according to the diagnosis and treatment type and progress; The patient communication visualization sub-unit retrieves and presents the patient's oral scan data, CBCT images, 3D models of the teeth and jaws, animations of treatment plans, prognostic comparison data and cost details through the intelligent diagnostic agent interaction module. The medical technology collaboration sub-unit is used for real-time screen sharing, 3D annotation, and audio-visual communication between doctors and technicians, and synchronously pushes oral scan data, restoration design plans, implant guide parameters, and occlusal relationship data. The diagnostic and treatment operation assistance subunit has a built-in database of oral diagnostic and treatment operation standards, complication management plans, and instrument usage guidelines. It is used to push corresponding prompts based on the current diagnostic and treatment steps, including operation standards, instrument usage prompts, and complication warning information. It also supports voice commands to trigger the retrieval and display of corresponding content. The audio and video archiving subunit automatically records, edits, generates summaries, encrypts and archives, and retrieves audio and video recordings of the diagnosis and treatment process. It also supports patients to scan codes to obtain diagnosis and treatment summaries and postoperative education content. The intelligent medical record automatic generation sub-unit has a built-in standard medical record template library for all categories of oral diagnosis and treatment, a fine-tuned large language model for oral specialties, and a medical semantic processing engine. It is used to process non-standardized spoken language content acquired by the audio and video acquisition and communication module in real time, and complete speech transcription, spoken language semantic disambiguation, diagnosis and treatment entity recognition, fragmented information integration and summarization, and standardized structure conversion. It automatically fills the corresponding standard medical record template and generates electronic medical records that comply with national medical quality control standards. The entire process does not require doctors to verbally recite standardized medical record terminology or change their original diagnosis and treatment habits.

[0049] In one specific embodiment, the medical semantic processing engine includes: a spoken language preprocessing module, a medical semantic disambiguation module, a diagnosis and treatment entity normalization module, a fragmented information context integration module, and a medical record structure generation module; The spoken language preprocessing module filters out interjections, removes duplicate content, corrects punctuation, and corrects homophones in the transcribed spoken language text to obtain fluent medical text content. The medical semantic disambiguation module, based on a dental corpus, maps colloquial expressions to medical terms, eliminating semantic discrepancies between everyday speech and medical terminology. The diagnostic entity normalization module extracts key diagnostic entities from the text, including patient complaints, present medical history, past medical history, allergy history, oral examination results, imaging conclusions, diagnostic results, treatment procedures, medication, and postoperative medical orders, and completes entity classification and standardized mapping. The fragmented information context integration module performs timeline association and contextual semantic association on fragmented information at different stages of the entire diagnosis and treatment process, merges duplicate information, completes logical chains, and forms a complete diagnosis and treatment information chain; The medical record structuring generation module automatically fills the integrated standardized diagnosis and treatment information into the standard medical record template corresponding to the diagnosis and treatment category, generating a complete structured electronic medical record.

[0050] Specifically, such as Figure 4 As shown, the core algorithm flow of the intelligent medical record automatic generation subunit is as follows: Step 1: Real-time speech transcription: Using the Whisper dental specialist fine-tuned speech model, real-time speech collected by the noise-canceling microphone array is transcribed with a word error rate of ≤3% and supports dialect accent adaptation. Step 2: Colloquial text preprocessing: Filtering out interjections and repetitive content, correcting homophones and punctuation errors, and obtaining standardized text; Step 3: Medical semantic disambiguation and entity extraction: Using a large language model for oral medicine, semantic understanding of the text is performed to extract key diagnostic entities and complete the mapping of colloquial expressions to medical terminology; Step 4: Fragmented information integration: Based on the timeline and contextual semantics, integrate the entity information of the entire diagnosis and treatment process, complete the logical chain, and remove duplicate information; Step 5: Structured medical record generation: The integrated information is automatically filled into the corresponding standard medical record template to generate a complete electronic medical record; Step 6: Level 3 Quality Control Verification: Verify the completeness, logical compliance, and quality control standards of the generated medical records, and provide correction suggestions.

[0051] In one specific embodiment, the intelligent medical record automatic generation subunit has a standard medical record template library, which contains a full range of treatment templates that conform to the "Basic Specifications for Medical Record Writing" and the "Specifications for Medical Record Writing in Dental Clinics". The standard medical record templates include: dental pulp disease, periodontal disease, dental restoration, dental implant, orthodontics, oral surgery, and pediatric dental treatment templates. Each template has required fields, optional fields, and quality control rules to adapt to the medical record writing requirements of different treatment scenarios.

[0052] This invention also discloses a vision-guided robotic arm method for mouthwash care, breath detection, and holographic diagnostic assistance, comprising: S0: System standby, specifically including: the auxiliary robotic arm is parked in a safe position behind the dental chair, the intelligent treatment agent interaction module is in a low-power standby state, the visual sensing module continuously monitors scene trigger signals; the intelligent medical record automatic generation sub-unit preloads the standard medical record template corresponding to the treatment subject, starts the noise reduction microphone array to pick up sound at all times, and waits for the treatment process to be triggered.

[0053] S1: Pre-treatment breath test, doctor-patient communication, and pre-extraction of medical record information process, such as... Figure 5 As shown, it specifically includes: S11: When the visual sensing module recognizes a preset trigger gesture or receives a voice trigger command, the system enters the breath detection mode. S12: The visual sensing module acquires images of the patient's face, calculates the three-dimensional pose of the patient's oral cavity, and the main control module drives the auxiliary robotic arm to move the air nozzle to a preset sampling position near the patient's oral cavity. S13: The system issues an exhalation prompt through a speaker, collects the patient's exhalation through the blowing nozzle and airway hose, and transmits it to the gas detection module to complete the quantitative analysis of exhalation components; S14: After the test is completed, the intelligent diagnostic agent interaction module presents the test results and health interpretation, and the main control module drives the auxiliary robotic arm to return to the safe position; S15: The system enters the doctor-patient communication mode. The intelligent treatment agent interaction module presents the patient's 3D model of teeth and jaws, CBCT images and treatment plan. Doctors can use gestures or voice to control the rotation and zoom of the model to complete the visual explanation of the treatment plan. S16: The intelligent medical record automatic generation subunit collects non-standard spoken dialogue content from doctor-patient consultations in real time, completes speech transcription and semantic processing, automatically extracts key information such as the patient's chief complaint, present illness, past medical history, and allergy history, and pre-fills it into the corresponding columns of the standard medical record template; the diagnosis and treatment intelligent agent interaction module displays the medical record generation preview in sync, and supports doctors to quickly correct it through voice commands.

[0054] S2: The process of mouthwash care during diagnosis and treatment, intelligent agent assistance in diagnosis and treatment, and real-time generation of medical records, such as... Figure 6 As shown, it specifically includes: S21: When the visual sensing module recognizes the gesture that triggers the rinsing service, the system enters the rinsing service mode; S22: The visual sensing module collects the patient's oral cavity posture in real time, and the main control module drives the auxiliary robotic arm to move the spittoon to the receiving position near the patient's oral cavity, waiting for the patient to spit out water; S23: After the waste collection is completed, the auxiliary robotic arm switches the working position by rotating the end joint, and moves the mouthwash nozzle to the water supply position near the patient's mouth to complete the quantitative water supply; S24: After the water supply is completed, the auxiliary robotic arm switches to the working position again and moves the spittoon to the sewage receiving position to complete the secondary sewage receiving. S25: During the diagnosis and treatment process, the AI ​​diagnosis and treatment intelligent agent unit pushes operation specifications, instrument prompts and complication warnings through the diagnosis and treatment intelligent agent interaction module according to the current diagnosis and treatment steps, and at the same time completes the full-process audio and video recording of the diagnosis and treatment process through the audio and video acquisition and communication module; S26: The intelligent medical record automatic generation sub-unit collects non-standardized spoken language content such as doctors' self-talk and medical staff collaborative communication during the operation process in real time, and completes semantic disambiguation, diagnosis and treatment entity extraction, fragmented information integration, and automatically extracts key information such as oral examination results, operation steps, instrument models, medication, and special situations during the operation, and fills them into the corresponding columns of the medical record template in real time, and simultaneously completes the timeline archiving of operation records. Example: A doctor's colloquial description during the procedure, "Tooth 46 distal proximal caries, caries removal, no pulp exposure, acid etching, bonding, resin filling, occlusion, polishing," is automatically converted by the system into a standardized record: "Oral examination: Tooth 46 distal proximal deep caries, no pulp exposure detected; Treatment procedure: Tooth 46 undergoes caries removal, acid etching, bonding agent application, light-cured resin filling, occlusion, and polishing." S27: When a medical technology collaboration or remote consultation instruction is received, the system establishes a real-time connection through the remote communication unit, and synchronously pushes surgical area images, 3D models of teeth and jaws and treatment data to achieve multi-party simultaneous annotation and audio-visual communication.

[0055] Furthermore, in steps S22-S24, during the movement of the auxiliary robotic arm, posture constraint control is consistently implemented to ensure that the normal vector of the sputum cup remains parallel to the opposite direction of gravity; simultaneously, an S-shaped velocity curve is used for acceleration and deceleration planning to limit the maximum acceleration and prevent liquid sloshing and spillage. In step S25, the system executes visual servo closed-loop control, with the visual sensing module updating the patient's oral cavity posture in real time. When the patient's head makes a slight movement, the auxiliary robotic arm corrects the target posture in real time to achieve dynamic following; at the same time, Cartesian space impedance control is activated, and when the external force on the end of the auxiliary robotic arm exceeds a preset threshold, it yields in the direction of the external force to avoid hard collision injury.

[0056] S3: Post-treatment medical record final review, education, and archiving process, specifically including: S31: After the diagnosis and treatment are completed, the intelligent medical record automatically generates sub-units to automatically complete the diagnostic results, treatment plan, postoperative medical orders, and follow-up visit plan information. It integrates the information collected throughout the process and performs contextual verification and logical validation to generate a complete and standardized electronic medical record. The record is then presented to the doctor for review through the intelligent medical agent interaction module, triggering a three-level quality control check: Level 1 check is a check for the completeness of required fields, providing reminders for missing core required fields; Level 2 check is a medical logic check, providing compliance prompts for the matching of the diagnosis and treatment plan and medication guidelines; Level 3 check is a quality control standard check, ensuring that the medical record complies with national medical record writing standards. Doctors can make modifications via voice or touch, and confirm and archive with one click. S32: The AI ​​diagnostic and treatment intelligent agent interaction module presents before-and-after treatment comparisons, postoperative precautions, and follow-up visit plans. The audio and video archiving sub-unit of the AI ​​diagnostic and treatment intelligent agent unit automatically generates audio and video summaries and complete archive files for diagnosis and treatment. Patients can obtain the summary content by scanning the code. S33: The system will synchronize the confirmed electronic medical record to the hospital's HIS / EMR system to complete the data loop of the entire process, and then reset to standby mode to wait for the next trigger.

[0057] Furthermore, in step S13, the gas detection module monitors the carbon dioxide concentration of the sampled gas in real time during the sampling process. When the CO2 increment of the concentration relative to the environmental baseline is lower than the preset difference, it is determined that the sample is diluted, and a voice prompt to exhale again is issued through the speaker until the sampling is qualified.

[0058] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0059] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic assistance system, characterized in that, include: The main control module, the auxiliary robotic arm module, the gas detection module, the vision sensing module, the diagnostic and treatment intelligent interaction module, the audio and video acquisition and communication module, and the multimodal mobile nursing terminal are electrically connected to the main control module; The fixed end of the auxiliary robotic arm module is fixedly installed on the column or lamp arm base of the dental chair. The multimodal mobile nursing terminal is fixedly installed on the end joint of the auxiliary robotic arm module. It reaches the target position through the multi-degree-of-freedom movement of the auxiliary robotic arm to perform waste collection, mouthwash water supply, and breath sampling. The gas detection module performs quantitative analysis on the components of the exhaled breath samples collected by the multimodal mobile nursing terminal; The visual sensing module acquires images of the patient's facial and oral cavity postures, recognizes user gesture commands, and transmits them to the main control module. The intelligent diagnostic and treatment interaction module visualizes the diagnostic and treatment data to the patient and enables human-computer interaction. The audio and video acquisition and communication module collects audio, video, and voice interaction data throughout the entire diagnosis and treatment process and transmits it to the main control module. The main control module calculates the target motion trajectory of the auxiliary robotic arm based on the data output by the visual sensing module, and adjusts the imaging angle and display content of the diagnostic intelligent agent interaction module. Based on the audio and video and voice interaction data collected by the audio and video acquisition and communication module, real-time content processing is performed and electronic medical records are generated.

2. The vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic auxiliary system according to claim 1, characterized in that, The multimodal mobile nursing terminal is fixedly connected to the end joint of the auxiliary robotic arm module via a tray. The multimodal mobile nursing terminal integrates a sputum cup, a rinsing nozzle, an air blowing nozzle, a water hose, and an air hose. The sputum cup, rinsing nozzle, and air blowing nozzle are arranged circumferentially along the rotation center of the tray. The water hose is connected to the water supply and drainage pipes of the dental chair, and the air hose is connected to the gas detection module. The working positions of the sputum cup, rinsing nozzle, and air blowing nozzle are switched by rotating the end joint.

3. The vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic auxiliary system according to claim 1, characterized in that, The visual sensing module is a depth camera that updates the three-dimensional pose data of the patient's oral cavity in real time at a frequency of 30Hz. During the exhalation sampling process, the gas detection module monitors the carbon dioxide concentration of the sampled gas in real time using a carbon dioxide sensor. When the concentration is lower than a preset threshold, it is determined that the sampled gas has been diluted, and a voice prompt for resampling is issued through a speaker.

4. The vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic auxiliary system according to claim 1, characterized in that, The intelligent diagnostic and treatment interaction module employs at least one of a medium-free aerial imaging module, a holographic projection module, or a touch screen. When using a medium-free aerial imaging module, the module's light output direction is directly facing the patient's face and the doctor's operating position, respectively, presenting a medium-free 3D dental model, treatment plan animation, real-time detection data, and medical record generation preview in the air. When using a holographic projection module, a miniature holographic chassis and a holographic imaging film are provided to output interactive 3D holographic images. When using a touch screen, it is a foldable, rotatable, sterile, waterproof medical touch screen fixed to the movable support arm of the dental chair.

5. The vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic auxiliary system according to claim 1, characterized in that, The main control module has a built-in robotic arm control unit, which performs coordinate system transformation, posture constraint control, end-effector mode switching control, visual servo trajectory planning, and safety impedance control. The posture constraint control is used to ensure that the normal vector of the spittoon remains parallel to the opposite direction of gravity during the movement of the robotic arm. The end-effector mode switching control switches the tool center point TCP between the center of the spittoon, the tip of the gargling nozzle, and the tip of the blowing nozzle according to the working mode.

6. The vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic auxiliary system according to claim 1, characterized in that, The main control module has a built-in AI diagnosis and treatment intelligent agent unit, which includes a diagnosis and treatment process guidance subunit, a doctor-patient communication visualization subunit, a medical technology collaboration subunit, a diagnosis and treatment operation assistance subunit, an audio and video archiving subunit, and an intelligent medical record automatic generation subunit. The diagnosis and treatment process guidance subunit automatically pushes corresponding process nodes, key operation points and time node prompts according to the diagnosis and treatment type and progress; The patient-doctor communication visualization sub-unit presents the patient's oral scan data, CBCT images, 3D model of the teeth and jaws, treatment plan animation, prognosis comparison data and cost details through the intelligent treatment agent interaction module. The medical technology collaboration subunit is used for real-time screen sharing, 3D annotation and audio-visual communication between the doctor's end and the technician's end, and synchronously pushes oral scan data, restoration design plan, implant guide parameters and occlusal relationship data. The diagnostic and treatment operation assistance subunit pushes corresponding prompts based on the current diagnostic and treatment step. The audio and video archiving subunit automatically records, edits, generates summaries, encrypts and archives, and retrieves audio and video recordings of the diagnosis and treatment process. The intelligent medical record automatic generation subunit processes the non-standardized spoken language content acquired by the audio and video acquisition and communication module in real time, and completes speech transcription, spoken language semantic disambiguation, diagnosis and treatment entity recognition, fragmented information integration and summarization, standardized structure conversion, and automatically fills it into the corresponding standard medical record template to generate an electronic medical record.

7. The vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic auxiliary system according to claim 6, characterized in that, The medical semantic processing engine includes: a spoken language preprocessing module, a medical semantic disambiguation module, a diagnosis and treatment entity normalization module, a fragmented information context integration module, and a medical record structure generation module; The spoken language preprocessing module performs filtering of interjections, removal of duplicate content, sentence segmentation correction, and homophony correction on the transcribed spoken language text to obtain the diagnosis and treatment text content. The medical semantic disambiguation module, based on a dental corpus, maps colloquial expressions to medical terms, eliminating semantic discrepancies between everyday speech and medical terminology. The diagnostic entity normalization module extracts key diagnostic entities from the text and completes entity classification and standardized mapping. The fragmented information context integration module performs timeline association and contextual semantic association on fragmented information at different stages of the entire diagnosis and treatment process, merges duplicate information, completes logical chains, and forms a complete diagnosis and treatment information chain. The medical record structure generation module automatically fills the integrated diagnosis and treatment information into the standard medical record template corresponding to the diagnosis and treatment category to generate electronic medical records.

8. The vision-guided robotic arm mouthwash care, breath detection, and holographic diagnostic auxiliary system according to claim 6, characterized in that, The standard medical record templates include: templates for dental endodontics, periodontitis, prosthodontics, dental implants, orthodontics, oral surgery, and pediatric dentistry. Each template has required fields, optional fields, and quality control rules to adapt to the medical record writing requirements of different treatment scenarios.

9. A vision-guided robotic arm method for mouth rinsing care, breath detection, and holographic diagnosis and treatment, applied to the vision-guided robotic arm method for mouth rinsing care, breath detection, and holographic diagnosis and treatment as described in any one of claims 1 to 8, characterized in that, include: S0: System standby; S1: Pre-treatment breath test, doctor-patient communication and pre-extraction of medical record information process; S2: The process of mouthwash care during diagnosis and treatment, intelligent agent assistance in diagnosis and treatment, and real-time generation of medical records; S3: Post-treatment medical record final review, education and archiving process.

Images