Physical intelligent agent physical examination auscultation device

Abstract

This invention discloses an embodied intelligent physical examination auscultation device, relating to the field of medical robot technology. The invention includes a processor and a memory. The device comprises: a multimodal perception and positioning module for acquiring spatial information of the patient's body surface and determining the spatial coordinates of the target auscultation site; and an adaptive auscultation head fitting module for driving the end-effector auscultation head to move and fit against the target auscultation site according to the spatial coordinates. The advantages are: this invention integrates visual anatomical positioning, acoustic feedback pressure optimization, spatial array directional noise reduction, cardiopulmonary sound blind source separation, anatomical topology network collaborative reasoning, and historical baseline longitudinal comparison into one system. This solves the problems of existing auscultation devices relying on manual operation and lacking multi-site correlation diagnostic capabilities. It achieves full-process intelligentization from autonomous auscultation head fitting to structured report generation, significantly improving auscultation accuracy, diagnostic consistency, and clinical interpretability.

Description

Technical Field

[0001] This invention relates to the field of medical robot technology, and in particular to a personal intelligent physical examination and auscultation device. Background Technology

[0002] Physical examination and auscultation are fundamental clinical diagnostic methods. Doctors use a stethoscope to listen to heart and lung sounds at multiple specific anatomical sites on the patient's chest and back, and make a comprehensive judgment on heart and lung diseases based on the sound characteristics of each site. This process is highly dependent on the doctor's operating experience and auscultation skills, including precise positioning of the stethoscope head, appropriate control of the contact pressure, and comprehensive analysis and experience-based judgment of the auscultation results from multiple sites.

[0003] While existing electronic stethoscopes have achieved digital acquisition and amplification of sound signals, they still require manual operation by doctors and cannot solve the problems of automatic positioning of the auscultation site and adaptive adjustment of the fitting pressure. Some robot-assisted auscultation solutions that have emerged in recent years typically use robotic arms to move the auscultation head according to preset coordinates, lacking the ability to provide individualized anatomical positioning based on real-time vision, making it difficult to adapt to different body shapes and positions. Furthermore, in terms of sound processing, existing solutions mostly use single-channel filtering for noise reduction, failing to fully utilize the spatial information of multi-channel arrays for noise source localization and suppression. In terms of separating heart and lung sounds, they usually rely on frequency domain filtering, which is difficult to handle situations where heart and lung sounds overlap in the time and frequency domains.

[0004] Therefore, there is a need to design an embodied intelligent physical examination and auscultation device. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a holistic intelligent physical examination and auscultation device, which solves the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A personal intelligent physical examination and auscultation device includes a processor and a memory, the device comprising: The multimodal sensing and localization module is used to acquire spatial information of the patient's body surface and determine the spatial coordinates of the target auscultation site; The stethoscope head adaptive fitting module is used to drive the end stethoscope head to move and fit against the target auscultation site according to the spatial coordinates; The sound acquisition and processing module, integrated into the stethoscope head, is used to synchronously acquire sound signals from the body surface and to perform noise reduction and separation of heart and lung sound components on the acquired sound signals. The correlation diagnosis reasoning module is used to extract features from the separated heart sound and lung sound components, and to perform correlation and comprehensive analysis by combining features from multiple auscultation sites to output diagnostic results. The diagnostic report generation module is used to convert the diagnostic results into a structured medical text report.

[0007] Furthermore, the sound acquisition and processing module includes a multi-channel pickup unit and a blind source separation processing unit arranged in a preset spatial layout; The blind source separation processing unit is used to estimate the spatial azimuth angle of the environmental noise source by utilizing the spatial distribution information of the multi-channel pickup unit, and uses the spatial azimuth angle as a spatial prior to construct a directional constrained beamformer to form a spatial notch null in the direction of the noise source and maintain the gain in the direction of body surface auscultation.

[0008] Furthermore, the sound acquisition and processing module also includes a body surface contact sensor for detecting whether the stethoscope head is in contact with the body surface; When adhesion is detected, the blind source separation processing unit switches to noise suppression mode to perform noise source localization and directional suppression; When a misalignment is detected, switch to omnidirectional pickup mode.

[0009] Furthermore, the associated diagnostic reasoning module includes a shared encoder network, a heart sound feature extraction branch, a lung sound feature extraction branch, and a graph neural network reasoning unit; The shared encoder network receives heart sound components and lung sound components, extracts general acoustic feature representations through shared convolutional layers, and the heart sound feature extraction branch and lung sound feature extraction branch respectively draw out their respective task heads from the general acoustic feature representations, outputting heart sound feature vectors and lung sound feature vectors; The graph neural network inference unit takes the heart sound feature vectors and lung sound feature vectors from multiple auscultation sites as node inputs, aggregates the feature information of adjacent nodes through graph convolution, and outputs the diagnostic results.

[0010] Furthermore, the shared encoder network, the heart sound feature extraction branch, the lung sound feature extraction branch, and the graph neural network inference unit are jointly trained using a multi-task loss function, which is a weighted sum of the heart sound feature extraction loss, the lung sound feature extraction loss, and the diagnostic inference loss.

[0011] Furthermore, the multimodal perception and localization module includes a depth camera unit and a human anatomical key point detection unit, used to output the three-dimensional spatial coordinates of auscultatory landmarks including at least one of the midclavicular line, anterior axillary line and apical pulsation point.

[0012] Furthermore, the stethoscope head adaptive fitting module includes a force sensing unit and an admittance control unit. The force sensing unit is installed at the connection between the stethoscope head and the robotic arm to measure the contact force between the stethoscope head and the body surface in real time. The admittance control unit receives the contact force signal and the target pressure value and outputs fine-tuning displacement commands according to the admittance control model.

[0013] Furthermore, the associated diagnostic reasoning module is also used to connect with the hospital information system through the communication interface unit, obtain the historical auscultation feature records of the same patient, calculate the deviation vector between the current feature and the historical baseline, and use the deviation vector as the reasoning input.

[0014] Furthermore, the associated diagnostic reasoning module also includes a respiratory phase synchronization triggering unit, which is used to receive the patient's chest movement sequence and extract the respiratory phase curve, using the inspiratory phase start point and the expiratory phase start point as trigger boundaries, and to segment the lung sound components according to the respiratory cycle and extract phase-related features.

[0015] Furthermore, the diagnostic report generation module includes a medical large language model reasoning unit, which receives natural language description sequences transformed from auscultation findings and diagnostic results, and outputs a formatted auscultation report containing a description of auscultation findings, potential pathological association analysis, and suggestions for further examination.

[0016] Compared with existing technologies, the advantages of this invention are: 1. By combining a multimodal perception and positioning module with a depth camera and a human anatomical key point detection network, the auscultation landmarks are automatically identified and their spatial coordinates are output. Then, the force sensing and admittance control unit of the auscultation head adaptive fitting module executes the automatic pressure optimization design based on acoustic signal-to-noise ratio feedback. This achieves accurate autonomous positioning and optimal acoustic coupling of the auscultation head on the patient's body surface, which has the advantages of eliminating differences in manual operation and ensuring consistent auscultation conditions in various parts of the body.

[0017] 2: By integrating multiple pickup units and body surface contact sensors through the sound acquisition and processing module, a directional constrained beamformer is constructed using multi-channel spatial distribution information in the contact state to directionally trap environmental noise. Combined with a blind source separation algorithm, the design realizes the separation of heart sound and lung sound components, achieving high-quality sound signal acquisition in complex sound field environments. It has the benefits of significantly improving the auscultation signal-to-noise ratio and the accuracy of heart and lung sound separation.

[0018] 3: By constructing a shared encoder network through the associated diagnostic reasoning module, the joint extraction of cardiopulmonary sound features is realized. Furthermore, the design of graph convolution aggregation reasoning for features from multiple sites is carried out using an anatomical topology graph neural network based on the relationship between blood flow connectivity and airway connectivity. This realizes a paradigm shift from independent judgment at a single point to collaborative diagnosis of multiple sites, which has the benefits of capturing abnormal transmission patterns across sites and improving diagnostic accuracy.

[0019] 4: By connecting to the hospital information system through the correlation diagnosis reasoning module to obtain the patient's historical auscultation feature records, calculating the deviation vector between the current feature and the historical baseline, and fusing it with the input graph neural network, the design realizes the automatic modeling of the patient's individualized normal variation baseline and longitudinal comparison of the disease course. It has the benefits of sensitively reflecting the trend of disease evolution and reducing the risk of misjudgment caused by individual differences.

[0020] 5: The diagnostic report generation module utilizes a medical big language model to transform diagnostic labels, abnormal scores, and auscultation findings into a formatted report that includes auscultation findings descriptions, pathological correlation analysis, and suggestions for further investigation. This achieves an end-to-end mapping from numerical features to medical semantics, which improves the standardization and clinical operability of reports.

[0021] In summary, this invention integrates visual anatomical localization, acoustic feedback pressure optimization, spatial array directional noise reduction, cardiopulmonary sound blind source separation, anatomical topology network collaborative reasoning, and historical baseline longitudinal comparison into one system. This solves the problems of existing auscultation devices relying on manual operation and lacking multi-site correlation diagnostic capabilities. It realizes intelligent operation of the entire process from the autonomous fitting of the auscultation head to the generation of structured reports, significantly improving auscultation accuracy, diagnostic consistency, and clinical interpretability. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of an embodied intelligent physical examination and auscultation device proposed in this invention; Figure 2 This is a flowchart illustrating the operation of a embodied intelligent physical examination and auscultation device proposed in this invention. Detailed Implementation

[0023] Reference Figures 1-2 A personal intelligent physical examination and auscultation device is used to be deployed in medical service robots or automated physical examination terminals. The device includes a processor and a memory. The memory stores a computer program. When the processor calls and executes the computer program, it realizes the functions of each module. It also includes a system control bus, which is connected to the multimodal sensing and positioning module, the auscultator head adaptive fitting module, the sound acquisition and processing module, the correlation diagnosis reasoning module, and the diagnosis report generation module, and is responsible for coordinating the startup sequence, working mode switching and abnormal state handling of each module.

[0024] The multimodal perception and localization module includes a depth camera unit and a human anatomical key point detection unit. The depth camera unit acquires three-dimensional spatial information of the patient's body surface, including three-dimensional point cloud data and color-depth images. The human anatomical key point detection unit embeds a pre-trained deep learning network, receives the three-dimensional spatial information of the body surface, and outputs the three-dimensional spatial coordinates of auscultatory landmarks in real time through semantic recognition of skeletal anatomical structures such as the thorax, clavicle, and costal arch. Auscultatory landmarks include at least one of the following: the midclavicular line, the anterior axillary line, and the apex of the heart.

[0025] During the stethoscope head motion planning phase, the multimodal perception and localization module also transmits the patient's three-dimensional point cloud to the motion planning unit of the stethoscope head adaptive fitting module. The motion planning unit uses the point cloud as a safety constraint boundary and the auscultation markers as target points to plan a three-dimensional spatial transfer path within the configuration space that avoids collisions with the patient's surface. After path planning is completed, the multi-degree-of-freedom robotic arm transports the end auscultation head from the initial position to above the target auscultation site along the path to complete coarse positioning.

[0026] The stethoscope head adaptive fitting module includes a force sensing unit, an admittance control unit, and a multi-degree-of-freedom robotic arm. The force sensing unit is installed at the connection between the end stethoscope head and the robotic arm. During the fitting stage, it measures the contact force between the stethoscope head and the body surface in real time and feeds it back to the admittance control unit at a millisecond frequency.

[0027] The admittance control unit incorporates an automatic pressure optimization mechanism based on acoustic quality feedback, and its specific working process is as follows: The robotic arm first drives the stethoscope head to fit against the body surface with a preset initial contact force, so that the stethoscope head and the body surface form an acoustically sealed cavity; Subsequently, the sound acquisition and processing module begins to acquire sound signals and calculates the acoustic signal-to-noise ratio of the currently acquired signals in real time; The admittance control unit gradually adjusts the target contact force within a preset pressure range using fine-tuning steps, and simultaneously records the acoustic signal-to-noise ratio corresponding to each pressure value. After completing the pressure scan, the pressure value with the best acoustic signal-to-noise ratio is selected as the target pressure value for that auscultation site. Subsequently, the admittance control unit performs dynamic calculations based on the real-time contact force signal and target pressure value according to the preset admittance control model, and outputs fine-tuning displacement commands for the robotic arm, thereby driving the stethoscope head to stably adhere to the patient's body surface with the optimized target pressure, avoiding the stethoscope head from detaching or being subjected to excessive pressure due to the patient's breathing fluctuations or slight displacement.

[0028] The sound acquisition and processing module is integrated inside the stethoscope head. This module includes a body surface contact sensor, a multi-channel pickup unit arranged in a preset spatial layout, and a blind source separation processing unit. The multi-channel pickup unit is a microphone array composed of no less than three independent pickup sensors, preferably a microelectromechanical system microphone, arranged in an equilateral triangle or cross topological geometry at the top of the acoustic sealed cavity inside the stethoscope head. To further improve the accuracy of environmental noise location estimation, the multi-channel pickup unit also includes at least one air reference pickup unit installed on the outer side of the stethoscope head housing. This air reference pickup unit faces away from the human skin and is used to independently collect environmental noise reference signals that are conducted through the air to the vicinity of the stethoscope head.

[0029] The body surface contact sensor is used to detect whether the stethoscope head is in contact with the body surface. When the stethoscope head is not in contact with the body surface, such as when the robotic arm is in the spatial transfer stage, the blind source separation processing unit switches to the omnidirectional sound pickup mode. In this mode, the system is mainly used to evaluate the ambient sound field inside the examination room, monitor the overall sound pressure level of the background noise and perform baseline modeling. When the body surface contact sensor detects that the stethoscope head is in close contact with the body surface, the blind source separation processing unit switches to noise suppression mode. In noise suppression mode, the blind source separation processing unit synchronously links the air reference pickup unit and the microphone array in the cavity. Using the spatial distribution information of the multi-channel pickup unit and the transmission delay difference between the solid conduction path and the air conduction path, the unit performs cross-correlation calculation on the acquired multi-channel raw signals to accurately estimate the spatial azimuth of the environmental noise source. Subsequently, a directional constrained beamformer is constructed using this spatial azimuth angle as spatial prior information to create a spatial notch null in the direction of the noise source, while maintaining high gain in the direction perpendicular to the auscultation principal axis of the body surface. Based on this, the blind source separation processing unit uses a blind source separation algorithm to further purify and separate independent heart sound and lung sound components from the multi-channel audio stream.

[0030] The separated heart sound and lung sound components are synchronously input into the association diagnosis reasoning module, which contains a shared encoder network, a heart sound feature extraction branch, and a lung sound feature extraction branch.

[0031] The shared encoder network receives heart sound and lung sound components and extracts common low- and mid-order acoustic feature representations through multiple shared convolutional layers. This sharing mechanism enables the network to learn the physiological acoustic background of coexisting heart and lung sounds, improving the model's generalization ability. The heart sound feature extraction branch and the lung sound feature extraction branch each draw their own independent task heads from the above-mentioned common acoustic feature representations, focusing on mining their own specific high-order pathological features, and outputting heart sound feature vectors and lung sound feature vectors respectively.

[0032] During the model development phase, this multi-task network is jointly trained using a multi-task loss function, and the overall loss function L_total is calculated using the following formula: L_total=α·L_heart+β·L_lung+γ·L_diag In the formula, L_heart is the heart sound feature extraction loss, L_lung is the lung sound feature extraction loss, L_diag is the diagnostic inference loss, and α, β, and γ are preset weight coefficients. In terms of form, the heart sound feature extraction loss L_heart and the lung sound feature extraction loss L_lung are calculated using cross-entropy loss or mean squared error loss, respectively, according to their corresponding disease classification task or abnormal severity regression task. The diagnostic inference loss L_diag is calculated based on the comprehensive diagnostic label output by the graph neural network. Joint training enables the shared encoder to simultaneously meet the representation requirements of multiple tasks.

[0033] To address the technical characteristic that lung sounds are easily affected by the respiratory cycle, the correlation diagnosis reasoning module also includes a respiratory phase synchronization triggering unit. This unit receives the patient's chest movement time series captured by the depth camera unit, extracts the respiratory phase curve, and uses the inspiratory phase start and expiratory phase start in the curve as trigger boundaries to segment the continuous lung sound components according to the respiratory cycle. The lung sound feature extraction branch independently extracts phase-related feature parameters for each segmented segment, thereby accurately capturing dry and wet rales or wheezing sounds that only appear in specific respiratory phases.

[0034] The core reasoning logic of the correlation diagnosis reasoning module is implemented by the graph neural network reasoning unit. In the actual auscultation process, the embodied intelligent auscultation device drives the auscultation head to inspect multiple normal auscultation sites such as the front of the chest and the back.

[0035] The graph neural network inference unit uses multiple auscultation sites as nodes in the topology graph, and uses the extracted heart sound feature vectors and lung sound feature vectors from each node as node initialization features. The auscultation association graph construction unit constructs physically meaningful graph edges between corresponding graph nodes based on the blood flow connectivity or airway connectivity between each site in medical anatomy. Blood flow connectivity is, for example, the blood flow conduction path between each valve area, and airway connectivity is, for example, the extended branches of the trachea, main bronchus, and lobar bronchus.

[0036] The graph neural network inference unit dynamically aggregates the feature information of adjacent nodes along the edges of the aforementioned anatomical graph through graph convolution operations, and outputs the diagnostic results. The diagnostic results include at least one of the following: the type of cardiac abnormality, the type of lung abnormality, the location of the abnormality, and the severity score of the abnormality. This explicit integration of medical prior knowledge into graph network reasoning enables the system not only to identify abnormalities in a single part, but also to perform global multi-part collaborative diagnosis.

[0037] In addition, the correlation diagnosis reasoning module also interacts with the hospital information system through the communication interface unit to obtain the historical auscultation feature records of the same patient. After receiving the currently collected auscultation feature vector, the module reads the corresponding historical auscultation feature vector and calculates the deviation vector between the current feature and the historical baseline. After obtaining the deviation vector, the graph neural network inference unit concatenates the feature vector extracted from the current auscultation site with the deviation vector in the channel dimension, or adds them together by weighting elements, thereby fusing them into a composite node feature vector containing information on the longitudinal course of the disease. This composite node feature vector serves as the final initialization input for the corresponding node in the graph neural network. The introduction of this deviation vector enables the diagnostic results to sensitively reflect the development and evolution trend of the patient's condition.

[0038] The diagnostic report generation module receives the diagnostic results output by the associated diagnostic reasoning module. The structured description generation unit within the module first converts the diagnostic labels, abnormal scores, and auscultation findings into natural language description sequences that conform to medical standards. Subsequently, the medical large language model reasoning unit (a pre-trained language model based on the Transformer architecture) receives the natural language description sequence as prompt words and uses its medical text generation and reasoning capabilities to output a formatted auscultation report. The report fully includes a description of auscultation findings, potential pathological correlation analysis, and suggestions for further investigation, and can be directly integrated into the electronic medical record system.

[0039] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A personal intelligent physical examination and auscultation device, comprising a processor and a memory, characterized in that, The device includes: The multimodal sensing and localization module is used to acquire spatial information of the patient's body surface and determine the spatial coordinates of the target auscultation site; The stethoscope head adaptive fitting module is used to drive the end stethoscope head to move and fit against the target auscultation site according to the spatial coordinates; The sound acquisition and processing module, integrated into the stethoscope head, is used to synchronously acquire sound signals from the body surface and to perform noise reduction and separation of heart and lung sound components on the acquired sound signals. The correlation diagnosis reasoning module is used to extract features from the separated heart sound and lung sound components, and to perform correlation and comprehensive analysis by combining features from multiple auscultation sites to output diagnostic results. The diagnostic report generation module is used to convert the diagnostic results into a structured medical text report.

2. The embodied intelligent physical examination and auscultation device according to claim 1, characterized in that, The sound acquisition and processing module includes a multi-channel pickup unit and a blind source separation processing unit arranged in a preset spatial layout. The blind source separation processing unit is used to estimate the spatial azimuth angle of the environmental noise source by utilizing the spatial distribution information of the multi-channel pickup unit, and uses the spatial azimuth angle as a spatial prior to construct a directional constrained beamformer to form a spatial notch null in the direction of the noise source and maintain the gain in the direction of body surface auscultation.

3. The embodied intelligent physical examination and auscultation device according to claim 2, characterized in that, The sound acquisition and processing module also includes a body surface contact sensor for detecting whether the stethoscope head is in contact with the body surface; When adhesion is detected, the blind source separation processing unit switches to noise suppression mode to perform noise source localization and directional suppression; When a misalignment is detected, switch to omnidirectional pickup mode.

4. The embodied intelligent physical examination and auscultation device according to claim 1, characterized in that, The associated diagnostic reasoning module includes a shared encoder network, a heart sound feature extraction branch, a lung sound feature extraction branch, and a graph neural network reasoning unit; The shared encoder network receives heart sound components and lung sound components, extracts general acoustic feature representations through shared convolutional layers, and the heart sound feature extraction branch and lung sound feature extraction branch respectively draw out their respective task heads from the general acoustic feature representations, outputting heart sound feature vectors and lung sound feature vectors; The graph neural network inference unit takes the heart sound feature vectors and lung sound feature vectors from multiple auscultation sites as node inputs, aggregates the feature information of adjacent nodes through graph convolution, and outputs the diagnostic results.

5. The embodied intelligent physical examination and auscultation device according to claim 4, characterized in that, The shared encoder network, the heart sound feature extraction branch, the lung sound feature extraction branch, and the graph neural network inference unit are jointly trained using a multi-task loss function, which is a weighted sum of the heart sound feature extraction loss, the lung sound feature extraction loss, and the diagnostic inference loss.

6. The embodied intelligent physical examination and auscultation device according to claim 1, characterized in that, The multimodal perception and localization module includes a depth camera unit and a human anatomical key point detection unit, which is used to output the three-dimensional spatial coordinates of auscultatory landmarks including at least one of the midclavicular line, anterior axillary line and apex beat point.

7. The embodied intelligent physical examination and auscultation device according to claim 1, characterized in that, The auscultatory head adaptive fitting module includes a force sensing unit and an admittance control unit. The force sensing unit is installed at the connection between the auscultatory head and the robotic arm to measure the contact force between the auscultatory head and the body surface in real time. The admittance control unit receives the contact force signal and the target pressure value and outputs fine-tuning displacement commands according to the admittance control model.

8. The embodied intelligent physical examination and auscultation device according to claim 1, characterized in that, The associated diagnostic reasoning module is also used to connect with the hospital information system through the communication interface unit, obtain the historical auscultation feature records of the same patient, calculate the deviation vector between the current feature and the historical baseline, and use the deviation vector as the reasoning input.

9. The embodied intelligent physical examination and auscultation device according to claim 1, characterized in that, The associated diagnostic reasoning module also includes a respiratory phase synchronization triggering unit, which is used to receive the patient's chest movement sequence and extract the respiratory phase curve. Using the inspiratory phase start point and the expiratory phase start point as trigger boundaries, the lung sound components are segmented according to the respiratory cycle and phase-related features are extracted.

10. A embodied intelligent physical examination and auscultation device according to claim 1, characterized in that, The diagnostic report generation module includes a medical large language model reasoning unit, which receives natural language description sequences of auscultation findings and diagnostic results, and outputs a formatted auscultation report containing a description of auscultation findings, potential pathological correlation analysis, and suggestions for further examination.

Images