A virtual-real combined heart-lung auscultation teaching system and a control method thereof

Abstract

The application provides a virtual and real combined heart and lung auscultation teaching system and a control method thereof. The system comprises a physical simulation man subsystem, an intelligent stethoscope subsystem, a virtual and real interaction and communication subsystem and a virtual simulation teaching software subsystem. The physical simulation man chest is provided with a pressure sensing grid array, which collects an auscultation position and a pressing force, and calculates a two-dimensional coordinate and a normal pressure value of a contact surface through a gravity method. The intelligent stethoscope is provided with a dual-mode audio processing DSP, which supports two auscultation modes of pure virtual and mixed enhancement. The virtual simulation teaching software comprises a three-dimensional anatomical visualization engine, a heart and lung sound synthesis and playing engine and a teaching management and evaluation module, realizes accurate mapping of physical coordinates and a three-dimensional scene, independently controls a perspective factor according to a pressing force to drive the transparency of the front layer of tissues and the surface deformation, the teacher end can remotely switch pathological cases, monitor student operation trajectories in real time and give quantitative scores, and realizes real-time linkage of the auscultation position, the pressing force and the three-dimensional anatomical structure.

Description

Technical Field

[0001] This invention relates to the field of medical teaching simulation technology, specifically to a virtual-real integrated teaching system for training cardiopulmonary auscultation skills, and more particularly to an intelligent teaching device and method that can realize real-time linkage between auscultation position, pressure intensity, and three-dimensional anatomical structure. Background Technology

[0002] Cardiopulmonary auscultation is one of the core skills for clinicians during physical examinations, and its accuracy directly affects the preliminary diagnosis of diseases of the circulatory and respiratory systems. Currently, medical schools both domestically and internationally primarily rely on the following methods for teaching and training in cardiopulmonary auscultation skills: 1. Teaching methods based on standardized patients or real-person peer training: This method is closest to clinical practice, but has poor reproducibility. Trainees cannot hear abnormal pathological sounds in healthy individuals, and the occurrence of pathological sounds is sporadic, making it impossible to reproduce and reinforce training as needed. This results in low teaching efficiency and limited resources.

[0003] 2. Teaching methods based on pure audio or digital sound libraries: Trainees are trained by playing pre-recorded normal / abnormal heart and lung sounds. This method lacks a correspondence with human anatomical locations, and cannot train trainees' accurate auscultation location identification ability and operational feel, leading to the clinical predicament of "understanding the sounds but not being able to locate the positions accurately".

[0004] 3. Teaching methods based on traditional electronic mannequins: Existing electronic mannequins typically have a limited number of pre-set auscultation points on their surface, with audio playback modules pre-installed below each point. When the trainee places the stethoscope at that point, a fixed audio signal is triggered. While this approach provides a certain level of realism, its technical shortcomings are also obvious: (1) The position perception is rough: the auscultation points are fixed and sparse, and it is impossible to identify the positional changes of the stethoscope at non-preset points or the edge of the area, let alone perceive the continuous movement trajectory of the stethoscope on the body surface.

[0005] (2) Lack of pressure perception: unable to collect and provide feedback on the pressure applied by the stethoscope, which directly affects the sound quality of the auscultation and is an important part of auscultation skills.

[0006] (3) Disconnection of anatomical structure: The three-dimensional anatomical structure inside the chest (such as lung lobes, bronchial tree, and heart valve projection) is not visible, and trainees cannot establish a three-dimensional thinking of "surface location - internal structure - corresponding auscultation sound".

[0007] (4) Difficulty in switching pathological sounds: The switching of pathological sounds is mostly done manually by switching preset modes, which cannot achieve real-time linkage and seamless transition with the dynamic changes in the auscultation area.

[0008] In conclusion, existing technologies are insufficient to meet the demands of modern medical education for intelligent, precise, quantifiable, and highly immersive teaching methods. Summary of the Invention

[0009] To address the shortcomings of the existing technology, the main objective of this invention is to provide a virtual-real integrated cardiopulmonary auscultation teaching system to solve the following technical problems: (1) How to obtain the continuous three-dimensional position information and dynamic pressure information of the stethoscope on the simulated human chest with high precision and in real time.

[0010] (2) How to establish a precise mapping relationship between auscultation position, pressure and three-dimensional thoracic anatomical structure (including internal organs and body surface projection), and drive the three-dimensional model to perform synchronous visualization rendering in real time.

[0011] (3) How to intelligently and smoothly call up and play the corresponding normal or abnormal heart and lung sounds from a rich library of heart and lung sounds according to different auscultation positions and pressure, and achieve seamless transition of sound effects when switching pathological modes.

[0012] (4) How to construct a closed-loop teaching system that integrates quantitative evaluation of actions, dynamic switching of pathology, and traceability of the teaching process.

[0013] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: A virtual-real integrated cardiopulmonary auscultation teaching system includes: a physical simulator subsystem, an intelligent stethoscope subsystem, a virtual-real interaction and communication subsystem, and a virtual simulation teaching software subsystem. For example... Figure 1 As shown, the four subsystems interact with each other via a wireless communication protocol, forming a complete virtual-real integrated auscultation teaching system.

[0014] (a) Physical simulation human subsystem: The physical simulator is a model with surface landmarks and internal skeletal structure of the human chest cavity, and its chest surface is covered with a highly elastic biomimetic skin layer. Its core innovation lies in: Sensor array layout: Between the bionic skin layer and the internal rigid skeleton, a high-density flexible pressure sensing grid array composed of M×N (preferably 16×16) piezoresistive sensing units is arranged. The sensing units are evenly distributed in space, with a unit spacing of no more than 1.5 cm. For example... Figure 2 As shown, to facilitate later teaching assessment and location mapping, the system divides the aforementioned grid array into 12 functional regions based on commonly used clinical auscultation location logic, including: left apex, right apex, left upper lobe, right upper lobe, right middle lobe, left lower lobe, right lower lobe, aortic valve region, pulmonary valve region, aortic valve region II, tricuspid valve region, and apex of the heart. When the stethoscope contacts the body surface, multiple adjacent sensing units below the contact surface respond simultaneously. The system achieves sub-grid-level precision position interpolation by capturing the distribution pattern of this response cluster.

[0015] Position and Pressure Calculation: The system constructs a chest surface pressure distribution heatmap by collecting resistance changes from all sensor nodes. Using the center-of-gravity method, it calculates the stethoscope's two-dimensional coordinates (x, y) on the body surface and the normal pressure value (F) of the contact surface in real time. As the stethoscope moves, the system continuously tracks its movement trajectory by tracing the displacement of the center of gravity in the pressure heatmap.

[0016] Formula for calculating two-dimensional coordinates (x, y) (using the centroid method as an example): ; ; in Let be the resistance change (or pressure value) of the i-th triggered sensor. Let be the physical coordinates of the i-th sensor.

[0017] The formula for calculating the normal pressure value F is: ; Where k is the pressure-resistance conversion coefficient calibrated experimentally. Let be the relative change in resistance of the i-th sensor, and n be the total number of sensors that are triggered.

[0018] (II) Intelligent Stethoscope Subsystem: The intelligent stethoscope includes a pickup head, a sound tube, and an ear hook, all with the same shape as a regular medical stethoscope. Based on the traditional stethoscope structure, it integrates a miniature inertial measurement unit for auxiliary calibration and stable positioning. Simultaneously, its internally integrated pickup converts the collected physical vibration signals into digital audio signals, which can be used for mixing and enhancement with a virtual sound source or as a sound source in pure operation mode.

[0019] The intelligent stethoscope also integrates an analog-to-digital converter (ADC) and a digital signal processor (DSP). For example... Figure 5 As shown, the DSP has built-in dual-mode audio processing logic: (a) Pure Virtual Mode: Completely shields the signal from the physical microphone and plays only the sound source from the virtual simulation software; (b) Hybrid Enhancement Mode: The physical vibration signal acquired by the physical pickup is processed by bandpass filtering (center frequency range: 20Hz~2000Hz) and adaptive noise gate, and then superimposed with the virtual sound source in the time domain. The hybrid output formula is: ; in The mixing coefficient (0~1) is adjustable on the teacher's end. This is the delay compensation value introduced after system delay measurement (used to ensure synchronization of virtual and real audio). This refers to the real-time audio signal acquired by a physical microphone and processed by bandpass filtering and adaptive noise thresholding. Virtual audio source signals generated / played by virtual simulation software. The final output is a mixed audio signal to the student, where t is a time variable.

[0020] (III) Virtual-Real Interaction and Communication Subsystem: This subsystem uses low-power Bluetooth or Wi-Fi wireless communication protocols to transmit position and pressure data collected from the physical mannequin, as well as posture and audio data collected from the intelligent stethoscope, to the virtual simulation teaching software subsystem in real time with low latency (<100ms). The communication data packet format includes: timestamp, device ID, pressure array data, and IMU attitude angle data.

[0021] This subsystem adopts a C / S (client / server) network architecture, with physical mannequins and intelligent stethoscopes serving as client devices, and virtual simulation teaching software deployed on the server side. The server supports multi-session concurrent management, with each student's device corresponding to an independent communication session. The server routes data and isolates states by parsing the device ID field in the data packets. The teacher's software, as a specially authorized client, can subscribe to real-time data streams (position, pressure, audio waveforms) from any student's device, enabling centralized one-to-many monitoring and teaching guidance. The server uses a circular buffer to temporarily store each student's operation data for the most recent 5 minutes, supporting instant playback and scoring traceability on the teacher's end. The circular buffer only records the position and pressure sequence data (sampling rate not less than 30Hz) for each student's device, excluding audio waveform data to reduce storage overhead. The 5-minute buffer duration is set based on the average time (approximately 3-5 minutes) of a standard cardiopulmonary auscultation examination procedure (covering a complete examination of all 12 auscultation areas), sufficient to support the teacher's instant playback and scoring traceability of a complete operation. The cached data adopts a first-in-first-out (FIFO) strategy, and historical data older than 5 minutes is automatically overwritten.

[0022] (iv) Virtual simulation teaching software subsystem This subsystem is the core data processing and presentation center of this invention, and includes the following functional modules: 1. 3D Anatomical Visualization Engine: A high-precision 3D human thoracic cavity model was constructed, mapped 1:1 to a physical mannequin. The model includes not only the body surface outline but also layered perspective views, clearly displaying anatomical structures such as bones (ribs, sternum), muscles, lung lobes (lobes), bronchial tree, and heart (including four chambers and major valves). Figure 3As shown, when the stethoscope touches the chest of the physical simulated human and applies pressure, the corresponding area on the body surface in the virtual model is highlighted, and the transparency of the anterior tissue changes dynamically with the pressure, while the deep anatomical structures are clearly visible.

[0023] Virtual-to-real mapping logic: After receiving the real-time position coordinates (x, y) and pressure intensity F from the simulated human, the software employs a separate mapping strategy: (a) Spatial position mapping: Using only the entity's two-dimensional coordinates (x, y), the coordinates are mapped to the initial three-dimensional coordinates of the body surface in the three-dimensional virtual scene through a pre-calibrated homography transformation matrix H. The mapping relationship is as follows: ; in , These are the x-coordinate and y-coordinate (horizontal position) of the point on the surface of the body in the virtual 3D scene. These are the fixed thoracic depth coordinates corresponding to this body surface point in the virtual model.

[0024] (b) Dynamic Perspective and Deformation Control: The pressure intensity F does not participate in spatial coordinate transformation but is used independently to control the degree of transparent sectioning of the virtual thoracic cavity. Define the perspective factor. as follows: ; in The preset standard auscultatory reference pressure threshold (preferably 2.0N) is used. The system utilizes... The volume rendering opacity of the front tissues (chest wall, ribs) and deep structures (lungs, heart) of the 3D model is mixed in real time, and the surface of the thoracic cavity in the stethoscope contact area is simultaneously driven to undergo a small normal concave deformation proportional to F, thereby achieving a virtual-real linkage effect where "the greater the pressure, the clearer the internal anatomical structure is seen, and the more obvious the surface deformation is".

[0025] When F reaches At that time, the model surface produces a pre-set maximum value of concave deformation, while the deep anatomical structure is displayed with maximum clarity (complete de-occlusion).

[0026] 2. Cardiopulmonary sound synthesis and playback engine: The system has a built-in cardiopulmonary sound sample library, which contains cardiopulmonary sound samples collected in various pathophysiological states at standard auscultation locations, and has been processed by time-domain segmentation and frequency-domain labeling.

[0027] Virtual-Real Combination Mapping Module: Based on the mapped position (X,Y,Z) of the virtual stethoscope, the system calculates in real time the spatial distance and azimuth angle between it and various anatomical structures (such as valve projection and bronchial branches) in the 3D model.

[0028] Adaptive sound effects module: During playback, based on the calculated spatial relationships, three-dimensional panoramic sound technology (preferably based on HRTF head-related transfer function binaural rendering) is used to perform convolution reverberation and equalizer adjustment on the basic sound effect samples to simulate the propagation and attenuation characteristics of sound in the chest cavity, so that the sound heard by the student changes smoothly as the stethoscope moves.

[0029] Pathology switching module: Teachers can issue pathology mode switching commands. The engine calculates the phase and amplitude of the currently playing audio source and transitions to the audio source in the new pathology mode through a cross-gradient algorithm, avoiding discomfort caused by sudden audio changes.

[0030] 3. Teaching Management and Evaluation Module (Teacher's End): Remote case management: Teachers can push preset pathology case packages (including: normal breathing, pneumonia, asthma, chronic obstructive pulmonary disease, mitral stenosis, aortic regurgitation, pericardial friction rub, etc.) to all students.

[0031] In addition to audio tags, the case package also includes a set of visualization tags for three-dimensional anatomical structures. When the teacher issues a switching command, the system not only executes the cross-gradient of the audio but also drives the three-dimensional visualization engine to highlight the core anatomical region corresponding to the pathology (for example, when switching to "pneumonia" mode, the corresponding lung segment in the three-dimensional lung lobe model is automatically rendered as an inflammatory red translucent area with a ground-glass texture; when switching to "mitral stenosis" mode, the virtual left atrioventricular valve model automatically contracts and thickens, limiting its opening and closing range), achieving a synchronous and linked teaching demonstration integrating "audio-pathology-anatomical structure".

[0032] Real-time monitoring and annotation: The teacher-side software can simultaneously display the virtual stethoscope position, pressure value, and currently heard audio waveform / spectral graph for all online students. For example... Figure 4 As shown, teachers can make virtual marks directly on the 3D model and provide real-time guidance via voice or text; the system also displays the coverage of the student's auscultation trajectory in the form of a heat map, with uncovered areas and abnormal pressure intervals highlighted in different colors.

[0033] Action Quantification Scoring Engine: The system pre-sets a standard auscultation operation path, which is defined as a sequence of N key auscultation location nodes arranged in a clinically standardized order. At the same time, a standard pressure range is set for each node. and standard stay duration .

[0034] After the trainee completes the practical operation, the system records the sequence of their actual trajectory points. and corresponding pressure value sequence .

[0035] Among them, an effective auscultation sampling point is defined as a trajectory point that simultaneously meets the following two conditions. : (a) The pressure applied at that point ≥ ,in A preset effective contact pressure threshold (preferably 0.3N) is used to eliminate noise trails caused by the stethoscope being suspended or accidentally touched. (b) The point is located within the boundary of one of the 12 preset auscultation functional areas. Trajectory points that do not meet the above conditions will not participate. and The score is calculated, but it is retained in the trajectory playback record for teachers to view.

[0036] The scoring uses a weighted percentage system, and the specific calculation method is as follows: (1) Position accuracy score The Hausdorff distance is used to measure the spatial deviation between the actual trajectory and the standard path. The scoring formula is as follows: ; in The set of standard node indices that have been covered by at least one valid sampling point (i.e. (nodes with finite values). = This represents the number of nodes that have been covered. Let be the shortest Euclidean distance from the actual trajectory point to the j-th standard node. The preset acceptable error radius is (preferably 1.5cm).

[0037] Uncovered nodes (i.e.) (Not participating) The percentage score is recorded separately as "missed auscultation areas" and marked in the final score report. Simultaneously, the system calculates path coverage. = / N×100% is displayed separately as an auxiliary evaluation indicator to distinguish between errors of two different natures: "inaccurate positioning" and "not yet checked".

[0038] (2) Score for accuracy of pressure applied : Calculate the pressure values ​​at all valid auscultation sampling points to ensure they fall within the standard range for the corresponding nodes. The percentage within is directly converted to a percentage score. .

[0039] (3) Overall score The final score is determined by weighting, with the preferred weight being: ; The system also records auscultation areas not covered by effective sampling points (i.e. The system identifies nodes and pressure timeout / insufficiency intervals, generating a visual heatmap for teachers to review. Location accuracy score. With path coverage The evaluation assesses both "accuracy" and "completeness," which together constitute a comprehensive assessment of the trainee's operational completeness.

[0040] Compared with the prior art, the present invention has the following significant advantages: (1) High-fidelity virtual and real integration experience: Through a high-density flexible pressure sensor array and a three-dimensional anatomical visualization engine, the real-time linkage of "wherever the stethoscope is placed, that place will light up; the more pressure is applied, the deeper the internal structure can be seen" is realized, which greatly enhances the immersive learning experience and cognitive efficiency.

[0041] (2) Intelligent generation and switching of pathological sounds: The simple audio clip playback is abandoned. The three-dimensional sound effect synthesis technology based on spatial position and intensity is adopted to achieve smooth transition and enhanced realism of pathological sounds. At the same time, the one-click switching function on the teacher's end supports efficient and risk-free case simulation teaching.

[0042] (3) Precise and quantifiable teaching assessment: By recording and replaying the student's complete operation trajectory (position, force, time), and using a quantitative scoring algorithm, the assessment of auscultation skills, a traditional "experience-based subject," becomes objective, fair, and traceable.

[0043] (4) Support for large-scale standardized teaching: The system can get rid of the dependence on real patients and specific environments, and support multiple students to train and be assessed online at the same time, which greatly improves the accessibility of high-quality teaching resources and teaching efficiency. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the overall architecture of a virtual-real combined cardiopulmonary auscultation teaching system according to the present invention; Figure 2 This is a schematic diagram of the 12-zone logical partitioning layout of the high-density flexible pressure sensing array for simulating the human chest of the present invention; the numbers in the diagram are: 01-left apex of the lung, 02-right apex of the lung, 03-left upper lobe of the lung, 04-right upper lobe of the lung, 05-right middle lobe of the lung, 06-left lower lobe of the lung, 07-right lower lobe of the lung, 08-aortic valve area, 09-pulmonary valve area, 10-second aortic valve area, 11-tricuspid valve area, 12-apex of the heart (mitral valve area); Figure 3 This is a perspective view of the three-dimensional thoracic anatomy model in the virtual simulation teaching software of this invention. Figure 3a: Pressure F = 0 (no contact), body surface intact, internal structure not visible; Figure 3 b: Pressure 0 < F < The surface is bright, the front layer is semi-transparent, and the interior is faintly visible. Figure 3 c: Pressure F≥ The surface of the body is concave and deformed, and the internal structure is completely unobstructed and clearly displayed; Figure 4 This is a schematic diagram of the trajectory recording and scoring heatmap of the teacher-side monitoring interface in the teaching management and evaluation module of this invention. Figure 5 This is a block diagram of the circuit functional modules of the internal analog-to-digital conversion and dual-mode audio processing flow of the intelligent stethoscope of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments. The described embodiments are only some, not all, of the embodiments of this invention.

[0046] Example 1: System initialization and coordinate calibration.

[0047] 1. Hardware Initialization: Connect the physical mannequin to the computer host via Universal Serial Bus (USB) or wirelessly, and start the virtual simulation teaching software. The software automatically recognizes the device and establishes communication.

[0048] 2. Spatial Coordinate Calibration: During the software initialization phase, the system guides the user to use a standard locator (or the stethoscope itself) to sequentially contact at least four non-collinear preset positioning points on the physical simulated human body (preferably the suprasternal notch, xiphoid process, the intersection of the left anterior axillary line and the costal arch, and the intersection of the right anterior axillary line and the costal arch). The software records the physical coordinates (x, y) of these points on the pressure sensor array and their corresponding logical coordinates on the 3D virtual model. By solving the system of linear equations formed by these four sets of corresponding points, the eight independent parameters of the homography transformation matrix H are calculated, thereby achieving precise spatial coordinate alignment between the solid surface and the three-dimensional virtual model.

[0049] Example 2: A standard auscultation teaching procedure.

[0050] 1. Auscultation Operation: The trainee holds the intelligent stethoscope and places it on the "apex area" of the physical mannequin.

[0051] 2. Data Acquisition and Transmission: The stethoscope contacts the body surface, causing changes in the resistance of multiple pressure sensors in the apex region. The microcontroller (MCU) acquires these changes and calculates the two-dimensional coordinates (x, y) = (12.5 cm, 8.2 cm) of the contact location and the normal pressure value F = 2.5 N. This data packet is then transmitted to the host computer via Bluetooth.

[0052] 3. Virtual-Real Mapping and Vision-Driven: After receiving the data, the software executes the separation mapping logic. First, only the two-dimensional coordinates of the entity (x, y) = (12.5cm, 8.2cm) are taken. Using the homography transformation matrix H pre-calibrated in the invention, these coordinates are mapped to the initial three-dimensional coordinates of the body surface corresponding to the apex of the left ventricle in the virtual scene. Simultaneously, based on the received pressure F=2.5N, the perspective factor is calculated. = =1.0. The software detected that this coordinate is located within the projection area of ​​the "left ventricular apex" of the virtual heart.

[0053] 4. Visual Feedback: The corresponding body surface area in the 3D model is displayed with a bright blue halo. Simultaneously, the myocardial and valvular structures at the apex of the heart are automatically highlighted, accompanied by a dynamic animation of the heartbeat. This is because the pressure F=2.5N reaches the preset perspective threshold (…). =2.0N), the perspective factor calculated by the system, at which point the opacity of the ribs and soft tissue model covering the front of the heart is determined by the perspective factor. =1.0 drive reduced to its lowest level (nearly complete transparency), while the system automatically activates the sectioning plane mode in the 3D viewport, virtually sectioning the anterior tissue along the central axis of the stethoscope, allowing the deep cardiac structures to be clearly presented in a completely unobstructed manner. Both visualization effects are active simultaneously, achieving dual perspective feedback of semi-transparent surface and unobstructed internal structures. At the same time, a normal concave deformation of approximately 2mm appears on the surface of the thoracic cavity at the stethoscope contact point, clearly demonstrating the correspondence between the internal cardiac structures and their positions on the body surface.

[0054] 5. Auditory feedback: The system loads the corresponding characteristic sounds of mitral stenosis from the sound library based on the current location and pathological pattern (e.g., "mitral stenosis" pattern).

[0055] Based on the pressure value F, the loudness-pressure mapping function in the audio engine adopts a piecewise linear mapping model: when F is less than 0.5N (too light contact), the output gain is locked to mute (-∞dB); when F is between 0.5N and... When the value is between (2.0N), the output gain G increases linearly with F, as shown in the formula: When F exceeds 2.0N, the output gain remains at the maximum comfortable auscultation loudness (0dB attenuation), thus simulating the physical characteristics of "light pressure, light sound; heavy pressure, clear sound" in real clinical auscultation. Through the earpiece of the intelligent stethoscope, trainees hear a clear and realistic diastolic rumbling murmur resembling mitral stenosis.

[0056] 6. Teaching Assessment: The teacher's software captured the student's operation in real time and displayed on the software interface that the operation point (apex area) was accurate and the pressure (2.5N) was within the standard range (2.0N - 3.0N). The system recorded "accurate positioning" and "pressure met the standard" for this operation as reference data for the final skills assessment.

[0057] Example 3: Dynamic switching of pathological modes.

[0058] 1. Trainee A is auscultating the "aortic valve area". The current pathological pattern is "normal". The trainee hears the normal second heart sound of the aortic valve.

[0059] 2. On the teacher's side interface, in order to demonstrate the pathological situation, the instructor switched the system pathology mode of all students from "normal" to "aortic valve insufficiency" with one click.

[0060] 3. After receiving the switching command, Student A's software does not immediately cut off the current sound, but instead obtains the phase angle of the current audio stream. The amplitude envelope value V was loaded along with an audio sample of "aortic regurgitation".

[0061] 4. The engine executes a 200ms linear crossfade, with the following formula: ; Where t is the time from the switching moment, and T is the preset total duration of the transition (200ms). To switch to the original audio signal in the pre-pathological mode, The target audio signal under the target pathological pattern. This is the mixed audio signal that is actually output to the student after cross-gradient processing.

[0062] 5. The sound heard by student A smoothly transitioned from normal heart sounds to abnormal heart sounds accompanied by a soft, diastolic, decrescendo sighing murmur within 0.2 seconds. The entire process was natural and smooth, requiring no interruption from the student, thus ensuring the continuity and immersion of the teaching demonstration.

[0063] Example 4: An alternative pressure sensing scheme.

[0064] As an alternative to the aforementioned flexible pressure sensing array scheme, this invention can also employ a capacitive pressure sensing film. This film consists of two conductive layers on the top and bottom, and a dielectric layer in the middle. When the stethoscope is pressed, the film deforms locally, causing the distance between the two conductive layers in that area to decrease, thus increasing the capacitance. The system can also calculate the contact position (x, y) and pressure F by scanning the capacitance matrix. The calculation formula is: ; Where C is capacitance, A is the area of ​​the plates, and d is the distance between the plates. The vacuum permittivity, Let be the relative permittivity of the material. This is determined by measuring the change in capacitance. This allows for the inverse calculation of pressure changes. This method offers advantages such as high sensitivity and low power consumption.

[0065] Example 5: Specific implementation of the hybrid enhancement mode of the intelligent stethoscope.

[0066] Teachers enable the "blended enhancement" mode on the software and set the blending coefficient. =0.7. When the trainee slides the stethoscope across the simulated chest, the physical pickup picks up faint skin friction sounds and ambient noise. This signal is then subjected to a high-pass filter with a cutoff frequency of 100Hz (to eliminate slight DC offset from pressing) and a low-pass filter with a cutoff frequency of 2500Hz by the DSP to obtain... The software-side synchronous measurement end-to-end delay is... =45ms, and advance the virtual audio source by 45ms. Press the two signals... =0.3· +0.7· The (t+45ms) mixing allows trainees to hear clear pathological teaching sounds while retaining the tactile feedback and faint friction sounds of physical operations, significantly enhancing the sense of presence during the operation.

[0067] In this invention, the t in the cross-gradient formula defines the starting point (switching time), while the hybrid enhancement formula does not limit t. The latter is a more generalized time variable, while the former is a specific manifestation of the latter in a particular scenario. The general and the narrow are compatible, not contradictory.

[0068] The above description is merely a specific embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A virtual-real combined cardiopulmonary auscultation teaching system, characterized in that, include: The system comprises a physical human simulation subsystem, an intelligent stethoscope subsystem, a virtual-real interaction and communication subsystem, and a virtual simulation teaching software subsystem. The physical mannequin subsystem includes a mannequin body and a high-density flexible pressure sensor array disposed on the chest of the mannequin body. The high-density flexible pressure sensor array is used to collect contact position information and pressure information when the stethoscope contacts the body surface. The intelligent stethoscope subsystem includes a microphone, an analog-to-digital converter, a digital signal processor, and a wireless communication module. The digital signal processor has built-in dual-mode audio processing logic, which includes a pure virtual mode and a hybrid enhancement mode. The virtual-real interaction and communication subsystem adopts a client / server network architecture, including a wireless communication protocol module and a server-side session management module, which is used to realize low-latency data transmission between the physical simulated human, the intelligent stethoscope and the virtual simulation teaching software. The virtual simulation teaching software subsystem includes: A 3D anatomy visualization engine is used to construct a 3D human thoracic cavity model that is mapped 1:1 to a physical mannequin, and to map the contact position information and pressure information to a 3D virtual scene; The heart and lung sound synthesis and playback engine is used to synthesize and play corresponding normal or abnormal heart and lung sounds based on the mapped virtual stethoscope position. The teaching management and assessment module is used to enable remote case management, real-time monitoring and annotation, and quantitative scoring of actions.

2. The virtual-real combined cardiopulmonary auscultation teaching system according to claim 1, characterized in that, The high-density flexible pressure sensing array is a grid array composed of M×N piezoresistive sensing units. The sensing units are evenly distributed in space with a spacing of no more than 1.5cm. The grid array is divided into 12 functional regions according to the commonly used location logic in clinical auscultation. The 12 functional regions include: left lung apex, right lung apex, left upper lobe, right upper lobe, right middle lobe, left lower lobe, right lower lobe, aortic valve region, pulmonary valve region, aortic valve second region, tricuspid valve region, and apex of the heart.

3. The virtual-real combined cardiopulmonary auscultation teaching system according to claim 1, characterized in that, The physical human simulation subsystem calculates the two-dimensional coordinates (x, y) of the stethoscope on the body surface and the normal pressure value F of the contact surface in real time using the center of gravity method. The formula for calculating the two-dimensional coordinates (x, y) is as follows: ， ； The formula for calculating the normal pressure value F is: ； in Let be the resistance change of the i-th triggered sensor. Let be the physical coordinates of the i-th sensor, and k be the pressure-to-resistance conversion coefficient. Let be the relative change in resistance of the i-th sensor, and n be the total number of sensors that are triggered.

4. The virtual-real combined cardiopulmonary auscultation teaching system according to claim 1, characterized in that, The built-in hybrid enhancement mode of the digital signal processor in the intelligent stethoscope subsystem is as follows: the physical vibration signal collected by the physical pickup is processed by bandpass filtering and adaptive noise thresholding, and then superimposed with the virtual sound source in the time domain. The hybrid output formula is: ； in This is an adjustable mixing coefficient for the teacher, with a value range of 0 to 1; This is the delay compensation value; Real-time audio signals acquired by physical microphones and processed by bandpass filtering and adaptive noise thresholding; Virtual sound source signals generated by virtual simulation software; The final output is the mixed audio signal to the student; t is the time variable; the intelligent stethoscope subsystem also includes a miniature inertial measurement unit for auxiliary calibration and stable positioning; the center frequency range of the bandpass filter is 20Hz~2000Hz.

5. The virtual-real combined cardiopulmonary auscultation teaching system according to claim 1, characterized in that, The 3D anatomy visualization engine uses a separation mapping strategy to map entity coordinates to a 3D virtual scene: (a) Spatial position mapping: using only the entity's two-dimensional coordinates (x, y), the initial three-dimensional coordinates of the object's surface in the three-dimensional virtual scene are mapped to the object's initial three-dimensional coordinates through the homography transformation matrix H. The mapping relationship is as follows: ； in , These are the x and y coordinates of the point on the body surface in the virtual 3D scene, respectively. The fixed thoracic depth coordinates corresponding to this body surface point in the virtual model; (b) Dynamic perspective and deformation control: The pressure intensity F does not participate in spatial coordinate transformation, but is used independently to control the degree of transparent sectioning of the virtual thoracic cavity, defining the perspective factor. ,in The preset standard auscultation reference pressure threshold; The system utilizes the perspective factor The volume rendering opacity of the front tissue and deep structure of the 3D model is blended in real time, and the normal concave deformation of the thoracic surface in the stethoscope contact area is driven in a direction proportional to F.

6. The virtual-real combined cardiopulmonary auscultation teaching system according to claim 1, characterized in that, The cardiopulmonary sound synthesis and playback engine includes: The cardiopulmonary sound sample bank contains cardiopulmonary sound samples collected at standard auscultation positions under various pathophysiological conditions. The virtual-real combination mapping module calculates the spatial distance and azimuth angle between the mapped virtual stethoscope position (X,Y,Z) and each anatomical structure in the 3D model in real time. The adaptive sound effects module, based on the spatial distance and azimuth angle, uses binaural rendering technology based on head correlation transfer function to perform convolution reverberation and equalizer adjustment on the basic sound effect samples; The pathology switching module, upon receiving a pathology mode switching command from the teacher's end, uses a cross-gradient algorithm to achieve a seamless transition of pathology sounds. The cross-gradient formula is as follows: ； in To switch to the original audio signal in the pre-pathological mode, The target audio signal under the target pathological pattern. The mixed audio signal actually output to the student after cross-gradient processing, t is the time from the switching moment, and T is the preset total gradient duration.

7. The virtual-real combined cardiopulmonary auscultation teaching system according to claim 1, characterized in that, The quantitative scoring of actions in the teaching management and evaluation module includes: A pre-defined standard auscultation procedure path is defined as a sequence of N key auscultation location nodes arranged in a clinically standardized order. And set a standard pressure range for each node. Record the actual trajectory point sequence after the trainee's actual operation. and corresponding pressure value sequence Define an effective auscultation sampling point, which is one that simultaneously satisfies the pressure level. ≥ And the trajectory point located within the boundary range of one of the 12 preset auscultation functional areas. ,in The preset effective contact pressure threshold is used to calculate the position accuracy score. Hausdorff distance is used to measure the spatial deviation between the actual trajectory and the standard path: ； in A set of standard node indices that have been covered by at least one valid sampling point. = This represents the number of nodes that have been covered. Let be the shortest Euclidean distance from the actual trajectory point to the j-th standard node. Set the acceptable error radius; calculate path coverage. = / N×100% is used as an auxiliary evaluation indicator; the accuracy score of pressure intensity is calculated. The pressure values ​​at all valid auscultation sampling points must fall within the standard range of the corresponding nodes. Percentage within; Calculate the overall score .

8. The virtual-real combined cardiopulmonary auscultation teaching system according to claim 1, characterized in that, The virtual-real interaction and communication subsystem uses a circular buffer to temporarily store the operation data of each student for the most recent 5 minutes. The circular buffer only records the position sequence and pressure sequence data of each student's terminal, with a sampling rate of not less than 30Hz, and does not store audio waveform data. The cached data adopts a first-in-first-out strategy, and historical data exceeding 5 minutes is automatically overwritten.

9. A control method for a virtual-real combined cardiopulmonary auscultation teaching system, characterized in that, Includes the following steps: S1: The contact position and pressure information of the stethoscope when it contacts the body surface are collected by a high-density flexible pressure sensor array set on the chest of a simulated human. The high-density flexible pressure sensor array is a grid array composed of M×N piezoresistive sensing units. The sensing units are evenly distributed in space with a spacing of no more than 1.5cm. The grid array is divided into 12 functional areas according to the commonly used location logic of clinical auscultation, including: left lung apex, right lung apex, left upper lobe, right upper lobe, right middle lobe, left lower lobe, right lower lobe, aortic valve area, pulmonary valve area, aortic valve second area, tricuspid valve area, and apex of the heart. S2: Solve the two-dimensional coordinates (x, y) of the stethoscope on the body surface and the normal pressure value F using the center of gravity method; S3: The two-dimensional coordinates (x, y) and the normal pressure value F are transmitted to the virtual simulation teaching software in real time via wireless communication; S4: Map the entity's two-dimensional coordinates (x, y) to the initial three-dimensional coordinates of the object's surface in the three-dimensional virtual scene using the homography transformation matrix H. ; S5: Calculate the perspective factor based on the pressure F. Using the perspective factor The mixed opacity of the anterior tissue and deep structure of the 3D model is controlled, and the normal concave deformation of the thoracic surface in the stethoscope contact area is driven to be proportional to F. S6: Calculate the spatial distance and azimuth angle between the mapped virtual stethoscope position (X,Y,Z) and each anatomical structure in the 3D model, and use binaural rendering technology based on head correlation transfer function to synthesize the corresponding 3D panoramic audio signal and output it. S7: Record the trainee's actual operation trajectory, compare the actual operation trajectory with the preset standard auscultation operation path, calculate the position accuracy score and path coverage based on Hausdorff distance, calculate the pressure accuracy score, and calculate the comprehensive total score according to the preset weight to complete the quantitative scoring.

10. The control method for the virtual-real combined cardiopulmonary auscultation teaching system according to claim 9, characterized in that, In step S6, when the teacher sends a pathology mode switching command, the audio engine executes a cross-gradient algorithm to transition from the current audio signal to the target audio signal in a linear gradient of duration T: ； in To switch to the original audio signal in the pre-pathological mode, The target audio signal under the target pathological pattern. The mixed audio signal actually output to the student after cross-gradient processing, where t is the time from the switching moment and T is the preset total gradient duration; The intelligent stethoscope supports two audio output modes: pure virtual mode and hybrid enhancement mode. The pure virtual mode completely blocks the signal from the physical microphone and only plays virtual sound sources generated by virtual simulation software. The hybrid enhancement mode involves bandpass filtering and adaptive noise thresholding of the physical vibration signal collected by the physical pickup, followed by temporal superposition with the virtual sound source. The hybrid output formula is as follows: ； in The mixing coefficient is adjustable on the teacher's end; This is the delay compensation value; Real-time audio signals acquired by physical microphones and processed by bandpass filtering and adaptive noise thresholding; Virtual sound source signals generated by virtual simulation software; This is the final mixed audio signal output to the student; t is the time variable.

Images