Human body surface palpation simulation training system based on multi-modal ai interaction

Abstract

This invention discloses a human body surface palpation simulation training system based on multimodal AI interaction, belonging to the fields of medical simulation training and human-computer interaction technology. The system includes: a photoelectric-triboelectric fusion tactile perception and feedback unit, a palpation-specific micro-tactile feature library, an AI dynamic tactile rendering engine, and a hardware clock synchronization module. The photoelectric-triboelectric fusion tactile perception and feedback unit integrates modules such as multispectral imaging, synchronously collects four types of palpation data, and generates corresponding feedback. The feature library stores multi-dimensional features of real clinical palpation. The AI ​​engine is based on the Transformer architecture, combines the collected data and the feature library, and generates driving signals through calculation formulas. The synchronization module uniformly controls the sampling timing of each module. This invention integrates multiple modules to synchronously collect multi-dimensional palpation data, constructs a real clinical palpation feature library, and combines a customized AI algorithm that is deeply integrated with hardware to achieve high-fidelity, low-latency three-dimensional tactile simulation, ensuring the consistency of tactile feedback across different devices.

Description

Technical Field

[0001] This invention relates to the field of medical simulation training and human-computer interaction technology, specifically a human body surface palpation simulation training system based on multimodal AI interaction. Background Technology

[0002] Palpation is a core diagnostic skill that clinicians must master. From lymph node examination for the common cold to preliminary tumor screening, accurate palpation is indispensable. Traditional palpation teaching mainly relies on instructors demonstrating on patients, followed by student practice. This method is limited by the number of patients and their cooperation, and it also prevents students from repeatedly practicing the palpation feel of the same lesion. In recent years, simulation training systems combining haptic feedback technology and virtual reality have begun to enter medical schools, providing medical students with training methods that are not limited by time or location.

[0003] The tactile reproduction accuracy of existing palpation simulation training systems is generally low. Most products use only a single pressure sensor to detect the pressure applied, which cannot simulate the friction of skin sliding, the subtle vibrations of diseased tissue, or the temperature differences of inflamed areas. The tactile data used by these systems mostly comes from theoretical models and is not based on real clinical cases, resulting in very low differentiation in the feel of different lesions. Moreover, the algorithms and hardware of most systems are designed separately without optimization for hardware characteristics, leading to significant delays during operation, affecting training effectiveness, and failing to meet the actual needs of standardized teaching of clinical palpation skills. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a human body surface palpation simulation training system based on multimodal AI interaction. This system integrates a multispectral imaging module, a triboelectric sensing array, and a micro-thermoelectric cooling layer within a single bionic skin unit, along with a hardware clock synchronization module, to achieve precise synchronous acquisition of multi-dimensional palpation data. It constructs a palpation-specific microtactile feature library containing real clinical cases and utilizes an AI dynamic tactile rendering engine based on the Transformer architecture. Through multimodal feature fusion and dynamic signal generation formulas, it achieves deep collaboration between the algorithm and hardware. This solves the problems of existing palpation simulation systems, such as single tactile dimension, insufficient data realism, and high feedback latency. It achieves three-dimensional synchronous tactile simulation of force, texture, and temperature, reproducing key clinical microtactile features, ensuring consistent tactile feedback across different devices, and providing standardized support for palpation skills training.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a human body surface palpation simulation training system based on multimodal AI interaction, the system comprising:

[0006] Photoelectric-triboelectric fusion tactile sensing and feedback unit: Integrating a multispectral imaging module, triboelectric sensing array and micro thermoelectric cooling layer within the same bionic skin unit, it is used to synchronously collect the normal force, tangential shear force, skin texture roughness and local temperature changes during user touch, and generate corresponding multimodal tactile feedback signals to achieve integrated acquisition and synchronous output of force-texture-temperature three-dimensional tactile information;

[0007] Palpation-specific microtouch feature library: It stores real clinical palpation data, including the micro-vibration, viscoelasticity, and sliding friction characteristics of normal and diseased tissues, providing a real and traceable clinical data benchmark for subsequent tactile rendering;

[0008] AI Dynamic Tactile Rendering Engine: Based on the Transformer architecture, it calls corresponding features from the touch-specific micro-tactile feature library according to the real-time operation data collected by the photoelectric-triboelectric fusion unit. It generates tactile feedback signals through the multimodal tactile feature weighted fusion formula and the dynamic tactile feedback signal generation formula, which drives the output of the photoelectric-triboelectric fusion tactile perception and feedback unit, realizing deep collaborative adaptation between the algorithm and the hardware, and reducing system response latency.

[0009] Hardware clock synchronization module: It is electrically connected to the multispectral imaging module, triboelectric sensing array and micro thermoelectric cooling layer respectively, and is used to uniformly control the sampling timing of each module, ensure the time synchronization of multi-source modal data and eliminate data timing deviation.

[0010] Furthermore, the photoelectric-triboelectric fusion tactile sensing and feedback unit has an overall thickness of no more than 1mm, is flexible and bendable, and has a radius of curvature of no more than 5mm. From top to bottom, it consists of a bionic skin protective layer, a multispectral imaging module, a triboelectric sensing array, a micro thermoelectric cooling layer, and a flexible substrate, which can closely fit the physiological shape of the human fingertip, improving wearing comfort and the flexibility of tactile operation.

[0011] Furthermore, the multispectral imaging module consists of a dot matrix structure composed of near-infrared light-emitting devices with wavelengths of 900nm-1000nm and photodetectors of the same specifications arranged in an alternating pattern, with each light-emitting device corresponding one-to-one with an adjacent photodetector. When the module is working, the near-infrared light-emitting devices emit light at a set frequency, and the photodetectors collect the changes in the intensity of reflected light at the contact interface. The skin texture roughness is calculated by the spatial distribution of the reflected light intensity, and the tangential displacement and tangential shear force are calculated by the temporal change of the reflected light intensity. The sampling frequency is not less than 1kHz, which realizes high-precision detection of skin texture roughness and tangential shear force, and accurately captures the dynamic characteristics of sliding friction during the palpation process.

[0012] Furthermore, the triboelectric sensing array employs a polymer triboelectric layer and a metal electrode layer to form a capacitive triboelectric sensor matrix structure. Utilizing the principles of contact electrification and electrostatic induction, the electrode layer collects the charge changes generated when the triboelectric layer contacts and separates from the user's fingertip, thereby calculating the normal force distribution during the user's palpation. Simultaneously, it collects micro-vibration signals in the frequency range of 10Hz-10kHz to achieve high-resolution detection of the normal force distribution and accurately capture tissue micro-vibration signals, thus restoring key micro-tactile features such as twisting sensation and wave sensation during clinical palpation.

[0013] Furthermore, the micro thermoelectric cooling layer is composed of multiple thermoelectric units connected in series and encapsulated using a flexible substrate. By changing the direction and magnitude of the current flowing through the thermoelectric units, cooling and heating functions are achieved. The temperature adjustment range is 25°C to 40°C, and the temperature resolution is no higher than 0.1°C. It is used to simulate local temperature changes and continuous temperature gradient distribution in human tissues, accurately simulate the temperature difference between normal and diseased tissues, restore the continuous temperature gradient distribution of inflamed areas, and enhance the realism of palpation simulation.

[0014] Furthermore, the palpation-specific microtactile feature library stores a large sample of real clinical palpation data, covering multiple anatomical sites and various diseases. Each case data includes mechanical feature vectors, vibration feature vectors, and friction feature vectors. The mechanical feature vectors include elastic modulus, viscosity coefficient, and stress relaxation time; the vibration feature vectors include dominant frequency, harmonic components, and amplitude distribution; and the friction feature vectors include static friction coefficient, dynamic friction coefficient, and viscous drag coefficient. The data is stored in a three-level classification according to anatomical site, disease type, and lesion degree. All features are cross-annotated by multiple physicians, providing real and reliable tactile benchmark data for AI rendering and ensuring the distinguishability of tactile features between different diseased tissues.

[0015] Furthermore, the AI ​​dynamic haptic rendering engine is composed of an input layer, a feature matching layer, a dynamic rendering layer, and an output layer connected in sequence;

[0016] The input layer receives multimodal raw data synchronously output by the photoelectric-triboelectric fusion unit, performs data normalization and format conversion processing, eliminates the dimensional differences of data in different dimensions, and improves the efficiency and accuracy of subsequent feature processing;

[0017] The feature matching layer uses a cosine similarity algorithm to calculate the similarity between the processed real-time feature vector and the features in the palpation-specific microtouch feature library, and selects the feature with the highest similarity as the benchmark feature to achieve fast and accurate matching of the tactile features of the corresponding tissue.

[0018] The dynamic rendering layer applies a multimodal haptic feature weighted fusion formula, the mathematical expression of which is:

[0019]

[0020] in The fused unified tactile feature vector, For mechanical eigenvectors, For vibration characteristic vectors, For friction feature vectors, Let be the temperature feature vector, and α, β, γ, δ be the adaptive attention weight coefficients, satisfying... This enables adaptive weighted fusion of multimodal tactile features to generate a unified feature vector that conforms to the laws of human tactile perception.

[0021] Furthermore, after obtaining the fused unified haptic feature vector, the dynamic rendering layer applies a dynamic haptic feedback signal generation formula, the mathematical expression of which is:

[0022]

[0023] in The haptic feedback drive signal output to the hardware at time t is... This is a baseline tissue feature vector matched from a palpation-specific microtouch feature library. The adjustment coefficients for the operating parameters are the user's real-time pressing pressure F, sliding speed v, and touch angle. nonlinear functions, The signal used to compensate for hardware delay and nonlinearity is generated by an AI model based on the response characteristics of the photoelectric-triboelectric fusion unit. This enables dynamic real-time adjustment of the tactile feedback signal, while compensating for the inherent response delay and nonlinearity error of the hardware, thereby improving the accuracy and real-time performance of the tactile feedback.

[0024] Furthermore, the system incorporates a formula for quantitatively evaluating the accuracy of palpation simulation, the mathematical expression of which is:

[0025]

[0026] Where E represents the overall relative error of the tactile feedback of the system, and N represents the total number of test samples. Let i be the system simulation tactile feature value of the i-th sample. This represents the true clinical tactile characteristic value of the i-th sample. The system completes accuracy calibration using this formula before leaving the factory, enabling quantifiable calibration of the system's tactile feedback accuracy and ensuring the consistency of tactile output for each device.

[0027] Furthermore, the hardware clock synchronization module outputs a unified clock trigger signal to the multispectral imaging module and the triboelectric sensing array, controlling the two modules to start sampling simultaneously, with a sampling timing synchronization error of no more than 1μs; the output layer of the AI ​​dynamic haptic rendering engine decomposes the generated haptic feedback signal into force-driven signals and temperature-driven signals, which are output to the triboelectric sensing array and the micro thermoelectric cooling layer, respectively, to ensure the timing synchronization accuracy of multimodal acquisition data and realize the coordinated synchronous output of force feedback and temperature feedback.

[0028] Compared with existing technologies, this human body surface palpation simulation training system based on multimodal AI interaction has the following beneficial effects:

[0029] I. This invention integrates a multispectral imaging module, a triboelectric sensing array, and a micro-thermoelectric cooling layer within a single thin bionic skin unit. Combined with a hardware clock synchronization module, it achieves precise synchronous acquisition of multimodal data, with the acquisition time difference of the four signals not exceeding 1 microsecond. The system can simultaneously acquire normal force, tangential shear force, skin texture roughness, and local temperature changes during palpation. Combined with an AI dynamic tactile rendering algorithm customized and optimized for this hardware, it can reproduce the twisting, undulating, and frictional sensations experienced during real palpation, providing users with a tactile experience highly similar to actual clinical palpation.

[0030] Second, this invention constructs a palpation-specific microtactile feature library, storing a large amount of tactile data from real clinical cases, covering the micro-vibration, viscoelasticity, and sliding friction properties of normal tissues and various diseased tissues. All features have been labeled and verified, and are stored according to anatomical location, disease type, and lesion severity. AI rendering directly uses this real data as a benchmark, rather than relying on theoretical physical models for calculation. The tactile differences between different lesions are clearly distinguishable, allowing users to accurately differentiate between normal and diseased tissues.

[0031] Third, this invention uses a multimodal tactile feature weighted fusion formula and a dynamic tactile feedback signal generation formula, combined with hardware latency and nonlinear compensation mechanisms, to control the total system response time to within 10 milliseconds, resulting in no noticeable lag during operation. The system incorporates a quantification and evaluation formula for tactile simulation accuracy, and accuracy calibration is performed using a unified standard before leaving the factory to ensure consistent tactile feedback on every device, providing a essentially identical training experience for different users on different devices.

[0032] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of the core architecture and data flow of the human body surface palpation simulation training system based on multimodal AI interaction of the present invention;

[0035] Figure 2 This is a schematic diagram illustrating the workflow of the AI ​​dynamic haptic rendering engine of the present invention.

[0036] Figure 3 This is a schematic diagram of the stacked structure of the photoelectric-triboelectric fusion tactile sensing and feedback unit of the present invention. Detailed Implementation

[0037] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0038] This embodiment provides a specific implementation method for a human body surface palpation simulation training system based on multimodal AI interaction, such as... Figure 1 As shown, the system synchronously collects multi-dimensional palpation data through a photoelectric-triboelectric fusion tactile perception and feedback unit. A hardware clock synchronization module ensures data timing consistency, and a real clinical data benchmark is provided based on a palpation-specific micro-tactile feature library. An AI dynamic tactile rendering engine completes multi-modal feature fusion and dynamic tactile signal generation, ultimately outputting multi-modal tactile feedback. This embodiment fully realizes three-dimensional synchronous tactile simulation of force, texture, and temperature, adaptable to palpation training scenarios of different anatomical locations, including but not limited to the abdomen, chest, superficial lymph nodes, thyroid gland, and other routine clinical palpation sites, meeting the multi-level usage needs of basic teaching and advanced training.

[0039] The specific implementation process of this embodiment is as follows:

[0040] When the system is running, the user wears a photoelectric-triboelectric fusion tactile sensing and feedback unit and performs palpation operations on a virtual human body model under the guidance of a display device. The display device simultaneously shows the anatomical structure of the virtual human body model, palpation site guidance, and real-time operating parameters. The user can select the corresponding training subject, anatomical location, and disease type according to their training needs, and the system automatically loads the virtual model and feature library data for the corresponding area. For example... Figure 3As shown, the photoelectric-triboelectric fusion tactile sensing and feedback unit adopts an integrated flexible structure with an overall thickness of no more than 1mm. It can conform to the curvature of the human fingertip with a radius of curvature of no more than 5mm. From top to bottom, it consists of a bionic skin protective layer, a multispectral imaging module, a triboelectric sensing array, a micro thermoelectric cooling layer, and a flexible substrate. The bionic skin protective layer is made of flexible material, and its surface simulates the texture of human skin, providing users with a natural tactile experience while protecting the internal electronic components from damage. The coefficient of friction of the bionic skin protective layer is consistent with that of human skin, closely mimicking the tactile sensation of real touch.

[0041] The multispectral imaging module consists of a dot matrix structure composed of near-infrared light-emitting devices and photodetectors arranged alternately, with each light-emitting device corresponding one-to-one with an adjacent photodetector. During operation, the near-infrared light-emitting devices emit near-infrared light with wavelengths in the range of 900nm to 1000nm at a set frequency. This light penetrates the bionic skin protective layer and illuminates the contact interface between the user's fingertip and the bionic skin. Part of the light is reflected and collected by the photodetectors. The photodetectors convert the received light signals into electrical signals. By analyzing the spatial distribution of the reflected light intensity, the skin texture roughness of the contact area is calculated; by analyzing the change in reflected light intensity over time, the tangential displacement and tangential shear force of the user's fingertip are calculated. The multispectral imaging module has a sampling frequency of at least 1kHz, enabling real-time capture of texture changes and sliding movements during palpation. After preprocessing, the raw data collected by the module is time-stamped and synchronized with the mechanical data collected by the triboelectric sensing array via a hardware clock synchronization module, ensuring the temporal correspondence of the multi-source data.

[0042] The triboelectric sensing array employs a polymer friction layer and a metal electrode layer to form a capacitive triboelectric sensor matrix structure. When a user presses or slides their fingertip against the biomimetic skin, the polymer friction layer contacts and separates from the fingertip skin. Due to the contact electrification effect, equal amounts of opposite charges are generated on both surfaces. Simultaneously, due to the electrostatic induction effect, the metal electrode layer senses a corresponding charge change. By acquiring the charge change signal on the metal electrode layer, the normal force distribution during the user's palpation can be calculated. Furthermore, the triboelectric sensing array can also acquire micro-vibration signals in the frequency range of 10Hz to 10kHz. These micro-vibration signals correspond to subtle tactile features generated by tissue friction and fluid flow, including the fluctuation sensation of abscesses, the frictional sensation of fibrotic tissue, and the hardness of nodular tissue during clinical palpation, comprehensively covering the key micro-tactile signals in clinical palpation.

[0043] The miniature thermoelectric cooling layer consists of multiple thermoelectric units connected in series and encapsulated on a flexible substrate. The thermoelectric units operate based on the Seebeck and Peltier effects, achieving cooling or heating functions by changing the direction of the current flowing through them; the temperature variation is controlled by adjusting the current magnitude. The miniature thermoelectric cooling layer has a temperature regulation range of 25℃ to 40℃, with a temperature resolution of no more than 0.1℃. It can simulate local temperature changes in human tissue, generate continuous temperature gradient distributions, reproduce the temperature characteristics of inflamed tissue where the temperature gradually decreases from the center to the edge, and simulate the low-temperature characteristics of ischemic tissue, covering the abnormal temperature manifestations corresponding to different types of clinical lesions.

[0044] The hardware clock synchronization module is electrically connected to the multispectral imaging module, the triboelectric sensing array, and the micro thermoelectric cooling layer, respectively. It outputs a unified clock trigger signal to each module, controlling the multispectral imaging module and the triboelectric sensing array to simultaneously start sampling. This ensures that the data collected by the two modules are strictly aligned in time, with a synchronization error of no more than 1μs. The hardware clock synchronization module also synchronously controls the temperature adjustment timing of the micro thermoelectric cooling layer, ensuring that temperature feedback is consistent with force feedback and texture feedback in time. This avoids tactile feedback misalignment caused by timing deviations between different modules, guaranteeing the synchronization and consistency of multimodal tactile feedback.

[0045] The palpation-specific microtactile feature library stores a large sample of real clinical palpation data, covering multiple anatomical sites and various disease types. Each case data includes mechanical feature vectors, vibration feature vectors, and friction feature vectors. The mechanical feature vectors include three core parameters: elastic modulus, viscosity coefficient, and stress relaxation time. The vibration feature vectors include three core parameters: dominant frequency, harmonic components, and amplitude distribution. The friction feature vectors include three core parameters: static friction coefficient, dynamic friction coefficient, and viscous drag coefficient. All feature data have been cross-annotated and validated by multiple qualified clinicians and are stored in a three-level classification according to anatomical site, disease type, and lesion severity. It supports rapid retrieval and access. The system can automatically retrieve feature data for the corresponding anatomical site and disease type based on the user's selected training subject, eliminating the need for manual retrieval.

[0046] like Figure 2As shown, the AI ​​dynamic tactile rendering engine is built on the Transformer architecture and consists of an input layer, a feature matching layer, a dynamic rendering layer, and an output layer connected sequentially. The input layer receives multimodal raw data synchronously output by the photoelectric-triboelectric fusion tactile perception and feedback unit. First, the raw data is denoised to remove interference from environmental and hardware noise. Then, the data is normalized to map data from different dimensions to a unified numerical range. Finally, the data is converted to a format that can be processed by subsequent layers. The processed data retains the temporal and spatial distribution features of the original tactile operation, providing complete input data for subsequent feature matching.

[0047] The feature matching layer employs a cosine similarity algorithm to calculate the similarity between the real-time feature vector processed by the input layer and the feature vectors in the palpation-specific microtactile feature library. The cosine similarity algorithm measures the similarity between two vectors by calculating the cosine of the angle between them; the closer the cosine value is to 1, the more similar the two vectors are. The feature matching layer iterates through all feature vectors corresponding to the anatomical sites in the feature library, calculates their cosine similarity with the real-time feature vector, and selects the feature vector with the highest similarity as the baseline tissue feature vector. When the highest similarity between the real-time feature vector and the feature in the feature library is lower than a set threshold, the system automatically calls feature data from adjacent anatomical sites for interpolation fitting to generate a suitable baseline feature vector.

[0048] The dynamic rendering layer first applies a multimodal haptic feature weighted fusion formula to fuse the baseline tissue feature vectors. The formula is as follows:

[0049]

[0050] in The fused unified tactile feature vector, For mechanical eigenvectors, For vibration characteristic vectors, For friction feature vectors, The temperature feature vector is represented by α, β, γ, and δ, which are adaptive attention weight coefficients dynamically calculated by the AI ​​model based on the user's real-time operation status. The sum of the four weight coefficients is always equal to 1. The dynamic adjustment range of the weight coefficients matches the routine operation range of clinical palpation, ensuring that the fused feature vector conforms to the tactile change patterns of real clinical palpation.

[0051] The adaptive attention weighting coefficient can automatically adjust the weight of each tactile dimension according to different tactile operation types. When the user performs a pressing operation, the weight of mechanical features is increased; when the user performs a sliding operation, the weight of friction features is increased.

[0052] After obtaining the fused unified haptic feature vector, the dynamic rendering layer applies the dynamic haptic feedback signal generation formula to generate the final haptic feedback driving signal. The formula is as follows:

[0053]

[0054] in The haptic feedback drive signal output to the hardware at time t is... This is a baseline tissue feature vector matched from a palpation-specific microtouch feature library. The adjustment coefficients for the operating parameters are the user's real-time pressing pressure F, sliding speed v, and touch angle. The nonlinear function. The operation parameter correction coefficient can adjust the intensity and dynamic characteristics of haptic feedback according to the user's real-time operation parameters, so that the haptic feedback changes in real time with the operation. The signal is used to compensate for hardware delay and nonlinearity. It is generated by an AI model based on the response characteristics of the photoelectric-triboelectric fusion unit. It can compensate for the inherent response delay and nonlinear error of the hardware, control the hardware response delay within the human tactile perception threshold, and avoid the lag problem between operation and feedback.

[0055] The output layer receives the haptic feedback drive signal generated by the dynamic rendering layer and decomposes it into two parts: a force drive signal and a temperature drive signal. The force drive signal is output to the triboelectric sensing array to control the array to output corresponding force and vibration feedback; the temperature drive signal is output to the micro thermoelectric cooling layer to control the micro thermoelectric cooling layer to output corresponding temperature feedback. The output timing of the force drive signal and the temperature drive signal is synchronized with the clock signal of the hardware clock synchronization module.

[0056] The system incorporates a built-in formula for quantifying the accuracy of palpation simulation, used to calibrate the system's accuracy before shipment. The formula is as follows:

[0057]

[0058] Where E represents the overall relative error of the tactile feedback of the system, and N represents the total number of test samples. Let i be the system simulation tactile feature value of the i-th sample. Let be the actual clinical tactile feature value of the i-th sample. During calibration, the system is tested using standard tactile test samples. The simulated tactile feature values ​​output by the system are collected and compared with the actual clinical tactile feature values ​​to calculate the overall relative error of the system. Based on the calculation results, the parameters of the AI ​​dynamic tactile rendering engine are adjusted until the overall relative error of the system meets the design requirements. After calibration, the system will automatically lock the calibration parameters. Recalibration can be performed periodically during subsequent use to ensure the accuracy of tactile feedback during long-term use of the equipment.

[0059] This embodiment achieves synchronous acquisition of multi-dimensional tactile data through a hardware architecture that integrates photoelectric and triboelectric sensing. A tactile benchmark is provided by a tactile-specific microtactile feature library constructed from real clinical data. Multimodal tactile feedback signals are generated through an AI dynamic tactile rendering engine that deeply integrates algorithms and hardware. The system recreates the microtactile features in clinical tactile examination, enabling tactile differentiation of different lesions and ensuring consistency of tactile feedback across different devices. It provides users with a tactile training experience and can be widely applied in various clinical teaching scenarios, such as clinical skills teaching in medical colleges, standardized residency training, and skills enhancement for primary healthcare workers.

[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A human body surface palpation simulation training system based on multi-modal AI interaction, characterized in that, The system includes: Photoelectric-triboelectric fusion tactile sensing and feedback unit: Integrating a multispectral imaging module, triboelectric sensing array and micro thermoelectric cooling layer within the same bionic skin unit, it is used to simultaneously collect the normal force, tangential shear force, skin texture roughness and local temperature changes during user touch, and generate corresponding multimodal tactile feedback signals; Palpation-specific microtouch feature library: stores real clinical palpation data, including micro-vibration, viscoelasticity, and sliding friction characteristics of normal and diseased tissues; AI Dynamic Tactile Rendering Engine: Based on the Transformer architecture, it calls corresponding features from the touch-specific micro-tactile feature library according to the real-time operation data collected by the photoelectric-triboelectric fusion unit. It generates tactile feedback signals through the multimodal tactile feature weighted fusion formula and the dynamic tactile feedback signal generation formula, driving the output of the photoelectric-triboelectric fusion tactile perception and feedback unit. Hardware clock synchronization module: It is electrically connected to the multispectral imaging module, the triboelectric sensing array, and the micro thermoelectric cooling layer, respectively, and is used to uniformly control the sampling timing of each module.

2. The human body surface palpation simulation training system based on multi-modal AI interaction according to claim 1, characterized in that, The photoelectric-triboelectric fusion tactile sensing and feedback unit has an overall thickness of no more than 1 mm, is flexible and bendable, and has a curvature radius of no more than 5 mm. From top to bottom, it consists of a bionic skin protective layer, a multispectral imaging module, a triboelectric sensing array, a micro thermoelectric cooling layer, and a flexible substrate.

3. The human body surface palpation simulation training system based on multi-modal AI interaction according to claim 2, characterized in that, The multispectral imaging module consists of a dot matrix structure composed of near-infrared light-emitting devices with wavelengths of 900nm-1000nm and photodetectors of the same specifications arranged in an alternating pattern. Each light-emitting device corresponds one-to-one with an adjacent photodetector. When the module is working, the near-infrared light-emitting devices emit light at a set frequency, and the photodetectors collect the changes in the intensity of reflected light at the contact interface. The skin texture roughness is calculated by the spatial distribution of the reflected light intensity, and the tangential displacement and tangential shear force are calculated by the temporal change of the reflected light intensity. The sampling frequency is not less than 1kHz.

4. The human body surface palpation simulation training system based on multi-modal AI interaction according to claim 2, characterized in that, The triboelectric sensing array uses a polymer friction layer and a metal electrode layer to form a capacitive triboelectric sensor array structure. Utilizing the principles of contact electrification and electrostatic induction, it collects the charge changes generated when the friction layer separates from the user's fingertip through the electrode layer, and then calculates the normal force distribution during the user's touch. At the same time, it collects micro-vibration signals with a frequency range of 10Hz-10kHz.

5. The human body surface palpation simulation training system based on multi-modal AI interaction according to claim 2, characterized in that, The micro thermoelectric cooling layer consists of multiple thermoelectric units connected in series and is encapsulated using a flexible substrate. By changing the direction and magnitude of the current flowing through the thermoelectric units, it achieves cooling and heating functions. The temperature adjustment range is 25°C to 40°C, and the temperature resolution is no higher than 0.1°C. It is used to simulate local temperature changes and continuous temperature gradient distribution in human tissue.

6. The human body surface palpation simulation training system based on multimodal AI interaction according to claim 1, characterized in that, The palpation-specific microtouch feature library stores a large sample of real clinical palpation data, covering multiple anatomical sites and various diseases. Each case data includes mechanical feature vectors, vibration feature vectors, and friction feature vectors. The mechanical feature vectors include elastic modulus, viscosity coefficient, and stress relaxation time. The vibration feature vectors include dominant frequency, harmonic components, and amplitude distribution. The friction feature vectors include static friction coefficient, dynamic friction coefficient, and viscous drag coefficient. The data is stored in a three-level classification according to anatomical site, disease type, and lesion severity. All features are cross-annotated by multiple physicians.

7. The human body surface palpation simulation training system based on multimodal AI interaction according to claim 1, characterized in that, The AI ​​dynamic haptic rendering engine is composed of an input layer, a feature matching layer, a dynamic rendering layer, and an output layer connected in sequence. The input layer receives multimodal raw data synchronously output by the photoelectric-triboelectric fusion unit and performs data normalization and format conversion processing. The feature matching layer uses a cosine similarity algorithm to calculate the similarity between the processed real-time feature vector and the features in the palpation-specific microtouch feature library, and selects the feature with the highest similarity as the benchmark feature. The dynamic rendering layer applies a multimodal haptic feature weighted fusion formula, the mathematical expression of which is: in The unified tactile feature vector after fusion. For mechanical eigenvectors, For vibration characteristic vectors, For friction feature vectors, Let be the temperature feature vector, and α, β, γ, δ be the adaptive attention weight coefficients, satisfying... .

8. The human body surface palpation simulation training system based on multimodal AI interaction according to claim 1, characterized in that, After obtaining the fused unified haptic feature vector, the dynamic rendering layer applies the dynamic haptic feedback signal generation formula, the mathematical expression of which is: in The haptic feedback drive signal output to the hardware at time t is... This is a baseline tissue feature vector matched from a palpation-specific microtouch feature library. The adjustment coefficients are for the operating parameters, which are the user's real-time pressing pressure F, sliding speed v, and touch angle. nonlinear functions, The signal is used to compensate for hardware delay and nonlinearity, and is generated by an AI model based on the response characteristics of the photoelectric-triboelectric fusion unit.

9. The human body surface palpation simulation training system based on multimodal AI interaction according to claim 1, characterized in that, The system incorporates a formula for quantitatively evaluating the accuracy of palpation simulation. The mathematical expression of this formula is as follows: Where E represents the overall relative error of the tactile feedback of the system, and N represents the total number of test samples. Let i be the system simulation tactile feature value of the i-th sample. The true clinical palpation tactile characteristic value of the i-th sample is used to perform accuracy calibration of the system before it leaves the factory.

10. The human body surface palpation simulation training system based on multimodal AI interaction according to claim 1, characterized in that, The hardware clock synchronization module outputs a unified clock trigger signal to the multispectral imaging module and the triboelectric sensing array, controlling the two modules to start sampling simultaneously, with a sampling timing synchronization error of no more than 1μs; the output layer of the AI ​​dynamic tactile rendering engine decomposes the generated tactile feedback signal into force-driven signals and temperature-driven signals, which are output to the triboelectric sensing array and the micro thermoelectric cooling layer, respectively.

Images