Intelligent integrated man-machine interaction system of nervous system flexible sensor

Abstract

The invention relates to an intelligent integrated man-machine interaction system of a nervous system flexible sensor, and belongs to the field of medical engineering. The system comprises a convenient operation layer formed by combining various mechanisms and an interaction capture layer formed by integrating various interaction modules, according to the convenient operation layer, through an eye follow-up positioning system, a stroke ejection system, an automatic limiting device and a straight tactile operation system, the use difficulty of eye movement interaction, stroke test interaction, grabbing interaction and a switch of a user is reduced, and the operation convenience is greatly improved. The interaction capture layer is integrated in a man-machine interaction system through a grabbing interaction module, a pen test interaction module, a posture interaction module, a myoelectricity interaction module, an eye movement interaction module, a voice interaction module and other modules, and early warning and auxiliary diagnosis of nervous system diseases are achieved. And integration and miniaturization are taken into consideration, so that a patient can carry out detection at home. According to the invention, effective integration of various convenient operation mechanisms and various interaction modules can be realized. And a patient can predict nervous system diseases without going to a hospital.

Description

Technical Field

[0001] The invention relates to a nervous system flexible sensor intelligent integrated human-computer interaction system, belonging to the field of medical engineering. Background Art

[0002] Neurological diseases such as Alzheimer's disease, Parkinson's disease, epilepsy, cerebral palsy, etc., are mainly manifested by abnormal movement ability, abnormal physiological signals, impaired touch and hearing, and other abnormal problems, which have a great impact on the patient's ability to live.

[0003] Neurological diseases affect the patient's nervous system function, thereby affecting the patient's ability to move. The nervous system is one of the main systems in the human body that controls movement. It controls the contraction and relaxation of muscles through the connection between motor neurons and muscles, thereby achieving the body's motor function. When the nervous system is damaged or affected by disease, it affects the connection between neurons and muscles, resulting in impaired motor function.

[0004] Different neurological diseases can affect different motor abilities. For example, stroke can affect the patient's limb motor function, leading to limb paralysis, muscle weakness, etc., causing a significant decrease in the patient's hand muscle strength level, seriously affecting the patient's grasping ability; Parkinson's disease can affect the patient's motor coordination ability, leading to tremors and muscle stiffness, and affecting the patient's pen-holding posture and pen stroke traces, such as being unable to draw along a specific trajectory when drawing spiral lines. Parkinson's disease can also affect the eye's ability to move, and Parkinson's patients may encounter difficulties when trying to move them quickly; multiple sclerosis can affect the patient's motor coordination and balance ability, leading to limb incoordination, unstable gait, etc., resulting in a significant difference between the patient's body posture and that of ordinary people.

[0005] Human-computer interaction technology can play an important role in early warning and auxiliary diagnosis of neurological diseases. Various sensors and devices can be used to collect patients' interaction data, such as grip, pen style, posture, eye movement, etc. These interaction data can reflect changes in patients' motor function, cognitive function, etc., thereby helping doctors to make early warnings and auxiliary diagnoses. Traditional neurological disease detection methods often require human participation, including clinician consultations, imaging examinations, etc. The detection of many neurological diseases requires long-term physical examinations and complex data analysis. In addition, for some special cases, in order to ensure the diagnosis, some further tests may need to be performed. Therefore, it is important to develop a neurological disease prediction system that is easy to operate.

[0006] In summary, the development of an early warning system for neurological diseases that includes pen-based, eye movement, grasping, electromyography, posture, voice and other human-computer interaction technologies is of great significance for early warning and auxiliary diagnosis of neurological diseases. Users can use the convenient operation layer in the intelligent integrated human-computer interaction system of the present invention conveniently, and the built-in interactive capture layer senses the various functions of the user's body and compares them with the normal functions of the human body, so that users can prevent neurological diseases at home.

[0007] The purpose of the present invention is to solve the early warning and auxiliary diagnosis of nervous system diseases, and to propose a nervous system flexible sensor intelligent integrated human-computer interaction system for common nervous system functional abnormalities such as pen style, eye movement, grasping, electromyography, posture, and voice.

[0008] The purpose of the present invention is achieved through the following technical solutions.

[0009] A neural system flexible sensor intelligent integrated human-computer interaction system, including a convenient operation layer and an interactive capture layer.

[0010] The convenient operation layer includes: the first platform of audio-visual interaction of the intelligent integrated human-computer interaction system, and the second platform of somatosensory interaction of the intelligent integrated human-computer interaction system.

[0011] The first platform of audio-visual interaction of intelligent integrated human-computer interaction system includes: eye tracking positioning system, voice interaction system; the second platform of somatosensory interaction of intelligent integrated human-computer interaction system is divided into the first step and the second step. The first step is placed with posture detection sensors, and the second step includes: pen ejection system, automatic limit device, direct tactile operating system, and operation keyboard. The interactive capture layer is embedded in the convenient operation layer, including: central processing unit and the pen test interaction module connected to it, grasping interaction module, posture interaction module, eye movement interaction module, electromyography interaction module, voice interaction module, wireless communication module, data storage module, power supply module, digital-to-analog conversion module and external device interface circuit. The user's interaction data is detected through the cooperation between the convenient operation layer and the interactive capture layer.

[0012] The convenient operation layer of the intelligent integrated human-computer interaction system includes: an audio-visual interaction first platform of the intelligent integrated human-computer interaction system, and a somatosensory interaction second platform of the intelligent integrated human-computer interaction system;

[0013] The first audio-visual interaction platform of the intelligent integrated human-computer interaction system includes: an eye tracking positioning system and a voice interaction system;

[0014] The eye tracking positioning system is used to capture the direction in which the user's eyes are looking, and the positioning system is tracked by the follow-up servo, and the positioning result is displayed on the eye movement capture screen;

[0015] The eye tracking positioning system is located on the first audio-visual interaction platform of the intelligent integrated human-computer interaction system;

[0016] The eye tracking positioning system includes a tracking servo, an eyeball capture device, and an eye movement capture screen;

[0017] The follow-up servo is fixed on the first audio-visual interaction platform of the intelligent integrated human-computer interaction system;

[0018] The follow-up servo is connected to the eyeball capture device and is used to drive the eyeball capture device to rotate;

[0019] The eye movement capture screen is fixed on the first audio-visual interaction platform of the intelligent integrated human-computer interaction system, and is used to feed back to the user the specific position captured by the eyeball.

[0020] The voice interaction system is located in the first audio-visual interaction platform of the intelligent integrated human-computer interaction system, and is used to detect the sound made by the user and conduct voice interaction with the user;

[0021] The second platform of somatosensory interaction of the intelligent integrated human-computer interaction system includes: a pen ejection system, a pen test interaction collection pen, an automatic limit device, a direct tactile operating system, an operation keyboard, an interaction capture layer, electrodes such as posture sensors, electromyography sensors, grip sensors, pen touch sensors, and signal collection and detection sensors;

[0022] The second platform for somatosensory interaction of the intelligent integrated human-computer interaction system is located in the intelligent integrated human-computer interaction system and is used for detecting somatosensory interaction of the human body;

[0023] The second platform of somatosensory interaction of the intelligent integrated human-computer interaction system is divided into a first level and a second level;

[0024] The first step is located on the surface of the second platform of the somatosensory interaction of the intelligent integrated human-computer interaction system, and is used to place electrodes such as posture sensors, electromyography sensors, grip sensors, and pen touch sensors, pens, and signal acquisition and detection sensors, etc., to detect the user's posture, electromyography, grip, pen style and other interaction data;

[0025] The second step is used to store various mechanical structures;

[0026] The pen ejection system is equipped with a sensing touch panel. Once the pen test interaction collection pen is taken out, the pen ejection system will eject the sensing touch panel. The user writes on the sensing touch panel, thereby collecting the user's pen touch posture, including the user's pen touch trajectory and the user's pen holding posture.

[0027] The pen-touch ejection system comprises a pen-touch ejection drive motor, a drive motor linkage device, a pen-touch linkage gear, an ejection drive left rack, an ejection follower left slide rail, an induction touch panel, an ejection follower right rack, an ejection follower right slide rail, an induction touch panel front protection device, and an induction touch panel rear protection device;

[0028] The pen-stroke ejection driving motor is fixed on the second step of the second platform of the integrated human-computer interaction system to drive the entire pen-stroke ejection system;

[0029] The pen-stroke ejection drive motor, the drive motor linkage device and the pen-stroke linkage gear are sequentially connected for constant speed transmission;

[0030] The ejection drive left rack is meshed with the stroke linkage gear, and the ejection drive left rack is pushed out through the gear to realize the ejection of the mechanism;

[0031] One end of the ejection follower left slide rail is fixed on the second platform of the first step of the lower half of the intelligent integrated human-computer interaction system, and is used to be fixed to the second platform and connected to the ejection drive left rack.

[0032] The inductive touch panel is respectively fixed to the ejection-driven left rack and the ejection-driven right rack, and the ejection-driven left rack brings the inductive touch panel out;

[0033] The ejection follower right slide rail is connected to the second platform and the ejection drive right rack, and is used to support the induction touch panel, making it more stable when ejecting;

[0034] The inductive touch panel front protection device and the inductive touch panel rear protection device are respectively fixed at the front and rear of the inductive touch panel to play a protective role.

[0035] The automatic limit device is used to connect the first platform of audio-visual interaction of the intelligent integrated human-computer interaction system and the second platform of somatosensory interaction of the intelligent integrated human-computer interaction system. A switch button is installed on it. By pressing the switch button, the intelligent integrated human-computer interaction system automatically opens and stops at a fixed position, limiting the opening and closing angle of the intelligent integrated human-computer interaction system.

[0036] The automatic limit device is located inside the second platform of somatosensory interaction of the intelligent integrated human-computer interaction system;

[0037] The automatic limit device includes a two-stage speed regulating gear support, a support limit screw, a limit transmission motor, a first transmission coupling of the limit motor, a rear end propulsion first transmission gear, a limit rear end propulsion second transmission large gear, a limit rear end propulsion second transmission small gear, a rear end propulsion third transmission gear, an automatic limit mechanical arm first stage front end, and an automatic limit mechanical arm second stage propulsion rear end;

[0038] The secondary speed regulating gear support bracket is fixed on the second somatosensory interaction platform of the intelligent integrated human-computer interaction system through the support bracket limiting screw;

[0039] The limit transmission motor is fixed on the second somatosensory interaction platform of the intelligent integrated human-computer interaction system;

[0040] The limit transmission motor and the first transmission coupling of the limit motor are connected in sequence with the rear end propulsion first transmission gear to form a power source for the automatic limit device;

[0041] The rear-end propulsion first transmission gear meshes with the limited rear-end propulsion second transmission large gear in the transmission secondary gear set to form a first-level speed regulation transmission device. This realizes the first-level regulation of the motor speed and initially reduces the motor speed;

[0042] The limited rear-end propulsion second transmission pinion in the transmission secondary gear set meshes with the rear-end propulsion third transmission gear to form a secondary speed regulation transmission device. This realizes secondary regulation of the motor speed and further reduces the motor speed.

[0043] The rear end propulsion third transmission gear cooperates with the secondary propulsion rear end of the automatic limit mechanical arm, so that the secondary propulsion rear end of the automatic limit mechanical arm moves forward;

[0044] The first-stage front end of the automatic limit mechanical arm is connected to the second-stage propulsion rear end of the automatic limit mechanical arm through a rotating pair;

[0045] The first-level front end of the automatic limit robot arm is connected to the first audio-visual interaction platform of the intelligent integrated human-computer interaction system through a rotating pair, and the switch function is realized by extending the first-level front end of the automatic limit robot arm forward.

[0046] The direct tactile operating system is used to perform gripping interaction with the user, and a convenient interaction experience is achieved by running the direct tactile operating system to detect the user's gripping data.

[0047] The direct tactile operating system is located on the first step of the second platform of the somatosensory interaction of the intelligent integrated human-computer interaction system;

[0048] The direct tactile operating system includes a direct tactile operating system tactile perception platform, a system tactile perception secondary transmission mechanical arm, a direct tactile operating system tactile perception primary transmission mechanical arm, a direct tactile operating system tactile perception platform firmware, and a direct tactile operating system direct tactile operating system drive motor;

[0049] The firmware of the tactile perception platform of the direct tactile operating system is fixed on the first step of the second platform of the somatosensory interaction of the intelligent integrated human-computer interaction system;

[0050] The direct tactile operating system tactile perception platform rotates around the direct tactile operating system tactile perception platform firmware and is used to store the grasping interaction device;

[0051] The direct tactile operating system tactile perception secondary transmission mechanical arm, the direct tactile operating system tactile perception primary transmission mechanical arm and the direct tactile operating system tactile perception platform are connected in sequence, and the direct tactile operating system tactile perception primary transmission mechanical arm is rotated to raise the direct tactile operating system tactile perception platform.

[0052] The direct tactile operating system driving motor is connected to the direct tactile operating system tactile perception primary transmission mechanical arm and is used to drive the entire operating system.

[0053] The operation keyboard is located on the first step of the second platform of the somatosensory interaction of the intelligent integrated human-computer interaction system;

[0054] The interaction capture layer of the intelligent integrated human-computer interaction system includes: a central processing unit and a voice interaction module, a written test interaction module, a posture interaction module, a digital-to-analog conversion module, a data communication module, a data storage module, a power supply module, an external device socket circuit, and an electromyography interaction module, an eye movement interaction module, and a grasping interaction module connected to the central processing unit through the digital-to-analog conversion module.

[0055] The central processor receives signals such as electrooculogram signals, electromyography signals, posture acquisition, image processing, and voice recognition. The data storage module temporarily stores the detection data, which is then uploaded to the client through the data communication module and displayed on the acquisition client.

[0056] The written test interaction module includes a sensing touch panel and a written test interaction collection pen. The written test interaction collection pen includes a pen tip pressure collection sensor and a posture collection sensor. The written test interaction module mainly records the angle, direction and speed information during the writing process, collects the force during the writing process by collecting the pen tip pressure, and can collect the entire process characteristics of the written test taker's writing through two sensors.

[0057] The eye movement interaction module includes an electrode sheet, a signal shield, a preamplifier circuit, a bandpass filter circuit, a right leg drive circuit, and a final gain circuit. The eye movement signal is indirectly obtained by setting electrodes on the skin surface around the eyeball, and the electrooculogram signal is shielded by a signal shield. The shielding end is connected to the same signal as the input signal, and the equipotential shielding method is used to ensure that the electrooculogram signal reduces interference before entering the amplifier circuit, and the signal is amplified and filtered to meet the electrooculogram signal acquisition requirements.

[0058] The gripping interaction module includes a flexible sensor interface circuit and a flexible sensor filter circuit. The pressure signals at each location of the inner palm and the joints of the fingers are recorded when the palm exerts force to grasp the object, and the digital-to-analog conversion is performed through the digital-to-analog conversion module to read the pressure. The flexible sensor interface circuit is mainly used to connect to the flexible sensor. The circuit uses 16 flexible sensors to collect the pressure signals at the inner palm and each joint respectively, and the digital-to-analog conversion is performed through this part of the digital-to-analog conversion circuit to read the pressure signal. The flexible sensor filter circuit is used to filter the pressure signal.

[0059] The electromyographic interaction module mainly performs electromyographic signal pickup and differential amplification circuit. The DC signal generated by human sweating, constant electromagnetic field interference and other conditions is superimposed on the surface electromyographic signal. After high-multiple amplification by the subsequent main amplifier circuit, it is easy to cause the amplifier to saturate and fail to detect effective signals. High-pass filters are used to filter out DC signals and low-frequency noise. High-pass filters are filters that allow high-frequency signals to pass, but weaken signals with frequencies lower than the cutoff frequency.

[0060] The posture interaction module includes a posture acquisition circuit and a posture acquisition sensor, which mainly collects the user's body posture. The sensor can quickly respond to the user's posture status and give accurate data judgment.

[0061] The voice interaction module includes an audio acquisition circuit and an audio output circuit, which are mainly used for human-computer voice interaction recognition, collecting voice commands and voice information issued by humans, and sending them to the central processor for processing after recognition and conversion.

[0062] The digital-to-analog conversion module mainly converts the pressure signals, electrooculogram signals and electromyography signals collected and processed by the grasping interaction module, eye movement interaction module and electromyography interaction module into data signals readable by the central processing unit.

[0063] The data communication module realizes wireless transmission of signals between the central processor and the client.

[0064] The data storage module is used for flash memory storage of data.

[0065] The power supply module includes a DC adapter, a lithium battery charging circuit, and a voltage-stabilized power supply circuit. The power supply module supplies power to the central processor, eye movement interaction module, voice interaction module, written test interaction module, grasping interaction module, posture interaction module, electromyography interaction module, digital-to-analog conversion module, data communication module, and data storage module through 1.6V, 1.8V, 3V, 3.3V, 4V, -5V, and 5V voltage stabilization.

[0066] Beneficial Effects

[0067] 1. The intelligent integrated human-computer interaction system of the present invention realizes the effective integration of multifunctional interactive devices. Aiming at the early prevention of neurological diseases, it combines the grasping interaction module, the eye movement interaction module, the written test interaction module, the electromyography interaction module, the posture interaction module, and the voice interaction module to comprehensively detect the user's interaction situation, efficiently predict neurological diseases, and improve the prediction effect.

[0068] 2. The intelligent integrated human-computer interaction system of the present invention greatly improves convenience through the eye tracking positioning system, pen ejection system, automatic limit device, and direct tactile operating system, reduces the difficulty of eye movement interaction, pen test interaction, grip interaction, and switch use, and greatly improves the convenience of operation. While integrating and miniaturizing, patients can be tested at home

[0069] 3. The grasping interaction module, pen test interaction module, posture interaction module, electromyography interaction module, eye movement interaction module, voice interaction module and other modules are all integrated into a human-computer interaction system. While integrating, it also takes miniaturization into consideration, so that patients can be tested at home.

[0070] 4. At the circuit level, filter circuits, boost circuits and audio output circuits are designed to make the output and captured signals more accurate and realize audio interaction functions. Multiple data such as pen stroke information, electrooculographic signals, grip pressure signals, electromyographic signals, posture information, and language signals are used to jointly judge neurological diseases, and multi-sensor cross-validation is used to greatly improve the prediction probability. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is the overall architecture of the intelligent integrated human-computer interaction system of the present invention;

[0072] Figure 2 It is the intelligent integrated human-computer interaction system structure of the present invention;

[0073] Among them, 1.1 intelligent integrated human-computer interaction system is the first platform for audio-visual interaction, and 1.2 intelligent integrated human-computer interaction system is the second platform for somatosensory interaction.

[0074] 2.1 Intelligent integrated human-computer interaction system audio-visual interaction The first platform is deployed with 1.1.1 eye movement interaction module and 1.1.2 voice interaction module. 1.2 Intelligent integrated human-computer interaction system somatosensory interaction The second platform is deployed with 1.2.1 electromyography interaction module, 1.2.2 grasping interaction module, 1.2.3 keyboard, 1.2.4 written test interaction module, 1.2.5 posture interaction module, and 1.2.6 written test interaction collection pen.

[0075] Figure 3 It is a schematic diagram of the internal structure of the intelligent integrated human-computer interaction system of the present invention;

[0076] Among them, 2.1 automatic limit device, 2.2 pen ejection system, 2.3 direct tactile operating system, 2.4 eye tracking positioning system, 2.5 interactive capture layer integrated board;

[0077] Figure 4 It is a schematic diagram of an automatic limit device of an intelligent integrated human-machine interaction system of the present invention;

[0078] Among them, 3.1-the first-level front end of the automatic limit device mechanical arm, 3.2-the rear end of the automatic limit device mechanical arm, 3.3-the second transmission gear of the rear end of the automatic limit device, 3.4-the second transmission pinion of the rear end of the automatic limit device, 3.5-the second-level speed regulating gear support of the automatic limit device, 3.6-the limiting screw of the support support of the automatic limit device, 3.7-the transmission motor of the automatic limit device, 3.8-the first transmission coupling of the limit motor of the automatic limit device, 3.9-the first transmission gear of the rear end of the automatic limit device, 3.10-the third transmission gear of the rear end of the automatic limit device;

[0079] Figure 5 It is a pen-stroke ejection system of the intelligent integrated human-computer interaction system of the present invention;

[0080] Among them, 4.1-pen-touch ejection system ejection follower right slide rail, 4.2-pen-touch ejection system ejection follower right rack, 4.3-pen-touch ejection system induction touch panel, 4.4-pen-touch ejection system induction touch panel front protection device, 4.5-pen-touch ejection system drive motor linkage device, 4.6-pen-touch ejection system pen-touch ejection drive motor, 4.7-pen-touch ejection system ejection follower left slide rail, 4.8-pen-touch ejection system ejection drive left rack, 4.9-pen-touch ejection system pen-touch linkage gear, 4.10-pen-touch ejection system induction touch panel rear protection device;

[0081] Figure 6 is a schematic diagram of a direct tactile operating system of the intelligent integrated human-computer interaction system of the present invention;

[0082] Among them, 5.1 Direct Tactile Operating System Tactile Perception Platform, 5.2 Direct Tactile Operating System Tactile Perception Platform Firmware, 5.3 Direct Tactile Operating System Tactile Perception Secondary Transmission Robotic Arm, 5.4 Direct Tactile Operating System Tactile Perception First-Level Transmission Robotic Arm, 5.5 Direct Tactile Operating System Drive Motor;

[0083] Figure 7 It is a schematic diagram of the eye tracking positioning system of the intelligent integrated human-computer interaction system of the present invention;

[0084] Among them, 6.1 eye movement capture screen, 6.2 eyeball capture device, 6.3 follow-up servo;

[0085] Figure 8 It is the overall framework diagram of the interaction capture layer;

[0086] Fig. 9 It is a schematic diagram of the interactive capture layer PCB drawing board;

[0087] Fig.10 It is a circuit diagram of the pen tip pressure collection sensor and posture collection sensor of the pen test interaction module;

[0088] Fig.11 It is the schematic diagram of the preamplifier circuit, right leg drive circuit, bandpass filter circuit, and final gain circuit of the eye movement interaction module;

[0089] Fig.12 is a schematic diagram of a flexible sensor interface circuit and a flexible sensor filter circuit of a grasping interaction module;

[0090] Fig.13 It is a schematic diagram of the differential amplifier circuit, bandpass filter circuit and suffix gain circuit of the electromyographic interaction module;

[0091] Fig.14 It is the audio acquisition circuit and audio output circuit of the voice interaction module. DETAILED DESCRIPTION

[0092] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and embodiments.

[0093] Example 1

[0094] like Figure 1 , Figure 3 As shown, the intelligent integrated human-computer interaction system of the present invention includes 2.1 an intelligent integrated human-computer interaction system audio-visual interaction first platform, and 1.2 an intelligent integrated human-computer interaction system somatosensory interaction second platform.

[0095] Among them, the first platform of 1.1 intelligent integrated human-computer interaction system audio-visual interaction deploys 1.1.1 eye movement interaction module and 1.1.2 voice interaction module. The second platform of 1.2 intelligent integrated human-computer interaction system somatosensory interaction deploys 1.2.1 myoelectric interaction module,

[0096] 1.2.2 Grasp interaction module, 1.2.3 Keyboard, 1.2.4 Written test interaction module, 1.2.5 Posture interaction module, 1.2.6 Written test interaction collection pen.

[0097] Further, such as Figure 2 As shown, 1.2 The second platform of audio-visual interaction of intelligent integrated human-computer interaction system includes: 2.1 automatic limit device, 2.2 pen ejection system, 2.3 direct tactile operating system, 1.1 The first platform of audio-visual interaction of intelligent integrated human-computer interaction system includes, 2.4 eye tracking positioning system.

[0098] 2.1 The automatic limit device is used to connect the first audio-visual interaction platform of the intelligent integrated human-computer interaction system and the second audio-visual interaction platform of the intelligent integrated human-computer interaction system;

[0099] The opening and closing angle of the automatic limit device is between 0 and 120 degrees. When it reaches 120 degrees, it stops automatically to achieve automatic limit.

[0100] Further, such as Figure 4 As shown, the automatic limit device 2.1 includes: a primary front end 3.1 of an automatic limit mechanical arm, a secondary propulsion rear end 3.2 of an automatic limit mechanical arm, a second transmission large gear 3.3 of a limit rear end propulsion, a second transmission small gear 3.4 of a limit rear end propulsion, a secondary speed regulating gear support 3.5, a support support limiting screw 3.6, a limit transmission motor 3.7, a first transmission coupling 3.8 of a limit motor, a first transmission gear 3.9 of a rear end propulsion, and a third transmission gear 3.10 of a rear end propulsion;

[0101] In the embodiment, the secondary speed regulating gear support bracket 3.5 is fixed on the second somatosensory interaction platform of the intelligent integrated human-machine interaction system through the support bracket limiting screw 3.6;

[0102] The limit transmission motor 3.7 is fixed on the second platform of the somatosensory interaction of the intelligent integrated human-machine interaction system;

[0103] The limit transmission motor 3.7 and the first transmission coupling 3.8 of the limit motor are connected in sequence with the rear end propulsion first transmission gear 3.9 to form a power source for the automatic limit device;

[0104] In the embodiment, the rear-end propulsion first transmission gear 3.9 is meshed with the limited rear-end propulsion second transmission large gear 3.3 in the transmission secondary gear set to form a primary speed regulating transmission device. The primary regulation of the motor speed is achieved to initially reduce the motor speed;

[0105] The limited rear-end propulsion second transmission pinion 3.4 in the transmission secondary gear set meshes with the rear-end propulsion third transmission gear 3.10 to form a secondary speed regulation transmission device. This realizes secondary regulation of the motor speed, further reduces the motor speed, and realizes speed regulation of the click;

[0106] The rear end propulsion third transmission gear 3.10 cooperates with the automatic limit mechanical arm secondary propulsion rear end 3.2, so that the automatic limit mechanical arm secondary propulsion rear end 3.2 moves forward;

[0107] The first-stage front end 3.1 of the automatic limit robot arm is connected to the second-stage propulsion rear end 3.2 of the automatic limit robot arm through a rotary pair;

[0108] The first-level front end 3.1 of the automatic limit robot arm is connected to the first audio-visual interaction platform of the intelligent integrated human-machine interaction system through a rotary pair, and the first-level front end 3.1 of the automatic limit robot arm is extended forward to realize the switch function.

[0109] Further, such as Figure 5 As shown, the pen-touch ejection system 2.2 comprises: an ejection follower right slide rail 4.1, an ejection follower right rack 4.2, an inductive touch panel 4.3, an inductive touch panel front protection device 4.4, a drive motor linkage device 4.5, a pen-touch ejection drive motor 4.6, an ejection follower left slide rail 4.7, an ejection drive left rack 4.8, a pen-touch linkage gear 4.9, and an inductive touch panel rear protection device 4.10;

[0110] The pen ejection system is equipped with a sensing touch panel 4.3. Once the 1.2.6 pen test interaction collection pen is taken out, the pen ejection system will eject the sensing touch panel 4.3, and predict whether the user is ill by collecting the pen stroke trajectory and the user's pen holding posture.

[0111] The pen ejection drive motor 4.6 is fixed on the second step of the second platform of the integrated human-machine interaction system to drive the entire pen ejection system;

[0112] The pen-stroke ejection driving motor 4.6, the driving motor linkage device 4.5 and the pen-stroke linkage gear 4.9 are sequentially connected for constant speed transmission to move the induction touch panel 4.3 out at a constant speed;

[0113] The ejection drive left rack 4.8 is meshed with the stroke linkage gear 4.9, and the ejection drive left rack 4.8 is pushed out through the gear to realize the ejection of the mechanism;

[0114] One end of the ejection follower left slide rail 4.7 is fixed on the second platform of the first step of the lower half of the intelligent integrated human-machine interaction system, and is used to be fixed to the second platform and connected to the ejection drive left rack 4.8.

[0115] The inductive touch panel 4.3 is fixed to the ejection drive left rack 4.8 and the ejection drive right rack 4.2 respectively, and the ejection drive left rack brings out the inductive touch panel 4.3 to support the entire inductive touch panel 4.3;

[0116] In the embodiment, the inductive touch panel 4.3 adopts a 7-inch HDMI LCD capacitive touch screen IPS high-definition compatible with Raspberry Pi sold by Shenzhen Youxin Electronic Technology Co., Ltd. The size is 165*102mm, the total product weight is 510g, with a 1024*600 high-definition resolution, up to 178-degree full-viewing angle panel, and capacitive touch sensing. Built-in backlight adjustment button, the backlight brightness can be adjusted as needed, and a high-fidelity stereo dual-speaker speaker system is optional, supporting mainstream development boards such as Raspberry Pi, BananaPi, EAIDK, Jetson Nan, BB Black, etc.

[0117] The ejection follower right slide rail 4.1 is connected to the second platform and the ejection drive right rack 4.2, and is used to support the induction touch panel, making it more stable when ejecting;

[0118] The inductive touch panel front protection device 4.4 and the inductive touch panel rear protection device 4.10 are respectively fixed to the front and rear of the inductive touch panel 4.3 to play a protective role;

[0119] Further, such as Figure 6 As shown, the direct tactile operating system 2.3 includes: a direct tactile operating system tactile perception platform 5.1, a direct tactile operating system tactile perception platform firmware 5.2, a direct tactile operating system tactile perception secondary transmission mechanical arm 5.3, a direct tactile operating system tactile perception primary transmission mechanical arm 5.4, and a direct tactile operating system drive motor 5.5;

[0120] The direct tactile operating system is used to interact with the user through grasping. It runs through the direct tactile operating system to achieve a convenient interactive experience, detect the user's grasping data, and thus make predictions about the user.

[0121] In the embodiment, the grasping interaction uses the Meta Quest Touch Pro VR handle developed by Meat, which has the advantage of using an automatic tracking camera to convert gestures into VR more smoothly, reliably and accurately, so that a wider range of motion can be enjoyed in a wider range. The tracking ring of the traditional VR handle is omitted, which improves balance and allows movements closer to the face. Detailed operations are performed through fine motor control (such as precise pinching movements, joysticks and stylus tips), making precise small movements (including writing and drawing) possible, more accurately detecting the user's grasping interaction, and improving prediction accuracy.

[0122] The direct tactile operating system tactile perception platform 5.1 rotates around the direct tactile operating system tactile perception platform firmware 5.2, and is used to store the grasping interaction device, and to perform grasping interaction through the reserved notch;

[0123] The direct tactile operating system tactile perception secondary transmission robotic arm 5.3, the direct tactile operating system tactile perception primary transmission robotic arm 5.4 and the direct tactile operating system tactile perception platform 5.1 are connected in sequence. By rotating the direct tactile operating system tactile perception primary transmission robotic arm 5.4, the direct tactile operating system tactile perception platform 5.1 is raised, thereby realizing convenience for users.

[0124] The direct tactile operating system driving motor 5.5 is connected to the direct tactile operating system tactile perception primary transmission mechanical arm 5.4, and is used to drive the entire operating system and provide power for the entire system;

[0125] Further, such as Figure 3 As shown, the first audio-visual interaction platform of the intelligent integrated human-computer interaction system includes: eye tracking positioning system 2.4, voice interaction system 1.1.2;

[0126] like Figure 7 As shown, the eye tracking positioning system 2.4 includes: a tracking servo 6.1, an eyeball capture device 6.2, and an eye movement capture screen 6.3.

[0127] The eye tracking positioning system captures the user's eye movement data through an eye capture device, adjusts the direction of the eye capture device through a follow-up servo so that it is always positioned around the human eye, and finally displays the data on the screen in real time through an eye capture screen, forming eye movement interactive feedback.

[0128] In the embodiment, the eye movement capture device is the Tobii Eye Tracker 5 produced by Tobii Technology, which has an overall length of 285 mm, a width of 8.2 mm, and a height of 15 mm. An ISS illuminator with a custom Tobii sensor is used, with a viewing angle of 40*40 degrees, which can reach 80*80 degrees by connecting with a follow-up servo. The head tracking uses a combination of CPU+neural network technology, which can accurately locate the head.

[0129] like Figure 8 As shown, the interaction capture layer of the intelligent integrated human-computer interaction system includes: a central processing unit and a voice interaction module, a written test interaction module, a posture interaction module, a digital-to-analog conversion module, a data communication module, a data storage module, a power supply module, and an electromyography interaction module, an eye movement interaction module, and a grasping interaction module connected to the central processing unit through the digital-to-analog conversion module.

[0130] The central processor chip is preferably an STM32F103RCT6 chip. The STM32F103RCT6 chip has a comprehensive set of power saving modes that allow the design of low-power applications, and is widely used in motor drive, application control, medical and handheld devices, etc. It mainly receives signals such as electrooculogram signals and electromyography signals, gesture acquisition, image processing, and speech recognition, and then uploads them to the client through the data communication module and displays them on the acquisition client.

[0131] The written test interaction module includes a sensing touch panel and a 1.2.6 written test interaction collection pen.

[0132] The inductive touch panel module in the pen test interaction module is fixed on the pen stroke ejection system and is ejected by the pen stroke ejection system, and is mainly used for collecting pen stroke traces.

[0133] like Fig.10 As shown, the written test interactive collection pen has a built-in written test interactive collection circuit, including a pen tip pressure collection sensor and a posture collection sensor. The posture collection sensor uses MPU6050, and the pen tip pressure collection sensor uses FSR402 piezoresistor sensor. The MPU6050 sensor mainly records the angle, direction and speed information during the writing process. The FSR402 sensor mainly collects the force during the writing process through the pen tip pressure transmission. The two sensors can collect the entire process characteristics of the entire written test taker's writing.

[0134] The pen stroke traces collected by the pen test interaction module are directly displayed on the inductive touch panel. The pen stroke data of ordinary people are displayed on the display screen of the first platform of the intelligent integrated human-computer interaction system.

[0135] like Fig.11 As shown, the eye movement interaction module includes an electrode sheet, a signal shield, a preamplifier circuit, a bandpass filter circuit, a right leg drive circuit, and a final gain circuit.

[0136] The signal shielding shields the electrooculogram signal, and the shielding end is connected to a signal that is the same as the input signal, so as to minimize the interference of the electrooculogram signal before entering the amplification circuit.

[0137] The preamplifier circuit is composed of a high-pass filter circuit, a follower circuit and a differential amplifier circuit. The electrooculogram signal is a differential signal, and the acquisition process is easily interfered by other signals, such as power frequency interference, etc. Each eye has a collection circuit. The common-mode amplitude of the electrooculogram signal is mainly in the range of mV to tens of mV, which is less than the standard for accurate reading and conversion of the ADC. Therefore, the signal needs to be amplified while preventing interference. The electrooculogram signal passes through two high-pass filters composed of C43, R34 and C51, R38 to limit the polarization interference signal of DC. The cut-off frequency calculation formula is:

[0138]

[0139] Then the signal is buffered by two follower circuits composed of AD746 dual operational amplifiers to prevent the peak voltage from damaging the chip. The electrooculogram differential mode signal is input into the differential amplifier circuit composed of INA333 to amplify the signal by 197 times. The calculation formula is:

[0140]

[0141] The INA333 chip has a very high common mode rejection ratio and a large input impedance, and the amplification range is 0-1000. The INA333 amplifier is selected in the preamplifier circuit to meet the high common mode rejection ratio requirements of the preamplifier circuit. The higher the gain, the greater the common mode rejection ratio. Due to the possible interference components of the collected signal, in order to prevent the interference from being amplified, the gain of the preamplifier circuit should not be too high.

[0142] The right leg driving circuit selects a point far from the input signal as an equipotential point to achieve equipotential shielding. Adding a cable shielding signal to the right leg driving circuit is helpful to achieve equipotential shielding.

[0143] The bandpass filter circuit is composed of a first-order high-pass filter and a second-order low-pass filter. Since the frequency of the electrooculogram signal is mainly concentrated in 0-40Hz, a bandpass filter is used to filter out signals outside 0-40Hz. The pre-high-pass filter adopts a first-order high-pass filter, which is composed of C47 and R37, and the cut-off frequency is:

[0144]

[0145] Therefore, the cut-off frequency is 2Hz.

[0146] When the frequency of the signal is higher than the cut-off frequency of the filter, the signal will decay at a rate of 40dB / dec. Therefore, for high-frequency signals, the second-order low-pass filter has a better suppression effect than the first-order low-pass filter. Therefore, the low-pass filter of the present invention adopts a second-order low-pass filter, which is composed of R35, R36, C42, and C49 to mainly filter out high-frequency component signals. Since the amplification factor of the primary differential amplifier should not be too high, it is only amplified by 197 times. At this time, the electrooculogram signal still cannot meet the standard of ADC analog-to-digital conversion. While filtering out the high-frequency signal, the electrooculogram signal needs to be amplified again. The overall amplification factor is about 2500 times.

[0147] The final gain circuit uses AD746 to form a common-phase amplifier for secondary amplification. The amplification factor is mainly determined by R29 and R28:

[0148]

[0149] Therefore, the magnification is 5 times.

[0150] In order to ensure that the overall gain is about 2500 times, the second-order low-pass filter has a 2-3 times gain effect while filtering out high-frequency components. The cut-off frequency of the second-order low-pass filter is:

[0151]

[0152] Therefore, the cut-off frequency is 38.7Hz and the gain multiple is:

[0153]

[0154] Therefore, the gain is 2.5 times. Therefore, the pass frequency of the entire bandpass filter circuit is 2Hz-38.7Hz. The entire eye signal amplification factor is:

[0155] G z =197*5*2.5=2462.5

[0156] Therefore, the overall magnification is 2462.5 times, and the overall acquisition meets the requirements for electrooculogram signal acquisition.

[0157] like Fig.12 As shown, the gripping interaction module includes a flexible sensor interface circuit and a flexible sensor filter circuit. The flexible sensing signal acquisition device uses FRS flexible sensors. A total of 16 FRS flexible sensors are placed on the inner side of the human palm and at each joint of the fingers. When the palm exerts force to grip the object, each pressure signal can be recorded, and the digital-to-analog conversion is performed through the digital-to-analog conversion module to read the pressure. The number of grips per minute collected by the flexible sensor and the gripping force curve data are mainly transmitted to the display screen of the first platform of the intelligent integrated human-computer interaction system through wired communication, and the curve is drawn, and the flexible sensor filter circuit is used to filter it. The flexible sensor interface circuit is embedded in the direct tactile operating system, and is lifted up as the direct tactile operating system is lifted up. It is mainly used to connect to the flexible sensor. The flexible sensor filter circuit uses the AD8221 chip. AD8221ARZ-R7 is a high-performance instrumentation amplifier with programmable gain and a high common mode rejection ratio. The high CMRR at frequency enables the AD8221-EP to suppress broadband interference and line harmonics, greatly simplifying the filter requirements. Low voltage offset, low offset drift, low gain drift, high gain accuracy. A single resistor sets the gain from 1 to 1000. Operates on single and dual supplies, ideal for applications encountering ±10V input voltages. The AD8221ARZ-R7 is specified over the military temperature range of -55°C to +125°C. It has an 8-lead MSOP package. A single resistor sets the gain from 1 to 1000. Available in an 8-lead MSOP package.

[0158] like Fig.13As shown, the myoelectric interaction module is mainly composed of electrode sheets, differential amplifier circuits, bandpass filter circuits, and suffix gain circuits.

[0159] The electrode sheet is connected to the circuit through the terminal H1.

[0160] The differential amplifier circuit and the common-mode and differential-mode suppression parts composed of C14, C10, and C16 complete the electromyographic signal acquisition. The differential amplifier circuit is composed of INA331IDGKT, which realizes differential amplification of the weak surface electromyographic signal obtained from the surface electrode. The electromyographic signal is first subjected to differential-mode interference suppression by the C14 differential-mode capacitor, and common-mode interference suppression by the C10 and C16 common-mode capacitors. Then, an instrument amplifier with a high common-mode rejection ratio is used for common-mode interference. At the same time, the internal gain of INA331 is set to 5, and an external gain adjustment circuit is formed by using R15 and R20. The gain formula is as follows:

[0161]

[0162] Therefore, the external gain of the differential amplifier circuit is 10 times.

[0163] The bandpass circuit uses an AD746 operational amplifier chip. The electromyographic signal on the surface of human skin is mainly concentrated in 10-150Hz. The cut-off frequency of the low-pass filter of the present invention is:

[0164]

[0165] Therefore, the cutoff frequency of the low-pass filter is designed to be 153.1Hz.

[0166] The high-pass filter uses an AD746 operational amplifier chip. The DC signal generated by human sweating, constant electromagnetic field interference and other conditions is superimposed on the surface electromyography signal. After high-power amplification in the subsequent main amplifier circuit, it is easy to cause the amplifier to saturate and fail to detect effective signals. High-pass filters are used to filter out DC signals and low-frequency noise. High-pass filters are filters that allow high-frequency signals to pass through, but weaken signals with frequencies lower than the cutoff frequency. The cutoff frequency of the high-pass filter is:

[0167]

[0168] Therefore, the cutoff frequency of the low-pass filter is designed to be 15.3Hz.

[0169] The suffix gain circuit uses the AD746 operational amplifier chip. Since the human electromyographic signal is weak, only at the millivolt level, the signal after filtering cannot be collected by the ADC chip, so it needs to be amplified by an amplifier before it can be input into the ADC chip. The gain calculation is:

[0170]

[0171] Therefore, the gain of the suffix gain amplifier is 101 times, so the gain of the overall electromyographic signal is G1×G S =10×101=1010 times.

[0172] The posture interaction module mainly collects the user's body posture. Through this sensor, it can quickly respond to the user's posture status and give accurate data judgment. The LSM6DSLTR chip is selected, which has a system-level packaged digital accelerometer and 3D digital gyroscope with 3D function. The performance in high-performance mode is 0.65mA, realizing the low-power function that is always on. The full-scale acceleration range is ±2 / ±4 / ±8 / ±16g, and the angular velocity range is ±125 / ±245 / ±500 / ±1000 / ±2000dps, which meets the high durability against mechanical shock. The posture interaction module transmits the collected human posture data to the controller, and the controller draws the human body line diagram and draws it on the display screen of the first platform of the intelligent integrated human-computer interaction system.

[0173] like Fig.14 As shown, the voice interaction module includes an audio acquisition circuit and an audio output circuit, which are mainly used for human-computer voice interaction recognition. The voice commands and voice information issued by humans are collected through sensors, and are converted to the central processor for processing after recognition. The audio acquisition circuit uses the MP34DT05 chip, an ultra-compact, low-power, omnidirectional, digital, low-distortion MEMS microphone, and uses capacitive sensing elements and IC interfaces. The sensing element can detect sound waves, and the IC interface is manufactured using CMOS technology, allowing the design of a dedicated circuit that can provide digital signals in PDM format externally. The signal-to-noise ratio is 64dB and the sensitivity is -26dBFS±3dB. It uses a top port, is compatible with SMD and EMI shielded packages, and operates in an extended temperature range of -40°C to +85°C. The audio output circuit is used for human-computer voice prompts. The audio output circuit uses the AD706AR chip, a dual low drift operational amplifier that provides BiFET-level input bias current, but has the low IB drift of a bipolar amplifier for use in circuits using dual operational amplifiers such as the LT1024, providing low drift and high DC accuracy. The present invention is used for the amplified output of audio signals.

[0174] The external device socket circuit uses the USB3300 chip and adopts the optimized ULPI design. Through the in-band signaling and status byte conversion between the link layer and the physical layer, the number of pins of the UTMI+ interface is compressed from 54 to 12. With a ULPI interface, a seamless connection from ULPI to the link layer interface is provided without the need for a UTMI-to-ULPI wrapper.

[0175] The ADC analog-to-digital conversion circuit uses the AD7689 chip and the ADS1255 chip. The AD7689 chip is an 8-channel 16-bit charge redistribution successive approximation register (SAR) analog-to-digital converter (ADC) that can operate using a single power supply VDD. It can operate in a temperature range of -40°C to +85°C. In this design, the flexible sensor is set to collect 16 points. The pressure generated by the change of the human hand is sensed by the flexible sensor, which acts on the flexible sensor, thereby generating changes in electrical signals, and the analog-to-digital conversion is performed by the AD7689. By adopting a dual ADC chip design, the needs of this design can be met. The ADS1255 chip is low-noise, and the onboard register stores setting variables, such as multiplexer options, sensor detection current, etc. The clock source is provided by an external crystal oscillator. The general digital I / O provides static read and write control, up to 4 pins. One of the pins can also be used to provide a programmable clock output. The analog-to-digital conversion circuit mainly converts the signals processed by the electromyographic signal acquisition circuit and the electrooculographic signal acquisition circuit into data signals readable by the MCU.

[0176] The wireless communication circuit preferably uses the Air724UG chip, which is an IoT communication module based on the UIS8910DM chipset. The module has excellent communication performance, complies with the Cat.1 communication standard, and supports data transmission at a maximum downlink rate of 10Mbps and a maximum uplink rate of 5Mbps. It has full-function audio input and output interfaces, LCD interface, Cammer interface, memory card interface, matrix keyboard interface, ADC interface, and complete peripherals. The Air724UG module is mainly used to realize wireless signal transmission with the client, and the data signal collected by the flexible sensor is received through the module and given to the central processing unit module. Because it has a very strong communication function, it can meet the wireless communication needs of this design.

[0177] The data storage module uses FLASH storage composed of W25Q64 and flash memory data storage composed of W25N01GVZEIG. The serial FLASH circuit uses the W25Q64 chip. It is powered by a single power supply of 3.6-4.7V, consumes current as low as 1μA, and has an operating range of 40°C to +85°C. It supports 1 2,048-byte unique ID page, 1 2,048-byte parameter page, and 10 2,048-byte OTP pages for flash memory storage of data. The W25N01GVZEIG chip integrates an SPI interface and NAND non-volatile storage space. It supports 1 2,048-byte unique ID page, 1 2,048-byte parameter page, and 10 2,048-byte OTP pages for flash memory storage of data.

[0178] The power supply module includes a DC adapter, a lithium battery charging circuit and a voltage-stabilizing power supply circuit.

[0179] The DC power adapter converts an external power supply with an AC voltage of 100-240V into a DC mode for normal operation of the neural system flexible sensor intelligent integrated human-computer interaction system.

[0180] The lithium battery charging circuit uses the TP4056X linear lithium-ion battery charger. It has positive and negative reverse connection protection, input power supply voltage of -6.5~12V, programmable charging current up to 1000mA, 4.9V trickle charging, 6.2V preset charging voltage with an accuracy of ±1%, charging current controller output for battery power detection, dual output of charging status, no battery and fault status display.

[0181] The voltage-stabilizing power supply circuit is mainly used to stabilize the 6.2V voltage of the lithium battery to the 1.6V, 1.8V, 3V, 3.3V, 4V, -5V, and 5V voltages required by the system. At the same time, it maintains good stability. The 5.3V voltage-stabilizing circuit uses the RT9013-33GB chip. The RT9013-33GB chip has a fixed positive output, a voltage output of 5.3V, a current output of 500mA, a voltage drop (typical value): 0.25V@500mA, 1 voltage regulator, a maximum voltage input of 5.4V, and an operating temperature of -40 to 85°C.

[0182] The boost circuit uses NMOS as the main switch and PMOS as the synchronous switch. It can convert an output voltage of up to 5.5V, and convert an input voltage of 2.0-5.0V into a voltage of 2.5V-5.5V for output. The boost circuit uses the SY7088 chip, with a maximum input voltage of 5.0V, a maximum output of 5.5V, and a maximum current output of 3A peak current. It has the characteristics of input undervoltage lockout, automatic disconnection when shutting down, and output overvoltage protection. The operating temperature is -40 to 80°C.

[0183] Embodiment 2:

[0184] Place the neural system flexible sensor intelligent integrated human-computer interaction system flat on the table, and the user presses the power button, the automatic limit device will automatically open, and the intelligent integrated human-computer interaction system audio-visual interaction first platform will be pushed to a certain position and then stop.

[0185] The user selects the eye movement interaction mode in the intelligent integrated human-computer interaction system, and the eye movement data of the user is captured by the eye movement capture device. The direction of the eye movement capture device is adjusted by the follow-up servo so that it is always positioned around the human eye. Finally, the data is displayed on the screen in real time through the eye movement capture screen, forming eye movement interaction feedback.

[0186] The user selects the electromyographic interaction mode in the intelligent integrated human-computer interaction system, and the patient wears the electromyographic interaction module. The sensors deployed in the electromyographic interaction module begin to detect the patient's electromyographic signals, store and record the electromyographic signals, and display them on the display screen.

[0187] The user selects the posture interaction mode in the intelligent integrated human-computer interaction system, and the user wears the posture interaction module on the body, and the sensor deployed in the electromyography interaction module starts to detect the user's posture.

[0188] The user selects the gripping interaction mode in the intelligent integrated human-computer interaction system, and the direct tactile operating system starts to operate, lifting the gripping interaction sensor to facilitate the user to take it out. The flexible sensor on the gripping interaction module detects various data of the patient's hand grip and evaluates the user's muscle strength level.

[0189] When the user selects the pen test interaction mode in the intelligent integrated human-computer interaction system, the pen stroke ejection system will pop up. By using the pen test interaction collection pen to draw on the sensor touch panel, the pen test interaction module will compare the pen stroke traces with the standard pattern to achieve the prediction of neurological diseases.

Claims

1. Nervous system flexible sensor intelligent integrated human-computer interaction system, characterized in that: It includes convenient operation layer and interactive capture layer; among them, The convenient operation layer includes: the first platform of audio-visual interaction of the intelligent integrated human-computer interaction system, and the second platform of somatosensory interaction of the intelligent integrated human-computer interaction system; The first platform of audio-visual interaction of intelligent integrated human-computer interaction system includes: eye tracking positioning system and voice interaction system; the second platform of somatosensory interaction of intelligent integrated human-computer interaction system is divided into the first step and the second step, the first step is for placing posture detection sensors, and the second step includes: pen ejection system, automatic limit device, direct tactile operating system, and operation keyboard; the interactive capture layer is embedded in the convenient operation layer, including: central processing unit and the pen test interaction module connected to it, grasping interaction module, posture interaction module, eye movement interaction module, electromyography interaction module, voice interaction module, wireless communication module, data storage module, power supply module, digital-to-analog conversion module and external device interface circuit; the user's interaction data is detected by cooperating with the convenient operation layer and the interactive capture layer; The convenient operation layer of the intelligent integrated human-computer interaction system includes: an audio-visual interaction first platform of the intelligent integrated human-computer interaction system, and a somatosensory interaction second platform of the intelligent integrated human-computer interaction system; The first audio-visual interaction platform of the intelligent integrated human-computer interaction system includes: an eye tracking positioning system and a voice interaction system; The eye tracking positioning system is used to capture the direction in which the user's eyes are looking, and the positioning system is tracked by the follow-up servo, and the positioning result is displayed on the eye movement capture screen; The eye tracking positioning system is located on the first audio-visual interaction platform of the intelligent integrated human-computer interaction system; The eye tracking positioning system includes a tracking servo, an eyeball capture device, and an eye movement capture screen; The follow-up servo is fixed on the first audio-visual interaction platform of the intelligent integrated human-computer interaction system; The follow-up servo is connected to the eyeball capture device and is used to drive the eyeball capture device to rotate; The eye movement capture screen is fixed on the first audio-visual interaction platform of the intelligent integrated human-computer interaction system, and is used to feed back to the user the specific position captured by the eyeball; The voice interaction system is located in the first audio-visual interaction platform of the intelligent integrated human-computer interaction system, and is used to detect the sound made by the user and conduct voice interaction with the user; The second platform of somatosensory interaction of the intelligent integrated human-computer interaction system includes: a pen ejection system, a pen test interaction collection pen, an automatic limit device, a direct tactile operating system, an operation keyboard, an interaction capture layer, electrodes such as posture sensors, electromyography sensors, grip sensors, pen touch sensors, and signal collection and detection sensors; The second platform for somatosensory interaction of the intelligent integrated human-computer interaction system is located in the intelligent integrated human-computer interaction system and is used for detecting somatosensory interaction of the human body; The second platform of somatosensory interaction of the intelligent integrated human-computer interaction system is divided into a first level and a second level; The first step is located on the surface of the second platform of the somatosensory interaction of the intelligent integrated human-computer interaction system, and is used to place electrodes such as posture sensors, electromyography sensors, grip sensors, and pen touch sensors, pens, and signal acquisition and detection sensors, etc., to detect the user's posture, electromyography, grip, pen style and other interaction data; The second step is used to store various mechanical structures; The pen ejection system is equipped with a sensing touch panel. Once the pen test interaction collection pen is taken out, the pen ejection system will eject the sensing touch panel. The user writes on the sensing touch panel, thereby collecting the user's pen touch posture, including the user's pen touch trajectory and the user's pen holding posture. The pen-touch ejection system comprises a pen-touch ejection drive motor, a drive motor linkage device, a pen-touch linkage gear, an ejection drive left rack, an ejection follower left slide rail, an induction touch panel, an ejection follower right rack, an ejection follower right slide rail, an induction touch panel front protection device, and an induction touch panel rear protection device; The pen-stroke ejection driving motor is fixed on the second step of the second platform of the integrated human-computer interaction system to drive the entire pen-stroke ejection system; The pen-stroke ejection drive motor, the drive motor linkage device and the pen-stroke linkage gear are sequentially connected for constant speed transmission; The ejection drive left rack is meshed with the stroke linkage gear, and the ejection drive left rack is pushed out through the gear to realize the ejection of the mechanism; One end of the ejection follower left slide rail is fixed on the second platform of the first step of the lower half of the intelligent integrated human-computer interaction system, and is used to be fixed to the second platform and connected to the ejection drive left rack. The inductive touch panel is respectively fixed to the ejection-driven left rack and the ejection-driven right rack, and the ejection-driven left rack brings the inductive touch panel out; The ejection follower right slide rail is connected to the second platform and the ejection drive right rack, and is used to support the induction touch panel, making it more stable when ejecting; The inductive touch panel front protection device and the inductive touch panel rear protection device are respectively fixed at the front and rear of the inductive touch panel to play a protective role. The automatic limit device is used to connect the first platform of audio-visual interaction of the intelligent integrated human-computer interaction system and the second platform of somatosensory interaction of the intelligent integrated human-computer interaction system. A switch button is installed on it. By pressing the switch button, the intelligent integrated human-computer interaction system automatically opens and stops at a fixed position, limiting the opening and closing angle of the intelligent integrated human-computer interaction system. The automatic limit device is located inside the second platform of somatosensory interaction of the intelligent integrated human-computer interaction system; The automatic limit device includes a two-stage speed regulating gear support, a support limit screw, a limit transmission motor, a first transmission coupling of the limit motor, a rear end propulsion first transmission gear, a limit rear end propulsion second transmission large gear, a limit rear end propulsion second transmission small gear, a rear end propulsion third transmission gear, an automatic limit mechanical arm first stage front end, and an automatic limit mechanical arm second stage propulsion rear end; The secondary speed regulating gear support bracket is fixed on the second somatosensory interaction platform of the intelligent integrated human-computer interaction system through the support bracket limiting screw; The limit transmission motor is fixed on the second somatosensory interaction platform of the intelligent integrated human-computer interaction system; The limit transmission motor and the first transmission coupling of the limit motor are connected in sequence with the rear end propulsion first transmission gear to form a power source for the automatic limit device; The rear-end propulsion first transmission gear meshes with the limited rear-end propulsion second transmission large gear in the transmission secondary gear set to form a first-level speed regulation transmission device. This realizes the first-level regulation of the motor speed and initially reduces the motor speed; The limited rear-end propulsion second transmission pinion in the transmission secondary gear set meshes with the rear-end propulsion third transmission gear to form a secondary speed regulation transmission device. This realizes secondary regulation of the motor speed and further reduces the motor speed. The rear end propulsion third transmission gear cooperates with the secondary propulsion rear end of the automatic limit mechanical arm, so that the secondary propulsion rear end of the automatic limit mechanical arm moves forward; The first-stage front end of the automatic limit mechanical arm is connected to the second-stage propulsion rear end of the automatic limit mechanical arm through a rotating pair; The first-level front end of the automatic limit robot arm is connected to the first audio-visual interaction platform of the intelligent integrated human-computer interaction system through a rotating pair, and the switch function is realized by extending the first-level front end of the automatic limit robot arm forward. The direct tactile operating system is used to interact with the user through grasping, and realizes a convenient interactive experience and detects the user's grasping data through the operation of the direct tactile operating system; The direct tactile operating system is located on the first step of the second platform of the somatosensory interaction of the intelligent integrated human-computer interaction system; The direct tactile operating system includes a direct tactile operating system tactile perception platform, a tactile perception secondary transmission mechanical arm, a direct tactile operating system tactile perception primary transmission mechanical arm, a direct tactile operating system tactile perception platform firmware, and a direct tactile operating system drive motor; The firmware of the tactile perception platform of the direct tactile operating system is fixed on the first step of the second platform of the somatosensory interaction of the intelligent integrated human-computer interaction system; The direct tactile operating system tactile perception platform rotates around the direct tactile operating system tactile perception platform firmware and is used to store the grasping interaction device; The direct tactile operating system tactile perception secondary transmission mechanical arm, the direct tactile operating system tactile perception primary transmission mechanical arm and the direct tactile operating system tactile perception platform are sequentially connected, and the direct tactile operating system tactile perception primary transmission mechanical arm is rotated to raise the direct tactile operating system tactile perception platform; The direct tactile operating system driving motor is connected to the direct tactile operating system tactile perception primary transmission mechanical arm and is used to drive the entire operating system. The operation keyboard is located on the first step of the second platform of the somatosensory interaction of the intelligent integrated human-computer interaction system; The interaction capture layer of the intelligent integrated human-computer interaction system includes: a central processing unit and a voice interaction module connected thereto, a written test interaction module, a posture interaction module, a digital-to-analog conversion module, a data communication module, a data storage module, a power supply module, an external device socket circuit, and an electromyographic interaction module, an eye movement interaction module, and a grasping interaction module connected thereto through the digital-to-analog conversion module; The central processor receives signals such as electrooculogram signals, electromyography signals, gesture acquisition, image processing, and speech recognition, and the data storage module temporarily stores the detection data, which is then uploaded to the client through the data communication module and displayed on the acquisition client; The written test interaction module includes a sensing touch panel and a written test interaction collection pen; wherein the written test interaction collection pen includes a pen tip pressure collection sensor and a posture collection sensor. The written test interaction module mainly records the angle, direction and speed information during the writing process, collects the force during the writing process by collecting the pen tip pressure, and can collect the entire process characteristics of the entire written test taker's writing through two sensors. The eye movement interaction module includes an electrode sheet, a signal shield, a preamplifier circuit, a bandpass filter circuit, a right leg drive circuit, and a final gain circuit. The eye movement signal is indirectly obtained by setting electrodes on the skin surface around the eyeball, and the electrooculogram signal is shielded by a signal shield. The shielding end is connected to the same signal as the input signal, and the equipotential shielding method is used to ensure that the electrooculogram signal reduces interference before entering the amplifier circuit, and the signal is amplified and filtered to meet the electrooculogram signal acquisition requirements; The grasping interaction module includes a flexible sensor interface circuit and a flexible sensor filter circuit; each pressure signal of the inner palm and each joint of the finger is recorded when the palm exerts force to grasp the object, and the digital-to-analog conversion is performed through the digital-to-analog conversion module to read the pressure. The flexible sensor interface circuit is mainly used to connect to the flexible sensor. The circuit uses 16 flexible sensors to collect the pressure signals of the inner palm and each joint respectively, and the digital-to-analog conversion is performed through this part of the digital-to-analog conversion circuit to read the pressure signal; the flexible sensor filter circuit is used to filter the pressure signal; The electromyographic interaction module mainly performs electromyographic signal pickup and differential amplification circuit; the DC signal generated under conditions such as human sweating and constant electromagnetic field interference is superimposed on the surface electromyographic signal, and after high-power amplification in the subsequent main amplifier circuit, it is easy to cause the amplifier to be saturated and unable to detect effective signals; high-pass filters are used to filter out DC signals and low-frequency noise. High-pass filters are filters that allow high-frequency signals to pass, but weaken signals with frequencies lower than the cutoff frequency. The posture interaction module includes a posture acquisition circuit and a posture acquisition sensor; the user's body posture is collected, and the sensor quickly responds to the user's posture status and gives accurate data judgment; The voice interaction module includes an audio acquisition circuit and an audio output circuit, which are used for human-computer voice interaction recognition, collects voice commands and voice information issued by humans, and sends them to the central processor for processing after recognition and conversion; The digital-to-analog conversion module converts the pressure signals, electrooculogram signals and electromyography signals collected and processed by the grasping interaction module, eye movement interaction module and electromyography interaction module into data signals readable by the central processor; The data communication module realizes wireless transmission of signals between the central processor and the client; the data storage module is used for flash memory storage of data; the power supply module includes a DC adapter, a lithium battery charging circuit, and a voltage-stabilized power supply circuit.

2. The neural system flexible sensor intelligent integrated human-computer interaction system according to claim 1 is characterized in that: The power supply module supplies power to the central processing unit, eye movement interaction module, voice interaction module, written test interaction module, grasping interaction module, posture interaction module and electromyography interaction module through 1.6V, 1.8V, 3V, 3.3V, 4V, -5V and 5V voltage regulation.

Images