Intelligent interaction device based on jaw multi-mode touch control

Abstract

The invention discloses an intelligent interaction device based on jaw multi-mode touch control, which comprises a helmet body, a flexible arc-shaped touch control belt, a touch control and voice collaborative confirmation module, a sliding path detection module and an eye recognition system, and is characterized in that the helmet body is composed of a shell and a lining, the bottom of the shell is provided with the flexible arc-shaped touch control belt, and the flexible arc-shaped touch control belt fits a mandible curve of a user; the touch control and voice collaborative confirmation module is fixed at the rear part of the right side of the helmet body and is electrically connected with the flexible arc-shaped touch control belt through a wire; and the sliding path detection module is integrated in the collaborative confirmation module. Therefore, according to the intelligent interaction device, by means of double confirmation of'touch control and voice 'or'touch control and eye recognition', the misoperation rate is reduced to be lower than 5%, flexible switching of voice, touch control and eye recognition modes is supported, jaw touch control is combined with sliding path recognition, the efficiency is improved remarkably, the ISO80369-1 standard is met, and cross infection is reduced. Through the Bluetooth and the USB-C interface, the system can be widely applied to multiple fields.

Description

Technical Field

[0001] This application relates to the technical field of smart wearable devices, and in particular to an intelligent interaction device based on multi-modal touch control of the jaw. Background Art

[0002] Existing intelligent interaction devices have the following significant defects in scenarios where users' hands are liberated:

[0003] 1. Single interaction mode: Traditional touch devices only support click operations and cannot achieve complex interaction functions such as swipe navigation and long-press wake-up. For example, existing touch bands usually limit the number of touch points (such as three-point touch), and it is easy for others to crack the operation logic by increasing or decreasing the touch points, resulting in insufficient security.

[0004] 2. Poor environmental adaptability: The misrecognition rate of voice confirmation in noisy environments is over 40%, and it requires users to speak frequently, making it difficult to be applied in high-noise scenarios such as industrial workshops or quiet environments such as operating rooms.

[0005] 3. Operational safety hazards: In the cycling scenario, users need to operate the mobile phone with one hand, resulting in frequent traffic accidents (statistics in some cities show that 30% of rider accidents are related to mobile phone operations); in the medical scenario, device contact may cause cross-infection (data from The Lancet shows that 75% of operating room infections originate from device contact).

[0006] 4. Insufficient functional expandability: Existing devices lack a multi-modal collaborative verification mechanism and cannot flexibly switch operation modes according to scenario requirements (such as voice, touch, or eye recognition), resulting in limited application scope.

[0007] To address the above problems, a multi-modal interaction device integrating jaw touch control, voice collaborative confirmation, sliding path detection, and eye recognition is proposed. Through a flexible touch band, a dynamic path database, and biometric verification technology, contactless operation with high security and high adaptability is achieved. Summary of the Invention

[0008] This application aims to solve at least one of the technical problems in the related technologies to some extent.

[0009] To this end, the first objective of this application is to provide an intelligent interaction device based on multi-modal touch control of the jaw, which reduces the misoperation rate to less than 5% through a dual confirmation mechanism of "touch + voice" or "touch + eye recognition", significantly improving security.

[0010] The second objective of this application is to provide an intelligent interaction device based on multi-modal touch control of the jaw, which supports flexible switching among voice mode (suitable for quiet scenarios), touch mode (suitable for noisy environments), and eye recognition mode (suitable for sterile environments).

[0011] The third object of the present application is to provide an intelligent interaction device based on multi-modal jaw touch. By combining jaw touch with sliding path recognition, the average single operation saves 10 seconds, and the daily order volume increases by 5%-10%.

[0012] The fourth object of the present application is to provide an intelligent interaction device based on multi-modal jaw touch, which complies with the ISO80369-1 standard, reduces the risk of cross-infection in medical scenarios, and the eye recognition system breaks through the limitations of traditional devices through non-contact verification.

[0013] The fifth object of the present application is to provide an intelligent interaction device based on multi-modal jaw touch, which realizes seamless connection with a variety of external devices through Bluetooth and USB-C interfaces, and supports multi-field applications such as industrial control, medical care, and drone operation.

[0014] To achieve the above object, an embodiment of the first aspect of the present application provides an intelligent interaction device based on multi-modal jaw touch, including a helmet body, a flexible arc touch band, a touch and voice collaborative confirmation module, a sliding path detection module, and an eye recognition system. Among them, the helmet body is composed of a shell and a lining. A flexible arc touch band is provided at the bottom of the shell, which fits the curve of the user's mandible; the touch and voice collaborative confirmation module is fixed to the rear part on the right side of the helmet body and is electrically connected to the flexible arc touch band through a wire; the sliding path detection module is integrated inside the collaborative confirmation module, and obtains path data through the sliding of the flexible arc touch band; the eye recognition system is rotatably installed on the helmet body through a rotating bracket, and can be maximally placed at a forty-five-degree angle below the cap visor, and includes a micro camera and a biometric processor, and communicates with the touch and voice collaborative confirmation module through a wireless signal.

[0015] An intelligent interaction device based on multi-modal jaw touch according to an embodiment of the present application. By means of double confirmation of "touch + voice" or "touch + eye recognition", the misoperation rate is reduced to less than 5%. It supports flexible switching between voice, touch, and eye recognition modes. The combination of jaw touch and sliding path recognition has a significant effect on improving efficiency, complies with the ISO80369-1 standard, and reduces cross-infection. Through Bluetooth and USB-C interfaces, it is widely used in multiple fields.

[0016] In addition, an intelligent interaction device based on multi-modal jaw touch proposed above according to the present application may also have the following additional technical features:

[0017] In an embodiment of the present application, the width of the flexible arc touch band is 3-5 cm, and it is internally provided with a pressure sensor array and a capacitive sliding trajectory recognition unit. The surface is divided into three touch areas on the left, middle, and right, corresponding to functions such as navigation, page switching, and order receiving respectively.

[0018] In one embodiment of the present application, the touch and voice collaborative confirmation module includes a main controller, a voice interaction unit, a miniature microphone, a bone conduction speaker, and a secondary confirmation logic unit. Among them, the main controller receives click signals from the pressure sensor array; the voice interaction unit collects user voices through the miniature microphone and plays prompt sounds through the bone conduction speaker; the secondary confirmation logic unit presets instruction execution conditions (such as "voice + blink" dual verification).

[0019] In one embodiment of the present application, the sliding path detection module includes a dynamic path database that stores various sliding trajectories (such as "L" shape, spiral shape) and is compared in real time through the coordinate tracking algorithm of the capacitive sliding trajectory recognition unit.

[0020] In one embodiment of the present application, the viewing angle range of the miniature camera of the eye recognition system is 40°×30°. The biometric processor is built-in with a blink frequency detection algorithm and a pupil gaze direction recognition algorithm, and is linked with the secondary confirmation logic unit of the touch and voice collaborative confirmation module.

[0021] In one embodiment of the present application, it further includes an external device interface including a USB-C interface and a low-power Bluetooth module, which is integrated at the bottom of the touch and voice collaborative confirmation module and is used for communication with mobile phones, medical devices, or industrial terminals.

[0022] In one embodiment of the present application, the touch and voice collaborative confirmation module is connected to the helmet body through a magnetic quick-release structure. The magnetic quick-release structure includes a magnetic base embedded in the outer shell of the helmet body and a buckle provided on the back of the touch and voice collaborative confirmation module, supporting quick disassembly and assembly.

[0023] Touch and voice collaborative confirmation module: Main controller model: STM32H743ZI (high-performance 32-bit microcontroller, supporting multitasking);

[0024] Function: Receive pressure sensor signals and coordinate the work of each module.

[0025] Voice interaction unit: Chip: Intel Movidius VPUMYRIADX (edge computing voice recognition);

[0026] Function: Process voice commands in real time and support offline voice wake-up.

[0027] Miniature microphone model: Knowles SPH0641LM4H-B (high-sensitivity MEMS microphone, -26dBFS sensitivity);

[0028] Function: Collect user voice signals.

[0029] Bone conduction speaker model: Shokz OpenRun Pro (bone conduction technology, IP55 waterproof);

[0030] Function: Play prompt tones and voice feedback.

[0031] Secondary confirmation logic unit chip: Xilinx Artix-7 FPGA (XC7A35T);

[0032] Function: Implement the "voice + blink" dual verification logic.

[0033] Sliding path detection module: Dynamic path database storage chip: Microchip 25LC256 (256KB SPI Nor Flash);

[0034] Function: Store preset sliding trajectories (such as "L" shape, spiral shape).

[0035] Capacitive sliding trajectory recognition unit: Cypress CapSense (CY8C4245AXI-483);

[0036] Function: Real-time track the touch trajectory through capacitance changes.

[0037] Eye recognition system: Miniature camera model: OmniVision OV7740 (1 / 4 inch, 720P resolution, 40°×30° viewing angle);

[0038] Function: Collect users' eye images.

[0039] Biometric processor chip: Qualcomm QCS605 (integrated AI accelerator);

[0040] Function: Run the blink frequency detection algorithm and the pupil gaze direction recognition algorithm.

[0041] External device interface: USB-C interface chip: Texas Instruments TUSB320 (support USB3.1 Gen1);

[0042] Function: High-speed data transfer with external devices.

[0043] Low-power Bluetooth module model: Nordic nRF52832 (BLE5.0, support 2Mbps rate);

[0044] Function: Wireless communication with mobile phones, medical devices, etc.

[0045] Pressure sensor array model: Tekscan FlexiForce A201 (flexible pressure sensor, 0.5 - 50N range);

[0046] Function: Detect jaw click actions.

[0047] Power management: Maxim MAX17260 (low-power power management IC, supporting lithium battery charging);

[0048] Communication protocol: The modules use a hybrid communication of UART (Universal Asynchronous Receiver-Transmitter) and I2C (Inter-Integrated Circuit).

[0049] The advantages of this application compared with the existing technologies are as follows:

[0050] (1) Through the dual confirmation mechanism of "touch + voice" or "touch + eye recognition", the misoperation rate is reduced to less than 5%, significantly improving safety.

[0051] (2) Support flexible switching among voice mode (suitable for quiet scenarios), touch mode (suitable for noisy environments), and eye recognition mode (suitable for sterile environments).

[0052] (3) The combination of jaw touch and sliding path recognition saves an average of 10 seconds per single operation, increasing the daily order volume by 5% - 10%.

[0053] (4) Complies with the ISO80369-1 standard, reducing the risk of cross-infection in medical scenarios. The eye recognition system breaks through the limitations of traditional devices through non-contact verification.

[0054] (5) Achieves seamless connection with a variety of external devices through Bluetooth and USB-C interfaces, supporting multi-field applications such as industrial control, medical care, and drone operation.

[0055] The additional aspects and advantages of this application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and / or additional aspects and advantages of this application will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0057] Figure 1 FIG. is a perspective view of an intelligent interaction device based on jaw multi-modal touch according to an embodiment of this application;

[0058] Figure 2 FIG. is a perspective view of an intelligent interaction device based on jaw multi-modal touch according to another embodiment of this application;

[0059] Figure 3 FIG. is a rear cross-sectional view of an intelligent interaction device based on jaw multi-modal touch according to an embodiment of this application;

[0060] Figure 4 Schematic cross-sectional structure diagram of an intelligent interaction device based on multi-modal jaw touch according to an embodiment of the present application;

[0061] Figure 5 Magnetic fast detachment structure diagram of an intelligent interaction device based on multi-modal jaw touch according to an embodiment of the present application;

[0062] Figure 6 Schematic control connection diagram of an intelligent interaction device based on multi-modal jaw touch according to an embodiment of the present application.

[0063] As shown in the figure: 1. Helmet body; 2. Flexible arc touch belt; 3. Touch and voice collaborative confirmation module; 4. Sliding path detection module; 5. Eye recognition system; 6. External device interface; 7. Magnetic fast detachment structure; 11. Shell; 12. Inner lining; 50. Rotating bracket; 51. Miniature camera; 52. Biometric processor; 21. Pressure sensor array; 22. Capacitive sliding trajectory recognition unit; 31. Main controller; 32. Voice interaction unit; 33. Miniature microphone; 34. Bone conduction speaker; 35. Secondary confirmation logic unit; 41. Dynamic path database; 61. USB-C interface; 62. Low-power Bluetooth module; 71. Magnetic base; 72. Buckle. Detailed implementation manners

[0064] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as limiting the present application. On the contrary, the embodiments of the present application include all changes, modifications and equivalents falling within the spirit and scope of the appended claims.

[0065] An intelligent interaction device based on multi-modal jaw touch according to an embodiment of the present application will be described below with reference to the accompanying drawings.

[0066] As Figures 1-6 shown, an intelligent interaction device based on multi-modal jaw touch according to an embodiment of the present application may include a helmet body 1, a flexible arc touch belt 2, a touch and voice collaborative confirmation module 3, a sliding path detection module 4, and an eye recognition system 5.

[0067] It can be understood that in this intelligent interaction device, the helmet body 1 serves as a basic bearing structure and is composed of a shell 11 and an inner lining 12. The flexible arc touch belt 2 provided at the bottom of the shell 11 can well fit the curve of the user's mandible and provide a convenient touch interaction area for the user.

[0068] When the user performs a jaw touch operation, the flexible arc touch belt 2 will collect relevant signals. For example, when the user slides or clicks on the touch belt 2, the signals generated by these actions are transmitted through a wire to the touch and voice collaborative confirmation module 3 fixed to the rear right side of the helmet body 1.

[0069] The touch and voice collaborative confirmation module 3 is one of the core processing units of the entire device. The sliding path detection module 4 is integrated inside the collaborative confirmation module 3, and it obtains sliding path data from the flexible arc touch belt 2. For example, if the user makes a specific sliding trajectory on the touch belt 2, the sliding path detection module 4 starts to work and determines whether the trajectory matches the preset path.

[0070] When performing some operation instructions, a confirmation link is often required. At this time, the touch and voice collaborative confirmation module 3 plays a role. It will, according to preset rules, such as after detecting a touch operation, perform secondary confirmation through voice interaction and other means, and only execute the corresponding instruction after confirmation is correct.

[0071] The eye recognition system 5 provides more confirmation methods and function expansions for the interaction. It is rotatably installed on the helmet body 1 through a rotating bracket 50 and can be placed up to forty-five degrees below the visor at most. During use, the micro camera 51 is responsible for collecting the user's eye image information, and this information is transmitted to the biometric processor 52. The biometric processor 52 analyzes the collected information, such as detecting biometric data such as the user's blinking action and gazing direction. After that, the eye recognition system 5 communicates with the touch and voice collaborative confirmation module 3 through a wireless signal and transmits the analyzed biometric data over, providing more basis for the secondary confirmation of the instruction and further improving the accuracy and security of the interaction.

[0072] In an embodiment of the present application, as Figures 1-6 shown, the flexible arc touch belt 2 has a width of 3 - 5 cm, and is internally provided with a pressure sensor array 21 and a capacitive sliding trajectory recognition unit 22. The surface is divided into three touch areas on the left, middle, and right, corresponding to functions such as navigation, page switching, and order receiving respectively.

[0073] It can be understood that the flexible arc touch belt 2 is internally provided with a pressure sensor array 21 and a capacitive sliding trajectory recognition unit 22. When the user touches the touch belt 2 with the jaw for operation, the pressure sensor array 21 starts to work. For example, when the user clicks on the left touch area on the surface of the touch belt 2, the pressure sensor array 21 can quickly detect the pressure change and convert it into an electrical signal. This signal will be transmitted to the associated touch and voice collaborative confirmation module 3, informing the system that the user has performed a click operation and it is in the left touch area. Since the left touch area corresponds to the navigation function, after receiving the signal, the touch and voice collaborative confirmation module 3 may trigger the voice interaction unit to emit a prompt sound through the bone conduction speaker 34, such as "Do you want to turn on the navigation? Please confirm by voice."

[0074] The capacitive sliding trajectory recognition unit 22 is mainly responsible for recognizing the sliding operation of the user on the touch belt 2. When the user slides in the middle touch area, the capacitive sliding trajectory recognition unit 22 uses the principle of capacitance change to real-time track the movement trajectory of the touch point and convert these trajectory data into digital signals. These signals will also be transmitted to the sliding path detection module 4, which is integrated inside the touch and voice collaborative confirmation module 3. The sliding path detection module 4 will compare the received sliding trajectory data with the preset path. If the match is successful, such as conforming to the preset sliding trajectory for switching pages, the touch and voice collaborative confirmation module 3 will execute the instruction to switch pages and achieve the corresponding function.

[0075] The right touch area corresponds to functions such as receiving orders. When the user clicks on the right touch area, the pressure sensor array 21 detects the click action and transmits the signal to the touch and voice collaborative confirmation module 3. The system will guide the user to confirm through voice interaction and other means according to the preset process, and execute the order receiving operation after the confirmation is completed. Through such a design, the user only needs to perform simple click or sliding operations on the flexible arc touch belt 2 with the jaw to achieve multiple functions such as navigation, page switching, and order receiving, providing a convenient and efficient interaction experience for the user in scenarios such as cycling, industrial operations, and medical care.

[0076] In an embodiment of the present application, as Figures 1-6 shown, the touch and voice collaborative confirmation module 3 includes a main controller 31, a voice interaction unit 32, a miniature microphone 33, a bone conduction speaker 34, and a secondary confirmation logic unit 35. Among them, the main controller 31 receives the click signal from the pressure sensor array 21, the voice interaction unit 32 collects the user's voice through the miniature microphone 33 and plays the prompt sound through the bone conduction speaker 34, and the secondary confirmation logic unit 35 presets the instruction execution conditions (such as "voice + blink" double verification).

[0077] It is understandable that when the user operates the flexible arc touch band 2, the pressure sensor array 21 will detect the click action and generate corresponding click signals. These signals are transmitted to the main controller 31 in the touch and voice collaborative confirmation module 3. For example, when the user clicks on the left touch area (corresponding to the navigation function) of the flexible arc touch band 2, the main controller 31 receives the click signal from the pressure sensor array 21, and it will perform preliminary processing and analysis on this signal to identify that this is an operation request related to the navigation function.

[0078] After the main controller 31 processes the click signal, it will trigger the voice interaction unit 32 to start working. The voice interaction unit 32 collects the user's voice information through the miniature microphone 33. For example, in the above navigation operation scenario, the voice interaction unit 32 will collect the user's voice reply to "whether to start navigation". The miniature microphone 33 converts the collected voice signal into an electrical signal and transmits it to the voice interaction unit 32 for processing.

[0079] At the same time, the voice interaction unit 32 is also responsible for playing a prompt tone through the bone conduction speaker 34. When the user's operation triggers the corresponding function, the bone conduction speaker 34 will send a voice prompt to the user. For example, after clicking on the left touch area, the bone conduction speaker 34 will play a prompt tone of "whether to start navigation, please confirm by voice" to guide the user to perform the next operation.

[0080] The secondary confirmation logic unit 35 is responsible for ensuring the accuracy and security of the instruction execution during the entire interaction process. It presets the instruction execution conditions, taking "voice + blink" double verification as an example. When the main controller 31 receives the click signal and the voice interaction unit 32 collects the user's voice reply, the secondary confirmation logic unit 35 will first verify whether the voice content meets the preset requirements. Suppose the user replies "yes" and the voice verification passes, but the entire confirmation process has not been completed yet.

[0081] The miniature camera 51 of the eye recognition system 5 will collect the user's eye image information in real time, transmit it to the biometric processor 52 for analysis, and detect whether the user makes a blinking action. The biometric processor 52 transmits the analysis result to the secondary confirmation logic unit 35 of the touch and voice collaborative confirmation module 3 through a wireless signal. Only when both the voice collected by the voice interaction unit 32 and the blinking action detected by the eye recognition system 5 meet the preset conditions, the secondary confirmation logic unit 35 will determine that the confirmation is successful and send an instruction execution signal to the main controller 31.

[0082] After receiving the execution signal from the secondary confirmation logic unit 35, the main controller 31 will send a control instruction to an external device (such as a mobile phone, a medical device, or an industrial terminal) through the external device interface 6 according to the function corresponding to the initially received click signal, and complete corresponding operations, such as turning on the navigation function. Through such a series of rigorous processes, the touch and voice collaborative confirmation module 3 realizes the collaborative work of multimodal interaction, effectively improving the accuracy and reliability of the interaction.

[0083] In an embodiment of the present application, as Figures 1-6 shown, the sliding path detection module 4 includes a dynamic path database 41, and the dynamic path database 41 stores various sliding trajectories such as "L" shape and spiral shape, which are compared in real time through the coordinate tracking algorithm of the capacitive sliding trajectory recognition unit 22.

[0084] It can be understood that when the user performs a sliding operation on the flexible arc touch band 2, the capacitive sliding trajectory recognition unit 22 will immediately sense this action. As the direct interaction area for the user's operation, the flexible arc touch band 2 has a built-in capacitive sliding trajectory recognition unit 22 that can sensitively capture the capacitance changes caused by the user's sliding.

[0085] The capacitive sliding trajectory recognition unit 22 uses the coordinate tracking algorithm to track the sliding path of the user on the flexible arc touch band 2 in real time. During the user's sliding process, this unit continuously collects the coordinate information of the sliding points, and combines these coordinate information into a complete sliding trajectory data in chronological order. For example, when the user slides out an "L" - shaped trajectory on the touch band 2, the capacitive sliding trajectory recognition unit 22 will record the coordinates of the sliding points at each moment, forming a coordinate sequence representing the "L" - shaped trajectory.

[0086] The capacitive sliding trajectory recognition unit 22 transmits the collected sliding trajectory data to the sliding path detection module 4. In the sliding path detection module 4, the dynamic path database 41 stores various preset sliding trajectories, such as "L" shape, spiral shape, etc. These preset trajectories are pre - entered into the dynamic path database 41 during the device development or configuration stage.

[0087] After receiving the real - time sliding trajectory data transmitted by the capacitive sliding trajectory recognition unit 22, the sliding path detection module 4 will compare it with the preset sliding trajectories stored in the dynamic path database 41. During the comparison process, the system analyzes the similarity of the real - time trajectory and the preset trajectory in terms of features such as shape, direction, and length. Taking the "L" - shaped trajectory as an example, the system will check whether the real - time trajectory has the characteristics of an "L" shape, such as whether there is an obvious turning point and whether the turning angle meets the preset range.

[0088] If the real-time sliding trajectory successfully matches a certain preset trajectory in the dynamic path database 41, the sliding path detection module 4 will output the corresponding matching result information. This information will be sent to the main controller 31 in the touch and voice collaborative confirmation module 3. The main controller 31 determines the operation that the user wants to perform according to the matching result, and further triggers the corresponding functions, such as navigation, page switching, etc., in accordance with the preset program flow.

[0089] If the real-time sliding trajectory does not match all the preset trajectories in the dynamic path database 41, the sliding path detection module 4 will determine it as an invalid operation, and may emit a prompt sound to the user through the voice interaction unit 32 and the bone conduction speaker 34, informing the user that the operation is not recognized and guiding the user to perform the operation again.

[0090] Through the above process, the sliding path detection module 4 utilizes the dynamic path database 41 and the capacitive sliding trajectory recognition unit 22 to achieve accurate recognition and response to the user's sliding operation, providing strong support for the multi-modal touch function of the intelligent interaction device.

[0091] In an embodiment of the present application, as Figures 1-6 shown, the viewing angle range of the micro camera 51 of the eye recognition system 5 is 40°×30°, and the biometric processor 52 is built-in with a blinking frequency detection algorithm and a pupil gaze direction recognition algorithm, which is linked with the secondary confirmation logic unit of the touch and voice collaborative confirmation module 3.

[0092] It can be understood that the eye recognition system 5 is installed on the helmet body 1 through the rotating bracket 50 and can be adjusted up to 45 degrees below the visor at most to ensure that the micro camera 51 can clearly capture the user's eye information. When the intelligent interaction device is started, the eye recognition system 5 also enters the working state, and the micro camera 51 starts to collect image data.

[0093] The viewing angle range of the micro camera 51 is 40°×30°. Within this range, it continuously collects the image information of the user's eyes. During the user's interaction operation, for example, after a touch operation on the flexible arc touch strip 2 triggers an instruction that requires secondary confirmation, the micro camera 51 will capture the subtle movements and state changes of the user's eyes in real time.

[0094] The collected eye image data is transmitted to the biometric processor 52. The biometric processor 52 is built-in with a blink frequency detection algorithm and a pupil gaze direction recognition algorithm. The blink frequency detection algorithm analyzes the blink actions of the user's eyes in the image and calculates the number of blinks per unit time. The pupil gaze direction recognition algorithm identifies the orientation of the user's pupils and determines the user's gaze direction. For example, when the user performs operations such as adjusting the hospital bed angle (left forehead touch + blink) or calling the nurse station (right forehead swipe + gaze), the biometric processor 52 can accurately analyze the corresponding biometric data.

[0095] The biometric processor 52 transmits the biometric data such as the analyzed blink frequency and pupil gaze direction to the secondary confirmation logic unit 35 of the touch and voice collaborative confirmation module 3 via a wireless signal. The secondary confirmation logic unit 35 combines this biometric data with other confirmation methods (such as voice confirmation) to determine whether the instruction execution condition is met. For example, when performing "voice + blink" dual verification, if the user's blink action is detected by the biometric processor 52 while the voice interaction unit 32 confirms the voice collected by the miniature microphone 33, and the blink frequency is within the preset range, the secondary confirmation logic unit 35 will determine that the confirmation is successful.

[0096] When the secondary confirmation logic unit 35 determines that the confirmation is successful based on the biometric data and other confirmation information provided by the eye recognition system 5, it sends a signal allowing the execution of the instruction to the main controller 31. After receiving this signal, the main controller 31 sends a control instruction to an external device (such as a medical device, an industrial terminal, etc.) via the external device interface 6 according to the instruction corresponding to the initial touch operation to complete the corresponding operation. If the biometric data does not match the preset conditions, the secondary confirmation logic unit 35 will determine that the confirmation fails, prevent the instruction from being executed, and may prompt the user to re-confirm the operation through the voice interaction unit 32 and the bone conduction speaker 34.

[0097] In an embodiment of the present application, as Figures 1-6 shown, the external device interface 6 further includes a USB-C interface 61 and a low-power Bluetooth module 62, which are integrated at the bottom of the touch and voice collaborative confirmation module 3 and are used for communicating with a mobile phone, a medical device, or an industrial terminal.

[0098] It can be understood that if an external device supporting the USB-C interface, such as some mobile phones, industrial terminals, etc., is to be connected, the user only needs to insert one end of the corresponding USB-C data cable into the USB-C interface 61 of the intelligent interaction device and the other end into the USB-C interface of the external device. After insertion, the USB-C interface 61 will automatically detect the connection status and perform a handshake of the communication protocol with the external device to complete the establishment of the physical connection and the data transmission channel.

[0099] For external devices supporting Bluetooth communication, such as Bluetooth headsets and medical monitoring devices, the low-power Bluetooth module 62 will automatically turn on the broadcast and search for available Bluetooth devices in the vicinity. The user needs to turn on the Bluetooth function on the external device, search the list of available devices, and pair with the Bluetooth name corresponding to the intelligent interaction device after finding it. After successful pairing, the low-power Bluetooth module 62 will establish a stable Bluetooth connection with the external device to prepare for subsequent data transmission.

[0100] Transmission through the USB-C interface: If the connected external device is connected through the USB-C interface 61, the main controller 31 will send the data in a format compliant with the USB communication protocol to the USB-C interface 61. The USB-C interface 61 will transmit the data to the external device through the data cable to issue commands. For example, when the user operates on the intelligent interaction device to send a navigation command, the main controller 31 sends the navigation-related data to the connected mobile phone through the USB-C interface 61, and the mobile phone starts the navigation application after receiving the data.

[0101] Transmission through the low-power Bluetooth module: If the connected external device is connected via Bluetooth, the main controller 31 will convert the data into a format supported by the Bluetooth communication protocol and send it to the low-power Bluetooth module 62. The low-power Bluetooth module 62 transmits the data to the paired external device through a wireless signal. For example, in a medical scenario, when the user operates the intelligent interaction device to send a command to query medical device data, the main controller 31 sends the query command data to the medical monitoring device through the low-power Bluetooth module 62, and the device returns the corresponding monitoring data after receiving the command.

[0102] After the main controller 31 receives the data returned by the external device, it will analyze and process the data. According to the data content, the main controller 31 will decide whether further operations are needed or feedback to the user.

[0103] If feedback information needs to be sent to the user, the main controller 31 will send the feedback information to the voice interaction unit 32. The voice interaction unit 32 uses the bone conduction speaker 34 to inform the user of the feedback information in the form of a voice prompt. For example, when the medical monitoring device returns abnormal data, the main controller 31 sends the abnormal information to the voice interaction unit 32, and the bone conduction speaker 34 will issue a voice prompt of "Medical data is abnormal, please pay attention".

[0104] In an embodiment of the present application, as Figures 1-6 shown, the touch and voice collaborative confirmation module 3 is connected to the helmet body 1 through a magnetic quick-release structure 7. The magnetic quick-release structure 7 includes a magnetic base 71 embedded in the outer shell 11 of the helmet body 1 and a buckle 72 provided on the back of the touch and voice collaborative confirmation module 3, supporting quick disassembly and assembly.

[0105] It is understandable that before installation, it is necessary to ensure that both the helmet body 1 and the touch and voice collaboration confirmation module 3 are in normal conditions. At this time, the magnetic base 71 has been pre-embedded in the outer shell 11 of the helmet body 1, and the buckle 72 is also installed on the back of the touch and voice collaboration confirmation module 3.

[0106] The user picks up the touch and voice collaboration confirmation module 3 and turns the side with the buckle 72 on its back towards the position of the magnetic base 71 on the outer shell 11 of the helmet body 1. With the magnetic attraction of the magnetic base 71, it can assist the user to quickly and accurately align the buckle 72 with the magnetic base 71.

[0107] When the buckle 72 is aligned with the magnetic base 71, the magnetic force of the magnetic base 71 will firmly adsorb the buckle 72, fixing the touch and voice collaboration confirmation module 3 firmly on the helmet body 1. At the same time, the mechanical structure between the buckle 72 and the magnetic base 71 will further strengthen the connection stability, preventing the module from loosening or falling off easily during use.

[0108] While the touch and voice collaboration confirmation module 3 is connected to the helmet body 1 through the magnetic quick-release structure 7, the internal circuit will also be completed. In this way, an electrical connection can be achieved between the flexible arc-shaped touch band 2 and the touch and voice collaboration confirmation module 3 through wires, ensuring the normal transmission of signals.

[0109] During the normal use of the intelligent interaction device, the magnetic quick-release structure 7 can ensure the connection stability between the touch and voice collaboration confirmation module 3 and the helmet body 1. The combined structure of the magnetic base 71 and the buckle 72 can withstand a certain amount of external force impact and vibration. For example, the shaking generated by the user during exercise or work will not cause the module to separate from the helmet, ensuring the normal operation of the device.

[0110] Since the touch and voice collaboration confirmation module 3 is firmly connected to the helmet body 1, the user operation signals (such as click and slide signals) collected by the flexible arc-shaped touch band 2 can be stably transmitted through wires to the touch and voice collaboration confirmation module 3 for processing. At the same time, the command signals sent by this module can also be smoothly transmitted to other relevant components to achieve the multi-modal touch interaction function.

[0111] It should be noted that the control method of this application can be automatically controlled by a controller. The control method of the controller can be realized by simple programming by those skilled in the art and belongs to the common knowledge in this field. And this application mainly aims to protect the mechanical structure, so the control method and circuit connection of this application will not be explained in detail.

[0112] Specifically, assume that a food delivery rider is using this intelligent interaction device. During the delivery process, the rider receives a new order notification. At this time:

[0113] 1. Order receiving operation: The rider clicks on the right touch area of the flexible arc touch band 2 with the jaw. The pressure sensor array 21 quickly detects the pressure change, converts it into an electrical signal, and transmits it through a wire to the main controller 31 in the touch and voice collaborative confirmation module 3. The main controller 31 recognizes that this is a signal for the order receiving operation and triggers the voice interaction unit 32 to work. The voice interaction unit 32 emits a prompt sound through the bone conduction speaker 34: "Order receiving operation detected. Do you confirm to receive the order? Please reply verbally." The rider replies "Confirm to receive the order", and the miniature microphone 33 collects the rider's voice signal and transmits it to the voice interaction unit 32 for processing.

[0114] 2. Secondary confirmation: The secondary confirmation logic unit 35 has preset the dual verification conditions of "voice + blink". After receiving the voice confirmation information, the eye recognition system 5 starts to work. The miniature camera 51 collects the rider's eye image information and transmits it to the biometric processor 52. The biometric processor 52 analyzes whether the rider blinks through the blink frequency detection algorithm. If the rider blinks and the blink frequency meets the preset range, the biometric processor 52 transmits the analysis result to the secondary confirmation logic unit 35 through a wireless signal. When both the voice and blink verifications pass, the secondary confirmation logic unit 35 determines that the confirmation is successful and sends an instruction execution signal to the main controller 31.

[0115] 3. Instruction execution and feedback: After receiving the execution signal, the main controller 31 sends an order receiving instruction to the paired mobile phone through the low-power Bluetooth module 62 of the external device interface 6. After the mobile phone successfully receives the order, it returns the feedback data of successful order receiving. The low-power Bluetooth module 62 receives this data and transmits it to the main controller 31. The main controller 31 analyzes and processes the feedback data, and then sends the feedback information to the voice interaction unit 32. The voice interaction unit 32 informs the rider through the bone conduction speaker 34: "Order received successfully".

[0116] 4. Operation to view order details: When the rider wants to view the order details, a sliding operation is performed in the middle touch area of the flexible arc touch band 2. The capacitive sliding trajectory recognition unit 22 senses the sliding action, uses the coordinate tracking algorithm to collect the sliding point coordinate information in real time, combines it into sliding trajectory data and transmits it to the sliding path detection module 4. The sliding path detection module 4 compares the received trajectory data with the preset sliding trajectory for viewing order details in the dynamic path database 41. If the comparison is successful, the sliding path detection module 4 sends a matching success message to the main controller 31.

[0117] 5. Execution and feedback of the order details viewing instruction: Based on the successful matching information, the main controller 31 sends an order details viewing instruction to the mobile phone through the low-power Bluetooth module 62. The mobile phone returns the order details data, which is received by the low-power Bluetooth module 62 and then transmitted to the main controller 31. The main controller 31 then sends the order details data to the voice interaction unit 32, and the voice interaction unit 32 informs the rider of the order details in the form of voice broadcast through the bone conduction speaker 34, such as "The order starting point is Building XX, XX Community, the ending point is XX Building, and the order amount is XX yuan".

[0118] Examples of each scenario:

[0119] (1) Nursing scenario for the elderly in long-term bedridden state:

[0120] When the elderly in long-term bedridden state feel thirsty during operation, they click on the left touch area of the flexible arc touch band 2, and the pressure sensor array 21 transmits the signal to the main controller 31 of the touch and voice collaborative confirmation module 3. The main controller 31 triggers the voice interaction unit 32, and the bone conduction speaker 34 emits "Do you want to drink water? Please answer by voice". If the elderly reply "Yes", the voice interaction unit 32 collects the voice and verifies it through the secondary confirmation logic unit 35. If it passes, the main controller 31 sends an instruction to the nursing robot through the external device interface 6, and the robot delivers water to the elderly.

[0121] If the elderly need to adjust the angle of the hospital bed, after clicking on the left touch area, the eye recognition system 5 is activated. The micro camera 51 collects the eye image and transmits it to the biometric processor 52 to detect the blinking action of the elderly. After confirming the blink, the biometric processor 52 transmits the signal to the secondary confirmation logic unit 35. After verification by combining with the touch signal and passing, the main controller 31 controls the hospital bed to adjust the angle with an accuracy of 0.5°.

[0122] When the elderly need to call the nurse station, they slide at the right forehead position of the flexible arc touch band 2, and the capacitive sliding trajectory recognition unit 22 transmits the trajectory data to the sliding path detection module 4. After confirming that the sliding operation is effective, at the same time, the eye recognition system 5 detects the gazing direction of the elderly. If the gazing direction is towards the nurse station direction (predetermined area), the biometric processor 52 transmits the data to the secondary confirmation logic unit 35. After verification and passing, the main controller 31 sends an emergency call signal to the nurse station through the external device interface 6, and attaches the elderly's illness grading information (determined according to the preset rules).

[0123] (2) Factory equipment operation scenario:

[0124] In a noisy factory environment, when workers operate equipment, they operate on the flexible arc-shaped touch band 2 through the jaw. For example, to start the equipment, click on the left touch area, and the pressure sensor array 21 transmits a signal to the main controller 31. Since voice control is not suitable, at this time, it mainly relies on the eye recognition system 5 for confirmation. The micro camera 51 collects eye images, and the biometric processor 52 detects the blinking action and transmits the result to the secondary confirmation logic unit 35. After passing the verification, the main controller 31 sends a start instruction to the equipment through the external device interface 6.

[0125] When adjusting the equipment parameters, the worker performs a specific sliding operation on the flexible arc-shaped touch band 2, and the capacitive sliding trajectory recognition unit 22 transmits the trajectory data to the sliding path detection module 4 to compare with the preset path to confirm the operation intention. At the same time, the eye recognition system 5 detects blinking or gazing actions for secondary confirmation. After passing the confirmation, the main controller 31 sends a parameter adjustment instruction to the equipment, realizing hands-free and accurate operation of the equipment.

[0126] (III) Hospital surgeon's operation scenario:

[0127] During the operation, the hospital surgeon is in a quiet environment and needs to avoid facial contact with the equipment. When operating the surgical assistance equipment, the doctor clicks on the left touch area of the flexible arc-shaped touch band 2 with the jaw, and the pressure sensor array 21 transmits a signal to the main controller 31 of the touch and voice collaborative confirmation module 3. At this time, the "left touch + blink" recognition and confirmation method is adopted. The micro camera 51 of the eye recognition system 5 collects eye images, and the biometric processor 52 detects the blinking action and transmits the data to the secondary confirmation logic unit 35. After passing the verification, the main controller 31 sends an instruction to the surgical assistance equipment through the external device interface 6, realizing precise operation, meeting the ISO80369-1 standard, and reducing the risk of cross-infection caused by equipment contact.

[0128] (IV) Disaster relief site scenario:

[0129] At the disaster relief site, the rescue workers wear a device with relevant equipment added. If they need to make a call for support, click on a specific area of the flexible arc-shaped touch band 2 (for example, the right touch area is set as the communication function area), and the pressure sensor array 21 transmits a signal to the main controller 31. The main controller 31 triggers the voice interaction unit 32, and the bone conduction speaker 34 prompts "Please state the call object". The rescue worker makes a voice reply. After the voice interaction unit 32 collects the voice and passes the verification by the secondary confirmation logic unit 35, the main controller 31 connects to the mobile phone through the low-power Bluetooth module 62 of the external device interface 6 to make a call, and at the same time, both hands can continue the rescue work.

[0130] If it is necessary to control the drone to transport the wounded or equipment, a specific sliding operation is performed on the flexible arc touch belt 2, and the capacitive sliding trajectory recognition unit 22 transmits the trajectory data to the sliding path detection module 4 to confirm the operation intention. The eye recognition system 5 detects blinking or gazing actions for secondary confirmation. After the confirmation is passed, the main controller 31 sends a control command to the drone through the external device interface 6 to achieve efficient transportation of rescue supplies and personnel.

[0131] (5) Agricultural harvester operation scenario:

[0132] When operating an agricultural harvester, the operator's hands need to constantly control the harvester. If you want to switch the working mode of the harvester (such as switching from the harvesting mode to the threshing mode), click on the middle touch area of the flexible arc touch belt 2, and the pressure sensor array 21 transmits the signal to the main controller 31 of the touch and voice collaborative confirmation module 3. The main controller 31 triggers the voice interaction unit 32, and the bone conduction speaker 34 prompts "Do you want to switch to the threshing mode? Please confirm by voice". The operator replies "Yes" by voice. After the voice interaction unit 32 collects the voice and verifies it through the secondary confirmation logic unit 35, the main controller 31 sends a switching command to the harvester control system through the external device interface 6.

[0133] When adjusting the harvesting height of the harvester, the operator slides on the flexible arc touch belt 2, and the capacitive sliding trajectory recognition unit 22 transmits the trajectory data to the sliding path detection module 4 to compare with the preset path to confirm the operation intention. At the same time, it can be combined with the blinking or gazing confirmation of the eye recognition system 5 (such as gazing at the height adjustment indication area). After the secondary confirmation logic unit 35 verifies and passes, the main controller 31 sends a height adjustment command to the harvester, enabling precise control of the harvester without the hands leaving the control lever.

[0134] In summary, an intelligent interaction device based on jaw multi-modal touch in the embodiment of the present application. This intelligent interaction device reduces the misoperation rate to less than 5% through double confirmation of "touch + voice" or "touch + eye recognition", supports flexible switching between voice, touch, and eye recognition modes, and significantly improves efficiency with jaw touch combined with sliding path recognition. It conforms to the ISO80369-1 standard and reduces cross-infection. Through Bluetooth and USB-C interfaces, it is widely used in multiple fields.

[0135] In the description of this specification, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0136] In the description of this specification, the descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0137] Although the embodiments of this application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting this application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An intelligent interaction device based on multi-modal jaw touch control, characterized in that, It includes a helmet body (1), a flexible arc-shaped touch band (2), a touch and voice collaborative confirmation module (3), a sliding path detection module (4), and an eye recognition system (5). Among them, the helmet body (1) consists of a shell (11) and a lining (12). The flexible arc-shaped touch band (2) is provided at the bottom of the shell (11) and conforms to the curve of the user's mandible; the touch and voice collaborative confirmation module (3) is fixed to the rear part on the right side of the helmet body (1) and is electrically connected to the flexible arc-shaped touch band (2) through a wire; the sliding path detection module (4) is integrated inside the collaborative confirmation module (3) and obtains path data through the sliding of the flexible arc-shaped touch band (2); the eye recognition system (5) is rotatably mounted on the helmet body (1) through a rotating bracket (50) and can be rotated up to a forty-five-degree angle under the visor at most. It includes a micro camera (51) and a biometric processor (52), and communicates with the touch and voice collaborative confirmation module (3) through a wireless signal.

2. The intelligent interaction device based on multi-modal jaw touch control according to claim 1, characterized in that, The flexible arc-shaped touch band (2) has a width of 3-5 cm and is internally provided with a pressure sensor array (21) and a capacitive sliding trajectory recognition unit (22). The surface is divided into three touch areas: left, middle, and right, which respectively correspond to functions such as navigation, page switching, and order receiving.

3. The intelligent interaction device based on multi-modal jaw touch control according to claim 1 or 2, characterized in that, The touch and voice collaborative confirmation module (3) includes a main controller (31), a voice interaction unit (32), a micro microphone (33), a bone conduction speaker (34), and a secondary confirmation logic unit (35). Among them, the main controller (31) receives the click signal from the pressure sensor array (21); the voice interaction unit (32) collects the user's voice through the micro microphone (33) and plays a prompt tone through the bone conduction speaker (34); the secondary confirmation logic unit (35) presets instruction execution conditions (such as "voice + blink" double verification).

4. An intelligent interaction device based on multi-modal jaw touch according to claim 1, characterized in that, The sliding path detection module (4) includes a dynamic path database (41). The dynamic path database (41) stores various sliding trajectories (such as "L" shape, spiral shape) and is compared in real time through the coordinate tracking algorithm of the capacitive sliding trajectory recognition unit (22).

5. An intelligent interaction device based on multi-modal jaw touch according to claim 1, characterized in that, The viewing angle range of the micro camera (51) of the eye recognition system (5) is 40°×30°. The biometric processor (52) is internally provided with a blink frequency detection algorithm and a pupil gaze direction recognition algorithm, and is linked with the secondary confirmation logic unit of the touch and voice collaborative confirmation module (3).

6. The intelligent interaction device based on multi-modal jaw touch control according to claim 1, characterized in that, It also includes an external device interface (6) including a USB-C interface (61) and a low-power Bluetooth module (62), which is integrated at the bottom of the touch and voice collaborative confirmation module (3) and is used for communication with mobile phones, medical devices, or industrial terminals.

7. An intelligent interaction device based on multi-modal jaw touch according to claim 1, characterized in that, The touch and voice collaborative confirmation module (3) is connected to the helmet body (1) through a magnetic quick-release structure (7). The magnetic quick-release structure (7) includes a magnetic base (71) embedded in the shell (11) of the helmet body (1), and a buckle (72) provided on the back of the touch and voice collaborative confirmation module (3), which supports quick disassembly and assembly.

Images