A bidirectional sign language communication system with a multi-sensor glove and haptic feedback

Abstract

A bidirectional sign language communication system (100) with multisensor glove and haptic feedback, comprising: a wearable glove with a variety of sensors configured to detect hand, finger and wrist movements of a user, wherein the sensors include at least curvature sensors, a pinch detector, a strain sensor, a pressure measuring arrangement and an inertial measurement unit; a processing unit that is functionally coupled with the sensors and configured to process sensor signals and derive sign language tokens using a machine learning model; a wireless communication interface configured to transmit the derived tokens to a companion device for playback as text or speech; wherein the companion device is further configured to receive spoken or textual inputs, convert the inputs into coded tactile patterns, and transmit the patterns to the wearable glove; and a distributed haptic feedback subsystem integrated into the wearable glove and configured to present tactile patterns to the user via fingertip actuators, vibration motors, or a tactile matrix, wherein the system (100) enables bidirectional real-time offline communication between a sign language user and a non-sign language user.

Description

[0001] The present invention relates generally to human-computer interaction systems and assistive communication technologies, in particular a bidirectional sign language communication system with a multisensor glove and haptic feedback.

[0002] Sign language is frequently used as a primary form of communication by people with hearing or speech impairments. Despite its importance, significant communication barriers persist, as the majority of the population does not know sign language. This leads to challenges in daily life, education, and professional settings. To address this problem, various technologies have been developed. Camera-based recognition systems use computer vision to interpret gestures, but they typically require controlled environments, good lighting, and high computing power, and are susceptible to obstructions, making them impractical for mobile or practical use. Wearable solutions, such as interpreter gloves with flex sensors or inertial measurement units, have been proposed to capture hand and finger movements.These systems are generally limited in scope and offer only unidirectional translation—they convert signs into text or speech. Such devices do not provide a feedback channel to send spoken or textual responses back to the sign language user in an accessible format. Furthermore, most existing systems lack accuracy due to their reliance on a small number of sensors, suffer from high latency when dependent on cloud computing, and consume excessive power, limiting their portability. Crucially, they also fail to meet the needs of deafblind users, as they lack integrated tactile or haptic mechanisms for conveying information through touch.

[0003] To solve this problem, the present invention offers a bidirectional sign language communication system with a multisensor glove and haptic feedback.

[0004] The system enables bidirectional communication, encompassing both the translation of signs into speech and of speech into tactile information, thus bridging the gap between sign language users and non-sign language users.

[0005] The system supports accessibility for deafblind people through an integrated multimodal haptic feedback array that transmits textual and spoken information in tactile patterns such as Braille or Lorm.

[0006] The system achieves low latency, with internal inference and Bluetooth Low Energy (BLE) connectivity ensuring near real-time communication without dependence on cloud servers.

[0007] The system is portable and energy-efficient, featuring a compact glove design with a low-power microcontroller and a quantized neural network for extended battery life during continuous use.

[0008] The system offers scalability and adaptability, and allows integration with smartphones and other companion devices for additional services such as language translation, storage of conversation logs, or cloud connectivity, if desired.

[0009] The system is independent of environmental conditions such as lighting or obstruction and is therefore reliable for indoor and outdoor applications.

[0010] In one embodiment, the present invention provides a bidirectional sign language communication system with a multisensor glove and haptic feedback. The present invention discloses a bidirectional sign language communication system comprising a wearable multisensor glove, a processing unit, wireless communication, and a multimodal haptic feedback subsystem. The glove detects hand and finger gestures through a combination of flexion, pinch, stretch, pressure, and motion sensors. An embedded machine learning model processes the signals and transmits sign language tokens to a companion device for text or speech playback. Conversely, the companion device converts spoken or textual input into tactile patterns, which are transmitted to the user via fingertip actuators, vibration motors, and a wrist-mounted micro-pin matrix.The system works offline with low latency and provides a portable and inclusive platform for real-time communication between sign language users, including deaf-blind individuals, and non-sign language users.

[0011] The invention is explained again below with reference to the figure. This shows: Fig. : a block diagram of a bidirectional sign language communication system with multisensor glove and haptic feedback.

[0012] Fig.Figure 1 shows a block diagram of a bidirectional sign language communication system with a multisensor glove and haptic feedback. The present invention relates to a bidirectional sign language communication system (100) comprising a wearable glove, a processing and communication unit, an accompanying device, and a multimodal haptic feedback subsystem. The glove integrates several sensors, including curvature sensors on the finger joints, a Hall-effect pinch detector, a liquid metal strain sensor around the wrist, a palm pressure sensor matrix, and an inertial measurement unit. Together, these sensors detect finely graduated hand and wrist movements to enable accurate recognition of sign language gestures. The sensor signals are processed locally by a low-power embedded unit running a quantized recurrent neural network or a similar machine learning model.The derived sign language tokens are transmitted via Bluetooth Low Energy (BLE) or other wireless protocols to a companion device such as a smartphone or tablet, where they are converted into text or synthesized speech. This allows sign language users to communicate seamlessly with people who do not understand sign language.

[0013] Conversely, the companion device receives spoken or textual input, converts it into coded tactile patterns, and sends them back to the glove. A distributed haptic feedback array, consisting of fingertip actuators, palm vibration motors, and a wrist-mounted micro-pin matrix, provides the user with intuitive tactile signals, including Braille or Lorm patterns. By combining robust multisensor input, in-device learning, low-latency wireless communication, and multimodal haptic output, the system (100) enables portable, offline-available, and inclusive two-way communication for deaf and deafblind individuals. Reference symbol list 100 System

Claims

[1] A bidirectional sign language communication system (100) with multisensor glove and haptic feedback, comprising: a wearable glove with a variety of sensors configured to detect hand, finger and wrist movements of a user, wherein the sensors include at least curvature sensors, a pinch detector, a strain sensor, a pressure measuring arrangement and an inertial measurement unit; a processing unit that is functionally coupled with the sensors and configured to process sensor signals and derive sign language tokens using a machine learning model; a wireless communication interface configured to transmit the derived tokens to a companion device for playback as text or speech; wherein the companion device is further configured to receive spoken or textual inputs, convert the inputs into coded tactile patterns, and transmit the patterns to the wearable glove; and a distributed haptic feedback subsystem integrated into the wearable glove and configured to present tactile patterns to the user via fingertip actuators, vibration motors, or a tactile matrix, wherein the system (100) enables bidirectional real-time offline communication between a sign language user and a non-sign language user. [2] System (100) according to claim 1, wherein the curvature sensors comprise optical polymer fiber elements positioned along the finger joints to detect bending movements. [3] System (100) according to claim 1, wherein the pinch detector comprises a magnetic element on the thumb and a Hall effect sensor on an adjacent finger for detecting contact. [4] System (100) according to claim 1, wherein the strain sensor comprises a liquid metal conductor embedded in an elastomeric channel positioned around the wrist, wherein the change in resistance indicates a strain or rotation. [5] System (100) according to claim 1, wherein the pressure measuring arrangement comprises a 4×4 piezoelectric resistor grid located on the palm surface of the glove. [6] System (100) according to claim 1, wherein the inertial measurement unit comprises a sensor with 9 degrees of freedom, comprising an accelerometer, a gyroscope and a magnetometer. [7] System (100) according to claim 1, wherein the processing unit executes a quantized recurrent neural network model stored in non-volatile memory to derive sign language tokens locally. [8] System (100) according to claim 1, wherein the wireless communication interface for communication with a companion smartphone operates with Bluetooth Low Energy. [9] System (100) according to claim 1, wherein the haptic feedback subsystem comprises linear resonance actuators at the fingertips configured to generate short-term tactile impulses.

Images