Oral input device

Abstract

The present invention relates to an oral input device with a tactile sensor array which is placed over a lingual surface of a first jaw. A support structure serves as an interface between the first jaw and the tactile sensor array and serves as point of attachment for the tactile sensor array to allow it to be placed over the lingual surface. The tactile sensor array receives pressure input from a tongue.

Description

[0001] ORAL INPUT DEVICE

[0002] TECHNICAL FIELD

[0003] The present disclosure generally relates to an input device, specifically, a tactile oral pad for receiving user input from within an oral cavity.

[0004] BACKGROUND

[0005] The advent of touchscreen devices such as smartphones, tablets, and intelligent watches has significantly impacted daily life by simplifying tasks like gaming and typing with a slide of fingertips. However, these devices present challenges for individuals with limited mobility or physical disabilities and can pose contamination risks in surgical operations due to contact with the keyboard. To address these limitations, several technologies have emerged, including voice recognition, eye-blink tracking, head-movement tracking, and action recognition using flexible tactile sensors. While these alternatives offer solutions, they lack the flexibility and intuitiveness of touchscreens, making them less suitable for social settings. For example, voice-to-text interaction may be impracticable in noisy environments or during meetings and public events. Although gesture recognition through soft tactile sensors holds promise for tasks such as typing and device operation, these sensors often struggle to convert a wide range of pressure patterns into specific control commands, thereby limiting their multitasking capabilities.

[0006] Assistive wearable systems incorporating soft tactile sensors to transmit spatiotemporal touch patterns have become increasingly prevalent, particularly in biomedical robotics. Although these systems have shown promise in tasks such as typing and device operation, their ability to convert pressure patterns into specific control commands has been limited, resulting in less dexterity and intuitiveness than finger-operated electronic devices. Accordingly, there is a need for a solution to address these limitations. SUMMARY OF DISCLOSURE

[0007] One aspect of the disclosure is an input device for capturing input from an oral cavity of a user comprising a support structure configured to attach to a first jaw in the oral cavity, the first jaw having at least one tooth, a tactile sensor array attached to the support structure, the tactile sensor array having a first surface functionally disposed over at least a lingual surface of the at least one tooth. Optionally, the at least one tooth is an anterior tooth.

[0008] Optionally, the tactile sensor array further comprises a second surface functionally disposed over an occlusal surface of the at least one tooth. Optionally, the tactile sensor array comprises a sensor matrix. Optionally, the tactile sensor array further comprises an electrode array in communication with the sensor matrix. Optionally, the sensor matrix comprises spinous microstructures. Optionally, the sensor matrix and the spinous microstructures comprise carbon nanotubes of about 15 wt%. Optionally, the electrode array is in communication with the sensor matrix by way of the spinous microstructures.

[0009] Optionally, the pressure input is a tongue movement comprising movement of the tongue that is transmitted to the tactile sensor array in at least substantially a first trajectory. Optionally, wherein the tactile sensor array is configured to recognize the tongue movement, the tongue movement comprising at least one tongue touch pattern. Optionally, the at least one tongue touch pattern corresponds to predetermined regions on the tactile sensor array.

[0010] Optionally, the pressure input is a combination of the tongue movement and at least one occlusal pattern. Optionally, the pressure input corresponds to a character, wherein the character or numeral is predicted based on at least the first trajectory in a time-dependent manner. Optionally, the combination of the tongue movement and at least one occlusal pattern occur simultaneously. Optionally, the tongue movement further comprises at least one tongue movement pattern and wherein each of the at least one tongue movement pattern comprises at least one of the at least one tongue touch pattern.

[0011] Optionally, the at least one tongue movement pattern is derived based on their statistical correlation between each of the at least one tongue touch pattern and each of the at least one occlusal pattern. Optionally, the support structure is shaped to fit a contour of the first jaw. Optionally, the pressure input further comprises intraoral movement, the intraoral movement comprising at least movement of orthodontics and respiration.

[0012] BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Some non-limiting exemplary embodiments or features of the disclosed subject matter are illustrated in the drawings.

[0014] In the drawings:

[0015] FIG. 1 is an exploded view of input device 100 according to some embodiments of the present disclosure.

[0016] FIGS. 2A-2B schematically illustrate the input device 100 as depicted within an oral cavity according to some embodiments of the present disclosure. FIG. 2A is a schematic view of the input device 100 where first surface 107 receives pressure input 200 directly from tongue 202; and FIG. 2B is a schematic view of the input device 100 where second surface 109 receives pressure input 200 during an occlusal action.

[0017] FIG 3 depicts an exploded view depicting the multi-layer structure of the tactile sensor array 106 according to some embodiments of the present disclosure.

[0018] FIG. 4 depicts schematic view of the multi-layer structure of the tactile sensor array 106 including spinous microstructure 402 according to some embodiments of the present disclosure.

[0019] FIG. 5 depicts schematic view of the multi-layer structure of the tactile sensor array 106 including spinous microstructure 402 according to some embodiments of the present disclosure.

[0020] FIG. 6A are graphs depicting various performance perimeters of the tactile sensor array 106 according to some embodiments of the present disclosure. FIG. 6 A depicts current registered by the tactile sensor array 106 in response to continuous application of water droplets over time. FIG. 6B depicts intensity over time when a force is applied to the tactile sensor array 106.

[0021] FIG. 7 depicts sensor matrix 304 as viewed under a scanning electron microscope before and after being subjected to more than 10000 compression cycles according to some embodiments of the present disclosure. FIG. 8 is a schematic view of the sensor matrix 304 and predetermined areas 802 (sensor partitions) according to some embodiments of the present disclosure.

[0022] FIG. 9 depicts a circular diagram displaying Pearson correlation coefficients between the 9 tongue touch patterns and 6 occlusal patterns which are each predetermined areas 802 on first surface 107 and second surface 109 respectively.

[0023] FIG. 10 depicts a 2D t-SNE visualization of different teeth and tongue patterns derived from the 9 tongue touch patterns and 6 occlusal patterns.

[0024] Fig. 11 depicts bode plots of the sensor matrix with varying CNT content (10-20%wt).

[0025] Fig. 12 depicts a frequency response graph when low, medium or high pressure was applied to the sensor matrix 304.

[0026] Fig. 13 depicts sensor data matrix and corresponding temporal trajectories for several letters.

[0027] DETAILED DESCRIPTION

[0028] Described herein are various non-limiting embodiments and configurations of oral input devices.

[0029] Referring to Fig. 1 , an aspect of the present disclosure is an input device 100 for capturing user input from within an oral cavity. The input device 100 comprises support structure 102 which is shaped to fit over first jaw 104 with at least one tooth 105. While depicted in Fig. 1 as an upper jaw, the first jaw 104 may instead be a lower jaw. Alternatively, the input device 100 may be fitted to both an upper jaw and a lower jaw. The support structure 102 is substantially (J-shaped to conform to the shape of the first jaw 104. Alternatively, the support structure 102 may be shaped to encompass substantially a region corresponding to an anterior tooth 112 without substantially encompassing the entirety of teeth present in the first jaw 104 as would be the case with a U- shaped support structure 102. In some embodiments, the support structure 102 may encompass, substantially, a region corresponding to anterior teeth. The support structure 102 serves to attach the tactile sensor array 106 to the first jaw 104. The tactile sensor array 106 has first surface 107 disposed over at least a lingual surface 108, that is, a surface of the at least one tooth 105 that faces the tongue. The first surface 107 is a pressuresensitive element and is functional in that in response to pressure applied to the first surface 107, the tactile sensor array 106 proper receives a pressure input (200 of Fig. 2A-2B) by way of the first surface 107, the pressure input (200 of Fig. 2A-2B) received through the support structure 102. The first surface 107 spans a region that corresponds substantially to anterior teeth of the first jaw 104. Alternatively, the first surface 104 may span region that corresponds to at least one tooth. In some embodiments, the support structure 102 may be an adhesive film (not depicted) that attaches the tactile sensor array 106 to the lingual surface 108 of the at least one tooth 105. The adhesive film may prevent liquid from reaching the tactile sensor array 106 while the input device 100 is in use.

[0030] The support structure 102 may substantially comprise an elastomeric material. The elastomeric material may be resilient such that pressure exerted on the support structure 102 can be transmitted to the first surface 107. Given the expected exposure to an aqueous environment, the support structure 102 may resist infiltration of aqueous substances that may cause the support structure 102 to swell. Further, and given direct contact with an oral cavity, the support structure 102 may also demonstrate biocompatibility. In some embodiments, and while attached to the support structure 102, pressure may be applied directly to the first surface 107. In some embodiments, the elastomeric material is: a polyurethane elastomer. It will be understood that any resilient material that demonstrates at least some of the above characteristics may be employed as a material for the support structure 102.

[0031] In addition to the first surface 107, the tactile sensor array 106 may optionally further comprise a second surface 109 as depicted in Fig. 1 that is also a pressure-sensitive element. The second surface 109 is functionally disposed over at least an occlusal surface 110, the occlusal surface 110 being the chewing surface of at least one tooth on the first jaw 104. While depicted in Fig. 1 as spanning the same teeth as the first surface 107, the second surface 109 may alternatively span a different tooth or teeth from that of the first surface 107.

[0032] The input device 100 may further comprise circuitry 114 that is disposed on the support structure 102 and in communication with the tactile sensor array 106 to facilitate data acquisition, transmission and communication with a remote device (not depicted) that may be outside the oral cavity. The circuitry 114 may comprise a printed circuit board. The printed circuit board may be a flexible printed circuit board to better withstand the regular flexing of the input device 100 while in use. An onboard power supply (not depicted) may be present to provide power to the circuitry.

[0033] The tactile sensor array 106 and the circuitry 114 may be at least partially encapsulated within the support structure 102 by way of encapsulating structure 116 that is fitted over the support structure 102. In embodiments where pressure may be applied directly to the first surface 107, the encapsulating structure 116 may not encompass at least part of the first surface 107. In this manner, the tactile sensor array 106 and the circuitry 114 are housed within the support structure 102. As described above, pressure is then transmitted to the tactile sensor array 106 through the support structure 102. Alternatively, the tactile sensor array 106 and the circuitry 114 may be attached within the support structure 102. In some embodiments, where pressure is applied directly to the tactile sensor array 106, the tactile sensor array 106 is instead directly disposed on external surface 118 of the support structure 102. In some embodiments the circuitry 114 may be disposed outside of the oral cavity.

[0034] Referring to Fig. 2A, the input device 100 is depicted with the tactile sensor array 106 as disposed on the support structure 102. Pressure input 200 is applied to the first surface 107 by way of tongue 202. Pressure is registered by the tactile sensor array 106 when the tongue 202 slides into contact with the first surface 107. The pressure input 200 may also be in the form of tongue movement where while in contact with the first surface 107, the tongue 202 moves across the first surface 107 in at least substantially in a first direction. The first direction may be substantially horizontal, vertical or diagonal. The circuitry 114 encapsulated within the support structure and is not visible.

[0035] Referring Fig. 2B, the input device 100 is depicted with the tactile sensor array 106 disposed over the first jaw 104, the first jaw 104 being the lower jaw. Pressure input 200 is applied to the second surface 109 via an occlusal action, i.e. when the lower jaw comes is brought into contact with upper jaw 204. Alternatively, the input device 100 may instead be disposed over the upper jaw 204.

[0036] Referring to Fig. 3, The first surface 107 and the second surface 109 of the tactile sensor array 106 are each multi-layered comprising interface layer 302, sensor matrix 304, electrode array 306 and insulation layer 308. The interface layer 302 comprises substantially a resilient material that may deform when subjected to the pressure input 200, be it a tongue movement or an occlusal action. In embodiments where the interface layer 302 is intended to be in direct contact with the pressure input 200, the interface layer 302 may resist infiltration of aqueous substances that may cause the support structure 102 to swell. Further, and given direct contact with an oral cavity, the interface layer 302 may also demonstrate biocompatibility. In some embodiments the interface layer 302 comprises poly dimethylsiloxane (PDMS).

[0037] The sensor matrix 304 may be piezoresistive and comprises a carbon nanotube (CNT) polymer and silicone composite where the CNT polymer is 10-20 wt%. In some embodiments the CNT polymer is 10, 11, 12, 13, 14, 15, 16, 17, 17.5, 18, 19 or 20 wt%. The portion of the sensor matrix 304 that is disposed over the first surface 107 is 10x12 while the portion of the sensor matrix 304 disposed over the second surface is 4x12 It will be understood that the size of the sensor matrix 304 may vary depending on specific input requirements. Alternatively, the piezoresistive sensor matrix 304 may be substituted with any sensor or sensor matrix capable of outputting a voltage that is proportional to the pressure input 200. In order to measure the voltage generated by the sensor matrix 304, the electrode array 306 is operably coupled to the sensor matrix 304. The electrode array 306 may have a single-sided interconnection structure of surface electrodes 308 that allows for pixelated pattern recognition with minimal interference from adjacent pixels, thus enabling clear recognition of patterns written by finger pressure (~20 kPa). In some embodiments, the electrode array 306 is an interdigitated array electrode. Insulation layer 308 is provided to insulate the electrode array 306. The insulation layer 308 may be selected from polyimide or parylene. It will be understood that an insulator of appropriate thickness that can prevent liquids from coming into direct contact the sensor matrix 304 can be used. This singlesided electrode connection structure was found to detect a smaller threshold pressure, and the independence of a single pressure detection point is higher.

[0038] Referring to Fig. 4, the sensor matrix 304 may further comprise spinal microstructures 402 which are disposed between the electrode array 306 and the sensor matrix 304. In some embodiments the spinal microstructures 402 are randomly distributed. The spinous microstructures 402 enhance sensitivity and detection accuracy by creating concentrated local stress. Referring to Fig. 5, the tactile sensor array 106 is depicted with the interface layer 302, the sensor matrix 304 in the form of a piezoresistive film with integrated spinous microstructures 402. Both the sensor matrix 304 and the spinous microstructures 402 comprise a carbon nanotube structure 502 while the interface layer 302 comprises substantially PDMS. The piezoresistive film with integrated spinous microstructures 402 demonstrated high sensitivity and resistance decreased under pressure due to the spinous microstructures 402 as evidenced in Figs. 6A-6B.

[0039] Referring to Fig. 6A, the current-time graph demonstrates the consistent detection of continuous water droplets (150pl) by the tactile sensor array 106. Referring to Fig. 6B, the intensity-time graph indicates a response time of about 30ms when pressure is applied to the tactile sensor array 106, indicating sufficient sensitivity for real world applications.

[0040] The durability of the multi-layer tactile sensor array 106 in scanning electron microscope images of the piezoresistive film before and after being subjected to over 10000 compressions cycles as depicted in Fig 7

[0041] Referring to Fig. 8, the input device 100 is configured to recognise the pressure input (200 of Fig. 2A) as a tongue movement comprising at least one tongue touch pattern. In some embodiments as depicted in Fig. 8, the at least one tongue touch pattern Pi - P9 each correspond to a predetermined area 802 on the sensor matrix 304 spanning the first surface 107 in a 3x3 configuration. In some embodiments, instead of a 3x3 configuration, a 2x2, 2x3 ,2x4, 3x4, 4x4, 4x5 or 5x5 configuration. In other words, the input device 100 can differentiate the nine distinct touch patterns Pi - P9. In some embodiments, each of the predetermined area 802 are spaced out throughout the sensor matrix 304 spanning the first surface 107.

[0042] In some embodiments, and where the sensor matrix 304 is further disposed over the second surface 109, the input device 100 is configured to recognise at least one occlusal pattern. In some embodiments as depicted in Fig. 8, the at least one occlusal pattern 1-6 each correspond to a predetermined area 802 on the sensor matrix 304 spanning the first surface 107 in a 3x2 configuration. In some embodiments and instead of a 3x2 configuration, a 2x2, 2x3, 2x4, 3x4, 4x4, 4x5 or 5x5 configuration may be employed.

[0043] The inventors have observed that common control of tongue movements and occlusal actions by the cranial nerve opens the possibility of generating unique pressure input that is a combination of at least one tongue movement and at least one occlusal pattern. The combination of the at least one tongue movement and the at least one occlusal pattern may occur sequentially or concurrently i.e. simultaneously.

[0044] In some embodiments the at least one tongue movement comprises at least one touch pattern and at least one occlusal pattern. Referring to Fig. 9, the circular diagram displays Pearson correlation coefficient was calculated to analyse the correlation between the 15 different data matrices (tongue touch patterns Pi - Pg and six occlusal patterns 1-6). Correlations range from 0 to 1. The results reveal that correlations among different tongue patterns are relatively large but remain below 0.4, while the correlation coefficients between tongue touch patterns Pi - Pg (9 tongue touch patterns) and six occlusal patterns 1-6 are all less than 0.1, indicating minimal correlation between tongue and occlusal patterns. The tongue movement may comprise at least one tongue movement pattern. The tongue at least one movement pattern may be derived / encompass the tongue touch patterns Pi - Pg. In some embodiments, about 50 tongue movement patterns were derived from the tongue touch patterns Pi - Pg. Each of the about 50 tongue movement patterns may comprise at least one of the nine tongue touch patterns. In some embodiments, the about 50 tongue movement patterns were derived based on their statistical correlation between each of the tongue touch patterns Pi - Pg and each of the six occlusal patterns 1-6. In some embodiments, the statical correlation may obtained by Pearson correlation coefficient.

[0045] Accurate recognition of the pressure input 200 of the tongue movement patterns and / or occlusal patterns 1-6 are facilitated by a trained CNN network where an approximately 98.7% accuracy in pattern recognition was achieved.

[0046] In some embodiments, the tactile sensor array 106 may function as a touch screen. The pressure input 200, specifically, tongue movement may correspond to character. The character may be an alphabet or a numeral. The alphabet may any one of the 26 letters in the English Alphabet and the numerals may be Arabic numerals. The tongue movement may correspond to other forms of writing. It was observed that there was a correlation between consecutive frames. On this basis, sequential data processing, for example recurrent neural networks can be used to predict the character / numeral based on the trajectory of the tongue while in functional contact with the first surface 107 in a time-dependent manner. Prediction of the character / numeral can be achieved by predicting subsequent trajectories of the tongue movement based on current / accumulated trajectories for each letter / numeral.

[0047] The input device 100, may be further configured to detect a variety of mtraoral movement. Examples of intraoral movement include at least movement of orthodontics within the oral cavity movement of air that occurs with the inspiration and / or expiration during respiration.

[0048] Example 1: Fabrication of the tactile sensor array

[0049] After premixing the silicone elastomer base with the curing agent (mass ratio: 10:1), the mixture of elastomer and carbon nanotubes (CNTs) in different concentrations was evenly scraped onto an abrasive paper using a knife applicator. The resulting composite was degassed in a vacuum container and then heated at 80 °C for 4 h. After cooling to room temperature, the as-fabricated film (thickness: 100 pm) was peeled off from the square acrylic template and used as a pressure sensing film. To further evaluate the effect of spinous microstructures on sensor performance, we conducted a systematic investigation on randomly distributed spinous microstructures with increasing roughness. The roughness was achieved using abrasive paper No.120-2000. Polydimethylsiloxane (PDMS, SYLGARDTM 184) silicone was coated on the surface of the pressure sensing film and cured in an oven at 70 °C for 0.5 h, with a film thickness of 50 - 100 pm. The detailed fabrication process of the piezoresistive sensor based on the CNT / silicone composition with randomly distributed microstructures. The desired interdigital electrode patterns of gold are printed on the pretreated polyimide (PI) film. Finally, the edge of the microstructured piezoresistive film was connected to the edge of the interdigitated electrode array by ultrathin double-sided adhesive tape. The edges and top of the integrated sensor array film are sealed with PDMS to prevent liquids from entering the interior.

[0050] Example 2: Characterization of the tactile sensor array.

[0051] The thickness and morphology of CNT nanofiber and the CNT / silicone composite piezoresistive film with randomly distributed spinous microstructures were characterized using a scanning electron microscope (FEI NOVA NanoSEM 450, Extend data Fig. 1). The performance of the microstructured piezoresistive film was tested on a system consisting of a computer-controlled three-axis translation stage, a force gauge, and a source meter (Agilent B2091 A) (Extend data Fig. 2). Specifically, we tested the current versus time curves of the sensor processed with a No. 1200- sandpaper under a continuously applied pressure of less than 15 kPa and a loading voltage of 1 V As the pressure increases, the current gradually increases. To measure the force response, we used the z-axis stage to repeatedly press the force gauge on the piezoresistive sensor. To further evaluate the durability and stability of the piezoresistive sensor film, 10,000 loading and unloading cycles were performed at a pressure of approximately 20 kPa, and the source meter was used to record the current versus time curve. The temperature characteristics of the piezoresistive sensors were demonstrated by measuring Bode plots at 25-45 °C by placing the sensor film on a heating stage with every 5°. Application tests of piezoresistive sensors in real-time monitoring of human activities were demonstrated by measuring current signals that changed with repeated finger bending and wrist bending. Referring to Figs. 11 and 12, the sensor matrix 304 with CNT polymer 15 wt% demonstrated optimal signal-to-noise ratios for detecting different pressure levels, where low pressure is about 50 kPa, medium pressure is about 100 kPa and high pressure is about 150 kPa.

[0052] Example 4: Circuitry design and characteristics

[0053] The circuit comprises a digital circuit and an analog circuit (Extend data Fig. 4). The main chips include a DA14585 Bluetooth chip, two SN74LVC138A chips, two TMUX1574RSV chips, one TLV9004 chip and one TLV62569 chip. The DA14585, with an ARM Cortex-MO microcontroller and 4-channels of 12bits ADC, is a low-power 5.0 Bluetooth chip with -20 dB transmit sensitivity and -93 dB receive sensitivity. The DAI 4585 is an ultra-low power SoC integrating a 2.4 GHz transceiver and an ARM Cortex-MO® microcontroller with 96 kB of RAM, 128 kB ROM, and 64kB of One-Time Programmable (OTP) memory. It offers a very fast boot time (<50ms) and supports up to 8 BLE connections. It has processing frequency of 16 MHz and can be used as a standalone application processor or as a data pump in hosted systems. SN74LVC138A is a 3-to-8- hne decoder, a combinational circuit that can be used as both a decoder and a demultiplexer. The TMUX1574 is a high-speed 2: 1 single-pole double throw (SPDT) 4-channel switch with powered- off protection up to 3.6 V. TLV9004 is a low-power RRIO 1-MHz chip for cost-sensitive systems with four operational amplifiers. The circuit is flexible and can read data from 16 16 pressure points. Detection of higher- density sensor arrays can be easily realized by simply increasing the number of chips. A signal isolation circuit is required to overcome the extensive crosstalk between sensors. By adjusting the resistance value of the voltage dividing resistor and the low-pass filter, flexible and sensitive pressure detection can be realized. Rastering through the readout resistors in the matrix is controlled by a DA14585 Bluetooth chip by switching the TMUX1574 switches and the SN74LVC138A decoders. The single sensor measurement at the output of the low-pass filter is converted to a digital signal (12-bit resolution; 0-4096 corresponding to 0-3.3 V) and transmitted serially to the computer.

[0054] All circuit components were arranged on a flexible printed circuit board with a thickness of 200 pm. The working voltage of the circuit was 3.3 V and the working current was 1.2 mA. Thus, the total average power consumption of the input device under normal operating conditions was calculated to be 3.96 mW. To reduce the size of the device, we selected a small lithium battery (10 x 10 x 1.6 mm) with a battery capacity of 22 mAh as the power supply. When the input device operates with real-time data processing and Bluetooth communication, the entire system can continue to work for approximately 20 h without recharging the battery. The size of the mam circuit was 10.0 x 14.0 xl .6 mm, which was embedded in custom thermoplastic polyurethane support structure. The sensitive part weight of the input device was approximately 1 .2 g.

[0055] Example 4: Device integration and assembly

[0056] The tactile sensor array is connected to the circuitry, specifically a flexible circuit board via copper- insulated flexible wires with a diameter of 80 pm, and the Bluetooth chip transmits the data to the mobile phone / computer The entire system was integrated into a 3D-printed mouthguard made of a thermoplastic polyurethane elastomer. The mouthguard is designed with the first and second surfaces on its upper and inner surfaces, while the battery and a small circuit board are placed in its hollowed-out section. To prevent the sensor matrix from being damaged by the excessive force of a bite, an elastomer buffer layer was glued to the upper part of the tactile sensor array. Before use, the encapsuled mouthguard is sterilized.

[0057] Example 5: Sensor correlations and t-SNE embedding

[0058] To ensure rationality and high-precision recognition of the designed patterns, we divided the teeth sensor array into 6 regions and the tongue sensor array into 9 regions. We calculated the crosscorrelation of each region of the sensor array during data acquisition. The 20 images acquired during the period each region was pressed were superimposed to remove random noise. The 15 tactile data matrices ultimately obtained are straightened into 15 vectors. The Pearson correlation coefficient was calculated to analyse the correlation of tactile data from different regions using the following formula:

[0059] Where X and Y are any two of the 15 vectors, and N is the length of each vector, which in this case is 784.

[0060] The 50 tactile tongue and 6 teeth data matrices were straightened into 56 vectors, and then the data dimensionality was reduced using MATLAB's built-in tsne function. The cosine distances between any two of the 56 vectors are defined as follows:

[0061] Referring to Fig. 10, the 2D t-SNE visualization of 50 tactile tongue and 6 teeth data matrices designed according to sensor partitions is depicted. Each dot represents an individual trial (30 trials shown for each of the 56 patterns).

[0062] Example 6: Data acquisition and preprocessing for pattern recognition

[0063] Four participants, including two male and two female researchers from the National University of Singapore, took part in this study. Their ages ranged from 28 to 32 years. The participants were firstly trained for about three days, with sessions lasting 2-3 hours each day. This training covered aspects such as wearing the input device, connecting the hardware, using the software, and developing proficiency in controlling the device with their teeth and tongue. The input device prototype was used to record real-time data from participant 1 , who slid 50 different patterns with the tongue, and performed 6 different patterns with the teeth, and created 356 different patterns with cooperative control of the teeth and tongue. After the participant completes each pattern, the teeth and tongue must leave the sensor array for longer than the time required to acquire a single frame of data (approximately 100 ms) before operating the next pattern. All frames from the time the sensor is pressed until the sensor is released are combined to form the final image for each pattern. The customized algorithm compares the pressed position for each frame in real time. If the pressed position is the same for multiple images collected continuously, only one image was retained to ensure the continuity of each pattern. To collect the training dataset and reduce the training time, participants performed 56 different patterns using their teeth and tongue (Extended Data Fig. 5). Each pattern was performed 100 times in different external environments (including sitting, standing, and running) with different forces. To tram the system, a dataset of 356 patterns was created by merging 50 tongue patterns with 6 tooth patterns. The training dataset for the CNN consisted of 35,600 images in total (356 patterns x] 00 images per pattern), with 30% of these images designated as the validation set ratio. In addition to the data recorded when the participants perform the patterns, we also record no-operation data for training. After training the model, the test data was collected. Participants performed 356 different patterns through coordinated control of their teeth and tongue, with each pattern executed approximately 20 times.

[0064] Example 7: Convolutional neural network training

[0065] Network training is conducted on a computer equipped with an E5-2680V3 CPU and an RTX 2080 Ti GPU. The convolutional neural network (CNN) used is a modified Resnet-18 structure compiled using the python programming language and the TensorFlow software library, with high performance in terms of accuracy and processing time (Extend data Fig. 6). The deep convolutional network first extracts the features of the input images with different patterns, and then inputs the extracted features into the fully connected network for classification. The dimension of the input feature map is 28 x 28 x 1, which is obtained by interpolating the matrix of the tactile input data of the sensor with a dimension of 12 x 14 using MATLAB software. No data augmentation was used in the process. The network initially uses 64 convolution kernels with a size of 3 x 3, and the number of convolution kernels is doubled for each subsequent block. Striding is 1 and padding is performed. Normalization is performed between the convolutional layer and the "Relu" activation layer. The fully connected layer uses the "Softmax" activation function to determine the class of the input feature map among the 356 classes.

[0066] Example 8: Predicting a character based on current / accumulated trajectory The tactile sensor array can also serve as a touch screen, enabling direct writing of characters with the tongue. Accurate recognition was achieved for 26 English characters and 8 numbers directly written by the tongue. Our trained RNN architecture predicts subsequent trajectory based on the current trajectory, facilitating character identification. Specifically, the RNN predicts the subsequent trajectory (xT+l~xN) according to the current trajectory (xl...xt...xT) of the input pattern, and then the predicted complete trajectory (yl~yN) is fed into an artifical neural network (ANN) to identify the pattern. The parameter describing the temporal trajectory of each character is the number of the central pixel pressed by the tongue in each frame. Taking number "3" as an example, the size of the sensor data matrix is interpolated into 28x28, and each pixel is numbered as Row -'28 Col (Row, Col = 1, 2, ... 28). The red curve represents the pixel number of each frame during the tongue sliding of number "3", interpolated into 30 frames. Specifically, the moments when the sensor array is pressed and 15 released are used to define the starting and ending frames for each character. Subsequently, regardless of the actual number of frames captured during the writing of each character, we interpolate all the frames to a total of 30 to match the matrix size required by the RNN network. Examples of sensor data matrix and corresponding temporal trajectories for several letters are shown in Fig. 13.

[0067] Parameter extraction of the temporal trajectories of each character

[0068] Real-time sensor array data was recorded using the input device prototype as participant 1 slid 26 English letters and 8 Arabic numerals with tongue. Similarly, after the participant operates each character, the teeth and tongue need to leave the sensor array longer than the time required for one frame of data to be collected before new operation. To obtain the temporal trajectory xt, the pixel point (Low, Col) with the minimum value in each frame image during character writing was identified and encoded as x = Low * 28 + Col. For each character, only one frame image was reserved for consecutive frames with the same xt, and the temporal trajectory vector x of each character needs to be interpolated to the 21 same length for the RNN.

[0069] Recurrent neural networks

[0070] The recurrent neural network (RNN) was based on the function SimpleRNN compiled by the python programming language and the TensorFlow software library (Extend data F ig. 9). The RNN employed share time parameters among recurrent kernels. Specifically, after the network receives the input xt at time t, the value ht of the hidden layer not only depends on xt, but also depends on ht- 1. In the designed network, the hidden layer output ht at time t is:

[0071] Where wXh and WM are the weight matrix, s is the bias, xtis the input feature at the current moment, ht-i is the state information at the previous moment, and tanh is the activation function. The output of the recurrent kernel at the current moment is: where w / iyis the weight matrix, s is the bias, and softmax is the activation function. Weights and biases are updated in real-time during backpropagation, and the loss function used is:

[0072] There are two hidden layers in the designed network, so in the first SimpleRNN function in the program, the output ht at each moment needs to be sent to the next hidden layer. The second SimpleRNN function only needs to output the yi at the last moment because it is followed by a dense layer.

[0073] Calculation method of average absolute relative deviation

[0074] We calculate the average absolute relative deviation between the predicted and real temporal trajectories using the following formula: where xtis the predicted temporal trajectory, x is the real temporal trajectory. Both vectors xtand x have a size of n. Several daily tasks have conducted to demonstrate the input device’s capabilities. These tasks included playing games, writing with the tongue, and navigating a wheelchair in confined spaces. After approximately an hour of training, users could play the game Fruit Ninja by simultaneously biting down on the left part of the teeth sensor array (representing the left mouse click) and sliding the tongue in the desired direction to control cursor movement. By utilizing the tongue sensor array as a touchscreen in combination with the RNN, users could directly write sentences with the tongue. Each character slid on the tongue sensor array corresponds to a direct click on the virtual keyboard, enabling the user to write sentences in a Word document. This approach eliminates the need for memorizing patterns and significantly reduces training time. It differs greatly from conventional interactive devices that rely on pressure sensors to transform specific patterns into limited instructions. Minor typos that may occur during sentence typing can be readily corrected using open-source word correction libraries. Moreover, we developed a wheelchair navigation system utilizing the input device, a computer for data processing and remote wheelchair control, and a custom motorized wheelchair that receives wireless instructions from the computer. By using the input device to emulate mouse function, we demonstrated flexible remote control of the wheelchair in a confined space.

[0075] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprising", “including” and / or "having" and other conjugations of these terms, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0076] The terminology used herein should not be understood as limiting, unless otherwise specified, and is for the purpose of describing embodiments only and is not intended to be limiting of the disclosed subject matter. While certain aspects of the disclosed subject matter have been illustrated and described, it will be clear that the disclosure is not limited to the aspects described herein. Numerous modifications, changes, variations, substitutions and equivalents are not precluded. It will also be appreciated by a person skilled in the art that the present disclosure is not limited to what has been particularly shown and described hereinabove.

Claims

CLAIMS1. An input device for capturing input from an oral cavity of a user comprising: a support structure configured to attach to a first jaw in the oral cavity, the first jaw having at least one tooth; a tactile sensor array attached to the support structure, the tactile sensor array having a first surface functionally disposed over at least a lingual surface of the at least one tooth.

2. The input device according to claim 1 , wherein the at least one tooth is an anterior tooth.

3. The input device according to claim 1, wherein the tactile sensor array further comprises a second surface functionally disposed over an occlusal surface of the at least one tooth.

4. The input device according to claim 1, wherein the tactile sensor array comprises a sensor matrix.

5. The input device according to claim 4, wherein the tactile sensor array further comprises an electrode array in communication with the sensor matrix.6 The input device according to claim 5, wherein the sensor matrix comprises spinous microstructures.

7. The input device according to claim 6, wherein the electrode array is in communication with the sensor matrix by way of the spinous microstructures.

8. The input device according to claim 6, wherein the sensor matrix and the spinous microstructures comprise carbon nanotubes of about 15 wt%.

9. The input device according to claim 1 , wherein the pressure input is a tongue movement comprising movement of the tongue that is transmitted to the tactile sensor array in at least substantially a first trajectory.

10. The input device according to claim 9, wherein the tactile sensor array is configured to recognize the tongue movement, the tongue movement comprising at least one tongue touch pattern.

11. The input device according to claim 10, wherein the at least one tongue touch pattern corresponds to predetermined regions on the tactile sensor array.

12. The input device according to claim 11 , wherein the pressure input is a combination of the tongue movement and at least one occlusal pattern.

13. The input device according to claim 12, wherein the pressure input corresponds to a character, wherein the character or numeral is predicted based on at least the first trajectory in a time-dependent manner.

14. The input device according to claim 12, wherein the combination of the tongue movement and at least one occlusal pattern occur simultaneously.

15. The input device according to claim 12, wherein the tongue movement further comprises at least one tongue movement pattern and wherein each of the at least one tongue movement pattern comprises at least one of the at least one tongue touch pattern.

16. The input device according to claim 15, wherein the at least one tongue movement pattern is derived based on their statistical correlation between each of the at least one tongue touch pattern and each of the at least one occlusal pattern.

17. The input device according to claim 1, wherein the support structure is shaped to fit a contour of the first jaw.

18. The input device according to claim 1, wherein the pressure input further comprises intraoral movement, the intraoral movement comprising at least movement of orthodontics and respiration.

Images