Scalable sensor arrays through row column compressive sensing

Abstract

Compressive sensing is used to create a novel sensor matrix for the device and method of the present invention. Conventionally, sensor arrays have required individual connections to each sensing pixel of the array. By manipulating only, the rows and columns of the sensor matrix of the present invention, the system 5 and method of the present invention allows each measurement to include data from multiple sensors. This method is also compatible with existing sensor matrices and does not require modifying the design of existing sensor arrays and only requires the use of new interfacing electronics. As the size of the sensor gets very large, the number of measurements using this method becomes very 10 small compared to sampling each sensor. Therefore, the present invention includes novel sensing devices as well as the method of implementing compressive sensing and recovery on existing devices.

Description

SCALABLE SENSOR ARRAYS THROUGH ROW COLUMN COMPRESSIVE SENSINGGOVERNMENT RIGHTS

[0001] This invention was made with government support under grant number W81XWH-20-1-0842. awarded by the Department of Defense. The government has certain rights in the invention.CROSS REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 653.517 filed on May 30. 2024, which is incorporated by reference, herein, in its entirety.FIELD OF THE INVENTION

[0003] The present invention relates generally to sensors. More particularly the present invention relates to sensor arrays that include row column compressive sensing and sensor data reconstruction.BACKGROUND OF THE INVENTION

[0004] Large artificial tactile sensor arrays use synchronous, clock-based transmission schemes that rely on time-division multiple access (TDMA). These TDMA-based transmission schemes utilize rapid raster scanning of the sensor array, requiring a controller to individually address each tactile sensing ‘pixel’ (taxel) in the sensor array before proceeding to the next taxel.

[0005] They are conventionally arranged in a matrix of rows and columns so individual sensors can be row-column addressed (raster-scanned). Row-column addressed matrixsensors use substantially less connections, which can be further reduced with multiplexing, but the sensors must be individually scanned to avoid cross talk.

[0006] Because of this raster-scanned process, TDMA interfaces suffer from scalability limitations, such as increasing wiring complexity and latency that worsens with an increasing number of matrix sensor arrays. Large matrix sensor arrays in robotics with over 100 sensors rarely have scanning rates higher than 40 Hz. Flexible tactile sensor arrays with numerous sensing elements have significant wiring complexity and sensor sampling rates on the order of ~10 Hz.

[0007] To mitigate this problem, several groups have developed asynchronous tactile sensor arrays with neural-inspired multiplexing. For example, the CellulARSkin. asynchronously coded electronic skin, and RFIDHand, all enable asynchronous transmission of tactile events by integrating ID generating electronics inside each taxel of the tactile sensor. However, the integration of electronics into each taxel significantly limits the density and flexibility of the sensor array and results in significant readout complexity.

[0008] It would therefore be advantageous to provide an improved design for scalable, sensor matrices.SUMMARY OF THE INVENTION

[0009] In accordance with an embodiment, the present invention provides a system including a sensing array. The sensing array includes a number of sensors, and those sensors can be arranged in an array .arranged in m rows and n columns. The system also includes a processing device programmed for receiving data from the sensing matrix; and processing thedata from the sensing matrix, such that each measurement includes data from multiple sensors.

[0010] In accordance with an aspect of the present invention, the processing device is programmed to apply one of a weighted row summation method, although the weighting does not have strictly be along rows, and generally involves summing the activity of numerous sensors where each sensor contributes to the sum with a particular weight or bias. The processing device is programmed for calculating an analog driving voltage, which is the weighing or bias voltage, and performing row-wise summation. The processing device is further programmed for executing a recovery algorithm. The processing device uses programmable mixing, such that each measurement includes data from more than one sensor. The sensor array can take the form of any sensing surface.

[0011] In accordance with another aspect of the present invention, a method for compressive sensing includes applying an analog driving voltage across rows and columns of a sensing matrix. The method includes compressing data across the rows and columns of the sensing matrix using weighted row summations and processing the compressed data. The method also may include outputting the processed data onto a single or a few output wires.

[0012] In accordance with another aspect of the present invention, the method includes applying one of a weighted row summation method and a dual tone multi frequency method to the compressed data. The method includes applying a recovery algorithm. The recovery algorithm can include a sparse recovery algorithm.

[0013] In accordance with another aspect of the present invention, a device for processing sensor data includes a sensor interface. The sensor interface is configured to receive data from a sensing matrix. A processing device is programmed for receiving data from thesensing matrix and processing the data from the sensing matrix, such that each measurement includes data from multiple sensors and an algorithm is used to reconstruct the full sensor data.BRIEF DESCRIPTION OF THE FIGURES

[0014] FIG. 1 illustrates a schematic diagram of a control module configured for programmable mixing, according to an embodiment of the present invention.

[0015] FIG. 2 illustrates a schematic diagram of a 3x3 sensor matrix implementing the described method using analog driving voltages, row-wise summations, and frequency multiplexing, according to an embodiment of the present invention.

[0016] FIG. 3 illustrates a perspective view of the sensor matrix with rows and columns and the phase shift oscillators at the perimeters, with each row and column having an independent frequency of at least 2kHz apart, according to an embodiment of the present invention.

[0017] FIG. 4 illustrates a schematic diagram of tactile DTMF system, according to an embodiment of the present invention.

[0018] FIG. 5 illustrates a perspective view of a sensor matrix incorporated on a Taska Hand device, with sensors positioned on the palm and on each fingertip, according to an embodiment of the present invention.

[0019] FIG. 6 illustrates a schematic diagram of an overview of an embodiment of the technology in the form of a high-speed high-density tactile sensor array for a prosthetic handwith frequency multiplexing, binary driving voltages (all set to 1), and frequency multiplexing, according to an embodiment of the present invention.DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The presently disclosed subject matter now will be described more fully hereinafter with reference to the accompanying Drawings, in which some, but not all embodiments of the inventions are shown. Like numbers refer to like elements throughout. The presently- disclosed subject matter may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Indeed, many modifications and other embodiments of the presently disclosed subject matter set forth herein will come to mind to one skilled in the art to which the presently disclosed subject matter pertains having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the presently disclosed subject matter is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims.

[0021] The fast readout of large sensor matrices in rows and columns has been an inherently difficult problem because of cross talk errors between rows and columns. Cross talk can be used as an advantage through the use of sparse recovery algorithms and knowledge of the driving signals behind cross-talking rows or columns. In the present invention, each measurement made by the system, contains data for more than one sensor.

[0022] Therefore, in one exemplary implementation of the present invention, given a sensor matrix with M rows, N columns, and M*N sensors an analog or signed or unsigned, binary voltage is provided to each row of a sensor matrix, measure the summation of activityalong each column, and repeat the method, possibly swapping the roles of rows and columns, until the desired level of measurements are taken. This method produces at least M or N measurements and can be iterated until M*N measurements are taken or the desired number of measurements are taken. Sparse recovery7algorithms are then used to reconstruct the activity7over the entire array of sensors.

[0023] In another exemplary' implementation of the present invention, a dual tone multi frequency (DTMF), M*N tactile sensor matrix and method can be used provides detection of touch events on a two-dimensional manifold and have the capability for multiple simultaneous touch events to be detected and distinguished from each other in a single measurement.

[0024] The method of the present invention is a new example of compressive sensing for a sensor matrix, which has conventionally' required individual connections to each sensing pixel of the array. By manipulating only, the rows and columns of the sensor matrix, this method is compatible with existing sensor matrices and does not require modifying the design of existing sensor arrays and only requires the use of new interfacing electronics. As the size of the sensor gets very large, the number of measurements using this method becomes very small compared to sampling each sensor.

[0025] The present invention provides hybrid solution that doesn't require internally modifying the sensor with electronics while also significantly improving the latency of the normal sensor matrix using a new simple method of compressive sensing that is compatible with existing sensor matrices. Consequently, the present invention allows for each measurement to include data from more than one sensor with programmable mixing. described further, herein. There are a number of methods to achieve data from multiple sensors in each measurement. While a number of exemplary’ methods are detailed herein, itshould be understood that these methods are included to further illustrate the invention and are not meant to be considered limiting. Any modification to the methods discussed herein known to or conceivable to one of skill in the art are included. Additionally, any additional methodologies known to or conceivable to one of skill in the art to achieve the outcome of data from multiple sensors in each measurement, are also included herein.

[0026] FIG. 1 illustrates a schematic diagram of a control module configured for programmable mixing, according to an embodiment of the present invention. A bias voltage or current is supplied to each sensor being measured by a programmable current or voltage source like a digital-to-analog converter (DAC). A summer (SUM) is used to add up the responses of the sensors being measured for example by using a summing circuit, The summed output is measured for example by using an analog-to-digital converter (ADC). A recovery algorithm such as sparse recovery is used to unmix the measurements based on the summed measurement and the known mixing conditions.

[0027] FIG. 2 illustrates a schematic diagram of a 3x3 sensor matrix implementing the described method using analog driving voltages, row-wise summations, and frequency multiplexing, according to an embodiment of the present invention. A 3 x3 sensor matrix 10 includes dark grey lines 12 that represent the row and column traces of the sensor array, the light grey areas 14 within the dark grey lines represent the sensing material (piezoresistive in this drawing). The sensing material can take the form of a specially designed piezo material that allows for equal contribution. Each A; produces an analog voltage along the columns of the sensor matrix. To not require any column addressing, each Ai will be preloaded with a list of analog weights that will be cycled with the CNTRL connection. Each S block represents summations on the connected rows. The summations control the gain of each variable gainoscillator f which allows the measurements to frequency multiplex onto a single common output wire. In this configuration only 4 wires are required to interface with the entire sensor array and arrays can be easily extended in size or daisy -chained together.

[0028] The basic elements of a tactile sensor according to the present invention, and its principle of operation may be understood by reference to FIG.2. FIG. 2 shows the schematic of a 3x3 sensor matrix 10 implementing the described method using analog driving voltages, row-wise summations, and frequency multiplexing. Each Ai produces an analog voltage along the columns of the sensor matrix. To avoid any column addressing, each Ai will be preloaded with a list of analog weights that will be cycled with the CNTRL connection. Each S block represents summations on the connected rows. The summations control the gain of each variable gain oscillator fi which allows the measurements to frequency multiplex onto a single common output wire. In this configuration only 4 wires over 16 and 18 combined are required to interface with the entire sensor array and arrays can be easily extended in size or daisy-chained together.

[0029] In an exemplary implementation of the present invention, a dual tone multi frequency (DTMF), M*N tactile sensor matrix and method can be used provides detection of touch events on a two-dimensional manifold and have the capability for multiple simultaneous touch events to be detected and distinguished from each other in a single measurement. In an exemplary embodiment, touch events may be detected and processed with very high-density, high-speed and low latency through compressed sensing that only incorporates few measurements which greatly facilitates increase in readout speed of sensor arrays while minimizing wiring and preserving resolution.

[0030] In another exemplar}' implementation of the present invention, weighted row summations are used to generate measurements containing data from more than one sensor. The method of weighted row sums works as follows: apply an analog voltage to each column of a sensor array and sum the activity on each row; then repeat the process until the desired number of measurements is taken. Compressive sensing by weighted row summation (CSWRS) can be theoretically applied to any sensor matrix, and therefore is generically about revolutionizing the sampling of the sensor matrix to require less measurements without modifying any of the sensor matrix design. The method of the present invention is extremely practical because the number of measurements can be adjusted, and if a single column is activated at a time this method reduces back to the traditional raster-scan. This allows for high-speed compressive sensing when data is simple, but also allows for low-speed fullsampling when interactions are complex.

[0031] FIG. 3 illustrates a perspective view of the sensor matrix with rows and columns and the phase shift oscillators at the perimeters, with each row and column having an independent frequency of at least 2kHz apart, according to an embodiment of the present invention. FIG. 3 shows the 3x3 sensor matrix 10 and the phase shift oscillators 20 at the perimeters. Each row and column is designed to have an independent carrier wave frequency 4 of at least 2kHz apart. The earner wave frequency is generated by a simple phase shift oscillator. The components on the flex PCB 10 are powered with three wires. + / - 3.3V and grounds 22, 24.

[0032] FIG. 4 illustrates a schematic diagram of a tactile system, according to an embodiment of the present invention. Each row and column conductive traces contain the full amplitude carrier waves and a piezoresistive layer amplitude modulates these carrier waves to the central conductive layer as pressure is applied to the sensor array. The modulated carrier waves are combined onto a single wire and a summing amplifier is used to combine the signals without interference. FIG. 4 show the different carrier wave frequencies at the perimeter of the rows and columns 10 and a single- wire frequency multiplexed output readout 26. The singles wire is connected to a summing amplifier 28 that sums the frequencies at the input of the amplifier 26 which gives a signal of all summed frequencies 30. Therefore, each measurement includes data from more than one sensor. The tactile system of FIG. 4 can be coupled with CSWRS, DTMF, or any other processing method known to or conceivable to one of skill in the art.

[0033] FIG. 5 illustrates a perspective view of a sensor matrix incorporated on a Taska Hand device, with sensors positioned on the palm and on each fingertip, according to an embodiment of the present invention. FIG. 5 Referring now to FIG. 5, an embodiment of the Taska hand 100 incorporates the high-density and high-speed tactile sensors flex PCB 102 cut to the size of the palm and the fingertips. The flex PCB 102 on the fingertips and the palm flex PCB are connected to each other via wires 104, 106 while still ensuring the rows and columns movements.

[0034] FIG. 6 illustrates a schematic diagram of an overview of an embodiment of the technology7in the form of a high-speed high-density tactile sensor array for a prosthetic hand with frequency multiplexing, binary7driving voltages (all set to 1), and frequency multiplexing, according to an embodiment of the present invention. The schematic diagramof FIG. 6 shows the original applied pressure to the hand in (1), how summations are made in rows and columns in (2), how the summations are encoded onto 1 wire through frequency multiplexing in (3), how Fourier analysis can be used to demodulate the measurements in (4), how the sensor activity' can be reconstructed using sparse recovery' algorithms in (5), and how the final data reconstruction is achieved in (6).

[0035] FIG. 6 further provides an overview of the concept with reconstruction of the sensor array data. The graph in (1) shows Taxel numbers versus the applied pressure overtime. It show s the compressive sampling and multi measurement recovery of the applied pressure. The graph in (2) shows the aDTMF sensor sums the pressure over rows and columns of array. The graph depicts the aDTMF signaling scheme with the effect of timewindow' and stimulus complexity' on reconstruction accuracy for 32x32 matrix. The graph in (3) show s the voltage versus time graph which is the high density' sensorization of the Taska Prosthetic Hand 3. The summations are encoded as a mixed frequency output on a single wire 8. The mixed frequency output is sampled using a 2-MSPS ADC. Goertzel algorithm is used to compute the magnitude of each frequency which is a key component in determining the sensors that are actively touched. The graph at (4) show s the results of the Goertzel algorithm. (5) show s the multi -measurement sparse recovery algorithm used to solve for sparse applied pressure. (6) shows the final reconstructed pressure over time depicting the data that is transformed to the applied pressure. Sparse recovery' is used to reconstruct the signal over the entire sensor array.

[0036] Further, FIG. 6 illustrates a novel device and method for touch sensing. The present invention provides compression of data over row s and columns. The sensor arrays use a multiplexing sensor for frequency encoding on the perimeter of the array. This data is thentransmittable over a single wire. Each sensor in the sensor array has two frequencies behind it. It should be noted that the novel methods of compression also necessitates a novel method of recovery. One such method of recovery is the modified sparse recovery' algorithm described above.

[0037] The benefit of this method is that as the size of the sensor gets very large, the number of measurements using this method becomes very small compared to sampling each sensor. Another advantage of this method is that the degree of measurements can be dynamically adjusted. For example, when stimuli are simple, they may require a few measurements but when stimuli are complex more measurements can be taken. Furthermore, if complete certainty is required in the sensing application, the method can still produce raster scanning by setting all but 1 row or column at a time to a driving voltage of 0. This method is also readily accessible to integrate multiplexing schemes which can be used to reduce the required wiring for each row and column of the array, for example using frequency, code, or time multiplexing to address each row / column. Other techniques have been developed to reduce the number of measurements required for sampling a sensor array, however they have called for modifying the sensor design by incorporating additional electronics into the sensor or have called for doing sub-sampled raster scanning (measuring subsets of the pixels). This method does not require augmentation of the sensor array, and algorithmic methods such as sub-sampling can be incorporated into this method still.

[0038] The sensor array of utilizing compressed sensing and spare recover}' can be implemented in various applications as follows:

[0039] A preferred embodiment of this technology is in robots, for example for the sensorization of robotic hands. The construction of a prosthetic device needs various important factors to be considered such as to streamline assembly and use, tactile sensor integration, management of heaps of w ires, activity level of the sensors, general prosthesis structure and components, ensuring patient comfort and ease of movement and interaction of objects with the prosthetic hand. This disclosure discusses the high-density and high-speed aDTMF tactile sensors for a coordinated prosthetic device.

[0040] However, it should be noted that the invention is not limited to prosthesis and has applications in the following domains. Applications to Tactile sensor arrays include:• Vibration feedback through motors at different locations on the amputee’s arm to enhance user experience on sense of touch when the matrix of high-density and high-speed tactile sensors on the prosthetic hand is used to perform a task, for example, grasping a cup.• Haptic feedback: It can be used to provide targeted feedback or electrical stimulation with various existing or proprietary haptic feedback technologies.• Robotic Grippers: It can be used in robot grippers in different industrial settings to grasp and manipulate objects by providing feedback, allowing the robot to adjust its grip strength and avoid damaging the object.Safety and Monitoring: Devices equipped with tactile sensing can monitor forces and pressures that could indicate a potential problem. For example, in industrial settings, these sensors can detect when equipment is subjected to forces beyond its design limits.Flexible and Scalable: They can be attached to various surfaces and objects, making integration into different systems easier.• In manufacturing and industrial settings, it can be used to monitor the quality of products by detecting variations in surface texture or consistency.• It can be used in wearable devices to provide alerts or notifications through vibration or touch-based feedback.• It can be used in smartphones, tablets or laptops to increase the tactile sensing speed of the screen and improve user experience.• Another application is in medical surgical robots to achieve tactile feedback to assist surgeons during minimally invasive procedures, enhancing their precision and reducing the risk of tissue damage.• The high-density and high-speed tactile sensors can be essential in medical training simulators. They replicate the feel of various medical procedures and help trainees develop the necessary skills and sensitivity for tasks like catheter insertion, or surgery.• Augmented reality gloves wi th the matrix of high-density and high-speed tactile sensors can enable users to interact with virtual objects as if they were real.• AR / VR: One of the challenges of creating a truly immersive AR or VR experience is the lack of physical feedback. With the sensor matrix, users could feel virtual objects, making the experience more realistic and engaging. This could be a game-changer for industries like gaming, education, and training simulations.The sensor matrix can be used to encode / develop a database of objects to be used for touch feedback in AR / VR.• Tactile sensors in wearables can monitor and provide feedback on posture and exercise form. • It can be used in pressure mattresses in healthcare settings to prevent pressure ulcers by monitoring and adjusting support based on a patient’s position and pressure points.• It can be used in exoskeletons to enhance mobility, balance, and safety by detecting terrain and adjusting support accordingly.• It can be integrated into assistive technologies to provide tactile feedback to individuals with visual impairments by using the matrix of sensors to reconstruct the Braille display touched and provide feedback through audio of the Braille messages.• It can be used in remote-controlled devices and teleoperation systems to convey tactile information to operators. This is especially important in applications like remote surgery or bomb disposals. • It can be used for a number of military, government, and national security applications.• It can be used for Fine texture perception.• It can be used in a pressure sensing insole• Can be used in wearable pressure sensors• Can be used in pressure sensing surfaces

[0041] Applications of non-tactile sensor arrays:• electrode sensor arrays such as the UTAH arrays.• Imaging sensor array that can utilize CMOS cameras for imaging purposes.• Audio sensing

[0042] It should be noted that the communications protocols described herein can be executed with a program(s) fixed on one or more non-transitory computer readable medium. The non-transitory computer readable medium can be loaded onto a computing device, server, imaging device processor, smartphone, tablet, phablet, or any other suitable device known to or conceivable by one of skill in the art.

[0043] It should also be noted that herein the steps of the method described can be carried out using a computer, non-transitory computer readable medium, or alternately a computing device, microprocessor, or other computer type device independent of or incorporated with the present invention. An independent computing device can be networked together with the device either with wires or wirelessly. Indeed, any suitable method of analysis known to or conceivable by one of skill in the art could be used. It should also be noted that while specific equations are detailed herein, variations on these equations can also be derived, and this application includes any such equation known to or conceivable by one of skill in the art.

[0044] A non-transitory computer readable medium is understood to mean any article of manufacture that can be read by a computer. Such non-transitory computer readable media includes, but is not limited to, magnetic media, such as a floppy disk, flexible disk, hard disk, reel-to-reel tape, cartridge tape, cassette tape or cards, optical media such as CD-ROM,writable compact disc, magneto-optical media in disc, tape or card form, and paper media, such as punched cards and paper tape.

[0045] Although the present invention has been described in connection with preferred embodiments thereof, it will be appreciated by those skilled in the art that additions, deletions, modifications, and substitutions not specifically described may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

CLAIMS1. A system comprising: a sensing array, wherein the sensing array comprises a number of sensors; a processing device programmed for receiving data from the sensing array; and wherein the processing device is further programmed for processing the data from the sensing array, such that each measurement includes data from multiple sensors.

2. The system of claim 1 wherein the sensing array takes the form of a sensing matrix.

3. The system of claim 1 comprising the processing device being programmed to apply a weighted row summation method.

4. The system of claim 3 wherein the weighted row summation is encoded as a mixed frequency output on a single wire.

5. The system of claim 2 wherein the processing device is programmed for calculating an analog driving voltage based on a desired sensing matrix.

6. The system of claim 1 wherein the processing device performs summations of numerous sensing pixels that have been biased by the analog driving voltage.

7. The system of claim 1 wherein the processing device is further programmed for executing a recovery algorithm.

8. The system of claim 2 wherein the sensing matrix takes the form of a sensing surface.

9. The system of claim 1 wherein the processing device uses programmable mixing, such that each measurement includes data from more than one sensor.

10. The system of claim 2 wherein the sensing matrix is configured to encode a database of objects to be used for touch feedback in AR / VR.

11. The system of claim 2 wherein the sensing matrix is configured to enable users to interact with virtual objects, as if the virtual objects were real.

12. A method for compressive sensing comprising: applying an analog driving voltage across rows and columns of a sensing matrix; compressing data across the rows and columns of the sensing matrix using the mixing method.

13. The method of claim 12 comprising applying one of a weighted row summation method and a dual tone multi frequency method to the compressed data.

14. The method of claim 12 further comprising applying a recovery algorithm.

15. The method of claim 14 wherein the recovery algorithm comprises a sparse recovery algorithm.

16. The method of claim 12 further comprising applying programmable mixing, such that each measurement includes data from more than one sensor.

17. The method of claim 12 wherein the sensing matrix is configured for encoding a database of obj ects to be used for touch feedback in AR / VR.

18. The method of claim 12 wherein the sensing matrix is configured for enabling users to interact with virtual objects, as if the virtual objects were real.

19. The method of claim 12 applying a weighted row summation method.

20. The method of claim 12 further comprising calculating an analog driving voltage based on a desired sensing matrix.

Images