Tracking and analysis of braille reading

Abstract

A method of tracking and analyzing tactile includes, receiving video data from a camera arranged to image at least one finger of a reader and to image a document encoded with a plurality of tactile symbols while being read by the reader by tactile sensing; segmenting the at least one finger using the video data; segmenting at least locations of the plurality of tactile symbols within the document; tracking the at least one finger relative to the tactile symbol locations while the document is being read by the reader to provide tracking data; and processing the tracking data to provide at least one measure of tactile reading ability.

Description

TRACKING AND ANALYSISOF BRAILLE READINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present patent application claims priority benefit to U.S. Provisional Patent Application No. 63 / 669,581, filed on July 10, 2024, the entire content of which is incorporated herein by reference. All references cited anywhere in this specification, including the Background and Detailed Description sections, are incorporated by reference as if each had been individually incorporated.BACKGROUND1. Technical Field

[0002] The currently claimed embodiments of the present invention relate to systems and methods related to Braille reading, and more particularly to systems and methods for tracking and analysis of Braille reading.2. Discussion of Related Art

[0003] Tactile reading of Braille — a primary' modality through which blind individuals access written material — holds unique promise for the study of language processing (e.g., Millar, 2003). As in reading of print, and unlike ordinary comprehension of speech or sign, Braille reading involves (implicit) control over the order and rate at which linguistic material enters peripheral processing. This allows reading time (RT) to serve as a dependent measure of processing operations and costs, playing the same role for tactile reading that it does for eyetracking and self-paced reading of print. However, the fingers move smoothly over the Braille page, often touching each character of a word, rather than making discrete saccades to a relatively small number of locations in a text. Relatedly, the tactile modality has no direct counterpart to parafoveal processing found in reading by eye: while multiple fingers on both hands may be used in Braille reading, linguistic material that has not yet been directly touched is perceptually inaccessible. In these respects, Braille reading is more similar to processing of speech or sign, where the incoming signal evolves smoothly in time and access to upcoming material is limited. Indeed, because Braille text is rendered with isolated and stereotypical cells free of anticipatory' coarticulation, tactile reading could provide one of the most precise windows into predictive and minimal unit-by-unit processing of linguistic material. There thus remains a need for improved systems and methods for tracking and analysis of Braille reading.SUMMARY

[0004] A method of tracking and analyzing tactile reading according to an embodiment of the current invention includes, receiving video data from a camera arranged to image at least one finger of a reader and to image a document encoded with a plurality of tactile symbols while being read by the reader by tactile sensing; segmenting the at least one finger using the video data; segmenting at least locations of the plurality' of tactile symbols within the document; tracking the at least one finger relative to the tactile symbol locations while the document is being read by the reader to provide tracking data; and processing the tracking data to provide at least one measure of tactile reading ability.

[0005] According to another embodiment of the current invention, non-transient computerexecutable code, which when executed by a computer, causes the computer to perform the method of any one of the embodiments of the current invention.

[0006] A computer according to an embodiment of the current invention includes nontransient computer-executable code, which when executed by the computer causes the computer to perform the method of any one of the embodiments of the current invention.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the present invention, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention.

[0008] FIG. 1 shows left (green) and right (purple) index-finger locations for a single clip (one participant’s reading of a single embossed page) with squares indicating non-blank Braille cells for an embodiment of the current invention.

[0009] FIG. 2 and FIG. 3 show, according to an embodiment of the current invention, a strong positive relationship with length (r = 0.77) and negative relationship with frequency (r = -0.62) for word types averaged over participants and tokens; the relationship with surprisal for word tokens, averaged over participants, was also strong (r = 0.64. all / is < .001).

[0010] FIG. 4 shows, according to an embodiment of the current invention, that there were significant negative effects of frequency (-0.026, t = -35.82) and position (-0.012. t = -18.85). and significant positive effect of complexity (0.012, t = 7.17) and surprisal (0.001, t = 2.4).DETAILED DESCRIPTION

[0011] Some embodiments of the current invention are discussed in detail below. In describing embodiments, specific terminology is employed for the sake of clarity. However, the invention is not intended to be limited to the specific terminology so selected. A person skilled in the relevant art will recognize that other equivalent components can be employed, and other methods developed, without departing from the broad concepts of the present invention. All references cited anywhere in this specification are incorporated by reference as if each had been individually incorporated.

[0012] Reading Braille through the tactile modality is a primary way in which blind adults and children access written language. Some embodiments of the current invention provides a low- cost, widely-applicable, and primarily software-based method of (i) tracking finger movement in recorded videos of Braille reading, (ii) mapping the positions of the fingers to individual words and characters (or ’cells') in each frame of video (approx, every 10-20 milliseconds), and (iii) calculating a range of statistical measures that are relevant for assessing the cognitive mechanisms involved in reading and comprehending braille text. The inputs to the system can be single-page overhead videos of an individual reading Braille. In some experiments we have found videos from standard cellphones to be of sufficient quality for this purpose - together with basic information about the text on each page (e.g., a standard BRF [Braille Ready Format] file). The fingers of both hands are then tracked using an open-source computer vision system (e.g., MediaPipe from Google). The software developed here automatically maps the tracked finger locations to the text content for the purpose of calculating coarse-grained measures of reading - such as speed in words-per- minute and hand movement patterns (e.g., left hand marks / parallel / split / scissors) — as well as fine-grained effects of word frequency, contextual prediction, and other linguistic properties. This information can be presented in static and dynamic visual figures, as well as in tabular form for subsequent analysis. The result is a permanent, detailed, and individualized record of reading that can be used to assess Braille literacy at many levels oflinguistic analysis, both longitudinally and cross-sectionally, as well as to develop instructional materials, techniques, and interventions.

[0013] Braille reading provides critical access to written language for children and adults who are blind or have low vision. However, the availability of TVIs (teachers of students with visual impairments) and other experts in Braille literacy instruction is highly limited, and there is currently no scalable method for providing them with fine-grained quantitative metrics of individual Braille reading skill. Some embodiments of the current invention can provide such a method. For example, it can be applied in any school district, classroom, learning center, or even home to provide a wealth of quantitative information that is relevant for evaluating the reading ability of learners, standardizing or validating teaching materials, designing targeted teaching interventions, and related purposes. It can make Braille instruction more accessible and evidencebased: a paraprofessional, classroom aide, or parent could video record a session of braille reading, send the video to be processed by the software, after which the resulting graphical and statistical information could be evaluated by a TV1 who may be in another school district or state (e.g., a TVI located in Boston could evaluate Braille reading performance of children living in remote Alaskan towns), and an accumulation of such results could inform TVI practice and training. Currently TVIs must be present (physically or virtually) in order to assess Braille reading; they typically work itinerantly with at most a few hours per week to spend with each child, and they must perform their work in the moment — that is, without permanent session recordings or derived statistics that can be reviewed and tracked for progress.

[0014] A product according to an embodiment of the current invention can take the form of a software system (e.g., accessible by a web portal) through which TVIs and others would be able to submit videos of Braille reading and receive back detailed, individualized measures of performance (e.g., in a dashboard). The information provided to them would include graphical display of finger tracks over individual pages, as well as metrics of the time spent reading particular units of language (e.g., words and characters), and the effect of linguistic propertiessuch as frequency, predictability, and complexity7on those metrics. A secondary7product can be a series of standardized texts for Braille reading assessment using the invention (though the method can be applied to any text of interest), along with instructions about proper procedures for video recording. The method is not limited to prose text: it could be straightforwardly extended to reading of math equations, computer code, scientific figures, tactile graphics, and other material relevant to STEAM education. The same system could also be used by psychologists, linguists, and other researchers who study reading — in the same way7that eye-tracking technology is currently used to study reading by sighted individuals — though that is not its primary purpose.

[0015] More generally, a method of tracking and analyzing tactile reading according to an embodiment of the current invention includes receiving video data from a camera arranged to image at least one finger of a reader and to image a document encoded with a plurality of tactile symbols while being read by the reader by tactile sensing; segmenting the at least one finger using the video data; segmenting at least locations of the plurality of tactile symbols within the document; tracking the at least one finger relative to the tactile symbol locations while the document is being read by the reader to provide tracking data; and processing the tracking data to provide at least one measure of tactile reading ability7.

[0016] The terms ‘'tactile symbol” or '‘tactile reading” are intended to have broad meanings to include symbols for expressing written text as well as other content such as, but not limited to, mathematical equations, charts, diagrams, schematic illustrations, flow charts, pictures, etc. and the “reading” thereof.

[0017] In an embodiment, the document has the plurality of tactile symbols arranged in a standard pattern of sizes and separations such that the segmenting at least tactile symbol locations of the document is performed by first segmenting the document as a whole followed by assigning tactile symbol locations within the document.

[0018] In an embodiment, the tactile symbol is assigned to each symbol location and the processing of the tracking data to provide at least one measure of tactile reading ability uses the tactile symbols. In an embodiment, the tactile symbol represents, either alone or group together, components of written text including at least one of letters, words, or punctuation. In some embodiments, the method further includes mapping the tracking data, the at least each one of letters, words, or punctuation.

[0019] In some embodiments, the document encoded with said plurality of tactile symbols is encoded with Braille symbols.

[0020] In some embodiments, the receiving video data from the camera is arranged to image substantially an entire hand of the reader, the segmenting the video data segments the entire hand and a plurality of fingers on the entire hand, and the tracking tracks the plurality of fingers on the entire hand to provide the tracking data.

[0021] In some embodiments, the receiving video data from the camera is arranged to image substantially an entirety of both hands of the reader, the segmenting the video data segments both hands and a plurality of fingers on each of both hands, and the tracking tracks the plurality of fingers on both hands to provide the tracking data.

[0022] In some embodiments, the at least one measure of tactile reading ability' includes at least one of document total reading time, words per minute, time for reverse motions for repeat reading, reading times per word category, reverse times per word category.

[0023] In some embodiments, the method further includes displaying at least one of the tracking data or the at least one measure of tactile reading ability for use by at least one of the reader or an observer.

[0024] In some embodiments, the method further includes recording at least one of the tracking data or the at least one measure of tactile reading ability for later use.

[0025] Some embodiments are directed to non-transient computer-executable code, which when executed by a computer causes the computer to perform a method of any one of the embodiments of the current invention.

[0026] Some embodiments are directed to a computer that has non-transient computerexecutable code, which when executed by the computer causes the computer to perform a method of any one of the embodiments of the current invention.

[0027] The following describes some examples according to embodiments of the current invention to help further describe some concepts of embodiments of the invention. However, the general concepts of the current invention are not limited to these examples.

[0028] According to some embodiments of the current invention, we provide benchmark findings on contextual, lexical, and sub-lexical effects in a densely sampled data set of naturalistic Braille reading. While some of our results replicate findings of Millar (2003) and parallel studies of print reading, our low-cost and automatic method of tracking Braille reading is original and we give novel evidence bearing on character-level effects.

[0029] Data. Adult congenitally-blind individuals ( V=9) read six passages from the Natural Stories Corpus (Futrell et al., 2020) embossed in Contracted Unified English braille (UEB). Hand movements were video recorded from above at approximately 60fps with a consumer cellphone. In each frame, the locations of fingers on both hands were identified using MediaPipe (Lugaresi et al, 2019), placed in standard text coordinates by automatic affine transformation (Thevenaz et al, 1998), and mapped to the closest Braille cells. Subsequent to this mapping, a finger- specific RT for each cell and word token was calculated by summing frames and converting to milliseconds. FIG. 1 shows left (green) and right (purple) index-finger locations for a single clip (one participant’s reading of a single embossed page) with squares indicating nonblank Braille cells. In total, the data set contained more than 3 million tracked index-fingerlocations (> 110,000 per participant hand) and RTs for approximately 178,000 cell tokens (> 19,000 per participant) and 50,000 word tokens (> 5,000 per participant).

[0030] Analysis. We first examined whether three major effects repeatedly found in eyetracking studies of print reading — word length (here, number of Braille cells), word frequency (taken from SUBTLEX-US), and contextual word predictably (surprisals provided with the Natural Stories Corpus) — had expected effects on word RTs. FIGS. 2 and 3 show a strong positive relationship with length (r = 0.77) and negative relationship with frequency (r = -0.62) for word types averaged over participants and tokens; the relationship with surprisal for word tokens, averaged over participants, was also strong (r = 0.64, all ps < .001). Mixed-effects analyses were performed at the level of word tokens using lme4 (Bates, 2015) with centered predictors, log RT summed over index fingers as the outcome, and random intercepts for participants and words. There were significant effects of length (0.076, t = 42.84), frequency (- 0.044, t = -12.32), and surprisal (0.006, t = 9.7). These were in the same direction and significant (| 1 1 > 2) for all but one of the participants analyzed separately. We further analyzed the log RTs of character (Braille cell) tokens as a function of their containing word frequency, position within the word (Millar, 2003), complexity (number of raised dots), and cohort surprisal (e.g., Brodbeck, 2018). There were significant negative effects of frequency (-0.026, t = -35.82) and position (-0.012, t = -18.85). and significant positive effect of complexity (0.012. t = TAI') and surprisal (0.001, t = 2.4; see FIG. 4). These results suggest that cells are processed predictively, as information about their containing words unfold, and that detection of raised dots is a perceptual bottleneck.References

[0031] Bates, D.. Maehler. M., Bolker, B.. & Walker, S. (2015). Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1), 1-48. • Brodbeck, C., Hong, L. E., & Simon, J. Z. (2018). Rapid transformation from auditory to linguistic representations of continuous speech. Current Biology, 28(24), 3976-3983. • Futrell, R., Gibson, E., Tily, H. J., Blank, I.,Vishnevetsky, A., Piantadosi, S. T., & Fedorenko, E. (2021). TheNatural Stories corpus: areading- time corpus of English texts containing rare syntactic constructions. Language Resources and Evaluation, 55. 63-77. • Millar, S. (2003). Reading by Touch. New York: Routledge. • Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Flays, M., Zhang, F., Chang, C.-L., Yong, M. G., Lee, J., Chang, W.-T., Hua, W., Georg, M., and Grundmann, M. (2019). MediaPipe: A framework for building perception pipelines. arXiv: 1906.08172 [cs]. • Thevenaz, P., Ruttimann, U.E.. & Unser. M. (1998). A pyramid approach to subpixel registration based on intensity. IEEE Transactions on Image Processing, 7(1), 27-41.

[0032] While various embodiments of the present invention have been described above, they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present invention should not be limited by any of the above-described illustrative embodiments but should instead be defined only in accordance with the following claims and their equivalents.

[0033] The embodiments illustrated and discussed in this specification are intended only to teach those skilled in the art how to make and use the invention. In describing embodiments of the disclosure, specific terminology is employed for the sake of clarity. However, the disclosure is not intended to be limited to the specific terminology so selected. The above-described embodiments of the disclosure may be modified or varied, without departing from the invention, as appreciated by those skilled in the art considering the above insights. It is therefore to be understood that, within the scope of the claims and their equivalents, the invention may be practiced otherwise than as specifically described. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment can be combined with one or more features of any other embodiment.

Claims

WE CLAIM:

1. A method of tracking and analyzing tactile reading, comprising: receiving video data from a camera arranged to image at least one finger of a reader and to image a document encoded with a plurality of tactile symbols while being read by said reader by tactile sensing; segmenting said at least one finger using said video data; segmenting at least locations of said plurality of tactile symbols within said document; tracking said at least one finger relative to said tactile symbol locations while said document is being read by said reader to provide tracking data; and processing said tracking data to provide at least one measure of tactile reading ability.

2. The method according to claim 1, wherein said document has said plurality of tactile symbols arranged in a standard pattern of sizes and separations such that said segmenting at least tactile symbol locations of said document is performed by first segmenting said document as a whole followed by assigning tactile symbol locations within said document.

3. The method according to claim 1, wherein a tactile symbol is assigned to each symbol location, and wherein said processing said tracking data to provide at least one measure of tactile reading ability uses said tactile symbols.

4. The method according to claim 3, wherein said tactile symbol represent, either alone or group together, components of written text including at least one of letters, words or punctuation.

5. The method according to claim 4, further comprising mapping said tracking data said at least each one letters, words or punctuation.

6. The method according to any one of claims 1-5, wherein said document encoded with said plurality of tactile symbols is encoded with Braille symbols.

7. The method according to any one of claims 1-6, wherein said receiving video data from said camera is arranged to image substantially an entire hand of said reader, said segmenting said video data segments said entire hand and a plurality of fingers on said entire hand, and said tracking tracks said plurality of fingers on said entire hand to provide said tracking data.

8. The method according to any one of claims 1-6, wherein said receiving video data from said camera is arranged to image substantially an entirety of both hands of said reader,said segmenting said video data segments said both hands and a plurality of fingers on each of said both hands, and said tracking tracks said plurality of fingers on said both hands to provide said tracking data.

9. The method according to any one of claims 1-8, wherein said at least one measure of tactile reading ability includes at least one of document total reading time, words per minute, time for reverse motions for repeat reading, reading times per word category, reverse times per word category.

10. The method according to any one of claims 1-9, further comprising displaying at least one of said tracking data or said at least one measure of tactile reading ability for use by at least one of said reader or an observer.

11. The method according to any one of claims 1-10, further comprising recording at least one of said tracking data or said at least one measure of tactile reading ability for later use.

12. Non-transient computer-executable code, which when executed by a computer causes the computer to perform the method of any one of claims 1-11.

13. A computer comprising non-transient computer-executable code, which when executed by said computer causes the computer to perform the method of any one of claims 1-11.

Images