Inspection device, display method, and display program

Abstract

This invention provides a testing device, display method, and display program that allow for easy understanding of literacy test results. [Solution] The inspection device comprises a display unit, a display control unit that displays inspection text on the display unit, a viewpoint detection unit that detects the user's viewpoint on the display unit, and a speech recognition unit that recognizes the voice spoken by the user. The display control unit overlays the detection result of the viewpoint detection unit and the recognition result of the speech recognition unit onto the text displayed on the display unit in a time-series correspondence.

Description

Technical Field

[0001] The present invention relates to an inspection device, a display method, and a display program.

Background Art

[0002] The inspection of literacy ability associated with learning disabilities and the like is performed by a doctor at a specialized medical institution or the like. If the inspection of literacy ability can be easily performed by a dedicated device without relying on a medical institution or the like, it becomes possible to provide appropriate learning support to users with learning disabilities. Patent Document 1 describes a device that allows a user to read aloud a text displayed on a display unit and recognizes and evaluates the user's reading aloud.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the configuration described in Patent Document 1, a configuration that can easily grasp the inspection result of literacy ability is required.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an inspection device, a display method, and a display program that can easily grasp the inspection result of literacy ability.

Means for Solving the Problems

[0006] The inspection apparatus according to the present invention comprises a display unit, a display control unit that displays inspection text on the display unit, a viewpoint detection unit that detects the user's viewpoint on the display unit, and a speech recognition unit that recognizes the voice spoken by the user. The display control unit displays the detection result of the viewpoint detection unit and the recognition result of the speech recognition unit overlaid on the text when the text is displayed on the display unit, so as to correspond in chronological order.

[0007] The display method according to the present invention includes displaying inspection text on a display unit, detecting the user's gaze and recognizing the voice spoken by the user while the text is displayed on the display unit, and displaying the results of the gaze detection and voice recognition in chronological order on top of the text while the text is displayed on the display unit.

[0008] The display program according to the present invention causes a computer to perform the following processes: displaying inspection text on a display unit; detecting the user's viewpoint and recognizing the voice spoken by the user while the text is displayed on the display unit; and displaying the results of the viewpoint detection and voice recognition in a time-series correspondence on the text while the text is displayed on the display unit. [Effects of the Invention]

[0009] According to the present invention, it becomes possible to easily grasp the results of literacy tests. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a block diagram showing an example of the configuration of the inspection apparatus according to this embodiment. [Figure 2] Figure 2 is a diagram illustrating the overview of the processing of the inspection apparatus according to this embodiment. [Figure 3] Figure 3 is a flowchart showing the flow of a basic literacy test method using the testing device according to this embodiment. [Figure 4]Figure 4 illustrates an example of a method for detecting the user's viewpoint. [Figure 5] Figure 5 illustrates another example of how to detect the user's viewpoint. [Figure 6] Figure 6 shows an example of a method for displaying inspection results according to this embodiment. [Figure 7] Figure 7 shows an example of the display content of the display unit when the slider position is changed from the state shown in Figure 6. [Figure 8] Figure 8 shows an example of the display content of the display unit when the slider position is changed from the state shown in Figure 6. [Figure 9] Figure 9 shows an example of the display content of the display unit when the slider position is changed from the state shown in Figure 6. [Figure 10] Figure 10 is a schematic diagram illustrating an example of the display operation of the test results. [Figure 11] Figure 11 is a flowchart showing an example of a display method according to this embodiment. [Modes for carrying out the invention]

[0011] The following describes embodiments of the inspection apparatus, display method, and display program according to the present invention with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are easily substituted or substantially identical to those that are easily substituted by those skilled in the art.

[0012] As shown in Figure 1, the testing device 10 comprises an input unit 12, an eye-tracking sensor 14, a microphone 16, a display unit 18, an audio output unit 20, a communication unit 22, a storage unit 24, and a control unit 26. The testing device 10 is, but is not limited to, a notebook PC (Personal Computer), a desktop PC, a tablet terminal, a smartphone, an HMD (Head Mounted Display), or smart glasses. The testing device 10 may also be a dedicated device. The testing device 10 is a device for testing a user's literacy ability.

[0013] Figure 2 is a diagram illustrating the overview of the testing device's process. The testing device 10 tests the literacy ability of the subject, user U. The testing device 10 displays a text for the literacy test on the display unit 18 for user U to read aloud. For example, as shown in Figure 2, the display unit 18 displays the sentence "saita saita sakura ga saita" (The cherry blossoms have bloomed, they have bloomed), and user U visually recognizes the sentence and says "saita saita sakura ga saita" (The cherry blossoms have bloomed, they have bloomed). Here, if the sentence displayed on the display unit 18 is in Japanese, it is assumed that the language spoken by user U is also Japanese. In other words, the testing device 10 tests literacy ability by having user U speak the sentence displayed on the display unit 18 in the language displayed. The testing device 10 detects user U's gaze toward the characters displayed on the display unit 18. The testing device 10 recognizes the voice of user U reading the sentence displayed on the display unit 18. Based on the detected gaze and the recognized voice, the testing device 10 tests user U's literacy ability.

[0014] The input unit 12 receives various input operations for the inspection device 10. The input unit 12 can be implemented using various input devices such as a keyboard, mouse, touch panel, buttons, and switches.

[0015] The line-of-sight sensor 14 detects the line of sight of the user. The line-of-sight sensor 14 outputs the detected line of sight of the user to the control unit 26. The line-of-sight sensor 14 has at least an imaging unit that images the user. The line-of-sight sensor 14, for example, images the user's eyeballs when characters are displayed on the display unit 18. The line-of-sight sensor 14 can be configured to include, for example, an infrared light emitting unit composed of an infrared LED (Light Emitting Diode) group and a pair of infrared cameras. Note that the line-of-sight sensor 14 may not include an infrared light emitting unit. The line-of-sight sensor 14 may be, for example, a visible light camera. The line-of-sight sensor 14 may be provided separately from the inspection device 10. The line-of-sight sensor 14 outputs imaging data to the control unit 26. The line-of-sight sensor 14 may be provided, for example, in smart glasses worn by the user. In this case, the line-of-sight sensor 14 outputs the captured imaging data to the control unit 26 by a wired or wireless communication method. The line-of-sight sensor 14 can detect the line of sight of the user, for example, at a predetermined period.

[0016] The microphone 16 detects the voice of the user. The microphone 16, for example, detects the voice of reading out the characters displayed on the display unit 18 when characters are displayed on the display unit 18. The microphone 16 outputs the detected voice to the control unit 26. The microphone 16 may be provided separately from the inspection device 10.

[0017] The display unit 18 displays various kinds of information. The display unit 18 is realized by a display including, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0018] The voice output unit 20 is a speaker that outputs various kinds of voices. The voice output unit 20 outputs, for example, a guide voice when the user inspects literacy ability.

[0019] The communication unit 22 is a communication interface that performs communication between the inspection device 10 and an external device. The communication unit 22 is realized, for example, by a wireless LAN (Local Area Network), a wired LAN, Wi-Fi (registered trademark), or the like.

[0020] The memory unit 24 stores various types of information. The memory unit 24 stores the calculation contents of the control unit 26 and information such as programs. The memory unit 24 includes, for example, at least one of the following: RAM (Random Access Memory), main memory such as ROM (Read Only Memory), and external memory such as HDD (Hard Disk Drive).

[0021] The memory unit 24 stores a display program that causes the computer to execute the following processes: displaying inspection text on the display unit 18; detecting the user's gaze and recognizing the voice spoken by the user while the text is displayed on the display unit 18; and displaying the results of gaze detection and voice recognition overlaid on the text in a time-series correspondence while the text is displayed on the display unit 18.

[0022] The control unit 26 controls the operation of the inspection device 10. The control unit 26 includes a processor such as a CPU (Central Processing Unit) and memory such as ROM (Read Only Memory) and RAM (Random Access Memory).

[0023] The control unit 26 includes a display control unit 30, a viewpoint detection unit 32, a voice recognition unit 34, a measurement unit 36, and a biological information detection unit 38.

[0024] The display control unit 30 controls the display unit 18 to display various types of information on it. For example, the display control unit 30 displays test text on the display unit 18 to test the user's literacy ability. The test text includes text data such as characters, numbers, words, and sentences.

[0025] The display control unit 30 displays the detection results of the viewpoint detection unit 32 and the recognition results of the speech recognition unit 34 superimposed on the text displayed on the display unit 18.

[0026] The display control unit 30 can display a heatmap showing the dwell time of the viewpoint and a curve showing the trajectory of the viewpoint movement, based on the detection results of the viewpoint detection unit 32. In addition, the display control unit 30 can change the display mode based on the recognition results of the speech recognition unit 34, showing appropriate parts of the text that were read aloud correctly and inappropriate parts that were not read aloud correctly.

[0027] Here, the appropriate portion can be defined as the portion that did not fall under the category of an inappropriate portion. The inappropriate portion includes, for example, portions with stuttering, portions with repetition, portions with skipped readings, portions with misreadings, portions with blank spaces, etc. Portions with stuttering are portions of the test text that the speech recognition unit 34 recognized as having stuttering. Portions with repetition are portions of the test text that the speech recognition unit 34 recognized as having repeatedly read aloud a single word or phrase, and portions that the same line was recognized as having been repeatedly read aloud. Portions with skipped readings are portions that the speech recognition unit 34 recognized as having skipped words or phrases, and portions that the speech recognition unit 34 recognized as having skipped lines when a line break occurred. Portions with misreadings are portions of the test text that the speech recognition unit 34 recognized as having not been read aloud for a certain period of time.

[0028] Furthermore, the display control unit 30 can set the display mode for the examination using the biological information detected by the biological information detection unit 38. Examples of biological information include heart rate, blood pressure, pulse wave, skin temperature, sweat volume, electroencephalogram, and blood flow rate.

[0029] The viewpoint detection unit 32 detects the position of the user's viewpoint on the display unit 18 in association with time, based on the detection result of the user's gaze by the gaze sensor 14. The viewpoint detection unit 32 can, for example, detect the position of the intersection point between the user's gaze detected by the gaze sensor 14 and the display unit 18 as the viewpoint. The viewpoint detection unit 32 detects the position of the user's viewpoint in association with time, for example, at predetermined intervals, based on the detection result of the user's gaze by the gaze sensor 14. The viewpoint detection unit 32 stores the detection results in the storage unit 24. The storage unit 24 stores the detection results of the viewpoint detection unit 32 in association with time.

[0030] The speech recognition unit 34 recognizes the speech spoken by the user using the inspection device 10, associating it with the time, based on the results of the user's voice detection by the microphone 16. For example, the speech recognition unit 34 can recognize the speech of the user reading the characters displayed on the display unit 18, based on the results of the user's voice detection by the microphone 16. The speech recognition unit 34 stores the recognition results in the storage unit 24. The storage unit 24 stores the recognition results of the speech recognition unit 34 as associated with the time.

[0031] The measurement unit 36 ​​determines the literacy ability of the user using the testing device 10. The measurement unit 36 ​​determines the user's literacy ability based, for example, on viewpoint information indicating the position of the user's viewpoint detected by the viewpoint detection unit 32 and audio information indicating the voice recognized by the voice recognition unit 34. The measurement unit 36 ​​can determine the user's literacy ability based, for example, on the comparison result between the viewpoint information detected by the viewpoint detection unit 32 and the audio information recognized by the voice recognition unit 34 and the literacy ability determination information stored in the memory unit 24.

[0032] The biometric information detection unit 38 detects the user's biometric information in association with time. The biometric information detection unit 38 can detect the user's biometric information based, for example, on the detection result of the biometric information sensor 40. The biometric information sensor 40 may be provided in the inspection device 10, or it may be provided separately from the inspection device 10. Alternatively, a part of the configuration or function of the inspection device 10 may be used as the biometric information sensor 40. For example, it is possible to detect the user's heart rate based on image data taken of the user by the camera constituting the gaze sensor 14. In this case, the camera of the gaze sensor 14 can be used as the biometric information sensor 40. The biometric information detection unit 38 stores the detection result in the storage unit 24. The storage unit 24 stores the detection result of the biometric information detection unit 38 in association with time.

[0033] In the control unit 26, the processor reads various programs from each unit and loads them into memory, thereby executing information processing corresponding to the functions of each unit. Examples of various programs include programs received by the communication unit 22, programs stored in the storage unit 24, and programs recorded on an external recording medium. The control unit 26 functions as an information processing device (computer) that executes various information processing. Note that other information processing devices different from the control unit 26 may execute various programs, or the control unit 26 and other information processing devices may cooperate to execute various programs.

[0034] (Basic operation of the inspection device) Next, an example of the operation of the testing device 10 according to this embodiment will be described. First, the basic operation of detecting literacy ability using the testing device 10 will be described. Figure 3 is a flowchart showing the flow of a basic literacy ability testing method using the testing device 10.

[0035] As shown in Figure 3, first, the display control unit 30 displays characters for testing the user's literacy ability on the display unit 18 (step S10).

[0036] The gaze detection unit 32 detects the position of the user's gaze on the display unit 18 based on the gaze detection result from the gaze sensor 14 (step S12). Furthermore, the gaze detection unit 32 sets the time when characters are displayed on the display unit 18 (for example, the time in step S10) as time 0 and detects the correspondence between the time and the position of the user's gaze.

[0037] Figure 4 illustrates an example of a method for detecting the position of the user's viewpoint. Figure 4(a) shows the characters displayed on the display unit 18. For example, the display unit 18 displays "Bloomed, bloomed, the cherry blossoms have bloomed" in vertical writing. Figure 4(b) shows the position of the viewpoint relative to the display unit 18 and the transition of the viewpoint. As shown in Figure 4(b), the viewpoint detection unit 32 divides the display unit 18 into, for example, 5x5 blocks and detects which block the user has placed their viewpoint on and for how long. For example, the viewpoint detection unit 32 sets a time threshold of 0.25 seconds to determine whether the user's viewpoint has stopped, and detects blocks where the viewpoint has been placed for 0.25 seconds or more as the position where the user's viewpoint stopped.

[0038] In Figure 4(b), points P1, P2, P3, P4, and P5 represent the positions where the user's viewpoint stopped. Points P1 through P5 are each greater than 0.25 seconds, which is the time threshold for determining whether the user's viewpoint stopped or not. The size of points P1 through P5 is displayed according to the amount of time the user's viewpoint was stopped. The viewpoint detection unit 32 detects that the user's viewpoint has moved along the trajectory 41 in the order of point P1 → point P2 → point P3 → point P4 → point P5. The viewpoint detection unit 32 detects the time the user focused their viewpoint on each block. The viewpoint detection unit 32 adds time information 42 to each block indicating the time the user focused their viewpoint. The viewpoint detection unit 32 outputs viewpoint information indicating the viewpoint detection result in the manner shown in Figure 4(b).

[0039] Figure 5 illustrates another example of how to detect the user's viewpoint. As shown in Figure 5(a), the display unit 18 may display text horizontally, for example, "The weather today is sunny. Let's go somewhere." Even in this case, the viewpoint detection unit 32 detects that the user's viewpoint has moved along the trajectory 41 in the order of point P11 → point P12 → point P13 → point P14 → point P15.

[0040] Returning to Figure 3, the speech recognition unit 34 recognizes the voice of the user reading the text displayed on the display unit 18 based on the results of detecting the user's voice via the microphone 16 (step S14). Furthermore, the speech recognition unit 34 recognizes the correspondence between the time and the text or characters read by the user, with the time at which the text was displayed on the display unit 18 (for example, the time in step S10) as time 0. For example, in the example shown in Figure 4(a), the speech recognition unit 34 recognizes the voice of the user reading "The cherry blossoms have bloomed, they have bloomed" displayed on the display unit 18. In the example shown in Figure 5(a), the speech recognition unit 34 recognizes the voice of the user reading "Today's weather is sunny, let's go somewhere" displayed on the display unit 18.

[0041] The measurement unit 36 ​​determines whether the literacy measurement is complete or not (step S16). Specifically, the measurement unit 36 ​​determines that the literacy measurement is complete when the pre-set time limit has expired or when the reading aloud of the characters displayed on the display unit 18 is complete. If it is determined that the literacy measurement is complete (step S16; Yes), the process proceeds to step S18. If it is not determined that the literacy measurement is complete (step S16; No), the process proceeds to step S12 and is repeated.

[0042] The measurement unit 36 ​​determines the literacy ability (step S18). Specifically, the measurement unit 36 ​​determines the user's literacy ability based on viewpoint information indicating the position of the user's viewpoint detected by the viewpoint detection unit 32 and audio information indicating the voice recognized by the voice recognition unit 34.

[0043] More specifically, the measurement unit 36 ​​determines the user's literacy ability based on conditions such as "whether the text is read aloud correctly," "whether the gaze moves in accordance with the text," "whether there are any significant pauses in the gaze," "the position of the gaze when the text is started to be read aloud," "the relationship between the gaze position and the characters being read aloud," and "the relationship between the utterance and the part where the gaze is still." However, the conditions used by the measurement unit 36 ​​to determine literacy ability are not limited to these.

[0044] The measurement unit 36 ​​determines whether the user is reading the text correctly, for example, based on whether the text displayed on the display unit 18 and the speech recognition result recognized by the speech recognition unit 34 match to a predetermined degree or higher. The measurement unit 36 ​​determines whether there is a match, for example, when voice and text are input, using a trained model generated by machine learning to output the degree of match.

[0045] The measurement unit 36 ​​determines whether the user's gaze is moving in accordance with the text, for example, based on whether the position of the user's gaze detected by the gaze detection unit 32 is moving in accordance with the text displayed on the display unit 18 by a predetermined amount or more. The measurement unit 36 ​​determines whether there is a match, for example, by using a trained model generated by machine learning to output a degree of agreement, when it receives the position of the gaze detected by the gaze detection unit 32 and the displayed text as input. Here, the measurement unit 36 ​​may also determine whether the position of the user's gaze detected by the gaze detection unit 32 is moving appropriately with respect to the text displayed on the display unit 18. This is because users who are considered to have high literacy ability may not necessarily read with their gaze aligned with the text, but may skip characters to some extent.

[0046] The measurement unit 36 ​​determines, for example, whether there is a significant dwell time at the user's viewpoint, based on whether the position of the user's viewpoint detected by the viewpoint detection unit 32 remains at one location for a predetermined time or longer. The measurement unit 36 ​​uses a trained model generated by machine learning to determine whether there is a significant dwell time at the viewpoint, for example, when it receives the position of the viewpoint detected by the viewpoint detection unit 32 as input. Alternatively, the measurement unit 36 ​​may determine that there is a significant dwell time at the viewpoint if the information on the position of the viewpoint detected by the viewpoint detection unit 32 includes a point where the viewpoint remained for a longer time than a preset threshold.

[0047] The measurement unit 36 ​​determines the user's literacy ability based, for example, on the relationship between the position of the user's gaze detected by the gaze detection unit 32 and the characters the user is reading aloud, as recognized by the speech recognition unit 34.

[0048] The measurement unit 36 ​​determines the user's literacy ability based, for example, on the position of the user's viewpoint and the position where the character the user is reading aloud is displayed. For example, the measurement unit 36 ​​determines that the user's literacy ability is normal if the position of the user's viewpoint is located at a predetermined point further ahead in the characters displayed on the display unit 18 than the character the user is reading aloud. Here, the preceding character refers to a character located later in the sentence than the character being read aloud. For example, the measurement unit 36 ​​determines that the user's literacy ability is normal if the position of the user's viewpoint coincides within a predetermined range with the position where the character the user is reading aloud is displayed on the display unit 18. Specifically, in the example shown in Figure 4, if the character the user is reading aloud is "ga" in "sakura ga", the measurement unit 36 ​​determines the user's literacy ability based on the position of the user's viewpoint corresponding to the time when "ga" was pronounced. For example, consider the case in the example shown in Figure 4 where the position of the viewpoint corresponding to the time when the user pronounced "ga" is point P4. Here, point P4 corresponds to the "ta" in "sakura ga saita" (cherry blossoms bloomed), and is located at a predetermined distance ahead of the character being read aloud by the user among the characters displayed on the display unit 18. Furthermore, it matches the position of the character being read aloud by the user within a predetermined range, so it is determined that the literacy ability is normal.

[0049] The measurement unit 36 ​​determines the user's literacy ability based, for example, on the position of the user's viewpoint relative to the display unit 18 when the user begins to read aloud the characters. The measurement unit 36 ​​determines that the user's literacy ability is normal if, for example, the position of the user's viewpoint relative to the display unit 18 when the user begins to read aloud is within a predetermined range from the first character displayed on the display unit 18. The measurement unit 36 ​​also determines that the user's literacy ability is normal if, for example, the position of the user's viewpoint relative to the display unit 18 when the user begins to read aloud is located more than a predetermined distance ahead from the first character displayed on the display unit 18. Specifically, in the example shown in Figure 4, if the first "sa" in "saita saita" is the character the user is reading aloud, the measurement unit 36 ​​determines the user's literacy ability based on the position of the user's viewpoint corresponding to the time when "sa" was pronounced. For example, consider the case in the example shown in Figure 4 where the position of the viewpoint corresponding to the time when the user first pronounced "sa" is point P2. Here, point P2 corresponds to the second "i" in "saita saita" (bloomed, bloomed), and is located at a predetermined distance ahead of the character being read aloud by the user among the characters displayed on the display unit 18. Furthermore, it matches the position of the character being read aloud by the user within a predetermined range, so it is determined that the literacy ability is normal.

[0050] The measurement unit 36 ​​determines the location where the user's speech and gaze stopped, based on the position of the user's gaze detected by the gaze detection unit 32 and the characters the user is reading aloud, as recognized by the speech recognition unit 34. For example, by extracting characters that the user has difficulty with from the location where the speech and gaze stopped, and saving them as data, it can be used for subsequent diagnosis and learning support.

[0051] The measurement unit 36 ​​determines, for example, that the user has successfully read the text displayed on the display unit 18 if the text is read aloud correctly, the user's gaze follows the text, and there is no significant hesitation in the gaze. The measurement unit 36 ​​determines, for example, that there is a minor abnormality in literacy ability even if the text is read aloud correctly, if other conditions are not met. The measurement unit 36 ​​determines, for example, that the user has not successfully read the text if the text is not read aloud correctly and a predetermined time limit has elapsed. The measurement unit 36 ​​determines, for example, that if it performs literacy ability measurements multiple times and determines that there is a minor abnormality three times, it determines that the user has not successfully read the text once. The measurement unit 36 ​​determines, for example, that if it integrates the results of multiple measurements and determines that more than half of the texts are not read aloud correctly, the user needs learning support. Up to this point, the measurement unit 36 ​​has been described as only determining whether or not the text was read aloud correctly, but it is not limited to this. In other words, each of the following conditions can be judged and used as information for assessing literacy ability: "whether the text is read aloud correctly," "whether the gaze moves in accordance with the text," "whether there are any significant pauses in the gaze," "the position of the gaze when the text is started to be read aloud," "the relationship between the gaze position and the characters being read," and "the parts where the speech and gaze stop."

[0052] (Display method) As described above, when testing literacy, it is necessary to be able to properly understand the test results. Therefore, in this embodiment, the test results are displayed on the display unit 18 by superimposing the detection results of the gaze detection unit 32 and the recognition results of the speech recognition unit 34 onto the text.

[0053] Figure 6 shows an example of a method for displaying inspection results according to this embodiment. In Figure 6, the final results at the end of the inspection are shown on the display unit 18, with the detection results from the viewpoint detection unit 32 and the recognition results from the speech recognition unit 34 displayed on the display unit 18.

[0054] The display control unit 30 displays the detection results of the viewpoint detection unit 32 and the recognition results of the speech recognition unit 34 overlaid on the inspection text TX displayed on the display unit 18, so as to correspond in chronological order. In the example shown in Figure 6, the display control unit 30 displays five sentences as inspection text TX on the display unit 18, with each sentence or two sentences on a new line: "I am a cat.", "I don't have a name yet.", "I have no idea where I was born.", "All I remember is crying 'meow meow' in a dark, damp place.", and "This is where I saw a human for the first time."

[0055] The display control unit 30 can display, for example, a heat map 51, an eye-track line 52, and an eye point 53 on the display unit 18 as detection results from the viewpoint detection unit 32.

[0056] The heatmap 51 shows the dwell time of the viewpoint. The heatmap 51 can be, for example, a circle centered on the viewpoint's dwell position, but is not limited to this shape. The display control unit 30 can change the display of the heatmap 51 according to the viewpoint's dwell time. For example, the display control unit 30 can choose not to display the heatmap 51 if the viewpoint's dwell time is below a threshold. If the viewpoint's dwell time is above a threshold, the display control unit 30 can make the display range of the heatmap 51 wider as the dwell time lengthens (in the example in Figure 6, the diameter is increased).

[0057] The eye-track line 52 indicates the trajectory of the viewpoint movement. The display control unit 30 can display the eye-track line 52 as a shape formed by connecting the viewpoint positions detected at predetermined intervals on the display unit 18 with straight lines or curves.

[0058] The eye point 53 indicates the most recently acquired viewpoint position. The eye point 53 is located at the end of the eye track line 52. In a later embodiment in which the display of the detection results of the viewpoint detection unit 32 is changed according to the time, the viewpoint position for each time period can be easily grasped by displaying the eye point 53.

[0059] The display control unit 30 can display, for example, speech recognition text 61 on the display unit 18 as a recognition result of the speech recognition unit 34. The speech recognition text 61 is text converted by the speech recognition unit 34 based on the results of the user's pronunciation. When displaying the speech recognition text 61, the display control unit 30 detects whether there are any inappropriate parts, such as stuttering parts, repeated parts, skipped parts, misreading parts, or blank spaces. If there are inappropriate parts, the display control unit 30 can change the display mode to show the inappropriate parts in relation to the other parts that are not inappropriate (appropriate parts 60).

[0060] In the example shown in Figure 6, a stutter "wa wa wa" is recognized when reading the character "wa" at the beginning of the first line of the test text TX. The display control unit 30 can change the display mode for the character "wa" where the stutter was recognized and display it as the stutter portion 62. In this case, the display control unit 30 can overlay a mark or explanation indicating the stutter portion 62 onto the test text TX. The display control unit 30 may also change the display mode of the corresponding portion 62a in the speech recognition text 61 that corresponds to the stutter portion 62 of the test text TX compared to other portions.

[0061] Furthermore, when reading the words "What is your name?" at the beginning of the second sentence on the first line of the test text TX, the repetition "What is your name?" is recognized. The display control unit 30 can change the display mode of the recognized repetition of "What is your name?" as a repeating portion 63. In this case, the display control unit 30 can overlay a mark or explanation indicating the repeating portion 63 onto the test text TX. The display control unit 30 may also change the display mode of the corresponding portion 63a in the speech recognition text 61 that corresponds to the repeating portion 63 of the test text TX compared to other portions.

[0062] Furthermore, the second line of the test text TX is recognized as having been skipped entirely. The display control unit 30 can change the display mode of the entire line in which the skipping occurred, "I have no idea where I was born," as the skipped line portion 64. In this case, the display control unit 30 can overlay a mark or explanation indicating the skipped line portion 64 onto the test text TX. The display control unit 30 may also change the display mode of the corresponding section 64a in the speech recognition text 61 that corresponds to the skipped line portion 64 of the test text TX, relative to other sections.

[0063] Furthermore, regarding the third line of the test text TX, the phrase "remembering" at the end was pronounced as "kioku shiteita," and it was recognized that the character "ru" was misread as "ta." The display control unit 30 can change the display mode for the recognized misreading character "ru" as the misread portion 65. In this case, the display control unit 30 can overlay a mark or explanation indicating the misread portion 65 onto the test text TX. The display control unit 30 may also change the display mode of the corresponding portion 65a in the speech recognition text 61 that corresponds to the misread portion 65 of the test text TX compared to other portions.

[0064] Furthermore, it is recognized that in the fourth line of the test text TX, there is a period of time (blank time) between the beginning "wagahai wa" and "koko de hajimete" where the text is not read aloud for a certain period of time or longer. The display control unit 30 can change the display mode to show the blank time portion 66 between "wagahai wa" and "koko de hajimete" where the blank time was recognized. In this case, the display control unit 30 can superimpose a mark or explanation indicating the blank time portion 66 onto the test text TX. The display control unit 30 may also change the display mode of the corresponding portion 66a in the speech recognition text 61 that corresponds to the blank time portion 66 of the test text TX compared to other portions.

[0065] Furthermore, for the fourth line of the test text TX, it is recognized that the word "human" following "for the first time here" was skipped. The display control unit 30 can change the display mode of the word "human" that was recognized as being skipped and display it as the skipped word portion 67. In this case, the display control unit 30 can overlay a mark or explanation indicating the skipped word portion 67 onto the test text TX. The display control unit 30 may also change the display mode of the corresponding section 67a in the speech recognition text 61 that corresponds to the skipped word portion 67 of the test text TX compared to other sections.

[0066] As shown in Figure 6, the display control unit 30 can display different types of inappropriate parts using different types of marks, explanations, etc. This makes it easy to understand the content of the inappropriate parts.

[0067] Furthermore, the display control unit 30 can display a timeline seek bar 70. The horizontal axis of the timeline seek bar 70 indicates time (or elapsed time from the start of inspection). Figure 6 shows an example in which the timeline seek bar 70 is displayed along the bottom edge of the display unit 18, but the display position of the timeline seek bar 70 is not limited to the above position. In the example shown in Figure 6, the slider 71 is positioned at the time t4 when the inspection is completed. The timeline seek bar 70 can be moved left and right via the input unit 12 using the slider 71.

[0068] Furthermore, the display control unit 30 can display, for example, a curve L1 showing the elapsed time required for the user to read each character aloud (hereinafter referred to as the character reading time), along with the horizontal axis of the timeline seek bar 70, on the display unit 18 a curve L2 showing the pulse, a curve L3 showing the heart rate, and a curve L4 showing the blood pressure, as detection results detected by the biometric information detection unit 38.

[0069] The display control unit 30 can display the detection result of the gaze detection unit 32, the recognition result of the speech recognition unit 34, the time taken to read one character aloud, pulse rate, heart rate, and blood pressure on the display unit 18, corresponding to the time taken to read the slider 71. In this configuration, when the position of the slider 71 changes in the left-right direction, the display control unit 30 changes the displayed content of the gaze detection unit 32, the recognition result of the speech recognition unit 34, the time taken to read one character aloud, pulse rate, heart rate, and blood pressure according to the time taken to read the slider 71. Figures 7 to 9 show examples of the displayed content of the display unit 18 when the position of the slider 71 is changed from the state shown in Figure 6. Figure 7 shows an example at time t1, after a certain amount of time has elapsed since the start of the examination; Figure 8 shows an example at time t2, after a certain amount of time has elapsed from time t1; and Figure 9 shows an example at time t3, after a certain amount of time has elapsed from time t2. Note that time t4 in Figure 6 is a certain amount of time after time t3 shown in Figure 9. As shown in Figures 7 to 9, by moving the slider 71 left or right, the detection results of the viewpoint detection unit 32 corresponding to the time, the recognition results of the voice recognition unit 34, the time taken to read one character aloud, pulse rate, heart rate, and blood pressure can be displayed on the display unit 18.

[0070] Furthermore, the display control unit 30 may automatically advance time from the start to the end of the test, and change the displayed content of the eye detection unit 32, the recognition results of the voice recognition unit 34, the time taken to read each character, pulse rate, heart rate, and blood pressure. Figure 10 is a schematic diagram showing an example of the display operation of the test results. When performing the above operation, as shown in Figure 10, the display control unit 30 may also output the voice read by the user from the voice output unit 20.

[0071] Figure 11 is a flowchart showing an example of a display method according to this embodiment. As shown in Figure 11, first, the display unit 18 displays the inspection text TX (step S20), and then the user's gaze is detected, the voice spoken by the user is recognized, and the user's biometric information is detected while the inspection text TX is displayed on the display unit 18 (step S22).

[0072] Next, the display unit 18 displays the inspection text TX, and the results of gaze detection, speech recognition, and biometric information detection are superimposed on the inspection text TX in a time-series correspondence (step S24).

[0073] As described above, the inspection device 10 according to this embodiment includes a display unit 18, a display control unit 30 that displays inspection text TX on the display unit 18, a viewpoint detection unit 32 that detects the user's viewpoint on the display unit 18, and a speech recognition unit 34 that recognizes the voice spoken by the user. The display control unit 30 displays the detection result of the viewpoint detection unit 32 and the recognition result of the speech recognition unit 34 superimposed on the inspection text TX when the inspection text TX is displayed on the display unit 18.

[0074] The display method according to this embodiment includes displaying inspection text TX on the display unit 18, detecting the user's gaze and recognizing the voice spoken by the user while the inspection text TX is displayed on the display unit 18, and displaying the results of gaze detection and voice recognition on the inspection text TX in a time-series correspondence with the inspection text TX.

[0075] The display program according to this embodiment causes the computer to perform the following processes: displaying inspection text TX on the display unit 18; detecting the user's gaze and recognizing the voice spoken by the user while the inspection text TX is displayed on the display unit 18; and displaying the results of gaze detection and voice recognition overlaid on the inspection text TX in a time-series correspondence while the inspection text TX is displayed on the display unit 18.

[0076] With this configuration, the results of gaze detection and speech recognition are displayed superimposed on the test text TX in a time-series manner when the test text TX is displayed on the display unit 18, making it easy to grasp the results of the literacy test.

[0077] In the inspection apparatus 10 according to this embodiment, the display control unit 30 displays a heat map 51 showing the dwell time of the viewpoint and an eye-track line 52 showing the trajectory of the viewpoint movement as detection results from the viewpoint detection unit 32, and displays the appropriate parts 60 of the inspection text TX that were read aloud appropriately and the inappropriate parts (62, 63, 64, 65, 66, 67) that were not read aloud appropriately as recognition results from the speech recognition unit 34, with different display modes.

[0078] With this configuration, the heat map 51 and eye-track lines 52 are displayed as detection results from the gaze detection unit 32, and the display mode is changed for appropriate parts 60 and inappropriate parts (62, 63, 64, 65, 66, 67) as recognition results from the voice recognition unit 34, making it easier to understand the inspection results.

[0079] In the inspection device 10 according to this embodiment, a biometric information detection unit 38 is further provided to detect the user's biometric information, and the display control unit 30 displays the biometric information detected by the biometric information detection unit 38 superimposed on the inspection text TX so as to correspond in chronological order to the detection results of the viewpoint detection unit 32 and the recognition results of the voice recognition unit 34.

[0080] With this configuration, the detection results from the viewpoint detection unit 32 and the recognition results from the speech recognition unit 34 are superimposed on the inspection text TX in a time-series correspondence, making it easier to understand the inspection results.

[0081] The technical scope of the present invention is not limited to the embodiments described above, and modifications can be made as appropriate without departing from the spirit of the invention. [Explanation of Symbols]

[0082] L1, L2, L3, L4…Curve, U…User, TX…Text, 10…Inspection device, 12…Input unit, 14…Eye-gaze sensor, 16…Microphone, 18…Display unit, 20…Audio output unit, 22…Communication unit, 24…Storage unit, 26…Control unit, 30…Display control unit, 32…Eye-gaze detection unit, 34…Audio recognition unit, 36…Measurement unit, 38…Biometric information detection unit, 40…Biometric information sensor, 41…Trajectory, 42…Time information 51…Heatmap, 52…Eye Track Line, 53…Eye Point, 60…Appropriate Section, 61…Speech Recognition Text, 62…Stuttering Section, 63…Repeated Section, 64…Line Skipping Section, 65…Misreading Section, 66…Blank Time Section, 67…Phrase Skipping Section, 62a,63a,64a,65a,66a,67a…Corresponding Section, 70…Timeline Seek Bar, 71…Slider

Claims

1. Display unit and The display unit includes a display control unit that displays inspection text, The display unit includes a viewpoint detection unit that detects the user's viewpoint, The voice recognition unit recognizes the voice spoken by the user. Equipped with, The display control unit overlays the detection results from the viewpoint detection unit and the recognition results from the speech recognition unit onto the text displayed on the display unit, so that they correspond in chronological order. Inspection device.

2. The display control unit, As a result of the detection by the viewpoint detection unit, a heat map showing the dwell time of the viewpoint and a curve showing the trajectory of the viewpoint's movement are displayed. Based on the recognition results of the speech recognition unit, the display mode is changed to show the appropriate parts of the text that were read aloud correctly and the inappropriate parts that were not read aloud correctly. The inspection apparatus according to claim 1.

3. The system further includes a biometric information detection unit for detecting the user's biometric information, The display control unit displays the biological information detected by the biological information detection unit overlaid on the text, in a time-series correspondence with the detection results of the viewpoint detection unit and the recognition results of the speech recognition unit. The inspection apparatus according to claim 1.

4. The display unit will show text for inspection, The display unit detects the user's gaze while the text is displayed and recognizes the voice spoken by the user. The display unit displays the viewpoint detection results and the voice recognition results in a time-series correspondence with the text. Display methods that include this.

5. The process of displaying inspection text on the display unit, The process includes detecting the user's gaze while the text is displayed on the display unit and recognizing the voice spoken by the user. A process of overlaying the detection results of the viewpoint and the recognition results of the voice onto the text in a time-series correspondence when the text is displayed on the display unit. A display program that causes a computer to execute something.

Images