Support device and support system

Abstract

To provide a support device that allows a user to specify a selection portion while reducing the psychological burden on the user.SOLUTION: A support device 1 according to the present disclosure comprises a transparent sheet-shaped contact detection unit 3 that detects contact with an object by user movement, and sensors 2-1 to 2-4 that are installed at the edge of the contact detection unit 3 and have an array of image sensors in one direction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an assistance device and an assistance system that assist a user in input operations. [Background technology]

[0002] Learning disabilities (LD), a type of developmental disorder, are often consulted after children start school, particularly those with dyslexia, a literacy disorder that makes it difficult to read kanji or long sentences. Traditionally, adults have provided support to these children by reading aloud, providing furigana (guidance), highlighting important parts, hiding unnecessary parts, and segmenting text. Recently, digital technology has become capable of providing some of the manual support needed. This not only improves children's learning efficiency, but also reduces the burden on adults, allowing for more personalized support. While digitalized information makes it easy for users, including children, to specify the areas for which they want assistance, non-digital media, such as paper, require users to identify and digitize the areas they want assistance.

[0003] Patent Document 1 discloses an image recognition device including an imaging element, a transparent plate material in close contact with the surface of the imaging element, and a substrate on which a circuit for controlling the imaging element is mounted. When imaging is performed using the image recognition device described in Patent Document 1, a light source is disposed to the side of the imaging element, and the imaging element captures an image of an object on which Braille, letters, etc. are written through the transparent plate material. The substrate is disposed above the imaging element, and a band that can be worn around a finger is attached to the substrate. As a result, a user wraps the band around their finger and traces the Braille, letters, etc. on the object with their finger, thereby acquiring the Braille, letters, etc. written on the object as an image. For example, the image recognition device described in Patent Document 1 includes an audio output means, recognizes the Braille, letters, etc. in the acquired image, converts the results of the recognition process into audio, and outputs the converted audio from the audio output means, allowing the user to recognize the Braille, letters, etc. traced with their finger as audio. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-070530 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when the image recognition device described in Patent Document 1 is used by a child at a school or the like to specify a selected portion of a document printed on paper or the like for which supplementary information, such as furigana or the reading of kanji characters, is desired, special equipment, such as a light source provided on the side or a band worn on the finger, is required. This poses a psychological problem of standing out from other children around. Support for user input operations is sometimes desired so as to avoid attracting as much attention as possible from those around, not just for children at school.

[0006] The present disclosure has been made in view of the above, and aims to provide an assistance device that allows a user to specify a selection point while reducing the psychological burden on the user. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the objectives, the assistance device disclosed herein comprises a transparent sheet-like contact detection unit that detects contact with an object due to a user's movement, and a line sensor provided at the end of the contact detection unit and having image sensors arranged in one direction. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to achieve an effect that the user can specify a selection point while reducing the psychological burden on the user. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration example of a support device according to a first embodiment. [Figure 2] FIG. 1 is a side view of the support device according to the first embodiment. [Figure 3] FIG. 1 is a diagram showing an image of how the support device according to the first embodiment is used; [Figure 4] FIG. 1 is a diagram showing an image of how the support device according to the first embodiment is used; [Figure 5] FIG. 1 is a diagram showing an example of a support device according to a first embodiment, which is equipped with a transmitting / receiving unit. [Figure 6] FIG. 6 is a diagram showing an example of the configuration of a support system including the support device shown in FIG. 5. [Figure 7] 10 is a flowchart showing an example of a processing procedure in the control device of the first embodiment. [Figure 8] FIG. 1 is a diagram showing a configuration example of a support device according to a first embodiment that includes a displacement sensor; [Figure 9] FIG. 1 is a diagram showing an example of the configuration of a computer system that realizes a control device according to a first embodiment. [Figure 10] FIG. 10 is a diagram illustrating a configuration example of a support device according to a second embodiment. [Figure 11] FIG. 10 is a side view of the support device according to the second embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a functional configuration of a support device according to a second embodiment; [Figure 13] FIG. 10 is a diagram showing a configuration example of a support system including a support device according to a second embodiment. [Figure 14] FIG. 10 is a diagram showing a display example of the second embodiment in which furigana is displayed as auxiliary information. [Figure 15] FIG. 10 is a diagram showing another example of display in the second embodiment when furigana is displayed as auxiliary information. [Figure 16] FIG. 10 shows a display example of the second embodiment in which the meaning of the selected part is displayed as auxiliary information. [Figure 17] FIG. 10 is a diagram showing a display example of the second embodiment in which a translation is displayed as auxiliary information. [Figure 18] FIG. 10 is a diagram showing a display example of the second embodiment in which segment positions are displayed as auxiliary information. [Figure 19] FIG. 10 shows a display example of the second embodiment in which a mask is displayed as auxiliary information. [Figure 20] FIG. 10 is a diagram showing a configuration example of a support device according to a second embodiment when auxiliary information is output as sound. [Figure 21] FIG. 10 is a diagram showing a configuration example of a support system according to a third embodiment. [Figure 22] FIG. 10 is a diagram showing an example of a functional configuration of a support system according to a third embodiment. [Figure 23] FIG. 10 is a diagram showing another example of the configuration of the support system according to the third embodiment. [Figure 24] FIG. 10 is a diagram showing an example of information exchange between a student and a teacher according to the third embodiment. [Figure 25] FIG. 10 is a diagram showing a configuration example of a support system according to a fourth embodiment. [Figure 26] FIG. 10 is a diagram showing an outline of schedule management according to a fourth embodiment. [Figure 27] FIG. 13 is a diagram showing a configuration example of a support system according to a fifth embodiment. [Figure 28] FIG. 10 shows an example of use of the support system according to the fifth embodiment. [Figure 29] FIG. 13 is a diagram showing a configuration example of a support system according to a sixth embodiment. [Figure 30] FIG. 13 shows an example of use of the support system according to the sixth embodiment. [Figure 31] FIG. 13 is a diagram illustrating a configuration example of a support device according to a seventh embodiment. [Figure 32] FIG. 13 shows an example of the configuration of a support system according to a seventh embodiment. [Figure 33] FIG. 20 is a diagram showing the movement of the support device and the photographable range of the support device according to the seventh embodiment. [Figure 34] FIG. 13 is a diagram illustrating a configuration example of a support device according to an eighth embodiment. [Figure 35] FIG. 13 is a side view of the support device according to the eighth embodiment. [Figure 36] FIG. 13 shows an example of the configuration of a support system according to an eighth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The support device and support system according to the embodiment will be described in detail below with reference to the accompanying drawings.

[0011] Embodiment 1 FIG. 1 is a diagram showing an example of the configuration of an assistance device according to a first embodiment. The assistance device 1 of this embodiment is used, for example, in a school, a cram school, or the like, for a user to specify a selected location when the user wants to obtain auxiliary information for the selected location regarding a displayed object such as characters, figures, or images displayed on an object, but the use of the assistance device 1 of this embodiment is not limited to this. Examples of auxiliary information include, but are not limited to, audio, phonetic readings, and word meanings. Details of the auxiliary information will be described later. Note that users of the assistance device 1 are not limited to children. Examples of objects include, but are not limited to, paper such as textbooks, printouts, books, and notebooks.

[0012] The support device 1 of this embodiment has a rectangular shape when viewed from the front and includes sensors 2-1 to 2-4 and a contact detection unit 3. Each of the sensors 2-1 to 2-4 is a line sensor in which imaging elements are arranged in one direction and capture a line-shaped image. Each of the sensors 2-1 to 2-4 includes, for example, a plurality of imaging elements arranged in one direction and an A / D (Analog to Digital) conversion circuit that converts data captured by the imaging elements into a digital signal. The imaging elements may be, but are not limited to, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The A / D conversion circuit may be provided separately from the sensors 2-1 to 2-4. Hereinafter, when the sensors 2-1 to 2-4 are referred to individually without distinction, they will also be referred to as sensor 2. The contact detection unit 3 is a transparent sheet-like element and has a rectangular shape when viewed from the front. 1, the front shapes of the support device 1 and the contact detection unit 3 are rectangular, but they may be any shape, such as circular or triangular. Furthermore, the contact detection unit 3 is not limited to a transparent, colorless sheet, but may be a colored, transparent sheet. The sensor 2 detects contact by a finger or the like using, for example, a resistive film method, a capacitance method, an ultrasonic method, or an optical method, but the method of detecting contact by the contact detection unit 3 is not limited to these examples.

[0013] In the example shown in FIG. 1, a frame 4 is provided outside the contact detection unit 3, and sensors 2-1 to 2-4 are provided on the frame 4; however, the frame 4 need not be provided. The frame 4 is, for example, made of a transparent material, but it need not be transparent. Furthermore, in the example shown in FIG. 1, the sensors 2-1 to 2-4 are provided at the ends of the contact detection unit 3, i.e., at the portions contacting the four sides of a rectangle. However, the sensors 2-1 to 2-4 may be provided on the edges of the support device 1, i.e., around the periphery, and the arrangement positions of the sensors 2-1 to 2-4 are not limited to the example shown in FIG. 1. Furthermore, the number of sensors 2-1 to 2-4 is not limited to the example shown in FIG. 1. For example, the sensors 2-1 to 2-4 may be provided at the ends of the long sides and the short sides of the contact detection unit 3. Note that if the shape of the contact detection unit 3 is square, one of the two parallel sides of the square may be considered the long sides, and the other two sides may be considered the short sides.

[0014] For example, the sensors 2-1 to 2-4 may be arranged at positions that contact the end of the contact detection unit 3, as shown in FIG. 1. Alternatively, the sensors 2-1 to 2-4 may be arranged at positions that include the end of the contact detection unit 3. For example, the sensors 2-1 to 2-4 may be arranged inside the rectangular area of the contact detection unit 3 in FIG. 1 so that the outer ends of the sensors 2-1 to 2-4 in the contact detection unit 3 coincide with the end of the contact detection unit 3. Furthermore, the sensors 2-1 to 2-4 may be arranged so that the top side of the rectangle representing the contact detection unit 3 in FIG. 1 overlaps with the sensor 2-1, the bottom side of the rectangle representing the contact detection unit 3 in FIG. 1 overlaps with the sensor 2-3, the left side of the rectangle representing the contact detection unit 3 in FIG. 1 overlaps with the sensor 2-4, and the right side of the rectangle representing the contact detection unit 3 in FIG. 1 overlaps with the sensor 2-2. Furthermore, the sensors 2-1 to 2-4 may be arranged on the outer periphery of the support device 1, and the sensors 2-1 to 2-4 may form the outer frame of the support device 1.

[0015] FIG. 2 is a side view of the assistance device 1 according to the present embodiment. The contact detection unit 3 has a first surface that is a surface that an object, such as a user's finger, contacts, and a second surface that is the reverse side of the first surface. The contact detection unit 3 is placed on the object to be imaged by the imaging element with the first surface facing up. In FIG. 2, the upper surface of the contact detection unit 3 is the first surface. In the example shown in FIG. 2, the assistance device 1 includes a support unit 5, which is a transparent plate that contacts the lower part of the contact detection unit 3, i.e., the second surface. The support unit 5 may be integrated with the frame unit 4. Note that the support unit 5 is not limited to a colorless, transparent plate; it may also be a colored, transparent plate or something like a polarizing plate that allows the user to select a color during use. As mentioned above, the same applies to the contact detection unit 3. Some people with learning disabilities have unique perceptions of color. Using a colored, transparent plate for the support unit 5 makes it easier for such users to read text. 2, the sensors 2-1 to 2-4 are provided so as to protrude from the upper part of the frame portion 4, but this is not limiting, and the sensors 2-1 to 2-4 may be embedded in the frame portion 4 so that at least a portion of the sensors 2-1 to 2-4 is within the frame portion 4. Furthermore, even when the sensors 2-1 to 2-4 are arranged so as to overlap with the contact detection unit 3 when viewed from the front, the sensors 2-1 to 2-4 may similarly be provided so as to protrude from the upper part of the frame portion 4, or may be embedded in the contact detection unit 3 so that at least a portion of the sensors 2-1 to 2-4 is within the contact detection unit 3. Although an example in which the support portion 5 is provided is shown in FIG. 2, the support portion 5 does not necessarily have to be provided.

[0016] 3 and 4 are diagrams illustrating an image of how the support device 1 of this embodiment is used. First, as shown in FIG. 3, a user slides the support device 1 in at least one direction over the object 50. This allows the sensors 2-1 to 2-4 to read an image of the area over which the support device 1 has moved. The direction of movement is, for example, a direction perpendicular to the arrangement direction of the image pickup elements of at least one of the sensors 2-1 to 2-4, such as, but not limited to, the left-right direction or the up-down direction. Note that "perpendicular" does not necessarily mean that the angle formed with the arrangement direction is strictly 90 degrees, but it is sufficient that the angle formed with the arrangement direction is approximately 90 degrees.

[0017] For example, as indicated by the right-pointing arrow in FIG. 3 , when moving from the upper left corner of the object 50 to the right, sensors 2-2 and 2-4, each having a vertically arranged imaging element, move horizontally to acquire a two-dimensional image. Also, as indicated by the downward-pointing arrow in FIG. 3 , when moving from the upper left corner of the object 50 to the bottom, sensors 2-1 and 2-3, each having a horizontally arranged imaging element, move vertically to acquire a two-dimensional image. For example, when reading the top row of the object 50 from the right, the user moves the support device 1 from the upper left corner of the object 50 to the right, and when it reaches the right edge of the object 50, stops the support device 1 there. At the stopped position, the user can select a location from which they want to obtain auxiliary information using the support device 1, as needed, and obtain the auxiliary information. The initial movement is not limited to movement from the upper left corner to the right, but may be any movement that can acquire an image including the area that the user initially focuses on (for example, in the example of FIG. 3 , the area that can be viewed through the support device 1 when the right edge of the support device 1 coincides with the right edge of the top row). For example, the support device 1 may be moved to the left from a position where it is positioned beyond the object 50 so that the right edge of the object 50 is near the left edge of the support device 1. Then, after the user finishes reading the text of the object 50 in the area corresponding to the position where the support device 1 was first stopped, the user moves the support device 1 one column to the left and stops it, and, if necessary, selects a section from which the user wants to obtain supplemental information and obtains the supplemental information. By repeating this operation, the user can continue reading the text in the upper section of the object 50 while obtaining the supplemental information. Next, the user moves the support device 1 downward and then moves it from left to right to read the text in the lower section of the object 50. When the user reaches the right edge, the user stops the support device 1 and, if necessary, uses the support device 1 to select a section from which the user wants to obtain supplemental information and obtain the supplemental information. Thereafter, the user moves the support device 1 one column at a time and stops it, as in the upper section, to continue reading the text of the object 50 while obtaining the supplemental information. Note that the direction and amount of movement of the support device 1 are not limited to the above example. Furthermore, the content of the object 50, the arrangement of the sentences, the direction of the character strings, the way the user reads, and the like are not limited to the examples described above.The area where reading is performed is not limited to the entire object 50, but may be a part of the object 50. In this case, by moving the support device 1 over a part of the object 50, reading is similarly performed by the sensors 2-1 to 2-4.

[0018] Here, an example will be described in which the support device 1 includes four sensors 2, namely, sensors 2-1 to 2-4. However, two sensors 2 may be provided at the left and right ends or at the top and bottom ends. For example, the two sensors 2 may be sensors 2-1 and 2-3 in FIG. 1, or sensors 2-2 and 2-4 shown in FIG. 1. When two sensors 2 are provided at the top and bottom ends, a two-dimensional image is acquired by moving the support device 1 in the up and down direction when reading the object 50. When two sensors 2 are provided at the left and right ends, a two-dimensional image is acquired by moving the support device 1 in the left and right direction when reading the object 50. This is not a limitation, and the support device 1 may include three sensors 2. Furthermore, one sensor 2 may be provided on only one side. In this case, a two-dimensional image of the object 50 corresponding to the area through which the sensor 2 passes as the support device 1 moves is acquired. That is, the number of sensors 2 may be one or more, and the sensor 2 may be provided, for example, at the end of the contact detection unit 3.

[0019] The user moves the support device 1 over the object 50 and then stops the support device 1 at a position of interest. FIG. 4 shows a state in which the object 50 is stopped. In this state, the user can designate a selected location by touching the support device 1 with a finger. The following describes an example in which the user designates a selected location by touching the support device 1 with a finger. However, the present invention is not limited to this example. The user may designate a selected location by touching an implement such as a pen to the support device 1. That is, the contact detection unit 3 only needs to detect contact of an object by the user's action, and the object may be a finger or an implement. As described above, the contact detection unit 3 of the support device 1 is transparent. Therefore, as shown in FIG. 4, the user can visually recognize a designatable area 51, which is a part of the object 50, through the contact detection unit 3. The designatable area 51 is an area in which the user can input a selected location using the support device 1. The designatable area 51 moves as the support device 1 moves. The specifiable area 51 is also the area of the object 50 that can be seen by the user, and the contact detection unit 3 can detect the selection point, which is the point selected by the user within the specifiable area 51, as the contact position, where the object comes into contact.

[0020] As described above, the selected location (the location selected by the user within the specifiable area 51 of the object 50) corresponding to the contact position is determined based on a two-dimensional image of the object 50 obtained by moving the support device 1 in a direction perpendicular to the arrangement direction of the imaging elements while it is placed on the object 50.

[0021] The images captured by the sensors 2-1 to 2-4 are stored within or outside the support device 1. An image corresponding to a location designated by the user's finger touching the support device 1 is extracted from the image, and a predetermined process, such as character recognition, is performed on the extracted image. This allows the support device 1 to provide the user with auxiliary information corresponding to the processed characters. The auxiliary information may be displayed on a display device within or outside the support device 1, or may be output as audio. The support device 1 of this embodiment is transparent and thin, i.e., plate-like, and thus has a shape similar to a stationery pad or flashcard. Therefore, it is difficult to distinguish from stationery at first glance and is not conspicuous when used by users at schools, cram schools, etc. Therefore, when a user performs input operations using the support device 1 of this embodiment, the user's psychological burden can be reduced. Furthermore, because there is no need to attach a device to the user's finger, the device does not obscure the characters or the like on the object 50 when the user traces the designated location, allowing the user to trace accurately. Furthermore, the visual information on the object 50 and the auxiliary information are linked, making it easier for the user to deepen their understanding.

[0022] Next, an example of displaying auxiliary information using the support device 1 of this embodiment will be described. FIG. 5 is a diagram showing an example of the support device 1 of this embodiment equipped with a transmitter / receiver unit. As shown in FIG. 5, the support device 1 may include a transmitter / receiver unit 6. The transmitter / receiver unit 6 is, for example, a communication device capable of wireless communication. In the example shown in FIG. 5, the transmitter / receiver unit 6 is provided on the frame unit 4, but the arrangement position of the transmitter / receiver unit 6 is not limited to the example shown in FIG. 5. The transmitter / receiver unit 6 may be integrated with one or more of the sensors 2-1 to 2-4.

[0023] Fig. 6 is a diagram showing an example of the configuration of a support system including the support device 1 shown in Fig. 5. The support system shown in Fig. 6 includes the support device 1, a control device 20, and an output device 30. Fig. 6 shows an example of the functional configuration of each device, and does not show the frame 4 and support part 5 of the support device 1. Furthermore, Fig. 6 shows one sensor 2, but as mentioned above, there may be multiple sensors 2.

[0024] The sensors 2-1 to 2-4 of the support device 1 output line images, which are acquired line-shaped images, together with identification information of each of the sensors 2-1 to 2-4 to the transmitting / receiving unit 6 at regular intervals. The sensors 2-1 to 2-4 output the line images to the transmitting / receiving unit 6, for example, at regular intervals. The transmitting / receiving unit 6 transmits the line images to the control device 20. Furthermore, when the contact detection unit 3 detects contact, it outputs contact position information indicating the position where the contact was detected to the transmitting / receiving unit 6. The transmitting / receiving unit 6 transmits the contact position information to the control device 20.

[0025] The control device 20 acquires a plurality of line images acquired by the sensor 2 while being placed on the object 50 and moved in a direction perpendicular to the arrangement direction of the image sensor, generates a two-dimensional image of the object 50 using the plurality of line images, and identifies a selected portion corresponding to a contact position on the contact detection unit 3 using the two-dimensional image. As shown in FIG. 6 , the control device 20 includes a transceiver 21, a reading processor 22, a position identification unit 23, an information storage unit 24, a character recognition unit 25, and an auxiliary information generator 26. The transceiver 21 is capable of communicating with other devices. For example, the transceiver 21 receives line images acquired by the sensors 2-1 to 2-4 from the transceiver 6 of the support device 1 and outputs the received line images to the reading processor 22. Furthermore, when the transceiver 21 receives contact position information from the transceiver 6 of the support device 1, it outputs the received contact position information to the position identification unit 23. The control device 20 may communicate with a plurality of support devices 1, generate auxiliary information corresponding to the plurality of support devices 1, and transmit the auxiliary information to the corresponding support devices 1. In this case, the auxiliary information generated for each support device 1 may be different.

[0026] Furthermore, auxiliary information suitable for each user using the assistance device 1 may be learned by machine learning or the like, and the control device 20 may generate auxiliary information suitable for the user using a trained model. The trained model may be generated by the control device 20 or by a learning device (not shown). For example, it may be possible to evaluate what kind of auxiliary information would be easy for the user to understand each character string such as a document or a word, and generate a trained model by supervised learning using multiple training datasets including the input character string and auxiliary information that has been evaluated favorably.

[0027] The reading processing unit 22 generates a two-dimensional image using a plurality of line images acquired at different times. For example, the reading processing unit 22 stores line images corresponding to the sensors 2-1 to 2-4 for each sensor 2 in the information storage unit 24. For example, when the support device 1 is moved left and right, one two-dimensional image is generated using the plurality of line images acquired by the sensor 2-2, and one two-dimensional image is generated using the plurality of line images acquired by the sensor 2-4, and the generated two-dimensional images are stored in the information storage unit 24. The two-dimensional image generated from the line images acquired by the sensor 2-2 and the two-dimensional image generated from the line images acquired by the sensor 2-4 partially overlap, but the reading processing unit 22 generates one two-dimensional image by adding non-overlapping portions to the ends of the overlapping portions.

[0028] For example, in the example shown in FIG. 3, when the support device 1 is moved from left to right, the sensor 2-4 can read the vicinity of the left end of the object 50, but the sensor 2-2 cannot read the vicinity of the left end of the object 50. Also, the sensor 2-4 cannot read the vicinity of the right end of the object 50, but the sensor 2-2 can read the vicinity of the right end of the object 50. Therefore, the reading processing unit 22 adds an image of the vicinity of the left end based on the line image acquired by the sensor 2-4 to the left of the overlapping portion, and adds an image of the vicinity of the right end based on the line image acquired by the sensor 2-2 to the right of the overlapping portion. This allows the reading processing unit 22 to generate a two-dimensional image from the left end to the right end of the object 50.

[0029] When the position specifying unit 23 receives the contact position information from the transmitting / receiving unit 21, it specifies a position in the two-dimensional image of the object 50 that corresponds to the position indicated by the contact position information, and extracts an image that corresponds to the specified position from the two-dimensional images stored in the information storage unit 24. The position specifying unit 23 outputs the extracted image to the character recognition unit 25. The method of specifying a position in the position specifying unit 23 will be described later.

[0030] The character recognition unit 25 performs character recognition processing on the extracted image received from the position identification unit 23, converting characters in the extracted image into character data, and outputs the converted character data to the auxiliary information generation unit 26. If the extracted image received from the position identification unit 23 contains multiple characters, i.e., a character string, the character recognition unit 25 outputs character data indicating the character string to the auxiliary information generation unit 26. Note that, hereinafter, a character string containing only one character will also be referred to as a character string. The character data is, for example, a code defined for each character in a computer system. Note that, although an example has been described in which the position identification unit 23 extracts an image using the result of identifying the position of the object 50 in a two-dimensional image, this image extraction may also be performed by the character recognition unit 25. Alternatively, the character recognition unit 25 may be integrated with the position identification unit 23, and the character recognition unit 25 may function as the position identification unit 23. Note that, although the character recognition in the character recognition unit 25 is performed by machine learning, for example, the character recognition method in the character recognition unit 25 is not limited thereto. When character recognition in character recognition unit 25 is performed by machine learning, for example, a trained model is stored in information storage unit 24, and character recognition unit 25 performs character recognition using the trained model.

[0031] The auxiliary information generation unit 26 generates auxiliary information based on the character data received from the character recognition unit 25 and outputs the generated auxiliary information to the transmission / reception unit 21. The auxiliary information may be furigana (phonetic reading) when the character string indicated by the character data is kanji, the meaning of the character string indicated by the character data, information indicating the segmented parts of the character string, audio data obtained by converting the character string indicated by the character data into audio, a combination of two or more of these, or other information. The conversion information used by the auxiliary information generation unit 26 to generate the auxiliary information is stored, for example, in the information storage unit 24. The conversion information may be, for example, information indicating the correspondence between kanji and furigana, dictionary information indicating the meaning of the character string, information for converting the character string into audio data, rules for segmenting the character string or a trained model using machine learning, etc. The auxiliary information generation unit 26 generates auxiliary information corresponding to the character data received from the character recognition unit 25 using the conversion information stored in the information storage unit 24. The auxiliary information generation unit 26 may output the auxiliary information to the transmission / reception unit 21 together with the extracted character string.

[0032] The conversion information may be stored in an information storage device separate from the control device 20. In this case, the auxiliary information generation unit 26 may acquire information necessary for generating the auxiliary information or the auxiliary information itself from the information storage device via the transmission / reception unit 21. For example, dictionary information indicating the meaning of a character string may be stored in the information storage device. The auxiliary information generation unit 26 may then acquire the meaning corresponding to the selected character string, i.e., the character string indicated by the character data received from the character recognition unit 25, from the information storage device via the transmission / reception unit 21.

[0033] The auxiliary information generated by the auxiliary information generation unit 26 is transmitted to the output device 30 by the transmission / reception unit 21. The output device 30 is at least one of a display device such as a display or a monitor, and an audio output device such as a speaker, headphones, or earphones. The output device 30 is capable of outputting auxiliary information corresponding to a selection made by a user by touching the contact detection unit 3 with a finger or the like. If the output device 30 is a display device, the output device 30 is installed, for example, in a location visible to the user. If the output device 30 is an audio output device, the output device 30 is installed, for example, in a location where the user can hear the audio output from the output device 30. Note that while FIG. 6 illustrates an example in which the output device 30 is provided separately from the control device 20, the output device 30 may be integrated with the control device 20. Also, while FIG. 6 illustrates an example in which the control device 20 is provided separately from the assistance device 1, the control device 20 may be mounted on the assistance device 1. In this case, the transmission / reception unit 6 for exchanging information between the control device 20 and the assistance device 1 may not be provided.

[0034] When the control device 20 performs processing related to multiple support devices 1, a two-dimensional image is stored in the information storage unit 24 for each support device 1. For example, the two-dimensional image is stored together with the identification information of the support device 1. Furthermore, the contact position information is input to the position identification unit 23 together with the identification information of the support device 1, and the position identification unit 23, the character recognition unit 25, and the auxiliary information generation unit 26 perform processing for each support device 1 using the identification information of the support device 1. A corresponding output device 30 may be defined for each support device 1, and in this case, the transmission / reception unit 21 transmits the auxiliary information to the output device 30 corresponding to the auxiliary information.

[0035] Next, the operation of the control device 20 of this embodiment will be described. FIG. 7 is a flowchart showing an example of a processing procedure in the control device 20 of this embodiment. As shown in FIG. 7, the control device 20 acquires a line image (step S1). In detail, the user moves the support device 1 over the object 50, and the transmitting / receiving unit 21 of the control device 20 receives the line image acquired by the sensor 2 from the support device 1, thereby causing the control device 20 to acquire the line image. As described above, the transmitting / receiving unit 21 outputs the received line image to the reading processing unit 22, and the reading processing unit 22 stores the line image in the information storage unit 24.

[0036] The control device 20 generates a two-dimensional image (step S2). Specifically, the reading processing unit 22 generates a two-dimensional image using the line image stored in the information storage unit 24, and stores the generated two-dimensional image in the information storage unit 24.

[0037] The control device 20 determines whether or not an operation has been performed (step S3). Specifically, the control device 20 determines whether or not the position identification unit 23 has received contact position information from the transmission / reception unit 21. Note that the operation here refers to an operation using the contact detection unit 3, such as tracing the contact detection unit 3 with a finger.

[0038] If an operation has been performed (Yes in step S3), the control device 20 performs character recognition (step S4). In detail, the position identification unit 23 identifies a position in the two-dimensional image of the object 50 that corresponds to the position indicated by the contact position information, extracts an image corresponding to the identified position from the two-dimensional image stored in the information storage unit 24, and outputs the extracted image to the character recognition unit 25. The character recognition unit 25 converts the characters in the received image into character codes, and outputs the character string converted into the character codes to the auxiliary information generation unit 26.

[0039] Here, we will explain the position identification method used by the position identification unit 23. The support device 1 may include a sensor, such as a displacement sensor, that detects the movement of the support device 1, and the position identification unit 23 may use the results detected by this sensor to identify the position in the two-dimensional image of the target object 50 that corresponds to the position indicated by the contact position information.

[0040] FIG. 8 is a diagram showing a configuration example of the support device 1 of this embodiment that includes a displacement sensor 7. In the example shown in FIG. 8, the support device 1 includes a displacement sensor 7 at each of the upper left end and the lower right end. As the displacement sensor 7, a sensor that detects planar movement used in a mouse for a computer system can be used. The detection method of the displacement sensor 7 can be, for example, a ball type (mechanical type), an optical type, a laser type, etc., but is not limited to these.

[0041] For example, when the support device 1 is placed on the object 50, the displacement sensor 7 detects the amount of displacement at regular intervals. The amount of displacement is transmitted to the control device 20 via the transceiver 6. When the transceiver 21 of the control device 20 receives the amount of displacement, it outputs the received amount of displacement to the position identification unit 23. The position identification unit 23 sequentially accumulates the amount of displacement to calculate the amount of movement of the support device 1. This allows the support device 1 to calculate the amount of movement from a reference position, which is the position of the support device 1 on the object 50 at a reference time. The reference time is, for example, the time when the movement of the support device 1 begins to acquire a two-dimensional image. The reference time may be, for example, the time when a power switch or power button (not shown) for turning on the power of the support device 1 is operated to turn on the power, or the time when a reading start switch or button (not shown) for instructing the start of reading is operated to start reading. Alternatively, the reference time may be determined taking into account the presence or absence of characters on the object 50. For example, the control device 20 may update the two-dimensional image each time it acquires a line image of the object 50, perform character recognition using the updated two-dimensional image, and use the time when the character is first recognized as the reference time.

[0042] The position specifying unit 23 determines the current position of the support device 1 in the two-dimensional image using the calculated amount of movement, and extracts an image corresponding to the area directly below the contact detection unit 3 of the current support device 1 from the two-dimensional image based on the determined position. The position specifying unit 23 extracts an image corresponding to the position indicated by the contact position information from the extracted image based on the contact position information. As a result, an image corresponding to the part where the user traced the support device 1 with his / her finger is extracted.

[0043] For example, when a user moves the support device 1 from the upper left corner of the object 50 shown in FIG. 3 to the right, the displacement sensor 7 detects that the support device 1 is moving in a direction perpendicular to the arrangement direction of the imaging elements of the sensors 2-2 and 2-4 of the support device 1. The position identification unit 23 calculates the amount of movement of the support device 1 using the time when the support device 1 is located at the upper left corner of the object 50 shown in FIG. 3 as a reference time point, thereby determining the current position of the support device 1 in a two-dimensional image acquired by the sensors 2-2 and 2-4 scanning the object 50 from the upper left corner to the right. This allows the position identification unit 23 to extract an image corresponding to the area directly below the contact detection unit 3 of the current support device 1 from the two-dimensional image. That is, for example, the position identification unit 23 can extract an image of the designatable area 51 shown in FIG. 4. From the extracted image, the position identification unit 23 extracts an image corresponding to the position indicated by the contact position information from the image of the designatable area 51 based on the contact position information.

[0044] In the example shown in FIG. 8, two displacement sensors 7 are provided, but the number of displacement sensors 7 is not limited to two. Furthermore, the arrangement positions of the displacement sensors 7 are not limited to the example shown in FIG. 8. If it is desired to detect the rotation of the support device 1 on the object 50, two or more displacement sensors 7 are provided. Furthermore, if only the displacement is to be detected, one displacement sensor 7 may be used. If the rotation of the support device 1 is detected, the position identification unit 23 identifies the current position of the support device 1 in the two-dimensional image, taking the rotation into consideration. Any method may be used to calculate the amount of movement and rotation using the displacement sensors 7, and a detailed description thereof will be omitted since a common method can be used.

[0045] Furthermore, the position identification by the position identification unit 23 is not limited to the example using the displacement sensor 7, and may be performed, for example, by comparing the acquired two-dimensional image with line images acquired by the sensors 2-1 to 2-4 of the support device 1. That is, the position identification unit 23 determines, by image processing, which parts of the two-dimensional image the four line images acquired by the sensors 2-1 to 2-4 match. When the object 50 is an object with no blank spaces, such as a newspaper, textbook, or book, the position identification unit 23 can identify the position in the two-dimensional image of the support device 1 by the image processing described above.

[0046] Returning to the explanation of FIG. 7, after step S4, the control device 20 generates auxiliary information (step S5). Specifically, the auxiliary information generation unit 26 generates auxiliary information based on the character data received from the character recognition unit 25, and outputs the generated auxiliary information to the transmission / reception unit 21. The control device 20 outputs the auxiliary information (step S6) and repeats the processing from step S3. Specifically, in step S6, the transmission / reception unit 21 transmits the auxiliary information to the output device 30, thereby outputting the auxiliary information. Also, if the result of step S3 is No, the control device 20 repeats the processing from step S3.

[0047] When the control device 20 detects the movement of the support device 1, it repeats the process from step S3 or repeats the process from step S1 depending on the amount of movement. If the movement of the support device 1 is within the range where a two-dimensional image has already been acquired, the control device 20 repeats the process from step S3. If the movement of the support device 1 is outside the range where a two-dimensional image has already been acquired, the control device 20 repeats the process from step S1.

[0048] Through the above process, the user can obtain auxiliary information corresponding to the portion traced with the finger on the support device 1. As described above, there are multiple types of auxiliary information, such as phonetic readings and meanings, and the type of auxiliary information generated by the support device 1 may be predetermined or may be set in the support device 1 via the control device 20 or the like. The setting may be performed by the user using the support device 1 or by another user, such as the user's parent or teacher. Furthermore, multiple types of auxiliary information may be generated. For example, the output device 30 may include a display device and an audio output device, and the display device may display the meaning corresponding to the selected portion, and the audio output device may output the selected portion as audio.

[0049] Next, the hardware configuration of the control device 20 of this embodiment will be described. The control device 20 of this embodiment shown in FIG. 6 functions as the control device 20 by executing a computer program, which is a computer program that describes the processing to be performed by the control device 20, on a computer system. FIG. 9 is a diagram showing an example configuration of a computer system that realizes the control device 20 of this embodiment. As shown in FIG. 9, this computer system includes a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the storage unit 103 form a processing circuit.

[0050] In FIG. 9 , the control unit 101 is a processor such as a CPU (Central Processing Unit) and executes a program describing the processing of the control device 20 of this embodiment. Note that a portion of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 includes buttons, a keyboard, a mouse, a touchpad, and the like. The storage unit 103 includes various types of memory such as RAM (Random Access Memory) and ROM (Read Only Memory) and a storage device such as a hard disk, and stores programs to be executed by the control unit 101, necessary data obtained during processing, and the like. The storage unit 103 is also used as a temporary storage area for programs. The display unit 104 is, for example, a display such as an LCD (Liquid Crystal Display). Note that the display unit 104 and the input unit 102 may be integrated and realized by a touch panel or the like. The communication unit 105 is a receiver and a transmitter that perform communication processing. The output unit 106 is a speaker or the like. Note that Fig. 9 is just an example, and the configuration of the computer system is not limited to the example of Fig. 9. For example, the computer system that realizes the control device 20 does not need to include the input unit 102, the display unit 104, and the output unit 106.

[0051] Here, an example of the operation of the computer system until the program of this embodiment is ready to be executed will be described. In the computer system having the above-described configuration, for example, a computer program is installed in storage unit 103 from a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from storage unit 103 is stored in the main storage area of storage unit 103. In this state, control unit 101 executes processing as control device 20 of this embodiment in accordance with the program stored in storage unit 103.

[0052] In the above description, a program describing the processing in the control device 20 is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided via a transmission medium such as the Internet.

[0053] The reading processing unit 22, the position identification unit 23, the character recognition unit 25, and the auxiliary information generation unit 26 shown in FIG. 6 are realized by the control unit 101 shown in FIG. 9 executing a computer program stored in the storage unit 103 shown in FIG. 9. The storage unit 103 shown in FIG. 9 is also used to realize the reading processing unit 22, the position identification unit 23, the character recognition unit 25, and the auxiliary information generation unit 26 shown in FIG. 6. The transmitting / receiving unit 21 shown in FIG. 6 is realized by the communication unit 105 shown in FIG. 9. The information storage unit 24 shown in FIG. 6 is part of the storage unit 103 shown in FIG. 9. The control device 20 may be realized by multiple computer systems. Furthermore, for example, the control device 20 may be realized by a cloud computer system.

[0054] As described above, in this embodiment, the assistance device 1 includes a transparent contact detection unit 3 that has a rectangular sheet shape when viewed from the front and detects contact, and sensors 2 installed on at least one of the left and right ends and the top and bottom ends of the contact detection unit 3. By making the assistance device 1 rectangular, the shape of the assistance device 1 is similar to that of a piece of stationery such as a writing pad, making it possible to prevent a user using the assistance device 1 from standing out in places such as schools. Therefore, by using the assistance device 1 of this embodiment, the user can specify a selection point while reducing the psychological burden on the user.

[0055] Embodiment 2 FIG. 10 is a diagram showing an example of the configuration of a support device according to the second embodiment. FIG. 11 is a side view of the support device according to this embodiment. As shown in FIG. 10, a support device 1a according to this embodiment is obtained by adding a control device 8 to the support device 1 according to the first embodiment, and as shown in FIG. 11, a display device 9 is added. Components having the same functions as those in the first embodiment are given the same reference numerals as those in the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0056] The display device 9 is an example of an output device that outputs auxiliary information and is a transparent, plate-like display arranged to overlap the contact detection unit 3. Specifically, the display device 9 is a transparent, plate-like device provided on the second surface of the contact detection unit 3 so as to overlap the contact detection unit 3. The display device 9 is arranged to be located below the contact detection unit 3 when the assistance device 1a is placed on the target object 50. The transparent, plate-like support unit 5 described in the first embodiment may be further provided below the display device 9 shown in FIG. 11 . When the assistance device 1a is placed on the target object 50, the user can view the displayed object on the display device 9 via the contact detection unit 3. The display device 9 is not limited to a colorless, transparent plate, but may also be a colored, transparent plate, or a polarizing plate that allows the user to select a color when using the display device 9. Some people with learning disabilities have unique color perception. Using a colored, transparent plate as the display device 9 makes it easier for such users to read text. Furthermore, similar to the display device 9, the contact detection unit 3 and the support unit 5 can be made of a colored transparent member or a member such as a polarizing plate whose color can be selected by the user, thereby achieving the same effect.

[0057] 10, the control device 8 is provided on the frame 4, but the location of the control device 8 is not limited to the example shown in FIG. 10. For example, the control device 8 may be integrated with any of the sensors 2-1 to 2-4. In this embodiment, the number and locations of the sensors 2 are also not limited to the example shown in FIG. 10.

[0058] Fig. 12 is a diagram showing an example of the functional configuration of the support device 1a of this embodiment. As shown in Fig. 12, the control device 8 in the support device 1a is the same as the control device 20 of the first embodiment, except that the transceiver unit 21 is removed from the control device 20 of the first embodiment. Note that the control device 8 of the support device 1a may also include the transceiver unit 21 and be capable of communicating with other devices.

[0059] The reading processing unit 22, the position identification unit 23, the information storage unit 24, the character recognition unit 25, and the auxiliary information generation unit 26 are the same as those in the first embodiment, but in this embodiment, a line image is input from the sensor 2 to the reading processing unit 22, and contact position information is input from the contact detection unit 3 to the position identification unit 23. Also, in this embodiment, the auxiliary information generated by the auxiliary information generation unit 26 is output to the display device 9 and displayed by the display device 9.

[0060] 10 shows an example in which the support device 1a includes the control device 8, but the support device 1a may include the control device 20 similar to that of the first embodiment without including the control device 8. FIG. 13 is a diagram showing an example of the configuration of a support system including the support device 1a of this embodiment. In the example shown in FIG. 13, the support device 1a includes a transceiver 6 instead of the control device 8 of the support device 1a shown in FIG. 12. The transceiver 6 communicates with the control device 20 as in the first embodiment. The control device 20 is the same as in the first embodiment, but the auxiliary information generated by the auxiliary information generator 26 is transmitted from the transceiver 21 to the support device 1a. The transceiver 6 of the support device 1a outputs the received auxiliary information to the display device 9, which displays the auxiliary information.

[0061] FIG. 14 is a diagram illustrating a display example of this embodiment in which phonetic readings are displayed as auxiliary information. FIG. 14 illustrates an example in which the support device 1a including the control device 8 illustrated in FIG. 10 is used. As illustrated in FIG. 14, a specifiable area 51, which is a part of the target object 50, is visible via the contact detection unit 3 and the display device 9. In the example illustrated in FIG. 14, a user desires to know the phonetic readings of "kanji" (characters) in the specifiable area 51 as auxiliary information, and traces the part of the specifiable area 51 corresponding to "kanji" (characters) visible via the contact detection unit 3 over the support device 1a with his / her finger. The contact detection unit 3 outputs contact position information indicating the position of the finger contact to the position identification unit 23. As a result, the position identification unit 23, the character recognition unit 25, and the auxiliary information generation unit 26 perform the same operations as those in the first embodiment, and phonetic readings are generated as auxiliary information. In this way, the auxiliary information may include the phonetic readings of the kanji when the selected portion includes the kanji. Furthermore, as shown in FIG. 14, the display device 9 may display the auxiliary information so as to overlay it on the object 50 visually recognized via the contact detection unit 3 and the display device 9.

[0062] In the example shown in FIG. 14 , the auxiliary information is furigana, so the auxiliary information generation unit 26 generates information indicating the character string "kanji" which is the furigana of "kanji" as auxiliary information, and outputs the auxiliary information to the display device 9. As shown in FIG. 14 , the display device 9 displays the auxiliary information received from the auxiliary information generation unit 26 in the auxiliary information display area 52. The auxiliary information display area 52 is, for example, near the portion of the contact detection unit 3 where the user has traced their finger. In the example shown in FIG. 14 , it is to the right of the portion where the user has traced their finger. The auxiliary information display area 52 is not limited to the example shown in FIG. 14 . The position of the auxiliary information display area 52 is determined by the position identification unit 23 based on the contact position information, and the determined auxiliary information display area 52 is output to the display device 9. Alternatively, the position identification unit 23 may output the position of the auxiliary information display area 52 to the auxiliary information generation unit 26, and the auxiliary information generation unit 26 may output the position of the auxiliary information display area 52 together with the auxiliary information to the display device 9.

[0063] When the furigana is displayed on the display device 9, the user can see the furigana through the transparent contact detection unit 3. In FIG. 14, the background of the auxiliary information display area 52 on the display device 9 is displayed in white, but the background of the auxiliary information display area 52 may be made transparent so that the furigana is superimposed on the designatable area 51 and can be seen. The color, size, and font of the furigana may be selected for each user who uses the support device 1 to suit that user. The display device 9 may display the furigana portion in a transparent color so that the designatable area 51 can be seen through the portion where the furigana is displayed.

[0064] FIG. 15 is a diagram showing another display example of the present embodiment in which furigana is displayed as auxiliary information. FIG. 14 shows an example in which the support device 1a including the control device 8 shown in FIG. 10 is used. The example shown in FIG. 15 shows an example in which the location of the display device 9 where the auxiliary information is displayed is predetermined. In the example shown in FIG. 15, the size of the display device 9 seen from the front is smaller than the size of the contact detection unit 3, and the display device 9 is arranged so that it overlaps only a portion of the upper part of the contact detection unit 3. As a result, furigana is displayed in the auxiliary information display area 52 within the upper part of the display device 9. Note that in the example shown in FIG. 15, the size of the display device 9 seen from the front is smaller than the size of the contact detection unit 3. However, if the size of the display device 9 seen from the front and the size of the contact detection unit 3 match, the location where the auxiliary information is displayed may be predetermined regardless of the position selected by the user's finger, as shown in FIG.

[0065] FIG. 16 is a diagram showing a display example of this embodiment in which the meaning of the selected portion is displayed as auxiliary information. FIG. 16 shows an example in which the support device 1a including the control device 8 shown in FIG. 10 is used. In the example shown in FIG. 16, the user selects the portion "schooling," and information indicating the meaning corresponding to "schooling" in simple words (information indicating the meaning) is displayed in the auxiliary information display area 53. In this way, the auxiliary information may be information indicating the selected portion, i.e., the meaning of the words included in the selected portion. The method of generating the auxiliary information is the same as the example shown in FIG. 14, except that information indicating the meaning of words is generated as auxiliary information instead of furigana. In the example shown in FIG. 16, the auxiliary information display area 53 is located above the portion selected by the user, but the position of the auxiliary information display area 53 is not limited to this example.

[0066] FIG. 17 is a diagram showing a display example of this embodiment in which a translation is displayed as auxiliary information. In this manner, the auxiliary information may include a translation of the selected portion. FIG. 17 shows an example in which the support device 1a including the control device 8 shown in FIG. 10 is used. In the example shown in FIG. 17, an English sentence is written in the designatable area 51, and the user selects the part "ubiquitous." The Japanese translation, "It's everywhere," is displayed in the auxiliary information display area 54 as auxiliary information. In the examples shown in FIGS. 14 to 16, vertically written Japanese text is written on the object 50, so the user traces the characters vertically with their finger, as indicated by the arrow. In the case where horizontally written text is written on the object 50, as shown in FIG. 17, the user traces the characters horizontally with their finger, as indicated by the arrow. The method of generating auxiliary information is the same as the example shown in FIG. 14, except that a translation is generated as auxiliary information instead of phonetic readings. In the example shown in FIG. 17, the auxiliary information display area 54 is located above the portion selected by the user, but the position of the auxiliary information display area 54 is not limited to this example.

[0067] FIG. 18 is a diagram showing a display example of this embodiment in which a segmentation position is displayed as auxiliary information. FIG. 18 illustrates an example in which the support device 1a including the control device 8 shown in FIG. 1 is used. In the example shown in FIG. 18, a user selects the portion "especially kanji and long characters," and a segmentation position mark 55 indicating the position where "especially kanji and long characters" is segmented is displayed. Note that in the example shown in FIG. 18, the auxiliary information includes information indicating the segmentation position. The information indicating the segmentation position is information indicating the position where the segmentation position mark 55 is displayed in the designatable area 51, i.e., the position where the segmentation position mark 55 is displayed on the display device 9. In this way, the auxiliary information may include information indicating the position where the selected portion is segmented. When the auxiliary information generation unit 26 segments the character string input from the character recognition unit 25, the auxiliary information generation unit 26 determines the position where the segmentation position mark 55 is displayed based on the segmentation result and the position of each recognized character in the designatable area 51. The display device 9 displays the segmentation position mark 55 at the position indicated by the auxiliary information. The segment position mark 55 is an example of displaying a segment position, and the method of displaying a segment position is not limited to the example shown in FIG.

[0068] FIG. 19 is a diagram showing a display example of this embodiment in which a mask is displayed as auxiliary information. FIG. 19 illustrates an example in which the support device 1a including the control device 8 shown in FIG. 10 is used. In the example shown in FIG. 19, a mask 56 for hiding areas other than the selected area by the user is displayed as auxiliary information. Note that in the example shown in FIG. 19, the auxiliary information includes the mask 56 that covers areas other than the selected area. The auxiliary information generation unit 26 determines the position at which the mask 56 is displayed based on the position in the designatable area 51 of the character string input from the character recognition unit 25. The information indicating the position at which the mask 56 is displayed is information indicating the position at which the mask 56 is displayed in the designatable area 51, i.e., the position at which the mask 56 is displayed on the display device 9. The display device 9 displays the mask 56 based on the information indicating the position at which the mask 56 is displayed. Note that the information indicating the position at which the mask 56 is displayed is also part of the auxiliary information. In this way, by covering areas other than the area of interest with the mask 56, the user can more easily recognize the area of interest.

[0069] In the examples shown in Figures 14 to 19, the support device 1a equipped with the control device 8 shown in Figure 10 has been used as an example, but the support device 1a can also perform similar displays when the support system shown in Figure 13 is used.

[0070] The types of auxiliary information and the method of presenting the auxiliary information to the user are not limited to the examples shown in FIGS. 14 to 19. For example, the auxiliary information may include at least one of information indicating a subject and a predicate corresponding to the selected portion, information indicating a reason for a feeling when the character string corresponding to the selected portion includes a character string indicating the feeling, and images showing a number of objects corresponding to the number when the selected portion includes a number. The auxiliary information may also include at least one of a number indicating the number of objects in an image when the selected portion includes an image showing multiple objects, information obtained by searching for the name on a website when the selected portion includes a name, and the stroke order of the kanji when the selected portion includes kanji. Characters corresponding to the selected portion may also be output as audio as auxiliary information. Multiple types of auxiliary information may be switchably presented. Specifically, for example, the following presentation may be performed. (1) Displaying information to emphasize the subject and predicate of the selected sentence as auxiliary information. Specifically, for example, at least one of a graphic and / or text is displayed to emphasize the subject and predicate. (2) When an expression related to a person's feelings is selected, the reason for the feelings is displayed as supplementary information. Specifically, for example, when a part of a story is selected and a person's feelings are written in the selected part, information indicating the reason for the feelings is displayed. (3) The number of shapes corresponding to the selected number is displayed as auxiliary information. For example, when a user traces a number in an arithmetic textbook or printout, an image of multiple objects, such as multiple apples, is displayed as auxiliary information. (4) When a portion of an image is selected, a number indicating the number of objects in the selected image is displayed as auxiliary information. For example, when a user traces an image of multiple apples, a number indicating the number of apples is displayed as auxiliary information. (5) Information such as the career history of the person with the selected name is displayed as supplementary information. For example, if a user traces the name on a business card, search results for the name are retrieved from websites, and the retrieved information is displayed as supplementary information. (6) The stroke order of the selected kanji is displayed as supplementary information. (7) The characters corresponding to the selected part are output as auxiliary information by voice. (8) A pointer to the next character after the selection is output as auxiliary information so that the user knows where to read next.

[0071] When the display of (4) above is performed, the auxiliary information generation unit 26 of the control device 8 or 20 receives the image extracted by the position identification unit 23 and performs image recognition processing on the received image to calculate the number of objects in the selected image. Alternatively, the control device 8 or 20 may be provided with an image recognition unit that performs image recognition instead of or in addition to the character recognition unit 25, and the image recognition unit may perform image recognition processing to calculate the number of objects in the selected image.

[0072] When the display (5) above is performed, the support device 1a shown in Fig. 10 includes the transmitting / receiving unit 6, and the auxiliary information generating unit 26 acquires search results by setting the name as a search word and searching websites via the Internet or other communication networks via the transmitting / receiving unit 6. When the support system shown in Fig. 13 is used, the auxiliary information generating unit 26 acquires search results by setting the name as a search word and searching websites via the Internet or other communication networks via the transmitting / receiving unit 21.

[0073] Furthermore, the support device 1a may be equipped with an audio output device, and the auxiliary information may be output as audio. FIG. 20 is a diagram showing an example of the configuration of the support device 1b of this embodiment when the auxiliary information is output as audio. In the example shown in FIG. 20, the support device 1b is obtained by adding a speaker 10, which is an example of an audio output device, to the support device 1a shown in FIGS. 10 and 12. The speaker 10 is an example of an output device that outputs auxiliary information, and is an example of an audio output device that outputs auxiliary information as audio. Note that the support device 1b may be equipped with an audio output device such as headphones or earphones instead of or in addition to the speaker 10, or may be equipped with a terminal for connecting headphones, earphones, etc.

[0074] In the example shown in FIG. 20, when the auxiliary information generation unit 26 generates audio data as auxiliary information, the audio data is output to the speaker 10, and the speaker 10 outputs the auxiliary data as audio. Similarly, in the configuration example shown in FIG. 13, the support device 1a may also include the speaker 10. Alternatively, the speaker 10 may be provided separately from the support device 1a. In this case, the support device 1a shown in FIG. 12 includes the transceiver 6, and the auxiliary information generation unit 26 outputs audio data to the speaker 10 via the transceiver 6. Alternatively, the auxiliary information generation unit 26 of the control device 20 shown in FIG. 13 outputs audio data to the speaker 10 via the transceiver 6. Note that the display device 9 may be removed from the support device 1b shown in FIG. 20, and the support device 1b may include only the speaker 10 as a means for outputting auxiliary information. Alternatively, the support device 1b may output audio in combination with other auxiliary information. For example, at the same time as outputting the audio of the relevant part, a highlight of the relevant part may be displayed on the display device 9, or bold text may be displayed on the display device 9 so as to overlap the relevant part. Note that auxiliary information for highlighting the relevant part is not limited to these. Furthermore, the timing for displaying the auxiliary information for highlighting the relevant part on the display device 9 does not have to be simultaneous with the audio output.

[0075] The control device 8 of this embodiment is realized by, for example, the computer system illustrated in Fig. 9, similar to the control device 20 of the first embodiment. Note that the control device 8 does not necessarily have to include the display unit 104, the input unit 102, and the output unit 106.

[0076] As described above, in this embodiment, the support devices 1a and 1b are each equipped with a display device 9. Therefore, by using the support devices 1a and 1b of this embodiment, the same effects as those of the first embodiment can be achieved, and there is no need to use an output device 30 separate from the support devices 1a and 1b. Furthermore, by using a transparent plate-like display overlapping the contact detection unit 3 as the display device 9, the position of the auxiliary information displayed on the display device 9 can be associated with the position where the finger touches the contact detection unit 3. Therefore, the support devices 1a and 1b can display the auxiliary information near a position corresponding to the position traced by the user's finger. This makes it easier for the user to associate and recognize the selected location with the auxiliary information.

[0077] Embodiment 3 FIG. 21 is a diagram showing an example of the configuration of a support system according to the third embodiment. The support system according to the present embodiment includes support devices 1-1 to 1-3 and an information terminal 40. Each of the support devices 1-1 to 1-3 is the support device 1 described in the first embodiment. Although three support devices 1 are shown in FIG. 21, the number of support devices 1 is not limited to the example shown in FIG. 21. Components having the same functions as those in the first and second embodiments are assigned the same reference numerals as those in the first and second embodiments, and redundant explanations will be omitted. Below, differences from the first and second embodiments will be mainly explained.

[0078] In this embodiment, it is assumed that, for example, in an educational setting such as a school or a cram school, a student uses the support device 1 and a teacher uses an information terminal 40. The information terminal 40 is, for example, a tablet terminal, but is not limited to a tablet terminal. In the following, an example in which the support system of this embodiment is applied to an educational setting will be described, but the application of the support system of this embodiment is not limited to educational settings.

[0079] A student, who is an example of a user, places the support device 1 on a textbook or the like, moves the support device 1 over the textbook as described in the first embodiment, and then selects a portion of a sentence written in the textbook by tracing the support device 1 with his / her finger. In this embodiment, as shown in FIG. 21, selection results acquired from each support device 1 are displayed on an information terminal 40 used by a teacher. In the example shown in FIG. 21, it is assumed that support devices 1-1, 1-2, and 1-3 are used by students A, B, and C, respectively, and each support device 1 is associated in advance with the student (student name) who uses it, and information indicating this association is stored as association information in the information terminal 40. As a result, the information terminal 40 can associate the selection results acquired from each support device 1 with each student using the association information and display them as selection results corresponding to each student.

[0080] FIG. 22 is a diagram illustrating an example of the functional configuration of the assistance system of this embodiment. In the example illustrated in FIG. 22, an information terminal 40 includes a transmitter / receiver 21, a reading processor 22, a position identifier 23, an information storage unit 24, and a character recognition unit 25, which are similar to those of the control device 20 of the first embodiment. The information terminal 40 further includes a display information generator 27, an input receiver 41, and a display unit 30a. Note that, in the example illustrated in FIG. 22, the information terminal 40 does not include the auxiliary information generator 26, but this is not a limitation, and the information terminal 40 may further include the auxiliary information generator 26. Also, in the example illustrated in FIG. 22, the information terminal 40 includes the display unit 30a, but this is not a limitation, and a display device may be provided separately from the information terminal 40.

[0081] The input receiving unit 41 receives input from a user such as a teacher. The information storage unit 24 stores correspondence information indicating the correspondence between the identification information of the support device 1 and the student who uses the support device 1. For example, the input receiving unit 41 receives input from a user such as a teacher, and the correspondence information is stored in the information storage unit 24, but the method by which the information terminal 40 acquires the correspondence information is not limited to this.

[0082] In this embodiment, a two-dimensional image is generated for each support device 1 and stored in the information storage unit 24 in association with the support device 1. The contact position information is input to the position identification unit 23 along with the identification information of the support device 1. The position identification unit 23 outputs the processing result together with the identification information of the support device 1 to the character recognition unit 25. The character recognition unit 25 outputs the processing result, a character string, together with the identification information of the support device 1 to the display information generation unit 27. The display information generation unit 27 associates the character string output from the character recognition unit 25 with a student using the correspondence information stored in the information storage unit 24, generates display information showing the selection result (character string) for each student, and outputs the generated display information to the display unit 30a. The display unit 30a displays the display information. Note that, although an example in which the information terminal 40 stores the correspondence information has been described here, the present invention is not limited to this. The support device 1 may store student identification information, such as the student's name, in the information storage unit 24 and transmit the line image and contact position information together with the student's identification information to the information terminal 40.

[0083] The support device 1 of this embodiment transmits the line image and contact position information acquired by the sensor 2 to the information terminal 40. For example, when each student selects a character string that is difficult to understand using the support device 1, the teacher can ascertain which student found which part difficult to understand by checking the display information displayed on the information terminal 40. Furthermore, since the teacher can ascertain which parts multiple students found difficult to understand, the teacher can take measures such as providing supplementary explanations for the parts that multiple students found difficult to understand.

[0084] 22 illustrates an example in which the information terminal 40 includes the reading processing unit 22, the position identification unit 23, and the character recognition unit 25. However, the assistance system may further include the control device 20 described in the first embodiment. In this case, the information terminal 40 does not necessarily include the reading processing unit 22, the position identification unit 23, and the character recognition unit 25. Instead, the information terminal 40 may receive a processing result from the character recognition unit 25 for each assistance device 1 from the control device 20 and generate display information using the received processing result. Alternatively, the control device 20 may also include the display information generation unit 27, generate display information, and display the display information received from the control device 20 on the information terminal 40. Furthermore, when the output device 30 described in the first embodiment is provided in a position visible to the students, the teacher may input information to be presented to each student by operating the input receiving unit 41, and the transmitting / receiving unit 21 may transmit the information to the output device 30, which may then display the information.

[0085] 21 and 22 illustrate an example in which the support device 1 of the first embodiment is used, but a support device including a display device 9 similar to the support device 1a of the second embodiment may also be used. FIG. 23 is a diagram showing another configuration example of the support system of this embodiment. In the example shown in FIG. 23, the support system includes a support device 1c and an information terminal 40a. The information terminal 40a is, for example, a tablet-type terminal and is used by a teacher similar to the information terminal 40. The support device 1c is used by a student, for example, similar to the support device 1 shown in FIGS. 21 and 22.

[0086] The support device 1c is similar to the support device 1a of the second embodiment except that a transmission / reception unit 28 is added to the support device 1a described in the second embodiment. In the example shown in FIG. 23, the support device 1c includes the auxiliary information generation unit 26, but this is not limiting, and the support device 1c does not necessarily include the auxiliary information generation unit 26. The transmission / reception unit 28 of the support device 1c communicates with the information terminal 40a. The transmission / reception unit 28 transmits, for example, the processing result of the character recognition unit 25 to the information terminal 40a.

[0087] The information terminal 40a includes a transmission / reception unit 21, an input acceptance unit 41, a display information generation unit 27, and a display unit 30a. The transmission / reception unit 21 communicates with the support device 1c. For example, when the transmission / reception unit 21 receives a processing result from the character recognition unit 25 from the support device 1c, the transmission / reception unit 21 outputs the received processing result together with identification information of the support device 1c to the display information generation unit 27. The display information generation unit 27 holds the above-mentioned correspondence information, generates display information for displaying the processing result of the character recognition unit 25 in association with the student based on the correspondence information, and displays the generated display information on the display unit 30a.

[0088] Alternatively, the teacher may input information that he or she wishes to present to each student by operating the input receiving unit 41, and the transmitting / receiving unit 21 may transmit the information to the support device 1c. Upon receiving this information, the transmitting / receiving unit 28 of the support device 1c outputs the information to the display device 9, which then displays the information. This allows the students and the teacher to exchange information, enabling two-way communication between the students and the teacher.

[0089] FIG. 24 is a diagram illustrating an example of information exchange between a student and a teacher according to this embodiment. For example, as shown in FIG. 24, a student selects a section about which they want to ask a question by tracing the support device 1c with their finger, and the support device 1c transmits the selected section as a question to the information terminal 40a. The student may handwrite their confusion, question, opinion, etc. on the support device 1c using their finger or a pen. The student's question is displayed on the information terminal 40a, and the teacher inputs an answer corresponding to the question by operating the input receiving unit 41. When the information terminal 40a transmits the answer to the support device 1c and the support device 1c displays the answer, the student can obtain the answer to the question. Alternatively, the teacher may input hints or supplementary information to help them arrive at the answer instead of the answer. In this embodiment, the support device 1c displays the answer, hints, supplementary information, etc., as described above, as examples of auxiliary information. When handwriting is performed on the support device 1c, for example, the position identification unit 23 or the character recognition unit 25 performs character recognition processing to recognize the contact position history indicated by the contact position information as characters.

[0090] Students can also ask questions anonymously. In this case, the information terminal 40a displays the question received from the support device 1c without using the corresponding information, i.e., without displaying the student's name. The teacher, for example, selects a displayed question and inputs an answer corresponding to the question into the information terminal 40a. This allows the teacher to answer the question without knowing which student asked it.

[0091] FIG. 24 shows an example in which an information terminal 40a and a support device 1c are used, but by placing an output device 30 in a position visible to the students and transmitting information input by the teacher to the information terminal 40, information can be exchanged between the students and the teacher in a similar manner using the support device 1 and the information terminal 40.

[0092] Furthermore, the information transmitted from the support device 1c to the information terminal 40a may be transmitted to another support device 1c, and the support device 1c may display the information, thereby allowing the students to share the information. This allows the students to understand the learning progress and thoughts of other students. The support devices 1 and 1c of this embodiment may be provided with a speaker 10, and the auxiliary information and the information received from the information terminals 40 and 40a may be output by the speaker 10.

[0093] The information terminals 40, 40a of this embodiment are realized, for example, by the computer system illustrated in Fig. 9, similar to the control device 20 of the first embodiment. When the information terminals 40, 40a are tablet terminals, the display unit 104 and the input unit 102 may be integrated and realized by a touch panel.

[0094] As described above, in this embodiment, the support device 1, 1c transmits the result of the selection made by the user to the information terminal 40, 40a, and the information terminal 40, 40a displays the received selection result. This allows the user of the support device 1, 1c to perform an input operation to ask a question to a teacher or the like while reducing the psychological burden.

[0095] Embodiment 4 FIG. 25 is a diagram showing an example of the configuration of a support system according to the fourth embodiment. The support system according to the present embodiment includes a support device 1a similar to that of the second embodiment, and a control device 20a. Although one support device 1a is shown in FIG. 25, the number of support devices 1a is not limited to the example shown in FIG. 25. Components having the same functions as those of the first and second embodiments are given the same reference numerals as those of the first and second embodiments, and redundant explanations will be omitted. Below, differences from the first and second embodiments will be mainly explained.

[0096] As shown in Fig. 25, the control device 20a of this embodiment is the same as the control device 20 shown in Fig. 13 of the second embodiment, except that a schedule management unit 29 is added to the control device 20. Note that Fig. 25 shows an example in which the control device 20a includes the auxiliary information generation unit 26, but this is not limiting, and the control device 20a does not necessarily have to include the auxiliary information generation unit 26.

[0097] The schedule management unit 29 manages the schedule of the user of the support device 1a. In this embodiment, the information storage unit 24 stores schedule information indicating the schedule of the user of the support device 1a. The schedule information may be transmitted from another device (not shown) via the transmission / reception unit 21, or the schedule management unit 29 may have a function of receiving input, and may receive input of schedule information from the user and store it in the information storage unit 24.

[0098] FIG. 26 is a diagram illustrating an overview of schedule management according to this embodiment. In this embodiment, the support device 1a displays, as an example of auxiliary information, information related to a schedule, such as whether or not a schedule can be added. For example, a user places the support device 1a on an object 60 on which the date, time, location, and other information of an event are written, moves the object 60, and then a two-dimensional image is acquired. The user then traces the portion of the object 60 indicating the date and time with his / her finger via the support device 1a. Examples of events include, but are not limited to, school events, events, and meetings. Similarly to the second embodiment, the support device 1a transmits a line image and contact position information corresponding to the portion indicating the date and time to the control device 20a. The user may also select information indicating the location, and contact position information corresponding to the portion indicating the location may be transmitted to the control device 20a. For simplicity, in FIG. 26, the contact position information corresponding to the portion indicating the date, time, and location is described as "date, time, and location."

[0099] When the control device 20a receives a line image from the support device 1a, it generates a two-dimensional image in the same manner as in the second embodiment. Furthermore, when the control device 20a receives contact position information corresponding to a location indicating a date and time (or a date and time and a location), it identifies the location in the two-dimensional image indicated by the contact position information and performs character recognition on the extracted image based on the identification result. The character string obtained by the character recognition, i.e., the character string indicating the date and time selected by the user, is stored in the information storage unit 24. The schedule management unit 29 uses the character string corresponding to the date and time selected by the user and the schedule information stored in the information storage unit 24 to determine whether or not an event for the date and time (or a date and time and a location) selected by the user can be added, and transmits the determination result, i.e., whether or not the event can be added, to the support device 1a via the transmission / reception unit 21. The support device 1a displays information indicating whether or not the event can be added, i.e., whether or not an event can be added to the user's schedule.

[0100] At this time, if it is determined that adding a schedule is possible, the support device 1a may display a display screen prompting the user to select whether or not to add a schedule. For example, the schedule management unit 29 may generate display information for displaying the display screen and transmit it to the support device 1a. When the transmission / reception unit 6 of the support device 1a receives the display information, it outputs it to the display device 9, which then displays the display information. When the contact detection unit 3 of the support device 1a receives from the user a selection result of whether or not to add a schedule, it transmits the received selection result to the control device 20a via the transmission / reception unit 6. When the transmission / reception unit 21 of the control device 20a receives the selection result from the support device 1a, it outputs the received selection result to the schedule management unit 29. When the selection result indicates that a schedule should be added, the schedule management unit 29 adds the schedule to the schedule information.

[0101] While an example in which the schedule management unit 29 is added to the control device 20 shown in Fig. 13 of the support system of the second embodiment has been described above, the present invention is not limited to this. Alternatively, the schedule management unit 29 may be added to the control device 8 of the support device 1a shown in Fig. 12 of the second embodiment and the schedule management process described above may be performed. Alternatively, the schedule management unit 29 may be added to the control device 8 of the support device 1b shown in Fig. 20 of the second embodiment and the schedule management process described above may be performed. Furthermore, the schedule management unit 29 may be added to the information terminal 40 or the support device 1c of the third embodiment and the schedule management process described above may be performed.

[0102] The control device 20a of this embodiment, like the control device 20 of the first embodiment, is realized by, for example, the computer system illustrated in Fig. 9. The control device 20a may be realized by a plurality of computer systems. Also, for example, the control device 20a may be realized by a cloud computer system.

[0103] As described above, in this embodiment, the schedule management unit 29 manages the user's schedule based on the event date and time input using the support device 1a. This allows the user to add the event schedule written on the object 60, such as a print, to the schedule information with a simple operation.

[0104] Embodiment 5. FIG. 27 is a diagram showing an example of the configuration of a support system according to a fifth embodiment. The support system according to this embodiment includes a support device 1d and a control device 20b. Although one support device 1d is shown in FIG. 27, the number of support devices 1d is not limited to the example shown in FIG. 27. Components having the same functions as those in the first and second embodiments are given the same reference numerals as those in the first and second embodiments, and redundant explanations will be omitted. Below, differences from the first and second embodiments will be mainly explained.

[0105] The support device 1d is similar to the support device 1a shown in FIG. 13 of the second embodiment, except that a self-location identification unit 11 is added. As the self-location identification unit 11, for example, a GPS receiver that performs GPS (Global Positioning System) positioning can be used, but is not limited to this. The self-location identification unit 11 identifies the position of the support device 1d and outputs position information indicating the identified position to the transmitter / receiver 6. The transmitter / receiver 6 transmits the position information received from the self-location identification unit 11 to the control device 20b. The control device 20b is similar to the control device 20 shown in FIG. 13 of the second embodiment, except that a route guidance unit 31 is added.

[0106] Here, an example of how the assistance system of this embodiment is used will be described. In this embodiment, the assistance device 1d displays, as an example of auxiliary information, a route to a destination specified by the user. In this manner, the auxiliary information may be a route from the current position (present location) to the destination. FIG. 28 is a diagram showing an example of how the assistance system of this embodiment is used. For example, the user moves the assistance device 1d of this embodiment over an object 70 on which a map is drawn, such as a paper map, to acquire a line image in the same manner as in the second embodiment, and transmits the acquired line image to the control device 20b. The control device 20b generates a two-dimensional image using the line image, similar to the control device 20 of the second embodiment. The control device 20b identifies the current position of the assistance device 1d in the two-dimensional image showing the map, and uses the identified position to extract from the two-dimensional image an image of an area corresponding to the display area of the display device 9 of the assistance device 1d. Based on the position information received from the assistance device 1d, the control device 20b determines the position of the assistance device 1d in the extracted image, i.e., the position of the assistance device 1d in the display area that the display device 9 can display.

[0107] The control device 20b generates display information for displaying a mark 71 indicating the current location of the support device 1d and the text "Current Location" at the determined position, and transmits the generated display information to the support device 1d. When the transmitter / receiver 6 of the support device 1d receives the display information, it outputs the display information to the display device 9, and the display device 9 displays the display information. As a result, as shown in FIG. 28, the round mark 71 indicating the current location and the text "Current Location" are displayed on the display device 9 of the support device 1d at a position corresponding to the current location of the support device 1d.

[0108] 28, the user uses his / her finger to specify a destination on a map of an object 70 visible through the support device 1d. The support device 1d transmits the position specified by the user to the control device 20b as contact position information. Upon receiving the contact position information specifying the destination, the control device 20b identifies the position of the destination in an area corresponding to the display area of the display device 9 of the support device 1d based on the contact position information, generates display information for displaying a mark 72 indicating the destination at the identified position, and transmits the generated display information to the support device 1d.

[0109] When the transmitting / receiving unit 6 of the support device 1d receives the display information, it outputs the display information to the display device 9, and the display device 9 displays the display information. As a result, as shown in FIG. 28, a round mark 72 indicating the destination is displayed at a position corresponding to the destination on the display device 9 of the support device 1d. Note that the marks and characters indicating the current position and the destination are not limited to the example shown in FIG. 28. The control device 20b further generates display information indicating a route 73 from the current position to the destination, and transmits the display information to the support device 1d. When the transmitting / receiving unit 6 of the support device 1d receives the display information, it outputs the display information to the display device 9, and the display device 9 displays the display information. As a result, the route 73 from the current position to the destination is displayed on the support device 1d.

[0110] Next, the operation of the control device 20b will be described in detail. In this embodiment, as in the second embodiment, the reading processing unit 22 generates a two-dimensional image using a line image and stores the image in the information storage unit 24. When the object 70 is a map, the position identification unit 23 identifies the current position of the support device 1d in the two-dimensional image, extracts an image of an area corresponding to the display area of the display device 9 of the support device 1d from the two-dimensional image using the identified position, and stores the extracted image in the information storage unit 24 as an extracted map. As described in the first embodiment, the position may be identified using the displacement sensor 7, or may be identified by comparing the two-dimensional image with a line image acquired by the sensor 2. Whether the object 70 is a map may be specified by the support device 1d receiving an input from the user, or the position identification unit 23 may automatically determine this from the two-dimensional image.

[0111] When the transmitter / receiver 21 of the control device 20b receives location information from the support device 1d, it outputs the received location information to the route guidance unit 31. In this embodiment, map information indicating a map is stored in the information storage unit 24. The route guidance unit 31 performs image processing on the extracted map stored in the information storage unit 24 to recognize roads, topography, structures, etc. on the extracted map, and compares the recognition results with the map information to identify which part of the map indicated by the map information the extracted map corresponds to. Note that the character recognition unit 25 may recognize characters on the extracted map and add the recognized characters to the extracted map, and the route guidance unit 31 may further use the characters recognized by the character recognition unit 25 to identify which part of the map indicated by the map information the extracted map corresponds to. The route guidance unit 31 calculates the latitude and longitude corresponding to the extracted map using the result of identifying which part of the map indicated by the map information the extracted map corresponds to. For example, the route guidance unit 31 calculates the latitude and longitude corresponding to the four corners of the extracted map. The route guidance unit 31 uses the calculated latitude and longitude and the position information received from the transmission / reception unit 21 to determine the position of the support device 1d within the extracted map (extracted image), i.e., the position of the support device 1d in the display area that can be displayed by the display device 9. The route guidance unit 31 generates display information for displaying a mark 71 indicating the current position of the support device 1d and the words "current position" at the determined position, and outputs the display information to the transmission / reception unit 21.

[0112] Furthermore, when the position specifying unit 23 receives contact position information indicating the destination from the assistance device 1d via the transmitting / receiving unit 21, the position specifying unit 23 stores the position of the destination in the display area of the display device 9 in the information storage unit 24 based on the contact position information. The route guidance unit 31 uses the position of the destination stored in the information storage unit 24 to generate display information for displaying a mark 72 indicating the destination at that position, and outputs the generated display information to the transmitting / receiving unit 21.

[0113] Furthermore, the route guidance unit 31 calculates the latitude and longitude of the destination using the location of the destination stored in the information storage unit 24 and the latitude and longitude corresponding to the extracted map. The route guidance unit 31 searches for a route from the current location to the destination using the current location of the assistance device 1d indicated by the location information, the latitude and longitude of the destination, and the map information stored in the information storage unit 24, generates display information for displaying the route obtained by the search, and outputs the generated display information to the transmission / reception unit 21. A general method can be used as the route search method, so a detailed description thereof will be omitted.

[0114] The transmitter / receiver 21 transmits the display information received from the route guidance unit 31 to the support device 1d. Upon receiving the display information, the support device 1d displays the display information. Note that in the example shown in FIG. 28, a destination is selected and a route 73 from the current location to the destination is displayed on the support device 1d, but this is not limiting, and only information indicating the current location may be displayed on the support device 1d. For example, only the mark 71 may be displayed on the support device 1d, or the mark 71 and the words "current location" may be displayed.

[0115] Here, an example has been described in which the route guidance unit 31 is added to the control device 20 shown in FIG. 13 of the support system of the second embodiment, but the present invention is not limited to this. Alternatively, the route guidance unit 31 may be added to the control device 8 of the support device 1a shown in FIG. 12 of the second embodiment, and the process related to the route search described above may be performed. Alternatively, the route guidance unit 31 may be added to the control device 8 of the support device 1b shown in FIG. 20 of the second embodiment, and the process related to the route search described above may be performed. Furthermore, the route guidance unit 31 may be added to the information terminal 40 or the support device 1c of the third embodiment, and the process related to the route search described above may be performed. Furthermore, this embodiment may be combined with the fourth embodiment, or this embodiment may be combined with the third and fourth embodiments.

[0116] The control device 20b of this embodiment, like the control device 20 of the first embodiment, is realized by, for example, the computer system illustrated in Fig. 9. The control device 20b may be realized by a plurality of computer systems. Also, for example, the control device 20b may be realized by a cloud computer system.

[0117] As described above, in this embodiment, when the object 70 read by the support device 1d is a map, the support device 1d is configured to display the current position of the support device 1d. This allows the user to check the current position on the map. Furthermore, the support device 1d accepts input of a destination and displays a route from the current position to the destination, allowing the user to understand the route to the destination.

[0118] Embodiment 6 FIG. 29 is a diagram showing an example of the configuration of a support system according to a sixth embodiment. The support system according to this embodiment includes a support device 1a and a control device 20c. Although one support device 1a is shown in FIG. 29, the number of support devices 1a is not limited to the example shown in FIG. 29. Components having the same functions as those in the first and second embodiments are given the same reference numerals as those in the first and second embodiments, and redundant explanations will be omitted. Below, differences from the first and second embodiments will be mainly explained.

[0119] The support device 1a is similar to the support device 1a shown in Fig. 13 of the second embodiment. The control device 20c is similar to the control device 20 shown in Fig. 13 of the second embodiment, except that a musical score recognition unit 32 is added.

[0120] Here, an example of how the assistance system of this embodiment is used will be described. In this embodiment, information for playing a selected note on a musical instrument is displayed on the assistance device 1a as an example of auxiliary information. FIG. 30 is a diagram showing an example of how the assistance system of this embodiment is used. For example, a user moves the assistance device 1a of this embodiment over an object 80 on which musical scores are written to acquire a line image in the same manner as in the second embodiment, and transmits the acquired line image to the control device 20c. The control device 20c generates a two-dimensional image using the line image, similar to the control device 20 of the second embodiment. The user stops the assistance device 1a and specifies a note on the object 80 for which auxiliary information is desired by tracing it with a finger or drawing a mark 81 such as a circle with a finger to surround the relevant portion on the assistance device 1a. In the example shown in FIG. 30, the note for which auxiliary information is desired is surrounded by a mark 81. In the example shown in FIG. 30, the position of the fingers on a keyboard instrument such as a piano when playing the note surrounded by the mark 81 is displayed as performance information 82, an example of auxiliary information. In the example shown in Fig. 30, the performance information 82 includes an image of a keyboard and finger positions on the keyboard. In addition, the auxiliary information may be information indicating the placement of fingers for playing the selected part on an instrument. In the example shown in Fig. 30, the finger positions on the keyboard are indicated by finger numbers 83, but the information indicating finger positions is not limited to this example.

[0121] 30 illustrates a keyboard instrument as an example of a musical instrument, but the musical instrument is not limited to a keyboard instrument and may be a wind instrument such as a recorder, a string instrument such as a guitar, or the like. For example, in the case of a recorder, an image of the recorder and information indicating the finger positions are displayed as performance information 82. In the case of a string instrument, for example, an image of the neck of the instrument and information indicating the finger positions are displayed. The manner in which performance information 82 is displayed is not limited to these examples.

[0122] Next, the operation of the control device 20c will be described in detail. In this embodiment, as in the second embodiment, the reading processing unit 22 generates a two-dimensional image using a line image and stores the image in the information storage unit 24. When the object 80 is a musical score, the position identification unit 23 identifies the current position of the support device 1a in the two-dimensional image, uses the identified position to extract an image of an area corresponding to the display area of the display device 9 of the support device 1a from the two-dimensional image, and stores the extracted image as a selected note in the information storage unit 24. As described in the first embodiment, the position may be identified using the displacement sensor 7, or may be identified by comparing the two-dimensional image with the line image acquired by the sensor 2. Whether the object 80 is a musical score may be specified by the support device 1a by receiving an input from the user, or the position identification unit 23 may automatically determine this from the two-dimensional image.

[0123] Next, the operation of the control device 20c will be described in detail. In this embodiment, as in the second embodiment, the reading processing unit 22 generates a two-dimensional image using a line image and stores the image in the information storage unit 24. When the object 80 is a musical score, the position identification unit 23 receives contact position information indicating a position selected by the user from the support device 1a via the transmission / reception unit 21. As in the second embodiment, the position identification unit 23 identifies the position indicated by the contact position information in the two-dimensional image and extracts an image corresponding to the identified position from the two-dimensional image. The position identification unit 23 outputs the extracted image to the score recognition unit 32. The score recognition unit 32 performs image processing on the selected note, which is the image received from the position identification unit 23, to identify the scale name of the note in the image. The score recognition unit 32 generates performance information 82 corresponding to the identified scale name as auxiliary information and outputs the generated performance information 82 to the transmission / reception unit 21. For example, the information storage unit 24 stores in advance images of musical instruments and finger positions corresponding to the scale names as instrument information. The score recognition unit 32 generates the performance information 82 using the instrument information stored in the information storage unit 24. As shown in FIG. 30, when multiple notes are selected as selected notes, the finger number 83 for playing the instrument for each combination of multiple scale names may be stored as instrument information.

[0124] The transmitting / receiving unit 21 transmits the performance information 82 received from the musical score recognition unit 32 to the support device 1a. As a result, the support device 1a can display the performance information 82 as shown in FIG. 30. The musical score recognition unit 32 may also generate audio data for outputting sounds corresponding to musical notes as auxiliary information, instead of or in addition to the performance information 82. In this case, the support device 1a may include a speaker 10, which outputs the audio data as audio, or an output device 30 may be provided separately from the support device 1a, which outputs the audio data as audio.

[0125] Here, an example has been described in which the score recognition unit 32 is added to the control device 20 shown in FIG. 13 of the support system of the second embodiment. However, the present invention is not limited to this. The score recognition unit 32 may be added to the control device 8 of the support device 1a shown in FIG. 12 of the second embodiment, and the above-mentioned processing related to the performance information 82 may be performed. Alternatively, the score recognition unit 32 may be added to the control device 8 of the support device 1b shown in FIG. 20 of the second embodiment, and the above-mentioned processing related to the performance information 82 may be performed. Furthermore, the score recognition unit 32 may be added to the information terminal 40 or the support device 1c of the third embodiment, and the above-mentioned processing related to the performance information 82 may be performed. Furthermore, this embodiment may be combined with one or more of the third to fifth embodiments.

[0126] The control device 20c of this embodiment, like the control device 20 of the first embodiment, is realized by, for example, the computer system illustrated in Fig. 9. The control device 20c may be realized by a plurality of computer systems. Also, for example, the control device 20c may be realized by a cloud computer system.

[0127] As described above, in this embodiment, when the object 80 read by the support device 1a is a musical score, the support device 1a is configured to display performance information 82 for playing an instrument. This makes it possible to support the user in playing an instrument. Furthermore, by outputting sounds corresponding to the notes as auxiliary information, the user can check whether or not their performance is correct.

[0128] Embodiment 7 FIG. 31 is a diagram illustrating an example of the configuration of an assistance device according to a seventh embodiment. In each of the above-described embodiments, an example has been described in which an image of an object 50 is acquired using a sensor 2 that is a line sensor. In this embodiment, an example will be described in which a camera 61 is used instead of the sensor 2. An assistance device 1e according to this embodiment includes a contact detection unit 3, a support unit 5, a frame unit 4, a camera 61, a contact sensor 62, and a transmission / reception unit 6. Note that the transmission / reception unit 6 is not illustrated in FIG. 31. The contact detection unit 3, the support unit 5, the frame unit 4, and the transmission / reception unit 6 are the same as those in the first embodiment. As described in the first embodiment, the support unit 5 may be integrated with the frame unit 4. Components having the same functions as those in the first embodiment are denoted by the same reference numerals as those in the first and second embodiments, and redundant description will be omitted. Below, differences from the first embodiment will be mainly described.

[0129] The camera 61 in this embodiment is a small camera capable of acquiring two-dimensional images. In the example shown in FIG. 31 , the support device 1e includes two cameras 61, but the number of cameras 61 included in the support device 1e is not limited to this, and may be one or more. The contact sensor 62 is a sensor that detects contact, i.e., a sensor that detects contact with an external object, etc. The contact sensor 62 may be integrated with the camera 61. In the example shown in FIG. 31 , the support device 1e includes two contact sensors 62, but the number of contact sensors 62 included in the support device 1e is not limited to this, and may be one or more. The arrangement positions of the cameras 61 and the contact sensors 62 shown in FIG. 31 are merely an example, and the arrangement positions of the cameras 61 and the contact sensors 62 are not limited to the example shown in FIG. 31 . The cameras 61 and the contact sensors 62 are provided, for example, at the end of the contact detection unit 3.

[0130] Fig. 32 is a diagram showing an example of the configuration of the support system of this embodiment. Fig. 32 shows an example of the functional configuration of each device, and omits the illustration of the frame unit 4 and support unit 5 of the support device 1. Also, while Fig. 6 shows one camera 61 and one contact sensor 62, as described above, there may be multiple cameras 61 and multiple contact sensors 62. The control device 20 is similar to the example shown in Fig. 6 of the first embodiment, except that the processing of the reading processing unit 22 and some of the processing of the character recognition unit 25 are different from the example shown in Fig. 6 of the first embodiment.

[0131] Returning to the explanation of FIG. 31 , the field of view range 202 shown in FIG. 31 indicates the range corresponding to the field of view of the camera 61, i.e., the range that can be captured by the camera 61. In the example shown in FIG. 31 , an object 50 is placed on a desk 201, and a user moves the support device 1e in a movement direction 203 from a position above the object 50 and away from the object 50. That is, the user moves the support device 1e so as to approach the object 50. During this movement, the camera 61 of the support device 1e continuously captures images of the object 50. Note that while the support device 1e moves so as to approach the object 50, the camera 61 continues to focus on the object 50. The captured images are transmitted to the control device 20 by the transmission / reception unit 6. The detection result of the contact sensor 62 is also transmitted to the control device 20 by the transmission / reception unit 6. The detection result may be information indicating whether or not contact has occurred, or may be information indicating that contact has occurred. In the former case, the detection result is transmitted periodically, and in the latter case, the detection result is transmitted when the contact sensor 62 detects contact.

[0132] When the control device 20 recognizes a recognition target such as characters in the received image, it starts saving the image and sequentially saves continuously acquired images until the support device 1e comes into contact with the target 50, i.e., until the contact sensor 62 detects contact. As a result, images taken continuously from the time the recognition target is recognized until the support device 1e comes into contact with the target 50 are saved. The control device 20 uses the saved images to generate an image (two-dimensional image) corresponding to the designatable area 51 of the support device 1e.

[0133] In detail, in this embodiment, character recognition unit 25 shown in FIG. 32 receives an image captured by camera 61 via transmission / reception unit 21, determines whether a recognition target such as a character has been recognized in the captured image, and, if a recognition target such as a character has been recognized in the image, stores the image in information storage unit 24. Then, character recognition unit 25 receives a detection result from contact sensor 62 via transmission / reception unit 21, determines whether contact sensor 62 has detected contact based on the detection result, and stores the received images in information storage unit 24 sequentially until contact sensor 62 detects contact. When contact sensor 62 detects contact, character recognition unit 25 notifies reading processing unit 22 of that fact. Note that the above processing of character recognition unit 25 may be performed by reading processing unit 22.

[0134] When the reading processing unit 22 is notified by the character recognition unit 25 that the contact sensor 62 has detected contact, the reading processing unit 22 generates an image corresponding to the designatable area 51 using continuously captured images stored in the information storage unit 24, and stores the generated image in the information storage unit 24. The range within the object 50 that the camera 61 can capture narrows as the support device 1e approaches the object 50. Therefore, when the support device 1e is in contact with the object 50, it is difficult to capture the entire designatable area 51, which is the area below the contact detection unit 3. On the other hand, when the support device 1e is located farther from the object 50, the camera 61 can capture a wider range than when the support device 1e is in contact with the object 50, and therefore it is possible to capture the entire designatable area 51. The reading processing unit 22 calculates the image corresponding to the designatable area 51 using, for example, multiple images taken at different distances from the object 50 and the position and orientation of the camera 61 on the support device 1e. At least a part of the processing in the reading processing unit 22 described above may be performed by the character recognition unit 25.

[0135] FIG. 33 is a diagram illustrating the movement and captureable range of the support device 1e according to the seventh embodiment. FIG. 33 illustrates an image of the angle of view 202 of the camera 63 as viewed from the side, and shows an example of the range of the object 50 corresponding to images captured while the support device 1e is moving closer to the object 50. Note that FIG. 33 illustrates three representative positional relationships between the support device 1e and the object 50 at different distances. However, this does not indicate the number of images captured when the support device 1e is brought closer to the object 50; it is sufficient that multiple images are captured while the support device 1e is brought closer. As shown in FIG. 33, as the support device 1e is brought closer to the object 50, the captureable range of the object 50 within the angle of view 202 of the camera 63 narrows depending on the distance between the support device 1e and the object 50, but overlapping areas of the object 50 are captured in each image. Therefore, by recognizing characters and the like in the image, the reading processing unit 22 can identify which area was captured within an image capturing a wider range that would have been acquired if the support device 1e had been further away from the object 50 when the support device 1e approached the object 50. The reading processing unit 22 can generate an image of the designatable area 51 when the support device 1e comes into contact with the object 50, using the identified result, the stored image, the position and orientation of the camera 61 in the support device 1e (the positional relationship between the camera 61 and the contact detection unit 3), and the like.

[0136] The processing after the image of the designatable area 51 is generated is the same as in the first embodiment. Note that in the above example, the camera 61 and the contact sensor 62 are used instead of the sensor 2 of the support device 1 of the first embodiment. However, this is not limited to this. The support devices 1a and 1b described in the second embodiment may use the camera 61 and the contact sensor 62 instead of the sensor 2, and the control device 20 or the control device 8 may perform the processing to generate an image corresponding to the designatable area 51 described above. Also, in the third embodiment, the support device 1e may be used instead of the support device 1, and the information terminal 40 may perform the processing to generate an image corresponding to the designatable area 51 described above. Alternatively, the support device 1c may be provided with the camera 61 and the contact sensor 62 instead of the sensor 2 and may perform the processing to generate an image corresponding to the designatable area 51. Similarly, the support devices 1a and 1d described in the third to fifth embodiments may perform the processing to generate an image corresponding to the designatable area 51. Also, this embodiment may be applied to a combination of two or more embodiments from the first to fifth embodiments.

[0137] In this embodiment, an image of the designatable area 51 of the object 50 can be generated without using the displacement sensor 7. Furthermore, when using the sensor 2 that is a line sensor as in the first embodiment, the user needs to slide the support devices 1, 1a, 1b, 1c, and 1d to acquire an image of the designatable area 51 of the object 50, but in this embodiment, the user does not need to slide the support device 1e.

[0138] Embodiment 8 FIG. 34 is a diagram showing an example of the configuration of a support device according to the eighth embodiment. In the seventh embodiment, camera 61, which is a small camera, is used, but in this embodiment, an example will be described in which camera 63, which is a close-up type small camera with a wide angle of view, is used. Support device 1f of this embodiment is the same as support device 1e of the seventh embodiment, except that it is equipped with camera 63 instead of camera 61. Components having the same functions as those of the first and seventh embodiments are assigned the same reference numerals as those of the first and seventh embodiments, and duplicated explanations will be omitted. Below, differences from the first and seventh embodiments will be mainly explained.

[0139] FIG. 35 is a side view of a support device 1f according to the eighth embodiment. The support device 1f according to the eighth embodiment includes a contact detection unit 3, a support unit 5, a frame unit 4, a camera 63, a contact sensor 62, and a transmission / reception unit 6. Note that the transmission / reception unit 6 is not shown in FIGS. 34 and 35. The contact detection unit 3, the support unit 5, the frame unit 4, the transmission / reception unit 6, and the contact sensor 62 are the same as those in the seventh embodiment. As described in the first embodiment, the support unit 5 may be integrated with the frame unit 4.

[0140] The angle of view range 204 shown in FIG. 35 indicates the range corresponding to the angle of view of the camera 63, i.e., the range that can be captured by the camera 63. As shown in FIG. 35, the camera 63 of this embodiment is a small, close-up camera with a wide angle of view, and therefore the angle of view range 204 is wider than that of the seventh embodiment. In this embodiment, as shown in FIG. 35, the support device 1f is slid while in contact with the object 50. The camera 63 continuously captures images when the support device 1f contacts the object 50. This makes it possible to capture an image of the entire designatable area 51 of the object 50. As in the seventh embodiment, the contact sensor 62 can detect that the support device 1f has contacted the object 50. Note that although two cameras 63 are used in FIGS. 34 and 35, the number of cameras 63 may be one or more, and the number, angle of view, and arrangement of the cameras 63 are not limited to the examples shown in FIGS. 34 and 35. Furthermore, if the camera 63 can capture the entire designatable area 51, the sliding operation does not need to be performed.

[0141] Fig. 36 is a diagram showing an example of the configuration of the support system of this embodiment. Fig. 36 shows an example of the functional configuration of each device, and omits the illustration of the frame unit 4 and support unit 5 of the support device 1. Also, Fig. 36 shows one camera 63 and one contact sensor 62, but as described above, there may be multiple cameras 63 and multiple contact sensors 62. The control device 20 is similar to the example shown in Fig. 6 of the first embodiment, except that the processing of the reading processing unit 22 is different from the example shown in Fig. 6 of the first embodiment.

[0142] The image captured by the camera 63 and the detection result by the contact sensor 62 are transmitted to the control device 20 by the transmitting / receiving unit 6. When there are multiple cameras 63, the reading processing unit 22 of the control device 20 generates an image of the designatable area 51 using the multiple images captured by each camera 63 received via the transmitting / receiving unit 21, and stores the generated image in the information storage unit 24. Note that when there is only one camera 63 and it is possible to capture an image of the entire designatable area 51, the control device 20 does not need to include the reading processing unit 22, and the image received from the support device 1f may be stored in the information storage unit 24.

[0143] Furthermore, similar to the seventh embodiment, the user may move the support device 1f from a position above the object 50 away from the object 50 so as to approach the object 50. In this case, the control device 20 can generate an image of the designatable area 51 by the same processing as in the seventh embodiment. Furthermore, the user may move the support device 1f from a position above the object 50 away from the object 50 so as to approach the object 50, and then bring the support device 1f into contact with the object 50 and slide it. In this case, the control device 20 can generate an image of the designatable area 51 by using images acquired both while the support device 1f is being moved closer to the object 50 and while the support device 1f is being slid.

[0144] The processing after the image of the designatable area 51 is generated is the same as in the first embodiment. Note that in the above example, the camera 63 and the contact sensor 62 are used instead of the sensor 2 of the support device 1 of the first embodiment. However, this is not limited to this. The support devices 1a and 1b described in the second embodiment may use the camera 63 and the contact sensor 62 instead of the sensor 2, and the control device 20 or the control device 8 may perform the processing to generate an image corresponding to the designatable area 51 described above. Also, in the third embodiment, the support device 1f may be used instead of the support device 1, and the information terminal 40 may perform the processing to generate an image corresponding to the designatable area 51 described above. Alternatively, the support device 1c may be provided with the camera 63 and the contact sensor 62 instead of the sensor 2 and may perform the processing to generate an image corresponding to the designatable area 51. Similarly, the support devices 1a and 1d described in the third to fifth embodiments may perform the processing to generate an image corresponding to the designatable area 51. Also, this embodiment may be applied to a combination of two or more embodiments from the first to fifth embodiments.

[0145] In this embodiment, an image of the specifiable area 51 of the object 50 can be generated without using the displacement sensor 7. Moreover, an image of the specifiable area 51 of the object 50 can be acquired even when the support device 1f is in contact with the object 50.

[0146] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0147] Various aspects of the present disclosure are summarized below as appendices.

[0148] (Appendix 1) a transparent sheet-like contact detection unit that detects contact with an object due to a user's movement; a line sensor provided at an end of the contact detection unit and having image pickup elements arranged in one direction; An assistance device comprising: (Appendix 2) the contact detection unit includes a first surface that is a surface that is in contact with the object, and a second surface that is a reverse surface of the first surface, the imaging element is placed on an object to be imaged with the first surface facing up; The assistance device according to claim 1, wherein the contact detection unit detects a position where the object contacts a selected area, which is a location selected by the user, within an area of the object visible to the user. (Appendix 3) The support device described in Appendix 2 is characterized in that the selected location corresponding to the contact position is determined based on a two-dimensional image of the object obtained by placing the support device on the object and moving it in a direction perpendicular to the arrangement direction of the imaging elements. (Appendix 4) an output device that outputs auxiliary information corresponding to the selected portion; 4. The assistance device according to claim 3, comprising: (Appendix 5) the output device includes a transparent plate-like display device provided on the second surface side so as to overlap the contact detection unit, The support device according to claim 4, wherein the display device displays the auxiliary information. (Appendix 6) 6. The assistance device according to claim 5, wherein the display device displays the auxiliary information so as to overlay it on an object visually recognized through the contact detection unit and the display device. (Appendix 7) The object has text written on it, The support device according to claim 5 or 6, wherein the auxiliary information includes furigana for a kanji character when the selected portion includes the kanji character. (Appendix 8) The object has text written on it, 8. The support device according to claim 5, wherein the auxiliary information includes information indicating the meaning of the selected portion. (Appendix 9) The object has text written on it, 9. The support device according to claim 5, wherein the auxiliary information includes a translation of the selected passage. (Appendix 10) The object has text written on it, 10. The support device according to any one of appendices 5 to 9, wherein the auxiliary information includes information indicating a segmented portion of the selected portion. (Appendix 11) The object has text written on it, 11. The assistance device according to any one of appendices 5 to 10, wherein the auxiliary information includes a mask that covers an area other than the selected portion. (Appendix 12) The object has text written on it, 12. The support device according to any one of appendices 5 to 11, wherein the auxiliary information includes information indicating a subject and a predicate corresponding to the selected portion. (Appendix 13) The object has text written on it, The support device according to any one of appendices 5 to 12, characterized in that the auxiliary information includes information indicating the reason for the feeling when the character string corresponding to the selected portion includes a character string indicating the feeling. (Appendix 14) The object has text written on it, The assistance device according to any one of appendices 5 to 13, wherein the auxiliary information includes an image showing an object of a number corresponding to the number when the selected location includes a number. (Appendix 15) The object has an image thereon; 15. The assistance device according to any one of appendices 5 to 14, wherein the auxiliary information includes a number indicating the number of objects in an image when the selected location includes an image showing multiple objects. (Appendix 16) The object has text written on it, An assistance device described in any one of Appendices 5 to 15, characterized in that the auxiliary information includes information obtained by searching for a name on a website if the selected portion contains a name. (Appendix 17) The object has text written on it, 17. The support device according to any one of appendices 5 to 16, wherein the auxiliary information includes the stroke order of a kanji character when the selected portion includes the kanji character. (Appendix 18) The object has the date and time of the event written on it; 18. The support device according to any one of appendices 5 to 17, wherein the auxiliary information includes information indicating whether the event can be added to the user's schedule. (Appendix 19) a self-location determination unit that determines a self-location; Equipped with The object has a map printed on it, 19. The support device of claim 5, wherein the display device includes information indicating a current position of the support device on the map. (Appendix 20) 20. The assistance device of claim 19, wherein the assistance information includes a route from a current location of the assistance device to a destination specified by the user. (Appendix 21) The object has musical notation written on it, 21. The assisting device according to any one of appendices 5 to 20, wherein the auxiliary information includes information indicating the placement of fingers for playing the selected part on a musical instrument. (Appendix 22) 22. The support device according to any one of appendices 4 to 21, wherein the output device outputs the auxiliary information as sound. (Appendix 23) 23. The support device according to any one of appendices 4 to 22, wherein the output device outputs characters corresponding to the selected portion as the auxiliary information by voice. (Appendix 24) a control device that acquires a plurality of line images acquired by the line sensor while the device is placed on the object and moved in a direction perpendicular to the arrangement direction of the image pickup elements, generates the two-dimensional image of the object using the plurality of line images, and identifies the selected location corresponding to the contact position using the two-dimensional image; 24. The assistance device according to any one of appendices 3 to 23, comprising: (Appendix 25) a displacement sensor for detecting displacement; Equipped with 25. The assistance device according to claim 24, wherein the control device uses the detection result of the displacement sensor to identify the selected location in the two-dimensional image that corresponds to the contact position. (Appendix 26) 26. The assistance device according to claim 25, wherein the control device generates auxiliary information corresponding to the selected portion. (Appendix 27) a transmitting / receiving unit that transmits contact position information indicating the contact position and a line image acquired by the line sensor to a control device and receives the auxiliary information from the control device; 5. The assistance device according to claim 4, comprising: (Appendix 28) 28. The assistance device according to any one of appendices 1 to 27, wherein the object is the user's finger. [Explanation of symbols]

[0149] 1, 1-1 to 1-3, 1a, 1b, 1c, 1d, 1e, 1f Support device, 2, 2-1 to 2-4 Sensor, 3 Contact detection unit, 4 Frame unit, 5 Support unit, 6, 21, 28 Transmitting / receiving unit, 7 Displacement sensor, 8, 20, 20a, 20b, 20c Control device, 9 Display device, 10 Speaker, 11 Self-positioning unit, 22 Reading processing unit, 23 Positioning unit, 24 Information storage unit, 25 Character recognition unit, 26 Auxiliary information generation unit, 27 Display information generation unit, 29 Schedule management unit, 30 Output device, 30a Display unit, 31 Route guidance unit, 32 Music score recognition unit, 40, 40a Information terminal, 41 Input acceptance unit, 61, 63 Camera, 62 Contact sensor.

Claims

1. a transparent sheet-like contact detection unit that detects contact with an object due to a user's movement; a line sensor provided at an end of the contact detection unit and having image pickup elements arranged in one direction; An assistance device comprising:

2. the contact detection unit includes a first surface that is a surface that is in contact with the object, and a second surface that is a surface opposite to the first surface, the imaging element is placed on an object to be imaged with the first surface facing up, 2. The assistance device according to claim 1, wherein the contact detection unit detects, as a contact position, a position at which the object contacts a selection point, which is a point selected by the user, within an area of the object visible to the user.

3. The support device according to claim 2, characterized in that the selected location corresponding to the contact position is determined based on a two-dimensional image of the object obtained by placing the support device on the object and moving the support device in a direction perpendicular to the arrangement direction of the imaging elements.

4. an output device that outputs auxiliary information corresponding to the selected portion; 4. The support device according to claim 3, further comprising:

5. the output device includes a transparent plate-like display device provided on the second surface side so as to overlap the contact detection unit, The support apparatus according to claim 4 , wherein the display device displays the auxiliary information.

6. The assistance device according to claim 5 , wherein the display device displays the auxiliary information so as to overlay it on an object visually recognized via the contact detection unit and the display device.

7. The object has text written on it, 6. The support device according to claim 5, wherein the auxiliary information includes, when the selected portion includes a kanji character, a reading of the kanji character.

8. The object has text written on it, The support device according to claim 5 , wherein the auxiliary information includes information indicating the meaning of the selected portion.

9. The object has text written on it, The support device according to claim 5 , wherein the auxiliary information includes a translation of the selected part.

10. The object has text written on it, The support device according to claim 5 , wherein the auxiliary information includes information indicating a segmented portion of the selected portion.

11. The object has text written on it, The support device according to claim 5 , wherein the auxiliary information includes a mask that covers an area other than the selected portion.

12. The object has text written on it, The support device according to claim 5 , wherein the auxiliary information includes information indicating a subject and a predicate corresponding to the selected portion.

13. The object has text written on it, 6. The support device according to claim 5, wherein the auxiliary information includes information indicating a reason for a feeling when the character string corresponding to the selected portion includes a character string indicating the feeling.

14. The object has text written on it, 6. The support device according to claim 5, wherein the auxiliary information includes an image showing a number of objects corresponding to a number included in the selected portion when the selected portion includes a number.

15. The object has an image thereon; 6. The support device according to claim 5, wherein the auxiliary information includes, when the selected location includes an image showing a plurality of objects, a number indicating the number of the objects in the image.

16. The object has text written on it, 6. The support device according to claim 5, wherein the auxiliary information includes information obtained by searching a website for a name when the selected portion includes the name.

17. The object has text written on it, 6. The support device according to claim 5, wherein the auxiliary information includes the stroke order of a kanji character when the selected portion includes the kanji character.

18. The object has the date and time of the event written on it; The support device according to claim 5 , wherein the auxiliary information includes information indicating whether the event can be added to the user's schedule.

19. a self-location determination unit that determines a self-location; Equipped with The object has a map printed on it, 6. The assistance device of claim 5, wherein the display device includes information indicating a current position of the assistance device on the map.

20. 20. The assistive device of claim 19, wherein the assistance information includes a route from a current location of the assistive device to a destination specified by the user.

21. The object has musical notation written on it, 6. The support device according to claim 5, wherein the auxiliary information includes information indicating a finger placement for playing the selected portion on a musical instrument.

22. The support device according to claim 4 , wherein the output device outputs the auxiliary information as sound.

23. The support device according to claim 4, wherein the output device outputs the characters corresponding to the selected portion as the auxiliary information by voice.

24. a control device that acquires a plurality of line images acquired by the line sensor while the device is placed on the object and moved in a direction perpendicular to the arrangement direction of the image pickup elements, generates the two-dimensional image of the object using the plurality of line images, and identifies the selected location corresponding to the contact position using the two-dimensional image; 4. The support device according to claim 3, further comprising:

25. a displacement sensor for detecting displacement; Equipped with 25. The support device according to claim 24, wherein the control device uses a detection result of the displacement sensor to identify the selected portion in the two-dimensional image that corresponds to the contact position.

26. 26. The support device according to claim 25, wherein the control device generates auxiliary information corresponding to the selected portion.

27. a transmitting / receiving unit that transmits contact position information indicating the contact position and the line image acquired by the line sensor to a control device and receives the auxiliary information from the control device; 5. The support device according to claim 4, further comprising:

28. The assistive device according to claim 1 , wherein the object is a finger of the user.

29. a transparent sheet-like contact detection unit that detects contact with an object due to a user's movement; a camera provided at an end of the contact detection unit and capable of acquiring images; a contact sensor that detects contact with an object photographed by the camera; Equipped with An assistance device characterized in that, when placed on the object, the user can visually recognize an area within the object corresponding to a lower part of the contact detection unit.

30. The camera is 30. The assistive device of claim 29, which is a contact-type wide-angle camera.

31. a support device that accepts input from a user; a control device; Equipped with The support device includes: a transparent sheet-like contact detection unit that detects contact with an object due to a user's movement; a line sensor provided at an end of the contact detection unit and having image pickup elements arranged in one direction; a transmitter / receiver; Equipped with the contact detection unit includes a first surface that is a surface that is in contact with the object, and a second surface that is a surface opposite to the first surface, the support device is placed on an object to be imaged by the imaging element so that the first surface faces up; the contact detection unit detects, as a contact position, a position at which the object contacts a selection point that is a point selected by the user within an area of the target object that is visible to the user; the transmitting / receiving unit transmits contact position information indicating the contact position and the line image acquired by the line sensor to a control device; The control device a support system that generates a two-dimensional image of the object using a plurality of line images acquired while the object is placed on the object and moved in a direction perpendicular to the arrangement direction of the imaging element, and that uses the two-dimensional image to identify the selected location corresponding to the contact position information.

32. The control device generating auxiliary information corresponding to the selected portion and transmitting the auxiliary information to the support device; The support device includes: a transceiver unit for receiving the auxiliary information; The support system according to claim 31, further comprising an output device for outputting the auxiliary information.

33. an information terminal that displays the selected portion; 32. The assistance system according to claim 31, comprising:

34. a support device that accepts input from a user; a control device; Equipped with The support device includes: a transparent sheet-like contact detection unit that detects contact with an object due to a user's movement; a camera provided at an end of the contact detection unit and capable of acquiring images; a contact sensor that detects contact with an object photographed by the camera; a transmitter / receiver; Equipped with When the touch panel is placed on the object, a user can visually recognize an area within the object that corresponds to a lower portion of the contact detection unit, the contact detection unit includes a first surface that is a surface that is in contact with the object, and a second surface that is a surface opposite to the first surface, the contact detection unit detects, as a contact position, a position at which the object contacts a selection point that is a point selected by the user within an area of the object that is visible to the user, in a state in which the object is placed on the object with the first surface facing up; the transmitting / receiving unit transmits contact position information indicating the contact position, a detection result by the contact sensor, and an image captured by the camera to a control device; The control device An assistance system characterized in that the selected portion corresponding to the contact position information is identified using an image acquired by the camera.

Images