Handwriting supporting device, handwriting supporting method, and program
The handwriting support device estimates character size from a predetermined stroke and adjusts predicted images to fit the writing area, addressing the lack of individuality in existing technologies and enhancing handwriting support.
Patent Information
- Application Number
- JP2023079207
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing handwriting support technologies do not allow users to write characters that utilize their individuality, as they primarily rely on tracing model characters.
A handwriting support device and method that estimates the size of a written character based on a predetermined stroke, supports the handwriting of unwritten portions, and displays predicted images adjusted to fit the writing area, respecting the user's individuality.
Enables handwriting that leverages the user's individuality by providing adjusted predicted images to ensure characters fit the writing area, allowing for more personalized and effective handwriting support.
Smart Images

Figure 2024163515000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a handwriting assistance device, a handwriting assistance method, and a program. [Background technology]
[0002] A technique has been disclosed for assisting handwriting by photographing handwritten characters and superimposing model characters on the photographed handwritten image (Patent Document 1).
[0003] This technology can assist users in writing by hand exactly as instructed. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2013-182211 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, the method disclosed in Patent Document 1 allows the user to trace a model, and there are cases where the user is unable to write characters that reflect his or her own individuality.
[0006] The present invention has been made in consideration of the above problems, and has as its object to provide a handwriting assistance device, a handwriting assistance method, and a program that enable a user to write characters that reflect the user's individuality. [Means for solving the problem]
[0007] A handwriting assistance device of one aspect of the present invention includes an estimation unit that estimates the size of the characters that a user will write when he or she finishes writing the characters from the size of a specified stroke of the characters that the user has written up to a certain point, and an assistance unit that assists in handwriting of unwritten parts of the characters so that the user can write the characters in the writing area at a size corresponding to the size of the specified writing area based on the size estimated by the estimation unit, and is characterized in that the assistance unit superimposes at least one of a predicted image of the written characters of the size estimated by the estimation unit and a modified predicted image obtained by converting the predicted image to a size that fits within the writing area on the display unit.
[0008] A handwriting assistance method according to one aspect of the present invention includes an estimation step of estimating the size of the characters to be written by the user when the user has finished writing the characters from the size of a specified stroke of the characters that the user has written up to a certain point, and an assistance step of assisting the user in writing the unwritten portions of the characters to be written so that the user can write the characters in the writing area at a size corresponding to the size of the specified writing area based on the size estimated by the estimation step, and is characterized in that the assistance step displays at least one of a predicted image of the written characters having the size estimated by the estimation step and a modified predicted image obtained by converting the predicted image to a size that fits within the writing area, superimposed on the writing area and displayed on the display unit.
[0009] A program according to one aspect of the present invention includes an estimation step of estimating the size of the characters to be written when the user finishes writing the characters from the size of a specified stroke of the characters that the user has written partway, and an assistance step of assisting the user in handwriting the unwritten parts of the characters to be written so that the user can write the characters in the writing area at a size corresponding to the size of the specified writing area based on the size estimated by the estimation step, and is characterized in that the assistance step displays at least one of a predicted image of the written characters of the size estimated by the estimation step and a modified predicted image obtained by converting the predicted image to a size that fits within the writing area, superimposed on the writing area and displayed on the display unit. Effect of the Invention
[0010] According to the present invention, it is possible to support handwriting that takes advantage of the individuality of the user. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 is a diagram showing the external configuration and a usage example of the AR glasses 1. [Diagram 2] FIG. 2 is a block diagram showing the internal configuration of the AR glasses 1. [Diagram 3] FIG. 3 is a flowchart for explaining the operation of the AR glasses 1. [Figure 4] FIG. 4 is a diagram showing an example of keystone correction for a sheet. [Diagram 5] FIG. 5 is a diagram showing an example of a predicted image and a corrected predicted image. [Figure 6] FIG. 6 is a diagram showing an example of a method for calculating the length of a stroke. [Figure 7] FIG. 7 is a flowchart for explaining the operation of the AR glasses 1 according to a second embodiment. [Figure 8] FIG. 8 is a diagram showing examples of modes and writing areas. [Figure 9] FIG. 9 is a diagram in which the types of writing areas are added to the writing areas shown in FIG. [Figure 10] FIG. 10 is a diagram showing the contents of the character database 15a registered by the user. [Figure 11] FIG. 11 is a diagram showing an example of a corrected predicted image of the address of a New Year's card, "1-1 Chiyoda, Chiyoda-ku, Tokyo." DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] [One embodiment] FIG. 1 is a diagram showing an external configuration and a usage example of AR (Augmented Reality) glasses 1 according to an embodiment of the present invention. The AR glasses 1 are worn by a user by being hung on both ear shells as shown in FIG.
[0013] The AR glasses 1 are provided with an imaging unit 12 facing forward so that, when worn by a user, the direction in which the user views is taken as the subject direction. The AR glasses 1 are also configured such that, when worn, a pair of display units 11 for the left and right eyes are disposed in front of both eyes of the user. The display unit 11 is a transmissive display, and the user wearing the AR glasses 1 can visually recognize the real space in the user's line of sight through the display unit 11 while looking at an image displayed on the display unit 11. Here, the real space includes a state in which a user is about to write characters on a piece of paper, actual handwritten characters, etc., which will be described later.
[0014] Note that the appearance of the AR glasses 1 shown in FIG. 1 is just one example, and various structures for allowing the user to wear the AR glasses 1 are possible.
[0015] The AR glasses 1 assist a user wearing the AR glasses 1 in writing characters on a writing surface such as a piece of paper. As will be described in detail later, the AR glasses 1 estimate the size of characters that the user will have when the user has finished writing them from a portion of a character that the user has written to a certain extent, and assist the user in writing the unwritten portion of a character based on the estimated character size so that the user can write characters in a writing area on a writing surface or the like at a size that corresponds to the size of the writing area.
[0016] The writing area is within the range of the paper surface on which characters are written, and refers to a place where the entire character string is written. The writing area may be the entire paper surface, or may be a part of the paper surface. The writing area may be set by the user inputting information in advance, or may be set automatically according to the image and situation captured by the imaging unit 12. The writing surface is not limited to paper, and may be made of a material other than paper (such as plastic).
[0017] [Internal structure of AR Glasses 1] 2 is a block diagram showing the internal configuration of the AR glasses 1. The AR glasses 1 include a display unit 11, an image capturing unit 12, a control unit 13, an input unit 14, and a storage unit 15.
[0018] The display unit 11 is, for example, a CRT display, a liquid crystal display, an organic EL display, a plasma display, etc. It displays various information based on the control by the control unit 13. Moreover, the display unit 11 can display various information so that the user can recognize it superimposed on the position of the writing area existing in the real space after the display unit 11 is put into a see-through state.
[0019] The image capturing unit 12 is, for example, a video camera, captures characters handwritten by the user and the user's handwriting actions, and supplies the captured video to the control unit 13.
[0020] Although not shown, the control unit 13 is composed of a CPU (Central Processing Unit), a storage section (such as a ROM (Read Only Memory), a RAM (Random Access Memory), or a non-volatile memory), and other elements including hardware. The control unit 13 executes a control program (not shown) stored in the storage unit 15 to control the AR glasses 1 as a whole and executes a handwriting assistance process (described later). The control unit 13 functions as an estimation unit 21 and an assistance unit 22 when executing the handwriting assistance process.
[0021] The estimation unit 21 refers to the character database 15a stored in the storage unit 15, and estimates the size of the written character when the user finishes writing the character from a part of the character that is halfway written, which is a character or a character string that the user is about to handwrite on a writing surface such as a paper sheet. Hereinafter, the character or character string that the user is about to handwrite on a writing surface such as a paper sheet will be referred to as the written character, as appropriate.
[0022] The support unit 22 supports handwriting of unfilled parts of characters to enable the user to write characters in a writing area (e.g., a writing surface) at a size corresponding to the size of the writing area (e.g., a writing surface) based on the size estimated by the estimation unit 21.
[0023] The input unit 14 is, for example, a touch panel or a button, and accepts information input from a user. The input unit 14 may be included in the AR glasses 1 body, or may be externally attached. If it is a touch panel, the user can input information by tapping, swiping, or the like. If the input unit 14 is externally attached, a mouse, a QWERTY keyboard, or the like may be used instead of a touch panel or a button.
[0024] The storage unit 15 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and a non-volatile memory. The storage unit 15 stores the above-mentioned control application program, various data including the character database 15a necessary for executing the program, and information generated by the processing of the program.
[0025] The character database 15a stores image data of model characters to be written and information regarding their sizes. Model characters are image data of handwritten characters.
[0026] The character database 15a may be stored in, for example, a terminal separate from the AR glasses 1, and the AR glasses 1 may acquire information from the character database 15a through wireless communication such as Bluetooth (registered trademark) or Wi-Fi (registered trademark) or through wired communication. The separate terminal may be a terminal such as a smartphone owned by the user. The character database 15a may be stored in a server, and the AR glasses 1 may acquire information from the character database 15a stored in the server.
[0027] [Handwriting support processing (first embodiment)] Next, the handwriting support process will be described with reference to the flowchart in Fig. 3. First, the flow of the process will be described with reference to the flowchart, and then a specific example will be described.
[0028] 1, a piece of paper is placed on a table, and a user wearing the AR glasses 1 is about to write characters on the paper. In addition, it is assumed that the characters to be written on the paper have been set in advance by the user via the input unit 14 and stored in the memory unit 15.
[0029] In step S1, the estimation unit 21 of the control unit 13 acquires from the storage unit 15 model characters to be written.
[0030] Next, in step S2, the estimation unit 21 acquires the captured image from the photographing unit 12 in a predetermined unit. Here, the predetermined unit is a time unit, which is set as a photographing interval such as every 0.1 milliseconds and a frame rate such as 30 fps. Then, in step S3, the estimation unit 21 specifies the position of the paper surface on the acquired photographed image, and in step S4, corrects the paper surface image to the actual shape of the paper. As shown in FIG. 4A, the photographing unit 12 photographs the paper P from oblique information, so that the image PV1 of the paper on the photographed image becomes a trapezoidal shape as shown in FIG. 4B. Therefore, the estimation unit 21 corrects the trapezoidal image to an image PV2 of the original shape (rectangle) of the paper P (hereinafter referred to as trapezoidal correction). Note that the trapezoidal correction can be performed using a known method such as trapezoidal correction by projective transformation.
[0031] In step S5, the estimation unit 21 judges whether the user has finished writing up to the Nth stroke (an integer equal to or greater than 1) of the first character (hereinafter referred to as the reference character) which is a part of the written character acquired in step S1. Specifically, the estimation unit 21 analyzes the image acquired in step S2, recognizes the lines written by the user, and judges whether the Nth stroke has been written by comparing the recognition result with the model character of the reference character stored in the character database 15a. Here, the Nth stroke refers to the number of a predetermined stroke among the strokes that constitute a character. In this embodiment, the first character of the written character consisting of multiple characters is set as the reference character, but this is not limited thereto. For example, it may be a character of a number of characters that can be assisted before the user finishes writing the entire written character.
[0032] If it is determined in step S5 that the user has not finished writing up to the Nth stroke of the reference character, the estimation unit 21 returns to step S2 and executes the subsequent processes.
[0033] If it is determined in step S5 that the user has finished writing up to the Nth stroke of the reference character, in step S6, the estimation unit 21 calculates the length of the Nth stroke of the reference character that the user has finished writing as the size of a specified stroke of the written character.
[0034] Next, in step S7, the estimation unit 21 estimates the size of the reference character when the user finishes writing the reference character based on the calculated length of the Nth stroke. Hereinafter, the size of the reference character when the user finishes writing the reference character is referred to as the size of the handwritten reference character. Specifically, for example, the length of the Nth stroke of the reference character finished by the user is compared with the length of the Nth stroke of the model character of the reference character stored in the character database 15a, and the size of the reference character when the user finishes writing the reference character (the size of the handwritten reference character) is estimated based on the comparison result.
[0035] Next, in step S8, the support unit 22 generates a model image based on the size of the handwritten reference character estimated in step S7. The model image is image data of the reference character recorded in the character database 15a, and the size is adjusted so that the entire written character, including characters other than the reference character, fits within the target writing area.
[0036] In step S9, the estimation unit 21 judges whether or not the size of the handwritten reference character estimated in step S7 differs from the size of the model image generated in step S8 by a predetermined threshold or more. For example, if the predetermined threshold is 5%, it is judged whether or not the size of the estimated handwritten reference character is 5% or more larger or smaller than the size of the model image.
[0037] If it is determined in step S9 that there is a difference equal to or greater than the predetermined threshold, in step S10, the support unit 22 generates a predicted image of the written character based on the size of the handwritten reference character estimated in step S7, or a corrected predicted image obtained by converting the predicted image to a size that fits on the paper together with the predicted image. The predicted image is image data of a model character of the written character recorded in the character database 15a, and has a size that matches the size of the estimated handwritten reference character.
[0038] Next, in step S11, the support unit 22 corrects at least one of the predicted image generated in step S10 and the corrected predicted image from a rectangular shape to a trapezoid shape, taking into account the angle between the AR glasses 1 and the paper. That is, a correction opposite to the trapezoid correction in step S4 is performed.
[0039] In step S12, the support unit 22 displays at least one of the predicted image corrected in step S11 and the corrected predicted image on the display unit 11 so that the user can visually recognize the image superimposed on the actual handwritten characters. The user handwrites the characters to be written by referring to the predicted image and the corrected predicted image superimposed on the actual handwritten characters. Note that the display of the predicted image continues, for example, until the user finishes writing all the characters to be written.
[0040] After the display of the predicted image starts in step S12, or if it is determined in step S9 that the estimated character size is not equal to or larger than the predetermined threshold, the handwriting assistance process ends. If it is determined in step S9 that the difference between the estimated character size and the model image having the predetermined character size is not equal to or larger than the predetermined threshold, that is, if the handwritten characters will fit on the paper even if the user continues to handwrite at the current size, the processes in steps S10 to S12 are not executed, and the predicted image and the modified predicted image are not displayed.
[0041] [Specific example] An example will be described in which a user practicing calligraphy is about to write the character string "Otehon" on a piece of paper, and N of the Nth stroke is 3. The estimation unit 21 acquires the model character "Otehon", which is the character to be written, from the storage unit 15 (step S1). Then, the estimation unit 21 acquires a captured image including a paper surface image from the imaging unit 12 (step S2), specifies the position of the acquired paper surface image (step S3), and performs keystone correction (step S4).
[0042] 5A shows a state in which the user has written up to the Nth stroke of the reference character " . When the user has written up to the third stroke of the reference character " , " as shown in FIG. 5A (YES in step S5), the estimation unit 21 calculates the length of the part extending up and down of the " , " (the part indicated by L2 in the figure) (step S6).
[0043] Then, the estimation unit 21 estimates the size (height H2, width W2) of the reference character when the user writes out the reference character, i.e., the size of the handwritten reference character, based on the calculated length L2 of the Nth stroke (step S7). In this example, the size of "O" is calculated using the following formula based on the length L1 of the third stroke of the model character "O" which is the reference character, and the size of the model character "O" (height H1, width W1). The box in Fig. 5A shows the size of the handwritten reference character corresponding to the calculated height L2 and width W2. Height H2 = L2 x H1 / L1 Width W2 = L2 × W1 / L1 The relationship between the stroke size and character size of each model character (the relationship between L1 and H1, and L1 and W1) is assumed to be registered in advance in the character database 15a.
[0044] Next, the support unit 22 generates a model image based on the estimated size of the handwritten reference character (step S8).
[0045] When it is determined that the size of the estimated handwritten reference character has a difference of a predetermined threshold value or more from the size of the template image (Yes in step S9), the support unit 22 generates at least one of a prediction image and a corrected prediction image (steps S10, S11). Then, the support unit 22 displays at least one of the generated prediction image and corrected prediction image on the display unit 11 (step S12).
[0046] Note that when generating the corrected prediction image, the support unit 22 adjusts the magnification / reduction ratio of the unfilled portion of the written characters, the character spacing, and the aspect ratio of each character. The magnification / reduction ratio of the written characters may be changed according to the number of strokes of the characters. The character spacing is the space between each character, and refers to the space between the frames that are the circumscribed rectangles of the characters. The aspect ratio of each character is the aspect ratio of each frame, and may be different for each character.
[0047] FIG. 5B shows an example of a prediction image, and FIG. 5C shows an example of a corrected prediction image. The frames around the characters shown in FIGS. 5B and 5C are frames for indicating the size of the characters, and represent the circumscribed rectangles of the characters without including the margins. The frames around the characters are not necessarily displayed on the display unit 11.
[0048] For example, as shown in FIG. 5B, when generating a prediction image based on the size of the handwritten reference character estimated by the estimation unit 21 (steps S10, S11), the support unit 22 displays the prediction image on the display unit 11 so that the user can visually recognize it by superimposing it on the actual handwritten characters. In this example, the user can recognize that if they continue writing from the third stroke of the reference character "御" as it is, it will not fit on the paper or will be too small. Therefore, the user can adjust the size of the unfilled portion so that it fits on the paper or is not too small.
[0049] 5C, when a corrected predicted image is generated by converting the predicted image so that the unfilled portion is adjusted to fit on the paper (steps S10 and S11), the support unit 22 can also display the corrected predicted image on the display unit 11 so that the user can visually recognize it superimposed on the actual handwritten characters (step S12). In this example, the user can write by hand as if tracing the corrected predicted image that has been adjusted to fit on the paper and superimposed on the actual handwritten characters.
[0050] Also, the predicted image of Fig. 5B and the corrected predicted image of Fig. 5C can be displayed alternately. Also, the predicted image and the corrected predicted image may be displayed simultaneously. Furthermore, the predicted image and the corrected predicted image may be displayed simultaneously in different display modes, such as in different colors.
[0051] (Identifying the Nth stroke) The Nth stroke can be set not only for a single character, but also for a character string consisting of multiple characters, such as "aiueo." Since "aiueo" consists of 12 strokes in total, if N is set to 6, the first stroke of "u" can be set as the Nth stroke. Also, N may be zero, that is, the Nth stroke may be the 0th stroke. In this case, the predicted image or the corrected predicted image is displayed from the beginning of writing. When it is not necessary to reflect the individuality of the user's characters, such as in an address, N may be set to 0. On the other hand, when it is desirable to utilize the individuality of the user's characters, such as in calligraphy, it is preferable to set N to a predetermined value or more and allow the user to freely write characters up to a predetermined number of strokes. Note that N may be set by the user, or the number of strokes that affects the size of each character may be determined as N and stored in character database 15a.
[0052] (Length of Nth stroke) The length of the Nth stroke of a character is defined for each character shape. In principle, it may be the length of one straight line part of the character as in the example of Figure 5A, but if the stroke is composed of two or more straight lines as in Figure 6A, it may be the length of the longest part, and if the stroke is composed of curved lines as in Figure 6B, it may be the length of the diagonal of the rectangle surrounding the character.
[0053] (Standard font size) In the above description, the estimation unit 21 calculates the size of a reference character handwritten by a user from the length of only the Nth stroke, but it may also calculate the size from the size up to the Nth stroke. For example, it may be calculated based on the total length from the 1st stroke to the Nth stroke, or it may be calculated based on the average of the character sizes estimated from the respective lengths from the 1st stroke to the Nth stroke. A character with a predetermined stroke size of a character written by a user is calculated as the size of the reference character.
[0054] [Handwriting support processing (second embodiment)] In the above description, the writing area in which characters are written is not specifically mentioned because the entire paper surface is the writing area, but the writing area should be limited depending on the situation in which characters are written. This is because it is expected that the user will handwrite characters in a predetermined writing area, for example, when writing the address on the right edge of a postcard and the recipient's name in the center. In such various situations, the user can use the AR glasses 1.
[0055] 7 is a flowchart illustrating another handwriting support process. This handwriting support process can support, in a case where a user needs to write characters, the user to determine a writing area in which to write characters and to write the characters in the writing area.
[0056] In step S31, the support unit 22 selects a mode. The mode can be determined according to the object on which characters are to be written. For example, the mode is determined according to a normal postcard (FIG. 8A), a New Year's postcard (FIG. 8B), a gift envelope or condolence envelope (FIG. 8C), a vertically written guest book (FIG. 8D), or a horizontally written guest book (FIG. 8E). When determining the mode, the support unit 22 can identify the object on which characters are to be written based on an image captured by the image capture unit 12 and select a mode based on the identified object, or the support unit 22 can select a mode manually selected by the user.
[0057] In step S32, the estimation unit 21 acquires model characters of characters to be written that are set in advance by the user. The user sets characters to be entered in advance via the input unit 14 and registers them in the character database 15a. For example, as shown in Fig. 10A to 10C, a plurality of destinations, sender names, and addressee names may be registered in the character database 15a in advance, and the user may select the destination and addressee name each time. Since the addressee name is accompanied by a title, the registered name followed by the title may be used as the input character. In addition, whether or not to add a title may be specified by the user, or may be determined according to the mode selected in step S31. For example, a title is added in the case of a postcard, but it is not necessary to add one on a shipping invoice, since "sama" is already printed on the invoice. In addition, if the input characters are character strings in which line breaks may be inserted, such as a destination or address, the model characters to be input obtained in step S32 may be image data in which line breaks are inserted at a specified position, or image data in which line breaks are not inserted. In this case, the user may register, for example, the destination address so that the specified positions in which line breaks may be inserted are known in advance. In the example of FIG. 10A, the first address, the second address, etc. are registered separately for each position in which line breaks may be inserted.
[0058] In steps S33 to S35, the same processes as in steps S2 to S4 in FIG. 3 are executed, and therefore the description thereof will be omitted.
[0059] In step S36, the support unit 22 judges whether or not the writing area has been determined. If it is judged that the writing area has not been determined, in step S37, the support unit 22 determines the writing area in which to write the characters based on the mode selected in step S31. Specifically, in the case of the normal postcard mode (FIG. 8A), the New Year's postcard mode (FIG. 8B), the mode for the gift envelope / condolence envelope (FIG. 8C), and the mode for the horizontally written guestbook (FIG. 8E), the support unit 22 determines the writing area Q1 for writing the address, the writing area Q2 for writing the recipient's name, and the writing area Q3 for writing the sender's name, as shown in each figure. In the case of the vertically written guestbook mode (FIG. 8D), the support unit 22 determines one area for writing the address and the name as the writing area Q4.
[0060] For example, the positions of the writing area Q1 for writing the address, the writing area Q2 for writing the recipient's name, the writing area Q3 for writing the sender's name, and the writing area Q4 for writing the address and name are determined in advance according to the object on which the characters are to be written. The support unit 22 can determine the writing area in which to write the characters based on the mode selected in step S31. In the case of the vertical writing guest book mode shown in FIG. 8D, the frame of the paper and the characters already written can be recognized, and the rightmost part on which no characters are written can be determined as the writing area Q4. Also, as shown in FIG. 8E, the writing areas Q1 and Q3 can be determined by recognizing the lines of the paper and the characters printed on the "name" and "address".
[0061] The frames shown on each sheet of paper in FIG. 8 indicate the writing area. To make the determined writing area easier to understand, the support unit 22 may display the outer frame of the writing area on the display unit 11 so that the user can see it superimposed on the actual handwritten characters. Also, as shown in FIG. 9, each writing area may be displayed with the notation "Address area" (Q1), "Recipient area" (Q2), "Name area" (Q3), and "Address and name area" (Q4). Note that, for example, if the sender's name is already written or printed on the back of the postcard, and there is no need for the user to write anything, the outer frame of the writing area may not be displayed.
[0062] The user may be allowed to specify the writing area, and may be allowed to adjust the writing area determined by the support unit 22 via the input unit 14.
[0063] Returning to FIG. 7, in step S38, the support unit 22 determines a single writing area (hereinafter referred to as a target writing area) in which to write characters. If multiple writing areas are determined in step S37, one of the multiple writing areas is determined as the target writing area. On the other hand, if there is only one writing area, the determined writing area is determined as the target writing area.
[0064] For example, when writing areas Q1, Q2, and Q3 are determined in step S37 as shown in Fig. 8A, Q1 is determined as the target writing area, and when only writing area Q2 is determined as shown in Fig. 8C, Q2 is determined as the target writing area. The support unit 22 can determine the target writing area based on the user's designation. The target writing area may also be determined based on information on the user's line of sight direction captured by the imaging unit 12.
[0065] If it is determined in step S36 that the writing area has already been determined, the process proceeds to step S39. In other words, after the writing area is determined, the processes of steps S37 and S38 are skipped, and the determination of the writing area and the target writing area are performed only once.
[0066] In steps S39 to S41, the same processes as those in steps S5 to S7 in FIG. 3 are executed, and therefore the description thereof will be omitted.
[0067] Next, in step S42, the support unit 22 generates a model image based on the size of the handwritten reference character estimated in step S41. The model image is image data of the reference character recorded in the character database 15a, and is adjusted in size so that the entire written character, including characters other than the reference character, fits within the target writing area.
[0068] In step S43, the estimation unit 21 judges whether or not the size of the handwritten reference character estimated in step S41 differs from the size of a predetermined character (the size of the model image generated in step S42) by a predetermined threshold or more. This is the same process as step S9 in FIG. 3.
[0069] In step S44, the support unit 22 generates a predicted image of the written characters based on the size of the handwritten reference characters estimated in step S41. It is also possible to generate a modified predicted image by converting the predicted image so that the unwritten portion is adjusted to fit into the target writing area. As described above, the modified predicted image can be created by converting the predicted image by adjusting not only the enlargement / reduction ratio of the written characters, the character spacing, and the aspect ratio of each character, but also the presence or absence of a line break and the position of the line break. Whether or not to perform a line break may be determined depending on the mode and the target writing area. As will be described later, when the New Year's postcard mode is selected, a line break is performed when writing an address, but a line break is not performed when writing a recipient's name.
[0070] Next, in step S45, the generated predicted image is corrected so that the paper is converted from a rectangular shape to a trapezoid, taking into account the angle between the AR glasses 1 and the paper.
[0071] In step S46, the support unit 22 displays the predicted image corrected in step S45 and the corrected predicted image on the display unit 11 so that the user can visually recognize the image superimposed on the actual handwritten characters.
[0072] FIG. 11 is a diagram showing an example of a corrected predicted image based on the registration information shown in FIG. 10, displayed on the display unit 11. The corrected predicted image is the "1-1 Chiyoda, Chiyoda-ku, Tokyo" portion in the writing area in FIG. 11. FIG. 11A is a corrected predicted image in which the enlargement / reduction ratio has been adjusted so that the entire address can be written in a single vertical line in the writing area Q1. FIG. 11B is a corrected predicted image in which a new line is added from the street address (the portion registered as the "third address" in FIG. 10A). FIG. 11C is a corrected predicted image in which a new line is added from the ward (the portion registered as the "second address" in FIG. 10A).
[0073] In step S46, after the display of the predicted image is started, or in step S43, if it is determined that there is no difference greater than or equal to a predetermined threshold between the size of the estimated handwritten reference character and the size of a predetermined character (the size of the model image generated in step S42), the handwriting assistance process ends.
[0074] (Creation of predicted image, corrected predicted image) In the above, the predicted image and the corrected predicted image were created based on the size of the estimated reference character. However, when the user's handwriting progresses and the next character (in the example of FIG. 5, "hand") is written, a predicted image can also be created and displayed using this character as the reference character.
[0075] Also, in the above, when it is determined that there is no difference greater than or equal to a predetermined threshold between the size of the estimated character and the size of a predetermined character (step S10), the predicted image and the corrected predicted image are not displayed. However, even in that case, the predicted image and the corrected predicted image may be displayed. For example, the predicted image and the corrected predicted image may be constantly displayed while the user is handwriting.
[0076] (Other assistance methods) In the above, the assistance unit 22 assisted the user's handwriting by displaying the predicted image corrected in step S11 on the display unit 11 so that it could be visually recognized superimposed on the user's actual handwritten characters. However, as another method, for example, the assistance unit 22 can also assist the user with messages such as "too large", "write larger", "too small", "write a little smaller" by voice or text display.
[0077] (Number of characters to be filled in) In the above example, the case where the characters to be filled in are multiple characters was described as an example, but it may also be a single character.
[0078] [Third Embodiment] As described above, the AR glasses 1 can assist in writing characters by hand, but can also assist in writing figures and pictures by hand as information to be written. In this case, each part that constitutes a figure or picture is considered to be a stroke, and the Nth stroke refers to the Nth part of the figure or picture with a set stroke order.
[0079] [Use case example] The AR glasses 1 are expected to be used by being owned by individuals and installed in public places. In the case of personal ownership, for example, character database 15a can be used to help learn calligraphy by storing a calligraphy model, can be used to help with signatures, address writing, and writing names on gift envelopes by storing personal information of the owner, and can be used to help with writing names and destinations by storing personal information of others. Also, if installed in government offices, banks, post offices, license centers, passport centers, etc., it can be useful for filling out various application forms, and if installed in convenience stores, department stores, supermarkets, etc., it can be useful for filling out addresses for parcel delivery and gift delivery. Furthermore, if installed in ceremonial halls, hotels, and inns, it can be useful for filling out guest books and guest lists.
[0080] [Supplementary explanation of the embodiment] The above-mentioned embodiments each show a preferred specific example of the present invention. The numerical values, components, the arrangement positions and connection forms of the components, the processing order in the flow charts, and the like shown in the following embodiments are merely examples and are not intended to limit the present invention. Also, each figure is not necessarily a strict illustration.
[0081] The above-mentioned series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs constituting the software are installed from a program recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose personal computer capable of executing various functions by installing various programs.
[0082] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.
[0083] [Summary of the invention] A handwriting assist device that assists a user in writing characters on a writing surface, comprising: an estimation unit 21 that estimates the size of a character that the user has written when the character is completed based on a predetermined stroke size of the character that the user has written to a certain point; a support unit 22 that supports handwriting of an unwritten portion of a character to be written so that the user can write the character in the writing area in a size corresponding to the size of the predetermined writing area based on the size estimated by the estimation unit 21; Equipped with The support unit 22 displays at least one of a predicted image of the written characters having the size estimated by the estimation unit 21 and a corrected predicted image obtained by converting the predicted image into a size that fits in the writing area on the display unit by superimposing the image on the writing area. It is characterized by:
[0084] With this configuration, the handwriting assist device can assist in handwriting by referring to a predicted image while respecting the individuality of even a portion of the written characters.
[0085] In addition, the support unit 22 generates a predicted image of a size that fits within the writing area by adjusting the enlargement ratio, reduction ratio, aspect ratio, character spacing, or the presence or absence of line breaks for each character of the written characters. It is characterized by:
[0086] With this configuration, the handwriting assist device can generate predicted images not only by enlarging or reducing characters but also by using various other methods.
[0087] In addition, the support unit 22 When the difference between the size estimated by the estimation unit 21 and the predetermined size is equal to or greater than a predetermined value, at least one of the predicted image and the corrected predicted image is displayed on the display unit so as to be superimposed on the entry area. It is characterized by:
[0088] With this configuration, the handwriting assistance device does not display predicted images and modified predicted images unless the size of the characters entered by the user deviates from the size of the entry area by a specified amount, allowing the user to handwrite characters more freely. [Explanation of symbols]
[0089] 1: AR glasses, 11: display unit, 12: shooting unit, 13: control unit, 14: input unit, 15: memory unit, 15a: character database, 21: estimation unit, 22: support unit
Claims
1. A handwriting assist device that assists a user in writing characters on a writing surface, comprising: an estimation unit that estimates a size of the written character when the user finishes writing the written character from a size of a predetermined stroke of the written character that the user has written to a certain point; a support unit that supports handwriting of an unwritten portion of the character to be written so that the user can write the character in the writing area in a size corresponding to a predetermined size of the writing area based on the size estimated by the estimation unit; Equipped with The support unit displays, on a display unit, at least one of a predicted image of the written character having a size estimated by the estimation unit and a corrected predicted image obtained by converting the predicted image into a size that fits within the writing area, superimposed on the writing area. A handwriting assistance device comprising:
2. 2. The handwriting assisting device according to claim 1, The support unit generates the corrected predicted image of a size that fits within the writing area by adjusting the enlargement rate, reduction rate, aspect ratio, character spacing, or the presence or absence of line breaks for each character of the written characters. A handwriting assistance device comprising:
3. 2. The handwriting assisting device according to claim 1, The support unit includes: When the difference between the size estimated by the estimation unit and a predetermined size is equal to or greater than a predetermined value, at least one of the predicted image and the corrected predicted image is displayed on a display unit so as to be superimposed on the writing area. A handwriting assistance device comprising:
4. A method for assisting a user in writing characters on a writing surface, comprising: an estimation step of estimating a size of the written character when the user finishes writing the written character from a size of a predetermined stroke of the written character that the user has written to a certain point; a support step of supporting the user in writing an unwritten portion of the character to be written, so that the user can write the character in the writing area in a size corresponding to a predetermined size of the writing area, based on the size estimated by the estimation step; Including, In the support step, at least one of a predicted image of the written characters having a size estimated in the estimation step and a corrected predicted image obtained by converting the predicted image into a size that fits in the writing area is displayed on a display unit so as to be superimposed on the writing area. A handwriting assistance method comprising:
5. In a program executable by a computer capable of transmitting and receiving information to and from an input / output device, an estimation step of estimating a size of the written character when the user finishes writing the written character from a size of a predetermined stroke of the written character that the user has written to a certain point; a support step of supporting the user in writing an unwritten portion of the character to be written, so that the user can write the character in the writing area in a size corresponding to a predetermined size of the writing area, based on the size estimated by the estimation step; Including, In the support step, at least one of a predicted image of the written characters having a size estimated in the estimation step and a corrected predicted image obtained by converting the predicted image into a size that fits in the writing area is displayed on a display unit so as to be superimposed on the writing area. A program for causing a computer to execute a process.
Citation Information
Patent Citations
Portable terminal, handwriting support method, and program
JP2013182211A
