Display device, display method, program, and display system

Abstract

To recognize handwritten input in a direction in which a user has performed handwritten input.SOLUTION: A display device comprises: a character recognition unit that performs character recognition on handwritten data input in a handwritten manner from a plurality of directions and calculates a recognition rate for each of the plurality of directions; and a recognition direction determination unit that determines, as the recognition direction of the handwritten input, the direction having the largest recognition rate when a difference between the top two recognition rates among the plurality of calculated recognition rates is equal to or greater than a threshold, causes the character recognition unit calculates, for each of the plurality of directions, recognition rates by including newly input handwritten data in the handwritten data for which it has been determined that a difference is smaller than the threshold when a difference between the top two recognition rates among the calculated recognition rates is smaller than the threshold, and determines, as the recognition direction of the handwritten data, the direction having the largest recognition rate when a difference between the top two recognition rates among the calculated recognition rates is equal to or greater than the threshold. Character recognition on the handwritten data is performed in the determined recognition direction.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to a display device, a display method, a program, and a display system.

Background Art

[0002] There is known a display device that uses handwriting recognition technology to recognize handwriting data and display it on a display. When the display device is placed flat, multiple users can write by hand so as to surround the display device.

[0003] Techniques for freely inputting handwritten characters from any direction have been devised (for example, see Patent Document 1). Patent Document 1 discloses a technique in which when receiving a two-character handwritten input from a user, recognition is performed from a plurality of directions, and the direction with the highest recognition rate is set as the recognition direction.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is a problem that the user may not be writing in the direction with the highest recognition rate. When setting the direction with the highest recognition rate as the recognition direction as in the conventional case, depending on the content written by the user, the vertical direction as seen by the user may not be set as the recognition direction. For example, when the user writes only the Chinese character "一" (ichi), the display device may recognize the number "1" with a recognition rate of 90% and the Chinese character "一" (ichi) with a recognition rate of 85%. In this case, even though the user has written the Chinese character "一" (ichi) with the second highest recognition rate, the display device sets the direction in which the number "1" with the highest recognition rate is recognized (a direction that is 90 degrees counterclockwise different from the vertical direction as seen by the user) as the recognition direction.

[0005] An object of the present invention is to recognize a handwritten input in the direction in which the user has made the handwritten input.

Means for Solving the Problems

[0006] In view of the above-described problems, the present invention includes a character recognition unit that performs character recognition on handwritten input data from a plurality of directions and calculates a recognition rate for each of the plurality of directions; a recognition direction determination unit that, when a difference between the top two recognition rates among the calculated plurality of recognition rates is equal to or exceeds a threshold, determines the direction with the highest recognition rate as the recognition direction of the handwritten data; when a difference between the top two recognition rates among the calculated plurality of recognition rates is smaller than the threshold, the character recognition unit calculates a recognition rate for each of the plurality of directions, including the handwritten data whose difference is determined to be smaller than the threshold and newly input handwritten data; and, when a difference between the top two recognition rates among the calculated recognition rates is equal to or exceeds the threshold, determines the direction with the highest recognition rate as the recognition direction of the handwritten data; and a display control unit that displays a character string obtained by character recognition of the handwritten data in the determined recognition direction, in the recognition direction determined by the recognition direction determination unit. [Effects of the Invention]

[0007] It can recognize handwriting input based on the direction in which the user writes. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram illustrating a method for determining a recognition direction. [Figure 2] 10A and 10B are diagrams illustrating a method for determining a recognition direction when the recognition direction cannot be identified. [Figure 3] FIG. 1 is an example of an overall configuration diagram of a display device. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Figure 5] FIG. 2 is an example of a functional block diagram illustrating functions of the display device, divided into blocks. [Figure 6] 3A and 3B are diagrams illustrating object data stored in an object data storage unit. [Figure 7] 10A and 10B are diagrams illustrating an example of an operation guide and selectable candidates displayed by the operation guide. [Figure 8] 10 is a diagram illustrating a method for determining a recognition direction when the recognition direction can be specified by a recognition rate (when D>T). FIG. [Figure 9] 10 is a diagram illustrating a method for determining a recognition direction when the recognition direction cannot be determined based on the recognition rate (when D≦T). FIG. [Figure 10] 10 is a diagram illustrating a method for determining the recognition direction when the user continues to write by hand from the state of FIG. 9. FIG. [Figure 11] 10 is a flowchart illustrating an example of a process performed by a recognition direction determination unit to determine the direction in which a user is writing by hand. [Figure 12] FIG. 10 is a diagram illustrating a method for determining a threshold value T. [Figure 13] FIG. 10 is a diagram showing an example of an identical recognition group when a predetermined time Tm has not elapsed since pen-up. [Figure 14] FIG. 10 is a diagram showing an example of an identical recognition group when a predetermined time Tm has elapsed since pen-up. [Figure 15] FIG. 10 is a diagram showing an example of handwritten data containing a mixture of handwritten characters and tables. [Figure 16] 10A and 10B are diagrams illustrating conditions under which stroke data are not classified into the same recognition group. [Figure 17] 10 is a flowchart illustrating an example of a procedure in which a character recognition unit determines stroke data in the same recognition group. [Figure 18] FIG. 18 is an example of a flowchart for determining whether or not the stroke data satisfies the condition for not being included in the same recognition group, as described in steps S1, S3, and S7 of FIG. 17. [Figure 19] FIG. 2 is an example of a functional block diagram of a display device. [Figure 20] FIG. 10 is a flowchart illustrating an example of changing conditions based on a result of determining a recognition direction. [Figure 21] FIG. 10 is a diagram showing an example of a determination result screen. [Figure 22] FIG. 1 is a diagram illustrating an example of a system configuration of a display system. [Figure 23] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing system. [Figure 24] FIG. 2 is an example of a functional block diagram illustrating functions of the display system in blocks. [Figure 25] 10 is an example of a sequence diagram illustrating a process in which a display device and an information processing system communicate with each other to display a recognition result of handwritten data. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A display device and a display method performed by the display device will be described below as an example of an embodiment of the present invention with reference to the accompanying drawings.

[0010] [First embodiment] <Method for determining recognition direction according to this embodiment> When performing character recognition on handwritten data, the display device of this embodiment recognizes the same handwritten data from multiple directions, and when the difference between the top two recognition rates exceeds a threshold (hereinafter referred to as a "recognition direction determination condition"), determines the direction with the highest recognition rate as the recognition direction. This will be described with reference to FIG.

[0011] 1 is a diagram illustrating a method for determining a recognition direction in this embodiment. A user inputs handwritten data 301 of "Sunny" from the negative direction of the y axis. First, the display device uses the following parameters for determining the recognition direction: Threshold T for determining the recognition direction (e.g. 50). The difference D between the top two recognition rates from each direction. When there are four directions, each direction is the up and down direction from the user's perspective, assuming that the user is on each side of a rectangular display device. The maximum recognition rate is 100, and the higher the number, the higher the recognition rate.

[0012] In the case of Figure 1, the recognition rates for each of the four directions are as follows: A. "y-axis positive direction" → recognition rate: 10 B. "x-axis positive direction" → recognition rate: 5 C. "y-axis negative direction" → recognition rate: 95 D. "Negative x-axis direction" → Recognition rate: 5 In the case of FIG. 1, the top two recognition rates are 10 and 95. Therefore, D = 95 - 10 = 85. Since the recognition direction determination condition "D > T" is satisfied, the recognition direction with the highest recognition rate is C. "Negative y-axis direction". The display device 2 displays the operation guide 500 including the character string candidate recognized in the "Negative y-axis direction" in the "Negative y-axis direction".

[0013] Subsequently, referring to FIG. 2, the case where the recognition direction cannot be specified (the case where D ≤ T) will be described. FIG. 2 is a diagram for explaining a method for determining the recognition direction when the recognition direction cannot be specified. The user inputs handwritten data 302 of "one" from the positive x-axis direction. In the case of FIG. 2, the recognition rates in each of the four directions are as follows. A. "Positive y-axis direction" → Recognition rate: 75 B. "Positive x-axis direction" → Recognition rate: 80 C. "Negative y-axis direction" → Recognition rate: 85 D. "Negative x-axis direction" → Recognition rate: 80 In the case of FIG. 2, the top two recognition rates are 80 and 85. Therefore, D = 85 - 80 = 5. Since the recognition direction determination condition "D > T" is not satisfied, the recognition direction is not specified. When the recognition direction cannot be specified, the display device adopts the latest recognition direction determined previously. If the previous recognition direction is not saved, the display device adopts the recognition direction of the default setting.

[0014] In FIG. 2, although the operation guide 500 is displayed in the "Positive y-axis direction", if the user continues writing without selecting the character string candidate, the operation guide 500 is erased. Then, by continuing to write, the user will eventually satisfy the recognition direction determination condition as in FIG. 1, and the correct recognition direction will be determined. ​​In this way, the display device 2 can determine the correct recognition direction by using the recognition direction determination condition "D > T" to determine the recognition direction. For example, when handwritten data such as "one" is input, it is possible to suppress the setting of an incorrect recognition direction. Even if an operation guide 500 including a string candidate recognized in an incorrect recognition direction is displayed, if the user continues writing, the recognition direction determination condition will eventually be satisfied, and the correct recognition direction can be determined. In this way, the display device 2 can determine the correct recognition direction by using the recognition direction determination condition "D > T" to determine the recognition direction.

[0016] <Regarding terms> A series of operations in which the user presses and continuously moves an input means on the display and then releases it from the display is called a stroke. Stroke data is information displayed on the display based on the locus of coordinates input by the input means. The stroke data may be interpolated as appropriate. Handwritten data is data having one or more stroke data. Handwriting input indicates that handwritten data is input by the user.

[0017] The string obtained by character recognition and conversion of the handwritten data may include, in addition to text data, fixed characters such as "completed", stamps displayed as marks, figures such as circles and stars, straight lines, etc., and data displayed based on the user's operations. Characters include numbers, letters, symbols, etc. Characters are also referred to as text data.

[0018] The recognition rate is the probability of converting handwritten data into a string. The recognition rate is, for example, the probability or likelihood of converting it into that string. The recognition rate is, for example, from 0 to 1.0, but may also be from 0 to 100, etc.

[0019] The plurality of directions for recognizing handwritten data are the vertical and horizontal directions as seen by the user when it is assumed that the user exists around the flat display device 2. When the user is present on each side of the display device 2, for example, the plurality of directions are the directions of the perpendicular lines dropped from the inside of the display device 2 to the sides.

[0020] In this embodiment, even if a threshold is exemplified as a comparison target, the threshold is not limited to the exemplified value. Therefore, in this embodiment, for all thresholds, the terms "less than threshold" and "below threshold" have the same meaning, and the terms "over threshold" and "above threshold" have the same meaning. For example, when the threshold is 11, "below threshold" has the same meaning as "below threshold" when the threshold is 10. Furthermore, when the threshold is 10, "over threshold" has the same meaning as "above threshold" when the threshold is 11.

[0021] <Device configuration example> The overall configuration of the display device 2 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing the overall configuration of the display device 2. Fig. 3(a) shows, as an example of the display device 2, a display device 2 used as a horizontally long electronic whiteboard hung on a wall.

[0022] As shown in Fig. 3(a), a display 220 as an example of the display device 2 is installed on the upper part of the display device 2. A user U can use a pen 2500 to handwrite (input or draw) characters or the like on the display 220.

[0023] FIG. 3(b) shows a display device 2 used as a vertically long electronic whiteboard hung on a wall.

[0024] 3(c) shows the display device 2 placed flat on a desk 230. The display device 2 is about 1 cm thick, so there is no need to adjust the height of the desk even when it is placed flat on a regular desk. In addition, the user can easily move the display device 2.

[0025] The pen 2500 can input coordinates using electromagnetic induction, active electrostatic coupling, etc. The pen 2500 may also have functions such as pressure detection, tilt detection, and a hover function (displaying the cursor before the pen touches the screen).

[0026] <Device hardware configuration> Next, the hardware configuration of the display device 2 will be described with reference to Fig. 4. As shown in the figure, the display device 2 has the configuration of an information processing device or a computer. Fig. 4 is an example of a hardware configuration diagram of the display device 2. As shown in Fig. 4, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0027] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201.

[0028] The SSD 204 stores various data such as the OS and programs for the display device. This program may be an application program that runs on an information processing device equipped with a general-purpose OS (Windows (registered trademark), Mac OS (registered trademark), Android (registered trademark), iOS (registered trademark), etc.). In this case, the device is normally used as a general-purpose information processing device, but when a user runs the application program, the user can write by hand, just like on a dedicated device for the display device 2.

[0029] The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I / F 223, a power control circuit 224, an AC adapter 225, and a battery 226.

[0030] The display controller 213 controls and manages the screen display in order to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500, the user's hand, etc. (the pen and the user's hand are input means) has come into contact with the display 220. The touch sensor 216 also receives a pen ID.

[0031] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. For example, in the case of an optical method, two light-emitting / receiving devices installed at both ends of the upper side of the display 220 emit multiple infrared rays parallel to the display 220, and receive the light that is reflected by a reflecting member installed around the periphery of the display 220 and returns along the same optical path as the emitted light by the light-receiving element. The touch sensor 216 outputs position information of the infrared rays emitted by the two light-emitting / receiving devices that are blocked by an object to the touch sensor controller 215, and the touch sensor controller 215 identifies the coordinate position, which is the contact position of the object. The touch sensor controller 215 also has a communication unit 215a and can wirelessly communicate with the pen 2500. For example, if communication is based on a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in advance in the communication unit 215a, the user can communicate with the display device 2 without having to perform connection settings to allow the pen 2500 to communicate with the display device 2.

[0032] The power switch 227 is a switch for switching ON / OFF the power of the display device 2. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. It is mainly used to detect whether the display device 2 is being used in the installation state shown in FIG. 3(a), FIG. 3(b), or FIG. 3(c), and can automatically change the orientation of characters, etc. depending on the installation state.

[0033] The serial interface 218 is a communication interface with the outside, such as a USB. The serial interface 218 is used for inputting information from the outside, etc. The speaker 219 is used for outputting audio, and the microphone 221 is used for inputting audio. The wireless communication device 222 communicates with a terminal carried by the user, and relays a connection to the Internet, for example. The wireless communication device 222 communicates using Wi-Fi, Bluetooth (registered trademark), or the like, but any communication standard is acceptable. The wireless communication device 222 forms an access point, and when the user sets the SSID (Service Set Identifier) ​​and password they have obtained in the terminal they carry, they can connect to the access point.

[0034] Preferably, the wireless communication device 222 is provided with two access points. a. Access point → Internet b. Access point → Internal network → Internet Access point a is for users outside the company, and users cannot access the company network but can use the Internet. Access point b is for users inside the company, and users can use the company network and the Internet.

[0035] The infrared I / F 223 detects the display device 2 arranged adjacently. The infrared I / F 223 can detect only the display device 2 arranged adjacently by taking advantage of the rectilinearity of infrared rays. It is preferable to provide one infrared I / F 223 on each side, and it can detect in which direction another display device 2 is arranged relative to the display device 2. This expands the screen, making it possible to display handwritten information that was previously handwritten on the adjacent display device 2 (handwritten information on another page, with the area of ​​one display 220 being one page), etc.

[0036] The power supply control circuit 224 controls an AC adapter 225 and a battery 226, which are power sources for the display device 2. The AC adapter 225 converts AC common to commercial power sources into DC.

[0037] If the display 220 is so-called electronic paper, it consumes little or no power to maintain the image display, and therefore can be driven by a battery 226. This makes it possible to use the display device 2 for purposes such as digital signage even in places where it is difficult to connect to a power source, such as outdoors.

[0038] The display device 2 further includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0039] The touch sensor 216 is not limited to an optical type, and various detection means may be used, such as a capacitive touch panel that identifies the contact position by detecting changes in capacitance, a resistive film touch panel that identifies the contact position by voltage changes between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. The touch sensor 216 may be of a type that does not require an electronic pen to detect whether or not the pen tip is touching. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not have to be a long, thin pen-shaped panel.

[0040] <About the function> Next, the functions of the display device 2 will be described with reference to Fig. 5. Fig. 5 is an example of a functional block diagram that explains the functions of the display device 2 in blocks. The display device 2 has an input acceptance unit 21, a drawing data generation unit 22, a character recognition unit 23, a display control unit 24, a data recording unit 25, a network communication unit 26, an operation acceptance unit 27, a recognition direction determination unit 28, a threshold determination unit 29, and an identical recognition group determination unit 30 (an area change unit 31, an exclusion unit 32). Each function of the display device 2 is a function or means that is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0041] The input receiving unit 21 receives input of stroke data (sequence of coordinate points) by detecting the coordinates of the position where an input means such as a pen 2500 touches the touch sensor 216. The drawing data generating unit 22 acquires the coordinates where the tip of the pen 2500 touches from the input receiving unit 21. The drawing data generating unit 22 generates stroke data by connecting the sequence of coordinate points through interpolation.

[0042] The character recognition unit 23 performs character recognition processing on one or more stroke data (handwritten data) handwritten by the user and converts them into character codes. The character recognition unit 23 recognizes characters (in multiple languages, including Japanese and English), numbers, symbols (%, $, &, etc.), shapes (lines, circles, triangles, etc.), etc., in parallel with the user's pen operation. Various algorithms have been devised for the recognition method, but this embodiment uses known techniques and will not go into detail. Furthermore, as will be described later, the character recognition unit 23 recognizes characters from multiple directions in the handwritten data input by hand, and calculates a recognition rate for each of the multiple directions.

[0043] The display control unit 24 displays handwritten data, character strings converted from the handwritten data, and operation menus for user operation on the display 220. The data recording unit 25 stores handwritten data handwritten on the display device 2, converted character strings, PC (Personal Computer) screens, files, etc. in the storage unit 40. The network communication unit 26 connects to a network such as a LAN and transmits and receives data to and from other devices via the network. As will be described later, the display control unit 24 also displays character strings obtained by character recognition of handwritten data input in the determined recognition direction in the recognition direction determined by the recognition direction determination unit 28.

[0044] The recognition direction determination unit 28 causes the character recognition unit 23 to perform character recognition processing in each of four directions (or eight directions) perpendicular to the four sides of the display device 2, and determines the handwriting direction based on the recognition rates for the four directions. Furthermore, as described below, when the difference between the top two recognition rates among the calculated multiple recognition rates is equal to or exceeds a threshold, the recognition direction determination unit 28 determines the direction with the highest recognition rate as the recognition direction of the handwritten input. Furthermore, as described below, when the difference between the top two recognition rates among the calculated multiple recognition rates is smaller than a threshold, the character recognition unit 23 calculates a recognition rate for each of the multiple directions, including handwritten data whose difference is determined to be smaller than the threshold and newly input handwritten data. When the difference between the top two recognition rates among the recognition rates calculated by the character recognition unit 23 is equal to or exceeds a threshold, the recognition direction determination unit 28 determines the direction with the highest recognition rate as the recognition direction of the handwritten data.

[0045] The threshold value determination unit 29 accumulates the top two recognition rates when a character string candidate is selected from the operation guide 500, and determines a threshold value for determining the recognition direction from the difference between the accumulated recognition rates.

[0046] The area change unit 31 changes the neighborhood rectangle 102 for determining whether stroke data is included in a recognition group depending on whether a predetermined time Tm has elapsed since the input tool was released from the touch panel.

[0047] If the stroke data received by the input receiving unit 21 satisfies a predetermined condition, the exclusion unit 32 excludes the stroke data from the recognition group even if the stroke data is handwritten in the neighboring rectangle 102.

[0048] The display device 2 also includes a storage unit 40 configured in the SSD 204, RAM 203, or the like shown in FIG.

[0049] <<Object data>> FIG. 6 is a diagram illustrating the object data stored in the object data storage unit 41. As shown in FIG. The object ID item is identification information that identifies the display data. The Type item is the type of object data, and includes handwriting, text, figure, image, table, etc. Handwriting is stroke data (a sequence of coordinate points). Text is one or more characters or symbols (character code) converted from handwritten data. Figures are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as JPEG, PNG, and TIFF imported from a PC or the Internet. Tables are one-dimensional or two-dimensional table-like objects. One screen of the display device 2 is called a page. The item on the page is the page number. The coordinates field indicates the position of the object data relative to a predetermined origin of the display device 2. The position of the object data is, for example, the upper left vertex of the bounding rectangle of the object data. The coordinates are expressed in pixel units of the display 220, for example. The size item is the width and height of the bounding rectangle of the object data. The input field item indicates the correspondence between the object entered in the input field and the input field. For example, the text with object ID=2 is associated with the input field with input field ID=001.

[0050] <Example of selectable candidates> Next, the operation guide 500 displayed when converting handwritten data will be described with reference to Fig. 7. Fig. 7 shows an example of the operation guide 500 and selectable candidates 530 displayed by the operation guide 500. The operation guide 500 is displayed when the user inputs handwritten data 504 by handwriting (by lifting the pen). The operation guide 500 is displayed when the operation guide 500 is set to be displayed for the input field.

[0051] The operation guide 500 displays operation command candidates 510, handwriting-recognized character string candidates 506, conversion character string candidates 507, and character string / predictive conversion candidates 508. Note that the selectable candidates 530 are the operation command candidates 510, handwriting-recognized character string candidates 506, conversion character string candidates 507, and character string / predictive conversion candidates 508. Note that the selectable candidates 530 excluding the operation command candidates 510 are referred to as character string candidates 539.

[0052] Handwritten data 504 is the character string "file" handwritten by the user. A handwritten data rectangular area display 503 is displayed surrounding handwritten data 504. In FIG. 7, operation guide 500 is displayed after four characters have been entered, but operation guide 500 is displayed when the user stops writing. Therefore, the number of characters in handwritten data 504 can be any number.

[0053] The character string candidates in handwriting recognition character string candidate 506, conversion character string candidate 507, and character string / predictive conversion candidate 508 are arranged in descending order of probability (for convenience of drawing, only one of each is shown in FIG. 7). The "file" in handwriting recognition character string candidate 506 is a candidate for the recognition result. In this example, the character recognition unit 23 correctly recognizes "file."

[0054] Conversion character string candidate 507 is a conversion character string candidate (e.g., a phrase containing "file") converted from the kana-kanji conversion result of "file" (e.g., "file"). Character string / predictive conversion candidate 508 is a predicted character string candidate converted from handwriting recognition character string candidate 506 or conversion character string candidate 507. In this example, "file server" is displayed, which is predicted from "file".

[0055] Operation command candidates 510 are predefined operation command candidates that are displayed according to the recognized characters. In the example of Fig. 7, the first character of the line, "■" 511, is shown as an operation command candidate. In Fig. 7, there are two pieces of command information that partially match the handwriting recognition character string candidate 506, "file," so two operation command candidates 510 are displayed. Operation command candidates 510 are displayed when command information containing the converted character string is found, so they may not always be displayed.

[0056] The operation header 520 has buttons 501, 509, 502, and 505. Button 501 accepts an operation to switch between predictive conversion and kana conversion. Button 502 operates the page of candidate display. In the example of FIG. 7, there are three candidate display pages, and the first page is currently displayed. Button 505 accepts an operation to erase the operation guide 500. When the user presses button 505, the operation acceptance unit 27 accepts it, and the display control unit 24 erases the display except for the handwritten data. Button 509 accepts an operation to erase all displays. When the user presses button 509, the operation acceptance unit 27 accepts it, and the display control unit 24 erases all displays shown in FIG. 7, including the handwritten data, allowing the user to start handwriting again from the beginning.

[0057] 7 is an example, and operation command candidates 510 may be displayed below character string candidates 539, or may be displayed in descending order of past operation history. The user may be able to set the display order.

[0058] <How to determine the recognition direction> A method for determining the recognition direction will be described with reference to Figures 8 to 10. First, T and D are defined as follows. The difference between the top two recognition rates for each direction is D The threshold T is used to compare the difference D. In the explanation of Fig. 8, T = 50. The maximum recognition rate is 100, and the larger the number, the higher the recognition rate is evaluated to be. The maximum recognition rate can be 1, and 100 is merely an example.

[0059] FIG. 8 is a diagram for explaining a method of determining a recognition direction when the recognition direction can be specified by recognition rates (when D > T). In the case of FIG. 8(a), the user inputs handwritten data 301 of "sunny" from the negative y-axis direction. The recognition rates in each of the four directions are as follows. Note that the operation guide 500 is not displayed at the stage where the recognition rates in each of the four directions are calculated. A. "Positive y-axis direction" → Recognition rate: 10 B. "Positive x-axis direction" → Recognition rate: 5 C. "Negative y-axis direction" → Recognition rate: 95 D. "Negative x-axis direction" → Recognition rate: 5 In the case of FIG. 8(a), the top two values of the recognition rates are 10 and 95. Therefore, D = 95 - 10 = 85. Since the recognition direction determination condition "D > T" is satisfied, C. "Negative y-axis direction" with the highest recognition rate is determined as the recognition direction.

[0060] When the recognition direction is determined to be C. "Negative y-axis direction", as shown in FIG. 8(b), the display device 2 displays an operation guide 500 including several string candidates 539 recognized in the negative y-axis direction in the negative y-axis direction. The user can select a string from the string candidates 539.

[0061] Subsequently, referring to FIG. 9, the case where the recognition direction cannot be specified (when D ≤ T) will be described. FIG. 9 is a diagram for explaining a method of determining a recognition direction when the recognition direction cannot be specified by the top two values of the recognition rates (when D ≤ T). In the case of FIG. 9(a), the user inputs handwritten data 302 of "one" from the positive x-axis direction. In the case of FIG. 9(a), the recognition rates in each of the four directions are as follows. A. "Positive y-axis direction" → Recognition rate: 75 B. "Positive x-axis direction" → Recognition rate: 80 C. "Negative y-axis direction" → Recognition rate: 85 D. "Negative x-axis direction" → Recognition rate: 80 In the case of FIG. 9(a), the top two values of the recognition rates are 80 and 85.

[0062] Therefore, D = 85 - 80 = 5. Since the recognition direction determination condition "D > T" does not hold, the recognition direction is not specified. When the display device cannot specify the recognition direction because D ≦ T, it adopts the latest recognized direction determined previously. When there is no previous recognition history, the display device adopts the recognition direction of the default setting.

[0063] In FIG. 9(b), the operation guide 500 is displayed in the "positive y-axis direction". However, if the user continues writing without selecting a character string candidate, the operation guide 500 is erased. Then, as shown in FIG. 10, when the user continues writing, eventually the recognition direction determination condition is satisfied and the correct recognition direction is determined. Thus, the display device 2 can determine the correct recognition direction by judging the recognition direction using the recognition direction determination condition "D > T".

[0064] FIG. 10 is a diagram for explaining a method of determining the recognition direction when the user continues to write by hand from the state of FIG. 9. The user continued writing by hand without selecting a character string candidate from the operation guide 500 in the state of FIG. 9. In the case of FIG. 10(a), the user input handwritten data 303 of "one night" from the positive x-axis direction. That is, the user additionally handwrote "night". Since "one night" belongs to the same recognition group, the character recognition unit 23 collectively performs character recognition on the handwritten data 303 in which "night" is added to the handwritten data 302 of "one". In the case of FIG. 10(a), the recognition rates in each of the four directions are as follows. A. "Positive y-axis direction" → Recognition rate: 10 B. "Positive x-axis direction" → Recognition rate: 80 C. "Negative y-axis direction" → Recognition rate: 20 D. "Negative x-axis direction" → Recognition rate: 10 In the case of FIG. 10(a), the top two values of the recognition rates are 20 and 80. Therefore, D = 80 - 20 = 65. Because the recognition direction determination condition "D > T" is met, B. "positive x-axis direction," which had the highest recognition rate, is determined as the recognition direction. When the recognition direction is determined to be B. "positive x-axis direction," the display device 2 displays, in the positive x-axis direction, an operation guide 500 including several character string candidates 539 recognized in the positive x-axis direction, as shown in FIG. 10(b). The user can select a character string from the character string candidates 539. In this way, the correct recognition direction is updated as the user continues to write by hand. The user does not need to manually determine the recognition direction; they can simply continue writing by hand until the character string is correctly recognized.

[0065] <Operation> 11 is a flowchart illustrating the process of determining the recognition direction based on handwritten input from the user by the recognition direction determination unit 28. The process in FIG. 11 starts when the input receiving unit 21 receives stroke data, for example.

[0066] First, the identical recognition group determination unit 30 identifies stroke data in the identical recognition group (S101). The method for determining stroke data in the identical recognition group will be described later.

[0067] Next, the recognition direction determination unit 28 requests the character recognition unit 23 to perform character recognition on the stroke data of the same recognition group from multiple directions (S102). The multiple directions are, for example, four directions perpendicular to the four sides as shown in Fig. 8, but they can also be further divided at 45 degrees to create eight directions. The character recognition unit 23 calculates the recognition rate for each of the four directions.

[0068] Next, the recognition direction determination unit 28 determines whether the difference D between the top two values ​​of the recognition rate is greater than a threshold T (S103).

[0069] If the determination in step S103 is Yes, the process proceeds to step S104, and if the determination is No, the process proceeds to step S106.

[0070] In step S104, the recognition direction determination unit 28 adopts the recognition result obtained by character recognition in the direction with the highest recognition rate (S104). Note that, although the operation guide 500 is displayed in the following step S105, if the determination in step S103 is Yes, the operation guide 500 does not have to be displayed. For example, there is an age input field in which handwritten data is converted only to numbers, but since the recognition accuracy of numbers is high, there is little need for the user to select numbers from the character string candidates 539.

[0071] The display control unit 24 displays, in the adopted recognition direction, the operation guide 500 including the recognition results (character string candidate 539, operation command candidate 510) obtained by character recognition in the adopted direction (S105).

[0072] In step S106, the recognition direction determination unit 28 determines whether or not the most recent recognition direction is stored (S106). The most recent recognition direction is the direction in which the display device 2 last performed character recognition on stroke data of the same recognition group, and is the direction in which character recognition was performed when the user selected the character string candidate 539 or the operation command candidate 510 from the operation guide 500. Note that the most recent recognition direction is not stored immediately after a firmware update or immediately after the power is turned on, for example.

[0073] If the determination in step S106 is Yes, the process proceeds to step S107, and if the determination is No, the process proceeds to step S108.

[0074] In step S107, the recognition direction determination unit 28 adopts the recognition result obtained by character recognition in the most recent recognition direction (S107). Furthermore, when a character string candidate displayed by the display control unit 24 is selected by the user, the recognition direction determination unit 28 saves the recognition direction of the character string candidate 539. The most recent recognition direction is the recognition direction that was last saved.

[0075] In step S108, the recognition direction determination unit 28 adopts the recognition result obtained by character recognition in the recognition direction of the default setting (S108). The recognition direction of the default setting is the direction of gravity detected by the tilt sensor 217. When the display device 2 is placed flat, the recognition direction of the default setting is the recognition direction predetermined when placed flat.

[0076] The display control unit 24 causes the operation guide 500 including the recognition results (string candidates 539, operation command candidates 510) obtained by character recognition in the direction adopted in steps S107 and S108 to be displayed in the adopted recognition direction (S105). Note that the method for determining the recognition direction in steps S106 to S108 is an example of a predetermined rule.

[0077] The display device 2 of the present embodiment can determine the correct recognition direction by determining the recognition direction using the recognition direction determination condition "D > T". For example, when handwritten data such as "one" is input, it is possible to suppress the setting of an incorrect recognition direction.

[0078] <Determination of Threshold T> Next, referring to FIG. 12, the method for determining the threshold T will be described. The threshold T may be a fixed value, but can be dynamically changed as follows. FIG. 12 is a diagram for explaining the method for determining the threshold T. FIG. 12 shows the top recognition rate and the second recognition rate calculated when a string candidate 539 or an operation command candidate 510 is selected from the operation guide 500. The threshold determination unit 29 accumulates the top recognition rate and the second recognition rate for the past n times. Further, the threshold determination unit 29 can calculate the difference between the top recognition rate and the second recognition rate.

[0079] 12 is a difference that satisfies the recognition direction determination condition "D > T", which means that the user has selected character string candidate 539 or operation command candidate 510, and therefore a threshold T that is equal to or greater than this difference is likely to be appropriate as threshold T. In other words, if the difference in FIG. 12 is determined as threshold T, it is highly likely that the correct recognition direction can be determined based on the recognition direction determination condition. Therefore, threshold determination unit 29 determines threshold T to be the minimum value of the past n differences (plurality of past differences) or a value slightly larger than the minimum value.

[0080] This tends to decrease the threshold value T, making it easier to determine "Yes" in step S103 and displaying the character string candidate 539 in the recognition direction intended by the user. Also, it is easier to determine the correct recognition direction even if the recognition rate changes due to quirks in the user's handwritten data, for example. Also, even if the user changes, the threshold value T can be dynamically determined according to the user. Note that the user may be able to set an arbitrary threshold value T.

[0081] The threshold value T may be updated, for example, whenever a character string candidate 539 or an operation command candidate 510 is selected from the operation guide 500, by the threshold value determination unit 29 calculating the minimum value of the differences over the past n times.

[0082] <Conditions for stroke data to be recognized as the same group> The conditions for forming the same recognition group will be described in detail with reference to Figures 13 and 14. The area change unit 31 changes the area for determining whether stroke data is included in a recognition group depending on whether stroke data input is continuously accepted. In other words, the area for determining whether stroke data is included in a recognition group is changed depending on whether a predetermined time has passed since the input tool was removed from the touch panel.

[0083] A. When the specified time Tm has not elapsed since the pen was lifted (during continuous writing) 13 is a diagram showing the same recognition group when a predetermined time Tm has not yet elapsed since pen-up. The circumscribing rectangle of the stroke data during continuous writing is the recognition group rectangle 101. In contrast, the area of ​​the neighboring rectangle 102 is defined as follows: "The top end of the neighborhood rectangle 102 = β1 from the top end of the recognition group rectangle 101" "Left end of neighborhood rectangle 102 = β2 from the left end of recognition group rectangle 101" "The bottom end of the neighborhood rectangle 102 = the width W of the recognition group rectangle 101 from the bottom end of the recognition group rectangle 101 + β3" "Right end of neighborhood rectangle 102 = height H+β4 of recognition group rectangle 101 from right end of recognition group rectangle 101" Even if some of the stroke data included in the nearby rectangle 102 protrudes, as long as the protruding portion of the stroke data is equal to or less than a predetermined ratio, it is determined that the stroke data was handwritten in the nearby rectangle 102. Furthermore, the stroke data handwritten in the recognition group rectangle 101 may or may not be considered to be part of the nearby rectangle 102.

[0084] Therefore, the stroke data of the recognition group rectangle 101 and the stroke data included in the neighboring rectangle 102 belong to the same recognition group. Therefore, it is possible to determine whether the stroke data handwritten in the upper, left, lower, and right areas of the circumscribing rectangle of the stroke data belong to the same recognition group.

[0085] For example, if the offsets are β1=β2=1.5 [cm] and β3=β4=2 [cm], when the width W of the recognition group rectangle is 1.5 [cm] and the height H is 0.5 [cm], the width and height of the neighborhood rectangle 102 are as follows:

[0086] Width: β2 + (Width W of recognition group rectangle 101) + (Height H of recognition group rectangle 101) + β4 = 5.5 [cm] Height: β1 + (height H of recognition group rectangle 101) + (width W of recognition group rectangle 101) + β3 = 5.5 [cm] Note that the offset varies depending on the size, number of pixels, and usage of the display 220. The above offset is an example assuming a display 220 with a size of about 40 inches and a pixel count of 2880 x 2160, and the size of handwritten data that can be shared among several people. The same applies when not in continuous writing.

[0087] Providing a margin for accepting the handwriting of stroke data by the offsets β1 and β2 to the "left" and "upper" sides of the recognition group rectangle 101 is for the purpose of recognizing rare left (such as "ふ") or upward (i, j, etc.) handwriting, although Japanese characters are mostly written downward or rightward. For this reason, only β1 and β2 have the neighboring rectangle 102 enlarged in the left and upward directions.

[0088] Providing a margin for accepting the handwriting of stroke data to the "right" of the recognition group rectangle 101 is due to the characteristics of the formation of Chinese characters. For example, in the case where a user continuously writes to the right of "イ" (にんべん), the height of "イ" can be assumed to be the character size, and the neighboring rectangle 102 is enlarged by the character size in the right direction.

[0089] Setting an additional length to the "lower" side of the recognition group rectangle 101 is due to the characteristics of the formation of Chinese characters. For example, in the case where a user continuously writes below "宀" (うかんむり), the width of "宀" can be assumed to be the character size, and the neighboring rectangle 102 is enlarged by the character size in the downward direction.

[0090] B. When a predetermined time Tm has elapsed since the pen was lifted FIG. 14 is a diagram showing the same recognition group when a predetermined time Tm has elapsed since the pen was lifted. The circumscribing rectangle of the stroke data written before the predetermined time Tm has elapsed is the recognition group rectangle 101. In contrast, the area of the neighboring rectangle 102 is defined as follows.

[0091] Height: The height H of the recognition group rectangle 101 Width: From the right end of the recognition group rectangle 101 to the height H + α of the recognition group rectangle 101 After the predetermined time Tm has elapsed, assuming the size of horizontally written Japanese characters, a neighborhood rectangle 102 is set to the right by the character size. The area modification unit 31 extends the recognition group rectangle 101 to the right by an offset α, assuming that the user will write to the right with a space in between. The area modification unit 31 determines only the area to the right of the circumscribing rectangle of the stroke data that has already been input as the area for determining whether the stroke data is included in the recognition group.

[0092] The stroke data of "o" in the recognition group rectangle 101 and the stroke data included in the neighboring rectangle 102 belong to the same recognition group.

[0093] For example, if the offset is α=3 [cm], and the width and height of the recognition group rectangle are 4 [cm] and 6 [cm], the width and height of the neighborhood rectangle 102 are as follows:

[0094] Width: (Height H of recognition group rectangle 101) + α = 9 [cm] Height: (Height H of recognition group rectangle 101) = 6 [cm] As described above, the area change unit 31 can change whether or not stroke data is included in a recognition group depending on whether or not the predetermined time Tm has passed since the input tool was released from the touch panel.

[0095] <Conditions for not becoming a group with the same identity> There are many cases where stroke data handwritten in the neighborhood rectangle 102 is not to be included in the same recognition group. This will be explained using FIG. 15. In FIG. 15, handwritten character 110 and table 120 are mixed together. If a user handwrites the frame of table 120 while handwriting character 110, the frame of table 120 may be handwritten before the predetermined time Tm has elapsed after handwriting character 110. This makes it difficult for display device 2 to recognize character 110 separately from table 120. In other words, there is a risk that the frame of table 120 will be recognized together with character 110 "a."

[0096] In addition, there are cases where a user writes explanatory text 111 outside table 120 and writes stroke data so that an arrow 113 points to a figure 112 or the like within table 120. In this case, since the arrow 113 is included in the neighboring rectangle 102 even after a predetermined time Tm has elapsed since the explanatory text 111 "this" was handwritten, there is a risk that "this" and the arrow 113 will be recognized as characters together.

[0097] Therefore, in a case like that shown in FIG. 15, it is preferable not to include stroke data included in the neighboring rectangle 102 in the same recognition group.

[0098] Therefore, the exclusion unit 32 of this embodiment determines the conditions for not forming an identical recognition group as follows.

[0099] 16 is a diagram illustrating the conditions under which stroke data does not belong to the same recognition group. When the stroke data satisfies either of the following conditions (i) or (ii), the exclusion unit 32 exceptionally excludes the stroke data from the same recognition group even if the stroke data is included in the neighboring rectangle 102. (i) The height of the stroke data is greater than threshold a. (ii) The width of the stroke data is greater than threshold b and the height is less than threshold c. The thresholds a and b may be, for example, 9 cm, and the threshold c may be, for example, 2.5 cm, but these values ​​vary depending on the size and number of pixels of the display 220, how many people share the text, and so on.

[0100] Condition (i) sets a threshold a as the maximum character height, and stroke data exceeding this threshold is determined to be a figure. Condition (ii) sets a threshold b as the maximum character width, and stroke data exceeding this threshold is determined to be a figure, or English cursive writing is included in one recognition group.

[0101] It will be explained whether or not stroke data is determined to belong to the same recognition group using areas 1 to 4 divided by thresholds a, b, and c in FIG.

[0102] Stroke data that falls entirely within areas 1 and 2 is a Japanese character because it does not satisfy (i) or (ii). Therefore, stroke data that falls entirely within areas 1 and 2 is not excluded from the same recognition group.

[0103] Stroke data that falls entirely within areas 1, 2, and 3 is a Japanese character because it does not satisfy (i) and (ii). Therefore, stroke data that falls entirely within areas 1, 2, and 3 is not excluded from the same recognition group. This makes it possible to handle English cursive writing. In other words, even if there is stroke data handwritten in cursive writing (one stroke), such as "English," it is not excluded from the same recognition group (it is not considered to be a figure), so the display device 2 can recognize it as a character. Note that stroke data that falls entirely within areas 1, 2, and 3 can also be recognized as English cursive writing.

[0104] Stroke data that falls entirely within areas 2 and 4 satisfies (ii) and is therefore a shape (for example, a horizontal line). Therefore, stroke data that falls entirely within areas 2 and 4 is excluded from the same recognition group.

[0105] Stroke data that falls entirely within areas 1 to 4 is a Japanese character because it does not satisfy (i) or (ii). Therefore, stroke data that falls entirely within areas 1 to 4 is not excluded from the same recognition group. In this case, English cursive writing can also be recognized.

[0106] In this way, the exclusion unit 32 forcibly considers stroke data included in the neighboring rectangle 102 as not belonging to the same recognition group depending on whether the stroke data satisfies the conditions (i) and (ii). This allows the character recognition unit 23 to recognize characters by separating them from figures, even when handwritten characters are mixed together.

[0107] In addition to (i) and (ii), there are the following conditions that prevent a group from being recognized as identical. If the stroke data is not included in the neighboring rectangle - When the previous operation of the Pen 2500 in use is a process other than stroke drawing, such as "character conversion." - In special cases such as area control, when writing is considered to be in a different area If the pen type is different <Procedure for determining the same group> 17 is a flowchart illustrating the procedure for determining stroke data in the same recognition group by the character recognition unit 23. The process of FIG.

[0108] The input receiving unit 21 detects the coordinates touched by the input tool, and the drawing data generating unit 22 generates stroke data. The display control unit 24 displays the stroke data on the display 220. The exclusion unit 32 determines whether the stroke data satisfies the condition for not being part of the same recognition group (S1). Only stroke data that does not satisfy the condition for not being part of the same recognition group is subject to subsequent processing. The determination in step S1 will be explained using the flowchart in FIG. 18.

[0109] The area change unit 31 determines whether a predetermined time Tm has elapsed since the handwriting of the stroke data in step S1 was completed and the pen was lifted (S2).

[0110] When the predetermined time Tm has not yet elapsed (Yes in S2), the input receiving unit 21 detects the coordinates touched by the input tool, and the drawing data generating unit 22 generates stroke data. The display control unit 24 displays the stroke data on the display 220. The exclusion unit 32 determines whether the stroke data of step S3 satisfies the condition for not being part of an identical recognition group (S3). Only stroke data that does not satisfy the condition for not being part of an identical recognition group is subject to subsequent processing.

[0111] Next, the area changing unit 31 sets a neighboring rectangle 102 during continuous writing based on the stroke data in step S1, and determines whether the handwritten stroke data in step S3 is included in this neighboring rectangle 102 (S4).

[0112] If the stroke data in step S3 is included in the neighboring rectangle 102 (Yes in S4), the area change unit 31 determines that the stroke data in steps S1 and S3 belong to the same recognition group (S5).

[0113] If the stroke data in step S3 is not included in the neighboring rectangle 102 (No in S4), the area changing unit 31 determines that the stroke data in steps S1 and S3 are not in the same recognition group (S6).

[0114] When a predetermined time Tm has elapsed since the handwriting of the stroke data in step S1 was completed and the pen was lifted (No in S2), the input receiving unit 21 detects the coordinates touched by the input tool, and the drawing data generating unit 22 generates stroke data. The display control unit 24 displays the stroke data on the display 220. The exclusion unit 32 determines whether the stroke data satisfies the condition for not being part of the same recognition group (S7). Only stroke data that does not satisfy the condition for not being part of the same recognition group is subject to subsequent processing.

[0115] Next, the area change unit 31 sets a nearby rectangle 102 after a predetermined time Tm has elapsed based on the stroke data of step S1, and determines whether the handwritten stroke data of step S7 is included in this nearby rectangle 102 (S8).

[0116] If the stroke data in step S7 is included in the neighboring rectangle 102 (Yes in S8), the area change unit 31 determines that the stroke data in steps S1 and S7 belong to the same recognition group (S5).

[0117] If the stroke data in step S7 is not included in the neighboring rectangle 102 (No in S8), the area change unit 31 determines that the stroke data in steps S1 and S7 are not in the same recognition group (S6).

[0118] FIG. 18 is a flowchart for determining whether or not the stroke data satisfies the conditions for not being in the same recognition group, as described in steps S1, S3, and S7 of FIG.

[0119] The input receiving unit 21 detects the coordinates where the input tool has been touched, and the drawing data generating unit 22 generates stroke data. The display control unit 24 displays the stroke data on the display 220 (S11).

[0120] The exclusion unit 32 determines whether the height of the stroke data is greater than a threshold value a (S12).

[0121] If the height of the stroke data is equal to or smaller than threshold a (No in S12), the exclusion unit 32 determines whether the width of the stroke data in step S11 is greater than threshold b and the height of the stroke data is smaller than threshold c (S13).

[0122] If the determination in either step S12 or step S13 is Yes, the exclusion unit 32 excludes the stroke data handwritten in step S11 from the identical recognition group (S14).

[0123] If the determination in step S13 is No, the area changing unit 31 determines whether the stroke data handwritten in step S11 belongs to the same recognition group. That is, the stroke data handwritten in step S11 is determined whether it is included in the neighboring rectangle 102 in the process of FIG. 17.

[0124] <Major Effects> As described above, the display device 2 can determine the correct recognition direction by using the recognition direction determination condition "D > T" to determine the recognition direction. For example, when handwritten data such as "one" is input, it is possible to suppress the setting of an incorrect recognition direction. Even if an operation guide 500 including a character string candidate 539 recognized in an incorrect recognition direction is displayed, the user can continue writing, and eventually the recognition direction determination condition will be satisfied, and the correct recognition direction can be determined. Thus, the display device 2 can determine the correct recognition direction by using the recognition direction determination condition "D > T" to determine the recognition direction. Also, in the conventional technology, the default recognition direction in which the display device recognizes handwritten data is fixed. Therefore, when the vertical direction seen by the user does not match the direction in which the display device recognizes handwritten data, it takes effort to specify or change in advance the direction in which the display device recognizes handwritten data before the user performs handwritten input. In this embodiment, since it is possible to save the effort of specifying or changing in advance the direction in which handwritten data is recognized, it is possible to reduce the user's effort.

[0125] [Second Embodiment] In this embodiment, an example using machine learning will be described. As described in the first embodiment, the display device 2 performs character recognition on the content handwritten by the user from a plurality of directions and calculates the recognition rate for each of the plurality of directions. When the difference between the recognition rate of the top two values is greater than or equal to a threshold, the display device 2 determines the direction with the highest recognition rate as the recognition direction for recognizing the handwritten data. In this embodiment, the display device 2 allows the user to input feedback on the past determination results of the recognition direction and corrects the conditions (e.g., threshold) for determining the recognition direction.

[0126] FIG. 19 is a functional block diagram of the display device 2 of this embodiment. The display device 2 in FIG. 19 newly includes an analysis unit 36 and a display reception unit 37. The analysis unit 36 accumulates the history of the determination results of the recognition direction and corrects the conditions (e.g., threshold) for determining the recognition direction according to the input from the user. The display reception unit 37 displays the determination result screen 350 in FIG. 21 and receives the user's operation on the determination result screen 350.

[0127] 20 is a flowchart for changing the conditions based on the accumulated results of the recognition direction determination. The process of FIG. 20 is performed, for example, every time the recognition direction is determined.

[0128] First, the analysis unit 36 ​​analyzes the number of times the recognition direction has been determined (S501). Next, the analysis unit 36 ​​determines whether the number of times the recognition direction has been determined is 10 or more (S502). 10 times is just an example, and it may be 1 or more. If the determination in step S502 is No, the process returns to step S501.

[0129] If the determination in step S502 is Yes, the display receiving unit 37 displays the determination result screen 350 (S503). An example of the determination result screen 350 is shown in FIG.

[0130] The display receiving unit 37 determines whether or not it has received a "No" operation from the user (S504). That is, the display receiving unit 37 displays the determination result screen 350 and receives input from the user as to whether or not the determination result is what the user desires. If the determination in step S504 is No, the process returns to step S501.

[0131] If the "No" operation is received (Yes in step S504), the analysis unit 36 ​​changes the conditions for determining the recognition direction from the next time onwards (S505).

[0132] Specifically, the analysis unit 36 ​​reduces the value of the threshold T that is compared with the difference D between the top two recognition rates. By reducing the value of the threshold T, it becomes easier to identify the recognition direction, allowing the user to perform the operation they desire. Alternatively, the value of the threshold T may be increased to identify the recognition direction with the most outstanding recognition rate.

[0133] Alternatively, the analysis unit 36 ​​may change the timing for comparing the difference D with the threshold T to a timing other than the timing when a "stroke recognized as belonging to the same recognition group" is drawn. Users often continue writing in the same direction. Therefore, the recognition direction determination unit 28 may compare the difference D with the threshold T when multiple strokes are handwritten (for example, when a "stroke recognized as belonging to the same recognition group" is drawn five times, or when the user writes five words). By identifying the recognition direction using multiple stroke data in this way, the recognition accuracy can be improved.

[0134] The judgment result screen 350 in Fig. 21 displays a list of the past 10 recognition direction judgment results. The judgment result screen 350 has a judgment table 351. The judgment table 351 has the number, the date and time of handwriting, the handwriting recognition direction, and the recognition result. The handwriting recognition direction is the recognition direction determined by the recognition direction determination unit 28. The recognition result is the result of character recognition.

[0135] Furthermore, the judgment result screen 350 displays a note 352 saying "The past 10 judgment results for the handwriting recognition direction are displayed. Are the judgment results what the operator intended?", a Yes button 353, and a No button 354. If necessary, the display receiving unit 37 may also display the judgment results (the content displayed on the display screen after the judgment) on the judgment result screen 350 as history information.

[0136] If the user presses the Yes button 353 on the determination result screen 350, the analysis unit 36 ​​does not correct the threshold value T for determining the recognition direction because the determination result is what the user intended.

[0137] On the other hand, if the user presses the No button 354, the result of the determination is not what the user intended, so the analysis unit 36 ​​corrects the threshold T for determining the recognition direction.

[0138] Note that the example of correcting the threshold value T is not limited to this. For example, when the user clicks the No button 354 on the screen, the threshold value T used by the display device 2 to determine the recognition direction may be displayed on the screen, and the user may change it by directly inputting a value.

[0139] 20, the display device 2 may determine whether user authentication has been performed, and if authentication has been performed, execute this flow. With this configuration, the display device 2 can identify the recognition direction using set conditions (thresholds) according to each operator. If authentication is not performed, the display device 2 is shared by multiple users, and therefore the flowchart in FIG. 20 is not executed.

[0140] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, the threshold value used by the display device 2 to determine the recognition direction can be adjusted by input from the user.

[0141] [Third embodiment] In this embodiment, a display system in a system form will be described in which an information processing system on a network performs processing such as character recognition and returns a character string as the recognition result to the display device 2.

[0142] In the description of this embodiment, components or contents of the figures that are given the same symbols as in the first embodiment perform the same functions, so the description of components that have already been described may be omitted or only the differences may be described.

[0143] 22 is an example of a system configuration diagram of a display system 100. The display system 100 has a display device 2 and an information processing system 10 that can communicate with each other via a network N.

[0144] The display device 2 is placed in a facility such as a company, and is connected to a LAN or Wi-Fi installed within the facility. The information processing system 10 is placed, for example, in a data center. The display device 2 is connected to the Internet i via a firewall 8, and the information processing system 10 is also connected to the Internet i via a high-speed LAN or the like within the data center.

[0145] The display device 2 may be connected to the Internet i using wireless communication such as a telephone line network. In this case, the wireless communication may be 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or the like.

[0146] The information processing system 10 has one or more information processing devices, and the one or more information processing devices act as servers to provide services to the display device 2. A server is a computer or software that performs the function of providing information and processing results in response to requests from clients. As will be described later, the information processing system 10 receives pen coordinates from the display device 2 and transmits information required to display the operation guide 500 shown in FIG. 7 to the display device 2.

[0147] The configuration of the display device 2 may be the same as that of the first embodiment, but in this embodiment, it is sufficient to have a touch panel, a display, and a communication function. The display device 2 may include multiple computing devices configured to communicate with each other.

[0148] In this embodiment, a general information processing device such as a PC or tablet can execute a web browser or a dedicated application. The web browser or the dedicated application communicates with the information processing system 10. When the web browser is running, the user inputs or selects the URL of the information processing system 10 to connect the display device to the information processing system 10. The display device 2 executes a web application provided by the information processing system 10 in the web browser. A web application refers to software or a mechanism thereof that operates through cooperation between a program written in a programming language (e.g., JavaScript (registered trademark)) that runs on the web browser and a program on the web server side.

[0149] When the dedicated application runs, it connects to a pre-registered URL of the information processing system 10. The dedicated application has a program and a user interface, so the program sends and receives necessary information to and from the information processing system 10 and displays it on the user interface.

[0150] The communication method may be a general-purpose communication protocol such as HTTP, HTTPs, or WebSocket, or a dedicated communication protocol.

[0151] <Hardware configuration example> The hardware configuration of the display device 2 may be the same as that shown in Fig. 6. In this embodiment, an example of the hardware configuration of the information processing system 10 will be described.

[0152] Fig. 23 is a hardware configuration diagram of the information processing system 10. As shown in Fig. 23, the information processing system 10 is constructed by a computer, and includes a CPU 601, a ROM 602, a RAM 603, an HD 604, an HDD (Hard Disk Drive) controller 605, a display 606, an external device connection I / F (Interface) 608, a network I / F 609, a bus line 610, a keyboard 611, a pointing device 612, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 616.

[0153] Of these, the CPU 601 controls the overall operation of the information processing system 10. The ROM 602 stores programs, such as an IPL, used to drive the CPU 601. The RAM 603 is used as a work area for the CPU 601. The HD 604 stores various data, such as programs. The HDD controller 605 controls the reading and writing of various data from and to the HD 604 under the control of the CPU 601. The display 606 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 608 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memories and printers. The network I / F 609 is an interface for data communication using a communication network. The bus line 610 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 601, shown in FIG. 23.

[0154] The keyboard 611 has a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 612 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 614 controls reading and writing of various data from a DVD-RW 613, which is an example of a removable recording medium. The DVD-RW 613 may be a DVD-R or the like. The media I / F 616 controls reading and writing (storing) of data from a recording medium 615, such as a flash memory.

[0155] <About the device functions> Next, functions of the display system 100 will be described with reference to Fig. 24. Fig. 24 is an example of a functional block diagram showing functions of the display system 100 in blocks. Note that the description of Fig. 24 will mainly focus on the differences from Fig. 5.

[0156] In this embodiment, the display device 2 has an input receiving unit 21, a drawing data generating unit 22, a display control unit 24, a data recording unit 25, a network communication unit 26, and an operation receiving unit 27. These functions may be the same as those in FIG.

[0157] <<Functions of the information processing system>> The information processing system 10 has a character recognition unit 23, a recognition direction determination unit 28, a threshold determination unit 29, a communication unit 33, and an identical recognition group determination unit 30. Each function of the information processing system 10 is a function or means realized when any of the components shown in Fig. 23 operates in response to an instruction from the CPU 601 in accordance with a program loaded from the HD 604 onto the RAM 603.

[0158] The communication unit 33 receives stroke data, gesture operations, etc. from the display device 2, and transmits the recognition result, the recognition direction, etc. to the display device 2. Other functions are the same as those in the first embodiment, or, even if they are different, they will not hinder the explanation of this embodiment.

[0159] <<Action or Processing>> FIG. 25 is a sequence diagram illustrating a process in which the display device 2 communicates with the information processing system 10 and displays the recognition result of handwritten data.

[0160] S201: The display device 2 repeats steps S201 to S203 while determining whether the display device 2 is in the same recognition group. The network communication unit 26 of the display device 2 transmits the stroke data to the information processing system 10.

[0161] S202: The communication unit 33 of the information processing system 10 receives the stroke data, and the character recognition unit performs character recognition each time a piece of stroke data is added. This allows the information processing system 10 to immediately return the recognition result when pen-up is detected. The character recognition unit performs character recognition in each of the four directions and calculates the recognition rate for each of the four directions.

[0162] S203: The display control unit 24 of the display device 2 causes the display 220 to display the stroke data in parallel with the transmission of the stroke data.

[0163] S204: When the user holds up the pen for a certain period of time or longer, the communication unit 33 of the information processing system 10 detects the pen-up by not receiving any stroke data. The fact that the pen-up has occurred may be explicitly transmitted. The stroke data of the same recognition group is identified by the pen-up.

[0164] S205: When the communication unit 33 of the information processing system 10 detects that the pen has been lifted, the recognition direction determination unit 28 determines the recognition direction, similarly to the first embodiment.

[0165] S206: The communication unit 33 of the information processing system 10 transmits to the display device 2 the recognition direction, the character string candidate 539, and the operation command candidate 510 stored in the character recognition unit.

[0166] S207: The network communication unit 26 of the display device 2 receives the character string candidates and operation command candidates, and the display control unit 24 displays the operation guide 500 including these candidates in the recognition direction of the received character string candidates and operation command candidates. In other words, the up-down direction as seen by the user in the recognition direction matches the up-down direction of the character string.

[0167] S208: When a character string candidate is selected from the operation guide 500, the operation receiving unit 27 receives it, and the display control unit 24 deletes the handwritten data and displays the selected character string candidate as the conversion result. When an operation command candidate 510 is selected, the operation receiving unit 27 receives it, and the display device 2 executes the operation command. When the operation command is executed, communication may occur between the display device 2 and the information processing system 10, or the processing may be completed within the display device 2.

[0168] S209: If the user continues handwriting without selecting the character string candidate 539 or the like from the operation guide 500, the display control unit 24 erases the operation guide 500. In this case, the process returns to step S201.

[0169] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, handwritten input is possible on any display device 2 connected to a network.

[0170] [Other application examples] The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0171] For example, instead of the difference between the top two recognition rates, the difference between the top and the average of the remaining three (excluding the top) may be compared with the threshold.

[0172] Alternatively, a learning device may generate a model in advance that associates the stroke order of stroke data constituting handwritten data with the recognition direction through machine learning, and the display device 2 may use this model to estimate the recognition direction. The display device 2 weights the recognition direction estimated by the model to the recognition rate of that direction and calculates the difference between the top two values. This improves the accuracy of the recognition direction.

[0173] For example, although an electronic whiteboard has been described as an example in this embodiment, any information processing device having a touch panel can be suitably applied. Furthermore, devices having similar functions to an electronic whiteboard are also called electronic whiteboards, electronic information boards, interactive boards, etc. An information processing device equipped with a touch panel may also be, for example, a display device using a projector.

[0174] In this embodiment, the coordinates of the pen tip are detected by detecting the coordinates of the pen tip on a touch panel, but the coordinates of the pen tip may also be detected by ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The display device 2 can identify the position of the pen based on the direction and distance. The projector draws (projects) the trajectory of the pen as a stroke.

[0175] In addition, the configuration examples in Figure 5 and the like are divided according to main functions to make it easier to understand the processing by the display device 2. The method of dividing the processing units and their names do not limit the present invention. The processing by the display device 2 can be divided into even more processing units depending on the processing content. Also, it can be divided so that one processing unit includes even more processes.

[0176] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions. Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, the methods by which information is stored and presented in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0177] <Additional Notes> [Appendix 1] a character recognition unit that recognizes characters in handwritten input data from a plurality of directions and calculates a recognition rate for each of the plurality of directions; determining, when a difference between the top two recognition rates among the calculated plurality of recognition rates is equal to or greater than a threshold, the direction with the highest recognition rate as the recognition direction of the handwriting input; a recognition direction determination unit that, when a difference between the top two recognition rates among the calculated plurality of recognition rates is smaller than a threshold, calculates a recognition rate for each of the plurality of directions, including the handwritten data whose difference is determined to be smaller than the threshold and newly input handwritten data, and, when a difference between the top two recognition rates among the calculated recognition rates is equal to or greater than the threshold, determines the direction with the highest recognition rate as the recognition direction of the handwritten data; a display control unit that displays a character string obtained by character recognition of the handwritten data in the determined recognition direction, in the recognition direction determined by the recognition direction determination unit; A display device having: [Appendix 2] the display control unit displays a plurality of character string candidates obtained by character recognition of the handwritten data in the determined recognition direction, in the recognition direction determined by the recognition direction determination unit; an operation receiving unit that receives a selection of one character string candidate from the plurality of character string candidates from a user; 2. The display device according to claim 1, wherein the display control unit displays the selected character string candidate as the character string in the recognition direction determined by the recognition direction determination unit. [Appendix 3] When the difference is smaller than the threshold value, the display control unit displays the plurality of character string candidates in a direction determined by a predetermined rule, When the operation acceptance unit does not accept the selection of the character string candidate, the character recognition unit calculates a recognition rate for each of the plurality of directions, including the handwritten data for which the difference is determined to be smaller than the threshold value and newly accepted handwritten data; The display device according to claim 2, wherein the recognition direction determination unit determines the direction with the highest recognition rate as the recognition direction of the handwritten data when a difference between the top two recognition rates among the calculated recognition rates is equal to or greater than the threshold value. [Appendix 4] the recognition direction determination unit stores a recognition direction of the character string when the character string displayed by the display control unit is selected by a user; If the difference is less than the threshold, 4. The display device according to claim 3, wherein the display control unit displays the character string in the last saved recognized direction as the predetermined rule. [Appendix 5] If the recognition direction determination unit does not hold the recognition direction, The display device according to claim 4, wherein the display control unit displays the character string in a recognition direction that is set as a default as the predetermined rule. [Appendix 6] The display device according to claim 5, wherein the default recognized direction is the direction of gravity determined by a tilt sensor provided in the display device. [Appendix 7] When the tilt sensor of the display device determines that the display device is placed flat, The display device according to claim 5, wherein the default recognition direction is a predetermined direction. [Appendix 8] the recognition direction determination unit accumulates a difference between the top two recognition rates when the character string displayed by the display control unit is selected by a user; 8. The display device according to any one of appendixes 1 to 7, further comprising a threshold value determining unit that determines the minimum value of a plurality of past differences as the threshold value. [Appendix 9] the character recognition unit performs character recognition on a plurality of stroke data included in a recognition group together; when a predetermined time has elapsed since the input means was removed from the touch panel, only the area to the right of the circumscribing rectangle of the stroke data that has already been input is determined as an area for determining whether the stroke data is included in the recognition group; The display device according to any one of appendices 1 to 8, further comprising an area change unit that, if a predetermined time has not elapsed since the input means was removed from the touch panel, determines the areas above, to the left, below, and to the right of the circumscribing rectangle of the stroke data that has already been input as areas for determining whether the stroke data is included in the recognition group. [Appendix 10] an analysis unit that accumulates a determination result of the recognition direction determined by the recognition direction determination unit and analyzes the determination result; a display receiving unit that displays a determination result by the analysis unit and receives a change of the threshold value for determining the recognition direction based on the determination result; 2. The display device according to claim 1, further comprising: [Explanation of symbols]

[0178] 2 Display device [Prior art documents] [Patent documents]

[0179] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-076380

Claims

1. a character recognition unit that recognizes characters in handwritten input data from a plurality of directions and calculates a recognition rate for each of the plurality of directions; determining a direction with the highest recognition rate as the recognition direction of the handwritten data when a difference between the top two recognition rates among the calculated plurality of recognition rates is equal to or greater than a threshold value; a recognition direction determination unit that, when a difference between the top two recognition rates among the calculated plurality of recognition rates is smaller than a threshold, calculates a recognition rate for each of the plurality of directions, including the handwritten data whose difference is determined to be smaller than the threshold and newly input handwritten data, and, when a difference between the top two recognition rates among the calculated recognition rates is equal to or greater than the threshold, determines the direction with the highest recognition rate as the recognition direction of the handwritten data; a display control unit that displays a character string obtained by character recognition of the handwritten data in the determined recognition direction, in the recognition direction determined by the recognition direction determination unit; A display device having:

2. the display control unit displays a plurality of character string candidates obtained by character recognition of the handwritten data in the determined recognition direction, in the recognition direction determined by the recognition direction determination unit; an operation receiving unit that receives a selection of one character string candidate from the plurality of character string candidates from a user; The display device according to claim 1 , wherein the display control unit displays the selected character string candidate as the character string in the recognition direction determined by the recognition direction determination unit.

3. When the difference is smaller than the threshold value, the display control unit displays the plurality of character string candidates in a direction determined by a predetermined rule, When the operation acceptance unit does not accept the selection of the character string candidate, the character recognition unit calculates a recognition rate for each of the plurality of directions, including the handwritten data for which the difference is determined to be smaller than the threshold value and newly accepted handwritten data; 3. The display device according to claim 2, wherein the recognition direction determination unit determines the direction with the highest recognition rate as the recognition direction of the handwritten data when the difference between the top two recognition rates among the calculated recognition rates is equal to or greater than the threshold value.

4. the recognition direction determination unit stores a recognition direction of the character string when the character string displayed by the display control unit is selected by a user; If the difference is less than the threshold, The display device according to claim 3 , wherein the display control unit displays the character string in the recognized direction that was last saved as the predetermined rule.

5. If the recognition direction determination unit does not hold the recognition direction, The display device according to claim 4 , wherein the display control unit displays the character string in a recognition direction that is set as a default as the predetermined rule.

6. The display device according to claim 5 , wherein the default recognized direction is the direction of gravity determined by a tilt sensor included in the display device.

7. When the tilt sensor of the display device determines that the display device is placed flat, The display device according to claim 5 , wherein the default recognized direction is a predetermined direction.

8. the recognition direction determination unit accumulates a difference between the top two recognition rates when the character string displayed by the display control unit is selected by a user; 8. The display device according to claim 1, further comprising a threshold value determining unit that determines the minimum value of a plurality of past differences as the threshold value.

9. the character recognition unit performs character recognition on a plurality of stroke data included in a recognition group together; when a predetermined time has elapsed since the input means was removed from the touch panel, only the area to the right of the circumscribing rectangle of the stroke data that has already been input is determined as an area for determining whether the stroke data is included in the recognition group; 2. The display device according to claim 1, further comprising an area change unit that, if a predetermined time has not elapsed since the input means was removed from the touch panel, determines the areas above, to the left, below, and right of the circumscribing rectangle of the stroke data that has already been input as areas for determining whether the stroke data is included in the recognition group.

10. an analysis unit that accumulates a determination result of the recognition direction determined by the recognition direction determination unit and analyzes the determination result; a display receiving unit that displays a determination result by the analysis unit and receives a change of the threshold value for determining the recognition direction based on the determination result; The display device according to claim 1 , further comprising:

11. A display method performed by a display device, A process of performing character recognition on handwritten input data from a plurality of directions and calculating a recognition rate for each of the plurality of directions; determining a direction with the highest recognition rate as the recognition direction of the handwritten data when a difference between the top two recognition rates among the calculated plurality of recognition rates is equal to or greater than a threshold value; when a difference between the top two recognition rates among the calculated plurality of recognition rates is smaller than a threshold value, the calculating process calculates a recognition rate for each of the plurality of directions, including the handwritten data whose difference is determined to be smaller than the threshold value and newly input handwritten data, and when a difference between the top two recognition rates among the calculated recognition rates is equal to or greater than the threshold value, determines the direction with the largest recognition rate as the recognition direction of the handwritten data; a process of displaying, in the determined recognition direction, a character string obtained by character recognition of the handwritten data in the determined recognition direction; Display method.

12. The display device a character recognition unit that recognizes characters in handwritten input data from a plurality of directions and calculates a recognition rate for each of the plurality of directions; determining a direction with the highest recognition rate as the recognition direction of the handwritten data when a difference between the top two recognition rates among the calculated plurality of recognition rates is equal to or greater than a threshold value; a recognition direction determination unit that, when a difference between the top two recognition rates among the calculated plurality of recognition rates is smaller than a threshold, calculates a recognition rate for each of the plurality of directions, including the handwritten data whose difference is determined to be smaller than the threshold and newly input handwritten data, and, when a difference between the top two recognition rates among the calculated recognition rates is equal to or greater than the threshold, determines the direction with the highest recognition rate as the recognition direction of the handwritten data; a display control unit that displays, in the recognition direction determined by the recognition direction determination unit, a character string obtained by character recognition of the handwritten data in the determined recognition direction; A program to function as a

13. A display system in which a display device and an information processing system communicate with each other via a network, The display device includes: a network communication unit that transmits the contents of handwritten data input by hand to the information processing system; The information processing system includes: a character recognition unit that recognizes characters in the handwritten data transmitted from the display device from a plurality of directions and calculates a recognition rate for each of the plurality of directions; determining the direction with the highest recognition rate as the recognition direction of the handwritten data when a difference between the top two recognition rates among the calculated plurality of recognition rates is equal to or greater than a predetermined threshold value; a recognition direction determination unit that, when a difference between the top two recognition rates among the calculated plurality of recognition rates is smaller than a threshold, calculates a recognition rate for each of the plurality of directions, including the handwritten data whose difference is determined to be smaller than the threshold and newly input handwritten data, and, when a difference between the top two recognition rates among the calculated recognition rates is equal to or greater than the threshold, determines the direction with the highest recognition rate as the recognition direction of the handwritten data; a communication unit that transmits the recognition direction determined by the recognition direction determination unit and a character string recognized in the recognition direction to the display device. The display device is a display system that displays the character string in the received recognition direction.

Images