Display device, display method, program, and information processing system

Abstract

To add a punctuation mark at the end of text data without dependence on the amount of input handwritten by a user.SOLUTION: A display device includes: a handwritten input unit which receives a handwritten input; a handwritten recognition control unit which recognizes the handwritten input, converts the recognized handwritten input into text data, and adds data indicating a blank at the end of the text data; a handwritten input display control unit which causes a display unit to display the text data with the blank data added at the end thereof and one or more operation command candidates for processing the text data; and an operation command receiving unit which receives an operation command selected by a user from among the operation command candidates. The handwritten input display control unit is configured to display, on the display unit, the text data with the blank data which has been converted into data indicating a punctuation mark when the operation command selected by the user is a command to convert the blank data added at the end of the text data into the data indicating a punctuation mark.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a display device, a display method, a program, and an information processing system. [Background technology]

[0002] 2. Description of the Related Art In a display device such as an interactive whiteboard device that accepts handwritten input from a user, text data is recognized from the handwritten input.

[0003] For example, in boxless handwritten character recognition technology that recognizes handwritten input and converts it into text, a technology that outputs a character string including spaces as a result of handwritten character recognition that takes into account the spaces between characters has been known for some time (see, for example, Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0004] A display device that recognizes text data from a user's handwritten input may accept the handwritten input up to a punctuation mark, such as a punctuation mark, question mark, or parentheses, and after recognizing the text data from the handwritten input, may accept the addition of a punctuation mark to the end of the text data via handwritten input.

[0005] However, even if the display device accepts the addition of punctuation marks to the end of recognized text data by handwriting input from the user, the amount of handwriting input available for the display device to recognize the text data is small, so the accuracy of the recognition of the handwritten punctuation marks does not improve, and the display device may not be able to display the recognition result expected by the user. In other words, the accuracy of the recognition of handwritten punctuation marks depends on the amount of handwriting input by the user. Note that Patent Document 1 does not mention such a problem.

[0006] An object of one embodiment of the present invention is to be able to add punctuation marks to the end of text data, regardless of the amount of handwritten input by a user. [Means for solving the problem]

[0007] One embodiment of the present invention is a display device comprising a handwriting input unit that accepts handwritten input, a handwriting recognition control unit that recognizes the handwritten input, converts the recognized handwritten input into text data, and adds data indicating a space to an end of the text data, a handwriting input display control unit that causes the text data with the data indicating a space added to the end and one or more operation command candidates for processing the text data to be displayed on a display unit, and an operation command accepting unit that accepts an operation command selected by a user from the operation command candidates, wherein if the operation command selected by the user is a command to convert the data indicating a space added to the end of the text data into data indicating a punctuation mark, the handwriting input display control unit causes the text data with the data indicating the space converted into the data indicating the punctuation mark to be displayed on the display unit. Effect of the Invention

[0008] According to an embodiment of the present invention, punctuation marks can be added to the end of text data regardless of the amount of handwritten input by a user. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device according to the present embodiment. [Diagram 2] FIG. 2 is a functional block diagram of an example of a display device according to the present embodiment. [Diagram 3] 10 is a flowchart illustrating an example of processing of the display device according to the embodiment. [Figure 4] 1 is a display example (part 1) of an example of the display device according to the present embodiment. [Diagram 5] 1 is a display example (part 1) of an example of the display device according to the present embodiment. [Figure 6] 1 is a display example (part 1) of an example of the display device according to the present embodiment. [Figure 7]13 is a display example (part 2) of the example of the display device according to the present embodiment. [Figure 8] 13 is a display example (part 2) of the example of the display device according to the present embodiment. [Figure 9] 13 is a display example (part 2) of the example of the display device according to the present embodiment. [Figure 10] 13 is a display example (part 3) of the example of the display device according to the present embodiment. [Figure 11] 13 is a display example (part 3) of the example of the display device according to the present embodiment. [Figure 12] 13 is a display example (part 3) of the example of the display device according to the present embodiment. [Figure 13] 13 is a fourth display example of the display device according to the present embodiment. [Figure 14] 13 is a fourth display example of the display device according to the present embodiment. [Figure 15] 5 is a display example (part 5) of the example of the display device according to the present embodiment. [Figure 16] 5 is a display example (part 5) of the example of the display device according to the present embodiment. [Figure 17] 10 is a flowchart illustrating an example of a process in which the display device according to the embodiment changes the display order of operation command candidates. [Figure 18] 13 is an image diagram of an example of a screen for proposing a change in the display order of operation command candidates. FIG. [Figure 19] 1 is a system configuration diagram of an example of an information processing system including a display device according to an embodiment of the present invention. [Figure 20] FIG. 2 is a hardware configuration diagram of an example of a server device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0011] <Terminology> Handwriting input refers to inputting handwritten data using an input means capable of designating coordinates, such as a pen, a human finger, or a rod-shaped member. A stroke is a series of operations in which the user presses the input means against a touch panel, moves it continuously, and then removes the input means from the touch panel. Note that a stroke also includes a series of operations in which the user moves the designation of coordinates without touching the input means to the touch panel.

[0012] The stroke data is information displayed on a display or the like based on a trajectory of coordinates input by an input means. The stroke data may be appropriately interpolated. The handwritten data is data having one or more stroke data. An object is a display object displayed on a display or the like based on the stroke data. Objects obtained by converting stroke data through handwritten recognition may include character strings, stamps displayed as marks, figures such as circles and stars, straight lines, and the like.

[0013] <Hardware configuration> Fig. 1 is a hardware configuration diagram of an example of a display device 2 according to this embodiment. The display device 2 shown in Fig. 1 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an SSD (Solid State Drive) 204.

[0014] The CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data, such as an OS and programs for the display device 2. The program may be, for example, an application program that runs on a computer equipped with a general-purpose OS. In this case, the display device 2 is used as a general-purpose computer before executing the application program, but after executing the application program, it can accept handwritten input from a user, just like a dedicated device for the display device 2.

[0015] The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a tilt sensor 217, a serial interface 218, a speaker 219, a display 220, a microphone 221, a wireless communication device 222, an infrared I / F 223, a power control circuit 224, an AC adapter 225, a battery 226, and a power switch 227.

[0016] The display controller 213 controls and manages screen display to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500 or the user's hand or the like (the pen or the user's hand serves as an input means) has touched the display 220. The touch sensor 216 may also receive a pen ID.

[0017] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. A method for inputting and detecting coordinates will be described. For example, in the case of an optical type, two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays in parallel to the display 220. The two light receiving and emitting devices receive light that is reflected by a reflecting member installed around the display 220 and returns along the same optical path as the optical path of the light emitted by the light receiving element.

[0018] The touch sensor 216 outputs position information of the infrared rays emitted by the two light receiving and emitting devices that are blocked by the object to the touch sensor controller 215. The touch sensor controller 215 identifies the coordinate position that is the contact position of the object. The touch sensor controller 215 has a communication unit 215a and may be able to wirelessly communicate with the pen 2500. For example, when communication is performed according to a standard such as Bluetooth (registered trademark), a commercially available pen 2500 can be used. If one or more pens 2500 are registered in advance in the communication unit 215a, the user can communicate without performing connection settings to allow the pen 2500 to communicate with the display device 2.

[0019] The power switch 227 is a switch for switching ON / OFF the power of the display device 2. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. The serial interface 218 is an interface for communication with the outside, such as a USB. The serial interface 218 is used for inputting information from the outside. The speaker 219 is used for outputting audio. The microphone 221 is used for inputting audio. The wireless communication device 222 communicates with a network, such as the Internet. The wireless communication device 222 communicates using Wi-Fi (registered trademark), Bluetooth (registered trademark), or the like, but any communication standard is acceptable.

[0020] The infrared I / F 223 detects the display device 2 arranged adjacently by utilizing the linearity of infrared rays. It is preferable that one infrared I / F 223 is provided on each side, and it is possible to detect in which direction of the display device 2 another display device 2 is arranged. The power supply control circuit 224 controls the AC adapter 225 and the battery 226 which are the power sources of the display device 2. The AC adapter 225 converts the alternating current provided by a commercial power source into direct current.

[0021] When the display 220 is electronic paper, little or no power is consumed to maintain the display of an image, and therefore the display device 2 can also be driven by a battery 226. When the display 220 is electronic paper, it becomes possible to use the display device 2 for purposes such as digital signage even in places such as outdoors where it is difficult to connect the AC adapter 225 to a commercial power source.

[0022] The display device 2 also includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0023] The touch sensor 216 is not limited to an optical type, and may be a capacitive touch panel that identifies the contact position by detecting a change in capacitance. The touch sensor 216 may be a resistive film type touch panel that identifies the contact position by a voltage change in two opposing resistive films. The touch sensor 216 may be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by a contact object touching the display unit, and may use various detection means. The touch sensor 216 may be of a type that does not require an electronic pen to detect the presence or absence of a touch with the pen tip. In the case of a type that does not require an electronic pen to detect the presence or absence of a touch with the pen tip, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not need to be a long, thin pen type.

[0024] <Functional configuration> Fig. 2 is a functional block diagram of an example of the display device 2 according to the present embodiment. The display device 2 shown in Fig. 2 includes a handwriting input unit 21, a display unit 22, a handwriting input display control unit 23, a candidate display timer control unit 24, a handwriting input storage unit 25, a handwriting recognition control unit 26, a handwriting recognition dictionary unit 27, a character string conversion control unit 28, a character string conversion dictionary unit 29, a predictive conversion control unit 30, a predictive conversion dictionary unit 31, an operation command reception unit 32, and an operation command definition unit 33. Each function of the display device 2 is a function or means realized by any of the components shown in Fig. 1 operating according to an instruction from the CPU 201 in accordance with a program expanded from the SSD 204 onto the RAM 203.

[0025] The handwriting input unit 21 is realized by a touch sensor 216 or the like, and accepts handwriting input by a user. The handwriting input unit 21 converts the user's pen input d1 into pen operation data d2 (pen up, pen down, or pen coordinate data) and transmits it to the handwriting input display control unit 23. The pen coordinate data is periodically transmitted as discrete values. Coordinates between discrete values ​​of the pen coordinate data are interpolated.

[0026] The display unit 22 is realized by a display 220 or the like, and displays handwritten input objects and operation menus. The display unit 22 converts the drawing data d3 written in the video memory by the handwritten input display control unit 23 into data according to the characteristics of the display 220, and displays the converted data on the display 220.

[0027] The handwriting input display control unit 23 performs overall control regarding handwriting input and display. The handwriting input display control unit 23 processes the pen operation data d2 from the handwriting input unit 21, and causes the display unit 22 to display the data.

[0028] The candidate display timer control unit 24 is a display control timer for selectable candidates. The candidate display timer control unit 24 starts or stops a timer to generate the timing for starting the display of selectable candidates and the timing for erasing the display. The candidate display timer control unit 24 receives a timer start request d4 (which may also be a timer stop request) from the handwriting input display control unit 23, and transmits a timeout event d5 to the handwriting input display control unit 23.

[0029] The handwriting input saving unit 25 has a storage function for saving user data (handwritten objects / character string objects). The handwriting input saving unit 25 saves user data d6-1 received from the handwriting input display control unit 23. The handwriting input saving unit 25 receives an acquisition request d6-2 from the handwriting input display control unit 23, and transmits user data d7 saved in the handwriting input saving unit 25 to the handwriting input display control unit 23. The handwriting input saving unit 25 transmits position information d36 of the confirmed object to the operation command receiving unit 32.

[0030] The handwriting recognition control unit 26 is a recognition engine that performs handwriting recognition. The handwriting recognition performed by the handwriting recognition control unit 26 is different from a general OCR (Optical Character Reader) in that it recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%, $, &, etc.), figures (lines, circles, triangles, etc.), punctuation marks in a sentence (punctuation marks, question marks, parentheses, etc.), or blanks (spaces) in parallel with the user's pen operation. Various algorithms have been devised for the handwriting recognition method. In this embodiment, a known technology is used, and details will be omitted.

[0031] The handwriting recognition control unit 26 receives the pen operation data d8-1 from the handwriting input display control unit 23, performs handwriting recognition, and holds the handwriting-recognized character string candidate. The handwriting recognition control unit 26 also holds the language string candidate d13 converted from the handwriting-recognized character string candidate d12 using the handwriting recognition dictionary unit 27. When the handwriting recognition control unit 26 receives an acquisition request d8-2 from the handwriting input display control unit 23, it transmits the handwriting-recognized character string candidate and the language string candidate d9 held by it to the handwriting input display control unit 23. The handwriting recognition control unit 26 adds, inserts, or gives data indicating a space to the end of the handwriting-recognized character string candidate and the language string candidate d9.

[0032] The handwriting recognition dictionary unit 27 is dictionary data for language conversion in handwriting recognition. The handwriting recognition dictionary unit 27 receives handwriting recognition character string candidates d12 from the handwriting recognition control unit 26, converts them into linguistically probable language character string candidates d13, and transmits them to the handwriting recognition control unit 26. For example, in the case of Japanese, hiragana is converted into kanji or katakana. In the case of English, it is converted into the alphabet.

[0033] The character string conversion control unit 28 controls the conversion of the conversion character string candidate into a character string. A conversion character string is a character string that is likely to be generated by including a handwriting recognition character string or a language character string. The character string conversion control unit 28 receives the handwriting recognition character string and the language character string candidate d11 from the handwriting recognition control unit 26, converts them into conversion character string candidates using the character string conversion dictionary unit 29, and holds them. When an acquisition request d14 is received from the handwriting input display control unit 23, the character string conversion control unit 28 transmits the conversion character string candidate d15 to the handwriting input display control unit 23.

[0034] The character string conversion dictionary unit 29 stores dictionary data for character string conversion. The character string conversion dictionary unit 29 receives the handwriting recognition character string and the language character string candidate d17 from the character string conversion control unit 28 and transmits the converted character string candidate d18 to the character string conversion control unit 28.

[0035] The predictive conversion control unit 30 receives the handwriting-recognized character string and the language string candidate d10 from the handwriting recognition control unit 26, and receives the converted character string candidate d16 from the character string conversion control unit 28. The predictive conversion control unit 30 converts the received handwriting-recognized character string and the language string candidate d10 and the converted character string candidate d16 into a predicted character string candidate using the predictive conversion dictionary unit 31. A predicted character string candidate is a character string that is likely to be generated by including a handwriting-recognized character string, a language string, or a converted character string. When the predictive conversion control unit 30 receives an acquisition request d19 from the handwriting input display control unit 23, it transmits the predicted character string candidate d20 to the handwriting input display control unit 23.

[0036] The predictive conversion dictionary unit 31 is dictionary data for predictive conversion. The predictive conversion dictionary unit 31 receives handwritten recognition character strings, language character string candidates, and converted character string candidates d21 from the predictive conversion control unit 30, and transmits predicted character string candidates d22 to the predictive conversion control unit 30.

[0037] The operation command receiving unit 32 receives the handwriting recognition character string and the language character string candidate d30 from the handwriting recognition control unit 26. The operation command receiving unit 32 receives the converted character string candidate d28 from the character string conversion control unit 28. The operation command receiving unit 32 receives the predicted character string candidate d29 from the predictive conversion control unit 30.

[0038] The operation command receiving unit 32 transmits an operation command conversion request d26 for each of the handwriting recognition character string and language character string candidate d30, the converted character string candidate d28, and the predicted character string candidate d29 to the operation command definition unit 33, and receives the operation command candidate d27 from the operation command definition unit 33. The operation command receiving unit 32 holds the operation command candidate d27. When the operation command conversion request d26 partially matches the operation command definition, the operation command definition unit 33 transmits the operation command candidate d27 to the operation command receiving unit 32.

[0039] In addition, the operation command receiving unit 32 receives pen operation data d24-1 from the handwriting input display control unit 23, sends a request d23 to the handwriting input storage unit 25 to obtain position information of a confirmed object that was previously input and confirmed, and stores the confirmed object specified by the pen operation data as a selected object (including position information).

[0040] Furthermore, when the operation command receiving unit 32 receives the acquisition request d24-2 from the handwriting input display control unit 23, it transmits the held operation command candidates and the identified selection object d25 to the handwriting input display control unit 23. The handwriting input display control unit 23 displays the received operation command candidates and the identified selection object d25 on the display unit 22. Note that the operation command receiving unit 32 may function as an operation command recognition control unit having a function of recognizing an operation command.

[0041] Details of the operation of each function in the functional block diagram of Figure 2 are described in, for example, JP 2020-064625 A.

[0042] <Processing> Fig. 3 is a flowchart of an example of processing of the display device 2 according to this embodiment. Figs. 4 to 6 are a display example (part 1) of the display device 2 according to this embodiment. In step S10, the handwriting input unit 21 accepts handwriting input (handwritten writing) from the user. Here, a case will be described as an example in which the handwriting input unit 21 of the display device 2 accepts handwritten input of "Well" from the user.

[0043] In step S12, the handwriting recognition control unit 26 recognizes the handwritten input received from the user and converts it into text data. For example, the handwriting recognition control unit 26 converts the handwritten input "Well" received from the user into text data "Well."

[0044] In step S14, the handwriting recognition control unit 26 adds data indicating a space to the end of the text data converted in step S12. For example, the handwriting recognition control unit 26 adds a space to the end (end of the word) of the converted text data "Well". Specifically, the handwriting recognition control unit 26 adds data indicating a space to the text data of the fourth character "l" of "Well" as the end of the word, following the text data of the fourth character "l". In other words, the handwriting recognition control unit 26 adds data indicating a space as the fifth character following the text data of "Well". In this way, when converting a handwritten input into text data, the display device 2 according to this embodiment adds a space after the converted text data. The text data "Well" to which the space has been added is displayed on the display 220 by the handwriting input display control unit 23.

[0045] In step S16, the handwriting input display control unit 23 determines whether or not selection of text data displayed on the display 220 has been accepted from the user. The user selects text data displayed on the display 220 by, for example, an operation of encircling the text data with handwriting input, or an operation of drawing a line with handwriting input so as to overlap the text data.

[0046] If it is determined that the selection of the text data displayed on the display 220 has been accepted, the handwriting input display control unit 23 proceeds to processing in step S18, and causes the display 220 to display candidate operation commands for the text data accepted in the processing in step S16.

[0047] FIG. 4 shows an image of a screen 1000 that displays text data 1002 with a space added to the end of the text data, and operation command candidates (operation command candidates) 1004 for the text data 1002 selected by the user.

[0048] The text data 1002 in Fig. 4 is "Well ," with a space added to the end of "Well." Candidate operation commands 1004 for the text data 1002 in Fig. 4 include an operation command "Remove Space" for deleting the space added to the end of the text data 1002, an operation command "Convert Space to ," for converting the space added to the end of the text data 1002 to a comma symbol ",", and an operation command "Convert Space to ." for converting the space added to the end of the text data 1002 to a period symbol ".".

[0049] The comma symbol and the period symbol are examples of punctuation marks. Punctuation marks are descriptive marks such as punctuation marks, question marks, parentheses, etc., which are written alone or in pairs at the beginning, middle, or end of a sentence to separate meanings or to improve readability. The operation command candidates 1004 are one or more operation commands for performing processing on the text data 1002 selected by the user. When handwriting inputting English, examples of punctuation marks include commas and periods. When handwriting inputting Japanese, examples of punctuation marks include periods and commas.

[0050] In step S20, the operation command receiving unit 32 judges whether or not a selection of an operation command has been received from the user. For example, Fig. 5 shows an example in which the user selects the operation command "Convert Space to," 1004-1 for converting a space added to the end of the text data 1002 into a comma symbol ",".

[0051] If it is determined that the selection of the operation command has been accepted, the process proceeds to step S22, where the operation command accepting unit 32 executes the operation command for the text data for which the selection has been accepted. The handwriting input display control unit 23 causes the display unit 22 to display the text data for which the operation command has been processed.

[0052] For example, FIG. 6 shows text data 1006 in which the space at the end of the word “Well,” which is text data 1002 in which a space has been added to the end when the operation command “Convert Space to,” 1004-1 is selected on screen 1000 in FIG. 5, has been converted into a comma symbol “,”.

[0053] In the examples shown in Figures 4 to 6, in the display device 2 which recognizes text data from the user's handwritten input, even after the handwritten input up to the position just before the handwritten input of sentence punctuation marks such as punctuation marks, question marks, or parentheses is recognized and converted into text data, the user can accurately add sentence punctuation marks (e.g., commas) to the end of the text data with a simple operation.

[0054] 7 to 9 show a display example (part 2) of one example of the display device 2 according to this embodiment. The display examples of FIGS. 7 to 9 will be described with reference to the flowchart of FIG.

[0055] 3, handwriting input unit 21 accepts handwritten input from a user. Here, a case will be described as an example in which handwriting input unit 21 of display device 2 accepts handwritten input of "Well, I guess so" from the user.

[0056] In step S12, the handwriting recognition control unit 26 recognizes the handwritten input received from the user and converts it into text data. For example, the handwriting recognition control unit 26 converts the handwritten input "Well, I guess so" received from the user into text data "Well, I guess so."

[0057] In step S14, the handwriting recognition control unit 26 adds data indicating a space to the end of the text data converted in step S12. For example, the handwriting recognition control unit 26 adds a space to the end (end of a word) of the converted text data "Well, I guess so". In this way, when converting handwritten input into text data, the display device 2 according to this embodiment adds a space after the converted text data. The text data "Well, I guess so" with the space added is displayed on the display 220 by the handwriting input display control unit 23.

[0058] In step S16, the handwriting input display control unit 23 determines whether or not a selection of text data displayed on the display 220 has been accepted from the user. The user selects the text data 1102 displayed on the display 220 by, for example, drawing a line 1106 by handwriting input so as to span the entire text data 1102, as shown on a screen 1100 in FIG.

[0059] If it is determined that the selection of the text data displayed on the display 220 has been accepted, the handwriting input display control unit 23 proceeds to processing in step S18, and causes the display 220 to display candidate operation commands for the text data accepted in the processing in step S16.

[0060] FIG. 7 shows an image of a screen 1100 that displays text data 1102 with a space added to the end of the text, and operation command candidates 1104 for the text data 1102 selected by the user.

[0061] 7 is "Well, I guess so" with a space added to the end of the text data 1102. Operation command candidates 1104 for the text data 1102 in FIG. 7 include the same commands as those in FIG.

[0062] In step S20, the operation command receiving unit 32 judges whether or not a selection of an operation command has been received from the user. For example, Fig. 8 shows an example in which the user selects the operation command "Convert Space to." 1104-1 for converting a space added to the end of the text data 1102 into a period symbol ".".

[0063] If it is determined that the selection of the operation command has been accepted, the process proceeds to step S22, where the operation command accepting unit 32 executes the operation command for the text data for which the selection has been accepted. The handwriting input display control unit 23 causes the display 220 to display the text data for which the operation command has been processed.

[0064] For example, Figure 9 shows text data 1108 in which the space at the end of the text data 1102, "Well, I guess so," which has been added when the operation command "Convert Space to." 1104-1 is selected on the screen 1100 of Figure 8, has been converted to a period symbol ".".

[0065] In the examples shown in Figures 7 to 9, in the display device 2 which recognizes text data from the user's handwritten input, even after the handwritten input up to the position just before the handwritten input of sentence punctuation marks such as punctuation marks, question marks, or parentheses is recognized and converted into text data, the user can accurately add sentence punctuation marks (e.g., periods) to the end of the text data with a simple operation.

[0066] 10 to 12 are display examples (part 3) of one example of the display device 2 according to the present embodiment. The display examples of FIGS. 10 to 12 will be described with reference to the flowchart of FIG.

[0067] 3, the handwriting input unit 21 accepts handwritten input from the user. Here, a case will be described as an example in which the handwriting input unit 21 of the display device 2 accepts handwritten input of "Today is " followed by "Friday" from the user.

[0068] In step S12, the handwriting recognition control unit 26 recognizes the handwritten input received from the user and converts it into text data. For example, the handwriting recognition control unit 26 converts the handwritten input "Friday" received from the user into text data "Friday."

[0069] In step S14, the handwriting recognition control unit 26 adds data indicating a space to the end of the text data converted in step S12. For example, the handwriting recognition control unit 26 adds a space to the end of "Friday" in the converted text data 1202. In this way, when converting handwritten input into text data, the display device 2 according to this embodiment adds a space after the converted text data. The text data "Friday" with the space added is displayed on the display 220 by the handwriting input display control unit 23.

[0070] In step S16, the handwriting input display control unit 23 determines whether or not a selection of text data displayed on the display 220 has been accepted from the user. The user selects the text data 1203 displayed on the display 220 by, for example, drawing a line 1206 by handwriting input across the entire text data 1203, as shown on a screen 1200 in Fig. 11 .

[0071] If it is determined that a selection of the text data displayed on the display 220 has been accepted, the handwriting input display control unit 23 proceeds to processing of step S18 and causes the display unit 22 to display candidate operation commands for the text data accepted in the processing of step S16.

[0072] FIG. 11 shows an image of a screen 1200 that displays text data 1203 with a space added to the end of the text, and operation command candidates 1204 for the text data 1203 selected by the user.

[0073] 11 is "Today is Friday" with a space added to the end of the text data 1203. Operation command candidates 1204 for the text data 1203 in FIG. 11 include the same commands as those in FIG.

[0074] In step S20, the operation command receiving unit 32 judges whether or not a selection of an operation command has been received from the user. For example, Fig. 11 shows an example in which the user selects the operation command "Remove Space" 1204-1 for deleting a space added to the end of the text data 1203.

[0075] If it is determined that the selection of the operation command has been accepted, the process proceeds to step S22, where the operation command accepting unit 32 executes the operation command for the text data for which the selection has been accepted. The handwriting input display control unit 23 causes the display 220 to display the text data for which the operation command has been processed.

[0076] For example, FIG. 12 shows text data 1208 in which the space at the end of “Today is Friday,” which is text data 1203 in which a space was added to the end, has been deleted when the operation command “Remove Space” 1204-1 is selected on screen 1200 in FIG. 11.

[0077] In the examples shown in Figures 10 to 12, in the display device 2 which recognizes text data from the user's handwritten input, even after the handwritten input up to the position just before the handwritten input of a sentence punctuation mark, question mark, or parentheses, etc., is recognized and converted into text data, the user can accurately delete spaces added to the end of the text data with a simple operation.

[0078] 13 and 14 are a display example (part 4) of an example of the display device 2 according to this embodiment. Fig. 13 and Fig. 14 are example screens to which a transition is made upon receiving selection of the text data 1208 from which spaces at the ends of words in Fig. 12 have been deleted.

[0079] For example, as shown on a screen 1300 in FIG. 13, the user selects text data 1302 displayed on the display 220 by drawing a line 1306 by handwriting input across the entire text data 1302 .

[0080] When it is determined that the selection of the text data displayed on the display 220 has been accepted, the handwriting input display control unit 23 causes the display 220 to display candidates for operation commands for the text data whose selection has been accepted.

[0081] Fig. 13 shows an image of a screen 1300 displaying text data 1302 with spaces deleted from the end of the text and operation command candidates 1304 for the text data 1302 selected by the user. The text data 1302 in Fig. 13 is "Today is Friday" with the space deleted from the end of the text.

[0082] Candidate operation commands 1304 for the text data 1302 in FIG. 13 include an operation command “Append Space” for adding a space to the end of the text data 1302, an operation command “Append ,” for adding a comma symbol “,” to the end of the text data 1302, and an operation command “Append .” for adding a period symbol “.” to the end of the text data 1302.

[0083] For example, Fig. 13 shows an example in which the user selects the operation command "Append." 1304-1 for adding a period to the end of text data 1302. For example, Fig. 14 shows text data 1308 in which the operation command "Append." 1304-1 is selected on the screen 1300 of Fig. 13 and a period symbol "." is added to the end of the text data.

[0084] In the examples shown in FIGS. 13 and 14, even after a space is deleted from the end of text data, the user can accurately add a sentence punctuation mark (for example, a period) to the end of text data with a simple operation.

[0085] 15 and 16 show a display example (part 5) of one example of the display device 2 according to this embodiment. For example, as shown in a screen 1400 of Fig. 15, the user selects text data 1402 displayed on the display 220 by, for example, drawing a line by handwriting input so as to straddle "is" in the text data 1402.

[0086] When it is determined that a selection of text data displayed on the display 220 has been accepted, the handwriting input display control unit 23 causes operation command candidates for the text data whose selection has been accepted to be displayed on the display 220. Fig. 15 shows an image of a screen 1400 that displays "is" in text data 1402 and operation command candidates 1404 for the text data 1402 whose selection has been accepted from the user.

[0087] Candidates 1404 for operation commands for text data 1402 in Fig. 15 include commands similar to those in Fig. 13. For example, Fig. 15 shows an example in which a user selects a command "Append Space" 1404-1 for adding a space to the end of text data 1402. Fig. 16 displays "is" in text data 1402 with a space added to the end of "is" in text data 1402.

[0088] 16, since the word "is" in the text data 1402 has been converted to "is" in the text data 1402 with a space added to the end, the operation command candidate 1406 has changed from the operation command candidate 1404 in FIG.

[0089] The operation command candidates 1406 in Figure 16 include an operation command "Remove Space" that deletes a space added to the end of the text data 1402, an operation command "Convert Space to," that converts the space added to the end of the text data 1402 to a comma symbol ",", and an operation command "Convert Space to." that converts the space added to the end of the text data 1402 to a period symbol ".".

[0090] In the examples shown in FIGS. 15 and 16, even after spaces have been deleted from the ends of words in the text data, the user can accurately add spaces to the ends of words in the text data with a simple operation.

[0091] <Display order of operation command candidates> The display order of the operation command candidates 1004, 1104, 1204, 1304, 1404, and 1406 may be fixed, or may be changed based on the operation command selection history by the user, or may be changed by inference using a machine learning model.

[0092] For example, the display device 2 may suggest to the user to change the display order of operation command candidates by collecting and analyzing the selection history of operation commands by the user. For example, the display device 2 may suggest to display operation commands operated by the user more frequently at the top and operation commands operated by the user less frequently at the bottom.

[0093] For example, if the user selects more comma conversion operation commands that convert spaces at the ends of text data into comma symbols than period conversion operation commands that convert spaces at the ends of text data into period symbols, the display device 2 may suggest displaying the comma conversion operation commands higher than the period conversion operation commands.

[0094] 17 is a flowchart of an example of a process in which the display device 2 according to the present embodiment changes the display order of operation command candidates. In step S30, the handwriting input display control unit 23 of the display device 2 analyzes the selection history of operation commands by the user.

[0095] In step S32, the handwriting input display control unit 23 determines whether or not a condition for changing the display order is met based on the analysis result of step S30. For example, the handwriting input display control unit 23 determines whether or not a condition for changing the display order is met based on whether or not the number of times the user has operated the operation command candidate is equal to or greater than a predetermined number of times.

[0096] If the condition for changing the display order is met, the handwriting input display control unit 23 displays, for example, a screen 2000 as shown in Fig. 18 on the display 220 in step S34, and proposes it to the user. Fig. 18 is an image diagram of an example of the screen 2000 for proposing a change in the display order of operation command candidates.

[0097] The user refers to the number of times the user operates each operation command candidate displayed on the screen 2000 of Fig. 18, and determines whether or not to change the display order of the operation command candidates. The user issues an instruction to change the display order of the operation command candidates according to the result of the determination.

[0098] If there is an instruction from the user to change the display order of the operation command candidates, the handwriting input display control unit 23 changes the display order of the operation command candidates. Note that if the condition for changing the display order is not met in step S32, or if there is no instruction from the user to change the display order of the operation command candidates in step S34, the handwriting input display control unit 23 does not change the display order of the operation command candidates.

[0099] In addition, when the handwriting input display control unit 23 changes the display order of operation command candidates by inference using a machine learning model, for example, the handwriting input display control unit 23 may use a machine learning model to determine the correspondence between the semantic position of the text data to which spaces have been added (such as in the middle or at the end of a sentence) and the operation command selected by the user at that position, infer operation command candidates that are easy for the user to select from the semantic position of the text data to which spaces have been added, and change the display order of the operation command candidates based on the results of the inference.

[0100] The display device 2 according to this embodiment can be used in the following situations, for example.

[0101] A user may perform handwritten input on the display device 2 while looking at a document at hand. For example, a rescue worker, who is an example of a user, may take notes on a piece of paper at hand about the medical history before the patient is transported by ambulance while in an ambulance, and after arriving at the hospital, may perform handwritten input on the display device 2 while looking at the paper.

[0102] The user handwrites the contents of the paper into the display device 2 while shifting his / her gaze alternately between the paper and the display 220 of the display device 2. In this case, the user may handwrite the text written on the paper up to the end of the sentence just before the punctuation mark, then return his / her gaze to the paper to check the contents of the note on the paper (the punctuation mark), and then handwrite the additional punctuation mark.

[0103] In this case, the display device 2 displays text data in which the content initially handwritten (up to the end of the sentence written on paper before the punctuation mark) is recognized. In this situation, the user handwrites and inputs the punctuation mark as a single character.

[0104] If a single character handwritten is a punctuation mark, the handwritten punctuation mark may not be accurately recognized by display device 2, and may be recognized as a shape such as a circle rather than a punctuation mark, and may not be converted into the text data that the user expects.

[0105] For example, if an ambulance worker, who is an example of a user, handwrites "2 people transported" and then handwrites a period, the display device 2 may recognize this as "2 people transported 〇" rather than "2 people transported." and convert it into text data.

[0106] In the display device 2 of this embodiment, the user can select an operation command to convert spaces contained in the text data converted from handwritten input into punctuation marks, etc., thereby making it possible to accurately and easily add desired punctuation marks, etc. to the end of recognized text data.

[0107] In the above embodiment, an example of handwritten input of English has been described, but the present invention is not limited to a specific language, and can also be applied to handwritten input of Japanese, for example. When handwriting input of English, it is essential to add spaces between words. For example, when handwriting input of "I", "guess", and "so", it is essential to add spaces between each word. Therefore, when handwriting English, when recognizing text data from handwritten input, it is particularly required to add data indicating a space to the end of the text data.

[0108] <Other embodiments> Fig. 19 is a system configuration diagram of an example of an information processing system 100 including the display device 2 according to this embodiment. The information processing system 100 in Fig. 19 is a system form in which a server device 10 on a network N performs processing such as handwriting recognition and returns a result of the processing from the server device 10 to the display device 2.

[0109] The information processing system 100 includes a display device 2 and a server device 10 that can communicate with each other via a network N. The display device 2 is disposed in a facility such as a company, and is connected to a LAN or Wi-Fi installed in the facility. The server device 10 is disposed in, for example, a data center. The display device 2 is connected to the Internet i via a firewall 8. The server device 10 is connected to the Internet i via a high-speed LAN or the like in the data center. The display device 2 may be connected to the Internet i using wireless communication such as a telephone line network. The wireless communication may be 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or the like.

[0110] The server device 10 has one or more computers, which act as servers to provide services to the display device 2. A server is a computer or software that provides information or processing results in response to a request from a client.

[0111] The system on the server side may be called a cloud system. A cloud system is a system that uses cloud computing. Cloud computing refers to a form of use in which resources on a network are used without being aware of specific hardware resources. A cloud system is not necessarily placed on the Internet. Although FIG. 19 shows an example in which the server device 10 is placed on the Internet, it may be placed on a local network such as on-premise.

[0112] Server apparatus 10 may include multiple computing devices, such as a server cluster, configured to communicate with each other over any type of communications link, including a network, shared memory, etc., to perform the processes disclosed herein.

[0113] The configuration of the display device 2 may be similar to the functional configuration shown in Fig. 2, but only needs to have a touch panel, a display, and a communication function. The display device 2 may include multiple computing devices configured to communicate with each other.

[0114] The display device 2 of this embodiment may be a general computer such as a PC or a tablet that executes a web browser or a dedicated application. The web browser or the dedicated application communicates with the server device 10.

[0115] When a web browser is running on the display device 2, the user inputs or selects the URL of the server device 10 to connect the display device 2 to the server device 10. The display device 2 executes a web application provided by the server device 10 in the web browser. A web application refers to software or a mechanism thereof that operates by cooperation between a program written in a programming language (e.g., JavaScript (registered trademark)) that runs on the web browser and a program on the web server side.

[0116] When a dedicated application runs on the display device 2, the display device 2 connects to a URL of the server device 10 that has been registered in advance. The dedicated application running on the display device 2 has a program and a user interface. The display device 2 transmits and receives information required by the program to and from the server device 10 and displays it on the user interface. The communication method may be a general-purpose communication protocol such as HTTP, HTTPs, or WebSocket, or may be a dedicated communication protocol.

[0117] 20 is a diagram showing an example of a hardware configuration of the server device 10. The server device 10 is constructed by a computer 500.

[0118] The computer 500 includes a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0119] The CPU 501 controls the overall operation of the computer 500 according to a program. The ROM 502 stores programs such as an IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 according to the control of the CPU 501.

[0120] The display 506 displays various information such as a cursor, a menu, a window, characters, or an image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory. The network I / F 509 is an interface for data communication using the network 18. The data bus 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

[0121] The keyboard 511 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513 as an example of a removable recording medium. Note that the medium is not limited to a DVD-RW, and may be a DVD-R or the like. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515 such as a flash memory.

[0122] It should be noted that the hardware configuration shown in FIG. 20 is merely an example, and it is not necessary for the configuration to include all of the components shown in FIG. 20, or the configuration may include components other than those shown in FIG.

[0123] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to the examples, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0124] For example, although the present embodiment has been described as an example of an electronic whiteboard, any information processing device having a touch panel can be suitably applied. Also, a device having the same function as an electronic whiteboard is also called an electronic whiteboard, an electronic information board, an interactive board, etc.

[0125] Examples of information processing devices equipped with a touch panel include output devices such as PJs (Projectors), digital signage, HUD (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network home appliances, notebook PCs (Personal Computers), mobile phones, smartphones, tablet terminals, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, or desktop PCs.

[0126] Moreover, the functional configuration example in FIG. 2 is divided according to main functions in order to facilitate understanding of the processing by the display device 2. The present invention is not limited by the manner in which the processing units are divided or the names thereof. The processing by the display device 2 can also be divided into more processing units according to the processing contents. Moreover, the processing by the display device 2 can also be divided so that one processing unit includes more processes.

[0127] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above. [Explanation of symbols]

[0128] 2 Display device 10. Server equipment 21 Handwriting input section 22 Display section 23 Handwriting input display control unit 24 Candidate display timer control unit 25 Handwriting input storage section 26 Handwriting recognition control section 27 Handwriting Recognition Dictionary Section 28 Character string conversion control section 29 String Conversion Dictionary 30 Predictive conversion control section 31 Predictive conversion dictionary section 32 Operation command reception section 33 Operation command definition section 100 Information Processing Systems [Prior art documents] [Patent documents]

[0129] [Patent Document 1] Patent No. 6772629

Claims

1. a handwriting input unit that accepts handwritten input; a handwriting recognition control unit that recognizes the handwritten input, converts the recognized handwritten input into text data, and adds data indicating a space to a tail of the text data; a handwriting input display control unit that displays on a display unit the text data to which data indicating a space has been added at the end of a word, and one or more operation command candidates for processing the text data; an operation command receiving unit that receives an operation command selected by a user from the operation command candidates; Equipped with the handwriting input display control unit, when the operation command selected by the user is a command to convert the data indicating the space added to an end of the text data into data indicating a punctuation mark, displays, on the display unit, the text data in which the data indicating the space has been converted into the data indicating the punctuation mark. A display device comprising:

2. The handwriting input display control unit, when the operation command selected by the user is a command to delete the data indicating the space added to an end of the text data, displays, on the display unit, the text data from which the data indicating the space has been deleted.

2. The display device according to claim 1,

3. The handwriting input display control unit, when the operation command selected by the user is a command to add data indicating the punctuation mark to a suffix of the text data, displays, on the display unit, the text data to which the data indicating the punctuation mark has been added.

3. The display device according to claim 2,

4. The handwriting input display control unit, when the operation command selected by the user is a command to add data indicating a space to an end of the text data, displays, on the display unit, the text data to which the data indicating a space has been added.

3. The display device according to claim 2,

5. The handwriting input display control unit changes a display order of the operation command candidates based on a selection history of the operation commands selected by the user. The display device according to claim 1.

6. The handwriting input display control unit infers the operation command to be selected by the user from the semantic position of the ending of the word to which the data indicating the space was added, using a machine learning model of a correspondence relationship between a semantic position of the text data to which the data indicating the space was added and a selection history of the operation command selected by the user, and changes a display order of the operation command candidates based on a result of the inference. The display device according to claim 1.

7. The punctuation marks are punctuation marks, question marks, or parentheses. A display device according to any one of claims 1 to 6.

8. The handwritten input is in English. The display device according to claim 1.

9. A display method performed by a display device, comprising: a handwriting input step of accepting handwritten input; a handwriting recognition control step of recognizing the handwritten input, converting the recognized handwritten input into text data, and adding data indicating a space to a tail of the text data; a handwriting input display control step of displaying on a display unit the text data to which data indicating a space has been added at the end of a word, and one or more operation command candidates for processing the text data; an operation command receiving step of receiving an operation command selected by a user from the operation command candidates; Equipped with The handwriting input display control step includes, when the operation command selected by the user is a command for converting the data indicating the space added to an end of the text data into data indicating a punctuation mark, displaying, on the display unit, the text data in which the data indicating the space has been converted into the data indicating the punctuation mark. A display method characterized by:

10. The display device includes: A handwriting input procedure for accepting handwritten input; a handwriting recognition control step of recognizing the handwritten input, converting the recognized handwritten input into text data, and adding data indicating a space to a tail of the text data; a handwriting input display control procedure for displaying on a display unit the text data, the text data having data indicating a space added to the end of the text data, and one or more operation command candidates for processing the text data; an operation command receiving step of receiving an operation command selected by a user from the operation command candidates; Run the command, The handwriting input display control step includes, when the operation command selected by the user is a command for converting the data indicating the space added to a tail of the text data into data indicating a punctuation mark, displaying, on the display unit, the text data in which the data indicating the space has been converted into the data indicating the punctuation mark. A program characterized by.

11. An information processing system in which a server device and a display device perform data communication via a network, The server device includes: a handwriting recognition control unit that recognizes handwritten input received by the display device from a user, converts the recognized handwritten input into text data, and adds data indicating a space to a tail of the text data; a handwriting input display control unit that displays on a display unit the text data to which data indicating a space has been added at the end of a word, and one or more operation command candidates for processing the text data; an operation command receiving unit that receives an operation command selected by a user from the operation command candidates; Equipped with The display device includes: a handwriting input unit that accepts the handwriting input; a display unit that displays the text data to which data indicating a space has been added at the end of a word, and one or more operation command candidates for processing the text data; Equipped with the handwriting input display control unit, when the operation command selected by the user is a command to convert the data indicating the space added to an end of the text data into data indicating a punctuation mark, displays, on the display unit, the text data in which the data indicating the space has been converted into the data indicating the punctuation mark. An information processing system comprising:

Images