Information processing apparatus, information processing system, information processing method, and program

Abstract

To store data on an option even when a user selects the corresponding option from among a plurality of options by handwriting.SOLUTION: An information processing apparatus is configured to convert an object input by handwriting on a display screen into text data, the information processing apparatus including; a receiving unit which receives an input handwritten on the display screen; and an output unit which outputs, when the handwriting is a predetermined selection operation for selecting an object displayed in a predetermined region on the display screen, text data corresponding to the object selected by the predetermined selection operation, out of a plurality of objects displayed in the predetermined region, to a predetermined output destination.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] There is an information processing device that converts objects such as characters handwritten on a display screen into text data. For example, a display device is known that converts handwritten data into a character string based on an attribute associated with a table and displays the character string (for example, see Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0003] For example, when checking in at a hospital, a medical questionnaire may be displayed on an information processing device, and the patient may select a corresponding option from a number of options by handwriting input.

[0004] When a patient selects a corresponding option from among multiple options, the patient selects the corresponding option by, for example, handwriting a circle around the option. However, the information processing device recognizes the patient's handwriting as a circle shape, and is unable to save the data of the option selected by the patient.

[0005] Note that this problem is not limited to medical questionnaires, but commonly exists when a user hand-selects options on various forms.

[0006] One embodiment of the present invention has been made in consideration of the above-mentioned problems, and enables a user to save data on options even when the user hand-selects a relevant option from among multiple options. [Means for solving the problem]

[0007] In order to solve the above problem, an information processing device according to one embodiment of the present invention is an information processing device that converts an object handwritten on a display screen into text data, and has an acceptance unit that accepts the handwritten input on the display screen, and an output unit that, when the handwriting is a predetermined selection operation for selecting an object displayed in a predetermined area of ​​the display screen, outputs text data corresponding to an object selected by the predetermined selection operation from among a plurality of objects displayed in the predetermined area to a predetermined output destination. Effect of the Invention

[0008] According to one embodiment of the present invention, even when a user selects a corresponding option from among a plurality of options by handwriting, the option data can be saved. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of an arrangement of information processing devices according to an embodiment. [Diagram 3] FIG. 1 is a diagram for explaining an overview of a process according to an embodiment; [Figure 4] FIG. 2 is a diagram for explaining an overview of the process according to an embodiment; [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer according to an embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device according to an embodiment. [Figure 8] 4 is a diagram illustrating an example of a functional configuration of an overall control unit according to the first embodiment. FIG. [Figure 9] 5 is a flowchart showing an example of processing of the information processing device according to the first embodiment. [Figure 10]FIG. 11 is a diagram illustrating an example of a predetermined selection operation according to an embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of attribute information according to an embodiment. [Figure 12] FIG. 13 is a diagram showing an example of an arrangement of items according to an embodiment. [Figure 13] FIG. 11 is a diagram for explaining a method for determining a selected item according to an embodiment. [Figure 14] FIG. 11 is a diagram illustrating an example of a functional configuration of an overall control unit according to a second embodiment. [Figure 15] FIG. 11 is a diagram showing an example of a notification screen according to the second embodiment. [Figure 16] 13 is a flowchart showing an example of a setting condition change process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A display device and a control method performed by the display device will be described below as an example of an embodiment of the present invention with reference to the drawings.

[0011] <System configuration> Fig. 1 is a diagram showing an example of a system configuration of an information processing system according to an embodiment. As shown in Fig. 1, the information processing system 1 includes an information processing device 100 that converts an object handwritten on a display screen into text data, and an external device 10 that can communicate with the information processing device 100 via a communication network N. An object represents a target. Examples of objects include handwritten data, characters, figures, tables, images, and the like.

[0012] The information processing device 100 has a display 101 that displays a display screen. A user 2 can handwrite objects such as characters or figures on the display screen displayed on the display 101 using a pen 102.

[0013] Furthermore, the information processing device 100 can convert an object input on the display screen into text data and output the converted text data to a predetermined output destination. For example, the information processing device 100 can transmit (output) the converted text data to the external device 10 via the communication network N. Furthermore, the information processing device 100 can convert handwritten characters into text using the converted text data and display (output) it. Furthermore, the information processing device 100 can store (output) the converted text data in a storage unit or the like provided in the information processing device 100.

[0014] As an example of a usage scenario, the information processing system 1 converts handwritten patient information on the information processing device 100 into text and displays it, and also converts the patient information into text data and transmits it to an electronic medical record management system (an example of an external device 10) and stores it.

[0015] The information processing device 100 is, for example, an electronic device such as electronic paper for handwriting or an IWB (Interactive White Board: an electronic whiteboard capable of mutual communication). However, the information processing device 100 is not limited to this, and may be a general-purpose information terminal such as a tablet terminal that executes an application program (hereinafter, referred to as an app) corresponding to the information processing system 1. Here, the following description will be given assuming that the information processing device 100 is an electronic device such as electronic paper for handwriting or an IWB.

[0016] Fig. 2 is a diagram showing an example of the arrangement of an information processing device according to an embodiment. For example, as shown in Fig. 2(A), the information processing device 100 can be hung on a wall and used as a horizontally long electronic whiteboard. Also, for example, as shown in Fig. 2(B), the information processing device 100 can be hung on a wall and used as a vertically long electronic whiteboard.

[0017] 2(C) shows the information processing device 100 placed flat on a desk 201. The information processing device 100 is preferably about 1 cm thick, and does not need to adjust the height of a normal desk even when placed flat on the desk. In addition, the user 2 can easily move the information processing device 100.

[0018] The input method of the pen 102 includes, for example, an electromagnetic induction method, an active electrostatic coupling method, etc. The pen 102 may also have functions such as pen pressure detection, tilt detection, or a hover function (displaying the cursor before the pen 102 touches the screen).

[0019] (Processing Overview) 3 and 4 are diagrams for explaining an overview of processing according to an embodiment. The information processing device 100 displays, for example, a medical questionnaire 300 as shown in Fig. 3 on the display 101, converts handwritten characters by a patient or the like into text and displays the text, and can transmit the converted text to the external device 10 as text data.

[0020] For example, as shown in FIG. 4(A), when a patient inputs the handwritten characters "diabetes" 401 in response to the question "Are you currently being treated for any illness?", the information processing device 100 converts this into text 403 as shown in FIG. 4(B) and displays it.

[0021] On the other hand, for example, as shown in Fig. 4(A), suppose that in response to a medical question about "vaccination against coronavirus," one answer must be selected from multiple options using a shape 402 such as a circle or a check. In this case, the conventional technology does not take into consideration the conversion of only the selected answer "no" into text, so that, for example, as shown in Fig. 4(B), text such as "yes" that is not selected may also be displayed.

[0022] Therefore, the information processing device 100 according to the present embodiment is able to convert an answer selected from a plurality of options into text data. As a result, the information processing device 100 is able to convert only the selected answer "No" into text 405 and display it in the answer field 407, for example, as shown in FIG. 4(C).

[0023] <Hardware configuration of the device> (Hardware configuration of information processing device) Fig. 5 is a diagram showing an example of a hardware configuration of an information processing device according to an embodiment. As shown in Fig. 5, the information processing device 100 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, and an SSD (Solid State Drive) 504.

[0024] Of these, the CPU 501 controls the overall operation of the information processing device 100. The ROM 502 stores programs used to start up the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The SSD 504 is an example of a storage device that stores various data such as programs for the information processing device 100. The programs may be downloaded by the information processing device 100 from a program distribution server, or may be installed from a storage medium. These programs may be called application software.

[0025] The information processing device 100 also includes a display controller 513, a touch sensor controller 515, a touch sensor 516, a display 101, a power switch 527, a tilt sensor 517, a serial interface 518, a speaker 519, a microphone 521, a communication device 522, an infrared I / F (Interface) 523, a power control circuit 524, an AC adapter 525, and a battery 526.

[0026] A display controller 513 controls and manages screen display in order to output an output image to the display 101 etc. A touch sensor 516 detects that the pen 102, a user's hand, etc. (the pen or the user's hand are input means) has touched the display 101.

[0027] The touch sensor controller 515 controls the processing of the touch sensor 516. The touch sensor 516 inputs and detects coordinates. For example, in the case of an optical method, two light receiving / emitting devices installed at both ends of the upper side of the display 101 emit a plurality of infrared rays in parallel to the display 101. The light receiving / emitting devices receive the emitted infrared rays reflected by a reflecting member provided around the display 101 and return along the same optical path as the emitted light with a light receiving element. The touch sensor 516 outputs position information of the infrared rays emitted by the two light receiving / emitting devices that are blocked by an object to the touch sensor controller 515, and the touch sensor controller 515 specifies the coordinate position that is the contact position of the object. The touch sensor controller 515 has a communication unit 515a and can wirelessly communicate with the pen 102. For example, when communication is performed according to a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 102 are registered in advance in the communication unit 515a, the user can communicate with the information processing device 100 without making connection settings for communicating with the pen 102.

[0028] The power switch 527 is a switch for switching ON / OFF the power supply of the information processing device 100. The tilt sensor 517 is a sensor for detecting the tilt angle of the information processing device 100. The tilt sensor 517 is mainly used for detecting whether the information processing device 100 is being used in any of the installation states of Fig. 2(A), Fig. 2(B), or Fig. 2(C), and the information processing device 100 can automatically change the thickness of characters, etc. according to the detected installation state.

[0029] The serial interface 518 is a communication interface with the outside, such as a USB (Universal Serial Bus). The serial interface 518 is used, for example, for inputting information from the outside. The speaker 519 is used for outputting sound, and the microphone 521 is used for inputting sound. The communication device 522 is a communication interface for connecting the information processing device 100 to a communication network N, such as the Internet or a LAN (Local Area Network).

[0030] The infrared I / F 523 detects the information processing device 100 arranged adjacently. By utilizing the linearity of infrared rays, only the adjacent information processing device 100 can be detected. In this case, it is preferable to provide one infrared I / F 523 on each side, and it is possible to detect in which direction of the information processing device 100 another information processing device 100 is arranged. This expands the screen, and it is possible to display handwritten information previously handwritten on the adjacent information processing device 100 (handwritten information on another page, with the area of ​​one display 101 being one page), etc.

[0031] A power supply control circuit 524 supplies power from an AC adapter 525, which is the power source for the information processing device 100, or a battery 526 to the CPU 501, etc. The AC adapter 525 converts AC supplied from a commercial power source into DC. If the display 101 is a so-called electronic paper, it consumes little or no power to maintain the display of an image, so it can also be driven by the battery 526. This makes it possible to use the information processing device 100 for purposes such as digital signage even in places where it is difficult to connect a power source, such as outdoors.

[0032] The information processing device 100 further includes a bus line 510. The bus line 510 includes an address bus, a data bus, various control signals, and the like for electrically connecting the components such as the CPU 501 shown in FIG.

[0033] The touch sensor 516 is not limited to an optical type, and may be a capacitive touch panel that identifies the contact position by detecting a change in capacitance, or a resistive touch panel that identifies the contact position by a voltage change between two opposing resistive films. The touch sensor 516 may use various detection means, such as an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contact object touching the display unit. Furthermore, the touch sensor 516 may be of a type that does not require an electronic pen to detect the presence or absence of a touch with the tip of the pen. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 102 does not need to be a long, thin pen type.

[0034] (Computer hardware configuration) The external device 10 has, for example, the hardware configuration of a computer 600 as shown in Fig. 6. Alternatively, the external device 10 may be configured by a plurality of computers 600.

[0035] Fig. 6 is a diagram showing an example of a hardware configuration of a computer according to an embodiment. For example, as shown in Fig. 6, a computer 600 includes a CPU 601, a ROM 602, a RAM 603, a HD (Hard Disk) 604, a HDD (Hard Disk Drive) controller 605, a display 606, an external device connection I / F (Interface) 607, a network I / F 608, a keyboard 609, a pointing device 610, a DVD-RW (Digital Versatile Disk Rewritable) drive 612, a media I / F 614, and a bus line 615.

[0036] Of these, the CPU 601 controls the overall operation of the computer 600. The ROM 602 stores, for example, a program used to start up the computer 600, such as an IPL. The RAM 603 is used, for example, as a work area for the CPU 601. The HD 604 stores, for example, an operating system (OS), applications, device drivers, and other programs, and various data. The HDD controller 605 controls the reading and writing of various data from and to the HD 604 under the control of the CPU 601. The HD 604 and the HDD controller 605 are examples of storage devices included in the computer 600.

[0037] The display 606 displays various types of information such as a cursor, a menu, a window, characters, or an image. The display 606 may be provided outside the computer 600. The external device connection I / F 607 is an interface for connecting various external devices to the computer 600. The network I / F 608 is an interface for connecting the computer 600 to a communication network N and communicating with other devices.

[0038] The keyboard 609 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 610 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The keyboard 609 and the pointing device 610 may be provided outside the computer 600.

[0039] The DVD-RW drive 612 controls reading and writing of various data from and to a DVD-RW 611, which is an example of a removable recording medium. The DVD-RW 611 is not limited to a DVD-RW, and may be another recording medium. The media I / F 614 controls reading and writing (storing) of data from and to a medium 613 such as a flash memory. The bus line 615 includes an address bus, a data bus, various control signals, and the like for electrically connecting the above components.

[0040] <Functional configuration> (Functional configuration of information processing device) 7 is a diagram showing an example of a functional configuration of an information processing device according to an embodiment. The information processing device 100 includes, for example, a reception unit 701, a display unit 702, an overall control unit 703, a candidate display timer control unit 704, a storage unit 705, a handwriting recognition control unit 706, a handwriting recognition dictionary unit 707, a character string conversion control unit 708, a character string conversion dictionary unit 709, a predictive conversion control unit 711, and a predictive conversion dictionary unit 712. Each functional configuration of the information processing device 100 is realized by operating any of the components shown in FIG. 5 according to an instruction from the CPU 501 in accordance with a program loaded from the SSD 504 onto the RAM 503.

[0041] Reception unit 701 is realized, for example, by a program executed by CPU 501, touch sensor 516, etc., and executes a reception process for receiving handwritten input on a display screen displayed on display 101. For example, reception unit 701 converts a user's pen input into pen operation data (pen up, pen down, or pen coordinate data) with a pen ID, and transmits it to overall control unit 703 (A). Reception unit 701 periodically transmits the pen coordinate data to overall control unit 703 as discrete values, and overall control unit 703 performs an interpolation calculation of coordinates between the discrete values.

[0042] The display unit 702 is realized, for example, by a program executed by the CPU 501 and the display 101, and executes a display process for displaying handwritten stroke data, recognized character strings, operation menus, etc. For example, the display unit 702 converts drawing data written in a video memory by the overall control unit 703 into data according to the characteristics of the display 101 and transmits it to the display 101.

[0043] The overall control unit 703 is realized, for example, by a program executed by the CPU 501, and controls the overall processing of the information processing device 100 including processing related to handwritten input and display. The functions of the overall control unit 703 will be described later with reference to several exemplary embodiments.

[0044] The candidate display timer control unit 704 is a display control timer for selectable candidates. The candidate display timer control unit 704 is realized, for example, by a program executed by the CPU 501, and generates the timing for starting and stopping the timer to start displaying the selectable candidates and the timing for erasing the display. The selectable candidates are handwriting recognition character string candidates, language character string candidates, conversion character string candidates, character string / predictive conversion candidates, operation command candidates, and the like, which are displayed selectably in the operation guide. The candidate display timer control unit 704 receives a timer start request (or a timer stop request) from the overall control unit 703, and transmits a timeout event to the overall control unit 703.

[0045] The storage unit 705 stores, for example, the programs executed by the CPU 501 and the SSD 504, etc. and has a storage function for saving user data (handwritten data / character string after recognition). For example, the storage unit 705 receives user data from the overall control unit 703 and stores the received user data in the SSD 504 or the like. The storage unit 705 also stores various information, data, programs, etc., such as attribute information or setting information, which will be described later.

[0046] The handwriting recognition control unit 706 is, for example, a recognition engine that is realized by a program executed by the CPU 501 and performs online handwriting recognition (handwritten data recognition processing). Unlike a typical OCR (Optical Character Reader), the handwriting recognition control unit 706 recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%, $, &, etc.), figures (lines, circles, triangles, etc.), etc. in parallel with the user's pen operation. Various algorithms have been devised for the recognition method, but details will be omitted since known techniques can be used in this embodiment.

[0047] Handwriting recognition control unit 706 is realized by, for example, a program executed by CPU 501, receives pen operation data from overall control unit 703, performs handwriting recognition, and stores handwriting recognition character string candidates. Handwriting recognition control unit 706 also stores language character string candidates converted from handwriting recognition character string candidates using handwriting recognition dictionary unit 707. When a separate acquisition request is received from overall control unit 703, handwriting recognition control unit 706 transmits the stored handwriting recognition character string candidates and language character string candidates to overall control unit 703.

[0048] The handwriting recognition dictionary unit 707 stores dictionary data for language conversion in handwriting recognition. The handwriting recognition dictionary unit 707 receives handwriting recognition character string candidates from the handwriting recognition control unit 706, converts them into language string candidates that are linguistically likely, and transmits them to the handwriting recognition control unit 706. For example, in the case of Japanese, hiragana is converted into kanji or katakana.

[0049] The character string conversion control unit 708 is realized by, for example, a program executed by the CPU 501, and controls the conversion of conversion character string candidates into character strings. A conversion character string is a character string that is likely to be generated by including a handwriting recognition character string candidate or a language string candidate. The character string conversion control unit 708 receives the handwriting recognition character string candidate and the language string candidate from the handwriting recognition control unit 706, converts them into conversion character string candidates using the character string conversion dictionary unit 709, and stores them. If an acquisition request is separately received from the overall control unit 703, the character string conversion control unit 708 transmits the stored conversion character string candidates to the overall control unit 703.

[0050] The character string conversion dictionary unit 709 stores dictionary data for character string conversion. The character string conversion dictionary unit 709 receives handwritten recognition character string candidates and language character string candidates from the character string conversion control unit 708, and transmits converted character string candidates to the character string conversion control unit 708.

[0051] The predictive conversion control unit 711 is realized by, for example, a program executed by the CPU 501, and receives a handwritten recognition string and a language string candidate from the handwriting recognition control unit 706, and receives a conversion string candidate from the string conversion control unit 708. The predictive conversion control unit 711 converts each of them into a predictive string candidate using the predictive conversion dictionary unit 712. A predictive string candidate is a string that is likely to be generated by including a handwritten recognition string candidate, a language string candidate, or a conversion string candidate. If an acquisition request is separately received from the overall control unit 703, the predictive conversion control unit 711 transmits the predictive string candidate to the overall control unit 703.

[0052] The predictive conversion dictionary unit 712 stores dictionary data for predictive conversion. The predictive conversion dictionary unit 712 receives handwritten recognition character string candidates, language character string candidates, and conversion character string candidates from the predictive conversion control unit 711, and transmits predicted character string candidates to the predictive conversion control unit 711.

[0053] [First embodiment] 8 is a diagram showing an example of the functional configuration of the overall control unit according to the first embodiment. The overall control unit 703 according to the first embodiment includes, for example, an input control unit 801, a display control unit 802, a determination unit 803, a conversion unit 804, an output unit 805, and a conversion control unit 806.

[0054] The input control unit 801 executes an input control process for processing handwritten data accepted by the accepting unit 701. The display control unit 802 outputs and displays handwritten characters or objects such as figures to the display unit 702 based on the handwritten data processed by the input control unit 801. The display control unit 802 also outputs and displays converted text, etc. to the display unit 702 based on converted text data output by the output unit 805. The display control unit 802 also controls the display of an operation menu, conversion candidates, etc.

[0055] The judgment unit 803 judges whether the handwriting accepted by the accepting unit 701 is a specified selection operation, and if the accepted handwriting is a specified selection operation, executes a judgment process to judge whether a specified attribute is set in the area where the specified selection operation was accepted.

[0056] For example, as shown in FIG. 4(A), an operation of selecting "no" from multiple options "yes" and "no" using a figure 402 such as a circle is an example of a predetermined selection operation. Note that the predetermined selection operation is not limited to an operation of encircling an answer, but may be an operation of selecting an answer using another figure such as a check mark. Also, as shown in FIG. 4(A), a predetermined attribute in which text information of the multiple options and area information are set is set in advance in an area 406 where one answer is selected from multiple options by a predetermined selection operation. Note that the predetermined attribute will be described in detail later.

[0057] If the handwriting accepted by the accepting unit 701 is a predetermined selection operation and a predetermined attribute is set in the area where the predetermined selection operation is accepted, the converting unit 804 converts the accepted handwriting into text data corresponding to the object selected by the predetermined selection operation.

[0058] 4A, for example, assume that "absent" is selected from multiple options "present" and "absent" in an area 406 in which a predetermined attribute is set by a predetermined selection operation using a figure 402 such as a circle. In this case, the conversion unit 804 converts the figure 402 into text data corresponding to "absent" based on the text information set in the predetermined attribute and the area information.

[0059] In addition, if the judgment unit 803 determines that the handwriting accepted by the accepting unit 701 is not a specified selection operation, and if it determines that a specified attribute is not set in the area in which the specified selection operation is accepted, the conversion unit 804 does not convert the accepted handwriting into text data.

[0060] The output unit 805 executes an output process to output the text data converted by the conversion unit 804 to a predetermined output destination. In this way, when the handwriting accepted by the acceptance unit 701 is a predetermined selection operation for selecting an object displayed in a predetermined area, the output unit 805 can output text data corresponding to the object selected by the predetermined selection operation to the predetermined output destination. Note that the output unit 805 does not output text data corresponding to an object not selected by the predetermined selection operation to the predetermined output destination.

[0061] As a result, for example, when "No" is selected from the multiple options "Yes" and "No" as shown in FIG. 4(A), the information processing device 100 can display only the selected "No" in the answer column 407 as shown in FIG. 4(C).

[0062] For example, when the predetermined output destination is the external device 10, the output unit 805 transmits the text data converted by the conversion unit 804 to the external device 10 using the communication device 522 or the like. Also, when the predetermined output destination is the display 101, the output unit 805 uses the display control unit 802 or the like to display the text data converted by the conversion unit 804 on the display 101. Furthermore, when the predetermined output destination is the storage unit 705, the output unit 805 stores the text data converted by the conversion unit 804 in the storage unit 705.

[0063] If the determination unit 803 determines that the handwriting accepted by the acceptance unit 701 is not a predetermined selection operation, or if the determination unit 803 determines that a predetermined attribute is not set in the area where the predetermined selection operation is accepted, the conversion control unit 806 converts the accepted handwriting into text data. Note that the process of converting the accepted handwriting into text data by the conversion control unit 806 may be the same as the conventional technology disclosed in, for example, Patent Document 1, etc.

[0064] The functional configuration of the overall control unit 703 shown in FIG. 8 is an example. For example, the determination unit 803 may be divided into a first determination unit that determines whether the handwriting accepted by the acceptance unit 701 is a predetermined selection operation, and a second determination unit that determines whether a predetermined attribute is set in the area where the predetermined selection operation is accepted, if the accepted handwriting is the predetermined selection operation. The function of the conversion unit 804 may be included in the output unit 805. The input control unit 801 and the display control unit 802 may be one input display control unit. Furthermore, the input control unit 801 may be included in the acceptance unit 701.

[0065] <Processing flow> Next, a process flow of the information processing method according to the first embodiment will be described. Fig. 9 is a flowchart showing an example of a process of the information processing device according to the first embodiment. This process shows an example of a process executed by the information processing device 100 having the functional configuration described in Figs. 7 and 8, for example.

[0066] In step S901, when the accepting unit 701 accepts handwritten input on the display screen displayed on the display 101, the information processing device 100 executes the processes in and after step S902.

[0067] In step S902, the determination unit 803 determines whether the handwriting accepted by the acceptance unit 701 is a predetermined selection operation.

[0068] Figure 10 is a diagram showing an example of a predetermined selection operation according to an embodiment. As shown in Figure 10(A), the predetermined selection operation includes a circle 1001 that encloses one of "Yes" and "No", which are examples of a plurality of objects displayed in a predetermined area 1000 of a display screen. In the example of Figure 10(A), "No" enclosed by the circle 1001 is the object selected by the predetermined selection operation.

[0069] 10(B), the predetermined selection operation includes a check mark 1002 for checking one of "Yes" and "No", which are examples of a plurality of objects displayed in a predetermined area 1000 of the display screen. In the example of FIG. 10(B), "No" checked with the check mark 1002 is the object selected by the predetermined selection operation.

[0070] In this case, when the handwriting accepted by the accepting unit 701 is a circle 1001 or a check mark 1002, the determining unit 803 determines that the handwriting accepted by the accepting unit 701 is a predetermined selecting operation. In addition, when the handwriting accepted by the accepting unit 701 is neither a circle 1001 nor a check mark 1002, the determining unit 803 determines that the handwriting accepted by the accepting unit 701 is not a predetermined selecting operation.

[0071] If the handwriting accepted by the accepting unit 701 is not a predetermined selection operation, the determining unit 803 shifts the process to step S903. On the other hand, if the handwriting accepted by the accepting unit 701 is a predetermined selection operation, the determining unit 803 shifts the process to step S904.

[0072] When the process proceeds to step 903, the conversion control unit 806 of the information processing device 100 converts the input handwritten data into text data. This process may be similar to the conventional technology disclosed in Patent Document 1 and the like.

[0073] For example, when the accepting unit 701 accepts handwritten input, the conversion control unit 806 controls the handwriting recognition control unit 706, the character string conversion control unit 708, and the predictive conversion control unit 711 to generate handwritten recognition character string candidates, language character string candidates, conversion character string candidates, and predicted character string candidates. Note that the handwriting recognition control unit 706, the character string conversion control unit 708, and the predictive conversion control unit 711 generate handwritten recognition character string candidates, language character string candidates, conversion character string candidates, and predicted character string candidates, for example, as described above in FIG.

[0074] Furthermore, the conversion control unit 806 starts a candidate display timer when the accepting unit 701 accepts handwritten input, and when the candidate display timer times out, causes the display 101 to display selectable candidates. The selectable candidates include, for example, handwritten recognition character string candidates, language character string candidates, conversion character string candidates, character string / predictive conversion candidates, and operation command candidates that are displayed in a selectable manner. The conversion control unit 806 converts the input handwritten text into text data according to a selection operation by the user 2 on the selectable candidates.

[0075] On the other hand, in step S904, the determining unit 803 determines whether or not a predetermined attribute is set in the area where the handwritten text accepted by the accepting unit 701 has been input.

[0076] 11 is a diagram showing an example of attribute information according to an embodiment. In the handwriting display area of ​​the display screen displayed on the display 101, for example, attribute information as shown in FIG.

[0077] As shown in Fig. 11, the attribute information includes a plurality of attributes and a setting value for each attribute. In the example of Fig. 11, the plurality of attributes include, for example, attributes such as field name, text data, font, font size, color, arrangement, alignment, and flag. The field name is information indicating the name of an area, etc. The text data is set to text data obtained by converting handwriting into text, etc. The font, font size, color, arrangement, and alignment are information indicating a method of displaying text based on the text data.

[0078] When the area corresponding to the attribute information 1100 is an area for selecting one answer from a plurality of options, for example, flag information 1110 as shown in FIG. 11(B) is set in the flag.

[0079] Fig. 11(B) shows an example of flag information 1110 in the case where two options, "Yes" and "No", as shown in Fig. 12, are displayed in an area corresponding to attribute information 1100. In the example of Fig. 11(B), the flag information 1110 includes information such as "item", "text information", and "area information".

[0080] In Fig. 11(B), "item 1" corresponds to the "Yes" option in predetermined area 1200 shown in Fig. 12, and "item 2" corresponds to the "No" option in predetermined area 1200. Furthermore, the "text information" of "item 1" indicates the text to be displayed in arrangement 1201 of item 1 in Fig. 12, and the "text information" of "item 2" indicates the text to be displayed in arrangement 1202 of item 2 in Fig. 12. Furthermore, the "area information" of "item 1" indicates the position on the display screen of arrangement 1201 of item 1 in Fig. 12, and the "area information" of "item 2" indicates the position on the display screen of arrangement 1202 of item 2 in Fig. 12.

[0081] With this attribute information 1100, the display control unit 802 can display multiple options in a selectable manner in a predetermined area 1200, as shown in Fig. 12. Note that the flag information 1110 set in the attribute information 1100 is an example of a predetermined attribute. Also, the "flag" and "flag information" may be named differently.

[0082] In step S904 of FIG. 9, if flag information is set in the attribute information 1100 of the area where the handwritten text accepted by the accepting unit 701 is input, the judgment unit 803 judges that a predetermined attribute is set in the input area.

[0083] If the input area does not have a predetermined attribute, the determination unit 803 shifts the process to step S903. On the other hand, if the input area has a predetermined attribute, the determination unit 803 shifts the process to step S905.

[0084] In step S905, the determination unit 803 determines whether or not the input handwriting exists in an area to which a predetermined attribute is set.

[0085] Fig. 13 is a diagram for explaining a method of determining a selected item according to an embodiment. As shown in Fig. 13(A), when the input handwriting 1301 is present in the layout 1202 of the item 2, the determination unit 803 determines that the input handwriting 1301 is present in the layout 1202 of the item 2, which is an example of an area to which a predetermined attribute is set. Also, as shown in Fig. 13(B), even if a part of the input handwriting 1302 protrudes from the layout 1202 of the item 2, the determination unit 803 may determine that the input handwriting 1301 is present in the layout 1202 of the item 2, which is an example of an area to which a predetermined attribute is set.

[0086] Preferably, when the input handwriting is included in an area to which a predetermined attribute is set at a predetermined ratio or more, the determination unit 803 determines that the input handwriting is present in the area to which a predetermined attribute is set. Also, when the input handwriting is included in a plurality of areas to which a predetermined attribute is set, the determination unit 803 determines that the input handwriting is present in an area that includes a larger amount of the input handwriting.

[0087] If the input handwriting does not exist in the area in which the predetermined attribute is set, the determination unit 803 shifts the process to step S903. On the other hand, if the input handwriting exists in the area in which the predetermined attribute is set, the determination unit 803 shifts the process to step S906.

[0088] When the process proceeds to step S906, the conversion unit 804 converts the input handwriting into text data corresponding to the selected object based on the area where the handwriting exists. For example, as shown in FIG. 13(A), it is assumed that "none" is selected by the handwriting 1301 from among multiple options "yes" and "no" displayed in the predetermined area 1200. In this case, the conversion unit 804 obtains text information "no" corresponding to the arrangement 1202 of the item 2, which is the area where the handwriting 1301 exists, from the flag information 1110 as shown in FIG. 11(B). Furthermore, the conversion unit 804 converts the handwriting 1301 into the text information "no" or text data corresponding to "no". Furthermore, the conversion unit 804 sets the text data corresponding to the text information "no" or "no" to the setting value 1101 of the attribute "text data" of the attribute information 1100 of the predetermined area 1200 as shown in FIG. 11(A), for example.

[0089] In this way, the conversion unit 804 changes the text data to be converted depending on which area of ​​the areas corresponding to each of the multiple objects the handwriting accepted by the acceptance unit overlaps.

[0090] In step S907, the output unit 805 outputs the text data converted in steps S906 and S903 to a predetermined output destination. For example, the output unit 805 uses the display control unit 802 to output the converted text data to the display 101, thereby making it possible to display only the text 405 "No" as the answer to the medical question "Vaccine vaccination against coronavirus," as shown in FIG. 4(C).

[0091] Preferably, the output unit 805 also transmits the converted text data to the external device 10 using the communication device 522. Note that the process shown in Fig. 9 is an example. For example, the output unit 805 may convert all handwritten input into the medical questionnaire 300 as shown in Fig. 300 into text data, and then transmit the converted text data collectively to the external device 10.

[0092] Through the process of FIG. 9, information processing device 100, which is an information processing device that converts an object handwritten on a display screen into text data, can convert an answer selected from a plurality of options into text data.

[0093] [Second embodiment] The information processing system 1 may analyze the collected information of the operation history in the information processing device 100, notify the user 2 of the analysis result, and make a suggestion to change the setting conditions.

[0094] <Functional configuration> Fig. 14 is a diagram showing an example of the functional configuration of an overall control unit according to the second embodiment. As shown in Fig. 14, an overall control unit 703 according to the second embodiment has an analysis unit 1401 and a display reception unit 1402 in addition to the functional configuration of the overall control unit 703 according to the first embodiment described in Fig. 8.

[0095] The analysis unit 1401 is realized, for example, by a program executed by the CPU 501, and executes an analysis process for analyzing the operation history of the user 2. For example, the analysis unit 1401 analyzes the operation history of the user 2 and detects that there is a predetermined number of times or more of handwriting indicating an action of selecting text data displayed in a predetermined area.

[0096] For example, the analysis unit 1401 may detect the action of selecting text data to be displayed in a specified area by using a machine learning model that has been trained in advance based on operation history information.

[0097] Here, machine learning is a technology for making a computer acquire human-like learning capabilities, in which the computer autonomously generates algorithms required for judgments such as data identification from learning data that is previously loaded, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods; any learning method for machine learning is acceptable.

[0098] The analysis unit 1401 may be included in the external device 10 instead of the information processing device 100. In this case, the information processing device 100 uses the analysis unit 1401 included in the external device 10 to detect that an operation of selecting text data displayed in a predetermined area has been performed a predetermined number of times or more.

[0099] The display reception unit 1402 is realized by, for example, a program executed by the CPU 501. The display reception unit 1402, for example, causes the display 101 or the like to display the analysis result by the analysis unit 1401, and executes a display reception process of receiving a change to the setting conditions of the information processing device 100 based on the analysis result.

[0100] For example, the display receiving unit 1402 may display operation command candidates before converting the input handwriting into text data, and may suggest to the operator whether or not to additionally display the attribute name set in the flag along with the character string candidates obtained by character recognition in the operation command.

[0101] As a specific example, assume that the user 2 inputs the circle 1001 or the check mark 1002 for performing a selection operation described in Fig. 10 into a predetermined area, but the input is not recognized correctly. In this case, if the information processing device 100 displays the text information set in the flag information 1110 as one of the operation command candidates together with the character string candidate data to be displayed in the case of erroneous recognition, the user 2 can easily select the text information. In this case, it is desirable for the display receiving unit 1402 to display, for example, a notification screen 1500 as shown in Fig. 15 on the display 101.

[0102] FIG. 15 is a diagram showing an example of a notification screen according to the second embodiment. In the example of FIG. 15, a notification screen 1500 displays a list of user operation histories 1501 and notification contents 1502 for the user. As an example of the notification contents 1502, the notification screen 1500 displays the following text: "There is a tendency for handwriting ("◯") indicating the action of selecting text data displayed in a predetermined area to be used. Would you like to display an operation command for selecting text data displayed in a predetermined area in the list of operation command candidates?". By selecting a "Yes" button 1503 on the screen, the user can change the settings of the information processing device 100 so that an operation command for selecting text data displayed in a predetermined area is displayed in the list of operation command candidates.

[0103] <Processing flow> Fig. 16 is a flowchart showing an example of a process of changing a setting condition according to the second embodiment. The information processing device 100 executes the process of Fig. 16 at predetermined time intervals, for example, in addition to the process of the information processing device described in Fig. 9.

[0104] In step S1601, the analysis unit 1401 analyzes the operation history of the user 2. For example, the analysis unit 1401 analyzes the number of times each operation is performed by the user 2, as shown in the operation history list 1501 in FIG.

[0105] In step S1602, the display receiving unit 1402 judges whether or not there is a predetermined number of times or more (for example, five times or more) of handwriting indicating an action of selecting an object displayed in a predetermined area. If there is a predetermined number of times or more of handwriting indicating an action of selecting an object displayed in a predetermined area, the display receiving unit 1402 shifts the process to step S1603. On the other hand, if there is not a predetermined number of times or more of handwriting indicating an action of selecting an object displayed in a predetermined area, the display receiving unit 1402 returns the process to step S1601 (or ends the process in FIG. 16).

[0106] In step S1603, the display receiving unit 1402 displays, for example, a notification screen 1500 as shown in FIG.

[0107] In step S1604, the display receiving unit 1402 determines whether or not a "Yes" operation (selection of the "Yes" button 1503) has been received from the user 2. If a "Yes" operation has been received, the display receiving unit 1402 shifts the process to step S1605. On the other hand, if a "Yes" operation has not been received, the display receiving unit 1402 returns the process to step S1601 (or ends the process in FIG. 16).

[0108] In step S1605, the display receiving unit 1402 changes the setting of the information processing device 100 so that an operation command for selecting an object displayed in a predetermined area is displayed in the list of operation command candidates.

[0109] Through the process of FIG. 16, the information processing system 1 can analyze collected information on the operation history in the information processing device 100, notify the user 2 of the analysis result, and make a proposal to change the setting conditions.

[0110] As described above, according to each embodiment of the present invention, even if a user selects a corresponding option from among a plurality of options by handwriting, the option data can be saved.

[0111] <Supplementary Information> Each function of each embodiment described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0112] Furthermore, the information processing system 1 can be configured to share the disclosed processing steps, for example, the processes shown in Figures 9 and 16, in various combinations. For example, the process executed by the information processing device 100 can be executed by an external device 10. Furthermore, each functional configuration of the information processing device 100 may be integrated into one device or may be divided into multiple devices.

[0113] <Additional Notes> This specification discloses the following information processing device, information processing system, information processing method, and program. (Section 1) An information processing device that converts an object handwritten on a display screen into text data, a reception unit that receives the handwritten input on the display screen; an output unit that outputs, when the handwriting is a predetermined selection operation for selecting an object displayed in a predetermined area of ​​the display screen, text data corresponding to an object selected by the predetermined selection operation among a plurality of objects displayed in the predetermined area to a predetermined output destination; An information processing device having the above configuration. (Section 2) The information processing device described in claim 1, wherein the output unit does not output to the specified output destination text data corresponding to an object that is not selected by the specified selection operation among the multiple objects displayed in the specified area. (Section 3) a determination unit that determines whether the handwriting accepted by the acceptance unit is the predetermined selection operation, and if the handwriting is the predetermined selection operation, determines whether a predetermined attribute is set in the area where the predetermined selection operation is accepted; a conversion unit that converts the handwriting into text data corresponding to an object selected by the predetermined selection operation when the handwriting is the predetermined selection operation and a predetermined attribute is set in an area where the predetermined selection operation is received; 3. The information processing device according to claim 1 or 2, (Section 4) The information processing device described in claim 3, wherein the specified attributes include area information indicating an area corresponding to each of the plurality of objects to be displayed in the specified area, and text information for identifying text data corresponding to each of the plurality of objects. (Section 5) 5. The information processing device according to claim 4, wherein the conversion unit changes the text data to be converted depending on which area of ​​the areas corresponding to each of the plurality of objects the handwriting accepted by the acceptance unit covers. (Section 6) 6. The information processing device according to any one of claims 1 to 5, wherein the output unit transmits text data corresponding to the object selected by the selection operation to an external device. (Section 7) An analysis unit that analyzes a user's operation history; a display receiving unit that causes a display unit to display an analysis result by the analysis unit and receives a change to a setting condition of the information processing device based on the analysis result; 7. The information processing device according to any one of items 1 to 6, comprising: (Section 8) An information processing system for converting an object handwritten on a display screen into text data, comprising: a reception unit that receives the handwritten input on the display screen; an output unit that outputs, when the handwriting is a predetermined selection operation for selecting an object displayed in a predetermined area of ​​the display screen, text data corresponding to an object selected by the predetermined selection operation among a plurality of objects displayed in the predetermined area to a predetermined output destination; An information processing system having the above configuration. (Section 9) An information processing device that converts an object handwritten on a display screen into text data, accepting the handwritten input on the display screen; a process of outputting, when the handwriting is a predetermined selection operation for selecting an object displayed in a predetermined area of ​​the display screen, text data corresponding to an object selected by the predetermined selection operation among a plurality of objects displayed in the predetermined area to a predetermined output destination; An information processing method. (Section 10) 10. A program for causing an information processing device to execute the information processing method according to item 9, or a storage medium storing the program.

[0114] Although an embodiment of the present invention has been described above, the present invention is not limited to such a specific embodiment, and various modifications and applications are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0115] 1. Information Processing Systems 2. Users 10 External device 100 Information processing device 102 pen 701 Reception 803 Judgment Department 804 Converter 805 Output section 1100 Attribute information 1110 Flag information (predetermined attribute) 1401 Analysis Department 1402 Display reception section [Prior art documents] [Patent documents]

[0116] [Patent Document 1] JP 2022-139957 A

Claims

1. An information processing device that converts an object handwritten on a display screen into text data, a reception unit that receives the handwritten input on the display screen; an output unit that outputs, when the handwriting is a predetermined selection operation for selecting an object displayed in a predetermined area of ​​the display screen, text data corresponding to an object selected by the predetermined selection operation among a plurality of objects displayed in the predetermined area to a predetermined output destination; An information processing device having the above configuration.

2. The information processing device according to claim 1 , wherein the output unit does not output, to the predetermined output destination, text data corresponding to an object that is not selected by the predetermined selection operation, among the plurality of objects displayed in the predetermined area.

3. a determination unit that determines whether the handwriting accepted by the acceptance unit is the predetermined selection operation, and if the handwriting is the predetermined selection operation, determines whether a predetermined attribute is set in the area where the predetermined selection operation is accepted; a conversion unit that converts the handwriting into text data corresponding to an object selected by the predetermined selection operation when the handwriting is the predetermined selection operation and a predetermined attribute is set in an area where the predetermined selection operation is received; The information processing device according to claim 1 .

4. 4. The information processing device according to claim 3, wherein the predetermined attribute includes area information indicating an area corresponding to each of the plurality of objects to be displayed in the predetermined area, and text information for identifying text data corresponding to each of the plurality of objects.

5. The information processing apparatus according to claim 4 , wherein the conversion unit changes the text data to be converted depending on which area of ​​areas corresponding to each of the plurality of objects the handwriting accepted by the acceptance unit covers.

6. The information processing apparatus according to claim 1 , wherein the output unit transmits text data corresponding to the object selected by the selection operation to an external device.

7. An analysis unit that analyzes a user's operation history; a display receiving unit that displays an analysis result by the analysis unit on a display unit and receives a change to a setting condition of the information processing device based on the analysis result; The information processing device according to claim 1 .

8. An information processing system for converting an object handwritten on a display screen into text data, comprising: a reception unit that receives the handwritten input on the display screen; an output unit that outputs, when the handwriting is a predetermined selection operation for selecting an object displayed in a predetermined area of ​​the display screen, text data corresponding to an object selected by the predetermined selection operation among a plurality of objects displayed in the predetermined area to a predetermined output destination; An information processing system having the above configuration.

9. An information processing device that converts an object handwritten on a display screen into text data, accepting the handwritten input on the display screen; a process of outputting, when the handwriting is a predetermined selection operation for selecting an object displayed in a predetermined area of ​​the display screen, text data corresponding to an object selected by the predetermined selection operation among a plurality of objects displayed in the predetermined area to a predetermined output destination; An information processing method.

10. A program for causing an information processing device to execute the information processing method according to claim 9.

Images