System for processing stroke data and method for processing stroke data

Abstract

A system and method capable of easily retrieving information represented by stroke data are provided. The system (1) of the present invention is a system having a processor (51). The processor (51) acquires stroke data generated by handwriting input, and generates semantic metadata including meaning data representing the meaning of the stroke data and purpose data determined based on the meaning data by performing semantic analysis on the stroke data.

Description

Technical Field

[0001] The present invention relates to a system for processing stroke data and a method for processing stroke data. Background Art

[0002] There is known an ink file containing stroke data generated by handwriting input. If an ink file is used, the handwriting situation can be reproduced by regenerating the stroke data contained therein.

[0003] An example of an ink file is disclosed in Patent Document 1. The ink file of this example is configured to include metadata for identifying an input device used in handwriting input. This metadata is used to implement decoding or regeneration processing of stroke data corresponding to different input devices.

[0004] In addition, in recent years, AI (Artificial Intelligence) assistants have attracted attention. An AI assistant performs tasks and services based on voice commands and is implemented by various intelligent devices that have been significantly popularized in recent years, such as smartphones, tablet terminals, and smart speakers. An example of an AI assistant is disclosed in Patent Document 2.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: International Publication No. 2016 / 157400

[0008] Patent Document 2: U.S. Patent No. 9,384,732 Summary of the Invention

[0009] Problems to be Solved by the Invention

[0010] However, retrieving information represented by stroke data contained in an ink file is a laborious task. Therefore, a technique that can simply retrieve information represented by stroke data is needed.

[0011] Therefore, one object of the present invention is to provide a system and method capable of easily retrieving information represented by stroke data.

[0012] Means for Solving the Problems

[0013] The system of the present invention is a system having a processor that acquires stroke data generated by handwriting input and generates semantic metadata including meaning data representing the meaning of the stroke data and purpose data determined based on the meaning data by performing semantic analysis on the stroke data.

[0014] The method of the present invention is a method for processing stroke data. Stroke data is generated based on a handwritten input, and semantic analysis is performed on the generated stroke data to generate semantic metadata including meaning data representing the meaning of the stroke data and target data determined based on the meaning data.

[0015] Advantageous Effects of the Invention

[0016] According to the present invention, information represented by stroke data can be easily retrieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 FIG. is a diagram showing the structure of system 1 according to the first embodiment of the present invention.

[0018] Figure 2 FIG. shows Figure 1 a schematic block diagram of the hardware structures of the AI assistant terminal 10 and the tablet terminal 20 shown respectively.

[0019] Figure 3 (a) to (c) of FIG. are diagrams showing tables pre-stored in the memory 12 of the AI assistant terminal 10 shown in Figure 2 FIG.

[0020] Figure 4 FIG. shows Figure 2 a diagram of the user table stored in the memory 22 of the tablet terminal 20 shown in FIG.

[0021] Figure 5 FIG. shows Figure 2 a processing flowchart of the processing executed by the processor 11 of the AI assistant terminal 10 shown in FIG.

[0022] Figure 6 FIG. is for explaining Figure 5 a specific example of the processing of the AI assistant terminal 10 shown in FIG.

[0023] Figure 7 (a) of FIG. is a diagram showing an example of a command generated in step S7 shown in Figure 6 FIG. for the example shown in Figure 5 FIG., and (b) is a diagram showing a modified example of the command generated in step S7 shown in Figure 6 FIG. for the example shown in Figure 5 FIG.

[0024] Figure 8 FIG. shows Figure 2 a processing flowchart of the processing executed by the processor 21 of the tablet terminal 20 shown in FIG.

[0025] Figure 9 FIG. is regarding a command sent out by the AI assistant terminal 10 having Figure 7(a) shows the content more specifically Figure 8 is a flowchart of the process.

[0026] Figure 10 is a diagram showing an example of a GUI (Graphical User Interface) in which an email application started by the Figure 9 process flow is displayed on the display surface of the display 24, that is, screen 100.

[0027] Figure 11 is a diagram showing system 1 of the first modification of the first embodiment of the present invention.

[0028] Figure 12 is a diagram showing system 1 of the second modification of the first embodiment of the present invention.

[0029] Figure 13 is a diagram showing the structure of system 2 of the second embodiment of the present invention.

[0030] Figure 14 is a diagram showing Figure 13 a schematic block diagram of the hardware structures of the AI assistant terminal 10, the tablet terminal 20, and the semantic / ink server 50 shown.

[0031] Figure 15 is a diagram showing Figure 14 the ink file database stored in the memory 12 of the AI assistant terminal 10 shown.

[0032] Figure 16 (a) and (b) of are diagrams showing tables pre-stored in the memory 12 of the AI assistant terminal 10 shown. Figure 14 shown.

[0033] Figure 17 is a flowchart of the process executed by the processor 21 of the tablet terminal 20 shown by Figure 14 shown.

[0034] Figure 18 is a diagram showing Figure 17 an example of a series of stroke data generated in step S21 shown.

[0035] Figure 19 is a flowchart of the process executed by the processor 51 of the semantic / ink server 50 shown by Figure 14 shown.

[0036] Figure 20 (a) of shows that for the ink file of the example shown by Figure 18 shown, Figure 19The figure showing the result of the grouping in step S31 shown, and (b) is a figure showing the semantic metadata generated from each group shown in (a).

[0037] Figure 21 In (a) shows the ink file for another example has been Figure 19 The figure showing the result of the grouping in step S31 shown, and (b) is a figure showing the semantic metadata generated from each group shown in (a).

[0038] Figure 22 Is shown by Figure 14 The processing flowchart of the processing executed by the processor 11 of the AI assistant terminal 10 shown.

[0039] Figure 23 Is shown by Figure 14 The processing flowchart of other processing executed by the processor 11 of the AI assistant terminal 10 shown.

[0040] Figure 24 Is a figure for explaining Figure 23 The figure of a specific example of the processing of the AI assistant terminal 10 shown.

[0041] Figure 25 Is about Figure 20 The figure showing an example of displaying an ink file on the display 14 for the example shown.

[0042] Figure 26 Is about Figure 20 The figure showing an example of displaying an alternative list constituted based on semantic metadata on the display 14 for the example shown.

[0043] Figure 27 Is a figure showing the system 2 which is a modification of the second embodiment of the present invention. Detailed Description of the Invention

[0044] Hereinafter, while referring to the drawings, embodiments of the present invention will be described in detail.

[0045] Figure 1 Is a figure showing the structure of the system 1 of the first embodiment of the present invention. As shown in this figure, the system 1 is a system installed, for example, in a home, and is configured to include an AI assistant terminal 10, a tablet terminal 20, and an electronic pen P.

[0046] The AI assistant terminal 10 is a computer that supports voice input and executes tasks and services according to commands based on the input voice (hereinafter referred to as "voice commands"). Specifically, as the AI assistant terminal 10, it is preferable to use a smart speaker that has received attention in recent years. The tablet terminal 20 is a computer that supports touch input based on an electronic pen P or a finger and is configured to be able to execute various applications such as a messaging application. The AI assistant terminal 10 and the tablet terminal 20 are connected to each other by wire or wirelessly to form the system 1 of this embodiment.

[0047] Figure 2 It is a schematic block diagram showing the hardware structures of the AI assistant terminal 10 and the tablet terminal 20 respectively. Hereinafter, while referring to this Figure 2 , the structures of the AI assistant terminal 10 and the tablet terminal 20 will be described in detail.

[0048] If we first focus on the AI assistant terminal 10, as Figure 2 shown, the AI assistant terminal 10 has a structure in which a processor 11, a memory 12, a communication unit 13, a display 14, a microphone 15, and a speaker 16 are connected to each other via an internal bus.

[0049] The processor 11 is a central processing unit that controls each part of the AI assistant terminal 10 and has a function of reading and executing a program stored in the memory 12. In the program executed by the processor 11, an AI engine for performing processing related to voice input such as the following-described voice recognition processing is included. The processor 11 functions as follows: by using this AI engine to recognize the voice input through the microphone 15, a command for starting an application in the handwriting input mode is generated.

[0050] The memory 12 is a storage device configured to be able to store various programs and data. In a typical example, the memory 12 includes a main storage device such as an LPDDR SDRAM (Low Power Double Data Rate Synchronous Dynamic Random Access Memory) and an auxiliary storage device such as a flash memory or an SSD (Solid State Drive).

[0051] In the memory 12, data associating the features of voices with strings and data associating the features of voices with speaker name data (hereinafter referred to as "user names") are pre-stored, for example, by machine learning. The AI engine performs voice recognition processing for converting the input voice into a string and speaker recognition processing for determining the speaker (user name) of the input voice by referring to these data.

[0052] Figure 3 (a) to (c) are diagrams showing tables pre - stored in the memory 12. Figure 3 (a) shows an action content table that correlates voice information, the content of the action to be started, and the application to be started. Note that the voice information is various information corresponding to the result of recognition processing performed by the AI engine, such as a string representing the voice, the speaker of the voice, etc.

[0053] The processor 11 is configured to determine the content of the action to be started and the application to be started based on the information obtained as a result of the recognition processing of the input voice using the AI engine and the voice information stored in the action content table. For example, if the string obtained as a result of the voice recognition processing contains "Write Email", the processor 11 determines to start the email application. Additionally, if the string obtained as a result of the voice recognition processing contains "What is on my*?(* represents an arbitrary string)", the processor 11 determines to prompt the ink file. In the present embodiment, the case of starting the email application in the former case will be described in detail. The case of prompting the ink file in the latter case will be described in detail in the second embodiment. (* represents an arbitrary string), the processor 11 determines to prompt the ink file. In the present embodiment, the case of starting the email application in the former case will be described in detail. The case of prompting the ink file in the latter case will be described in detail in the second embodiment.

[0054] In Figure 3 (b), a start mode table that correlates voice information and the start mode in the case of starting the email application is shown. Note that in this figure, only the start mode table for the email application is shown, but it is preferable to prepare the same start mode table for each application.

[0055] The processor 11 is configured to: when it is determined to start the email application according to the Figure 3 table in (a), determine its start mode by referring to Figure 3 (b) sequentially from the top. Here, as shown in Figure 3 (b), in the start mode of the email application, there are at least 3 start modes. The first is the unrestricted handwriting input mode, in which input based on an arbitrary electronic pen P or finger can be achieved. The second is the speaker's handwriting input mode, in which input based on the electronic pen P of the speaker of the input voice can be achieved. Input based on other electronic pens P or fingers is not allowed. The third is the keyboard input mode, in which input based on the virtual keyboard displayed on the display surface of the tablet terminal 20 or the physical keyboard externally provided to the tablet terminal 20 can be achieved.

[0056] In Figure 3In the example of (b), the processor 11 first determines whether the string obtained as a result of the speech recognition process contains the string "With pen" and the string "Pen is unlimited". If it is determined that they are included, it is decided to start the email application in the unrestricted handwriting input mode. On the other hand, if it is determined that they are not included, the processor 11 then determines whether the string obtained as a result of the speech recognition process contains the string "With pen". If it is determined that it is included, it is decided to start the email application in the speaker's handwriting input mode. Hereinafter, by repeatedly determining in the same way, the processor 11 executes the decision of the start mode of the email application. Finally, if the string obtained as a result of the speech recognition process does not contain any of "With pen", "Pen is unlimited", and "With keyboard", the processor 11 decides to start the email application in the speaker's handwriting input mode.

[0057] It should be noted that in this embodiment, an example of distinguishing between the "unrestricted handwriting input mode" and the "speaker's handwriting input mode" is described, but they may not be particularly distinguished and only treated as the "handwriting input mode". In addition, the Figure 3 start mode table shown in (b) may not be used. If the string obtained as a result of the speech recognition process contains "Write Email", the email application is started in the "handwriting input mode".

[0058] In Figure 3 (c), a setting data table that correlates the voice information with the setting data in the case of starting the email application is shown. It should be noted that only the setting data table for the email application is shown in this figure, but it is preferable to prepare the same setting data table for each application.

[0059] The processor 11 is configured to: when starting the email application according to the Figure 3 table in (a), determine the setting data by referring to Figure 3 (c). For example, if the string obtained as a result of the speech recognition process contains the string "To*", the processor 11 determines the string in the part of "*" as the recipient in the email application. In addition, the user name of the speaker obtained as a result of the speaker recognition process is determined as the sender in the email application.

[0060] Return Figure 2。The communication unit 13 is a functional unit for communicating with other computers via a network such as the Internet or in a peer-to-peer manner. Specifically, it is configured to be able to communicate via one or more of various communication standards such as wireless LAN, Bluetooth (registered trademark), and Ethernet (registered trademark).

[0061] The display 14 is a display device configured to be able to display various images according to the instructions of the processor 11, and is typically composed of a liquid crystal display or an organic EL display. The microphone 15 is a voice input device that converts the voice coming from the outside into a digital signal and supplies it to the processor 11. The speaker 16 is a voice output device that converts the electrical signal supplied from the processor 11 into a voice and outputs it.

[0062] Next, if we focus on the tablet terminal 20, as Figure 2 shown, the tablet terminal 20 has a structure in which a processor 21, a memory 22, a communication unit 23, a display 24, and a sensor 25 are interconnected via an internal bus.

[0063] The processor 21 is a central processing unit that controls each part of the tablet terminal 20, and has a function of reading and executing the programs stored in the memory 22. Among the programs executed by the processor 21, there is a messaging application configured to be able to send and receive handwritten data input by the electronic pen P. The processor 21 functions to start the application in the handwritten input mode based on the command generated by the AI assistant terminal 10.

[0064] Here, the messaging application is typically an email application configured to be able to send and receive emails, but it can also be other types of messaging applications such as an instant messaging tool configured to be able to send and receive short messages in real time between the connected users. In this embodiment, the case of using an email application is taken as an example for explanation.

[0065] The memory 22, the communication unit 23, and the display 24 are respectively the same devices as the memory 12, the communication unit 13, and the display 14 of the AI assistant terminal 10 as hardware.

[0066] Figure 4 is a diagram showing the user table stored in the memory 22. As shown in this diagram, this user table is a table that correlates the user name and the pen ID. As the user names in the user table, a series of user names that can be determined by the speaker recognition process performed by the AI engine are used. The pen ID is data of a specified number of digits pre-stored in the electronic pen P, and has the function of identifying each electronic pen P.

[0067] Return Figure 2, the sensor 25 is a position detection device configured to detect the positions of the electronic pen P and the finger on the touch surface. In a typical example, the touch surface is constituted by the display surface of the display 24. The sensor 25 is configured to periodically detect the positions of the electronic pen P and the finger present on the touch surface, and supply the coordinates indicating the detected positions to the processor 21 each time a detection is made.

[0068] As a specific method of position detection of the sensor 25, various methods such as the resistive film method, the electromagnetic induction method, and the capacitive method (active electrostatic method) can be used. Hereinafter, the case of using the capacitive method (active electrostatic method) will be taken as an example for explanation. According to this method, the sensor 25 can detect both the electronic pen P and the finger, and can transmit and receive signals bidirectionally with the electronic pen P5. Hereinafter, the signal transmitted by the electronic pen P in this bidirectional signal transmission and reception will be referred to as the "downlink signal", and the signal transmitted by the sensor 25 will be referred to as the "uplink signal".

[0069] The uplink signal is a signal that functions to synchronize the electronic pen P with the sensor 25 and send commands to the electronic pen P. The sensor 25 is configured to periodically send the uplink signal.

[0070] The downlink signal is configured to include a non-modulated signal, i.e., a burst signal, for enabling the sensor 25 to detect the position of the electronic pen P, and a data signal for sending various data obtained in the electronic pen P to the sensor 25. Among the various data sent through the data signal, there are data indicating the pressure applied to the pen tip (pen pressure value), data indicating the on / off information of the switch provided on the housing, and the above-mentioned pen ID, etc.

[0071] The electronic pen P continuously or intermittently performs the uplink signal reception operation in a state where the sensor 25 has not been detected yet. And, when receiving the uplink signal sent by the sensor 25, it determines the transmission and reception process synchronized with the sensor 25. After that, the electronic pen P performs the reception of the uplink signal and the reception of the downlink signal according to the determined transmission and reception process. In addition, when the uplink signal contains a command, the electronic pen P obtains the data required by the command and includes the data in the data signal and sends it to the sensor 25. However, regarding the pen pressure value, it is appropriately included in the data signal and sent to the sensor 25 in a manner of being periodically sent at a prescribed period regardless of the command.

[0072] When the sensor 25 detects a downlink signal transmitted by the undetected electronic pen P according to the uplink signal, it first causes the electronic pen P to transmit the pen ID by sending an uplink signal including a command requesting the pen ID. Then, it obtains the pen ID from the received data signal and supplies it to the processor 21. In addition, coordinates indicating the position of the electronic pen P are periodically obtained based on the reception intensity of the burst signal transmitted by the electronic pen P, and the pen pressure value is obtained based on the data signal transmitted by the electronic pen P. These data are successively supplied to the processor 21.

[0073] The processor 21 is configured to control the position of the cursor on the display screen displayed on the display 24 according to the coordinates supplied from the sensor 25. In addition, when receiving a handwritten input based on the electronic pen P, the processor 21 monitors the pen pressure value supplied from the sensor 25. When the pen pressure value becomes a value greater than 0 (i.e., when a pen-down occurs), it starts generating stroke data representing the trajectory of the electronic pen P. After that, during the period when the pen pressure value remains greater than 0, the continuously supplied coordinates and pen pressure value are successively appended to the stroke data. When the pen pressure value returns to 0 (i.e., when a pen-up occurs), the generation of the stroke data ends. After that, at a specified timing, the processor 21 generates a file (hereinafter referred to as an "ink file") including one or more stroke data generated so far and saves it in the memory 22.

[0074] Above, the structures of the AI assistant terminal 10 and the tablet terminal 20 have been described in detail. Next, regarding the processing performed by the AI assistant terminal 10 and the tablet terminal 20 of the present embodiment, it will be described in more detail while referring to their respective processing flows. Figure 2 Together with

[0075] Figure 5 It is a processing flow chart showing the processing executed by the processor 11 of the AI assistant terminal 10. As shown in this figure, the processor 11 first obtains the voice input to the microphone 15 (step S1), and transforms the voice into a character string and determines the speaker of the voice by performing recognition processing on the obtained voice (step S2).

[0076] Figure 6 It is shown for explaining Figure 5 A specific example of the processing of the AI assistant terminal 10 shown. As shown in this figure, in this example, the user issues a voice "Write E-mail to mom." The processor 11 transforms the above voice collected by the microphone 15 into a character string "Write E-mail to mom." and determines the speaker of the voice, "Tom", by performing a specified recognition processing using the above AI engine (step S2).

[0077] Return Figure 5 . The processor 11 then determines the start of the application based on the information obtained in step S2 (step S3), and further determines the application to be started (step S4). These determinations are performed by the processor 11 referring to Figure 3 the action content table shown in (a). In the case of Figure 6 an example, since the string obtained in step S2 contains the string "Write E-mail", the processor 11 refers to Figure 3 the first row of the action content table shown in (a). Then, according to the description content of the first row, the start of the application is determined, and the application to be started is determined as the email application.

[0078] The processor 11 further determines to start the application determined in step S4 in the handwriting input mode based on the information obtained in step S2 (step S5). In the Figure 6 example, since the string obtained in step S2 does not contain any of "With pen", "Pen is unlimited", and "With keyboard", the processor 11 refers to Figure 3 the fifth row of the start mode table shown in (b). Then, according to the description content of the fifth row, it is determined to start the application in the speaker's handwriting input mode.

[0079] The processor 11 then determines the data to be set for the application determined in step S5 based on the information obtained in step S2 (step S6). In the Figure 6 example, since the string obtained in step S2 contains the string "To Mom", the processor 11 refers to Figure 3 the first row of the start mode table shown in (c). Then, according to the description content of the first row, "Mom" is determined as the recipient. Additionally, when the speaker of the input voice is determined in step S2, the processor 11 refers to Figure 3 the second row of the start mode table shown in (c). Then, according to the description content of the second row, the user name of the determined speaker is determined as the sender.

[0080] After that, the processor 11 generates a command for starting the application determined in step S4 in the handwriting input mode determined in step S5 and setting the data determined in step S6 (step S7).

[0081] Figure 7 (a) is regarding Figure 6The figure shown illustrates an example of a command generated in step S7. The command of this example is configured to include information for determining the application to be started (start), information for determining the mode of the application to be started (mode setting), information for determining the sender of the email, and information for determining the recipient of the email. Figure 6 In this example, the specific contents of this information respectively become the email application, the handwriting input mode of the speaker (= Tom), Tom, and Tom's mother.

[0082] Figure 7 (b) is regarding Figure 6 The figure shown illustrates a modified example of a command generated in step S7. In this example, data representing the relationships between users and the email addresses of each user are pre-stored in the memory 12 shown in Figure 2 When the processor 11 determines that Tom is the sender, it reads out Tom's email address "Tom@xxx.com" from the memory 12. In addition, when it determines that Tom's mother is the recipient, the processor 11 first determines "Anna" as Tom's mother by referring to the information in the memory 12. Then, it reads out Anna's email address "Anna@xxx.com" from the memory 12. After that, the processor 11 sets the two read email addresses as the sender and the recipient respectively, replacing the strings representing "Tom" and "Tom's mother".

[0083] Return Figure 5 . The processor 11 that has generated the command in step S8 then sends the generated command to the tablet terminal 20 (step S9). Through the processing up to this point, the series of processing by the processor 11 based on the input voice ends.

[0084] Figure 8 The figure is a flowchart of the processing executed by the processor 21 of the tablet terminal 20. As shown in this figure, the processor 21 first receives the command sent from the AI assistant terminal 10 (step S10) and interprets it (step S11). Then, it starts the application indicated by the command (step S12) and sets the started application to the handwriting input mode according to the indication of the command (step S13). Moreover, after setting the data indicated by the command to the application (step S14), it starts accepting handwriting input (step S15).

[0085] Thus, according to the system 1 of the present embodiment, since the application is started in the handwriting input mode based on the voice recognition result, the user does not need to set the application to the handwriting input mode one by one through touch operations or the like, and can directly start handwriting input through voice commands.

[0086] Figure 9 Regarding the command sent from the AI assistant terminal 10 havingFigure 7 The situation of the content shown in (a) is described in more detail Figure 8 into a process flow chart of the processing. The steps S12a to 14a shown in this figure respectively correspond to Figure 8 the steps S12 to 14, and the steps S15a to 15e show the processing performed in Figure 8 the step S15.

[0087] As Figure 9 shown, in this case, the application started in step S12 becomes an email application (step S12a), the mode set in step S13 becomes Tom's handwriting input mode (a handwriting input mode that limits the input means to Tom's electronic pen P) (step S13a), and the data set in step S14 becomes the sender (Tom) and the recipient (Tom's mother) indicated by the command (step S14a).

[0088] Figure 10 It is a diagram showing an example of a GUI (Graphical User Interface) of a display surface on which an email application started by the processing flow through Figure 9 is displayed on the display 24 (refer to Figure 2 ). As shown in this figure, the screen 100 is configured to have a mode display bar 101, a sender selection bar 102, a recipient input bar 103, a carbon copy input bar 104, a subject input bar 105, and a body input bar 106.

[0089] In the mode display bar 101, the mode of the email application set by the processor 21 in step S13a is displayed. In addition, in the sender selection bar 102 and the recipient input bar 103, the sender and the recipient set by the processor 21 in step S13a are set respectively. It should be noted that the tablet terminal 20 is built-in with an address book application that associates and stores a user name and an email address, and the "Tom" and "Tom's mother" respectively set for the sender and the recipient are automatically converted by the email application into the email addresses associated and stored in the address book application.

[0090] The mode display bar 101, the sender selection bar 102, and the recipient input bar 103 are all configured to allow the user to change the set content. In this case, the mode display bar 101 and the sender selection bar 102 are configured such that the user uses the electronic pen P or a finger to select from the pre-set options. By changing the set content of the mode display bar 101, the user can change the mode of the email application to, for example, the keyboard input mode described above. On the other hand, the recipient input bar 103 is configured such that the user uses the virtual keyboard displayed by the processor 21 to input. It should be noted that the processor 21 preferably displays the virtual keyboard according to the user touching the recipient input bar 103 with the electronic pen P or a finger.

[0091] The carbon copy input bar 104 and the subject input bar 105 are respectively the bars for inputting the recipients of the carbon copy of the email and the subject of the email, and are both configured to allow the user to input. This input is performed by the user using the virtual keyboard in the same way as the recipient input bar 103.

[0092] The body input bar 106 is the bar for inputting the body of the email and is configured to allow the user to input. If the processor 21 sets the email application to the "handwriting input mode", the body input bar 106 becomes a state where input based on the electronic pen P can be achieved. In the case where the processor 21 sets the email application to the "keyboard input mode", the body input bar 106 becomes a state where input based on the keyboard can be achieved.

[0093] Return Figure 9 。In step S14a, the processor 21 that has set the sender and the recipient obtains the pen ID of Tom, the speaker of the input voice, by referring to the user table shown in Figure 4 (step S15a). Next, the processor 21 obtains the pen ID from the sensor 25 (step S15b). The pen ID obtained in step S15b is received by the sensor 25 from the electronic pen P approaching the touch surface as described above.

[0094] The processor 21 that has obtained the pen ID in step S15b determines whether the pen IDs obtained in steps S15a and S15b are the same (step S15c). And, in the case where it is determined that they are the same, acceptance of handwriting input is started (step S15d). After that, the processor 21 generates stroke data according to a series of coordinates sequentially supplied from the sensor 25 and sequentially displays it within the body input bar 106. On the other hand, in the case where it is determined in step S15c that they are not the same, wait until a new pen ID is supplied from the sensor 25. Thus, it is achieved that only the input of the electronic pen P of Tom, the speaker of the input voice, is allowed, and the input using other electronic pens P or fingers is not accepted.

[0095] As described above, in the system 1 according to the present embodiment, since the application is started in the handwriting input mode based on the speech recognition result, the handwriting input can be started by a voice command. For example, since the email application is started in the handwriting input mode based on the speech recognition result, the handwriting input of the email can be started by a voice command.

[0096] In addition, it is possible to allow only the input based on the electronic pen P corresponding to the speaker of the input voice, and not to accept the input using other electronic pens P or fingers.

[0097] In addition, based on the speech recognition result, data such as the sender / recipient of the email can be automatically set for the started application.

[0098] It should be noted that in the present embodiment, an example of starting an application in the handwriting input mode of the speaker is given, but it may also be that the AI assistant terminal 10 generates a command for starting an application in the handwriting input mode based on the speech recognition result, and the tablet terminal 20 starts the application in the handwriting input mode that can realize handwriting input even with an electronic pen different from the electronic pen corresponding to the speaker based on this command.

[0099] Figure 11 FIG. is a diagram showing the system 1 of the first modification of the present embodiment. As shown in this figure, the system 1 of this modification is configured to further include an AI server 30. In addition, the AI assistant terminal 10, the tablet terminal 20, and the AI server 30 are connected to each other via a network 40. The network 40 is, for example, the Internet. This modification is different from the present embodiment in that a part of the processing performed by the AI assistant terminal 10 in the present embodiment is executed in the AI server 30 with higher processing power. Hereinafter, the description will be centered on the differences from the present embodiment.

[0100] The AI server 30 is a server provided in, for example, a data center of a company that provides speech recognition services, as Figure 11 shown, having a structure in which a processor 31, a memory 32, and a communication unit 33 are connected to each other via an internal bus.

[0101] The processor 31 is a central processing device that controls each part of the AI server 30, and has a function of reading and executing a program stored in the memory 32. In the program executed by the processor 31, an AI engine that executes processing related to voice input such as speech recognition is included.

[0102] The memory 32 is a storage device configured to be able to store various programs and data. In a typical example, the memory 32 includes a main storage device such as a DDR4 SDRAM (Double Data Rate 4 Synchronous Dynamic Random Access Memory) and an auxiliary storage device such as a hard disk.

[0103] The communication unit 33 is a functional unit for communicating with other computers via a network such as the Internet or in a point-to-point manner. Typically, it is configured to be able to communicate according to the Ethernet (registered trademark) standard.

[0104] When the processor 11 of the AI assistant terminal 10 of this modification example is supplied with a digital signal representing speech from the microphone 15, it transmits this digital signal to the AI server 30 in real time. The processor 31 of the AI server 30 uses the digital signal thus transmitted in real time to execute Figure 5 the processes of steps S2 to S7 shown. Then, the command generated in step S7 is sent to the tablet terminal 20. The processing executed by the tablet terminal 20 that has received this command is the same as that of this embodiment.

[0105] According to this modification example, since the AI server 30 with high processing power can execute processing with a relatively large load such as speech recognition, the application can be started in the handwriting input mode at a higher speed.

[0106] Figure 12 FIG. shows the system 1 of the second modification example of this embodiment. The system 1 of this modification example is composed of one AI assistant terminal 10. If compared with Figure 2 it can be understood that: this AI assistant terminal 10 has a structure in which a sensor 17 is added to the AI assistant terminal 10 of this embodiment.

[0107] The sensor 17 is a position detection device similar to the Figure 2 sensor 25 shown, and is configured to be able to detect the positions of the electronic pen P and the finger on the touch surface. In a typical example, the touch surface is constituted by the display surface of the display 14.

[0108] After the processor 11 of this embodiment generates a command by performing the Figure 5 processes of steps S1 to S7 shown, it performs the Figure 8 processes of steps S11 to S15 shown on this command. Thus, the processing that is executed through the cooperation of the AI assistant terminal 10 and the tablet terminal 20 in this embodiment is executed by one AI assistant terminal 10 in this modification example.

[0109] According to this modification example, a voice command can be input to the microphone 15 of the AI assistant terminal 10 that supports touch input based on the electronic pen P, and handwriting input can be started using the display 14 of the AI assistant terminal 10.

[0110] It should be noted that, in this modification example, an example in which the system 1 is constituted by one AI assistant terminal 10 is described, but the system 1 can also be constituted by one tablet terminal 20. That is, by installing the functions of the AI assistant terminal 10 in the tablet terminal 20, a voice command can be input to the microphone of the tablet terminal 20, and handwriting input can be started using the display 24 of the tablet terminal 20.

[0111] Figure 13 FIG. shows the structure of the system 2 according to the second embodiment of the present invention. As shown in this figure, the system 2 is configured to include, for example, an AI assistant terminal 10, a tablet terminal 20, and an electronic pen P provided in a home, and a semantic / ink server 50 provided, for example, externally.

[0112] The structures of the AI assistant terminal 10 and the tablet terminal 20 are as described in the first embodiment. The semantic / ink server 50 is a server that performs semantic (meaning) analysis of the above-mentioned ink file. The AI assistant terminal 10, the tablet terminal 20, and the semantic / ink server 50 are connected to each other via a network 40. The network 40 is, for example, the Internet.

[0113] Figure 14 FIG. is a schematic block diagram showing the hardware structures of the AI assistant terminal 10, the tablet terminal 20, and the semantic / ink server 50, respectively. Among them, regarding the hardware structures of the AI assistant terminal 10 and the tablet terminal 20, they are the same as the hardware structures described with reference to Figure 2 and explained.

[0114] Figure 15 FIG. is a diagram showing an ink file database stored in the memory 12 of the AI assistant terminal 10. As shown in this figure, the ink file database is configured to store by establishing correspondence among an ink file, semantic metadata, and a pen ID. The semantic metadata is data obtained as a result of semantic analysis of the ink file, and includes purpose data indicating the purpose of the ink file. Regarding the generation of the semantic metadata, it will be described in detail later with reference to Figures 19 - 21 and explained.

[0115] Figure 16 (a) and (b) are diagrams showing tables stored in the memory 12 of the AI assistant terminal 10, respectively. Figure 16 (a) is Figure 3 (a) shown again. In Figure 16 (b), a prompt content table that establishes correspondence between voice information and the types of ink files to be prompted is shown.

[0116] The processor 11 of the AI assistant terminal 10 is configured to, when presenting ink files according to the table in Figure 16 (a), determine the type of ink file to be presented by referring to Figure 16 (b), and present the determined type of ink file. For example, when the string obtained as a result of speech recognition processing contains the string "Shopping List", the processor 11 determines to present an ink file with the target data being "Shopping List".

[0117] In addition, in the present embodiment, Figure 4 the user table shown is also stored in the memory 12 of the AI assistant terminal 10. The processor 11 of the AI assistant terminal 10 refers to this user table in step S22 described later Figure 23 . Details will be described later.

[0118] Return Figure 14 . As Figure 14 shown, the semantic ink server 50 has a structure in which a processor 51, a memory 52, and a communication unit 53 are interconnected via an internal bus.

[0119] The processor 51 is a central processing device that controls each part of the semantic ink server 50 and has a function of reading and executing programs stored in the memory 32. Among the programs executed by the processor 51, there are a character recognition process for recognizing characters based on a graphic composed of a plurality of stroke data in an ink file and an ink engine for performing semantic analysis of analyzing the meaning represented by the recognized characters.

[0120] The memory 52 is a storage device configured to be able to store various programs and data. In a typical example, the memory 52 includes a main storage device such as the above-mentioned DDR4 SDRAM and an auxiliary storage device such as a hard disk.

[0121] The communication unit 53 is a functional unit for communicating with other computers via a network such as the Internet or in a point-to-point manner. Typically, it is configured to be able to communicate according to the Ethernet (registered trademark) standard.

[0122] Figure 17 is a processing flow chart showing the processing executed by the processor 21 of the tablet terminal 20 in the present embodiment. As shown in this figure, the processor 21 first obtains a pen ID from the sensor 25 (step S20). This processing is the same as the processing in Figure 9 step S15b. Next, the processor 21 accepts a handwritten input (step S21). Specifically, as described above, one or more stroke data respectively including a series of coordinates and pen pressure values sequentially supplied from the sensor 25 are generated.

[0123] Next, the processor 21 generates an ink file containing one or more stroke data generated in step S20 (step S22). The ink file generated here typically contains one or more stroke data input through the handwritten input started in step S15 shown in Figure 8 However, it may also contain one or more stroke data input through handwritten input started in other processes. After that, the processor 21 associates the generated ink file with the pen ID obtained in step S20 and sends it to the semantic ink server 50 (step S23).

[0124] Figure 18 FIG. is an example showing a series of stroke data generated in step S21. In this example, within the display surface of the tablet terminal 20, the strings “-milk”, “-bread”, “-eggs”, and “-apples” are handwritten using the electronic pen P. The processor 21 saves the multiple stroke data constituting these strings in one ink file, associates it with the pen ID of the electronic pen P, and sends it to the semantic ink server 50.

[0125] Figure 19 FIG. is a process flow chart showing the process executed by the processor 51 of the semantic ink server 50. As shown in this figure, the processor 51 first obtains the ink file and the pen ID sent from the tablet terminal 20 (step S30). Then, it groups the multiple stroke data in the ink file by performing character recognition (step S31).

[0126] Figure 20 (a) FIG. shows the result of the grouping in step S31 for the ink file of the example shown in Figure 18 In addition, Figure 21 (a) FIG. shows the result of the grouping in step S31 for the ink file of another example. As shown in these figures, the processor 51 first generates a group g0 representing the whole ink file, and then generates a subgroup g1 in the group g0 by extracting the area where there are characters. Next, the processor 51 recognizes the symbol “-” and classifies each line starting with “-” into one grandchild group. In the examples of Figure 20 (a) and Figure 21 (a), as a result of this classification, four grandchild groups g2 to g5 are generated in the subgroup g1. It should be noted that here, it is assumed that the reference symbol is “-”, but of course, grandchild groups can also be generated based on other symbols. Regardless of which symbol is used, the user needs to know in advance the symbol to be referred to for grouping. By doing so, the user can generate groups purposefully.

[0127] Return Figure 19The grouped processors 51 then generate semantic metadata including purpose data representing the purpose of the ink file by performing semantic analysis on the grouped stroke data (step S32).

[0128] Figure 20 (b) is a diagram showing the semantic metadata generated according to Figure 20 each group shown in (a). Figure 21 (b) is a diagram showing the semantic metadata generated according to Figure 21 each group shown in (a). As shown in these diagrams, the processor 51 first sets the parent-child relationships of groups g0 to g5. Then, since there are 4 groups g2 to g5 classified by the identification symbol "-" in group g1, the meaning of "list" is associated with group g1, and the meaning of "list item" is associated with each of groups g2 to g5. Moreover, by performing character recognition processing and semantic analysis on one or more stroke data belonging to each of groups g2 to g5, in Figure 20 the example of (b), the meaning data representing the meaning of one or more stroke data belonging to each of groups g2 to g5, namely the words "milk", "bread", "eggs", "apples", are associated with each of groups g2 to g5. In Figure 21 the example of (b), the words "Provide an approval document", "Tel Company A", "Interview B", "Purchase summer gift" are associated with each of groups g2 to g5. It should be noted that, in terms of the relationship in the paper space, part of the description is omitted in Figure 21 (b).

[0129] The processor 51 further determines the purpose data representing the purpose of the list by analyzing the content of the words thus associated with each of groups g2 to g5, and associates it with group g1. In Figure 20 the example of (b), according to the content of the words "milk", "bread", "eggs", "apples", "shopping list" is determined as the purpose data and associated with group g1. In Figure 21 the example of (b), according to the content of the words "Provide an approval document", "Tel Company A", "Interview B", "Purchase summer gift", "ToDo list" is determined as the purpose data and associated with group g1.

[0130] Return Figure 19 。The processor 51 that has generated the semantic metadata as described above generates data including the ink file and pen ID obtained in step S30 and the semantic metadata generated in step S32 (step S33), and sends it to the AI assistant terminal 10 (step S34).

[0131] Figure 22 is a processing flow chart showing the processing executed by the processor 11 of the AI assistant terminal 10 of the present embodiment. If the AI assistant terminal 10 receives the data sent by the semantic·ink server 50 in Figure 19 step S34 of (step S40), the received data is saved in the Figure 15 shown ink file database (step S41). Thus, as will be described below, the AI assistant terminal 10 can be prompted to display an ink file by a voice command.

[0132] Figure 23 is a processing flow chart showing other processing executed by the processor 11 of the AI assistant terminal 10 of the present embodiment. In this figure, a process for prompting the ink file saved in the ink file database shown in Figure 21 according to a voice command is shown. Hereinafter, this process will be described in detail.

[0133] The processor 11 first executes steps S1 and S2 with reference to Figure 5 the description.

[0134] Figure 24 is a diagram showing a specific example of the processing of the AI assistant terminal 10 for explaining Figure 23 shown. As shown in this figure, in this example, the user issues the voice “What is on my shopping list?”. The processor 11 performs a prescribed recognition process using the above-described AI engine, transforms the above voice collected by the microphone 15 into the string “What is on my shopping list?”, and determines the speaker of the voice, “Tom” (step S2).

[0135] Return Figure 23 。Next, the processor 11 determines the prompting of the ink file based on the information obtained in step S2 (step S50). If it is Figure 24 the example of, since the string “What is on my*?” is included in the string obtained in step S2, the processor 11 refers to Figure 16 (a) the second row of the action content table shown. Then, the prompting of the ink file is determined according to the description content of the second row.

[0136] Next, the processor 11 determines the type of ink file to be prompted based on the information obtained in step S2 (step S51). In Figure 24 the example, by including the string "shopping list" in the string obtained in step S2, the processor 11 refers to Figure 16 the first row of the prompt content table shown in (b). Then, the prompt of the ink file as a shopping list is determined according to the description content of the first row.

[0137] Next, the processor 11 selects a pen ID based on the information obtained in step S2 (step S52). This selection is performed by referring to Figure 4 the user table shown. In the example of Figure 24 , the pen ID = 0001 of the speaker Tom determined in step S2 is selected.

[0138] Next, based on the pen ID selected in step S52, the processor 11 selects one or more data from the Figure 15 ink file database shown (step S53). That is, the data associated with the pen ID selected in step S52 and saved among the multiple data saved in the ink file database is selected. If, for example, in the example of Figure 15 , if the pen ID selected in step S52 is "0001", the data related to ink file 01 and the data related to ink file 03 are selected.

[0139] The processor 11 further selects the data of the type determined in step S51 by referring to the semantic metadata of each data selected in step S53 (step S54). That is, the data associated with the target data corresponding to the type determined in step S51 and saved in the ink file database among each data selected in step S53 is selected. In the example of Figure 24 , as described above, the type of the ink file determined in step S51 becomes a shopping list. Therefore, the processor 11 selects the data whose target data in the semantic metadata is "shopping list" from the data selected in step S53.

[0140] Finally, the processor 11 presents the data selected in step S54 to the user together with the target data (step S55). The information presented here can be the ink file itself, that is, one or more stroke data, or the semantic metadata, that is, the target data or meaning data of the stroke data. In addition, as a presentation method, it can be displayed on the Figure 14 display 14 shown, or the semantic metadata can be output as speech from the Figure 14 speaker 16 shown using speech reading software.

[0141] Figure 25 relates to Figure 20 The figure shown in Figure 20 is a diagram illustrating an example in which an ink file is displayed on the display 14. Figure 26 relates to Figure 20 The figure shown in Figure 20 is a diagram illustrating an example in which an alternative list formed based on semantic metadata is displayed on the display 14. According to Figure 25 the example of Figure 25 , since the content written by the user himself / herself is displayed, the writing content of the user can be reliably presented to the user. On the other hand, according to Figure 26 the example of Figure 26 , since the list is displayed in typeface, the ink file can be presented in a state that is easy for the user to read. It should be noted that the term "shopping list" shown at the upper part of each figure represents the target data presented together with the data selected in step S54.

[0142] As described above, according to the system 2 of the present embodiment, the purpose of the ink file automatically assigned through semantic analysis ("shopping list", "ToDo list", etc.) can be used as a search keyword to search for the ink file generated by handwritten input, so that the ink file can be easily retrieved.

[0143] In addition, according to the system 2 of the present embodiment, the ink file can be retrieved by stating the purpose of the ink file ("shopping list", "ToDo list", etc.).

[0144] Figure 27 The figure shows the system 2 of a modified example of the present embodiment. As shown in this figure, the system 1 of this modified example is configured to further include a cloud server 60. This modified example is different from the present embodiment in that Figure 15 the ink file database shown in Figure 15 is constructed in the cloud server 60 instead of in the AI assistant terminal 10. Hereinafter, the description will focus on the differences from the present embodiment.

[0145] The semantic & ink server 50 of this modified example replaces Figure 19 step S34 of Figure 19 and executes the process of sending the data generated in step S33 to the cloud server 60 (step S100). The cloud server 60 that receives the data sent in this way executes the same process as the process of the AI assistant terminal 10 shown in Figure 22 Figure 22 internally. Thereby, Figure 15 the ink file database shown in Figure 15 is constructed in the cloud server 60.

[0146] The AI assistant terminal 10 of this modified example replaces Figure 23The commands (e.g., SQL commands) including the pen ID selected in step S53 and the type of the ink file determined in step S51 shown in the figure are sent to the cloud server 60 according to steps S53 and S54 (step S101). The cloud server 60 that receives this command selects one or more pieces of data from the ink file database based on the received pen ID, and further selects the data of the received type from among them by referring to the semantic metadata of each selected piece of data. Then, the finally selected data is sent back to the AI assistant terminal 10 (step S102). The AI assistant terminal 10 that receives the data sent back in this way executes Figure 24 the process of step S55 shown in the figure.

[0147] According to this modification example, since the ink file database is constructed in the cloud server 60, it is possible to accumulate the ink files generated by each of a plurality of tablet terminals 20 owned by, for example, a certain company in one ink file database, and retrieve the plurality of ink files stored in this ink file database from a plurality of AI assistant terminals 10 owned by the same company.

[0148] As described above, although the preferred embodiments of the present invention have been described, the present invention is in no way limited to such embodiments, and the present invention can of course be implemented in various ways without departing from its gist.

[0149] For example, the strings ("Write E-mail", etc.) used in the above-described embodiments are merely examples, and of course, other strings can also be used.

[0150] In addition, in the above-described embodiments, the processor 11 of the AI assistant terminal 10 determines the content of the action to be started, etc. by comparing the string obtained through the speech recognition process with the string described in Figure 3 (a) to (c) or Figure 16 (a)(b) in the "voice information" column, but in the speech recognition process, it is also possible to determine the content of the action to be started, etc. by comparing the input voice with the voice obtained by speaking the string described in Figure 3 (a) to (c) or Figure 16 (a)(b) in the "voice information" column based on the voice.

[0151] The present invention can also be configured as follows. Thereby, it is possible to provide a system capable of realizing handwritten input using an AI assistant.

[0152] Solution 1]

[0153] A system,

[0154] generates a command for starting an application in a handwritten input mode by recognizing the voice input through a microphone,

[0155] Start the application in the handwriting input mode based on the generated command.

[0156] 2. The system according to Solution 1,

[0157] Determine the speaker of the voice by recognizing the voice,

[0158] In the handwriting input mode, it is possible to realize the input using an electronic pen corresponding to the determined speaker.

[0159] 3. The system according to Solution 1,

[0160] The application is a messaging application configured to be able to send the handwritten data input by the electronic pen.

[0161] 4. The system according to Solution 3,

[0162] Determine the speaker of the voice by recognizing the voice,

[0163] Set the speaker as the sender of the handwritten data.

[0164] 5. The system according to Solution 3,

[0165] Determine the recipient of the handwritten data by recognizing the voice,

[0166] Set the recipient as the receiving address of the handwritten data.

[0167] 6. The system according to Solution 1,

[0168] Have a first table that associates the voice information corresponding to the result of the voice recognition process with the application to be started,

[0169] Determine the application to be started based on the information obtained by recognizing the voice and the voice information stored in the first table.

[0170] 7. The system according to Solution 6,

[0171] Have a second table that associates the voice information corresponding to the result of the voice recognition process with the start mode of the application to be started,

[0172] Determine the start mode of the application to be started based on the information obtained by recognizing the voice and the voice information stored in the second table.

[0173] 8. The system according to Solution 7,

[0174] There is a third table that correlates speech information corresponding to the result of the recognition process of speech with setting data in the case of starting the application of the start object.

[0175] Based on the information obtained by recognizing the speech or the speaker of the speech obtained by recognizing the speech and the speech information stored in the third table, determine the data set for the application of the start object.

[0176] 9. The system according to Solution 2,

[0177] There is a fourth table that correlates the user name with the pen ID.

[0178] In the handwriting input mode, it is possible to realize input using an electronic pen with a pen ID that corresponds to the determined speaker and is stored in the fourth table.

[0179] 10. The system according to Solution 9,

[0180] There is a sensor that detects the position of the electronic pen and receives the pen ID from the electronic pen.

[0181] In the handwriting input mode, it is possible to realize input using an electronic pen whose pen ID corresponding to the determined speaker and stored in the fourth table is the same as the pen ID received by the sensor.

[0182] Reference Numeral Explanation

[0183] 1, 2 System

[0184] 10 AI Assistant Terminal

[0185] 11, 21, 31, 51 Processor

[0186] 12, 22, 32, 52 Memory

[0187] 13, 23, 33, 53 Communication Unit

[0188] 14, 24 Display

[0189] 15 Microphone

[0190] 16 Speaker

[0191] 17, 25 Sensor

[0192] 20 Tablet Terminal

[0193] 30 AI Server

[0194] 40 Network

[0195] 50 Semantic · Ink Server

[0196] 60 Cloud server

[0197] 100 Screens

[0198] 101 Mode display bar

[0199] 102 Sender selection bar

[0200] 103 Recipient input bar

[0201] 104 CC input bar

[0202] 105 Subject input bar

[0203] 106 Body input bar

[0204] Groups g0 - g5

Claims

1. A system for processing stroke data, configured to allow storage and retrieval of handwritten inputs, comprising: a processor, a sensor configured to receive touch inputs; and a non-transitory memory including instructions that, when executed by the processor, cause the processor to: generate stroke data in response to a handwritten input received via the sensor, generate semantic metadata including meaning data representing the meaning of the stroke data and purpose data determined based on the meaning data, where the purpose data represents the user purpose of the stroke data, by performing semantic analysis on the stroke data, store the stroke data, the semantic metadata, and a pen ID representing an electronic pen used to generate the stroke data, associated with each other, and retrieve the stroke data and the semantic metadata corresponding to the pen ID in response to a user prompt representing the pen ID.

2. The system according to claim 1, further comprising: a microphone, wherein the processor selects the stroke data or the meaning data associated with the purpose data corresponding to the recognized speech by recognizing the speech input through the microphone.

3. The system according to claim 1, further comprising: a microphone, wherein the processor selects the stroke data or the meaning data associated with the purpose data corresponding to the recognized speech by recognizing the speech input through the microphone, and the system further comprises a speaker configured to output the selected stroke data or meaning data as speech.

4. The system according to claim 1, comprising: a microphone, wherein the processor obtains the pen ID assigned to the electronic pen of the speaker who spoke the speech by recognizing the speech input through the microphone, and selects the stroke data or the meaning data associated with the obtained pen ID, and the system is configured to display the selected stroke data or meaning data on a display.

5. The system according to claim 4, wherein, the processor obtains the pen ID assigned to the electronic pen of the speaker by obtaining the pen ID stored in association with the user ID of the speaker in a user table that stores the user name and the pen ID associated with each other.

6. The system according to claim 1, wherein, the processor groups the stroke data and performs semantic analysis on the grouped stroke data.

7. A method for processing stroke data, comprising: generating stroke data in response to a handwritten input received via a sensor, generating semantic metadata including meaning data representing the meaning of the generated stroke data and purpose data determined based on the meaning data, where the purpose data represents the user purpose of the stroke data, by performing semantic analysis on the generated stroke data, storing the stroke data, the semantic metadata, and a pen ID representing an electronic pen used to generate the stroke data, associated with each other, in a non-transitory memory, and retrieving the stroke data and the semantic metadata corresponding to the pen ID in response to a user prompt representing the pen ID.

8. The method for processing stroke data according to claim 7 further comprises: identifying speech input through a microphone, and displaying the stroke data or the meaning data associated with the target data corresponding to the recognized speech on a display.

9. The method for processing stroke data according to claim 7 further comprises: identifying speech input through a microphone, and outputting, as speech from a speaker, the stroke data or the meaning data associated with the target data corresponding to the recognized speech.

10. The method for processing stroke data according to claim 7 further comprises: obtaining the pen ID assigned to the electronic pen of the speaker who uttered the speech by identifying the speech input through a microphone, and displaying the stroke data or the meaning data associated with the obtained pen ID on a display.

11. The method for processing stroke data according to claim 10 further comprises: obtaining the pen ID assigned to the electronic pen of the speaker by obtaining the pen ID stored in association with the user ID of the speaker in a user table that stores a user name and a pen ID in association with each other.

12. The method for processing stroke data according to claim 7 further comprises: performing the semantic analysis on the grouped stroke data.

Images