Data processing device, data processing method, and data processing program

Abstract

To provide a data processing device, a data processing method, and a data processing program capable of supporting communication between an ALS sufferer and others.SOLUTION: A data processing device includes an input unit 292 for acquiring user data, a processing unit 294 for performing specific processing using a data generation model for generating a prescribed inference result corresponding to the user data, and an output unit 296 for outputting a result of the specific processing to a prescribed device. The input unit 292 acquires character data showing characters inputted by using a user's visual line as the user data from the device, and the processing unit 294 performs processing for deriving sentence data relating to characters shown by the character data as the specific processing by using an output of the data generation model when the character data is inputted.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The technology of the present disclosure relates to a data processing apparatus, a data processing method, and a data processing program.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, there is room for improvement in assisting communication between ALS (Amyotrophic Lateral Sclerosis) patients and others. Hereinafter, ALS will be referred to as such.

Means for Solving the Problems

[0005] A first aspect of the technology according to the present disclosure includes an input unit that acquires user data, a processing unit that performs a specific process using a data generation model that generates a predetermined inference result according to the user data, and an output unit that outputs the result of the specific process to a predetermined device. The input unit acquires, as the user data from the device, character data indicating characters input using the user's line of sight. The processing unit performs, as the specific process, a process of deriving sentence data related to the characters indicated by the character data using the output of the data generation model when the character data is input. This is a data processing device.

[0006] A second aspect of the technology according to the present disclosure is a data processing method that acquires user data, performs a specific process using a data generation model that generates a predetermined inference result according to the user data, and outputs the result of the specific process to a predetermined device. The method acquires, as the user data from the device, character data indicating characters input using the user's line of sight, and performs, as the specific process, a process of deriving sentence data related to the characters indicated by the character data using the output of the data generation model when the character data is input.

[0007] A third aspect of the technology according to the present disclosure is a process that acquires user data, performs a specific process using a data generation model that generates a predetermined inference result according to the user data, and outputs the result of the specific process to a predetermined device. The process acquires, as the user data from the device, character data indicating characters input using the user's line of sight, and performs, as the specific process, a process of deriving sentence data related to the characters indicated by the character data using the output of the data generation model when the character data is input. This is a data processing program that causes a computer to execute the process.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 3C

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0009] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a data processing program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit), etc.

[0012] In the following embodiments, the labeled RAM (Random Access Memory) is a memory where information is temporarily stored and is used as a work memory by the processor.

[0013] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0014] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F manages communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] FIG. 1 shows an example of the configuration of a data processing system 10 according to an embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a user terminal 14. An example of the data processing device 12 is a server. An example of the user terminal 14 is a personal computer. In this embodiment, the data processing device 12 is an example of the "data processing device" according to the technology of the present disclosure, and the user terminal 14 is an example of the "device" according to the technology of the present disclosure.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of the "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The user terminal 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. Also, the reception device 38, the output device 40, and the camera 42 are connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A, a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by contact of an indicator (e.g., a pen or a finger, etc.) by detecting the contact. The microphone 38B receives user input by voice by detecting the user's voice. A control unit 46A, which will be described later, transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, a specific processing unit 290, which will be described later, acquires data indicating the user input.

[0021] The output device 40 includes a display 40A, a speaker 40B, etc., and presents data to the user by outputting the data in a form that can be perceived by the user (e.g., voice and / or text). The display 40A displays visible information such as text and images according to an instruction from the processor 46. The speaker 40B outputs voice according to an instruction from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, a diaphragm, and a shutter, and an imaging device such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / F 44 and 26 manage the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the user terminal 14.

[0024] As shown in FIG. 2, in the data processing device 12, specific processing is performed by the processor 28. The storage 32 stores a specific processing program 56. The specific processing program 56 is an example of the "data processing program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58. The data generation model 58 is used by the specific processing unit 290.

[0026] In the user terminal 14, input / output processing is performed by the processor 46. The storage 50 stores an input / output program 62. The input / output program 62 is used in combination with the specific processing program 56 by the data processing system 10. The processor 46 reads the input / output program 62 from the storage 50 and executes the read input / output program 62 on the RAM 48. The input / output processing is realized by the processor 46 operating as a control unit 46A according to the input / output program 62 executed on the RAM 48.

[0027] Next, the processing of the specific processing unit 290 when the data processing device 12 performs specific processing to assist communication between ALS parties and others will be described.

[0028] ALS patients often use a communication device that can be operated by eye movements when they are unable to speak on their own. However, when the disease progresses extremely, it takes a lot of time to operate, and it is difficult to convey things other than "yes / no" or fixed phrases. For this reason, it becomes difficult to enjoy daily conversations and have exchanges using the patient's own experiences and skills.

[0029] FIG. 3A shows an example of a display screen of a communication support device (hereinafter also referred to as a "user") for assisting communication between others and a conventional ALS user (hereinafter also referred to as a "user"). In FIG. 3A, as an example of the communication support screen, menu items selectable by the user's line of sight are displayed on the left side, and selection buttons that are gazed at by the user when the user makes a selection designation of a menu item are displayed on the right side.

[0030] In the conventional communication support screen shown in FIG. 3A, a plurality of menu items such as a "yes / no" item, a "write text" item, and a "fixed phrase" item are displayed as menu items selectable by the user's line of sight. And in this communication support screen, the display state of any one of the plurality of menu items is sequentially and repeatedly switched at predetermined time intervals (for example, 2 seconds) and in the arrangement direction of the menu items (in the example shown in FIG. 3A, the direction from top to bottom) so as to be different from that of the other menu items. In the example shown in FIG. 3A, the menu item with a different display state is displayed within the display area of the selection button.

[0031] In this communication support screen, using the detection result of the user's line of sight, it is specified that the menu item with a different display state when the line of sight gazes at the selection button for a predetermined period (for example, 0.5 seconds) or more is the menu item selected by the user.

[0032] In such a conventional communication support screen, there are problems such as a large burden on the user's eyes, difficulty in typing long sentences, being prone to only requests using fixed phrases or "yes / no" responses and having no room for daily conversations.

[0033] Therefore, in the data processing system 10 according to the present embodiment, using the detection result of the user's line of sight, sentence data indicating a response sentence to an utterance from another person is derived using the data generation model 58.

[0034] That is, in the data processing system 10 according to the present embodiment, as shown in FIG. 3B as an example, when a message arrives from another person (a conversation partner, hereinafter referred to as an "interlocutor") 82 such as a family member or a friend, the user 80 uses the display 40A of the user terminal 14 to display the message.

[0035] The data generation model 58 according to the present embodiment is pre-trained by machine learning to generate the sentence data by inputting character data (hereinafter simply referred to as "character data") in the range from one character to several characters (in this embodiment, in the range from one character to five characters), which is the key to the sentence data to be generated, and directionality data (hereinafter simply referred to as "directionality data") indicating the directionality of the sentence data to be generated.

[0036] Note that the character data is data indicating the content of the sentence data (hereinafter simply referred to as "sentence data") that the user 80 wants to generate (intends), or a representative word for what the user felt about the utterance by the interlocutor 82, or a part of the word, or a character string including the word. Further, the directionality data is data indicating the directionality of the content of the user 80's own response to the utterance by the interlocutor 82, that is, the directionality of the content of the sentence data, and is data selectively applied from a plurality of predetermined types of directionality such as affirmative, negative, interrogative, encouraging, gentle, emotional, etc.

[0037] When the above message is displayed on the display 40A of the user terminal 14, the user 80 selectively inputs, by sight line, character data corresponding to a reply sentence to the message and directionality data indicating the directionality of the reply sentence. The character data and the directionality data input by the user 80 are input to the data generation model 58, whereby sentence data that is a candidate for the reply sentence corresponding to the character data and the directionality data is generated, and candidates for the reply sentence indicated by the sentence data are displayed on the display 40A of the user terminal 14.

[0038] When candidates for the above reply sentence are displayed on the display 40A of the user terminal 14, if the content is as intended, or is close to what is intended and is to be adopted as the reply sentence, the user 80 selects the reply sentence by sight line. When the reply sentence is selected by the user, the reply sentence is displayed by the device used by the interlocutor 82.

[0039] In the data processing system 10 according to the present embodiment, the sight line of the user 80 is obtained by using a face image obtained by the camera 42 to photograph the face of the user 80 in real time, and is performed using a conventionally known eye tracking technique, but is not limited thereto. For example, it may be a form in which the user 80 wears smart glasses and the sight line of the user 80 is detected using the smart glasses.

[0040] FIG. 3C shows a specific application example of the data processing system 10 according to the present embodiment. In the example shown in FIG. 3C, a case is illustrated in which the interlocutor 82 is making a statement to the user 80 regarding the book that the interlocutor 82 has read.

[0041] As shown in FIG. 3C, in this application example, first, the interlocutor 82 makes a statement such as "I read a book called ABC yesterday. It was really interesting!". As a result, data indicating the content of the statement is input to the user terminal 14, and the content of the statement is displayed on the display 40A of the user terminal 14.

[0042] On the other hand, when the user 80 thinks, "What was interesting about the content of that book?", the user terminal 14 inputs, as an example, four-character text data "not like this" based on the user's line of sight, and selects direction data for "interrogative form".

[0043] When the user 80 finishes inputting the text data and selecting the direction data, these data are transmitted to the data processing device 12 and input into the data generation model 58. As a result, text data such as "What kind of story was 'ABC'?", "What was so interesting? Can you introduce it to me?" are generated as selection candidates for the response sentence, and the sentences indicated by these selection candidate text data are displayed on the user terminal 14.

[0044] Therefore, the user 80 selects the text data to be adopted using their line of sight. When the text data is selected by the user 80, the user terminal 14 reproduces, as an example, the sentence indicated by the text data by voice. When the reproduction is performed, the interlocutor 82 makes responses such as "Regarding the content of 'ABC'...", "The interesting parts are...".

[0045] By repeating the above exchanges, communication between the user 80 and the interlocutor 82 is carried out. At this time, the user 80 can respond according to their intention only by selecting, using their line of sight, characters of about several characters, direction, and the sentence to actually respond from the selection candidates of the response sentences generated by the data generation model 58. As a result, the user 80 can communicate with the interlocutor 82 in a less stressful state.

[0046] As shown in FIG. 4, the specific processing unit 290 according to this embodiment includes an input unit 292, a processing unit 294, and an output unit 296.

[0047] The input unit 292 acquires user data received by the user terminal 14. Specifically, as user data from the user terminal 14, it acquires text data indicating characters input using the line of sight of the user 80.

[0048] The processing unit 294 performs a specific process using the data generation model 58. Specifically, the processing unit 294 performs, as the specific process, a process of deriving sentence data related to the character indicated by the character data by using the output of the data generation model 58 when the character data acquired by the input unit 292 is input.

[0049] The output unit 296 transmits the result of the specific process to the user terminal 14. In the user terminal 14, the control unit 46A causes the output device 40 to output the result of the specific process.

[0050] Here, as described above, the input unit 292 according to the present embodiment further acquires, as user data from the user terminal 14, directionality data indicating the directionality of the sentence data, and the processing unit 294 performs, as the specific process, a process of deriving candidates for the sentence data by using the output of the data generation model 58 when the character data and the directionality data are input.

[0051] Note that, in the present embodiment, as the directionality data, data indicating each directionality of a positive directionality, a negative directionality, an interrogative directionality, an encouraging directionality, a directionality related to kindness, and a directionality related to emotion is applied, but the present invention is not limited to this. For example, it may be a form in which data of any one or a plurality of combinations excluding all of these directionality is applied as the directionality data.

[0052] Further, in the present embodiment, as the sentence data, data in which the relationship between the user 80 and the characters appearing in the sentence data is reflected is applied, and the relationship includes each relationship of family, friend, senior, junior, and lover. However, the present invention is not limited to this form. For example, it may be a form in which any one or a plurality of combinations excluding all of these relationships are included in the above relationship.

[0053] In addition, in the present embodiment, as the sentence data, data reflecting past sentences created by the user 80 is applied, and the past sentences include both sentences created by the user 80 during e-mail exchanges and sentences sent by the user 80 on a social networking service (hereinafter referred to as "SNS"). However, it is not limited to this form. For example, it may be in a form where any one of these sentences is included in the above-mentioned past sentences.

[0054] In addition, in the present embodiment, as the sentence data, data reflecting the tone of the user 80 is applied, and the tone of the user 80 includes both the tone of written language and the tone of spoken language. However, it is not limited to this form. For example, it may be in a form where any one of these tones is included in the tone of the user 80. Here, examples of the types of tones include tones representing the characteristics of word endings such as the usage of honorifics and polite language, tones representing grammatical characteristics such as the usage of punctuation marks, quotation marks, parentheses, etc., tones representing the characteristics of expressions, and tones representing an individual's personality.

[0055] Furthermore, the processing unit 294 according to the present embodiment derives a plurality of sentence data related to the above-described character data as selection candidates as the sentence data, the output unit 296 outputs the selection candidates to the user terminal 14, and the input unit 292 further obtains a selection instruction from the selection candidates by the user 80.

[0056] Thus, in the present embodiment, the user terminal 14 is applied as the device in the technology of the present disclosure, but it is not limited thereto. For example, it may be in a form where smart glasses capable of detecting the line of sight of the user 80 are applied as the device in the technology of the present disclosure. In this form, various information can also be displayed by the smart glasses, so the convenience for the user 80 can be further improved.

[0057] The data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of the data generation model 58 include generative AIs such as ChatGPT (Internet search <URL: https: / / openai.com / blog / chatgpt>). The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image (in this embodiment, text data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data (in this embodiment, text data). Here, inference refers to, for example, analysis, classification, prediction, and / or summarization, etc.

[0058] As an example, as shown in FIG. 5 in this embodiment, by decomposing the prompt into three phases, the uniqueness of the user 80 himself / herself is reflected in the response sentence generated by the data generation model 58, and as a result, the accuracy of the response sentence is improved.

[0059] That is, the first-phase is the analysis phase, which predicts the genre of the speech from the conversation history up to the previous moment, determines whether a search is necessary and generates a search word, determines whether a pre-registered character appears in the topic, etc.

[0060] Also, the second-phase is the search phase, which uses the search word generated in the analysis phase to perform vector search on the e-mails previously described by the parties of the ALS, SNS, sentences in documents, etc., to obtain information necessary for generating the response sentence.

[0061] And the third-phase is the candidate generation phase, which generates candidates for the response sentence from the character data and direction data of about several characters input by the parties by sight, referring to the results of the analysis phase and the search phase.

[0062] Next, the operation of the data processing system 10 will be described.

[0063] An example of the flow of a specific process will be described with reference to FIG. 6. Note that the flow of the specific process shown in FIG. 6 is an example of the "data processing method" according to the technology of the present disclosure, and is repeatedly executed by the processor 28 of the data processing apparatus 12. Hereinafter, a case where the interlocutor 82 uses a user terminal (not shown, hereinafter referred to as the "interlocutor terminal") such as a smartphone held by himself / herself to initiate a conversation with the user terminal 14 used by the ALS party will be described, but the present invention is not limited thereto. For example, the interlocutor 82 may directly talk to the ALS party to initiate a conversation. In this case, a form in which text data indicating the content spoken by the interlocutor 82 is derived and applied by the data processing apparatus 12 or the user terminal 14 using a conventionally known speech recognition technology, the data generation model 58, or the like is exemplified.

[0064] In step S300 of FIG. 6, the processor 28 determines whether the user terminal 14 has received information indicating an initiation of a conversation from the interlocutor terminal (information indicating the content spoken by the interlocutor first), thereby determining whether there has been an initiation of a conversation from the interlocutor 82 to the user 80. If the determination is negative, the process proceeds to step S322, while if the determination is positive, the process proceeds to step S302.

[0065] In step S302, the processor 28 controls the user terminal 14 to display, on the display 40A, a message transmission support screen having a predetermined configuration, using the information indicating the initiation of the conversation received by the user terminal 14 from the interlocutor terminal. FIG. 7 shows an example of the message transmission support screen displayed on the display 40A of the user terminal 14 by the process of step S302.

[0066] As shown in FIG. 7, on this communication support screen, a dialogue display area 70A, which is an area for displaying the content of the conversation between the interlocutor 82 and the user 80, is displayed with a spoken sentence 70A1 indicating the content first spoken by the interlocutor 82. Also, on this communication support screen, a menu display area 70B, which is an area for displaying various menu items that can be selectively specified by the line of sight of the user 80, is provided.

[0067] Note that also on this communication support screen, similar to the conventional one shown in FIG. 3A, the display state of any one of a plurality of menu items is different from that of other menu items (in the example shown in FIG. 7, a shaded state), and it is sequentially and repeatedly switched at predetermined time intervals (for example, 2 seconds) and in the arrangement direction of the menu items (in the example shown in FIG. 7, the direction from top to bottom).

[0068] Also, on this communication support screen, when the user 80 designates any one of the menu items displayed in the menu display area 70B, a selection button 70C that is gazed at by the user 80 is displayed, and a character data display area 70D, which is an area for displaying character data input by the line of sight of the user 80, is displayed. Note that in the present embodiment, information indicating a menu item having a display state different from that of other menu items is displayed in the display area of the selection button 70C. And in the present embodiment, the user 80 gazes at the selection button 70C at the timing when information indicating a desired menu item is displayed on the selection button 70C, thereby specifying the desired menu item, but it is not limited to this. For example, it may be a form in which the desired menu item is specified by gazing at the display area itself of the desired menu item for a predetermined period (for example, 3 seconds).

[0069] On the communication support screen shown in Fig. 7, a question sentence "Recently, I've been very busy with caregiving work... I'd like to consult with you. Would that be okay?" is displayed as the addressed sentence 70A1. Also, as shown in Fig. 7, on this communication support screen, for the menu display area 70B, an "OK / Cancel" item designated when "OK" or "Cancel" is specified, a "Write Characters" item designated when character data is input, and an "AI Assistance" item designated when receiving assistance from the data generation model 58 are displayed. Further, on this communication support screen, for the menu display area 70B, a "Standard Phrase" item designated when using a pre-registered standard phrase, a "Send" item designated when sending the sentence data determined to be applied to the interlocutor 82, and a "Delete" item selected when deleting some characters or character strings from the sentence data are displayed. Note that hereinafter, to avoid complication, the description of the cases where the "Standard Phrase" item and the "Delete" item are specified will be omitted.

[0070] In step S304, the processor 28 waits until any menu item is specified by the user 80's line of sight. In step S306, the processor 28 determines whether the "OK / Cancel" item has been specified by the user 80. If the determination is affirmative, the process proceeds to step S308 to execute the predetermined OK / Cancel process.

[0071] In the yes / no process according to this embodiment, instead of each menu item displayed in the menu display area 70B, a display is performed such that either "yes" or "no" can be selectively specified by the line of sight. Therefore, when the user 80 responds to the conversation by the interlocutor 82, the user 80 specifies "yes" by the line of sight, and when not responding to the conversation, the user 80 specifies "no" by the line of sight. In response to the specification by this user 80, as shown in FIG. 7 as an example, the processor 28 controls the user terminal 14 to display the "yes" or "no" specified by the user 80 following the conversation sentence 70A1, and transmits information indicating the "yes" or "no" to the interlocutor terminal. In the example shown in FIG. 7, the case where "yes" is specified by the user 80 and as a result, the response sentence 70A2 of "yes" is displayed is illustrated. Further, by transmitting the information indicating the above "yes" or "no" to the interlocutor terminal, a sentence similar to the response sentence 70A2 shown in FIG. 7 is displayed on the interlocutor terminal in the same manner as the conversation display area 70A shown in FIG. 7.

[0072] When a response sentence of "yes" is displayed on the interlocutor terminal, the interlocutor 82 makes a conversation with more specific content. In response to this, the interlocutor terminal transmits information indicating the content of the conversation to the user terminal 14. When the user terminal 14 receives the information indicating the content of the conversation, as shown in FIG. 8 as an example, the processor 28 controls the user terminal 14 to display the conversation sentence 70A3 indicating the content of the conversation following the response sentence 70A2, and then proceeds to step S322. In the example shown in FIG. 8, the case where the sentence "Thank you. ··· How can I get involved?" is displayed as the conversation sentence 70A3 is illustrated.

[0073] On the other hand, if a negative determination is made in step S306, the process proceeds to step S310. In step S310, as an example shown in the upper diagram of FIG. 9, the processor 28 determines whether the item "Write characters" is specified for the user 80 to input a response sentence to the speech sentence 70A3. If an affirmative determination is made, the process proceeds to step S312 to execute a predetermined line-of-sight character input process. In the examples shown in FIGS. 9 to 15, an image indicating the position of the pupil of the user 80 is schematically illustrated directly below the display 40A, and the movement (shift) of the pupil is drawn as the movement of the pupil as seen from the user 80 side for convenience, but the actual direction of movement of the pupil of the user 80 is opposite to that shown in the figures.

[0074] In the line-of-sight character input process according to the present embodiment, the processor 28 performs a display that allows characters to be selectively specified one by one by the line of sight instead of each menu item displayed in the menu display area 70B, as shown in the lower diagram of FIG. 9 as an example. Therefore, the user 80 specifies the above-mentioned several characters of character data by the line of sight. In response to this specification by the user 80, the processor 28 controls the user terminal 14 to display the character string indicated by the character data specified by the user 80 in the character data display area 70D, as shown in FIG. 10 as an example, and then proceeds to step S322. In the example shown in FIG. 10, it is assumed that the word "communication", which is the key to the content of the response sentence to the speech sentence 70A3, is specified by the user 80, and a character string of the first three characters "comu" thereof is specified, and as a result, the case where the character string is displayed in the character data display area 70D is illustrated.

[0075] On the other hand, if a negative determination is made in step S310, the process proceeds to step S314. In step S314, as an example shown in FIG. 10, the processor 28 determines whether the "AI assistance" item is specified by the user 80. If an affirmative determination is made, the process proceeds to step S316 to execute a predetermined AI assistance process.

[0076] In the AI-assisted processing according to this embodiment, first, as shown in FIG. 11 as an example, the processor 28 makes a display such that the above-described strings indicating a plurality of types of directions can be selectively specified by the line of sight in place of each menu item displayed in the menu display area 70B. Therefore, the user 80 specifies the string indicating the desired direction by the line of sight. In the example shown in FIG. 11, the case where the user 80 specifies the string indicating the direction of "encouragingly" is illustrated.

[0077] Next, the processor 28 uses the character data indicating the string specified by the user 80 and the direction data indicating the direction, and as described above, the data generation model 58 generates a plurality of sentence data indicating candidates for the response sentence, which are related to the string indicated by the character data and have the direction indicated by the direction data. Then, as shown in FIG. 12 as an example, the processor 28 controls the user terminal 14 to display the candidate sentence 70B1 of the response sentence indicated by any one of the plurality of generated sentence data in the menu display area 70B. At this time, as shown in FIG. 12, the processor 28 controls the menu display area 70B to display the next candidate display designation area 70B2, which is designated when displaying the next candidate of the response sentence. In the example shown in FIG. 12, the sentence "Carefully accumulate communication with your son... It is your warm understanding and support." is displayed as the candidate sentence 70B1.

[0078] Therefore, when the user 80 likes the candidate sentence 70B1 displayed in the menu display area 70B, the user 80 designates the candidate sentence 70B1 with their line of sight. When the user does not like it, the user 80 designates the next candidate display designation area 70B2 with their line of sight. When the next candidate display designation area 70B2 is designated by the user 80, as shown in FIG. 13 as an example, the processor 28 controls the user terminal 14 to display any one candidate sentence 70B3 of the remaining sentence data generated by the data generation model 58 in the menu display area 70B. In the example shown in FIG. 13, the sentence "In order for the son to have contact with society again,... and please do not blame yourself too much." is displayed as the candidate sentence 70B3.

[0079] On the other hand, when the candidate sentence being displayed is designated by the user 80, as an example, as shown in FIG. 14, the user terminal 14 is controlled to display the candidate sentence in the character data display area 70D, assuming that the candidate sentence is adopted, and then the process proceeds to step S322. In the example shown in FIG. 14, the case where the candidate sentence "In order for the son to have contact with society again,..." is designated by the user 80 and as a result, the candidate sentence is displayed in the character data display area 70D is illustrated.

[0080] On the other hand, if a negative determination is made in step S314, the process proceeds to step S318. In step S318, as shown in FIG. 14 as an example, the processor 28 determines whether the "Send" item has been designated by the user 80. If a negative determination is made, the process proceeds to step S322. If an affirmative determination is made, the process proceeds to step S320 to execute a predetermined transmission process.

[0081] In the transmission process according to this embodiment, the processor 28 controls the user terminal 14 to display the response sentence in the dialogue display area 70A by regarding the candidate sentence specified by the user 80 as the final response sentence, and transmits the data indicating the response sentence to the interlocutor terminal. As a result, as shown in FIG. 15 as an example, in the dialogue display area 70A on the intention transmission support screen of the user terminal 14, the response sentence 70A4 is displayed immediately below the speaking sentence 70A3, and on the interlocutor terminal, a sentence similar to the response sentence 70A4 shown in FIG. 15 is displayed in the same manner as the dialogue display area 70A shown in FIG. 15.

[0082] In step S322, the processor 28 determines whether the end timing predetermined as the timing to end this specific process has arrived. If the determination is negative, the process returns to step S304, while if the determination is positive, this specific process ends. In this embodiment, as the above end timing, the timing when the user 80 or the interlocutor 82 makes an instruction input to instruct the end of the dialogue is applied, but it goes without saying that it is not limited to this.

[0083] Next, with reference to FIG. 16, the overall content of the specific process according to this embodiment will be described. FIG. 16 is a diagram showing the overall content of the specific process according to this embodiment.

[0084] As shown in FIG. 16, in the specific process according to this embodiment, the party of the ALS uses the user terminal 14 to input character data of about several characters and directionality data indicating the directionality of the response sentence. In response, the user terminal 14 transmits the character data and the directionality data to the data processing device 12.

[0085] When the data processing device 12 receives character data and direction data from the user terminal 14, it uses the character data and direction data to generate candidate response sentences by the data generation model 58 based on the recent conversation history of the parties, information of related parties, characteristics of the parties' tones, and the parties' past sentences, etc. Then, the data processing device 12 causes the user terminal 14 to display the generated candidate response sentences, and the user 80 selects a response sentence to be adopted from among the displayed candidates.

[0086] Examples of the information of related parties used at this time can include information indicating the names and relationships of related parties such as family members and friends. In addition, examples of the characteristics of the parties' tones can include the characteristics of the endings of words used by the parties and the characteristics of turns of phrase. Furthermore, examples of the parties' past sentences can include the parties' LINE (registered trademark) and coherent sentences written by the parties in the past.

[0087] As described above, in the data processing system 10 according to the present embodiment, an input unit 292 that acquires user data, a processing unit 294 that performs a specific process using a data generation model 58 that generates a predetermined inference result according to the user data, and an output unit 296 that outputs the result of the specific process to a predetermined device are provided. The input unit 292 acquires, as user data from the device, character data indicating characters input using the user's line of sight. The processing unit 294 performs, as the specific process, a process of deriving sentence data related to the characters indicated by the character data using the output of the data generation model 58 when the character data is input. Therefore, it is possible to assist the ALS parties in communicating with others.

[0088] In addition, in the data processing system 10 according to the present embodiment, as user data from the device, direction data indicating the direction of sentence data is further acquired, and a process of deriving candidate sentence data using the output of the data generation model 58 when the character data and the direction data are input is performed as the specific process. Therefore, it is possible to derive more accurate candidate sentence data as compared with the case where the direction data is not used.

[0089] In addition, in the data processing system 10 according to the present embodiment, as directional data, data indicating at least one direction among a positive direction, a negative direction, an interrogative direction, an encouraging direction, a direction related to kindness, and a direction related to emotion is applied. Therefore, sentence data corresponding to the applied directional data can be derived.

[0090] In addition, in the data processing system 10 according to the present embodiment, as sentence data, data in which the relationship between the user and the characters appearing in the sentence data is reflected is applied. Therefore, more accurate sentence data can be derived as compared with the case where the above relationship is not applied.

[0091] In addition, in the data processing system 10 according to the present embodiment, the above relationship includes at least one relationship among family, friends, seniors, juniors, and lovers. Therefore, more accurate sentence data can be derived as compared with the case where the included relationship is not used.

[0092] In addition, in the data processing system 10 according to the present embodiment, as sentence data, data in which past sentences created by the user are reflected is applied. Therefore, more user-like sentence data can be derived as compared with the case where the above past sentences are not applied.

[0093] In addition, in the data processing system 10 according to the present embodiment, the above past sentences include at least one of sentences created by the user during email exchanges and sentences sent by the user on social networking services. Therefore, more user-like sentence data can be derived as compared with the case where the included past sentences are not used.

[0094] In addition, in the data processing system 10 according to the present embodiment, as sentence data, data in which the user's tone is reflected is applied. Therefore, more user-like sentence data can be derived as compared with the case where data in which the user's tone is not reflected is not applied as sentence data.

[0095] Also, in the data processing system 10 according to the present embodiment, the user's tone is set to include at least one of the tone of written language and the tone of spoken language. Therefore, more user-like sentence data can be derived as compared with the case where the included tone is not applied.

[0096] Furthermore, in the data processing system 10 according to the present embodiment, as sentence data, a plurality of sentence data related to the above character data are derived as selection candidates, the selection candidates are output to a device, and a selection instruction from the selection candidates by the user is further obtained. Therefore, more sentence data that matches the user's preference can be adopted as compared with the case where a selection instruction from the selection candidates by the user is not obtained.

[0097] As described above, the system according to the present disclosure has been mainly described in terms of the functions of the data processing device 12. However, the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program operating on a personal computer or as an application operating on a smartphone or the like. The method according to the present disclosure may be provided to a user in the form of SaaS (Software as a Service).

[0098] In the above embodiment, a configuration example in which specific processing is performed by one computer 22 has been given. However, the technology of the present disclosure is not limited to this, and distributed processing for specific processing by a plurality of computers including the computer 22 may be performed.

[0099] In the above-described embodiment, an example has been described in which the specific processing program 56 is stored in the storage 32. However, the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing according to the specific processing program 56.

[0100] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed in the computer 22 in response to a request from the data processing device 12.

[0101] Note that it is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32. A part of the specific processing program 56 may be stored.

[0102] As the hardware resources for executing the specific processing, the following various processors can be used. As the processor, for example, a general-purpose processor such as a CPU that functions as a hardware resource for executing specific processing by executing software, that is, a program, can be mentioned. In addition, as the processor, for example, a dedicated electric circuit that is a processor having a circuit configuration designed specifically for executing specific processing such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit) can be mentioned. A memory is built in or connected to any of the processors, and any of the processors executes specific processing by using the memory.

[0103] The hardware resources for executing the specific process may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources for executing the specific process may be a single processor.

[0104] As an example of being composed of a single processor, first, there is a form in which one processor is composed of a combination of one or more CPUs and software, and this processor functions as the hardware resources for executing the specific process. Second, as represented by a SoC (System-on-a-chip), there is a form in which a processor that realizes the functions of the entire system including a plurality of hardware resources for executing the specific process is used in one IC chip. Thus, the specific process is realized as the hardware resources using one or more of the above various processors.

[0105] Furthermore, as the hardware structure of these various processors, more specifically, an electric circuit combining circuit elements such as semiconductor elements can be used. Also, the above specific process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed within the scope of not departing from the gist.

[0106] The above-described description and illustration are detailed explanations of the part related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the explanations regarding the above-described configuration, function, operation, and effect are explanations regarding an example of the configuration, function, operation, and effect of the part related to the technology of the present disclosure. Therefore, it goes without saying that within the scope not departing from the gist of the technology of the present disclosure, the above-described description and illustration may be modified by deleting unnecessary parts, adding new elements, or making replacements. Also, in order to avoid complication and facilitate the understanding of the part related to the technology of the present disclosure, the above-described description and illustration omit the explanations regarding common technical knowledge and the like that do not particularly require explanation for implementing the technology of the present disclosure.

[0107] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually stated to be incorporated by reference.

[0108] Regarding the above, the following supplementary notes are further disclosed.

[0109] <Supplementary Note 1> An input unit that acquires user data; A processing unit that performs a specific process using a data generation model that generates a predetermined inference result according to the user data; An output unit that outputs the result of the specific process to a predetermined device, and the input unit acquires, as the user data from the device, character data indicating characters input using the user's line of sight, the processing unit performs, as the specific process, a process of deriving passage data related to the characters indicated by the character data using the output of the data generation model when the character data is input, A data processing device. <Supplementary Note 2> the input unit further acquires, as the user data from the device, directionality data indicating the directionality of the passage data, The processing unit performs, as the specific processing, a process of deriving candidates for the text data using the output of the data generation model when the text data and the directionality data are input. The data processing device according to Supplementary Note 1. <Supplementary Note 3> The directionality data is data indicating at least one directionality among positive directionality, negative directionality, interrogative directionality, encouraging directionality, directionality regarding kindness, and directionality regarding emotion. The data processing device according to Supplementary Note 2. <Supplementary Note 4> The text data is data in which the relationship between the user and the characters appearing in the text data is reflected. The data processing device according to any one of Supplementary Notes 1 to 3. <Supplementary Note 5> The relationship includes at least one relationship among family, friend, senior, junior, and lover. The data processing device according to Supplementary Note 4. <Supplementary Note 6> The text data is data in which the past texts created by the user are reflected. The data processing device according to any one of Supplementary Notes 1 to 5. <Supplementary Note 7> The past texts include at least one of the texts created by the user during the exchange of emails and the texts sent by the user on social networking services. The data processing device according to Supplementary Note 6. <Supplementary Note 8> The text data is data in which the tone of the user is reflected. The data processing device according to any one of Supplementary Notes 1 to 7. <Supplementary Note 9> The tone of the user includes at least one of the tone of written language and the tone of spoken language. The data processing device according to Supplementary Note 8. <Supplementary Note 10> The processing unit derives, as the text data, a plurality of text data related to the character data as selection candidates. The output unit outputs the selection candidates to the device, The input unit further obtains a selection instruction from the selection candidates by the user, The data processing device according to any one of Appendices 1 to 9. <Appendix 11> The device is smart glasses, The data processing device according to any one of Appendices 1 to 10.

Explanation of symbols

[0110] 10 Data processing system 12 Data processing device 14 User terminal 22 Computer 24 Database 26 Communication I / F 28 Processor 30 RAM 32 Storage 34 Bus 36 Computer 38 Reception device 38A Touch panel 38B Microphone 40 Output device 40A Display 40B Speaker 42 Camera 44 Communication I / F 46 Processor 46A Control unit 48 RAM 50 Storage 52 Bus 54 Network 56 Specific processing program 58 Data generation model 62 Reception / output program 70A Interaction display area 70B Menu display area 70C Selection button 70D Character data display area 80 User 82 Interlocutor 290 Specific processing unit 292 Input unit 294 Processing unit 296 Output unit

Claims

1. An input unit that acquires user data, A processing unit that performs a specific process using a data generation model that generates a predetermined inference result according to the user data, An output unit that outputs the result of the specific process to a predetermined device, and The input unit acquires, as the user data, character data indicating characters input using the user's line of sight from the device, The processing unit performs, as the specific process, a process of deriving sentence data related to the characters indicated by the character data using the output of the data generation model when the character data is input, A data processing device.

2. The input unit further acquires, as the user data, directionality data indicating the directionality of the sentence data from the device, The processing unit performs, as the specific process, a process of deriving candidates for the sentence data using the output of the data generation model when the character data and the directionality data are input, The data processing device according to claim 1.

3. The directionality data is data indicating at least one directionality of a positive directionality, a negative directionality, an interrogative directionality, an encouraging directionality, a directionality related to kindness, and a directionality related to emotion, The data processing device according to claim 2.

4. The sentence data is data in which the relationship between the user and the characters appearing in the sentence data is reflected, The data processing device according to claim 1 or claim 2.

5. The relationship includes at least one relationship of family, friend, senior, junior, and lover, The data processing device according to claim 4.

6. The sentence data is data in which past sentences created by the user are reflected, The data processing device according to claim 1 or claim 2.

7. The past sentences include at least one of the sentences created by the user during the e-mail communication and the sentences sent by the user on the social networking service. The data processing device according to claim 6.

8. The sentence data is data that reflects the user's tone. The data processing device according to claim 1 or claim 2.

9. The user's tone includes at least one of the tone of written language and the tone of spoken language. The data processing device according to claim 8.

10. The processing unit derives a plurality of sentence data related to the character data as selection candidates as the sentence data. The output unit outputs the selection candidates to the device. The input unit further obtains a selection instruction from the user from the selection candidates. The data processing device according to claim 1.

11. The device is smart glasses. The data processing device according to claim 1.

12. Obtain user data. Perform specific processing using a data generation model that generates a predetermined inference result according to the user data. A data processing method for outputting the result of the specific processing to a predetermined device, Obtain, as the user data from the device, character data indicating characters input using the user's line of sight. Using the output of the data generation model when the character data is input, perform the process of deriving sentence data related to the characters indicated by the character data as the specific processing. Data processing method.

13. Obtain user data, perform a specific process using a data generation model that generates a predetermined inference result according to the user data, a process of outputting the result of the specific process to a predetermined device, as the user data from the device, obtain character data indicating characters input using the user's line of sight, using the output of the data generation model when the character data is input, perform the process of deriving sentence data related to the characters indicated by the character data as the specific process, A data processing program for causing a computer to execute the process.

Images