Data processing apparatus, communication terminal, data processing method, and program

Abstract

To provide a data processing apparatus, a communication terminal, a data processing method, and a program that can support activities of a person who has difficulty in recognizing peripheral situations through the visual sense.SOLUTION: A data processing apparatus includes: an acquisition part that acquirers an image obtained by imaging the periphery of a user by a camera and request data indicating a request related to the periphery by the user; a generation part that uses a response generation model to generate response data indicating a response according to the request data, based on the image; and an output part that outputs the response data generated by the generation part.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The technology disclosed herein relates to a data processing device, a communication terminal, a data processing method, and a program. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, visually impaired people need more support in daily activities than people without vision. Visually impaired people have difficulty in grasping the shape, color, and size of objects, and in reading books. In addition, it is difficult for them to safely travel to their destination when going out. If a dedicated caregiver were always with the visually impaired person, the caregiver might be able to respond appropriately to the requests of the visually impaired person, but in reality, it is difficult to have a dedicated caregiver always with the visually impaired person.

[0005] The technology disclosed herein provides a data processing device, a communication terminal, a data processing method, and a program that can support the activities of people who have difficulty understanding their surroundings through vision. [Means for solving the problem]

[0006] A first aspect of the technology disclosed herein is a data processing device comprising: an acquisition unit that acquires an image obtained by capturing an image of a user's surroundings by a camera and request data indicating a request made by the user regarding the surroundings; a generation unit that generates response data indicating a response corresponding to the request data based on the image by using a response generation model; and an output unit that outputs the response data generated by the generation unit.

[0007] A second aspect of the technology of the present disclosure is the data processing device according to the first aspect, wherein the response data includes voice data indicating the response by voice.

[0008] A third aspect of the technology of the present disclosure is the data processing device according to the first aspect, wherein the request includes a traffic condition grasping request that causes the user to grasp the surrounding traffic condition.

[0009] A fourth aspect of the technique of the present disclosure is the data processing device according to the first aspect, wherein the request includes a route guidance request for providing route guidance to the user.

[0010] A fifth aspect of the technology disclosed herein is the data processing device according to the first aspect, wherein the request includes a book content understanding request that causes the user to understand the content of a book.

[0011] A sixth aspect of the technology disclosed herein is a data processing device according to the first aspect, in which the request is accepted by a communication terminal carried by the user, and the communication terminal acquires the response data output from the output unit, and outputs the acquired response data.

[0012] A seventh aspect of the technology of the present disclosure is the data processing device according to the sixth aspect, in which the communication terminal is a wearable terminal.

[0013] An eighth aspect of the technology of the present disclosure is a data processing device according to the seventh aspect, in which the wearable terminal is smart glasses.

[0014] A ninth aspect of the technique of the present disclosure is the data processing device according to the sixth aspect, in which the communication terminal includes a microphone, and the request is received by the microphone.

[0015] A tenth aspect of the technology of the present disclosure is the data processing device according to the sixth aspect, in which the output of the response data by the communication terminal is achieved by at least one of a voice output method and a bone conduction method.

[0016] An eleventh aspect of the technology of the present disclosure is a communication terminal comprising a communication interface that communicates with a data processing device relating to any one of the first to tenth aspects, thereby transmitting the image and the request data to the data processing device and receiving the response data output from the output unit, a receiving device that accepts the request, and an output device that outputs the response data.

[0017] A twelfth aspect of the technology of the present disclosure is a data processing method including acquiring an image of a user's surroundings captured by a camera and request data indicating a request made by the user regarding the surroundings, generating response data indicating a response corresponding to the request data based on the image by using a response generation model, and outputting the generated response data.

[0018] A thirteenth aspect of the technology of the present disclosure is a program for causing a computer to execute a process including acquiring an image of a user's surroundings captured by a camera and request data indicating a request made by the user regarding the surroundings, generating response data indicating a response corresponding to the request data based on the image by using a response generation model, and outputting the generated response data. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. [Diagram 2] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device and a smart device. [Diagram 3] FIG. 13 is a conceptual diagram showing an example of processing contents in a data processing system when an activity support process is performed by a processor of a data processing device and a reception output process is performed by a processor of a smart device. [Figure 4] 13 is a flowchart showing an example of the flow of an activity support process. [Diagram 5] 13 is a flowchart showing an example of the flow of a reception and output process. [Figure 6] 13 is a first variation of a user request and a response to the request. [Figure 7] 11 is a second variation of a user request and a response to the request. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, examples of embodiments of a data processing device, a communication terminal, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the attached drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0023] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the embodiment.

[0028] 1, the data processing system 10 includes a data processing device 12 and smart glasses 14. An example of the data processing device 12 is a server. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure, and the smart glasses 14 are an example of a "communication terminal," a "wearable terminal," and a "smart glasses" according to the technology of the present disclosure.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0030] The smart glasses 14 are worn and used by a user 20. Here, the user 20 refers to a user of the data processing system 10. For convenience of explanation, the following description will be given on the assumption that the user 20 is a visually impaired person (for example, a completely blind person).

[0031] The smart glasses 14 include a computer 36, a microphone 38, a speaker 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 38, the speaker 40, and the camera 42 are also connected to the bus 52. The microphone 38 is an example of a "microphone" and a "reception device" according to the technology of the present disclosure. The speaker 40 is an example of an "output device" according to the technology of the present disclosure.

[0032] The microphone 38 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 38 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 40 outputs the voice according to instructions from the processor 46.

[0033] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state. The communication I / F 44 is an example of a "communication interface" according to the technology of the present disclosure.

[0035] Fig. 2 shows an example of main functions of the data processing device 12 and the smart glasses 14. As shown in Fig. 2, in the data processing device 12, activity support processing is performed by the processor 28. An activity support program 56 is stored in the storage 32.

[0036] Activity support program 56 is an example of a "program" according to the technology of the present disclosure. Processor 28 reads activity support program 56 from storage 32, and executes the read activity support program 56 on RAM 30. The activity support process is realized by processor 28 operating as acquisition unit 28A, generation unit 28B, and output unit 28C in accordance with activity support program 56 executed on RAM 30.

[0037] A reply generation model 58 is stored in the storage 32. The reply generation model 58 is an example of a "reply generation model" according to the technique of the present disclosure. As will be described in detail later, the reply generation model 58 is used by the generator 28B.

[0038] In the smart glasses 14, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] FIG. 3 shows an example of processing contents in the data processing system 10 when activity support processing is performed by the processor 28 of the data processing device 12 and reception output processing is performed by the processor 46 of the smart glasses 14.

[0040] As shown in FIG. 3, in the smart glasses 14, the microphone 38 receives a request 62 regarding the surroundings of the user 20. That is, the microphone 38 captures a voice indicating the request 62. The voice indicating the request 62 is a voice emitted by the user 20. There is a book around the user 20. For example, the book is in front of the user 20. The request 62 is a request for the user 20 to understand the contents of the book. In the example shown in FIG. 4, the request 62 is indicated as "I want to read a book."

[0041] Here, the book is an example of a "book" according to the technology of the present disclosure. Also, request 62 is an example of a "book content understanding request" according to the technology of the present disclosure. In addition to books, the concept of a "book" according to the technology of the present disclosure includes printed media such as newspapers, advertisements, and posters, and media provided via a display such as electronic books.

[0042] The control unit 46A transmits request data 64 indicating the request 62 accepted by the microphone 38 to the data processing device 12 via the communication I / F 44 (see FIG. 1). In the data processing device 12, the request data 64 is received by the communication I / F 26. The acquisition unit 28A acquires the request data 64 received by the communication I / F 26.

[0043] On the condition that the request 62 is accepted by the microphone 38, the control unit 46A causes the camera 42 to capture an image of the surroundings of the user 20. The surroundings of the user 20 include a book that is in front of the user 20. The control unit 46A transmits an image 63 obtained by the camera 42 capturing an image of the surroundings of the user 20 to the data processing device 12 via the communication I / F 44 (see FIG. 1). In the data processing device 12, the image 63 is received by the communication I / F 26. The acquisition unit 28A acquires the image 63 received by the communication I / F 26.

[0044] The generation unit 28B generates an instruction statement 66 based on the request data 64. The instruction statement 66 is text data indicating an instruction to be given to the response generation model 58. For example, the generation unit 28B creates the instruction statement 66 by using a template that is the basis of the instruction statement 66. For example, the instruction statement 66 is created by incorporating the request 62 indicated by the request data 64 into the template.

[0045] The request data 64 is voice data. The generation unit 28B performs a conventionally known voice-to-text conversion process on the request data 64. By performing the voice-to-text conversion process on the request data 64, the request 62 indicated by the request data 64 is converted into text.

[0046] The response generation model 58 is a so-called generative AI (Artificial Intelligence). One example of the response generation model 58 is ChatGPT-4 (Internet search<https: / / openai.com / gpt-4> ) or ChatGPT-4-32k. The reply generation model 58 is obtained by performing deep learning on a neural network. An instruction sentence 66 is input to the reply generation model 58, and an image 63 acquired by the acquisition unit 28A is input as inference data.

[0047] The response generation model 58 performs inference on the input image 63 according to the instruction given by the instruction sentence 66, and outputs the inference result as voice data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0048] An example of the instruction 66 is text data indicating the following text: "The person you are speaking with is blind. You are an assistant who accompanies the blind person and explains the situation around the blind person in an easy-to-understand manner. Please read the contents of the book shown in the input image and express the read contents in voice. If the book contents are sentences, please read the sentences out loud. If the book contents are pictures, diagrams, or tables, please decipher the meaning of the pictures, diagrams, or tables and express the deciphered contents in voice." Note that the contents of the instruction 66 specifically illustrated here are merely examples, and any contents may be used as long as an appropriate response to the request 62 can be obtained from the response generation model 58.

[0049] The generation unit 28B uses the reply generation model 58 to generate reply data 70 indicating a reply 68 in response to the request 62 indicated by the request data 64, based on the image 63. When an instruction statement 66 and an image 63 are input to the reply generation model 58, the reply data 70 is generated by the reply generation model 58. The reply data 70 is voice data indicating the reply 68 to the request 62. For example, the reply 68 is the contents (e.g., a sentence) of a book shown in the image 63.

[0050] The output unit 28C outputs the response data 70 generated by the generation unit 28B. In the example shown in Fig. 3, the output unit 28C transmits the response data 70 generated by the generation unit 28B to the smart glasses 14 via the communication I / F 26 (see Fig. 1).

[0051] In the smart glasses 14, the response data 70 is received by the communication I / F 44 (see FIG. 1). The control unit 46A acquires the response data 70 received by the communication I / F 44. The control unit 46A causes the speaker 40 to output the response data 70. That is, the response 68 indicated by the response data 70 is output by voice from the speaker 40.

[0052] Next, the operation of the data processing system 10 will be described with reference to FIGS.

[0053] First, an example of the flow of the activity support process will be described with reference to Fig. 4. Note that the flow of the activity support process shown in Fig. 4 is an example of a "data processing method" according to the technique of the present disclosure.

[0054] In the activity support process shown in FIG. 4, first, in step ST10, the acquisition unit 28A determines whether or not the request data 64 transmitted from the smart glasses 14 is received by the communication I / F 26 by executing the process of step ST52 shown in FIG. 5 described later. In step ST10, if the request data 64 has not been received by the communication I / F 26, the determination is negative, and the determination of step ST10 is performed again. In step ST10, if the request data 64 has been received by the communication I / F 26, the determination is positive, and the activity support process proceeds to step ST12. In addition, the acquisition unit 28A acquires the request data 64 received by the communication I / F 26.

[0055] In step ST12, the acquisition unit 28A determines whether or not the image 63 transmitted from the smart glasses 14 is received by the communication I / F 26 by executing the process of step ST56 shown in FIG. 5 described later. In step ST12, if the image 63 is not received by the communication I / F 26, the determination is negative, and the determination of step ST12 is performed again. In step ST12, if the image 63 is received by the communication I / F 26, the determination is positive, and the activity support process proceeds to step ST14. In addition, the acquisition unit 28A acquires the image 63 received by the communication I / F 26.

[0056] In step ST14, the generating unit 28B generates an instruction statement 66 based on the request data 64 acquired by the acquiring unit 28A. After the process of step ST14 is executed, the activity support process proceeds to step ST16.

[0057] In step ST16, the generation unit 28B generates reply data 70 corresponding to the instruction sentence 66 based on the image 63 by using the reply generation model 58. After the process of step ST16 is executed, the activity support process proceeds to step ST18.

[0058] In step ST18, the output unit 28C transmits the response data 70 generated in step ST16 to the smart glasses 14 via the communication I / F 26. After the process of step ST18 is executed, the activity support process proceeds to step ST20.

[0059] In step ST20, the acquisition unit 28A determines whether or not a condition for terminating the activity support process (hereinafter, referred to as an "activity support process termination condition") is satisfied. An example of the activity support process termination condition is a condition in which an instruction to terminate the activity support process is received by the microphone 38 or the like.

[0060] In step ST20, if the activity support process end condition is not satisfied, the determination is negative, and the activity support process proceeds to step ST10. In step ST20, if the activity support process end condition is satisfied, the determination is positive, and the activity support process ends.

[0061] Next, an example of the flow of the reception and output process will be described with reference to FIG.

[0062] 5, first, in step ST50, the control unit 46A determines whether or not the request 62 has been accepted by the microphone 38. If the request 62 has not been accepted by the microphone 38 in step ST50, the determination is negative, and the determination of step ST50 is performed again. If the request 62 has been accepted by the microphone 38 in step ST50, the determination is positive, and the reception output process proceeds to step ST52.

[0063] In step ST52, the control unit 46A transmits request data 64 indicating the request 62 accepted by the microphone 38 to the data processing device 12 via the communication I / F 44. After the process of step ST52 is executed, the acceptance and output process proceeds to step ST54.

[0064] In step ST54, the control unit 46A causes the camera 42 to capture an image of the surroundings of the user 20. After the process of step ST54 is executed, the reception and output process proceeds to step ST56.

[0065] In step ST56, the control unit 46A transmits an image 63 obtained by capturing an image of the surroundings of the user 20 by the camera 42 to the data processing device 12 via the communication I / F 44. After the processing of step ST56 is executed, the reception and output processing proceeds to step ST58.

[0066] In step ST58, the control unit 46A judges whether or not the response data 70 transmitted by executing the process of step ST18 shown in FIG. 4 has been received by the communication I / F 44. If the response data 70 has not been received by the communication I / F 44 in step ST58, the judgment is negative, and the judgment of step ST58 is performed again. If the response data 70 has been received by the communication I / F 44 in step ST58, the judgment is positive, and the reception and output process proceeds to step ST60. In addition, the control unit 46A acquires the response data 70 received by the communication I / F 44.

[0067] In step ST60, the control unit 46A causes the speaker 40 to output the response data 70 acquired in step ST58. After the process of step ST60 is executed, the reception and output process proceeds to step ST62.

[0068] In step ST60, the control unit 46A determines whether a condition for terminating the reception / output process (hereinafter referred to as a "reception / output process termination condition") is satisfied. One example of the reception / output process termination condition is that an instruction to terminate the reception / output process is received by the microphone 38 or the like.

[0069] In step ST60, if the acceptance / output processing end condition is not satisfied, the determination is negative and the acceptance / output processing proceeds to step ST50. In step ST60, if the acceptance / output processing end condition is satisfied, the determination is positive and the acceptance / output processing ends.

[0070] As described above, in the data processing device 12, the acquisition unit 28A acquires the image 63 obtained by capturing the surroundings of the user 20 with the camera 42, and the request data 64 indicating the request 62 by the user 20 regarding the surroundings. In addition, the generation unit 28B uses the response generation model 58 to generate response data 70 indicating a response 68 in response to the request data 64 based on the image 63. Then, the output unit 28C transmits the response data 70 generated by the generation unit 28B to the smart glasses 14. In the smart glasses 14, the response data 70 is output by the speaker 40. This allows the user 20 to understand the response 68 in response to the request 62. In addition, since the response data 70 is audio data, it is possible to support the activities of a person who has difficulty in understanding the surroundings through vision.

[0071] In addition, in the data processing device 12, the contents of the book (e.g., a sentence) are output by voice as a response 68 in response to the user 20's request 62 of "I want to read a book," so that the user 20 can understand the contents of the book even if it is difficult for him or her to visually grasp the contents of the book.

[0072] Furthermore, since the request 62 of the user 20 is received by the microphone 38, even a person who has difficulty visually grasping the surroundings can transmit the request 62 to the data processing device 12.

[0073] In the above embodiment, an example of a form in which the user 20 hears the sound output from the speaker 40 (i.e., the user 20 perceives the reply 68 in a voice output manner) has been described, but the technology of the present disclosure is not limited to this. For example, the reply data 70 may be output in a bone conduction manner. Also, the output in the voice output manner and the output in the bone conduction manner may be selected according to an instruction given by the user 20 via the microphone 38.

[0074] In the above embodiment, an example in which the request 62 indicated by the request data 64 is converted into text by performing a speech-to-text conversion process on the request data 64 has been given, but this is merely one example. The request data 64A may be input to the response generation model 58 as speech data. In this case, an example of the instruction sentence 66 is "The person you are talking to is blind. You are a caregiver who accompanies the blind person and explains the situation around the blind person in an easy-to-understand manner. Please decode the speech of the blind person and understand the request of the blind person. Please create a response according to the request of the blind person based on the input image and respond by voice. If the request of the blind person is to understand the contents of a book, please read the contents of the book shown in the input image and read the read contents aloud. If the contents of the book are sentences, please express the sentences by voice. If the contents of the book are pictures, figures, or tables, please decode the meaning of the pictures, figures, or tables and express the decoded contents by voice."

[0075] Also, the request data 64A may be input to the response generation model 58 as voice data, so that the response generation model 58 can identify the emotion of the user 20. An example of the instruction 66 in this case is "The person you are talking to is blind. You are a caregiver who accompanies the blind person and explains the situation around the blind person in an easy-to-understand manner. Please decode the voice of the blind person to understand the request of the blind person. Please decode the voice of the blind person to identify the emotion of the blind person. Please create a response according to the request of the blind person based on the input image, and read the response aloud. Please make the content of the response sympathetic to the emotion of the blind person. If the request of the blind person is to understand the content of a book, please read the content of the book shown in the input image, and read the read content aloud. If the content of the book is a sentence, please express the sentence by voice. If the content of the book is a picture, a diagram, or a table, please decode the meaning of the picture, diagram, or table, and express the decoded content by voice."

[0076] Here, part of the instruction 66 reads, "If the blind person's request is to understand the contents of a book, read the contents of the book shown in the input image and read the read contents aloud. If the contents of the book are sentences, express the sentences aloud. If the contents of the book are pictures, diagrams, or tables, decipher the meaning of the pictures, diagrams, or tables and express the deciphered contents aloud." The content of this part is changed according to the content of the request 62 of the user 20.

[0077] In the above embodiment, an example is given in which the data processing device 12 provides the user 20 with a response 68 in response to the request 62 of the user 20 "I want to read a book," but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 6, the data processing device 12 may provide the user 20 with a response 68A in response to a traffic condition understanding request 62A. The traffic condition understanding request 62A is a request for the user 20 to understand the traffic conditions around the user 20. In this case, the request data 64A indicating the traffic condition understanding request 62A is acquired by the acquisition unit 28A, and the instruction sentence 66A based on the request data 64A is generated by the generation unit 28B. The instruction sentence 66A is input to the response generation model 58.

[0078] An example of the instruction sentence 66A is text data indicating the text, "The person you are talking to is blind. You are a caregiver who accompanies the blind person and explains the situation around the blind person in an easy-to-understand manner. Please decode the speech of the blind person and understand his / her request. Please explain the traffic situation shown in the input image by voice." When the instruction sentence 66A is input to the response generation model 58 and the image 63 is input, the response generation model 58 generates response data 70A indicating a response 68A to the traffic situation understanding request 62 (in the example shown in FIG. 6, the response is "the intersection is one meter ahead"). The response data 70A is voice data and is output from the speaker 40. This allows the user 20 to understand the traffic situation around him / her.

[0079] 7, for example, a response 68B may be provided to the user 20 by the data processing device 12 in response to a route guidance request 62B. The route guidance request 62B is a request to provide route guidance to the user 20. In this case, request data 64B indicating the route guidance request 62B is acquired by the acquisition unit 28A, and an instruction sentence 66B based on the request data 64B is generated by the generation unit 28B. The instruction sentence 66B is input to the response generation model 58.

[0080] An example of the instruction 66B is text data indicating the text, "The person you are talking to is blind. You are a caregiver who accompanies the blind person and explains the situation around the blind person in an easy-to-understand manner. Please decode the speech of the blind person and understand his / her requirements. Please analyze the contents of the input image and provide voice guidance to the destination. The destination is '...'." Information identifying the destination is included in the guidance request 62B. For example, the information identifying the destination is extracted by the RPA and incorporated into the instruction 66B.

[0081] When the instruction sentence 66B and the image 63 are input to the reply generation model 58, reply data 70B indicating a reply 68B to the guidance request 62B (in the example shown in FIG. 7, a reply of "Please go straight for a while" is generated by the reply generation model 58. The reply data 70B is voice data, and is output from the speaker 40. This enables the user 20 to move to the destination according to the reply 68B to the guidance request 62B. At this time, information for guidance to the destination may be acquired, for example, by using an API (Application Programming Interface) from a website that searches for walking routes or routes using public transportation, with the destination as a search query.

[0082] Note that the request 62, the traffic condition understanding request 62A, and the route guidance request 62B are merely examples, and may be any request necessary for the user 20 to be active, such as a request related to food, clothing, and shelter necessary for living. In this case, too, a response corresponding to the request given by the user 20 may be generated by the response generation model 58 and provided to the user 20 via the smart glasses 14. In this case, too, it is preferable to have the user 20 perceive the response by a voice output method or a bone conduction method.

[0083] In the above embodiment, an example has been given in which an instruction sentence 66 in response to a request 62 from a user 20 is generated by the generation unit 28B and input to the reply generation model 58, but this is merely one example. For example, the user 20 may utter a voice indicating the content of the instruction sentence 66, and voice data indicating the uttered voice may be input to the reply generation model 58. Alternatively, an instruction sentence 66 capable of responding to any request may already be determined, and reply data 70 indicating a reply 68 in response to the content of the request 62 may be generated by the reply generation model 58.

[0084] In the above embodiment, the smart glasses 14 are exemplified, but the technology of the present disclosure is not limited thereto. For example, the smart glasses 14 may be used in combination with a smartphone. For example, an image may be captured by the camera 42 of the smart glasses 14, and a microphone and a speaker mounted on the smartphone may be used instead of the microphone 38 and the speaker 40. Also, a smartphone may be used instead of the smart glasses 14, or a wearable terminal other than the smart glasses 14 may be used. An example of a wearable terminal other than the smart glasses 14 includes an ear-hanging speaker and a camera with a microphone.

[0085] In the above embodiment, an example in which the data processing device 12 assuming a server is implemented in the data processing system 10 is given, but the technology of the present disclosure is not limited to this. For example, the technology of the present disclosure may be implemented in a general information processing system. Also, for example, the technology of the present disclosure may be implemented in a software program that runs on a server and / or a personal computer, or an application that runs on a smart device. Also, the technology of the present disclosure may be provided to a user in the form of SaaS (Software as a Service). Also, in the above embodiment, a server is given as an example of the data processing device 12, but this is merely an example, and the technology of the present disclosure may be realized by a device other than a server (for example, a personal computer, etc.).

[0086] In the above embodiment, an example was given in which activity support processing is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of activity support processing may be performed by multiple computers, including computer 22.

[0087] In the above embodiment, an example in which the activity assistance program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the activity assistance program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The activity assistance program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the activity assistance process in accordance with the activity assistance program 56.

[0088] Also, the activity assistance program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the activity assistance program 56 may be downloaded in response to a request from the data processing device 12 and installed in the computer 22. It is not necessary to store the entire activity assistance program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entire activity assistance program 56 in the storage 32, and only a part of the activity assistance program 56 may be stored.

[0089] As hardware resources for executing the activity support process, various processors as shown below can be used. An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the activity support process by executing software, i.e., a program. Another example of a processor is a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as a field-programmable gate array (FPGA), a programmable logic device (PLD), or an application specific integrated circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the activity support process by using the memory.

[0090] The hardware resource for executing the activity support process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the activity support process may be a single processor.

[0091] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes the activity support processing. Second, there is a configuration in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute the activity support processing on a single IC chip, as typified by SoC (System-on-a-chip). In this way, the activity support processing is realized using one or more of the above-mentioned various processors as hardware resources.

[0092] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor elements. Also, the above activity support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the process.

[0093] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0094] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[0095] <Additional Notes> (1) an acquisition unit that acquires an image obtained by capturing an image of a user's surroundings by a camera and request data indicating a request by the user regarding the surroundings; a generation unit that generates response data indicating a response corresponding to the request data based on the image by using a response generation model; an output unit that outputs the response data generated by the generation unit; Data processing device.

[0096] (2) The response data includes voice data indicating the response by voice. A data processing device as described in (1).

[0097] (3) The request includes a traffic situation awareness request that allows the user to understand the surrounding traffic situation. A data processing device according to (1) or (2).

[0098] (4) The request includes a route guidance request for providing route guidance to the user. A data processing device according to any one of (1) to (3).

[0099] (5) The request includes a book content comprehension request for causing the user to comprehend the content of the book. A data processing device according to any one of (1) to (4).

[0100] (6) the request is accepted by a communication terminal carried by the user; The communication terminal includes: Acquire the response data output from the output unit; Output the acquired response data A data processing device according to any one of (1) to (5).

[0101] (7) The communication terminal is a wearable terminal. A data processing device according to (6).

[0102] (8) The wearable device is a pair of smart glasses. A data processing device according to (7).

[0103] (9) The communication terminal includes a microphone; The request is received by the microphone. A data processing device according to any one of (6) to (8).

[0104] (10) The output of the response data by the communication terminal is realized by at least one of a voice output method and a bone conduction method. A data processing device according to any one of (6) to (9).

[0105] (11) a communication interface that communicates with the data processing device according to any one of (1) to (10) to transmit the image and the request data to the data processing device and receive the response data output from the output unit; a reception device that receives the request; and an output device for outputting the response data. Communications terminal.

[0106] (12) acquiring an image of a user's surroundings captured by a camera and request data indicating a request by the user regarding the surroundings; generating response data indicating a response corresponding to the request data based on the image by using a response generation model; and and outputting the generated response data. Data processing methods.

[0107] (13) acquiring an image of a user's surroundings captured by a camera and request data indicating a request by the user regarding the surroundings; generating response data indicating a response corresponding to the request data based on the image by using a response generation model; and A program for causing a computer to execute a process including outputting the generated response data. [Explanation of symbols]

[0108] 10 Data Processing System 12 Data Processing Device 14 Smart Glasses 20 users 22,36 Computer 26,44 Communication I / F 28,46 processor 28A Acquisition Department 28B Generation part 28C Output section 30,48 RAM 32,50 Storage 34,52 Bus 38 Microphones 40 Speaker 42 Camera 46A Control Unit 54 Network 56 Activity Support Program 58 Response Generation Model 60 Reception Output Program 62 requests 62A Traffic situation awareness request 62B Directions Request 63 images 64 Request Data 66,66A,66B instructions 68,68A,68B Reply 70,70A,70B Reply data

Claims

1. an acquisition unit that acquires an image obtained by capturing an image of a user's surroundings by a camera and request data indicating a request by the user regarding the surroundings; a generation unit that generates response data indicating a response corresponding to the request data based on the image by using a response generation model; an output unit that outputs the response data generated by the generation unit; Data processing device.

2. The response data includes voice data indicating the response by voice.

2. A data processing apparatus according to claim 1.

3. The request includes a traffic situation awareness request that allows the user to understand the surrounding traffic situation.

2. A data processing apparatus according to claim 1.

4. The request includes a route guidance request for providing route guidance to the user.

2. A data processing apparatus according to claim 1.

5. The request includes a book content comprehension request for causing the user to comprehend the content of the book.

2. A data processing apparatus according to claim 1.

6. the request is accepted by a communication terminal carried by the user; The communication terminal includes: Acquire the response data output from the output unit; Output the acquired response data 2. A data processing apparatus according to claim 1.

7. The communication terminal is a wearable terminal.

7. A data processing apparatus according to claim 6.

8. The wearable device is a pair of smart glasses.

8. A data processing apparatus according to claim 7.

9. The communication terminal includes a microphone; The request is received by the microphone.

7. A data processing apparatus according to claim 6.

10. The output of the response data by the communication terminal is realized by at least one of a voice output method and a bone conduction method.

7. The data processing device according to claim 6.

11. a communication interface that communicates with the data processing device according to any one of claims 1 to 10, thereby transmitting the image and the request data to the data processing device and receiving the response data output from the output unit; a reception device that receives the request; and an output device for outputting the response data. Communications terminal.

12. acquiring an image obtained by capturing an image of a user's surroundings by a camera and request data indicating a request by the user regarding the surroundings; generating response data indicating a response corresponding to the request data based on the image by using a response generation model; and and outputting the generated response data. Data processing methods.

13. acquiring an image obtained by capturing an image of a user's surroundings by a camera and request data indicating a request by the user regarding the surroundings; generating response data indicating a response corresponding to the request data based on the image by using a response generation model; and A program for causing a computer to execute a process including outputting the generated response data.

Images