Terminal device used for communication between remote locations

Abstract

The object of the present invention is to provide a means for supporting smooth communication between users located in places far from each other. A system (1) of one embodiment of the present invention is a system for supporting remote proxy services, which refers to a client user going to a place (real site) that the client user cannot go to on behalf of the client user and performing things instructed by the client user. The system (1) has a terminal device (11A) used by the client user and a terminal device (11B) used by the client user. The terminal device (11A) displays in real time a dynamic image captured by the terminal device (11B) on the real site. When the client user selects an image included in the dynamic image displayed by the terminal device (11A), the selected image is displayed on the terminal device (11B). The client user can receive instructions related to the object captured by the image while observing the image selected by the client user.

Description

Technical Field

[0001] The present invention relates to a technology for supporting smooth communication between users located in places far away from each other. Background Art

[0002] With the popularization of communication networks such as the Internet or mobile communication networks and the increase in the communication speed of these communication networks, video calls can be made by sending and receiving dynamic image data representing dynamic images captured by a camera (accompanied by sounds picked up by a microphone) between terminal devices used by users (in this application, the term "user" is not limited to humans, but includes devices with artificial intelligence and other devices that can perform intellectual activities similar to humans) located in places far away from each other in substantially real time and reproducing these dynamic images by the terminal devices.

[0003] Utilizing the above-mentioned video call mechanism, the following service (hereinafter referred to as "remote agency service") is proposed: the entrusting user (hereinafter referred to as "entrusting user") finds a person (hereinafter referred to as "entrusted user") who can go to a place (hereinafter referred to as "the actual site") on his behalf that he cannot go to, and the entrusted user who has arrived at the actual site communicates with the entrusting user who is not at the actual site, so as to simulate the entrusting user's experience of going to the actual site, and let the entrusted user perform things that the entrusting user hopes to do at the actual site (for example, purchasing items, conducting negotiations, etc.).

[0004] As a patent document that proposes a mechanism for supporting the above-mentioned remote proxy service, there is Patent Document 1, for example.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2020-198474 Summary of the invention

[0008] Problems to be solved by the invention

[0009] In remote proxy services, it is sometimes difficult for the client user and the client user to communicate smoothly. For example, there is a situation where the client user finds an item of interest in a live video and wants the client user to confirm what the item is. In such a situation, since the client user does not know what the item of interest is, it is not easy to convey the item to the client user.

[0010] In view of the above circumstances, the present invention provides means for supporting smooth communication between users located in places far away from each other.

[0011] Means for solving problems

[0012] The present invention provides a terminal device as a first embodiment, which comprises: a receiving unit, which receives dynamic image data sent in real time from a device of a communication object; a selection image acquisition unit, which acquires image recognition data, which recognizes an image of a selection object among multiple images constituting a dynamic image represented by the dynamic image data received by the receiving unit; and a sending unit, which sends the image recognition data acquired by the selection image acquisition unit or image data representing an image recognized by the image recognition data to the device of the communication object.

[0013] According to the terminal device of the first aspect, the image selected by the user among the images constituting the moving image transmitted from the communication partner device can be notified to the communication partner device. As a result, the user can convey the item of interest to the communication partner user through the image.

[0014] In the terminal device of the first mode, the following structure can also be adopted as the second mode: the terminal device comprises: a display indication unit, which causes the display device to display the dynamic image represented by the dynamic image data received by the receiving unit; and an operation acceptance unit, which accepts the user's operation on the image displayed by the display device, and the selection image acquisition unit acquires image recognition data that identifies the image being displayed by the display device when the operation acceptance unit accepts the user's operation on the dynamic image displayed by the display device as image recognition data of an image of a selection object or image recognition data of an image of a candidate for a selection object.

[0015] According to the terminal device of the second mode, when an image of interest to the user is displayed when a dynamic image sent from the device of the communication partner is reproduced in substantially real time, the image can be selected as an image to be sent to the device of the communication partner or a candidate for an image to be sent to the device of the communication partner by performing a prescribed operation.

[0016] In the terminal device of the first mode, the following structure can also be adopted as the third mode: the terminal device comprises: a storage unit, which stores a portion of the dynamic image data received by the receiving unit corresponding to at least a specified amount in the most recent past; an operation acceptance unit, which accepts a user's operation on the image displayed by the display device; and a display indication unit, which causes the display device to display the dynamic image represented by the dynamic image data received by the receiving unit in real time, and causes the display device to display an image designated by the operation performed by the operation acceptance unit among the images constituting the past dynamic image represented by the dynamic image data stored in the storage unit, wherein the selected image acquisition unit acquires image recognition data of the selected image as image recognition data of an image of a selected object or image recognition data of an image of a candidate for a selected object when the operation acceptance unit accepts an operation by the user to select an image designated by the operation performed by the operation acceptance unit displayed on the display device.

[0017] According to the terminal device of the third mode, even if there is an image of interest in the dynamic image that is essentially played back in real time and sent from the device of the communication partner, the user can later select the image as an image to be sent to the device of the communication partner or as a candidate for an image to be sent to the device of the communication partner, if the user does not perform an operation for selecting the image when the image is displayed.

[0018] In the terminal device of the second or third mode, the following structure can also be adopted as the fourth mode: when the selection image acquisition unit acquires image recognition data of a candidate image of a selection object, the display indication unit causes the display device to display an image recognized based on the image recognition data as a candidate image of the selection object; when the operation acceptance unit accepts an operation by the user to select any image from one or more candidate images of the selection object displayed on the display device, the selection image acquisition unit acquires the image recognition data of the selected image as the image recognition data of the selection object.

[0019] According to the terminal device of the fourth aspect, after the user has selected candidates for an image to be transmitted to the device of the communication partner, the user can select an image to be actually transmitted to the device of the communication partner from among the selected one or more candidates.

[0020] In a terminal device of any of the second to fourth modes, the following structure may also be adopted as the fifth mode: the operation accepting unit accepts an operation specified by the user for any area contained in an image of a selection object identified based on image recognition data acquired by the selection image acquiring unit, and the sending unit sends area data representing the area specified by the user to the device of the communication object.

[0021] According to the terminal device of the fifth aspect, the user can notify the user of the communication partner's device of the region of interest in the image transmitted to the communication partner's device.

[0022] In a terminal device of any of the second to fifth modes, the following structure may also be adopted as the sixth mode: the terminal device has an image recognition unit that recognizes an item captured in an image of a selected object identified based on image recognition data acquired by the selected image acquisition unit, and the display indication unit causes the display device to display an image in which a display object representing a recognition result of the image recognition unit is appended to the image of the selected object identified based on image recognition data acquired by the selected image acquisition unit.

[0023] According to the terminal device of the sixth aspect, the user can easily know what is captured in the image to be transmitted to the device of the communication partner or in the candidate of the image to be transmitted to the device of the communication partner.

[0024] In a terminal device of any of modes 2 to 6, the following structure may also be adopted as mode 7: the display indication unit causes the display device to display a virtual operation button, and when the operation acceptance unit receives an operation of the operation button by the user, the sending unit sends message identification data identifying the message corresponding to the operation button or message data representing the message to the device of the communication object.

[0025] According to the terminal device of the seventh aspect, the user can accurately transmit the message by operating the operation button corresponding to the message that the user wants to transmit to the communication partner's device.

[0026] In the terminal device of the 7th mode, the following structure can also be adopted as the 8th mode: when the operation acceptance unit receives an operation to match the image selected by the user with the operation button, the sending unit matches the message identification data that identifies the message corresponding to the operation button or the message data representing the message and the image identification data acquired by the selection image acquisition unit or the image data representing the image identified by the image identification data and sends them to the device of the communication object.

[0027] According to the terminal device of the eighth aspect, it is possible to convey an image and a message associated with the image to a user of a communication partner device.

[0028] In the terminal device of the 7th or 8th mode, the following structure can also be adopted as the 9th mode: the message data is text data representing a sentence written in the language used by the user of the communication object device, or voice data representing the sound of speaking the sentence.

[0029] According to the terminal device of the ninth aspect, even when the language used by the user is different from the language used by the user of the communication partner device, it is possible to convey a message that the other party user can understand.

[0030] In the terminal device of any of the seventh to ninth aspects, the following configuration may be adopted as a tenth aspect: the message data is data indicating a status of the user.

[0031] According to the terminal device of the tenth aspect, the user can notify the user of the device of the communication partner of his / her own status.

[0032] In the terminal device of the seventh or eighth aspect, the following structure may be adopted as an eleventh aspect: the message data is action instruction data for instructing the device of the communication partner to perform a physical action.

[0033] According to the terminal device of the eleventh aspect, the user can remotely control the physical operation of the device of the communication partner.

[0034] Furthermore, the present invention provides a terminal device as a twelfth mode, which comprises: a dynamic image acquisition unit, which acquires dynamic image data representing a dynamic image captured by a photographic device; a sending unit, which sends the dynamic image data acquired by the dynamic image acquisition unit to a device of a communication object in real time; a storage unit, which stores at least a portion of the dynamic image data acquired by the dynamic image acquisition unit corresponding to a prescribed amount in the most recent past; a receiving unit, which receives image recognition data for identifying an image sent from the device of the communication object; and a display indication unit, which causes a display device to display an image recognized according to the image recognition data received by the receiving unit, from a plurality of images constituting the dynamic image represented by the dynamic image data stored in the storage unit.

[0035] According to the terminal device of the twelfth aspect, the user can view and confirm the image selected by the user of the communication partner device among the plurality of images constituting the moving image transmitted to the communication partner device.

[0036] In the terminal device of the 12th mode, the following structure can also be adopted as the 13th mode: when the receiving unit receives message identification data of an identification message sent corresponding to the image identification data or message data representing the message from the device of the communication object, the display indication unit causes the display device to display the image recognized based on the image recognition data, and display the message recognized based on the message identification data or the message represented by the message data.

[0037] According to the terminal device of the thirteenth aspect, the user can know the message of the user related to the image selected by the user of the communication partner device.

[0038] In the terminal device of the 13th mode, the following structure can also be adopted as the 14th mode: the terminal device has a translation unit, and when the translation unit obtains text data representing a sentence in a language that is not used by the user of the device, the translation unit obtains text data representing a sentence translated from the text data into the language used by the user, and when the message indicated by the message data represents a sentence in a language that is not used by the user of the device, the display indication unit causes the display device to display the sentence after the sentence indicated by the text data obtained by the translation unit is translated.

[0039] According to the terminal device of the fourteenth aspect, the user can read and understand the message from the user even if the user uses a language different from that of the user of the communication partner device.

[0040] In the terminal device of the 13th or 14th mode, the following structure can also be adopted as the 15th mode: the terminal device comprises: an interpretation unit which, when obtaining text data representing a sentence in a language that is not a language used by a user of the device, obtains voice data representing the sound of speaking a sentence after translating the sentence shown in the text data into the language used by the user; and a voice instruction unit which, when the message shown in the message data represents a sentence in a language that is not a language used by the user of the device, causes a voice device to emit the sound of speaking the sentence after the sentence is translated from the voice data obtained by the interpretation unit.

[0041] According to the terminal device of the fifteenth aspect, the user can hear and understand a message from the user even when the user uses a language different from that of the user of the communication partner device.

[0042] In a terminal device of any of the 12th and 15th modes, the following structure may also be adopted as the 16th mode: the terminal device comprises: a position acquisition unit which acquires position data indicating the position of the device; and a map acquisition unit which acquires map data indicating a map of an area including the position of the device, the display indication unit causing the display device to display an image in which a display object indicating the position indicated by the position data corresponding to the shooting timing of the image recognized by the image recognition data received by the receiving unit is added to the map represented by the map data.

[0043] According to the terminal device of the sixteenth aspect, the user can know the location where the image selected by the user of the communication partner device was taken.

[0044] In the terminal device of the 16th mode, the following structure can also be adopted as the 17th mode: the dynamic image acquisition unit acquires dynamic image data and shooting direction data indicating the shooting directions of each of the multiple images constituting the dynamic image represented by the dynamic image data, and the display indication unit causes the display device to display an image after a display object indicating the shooting direction indicated by the shooting direction data corresponding to the shooting timing of the image recognized according to the image recognition data received by the receiving unit is added to the map represented by the map data.

[0045] According to the terminal device of the seventeenth aspect, the user can know the shooting direction of the image selected by the user of the communication partner device.

[0046] In a terminal device of any of the 12th to 17th modes, the following structure may also be adopted as the 18th mode: the terminal device comprises: a position acquisition unit, which acquires position data indicating the position of the device; and a moving path acquisition unit, which acquires moving path data indicating a moving path from the position of the device indicated by the position data last acquired by the position acquisition unit to the position indicated by the position data corresponding to the shooting timing of the image recognized according to the image recognition data received by the receiving unit, and the display indication unit causes the display device to display a display object indicating the moving path indicated by the moving path data.

[0047] According to the terminal device of the eighteenth aspect, the user can know the movement route for moving from the current position to the place where the image selected by the user of the communication partner device is captured.

[0048] In the terminal device of the 12th mode, the following structure can also be adopted as the 19th mode: the terminal device has an action unit, and when the receiving unit receives message identification data for identifying an action indication indicating a physical action or message data indicating the action indication sent corresponding to the image recognition data from the device of the communication object, the action unit performs a physical action in accordance with the action indication identified by the message identification data or the action indication indicated by the message data.

[0049] According to the terminal device of the nineteenth aspect, a user of a communication partner device can cause the terminal device to perform an action that the user wants to perform on site by remotely operating the terminal device.

[0050] In the terminal device of the 12th mode, the following structure can also be adopted as the 20th mode: the terminal device has: a position acquisition unit, which acquires position data indicating the position of the device; a moving path acquisition unit, which acquires movement path data indicating the movement path from the position of the device indicated by the position data last acquired by the position acquisition unit to the position indicated by the position data corresponding to the shooting timing of the image recognized according to the image recognition data received by the receiving unit; and a moving unit, which moves according to the movement path indicated by the movement path data.

[0051] According to the terminal device of the twentieth aspect, the user of the communication partner device can move the terminal device to the place where the image was captured by sending the image selected by the user to the communication partner device.

[0052] Furthermore, the present invention provides a program as a 21st method, which is used to cause a computer to perform the following processing: receiving dynamic image data sent in real time from a device of a communication object; acquiring image recognition data, which recognizes an image of a selected object among multiple images constituting a dynamic image represented by the received dynamic image data; and sending the acquired image recognition data or image data representing an image recognized by the image recognition data to the device of the communication object.

[0053] According to the program of the twenty-first aspect, the terminal device of the first aspect can be realized by a computer.

[0054] Furthermore, the present invention provides a program as a 22nd method, which is used to cause a computer to perform the following processing: acquiring dynamic image data representing a dynamic image captured by a photographic device; sending the acquired dynamic image data to a device of a communication object in real time; storing a portion of the acquired dynamic image data corresponding to at least a specified amount in the most recent past; receiving image recognition data for identifying an image sent from the device of the communication object; and causing a display device to display an image recognized based on the received image recognition data from a plurality of images constituting a dynamic image represented by the stored dynamic image data.

[0055] According to the program of the 22nd aspect, the terminal device of the 12th aspect can be realized by a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a diagram showing the overall structure of a system according to one embodiment.

[0057] Figure 2 It is a diagram showing the structure of a computer for realizing a terminal device according to one embodiment.

[0058] Figure 3 This is a diagram showing a functional structure of a terminal device according to one embodiment.

[0059] Figure 4 This is a diagram showing a screen displayed on a terminal device used by a requesting user according to one embodiment.

[0060] Figure 5 This is a diagram showing a screen displayed on a terminal device used by a request target user according to one embodiment. DETAILED DESCRIPTION

[0061] [Implementation Method]

[0062] Hereinafter, a system 1 according to an embodiment of the present invention will be described. The system 1 is a system that functions as a platform for remote proxy services. Figure 1 1 is a diagram showing the overall structure of the system 1. The system 1 includes a terminal device 11A, a terminal device 11B, a management server device 12, a route search server device 13, a translation server device 14, a speech synthesis server device 15, and a speech recognition server device 16. These devices constituting the system 1 can communicate with each other via a network 9.

[0063] The terminal device 11A is a terminal device used by a user who requests the remote proxy service. The terminal device 11B is a terminal device used by a user who requests the remote proxy service. Figure 1In FIG. 1 , only one terminal device 11A and one terminal device 11B are shown, but in reality, the number of these terminal devices varies depending on the number of users utilizing the system 1 .

[0064] The management server device 12 manages users of the system 1 , supports matching between client users and client users, and establishes a communication connection between the client user's terminal device 11A and the client user's terminal device 11B.

[0065] The route search server device 13 is a server device as follows: when receiving a request including location data indicating the location of a departure point and a destination from any device, it determines a moving path from the departure point to the destination based on the request, and sends moving path data indicating the determined moving path and map data indicating a map of an area including the determined moving path to the device of the request source.

[0066] The translation server device 14 is a server device as follows: when a request including text data representing a sentence and language designation data representing a translation target language of the sentence is received from any device, the sentence represented by the text data is translated into the language represented by the language designation data according to the request, and the text data representing the translated sentence is sent to the device of the request source.

[0067] The speech synthesis server device 15 is a server device that, when receiving a request including text data representing a sentence from any device, synthesizes speech representing the sentence represented by the text data in accordance with the request and transmits speech data representing the synthesized speech to the device of the request source.

[0068] The speech recognition server device 16 is a server device that, when receiving a request including speech data representing speech from any device, recognizes the speech represented by the speech data in response to the request and transmits text data representing the recognized sentence to the requesting device.

[0069] The management server device 12, the route search server device 13, the translation server device 14, the speech synthesis server device 15, and the speech recognition server device 16 are each implemented by one or a plurality of computers operating in coordination. The computer used to implement these server devices is, for example, a general server device computer having a processor, a memory, and a communication interface.

[0070] The terminal device 11A is implemented by executing a program for the terminal device of the client user according to the present embodiment by a computer for terminal devices such as a desktop PC (Personal Computer), a notebook PC, a tablet PC, a smart phone, etc. The terminal device 11A is used by the client user, so the computer for implementing the terminal device 11A does not necessarily have to be portable. The computer for implementing the terminal device 11A has a built-in microphone (an example of a sound pickup device) for picking up the voice of the client user, a keyboard (an example of an operating device) for accepting the operation of the client user, a display (an example of a display device) for displaying various information to the client user, a speaker (an example of a sound generating device) for emitting sound to the client user, a camera (an example of a photographic device) for photographing the posture of the client user, etc., and a GNSS (Global Navigation Satellite System) unit (an example of a position measuring device) for measuring the position of the device on the earth (for example, latitude, longitude), or is connected to these devices and can transmit data between these devices.

[0071] The terminal device 11B is implemented by executing the processing of the program for the terminal device of the client user according to the present embodiment by a computer for a terminal device such as a tablet PC or a smart phone. Since the terminal device 11B is used by the client user who needs to move in the field, the computer used to implement the terminal device 11B needs to be portable. The computer used to implement the terminal device 11B has a built-in microphone (an example of a sound pickup device) for picking up the voice of the client user, a touch panel (an example of an operation device) for accepting the operation of the client user, a display (an example of a display device) for displaying various information to the client user, a speaker (an example of a sound emitting device) for emitting sound to the client user, a camera (an example of a photographic device) for photographing the surroundings of the client user, a GNSS unit (an example of a position measuring device) for measuring the position of the device on the earth (for example, latitude, longitude), and an orientation sensor (an example of an orientation measuring device) for measuring the orientation of the shooting direction of the camera of the device, or is connected to these devices and can transmit data between these devices.

[0072] As described above, the specifications required for the computer used to implement the terminal device 11A and the computer used to implement the terminal device 11B are different, but in the following description, for convenience, it is assumed that the computer used to implement the terminal device 11A and the computer used to implement the terminal device 11B have the same specifications.

[0073] Figure 21 is a diagram showing the structure of a computer 10 for realizing a terminal device 11A or a terminal device 11B. The computer 10 includes a processor 101 for performing various data processing according to a program, a memory 102 for storing various data including the program, a communication interface 103 for performing data communication with other devices, a touch screen 104 serving as a user interface, a microphone 105 for picking up surrounding sounds, a speaker 106 for emitting sounds, a camera 107 for taking pictures, a GNSS unit 108 for measuring the position of the device, and an orientation sensor 109 for measuring the orientation of the shooting direction of the camera 107.

[0074] The touch screen 104 is, for example, a device in which a liquid crystal display and a touch panel are stacked, and displays images and the like for the user and receives touch operations performed by the user on the computer 10 .

[0075] Although the functions of the terminal device 11A and the terminal device 11B are partially different, most of their functions are common. Therefore, in the following description, for convenience, it is assumed that the functional structure of the terminal device 11A is common to the functional structure of the terminal device 11B. For example, the terminal device 11A may not have the direction measuring device 1105 described below. And, for example, the receiving unit 111 of the terminal device 11A and the receiving unit 111 of the terminal device 11B receive different data. These differences can be explained later using Figure 4 and Figure 5 The description of the operations of the terminal device 11A and the terminal device 11B will be made clear.

[0076] Figure 3 1 is a diagram showing the functional structure of the terminal device 11A or the terminal device 11B. The storage unit 110 is realized by the memory 102 and stores various data. The receiving unit 111 is realized by the communication interface 103 and receives various data from other devices.

[0077] The display device 1101 is implemented by the touch screen 104 and displays images. The display instruction unit 112 is implemented by the processor 101 and causes the display device 1101 to display images. The sound device 1102 is implemented by the speaker 106 and emits sounds. The sound instruction unit 113 is implemented by the processor 101 and causes the sound device 1102 to emit sounds.

[0078] The operation device 1103 is implemented by the touch screen 104 and physically receives the user's operation. The operation receiving unit 114 is implemented by the processor 101 and receives the user's operation in software by determining the content of the physical operation received by the operation device 1103. The selection image acquisition unit 115 is implemented by the processor 101 and acquires image recognition data that recognizes the image of the selection object among the multiple images (still images) constituting the dynamic image represented by the dynamic image data received by the receiving unit 111 from other devices.

[0079] The transmission unit 116 is implemented by the communication interface 103 and transmits various data to other devices.

[0080] The photographing device 1104 is implemented by the camera 107, and takes pictures and generates dynamic image data representing a dynamic image. The dynamic image acquisition unit 117 acquires the dynamic image data generated by the photographing device 1104. The direction measuring device 1105 is implemented by the direction sensor 109, and measures the photographing direction of the photographing device 1104, and generates photographing direction data representing the measurement result. The dynamic image acquisition unit 117 acquires the photographing direction data generated by the direction measuring device 1105, and associates it with the dynamic image data acquired from the photographing device 1104.

[0081] The sound pickup device 1106 is implemented by the microphone 105, picks up the surrounding voice, and generates voice data mainly representing the user's voice. The voice acquisition unit 118 acquires the voice data generated by the sound pickup device 1106. In addition, the dynamic image data acquired by the dynamic image acquisition unit 117 (dynamic image data corresponding to the shooting direction data) and the voice data acquired by the voice acquisition unit 118 are synthesized into synchronized dynamic image data with voice (hereinafter referred to as "dynamic image data").

[0082] The position measuring device 1107 is implemented by the GNSS unit 108 , measures the position of the device, and generates position data indicating the measurement result. The position acquiring unit 119 is implemented by the processor 101 , and acquires the position data generated by the position measuring device 1107 .

[0083] In the present embodiment, it is assumed as an example that the requesting user speaks Japanese and the requesting target user speaks English.

[0084] In this case, the sending unit 116 of the terminal device 11B sends a request including the text data representing the Japanese sentence received from the terminal device 11A to the translation server device 14, and the receiving unit 111 of the terminal device 11B receives the text data representing the English sentence sent from the translation server device 14 as a response to the request. That is, the sending unit 116 and the receiving unit 111 of the terminal device 11B function as a translation unit that, when text data representing a sentence in a language other than the language used by the user of the terminal device is obtained, obtains text data representing a sentence translated from the text data into the language used by the user.

[0085] Furthermore, the sending unit 116 of the terminal device 11B sends a request including the text data representing the English sentence received by the receiving unit 111 of the terminal device 11B from the translation server device 14 to the speech synthesis server device 15, and the receiving unit 111 of the terminal device 11B receives the speech data representing the voice speaking the English sentence sent from the speech synthesis server device 15 as a response to the request. That is, the sending unit 116 and the receiving unit 111 of the terminal device 11B play the role of the following interpretation unit: when the text data representing the sentence in the language other than the language used by the user of the present device is obtained, the speech data representing the voice speaking the sentence translated from the text data into the language used by the user is obtained.

[0086] In addition, the sending unit 116 of the terminal device 11B sends a request to the route retrieval server device 13, which includes the latest location data among the location data acquired by the location acquisition unit 119 as the location data of the departure place and includes the location data corresponding to the shooting time of the image recognized according to the image recognition data received from the terminal device 11A by the receiving unit 111 of the terminal device 11B as the location data of the destination, and the receiving unit 111 of the terminal device 11B receives the moving path data and map data sent from the route retrieval server device 13 as a response to the request.

[0087] That is, the sending unit 116 and the receiving unit 111 of the terminal device 11B play the role of a map acquisition unit that acquires map data of a map representing an area including the location of the terminal device. In addition, the sending unit 116 and the receiving unit 111 of the terminal device 11B play the role of a moving path acquisition unit that acquires a moving path representing the position of the terminal device indicated by the position data last acquired by the position acquisition unit 119 of the terminal device 11B to the position indicated by the position data corresponding to the shooting timing of the image recognized by the image recognition data received by the receiving unit 111 of the terminal device 11B.

[0088] Next, the operation of the terminal device 11A and the terminal device 11B will be described. In the following description, the terminal device 11A and the terminal device 11B establish a communication connection according to an instruction of the management server device 12, for example.

[0089] Figure 4 1 is a diagram illustrating a screen displayed on the terminal device 11A (hereinafter referred to as a "client screen"). In the area A01 of the client screen, a moving image represented by moving image data transmitted from the terminal device 11B to the terminal device 11A is displayed substantially in real time.

[0090] Displayed in the text box in the area A02 of the client screen is a Japanese sentence represented by text data received from the translation server device 14 in response to the terminal device 11A transmitting a request including message data received from the terminal device 11B to the translation server device 14 .

[0091] When a sentence is displayed in the text box in the area A02, a voice speaking the sentence is emitted from the sound generating device 1102 of the terminal device 11A. The voice emitted by the sound generating device 1102 is a voice represented by voice data received from the speech synthesis server device 15 in response to the terminal device 11A sending a request including text data representing the sentence displayed in the text box in the area A02 to the speech synthesis server device 15. The client user can make the sound generating device 1102 emit the voice repeatedly by touching the "playback" button in the area A02.

[0092] In area A03 of the client screen, an image selected by the client user from among the multiple images constituting the dynamic image displayed in area A01 is displayed. The client user can select an image displayed in area A01 by touching area A01. The image selected in this way is displayed in area A03.

[0093] Furthermore, the client user can display a thumbnail P03 of an image corresponding to the position of the operation button P02 in the left-right direction among images constituting the dynamic image displayed in the area A01 within a specified time in the recent past near the operation button P02 by dragging the operation button P02 on the time bar P01 displayed on the dynamic image in the left-right direction in the area A01. Therefore, the storage unit 110 of the terminal device 11A stores a portion corresponding to the specified time in the recent past among the dynamic image data received by the receiving unit 111 from the terminal device 11B. The client user can select an image corresponding to the thumbnail P03 displayed when the touch operation is performed by touching the thumbnail P03. The image selected in this way is displayed in the area A03.

[0094] In area A04 of the client screen, thumbnails of images selected by the client user and displayed in area A03 and dragged and dropped from area A03 to area A04 by the client user are displayed. The images displayed as thumbnails in area A04 are images selected as candidates for selection by the client user. The client user can display an image corresponding to the thumbnail in area A03 by dragging and dropping any thumbnail from one or more thumbnails displayed in area A04 from area A04 to area A03.

[0095] A plurality of virtual operation buttons are displayed in the area A05 of the client screen. The four operation buttons on the left among these operation buttons are operation buttons for the client user to convey a fixed message to the client user. Figure 4 The four operation buttons displayed in the figure correspond to messages such as "What is this?", "What is the price?", "Let me see", and "Please buy" from the left.

[0096] The client user can register a plurality of groups of these operation buttons in advance in the terminal device 11A, and can switch the groups of these operation buttons by touching the up arrow button or the down arrow button displayed in the area A05. Various groups of operation buttons are stored in the management server device 12, and the client user can download a group including the operation buttons that are frequently used in the remote proxy service from these groups to the terminal device 11A. In addition, the client user can create a new group of operation buttons, or change the message corresponding to the operation button included in the group downloaded from the management server device 12 to the terminal device 11A.

[0097] When the client user touches any of the four operation buttons on the left side displayed in the area A05 , the terminal device 11A transmits message data indicating a message corresponding to the touched operation button to the terminal device 11B.

[0098] When the client user drags and drops the image displayed in area A03 onto any of the four operation buttons displayed on the left side of area A05, the terminal device 11A associates image recognition data for identifying the dragged image, area data indicating the area in the image touched by the client user when dragging and dropping the image (for example, an area of ​​a predetermined size and shape centered on the touched position), and message data indicating a message corresponding to the operation button of the drag-and-drop destination, and transmits them to the terminal device 11B. In addition, as the image recognition data, for example, data indicating the frame number of the image constituting the dynamic image shown in the dynamic image data can be used, but it is not limited to this.

[0099] When the client user touches the "Send" button displayed on the far right of area A05, the terminal device 11A transmits image recognition data identifying the image displayed in area A03 when the touch operation is performed to the terminal device 11B. In addition, when the client user drags and drops the image displayed in area A03 onto the "Send" button in area A05, the terminal device 11A matches the image recognition data identifying the dragged and dropped image with the area data indicating the area in the image touched by the client user when dragging and dropping the image, and transmits them to the terminal device 11B.

[0100] In area A06 of the client screen, a text box P04 for inputting a non-fixed message that the client user wants to send to the client user is displayed. After the client user touches the "voice" button in area A06, he speaks a message to the terminal device 11A, thereby inputting a text representing the spoken message into the text box P04. At this time, the terminal device 11A sends a request including the voice data acquired by the voice acquisition unit 118 to the voice recognition server device 16, and displays the content of the text data sent from the voice recognition server device 16 as a response to the request in the text box P04.

[0101] When the client user touches the "keyboard" button in the area A06, the terminal device 11A displays a virtual keyboard. The client user can input a message into the text box P04 by touching the displayed virtual keyboard.

[0102] When the client user inputs a message into the text box P04 and then touches the "Send" button in the area A06, the terminal device 11A sends the message data representing the message input into the text box P04 to the terminal device 11B. In addition, when the client user drags and drops the image displayed in the area A03 onto the "Send" button in the area A06 or into the text box P04, the terminal device 11A associates the image recognition data for identifying the dragged and dropped image, the area data representing the area in the image touched by the client user when dragging and dropping the image, and the message data representing the message input into the text box P04, and sends them to the terminal device 11B.

[0103] In area A07 of the client screen, virtual operation buttons are displayed for the client user to instruct the client user on the direction of movement, etc. The up arrow button in area A07 corresponds to the message data indicating the message "Please move forward", and the down arrow button corresponds to the message data indicating the message "Please move backward". In addition, the right-rotating arrow in area A07 corresponds to the message data indicating the message "Please rotate right", and the left-rotating arrow corresponds to the message data indicating the message "Please rotate left". When the client user touches any of these operation buttons, the terminal device 11A sends the message data corresponding to the touched operation button to the terminal device 11B.

[0104] A map showing the current position of the client user (terminal device 11B) is displayed in area A08 of the client screen. Therefore, the terminal device 11A receives the position data showing the current position of the terminal device 11B sent from the terminal device 11B at a predetermined time interval. The terminal device 11A sends a request including the position data received from the terminal device 11B to the route search server device 13, and displays the map in area A08 using the map data sent from the route search server device 13 as a response to the request. The terminal device 11A adds a circle mark showing the current position of the client user shown in the latest position data sent from the terminal device 11B to the map, determines the moving direction of the client user based on the temporal change of the position of the client user shown in the position data sent from the terminal device 11B, adds an arrow showing the determined moving direction to the map, and displays the map in area A08.

[0105] In area A09 of the client screen, radio buttons are displayed for the client user to choose whether to convey his or her current location "blurred" or "clearly" to the client user. When the client user selects "blurred" in area A09, the terminal device 11A sends location data that ambiguously represents the current location of the device to the terminal device 11B at a specified time interval. On the other hand, when the client user selects "clear" in area A09, the terminal device 11A sends location data that clearly represents the current location of the device to the terminal device 11B at a specified time interval. Here, location data that ambiguously represents the current location refers to, for example, data that represents the current location with an accuracy of hundreds of meters. In contrast, location data that clearly represents the current location refers to, for example, data that represents the current location with an accuracy of several meters.

[0106] In area A10 of the client screen, a list box is displayed for conveying the client user's current status to the client user. In the list box on the upper side of area A10, multiple icons representing emotions such as "happy", "sad", "angry" are displayed as options. In the list box on the lower side of area A10, multiple icons representing physical states such as "still have a lot of physical strength", "still have some physical strength", "little physical strength left", "no physical strength at all" are displayed as options. When the client user changes the icon representing his current status in these list boxes, the terminal device 11A sends the icon identification data identifying the changed icon to the terminal device 11B.

[0107] An icon representing the current status of the client user is displayed in area A11 of the client screen. The client user can change the icon representing the current status of the user in the list box displayed on the screen of the terminal device 11B in the same way as the client user. When these icons are changed, icon identification data for identifying the changed icons is sent from the terminal device 11B to the terminal device 11A. The terminal device 11A displays the icon identified by the icon identification data sent from the terminal device 11B in area A11.

[0108] Figure 5 The figure is a diagram showing an example of a screen (hereinafter referred to as "the entrusted object screen") displayed by the terminal device 11B. The entrusted object screen includes the same items as the entrusted party screen ( Figure 4 ) is the same or similar area for displaying the displayed object. Figure 5 In the figure, an area on the client screen that displays a display object that is the same as or similar to that on the client screen is labeled with a number with the same last two digits.

[0109] In the area B01 of the request target screen, a moving image represented by the moving image data transmitted from the terminal device 11A to the terminal device 11B is displayed substantially in real time.

[0110] In the text box in the area B02 of the request target screen, an English sentence represented by text data received from the translation server device 14 in response to the terminal device 11B transmitting a request including the message data received from the terminal device 11A to the translation server device 14 is displayed.

[0111] When a sentence is displayed in the text box in area A02 of the client screen, similarly to when a sentence is displayed in the text box in area B02 of the client screen, a voice speaking the sentence is emitted from the sound emitting device 1102 of the terminal device 11B. The client user can make the sound emitting device 1102 emit the voice repeatedly by touching the "playback" button in area B02.

[0112] A plurality of virtual operation buttons are displayed in area B05 of the client screen. These operation buttons are operation buttons for the client user to convey a fixed message to the client user, and each operation button corresponds to a different message. The operation buttons displayed in area B05 can be customized by the client user in the same way as the operation buttons displayed in area A05 of the client screen (except the "Send" button).

[0113] When the request object user touches any of the operation buttons displayed in the area B05 , the terminal device 11B transmits message data indicating a message corresponding to the touched operation button to the terminal device 11A.

[0114] In area B06 of the object of entrustment screen, a text box for inputting a non-fixed message that the object of entrustment user wants to send to the entrusting user is displayed. In area B06, it is not possible to drag and drop an image in the text box. Apart from this, the functions of the displayed objects included in area B06 of the object of entrustment screen are the same as those of the displayed objects included in area A06 of the entrusting user screen, so their description is omitted.

[0115] A map showing the current location of the client user (terminal device 11A) is displayed in area B08 of the client screen. Therefore, the terminal device 11B receives the location data showing the current location of the terminal device 11A sent from the terminal device 11A at a predetermined time interval. The terminal device 11B sends a request including the location data received from the terminal device 11A to the route search server device 13, and displays the map in area B08 using the map data sent from the route search server device 13 as a response to the request. The method by which the terminal device 11B generates an image of the map displayed in area B08 is the same as the method by which the terminal device 11A generates an image of the map displayed in area A08, so the description thereof is omitted.

[0116] In area B09 of the entrusted object screen, radio buttons are displayed for the entrusted object user to select whether to convey his / her current position "blurred" or "clearly" to the entrusting party user. The functions of the radio buttons contained in area B09 of the entrusted object screen are the same as those of the radio buttons contained in area A09 of the entrusting party screen, so their description is omitted.

[0117] A list box for notifying the current status of the client user to the client user is displayed in area B10 of the client screen. The function of the list box included in area B10 of the client screen is the same as that of the list box included in area A10 of the client screen, so its description is omitted.

[0118] Icons indicating the current status of the requesting user are displayed in area B11 of the requesting screen. The icons included in area B11 of the requesting screen have the same functions as those included in area A11 of the requesting screen, and thus description thereof is omitted.

[0119] In area B12 of the client screen, an image recognized based on the image recognition data sent from the terminal device 11A to the terminal device 11B is displayed. That is, the image displayed in area B12 is an image (still image) selected by the client user from the dynamic images. When the image recognition data received by the terminal device 11B from the terminal device 11A and the area data are matched, the terminal device 11B adds a display object ( Figure 5 The image after the dotted circle) is displayed in area B12.

[0120] When the image recognition data received by the terminal device 11B from the terminal device 11A is associated with the message data, an English sentence represented by the text data received from the translation server device 14 in response to the terminal device 11B sending a request including the message data to the translation server device 14 is displayed in the text box in the area B13 of the client screen. When the message is displayed in the text box in the area B13, the terminal device 11B speaks the voice of the message. The client user can listen to the voice repeatedly by touching the "playback" button in the area B13.

[0121] In area B14 of the request object screen, a moving path from the current position of the request object user (terminal device 11B) to the shooting position of the image displayed in area B12 and a map indicating the shooting direction of the image are displayed. The terminal device 11B sends a request to the route search server device 13, the request including the position data last acquired by the position acquisition unit 119 as the position data of the departure place and the position data acquired by the position acquisition unit 119 at the shooting time of the image displayed in area B12 as the position data of the destination, and the terminal device 11B receives the map data and moving path data sent from the route search server device 13 as a response to the request. The terminal device 11B generates an image in which a dotted line indicating the moving path indicated by the moving path data received from the route search server device 13, a circle mark indicating the current position of the request object user, an arrow indicating the moving direction of the request object user, and a sector mark indicating the shooting direction of the image displayed in area B12 are added to the map shown in the map data received from the route search server device 13, and displays the image in area B12.

[0122] For example, when the client user receives a request from the client user to purchase an item captured by the image displayed in area B12 (e.g., an item captured by the area indicated by the dotted circle in area B12), the client user can easily reach the location where the item indicated by the client user is located by moving along the moving path displayed in area B14. In addition, the client user can easily identify the item indicated by the client user while looking at the image displayed in area B12 at the location reached.

[0123] According to the system 1 of the above embodiment, the requesting user can easily convey to the requesting user an object that he wants the requesting user to do something by showing the requesting user an image among the moving images sent from the requesting user.

[0124] Furthermore, according to the system 1 , the requesting user can quickly and clearly convey a message to the requesting user by operating the operation button.

[0125] Furthermore, according to the system 1, even if the client user and the client user use different languages, communication can be performed between the client user and the client user.

[0126] Furthermore, according to the system 1, the requesting user can easily check the current position and current moving direction of the requesting user, and can therefore easily instruct the requesting user in the direction in which he or she wants the requesting user to move next.

[0127] In addition, according to the system 1, the client user can send the image and the message to the client user by a simple operation such as dragging and dropping the image selected from the dynamic image onto the operation button corresponding to the message or the text box in which the message is input. In addition, when dragging and dropping the image onto the operation button, etc., the client user can touch the object captured by the image, thereby easily conveying the object to the client user.

[0128] [Modifications]

[0129] The system 1 described above can be modified in various ways within the scope of the technical concept of the present invention. Modifications are shown below. Two or more of the following modifications can also be appropriately combined.

[0130] (1) Instead of image recognition data of an image selected by the client user from the moving images, image data representing the image may be sent from the terminal device 11A to the terminal device 11B. In this case, the terminal device 11B displays the image represented by the image data received from the terminal device 11A in the area B12 of the client screen.

[0131] (2) In the above embodiment, the terminal device 11A stores at least a portion of the dynamic image data received from the terminal device 11B corresponding to a prescribed time in the most recent past. The portion of the dynamic image data received from the terminal device 11B that needs to be stored in advance by the terminal device 11A only needs to correspond to a prescribed amount, and does not necessarily need to correspond to a prescribed time. For example, the terminal device 11A may store at least a portion of the dynamic image data received from the terminal device 11B corresponding to a prescribed amount of data in the most recent past.

[0132] (3) Alternatively, the terminal device 11A may display the result of recognizing the object captured by the image in the area A03 selected by the client user and displayed on the client screen on the image in the area A03. In this case, the terminal device 11A has an image recognition unit that recognizes the object captured by the image displayed in the area A03 using a known image recognition technology, and displays the image with the display object indicating the recognition result of the image recognition unit added in the area A03. In addition, the image recognition process may be performed by, for example, a server device that provides image recognition services and is connected to the system 1.

[0133] (4) Instead of sending and receiving message data representing a message between the terminal device 11A and the terminal device 11B, message identification data identifying a message may be sent and received. In this case, the terminal device 11A and the terminal device 11B store in advance message data representing a fixed message in correspondence with the message identification data common to the terminal device 11A and the terminal device 11B. For example, the terminal device 11A sends the message identification data of a message specified by the entrusting user to the terminal device 11B, and the terminal device 11B displays the message represented by the message data identified by the message identification data received from the terminal device 11A. Similarly, the terminal device 11B sends the message identification data of a message specified by the entrusting user to the terminal device 11A, and the terminal device 11A displays the message represented by the message data identified by the message identification data received from the terminal device 11B.

[0134] (5) In the above-described embodiment, the terminal device 11A and the terminal device 11B receive message data representing a sentence from the other party's device, and obtain voice data representing the voice speaking the sentence represented by the message data from the speech synthesis server device 15. Instead of this, or in addition to this, voice data representing the voice speaking the sentence of the message may be transmitted and received between the terminal device 11A and the terminal device 11B as the message data.

[0135] (6) In the above embodiment, when the client user is a device such as a robot having an action unit that performs physical actions, the terminal device 11A may transmit action instruction data as message data to the terminal device 11B, and the action instruction data may instruct the client user as a device to perform physical actions. In this case, the client user as a device performs actions according to the action instructions indicated by the message data. In addition, when the client user is a device such as a robot having a moving unit that performs movement, the client user as a device may move according to the moving path indicated by the moving path data.

[0136] (7) In the above embodiment, at least a part of the request made by the terminal device 11A to the route search server device 13, the translation server device 14, and the speech synthesis server device 15 may also be made by the terminal device 11B. In this case, the terminal device 11B only needs to send the data obtained from these server devices to the terminal device 11A. Similarly, in the above embodiment, at least a part of the request made by the terminal device 11B to the route search server device 13, the translation server device 14, and the speech synthesis server device 15 may also be made by the terminal device 11A. In this case, the terminal device 11A only needs to send the data obtained from these server devices to the terminal device 11B.

[0137] (8) In the above embodiment, at least part of the processing performed by the route search server 13, the translation server 14, the speech synthesis server 15, and the speech recognition server 16 may be performed by the terminal 11A or the terminal 11B.

[0138] (9) In the above embodiment, the client screen ( Figure 4 ) is an example, and operation buttons for instructing other actions may be displayed in area A07. For example, an operation button corresponding to a message "Please jump" or an operation button corresponding to a message "Please fast forward" or "Please slow forward" may be displayed in area A07.

[0139] (10) In the above-mentioned embodiment, the client user and the client user are each one person, but at least one of the client user and the client user may be multiple persons. In this case, the number of terminal devices that simultaneously perform data communication for performing the on-site agency service is three or more, and therefore the management server device 12 may relay the data communication between these terminal devices.

[0140] Furthermore, when there are multiple entrusting users, a problem may arise if the multiple entrusting users simultaneously give contradictory instructions to the entrusting user. To avoid such a problem, only one terminal device 11A among the multiple terminal devices 11A may have instruction authority, and the transmission of image recognition data or message data from the terminal device 11A that does not have instruction authority to the terminal device 11B may be restricted.

[0141] (11) In the above embodiment, the client user selects the client screen ( Figure 4 ) in area A10. Alternatively, for example, the terminal device 11A may infer the status of the client user based on an image of the client user's face captured by the terminal device 11A and the client user's voice picked up by the terminal device 11A, and the terminal device 11A automatically selects an icon in area A10 based on the inference result. Similarly, the terminal device 11B may infer the status of the client user based on an image of the client user's face captured by the terminal device 11B and the client user's voice picked up by the terminal device 11B, and the terminal device 11B automatically selects an icon in area A10 based on the inference result. Figure 5 ) in area B10.

[0142] Furthermore, the categories of the user's status displayed in the area A10 or the area B10 are not limited to mood and physical strength.

[0143] (12) Alternatively, the entrusting user can use the system 1 to entrust the entrusting target user with electronic settlement using an image code such as a QR code (registered trademark) or a bar code on the spot. For example, the terminal device 11A sends the image code for electronic settlement of the entrusting user to the terminal device 11B, and the clerk of the store on the spot uses a code reader to read the image code displayed on the terminal device 11B, so that the store can receive payment from the entrusting user.

[0144] In addition, the terminal device 11B sends an image of the image code for electronic settlement of the actual store to the terminal device 11A, and the client user performs an operation of paying the specified amount to the store identified by the image code displayed on the terminal device 11A, so that the store can receive payment from the client user. In this case, the store confirms that the client user has completed the processing of paying the specified fee through the image sent from the terminal device 11A to the terminal device 11B. Therefore, in order to prevent the image sent from the terminal device 11A to the terminal device 11B from being disguised, for example, it can be determined in the terminal device 11B whether the time displayed on the settlement screen is substantially consistent with the current time, and if it is substantially inconsistent, a warning display is performed.

[0145] (13) In the above-mentioned embodiment, when the client user wants to prompt the client user with any image in the dynamic image, it is necessary to temporarily display the image in area A03 of the client screen. The method by which the client user specifies the image that he wants to prompt the client user is not limited to this. For example, the image recognition data of the image may be sent from the terminal device 11A to the terminal device 11B by performing a prescribed operation such as double-clicking the area A01, without temporarily displaying the image in area A03. In addition, when the sending user performs an operation such as dragging and dropping from area A01 to the "Send" button in area A05, the image recognition data of the image displayed in the dynamic image may be sent from the terminal device 11A to the terminal device 11B.

[0146] (14) In the above embodiment, the candidate images can be stored in the area A04 by the client user performing an intentional operation such as dragging and dropping an image from the area A03 to the area A04 on the client screen. Instead of this, all the images displayed in the area A03 by the client user may be stored in the area A04. In this case, the automatically stored images and the intentionally stored images can be distinguished by adding bookmarks or the like to the images intentionally stored in the area A04 by the client user.

[0147] (15) When the client user is in a location where filming for streaming media distribution is allowed but recording is prohibited, the terminal device 11A and the terminal device 11B may be prevented from recording moving images (including recording of images constituting moving images). In this case, for example, a database may be pre-stored in the management server device 11, etc., which stores prohibited area data indicating an area where recording is prohibited, prohibited period data indicating a period during which recording is prohibited, and permission condition data indicating conditions for an exceptional authorized person (e.g., an organizer of an event where recording is prohibited) for which recording is permitted, and the terminal device 11A and the terminal device 11B may determine whether recording moving images is prohibited by referring to the database. Furthermore, if the client user or the client user is an exceptional authorized person for which recording is permitted, confirmation is performed to confirm that the user is the authorized person (e.g., inputting a password known only to the authorized person), and when the user is confirmed to be the authorized person, the terminal device 11A and the terminal device 11B may be enabled to record moving images. As a method of prohibiting the recording of moving images, for example, there is a method of adding a code prohibiting recording to the moving image data generated in the terminal device 11B and then transmitting the code to the terminal device 11A.

[0148] (16) In the above-mentioned embodiment, the client user can confirm the moving path from the current position to the shooting position of the image prompted by the client user on the map. Alternatively or in addition to this, the terminal device 11B may use AR (Augmented Reality) technology to display the moving path to the client user. In this case, when the client user uses the camera 107 of the terminal device 11B to shoot the surrounding environment, a display such as an arrow indicating the moving path and the image shot by the camera 107 are superimposed on the touch screen 104 of the terminal device 11B. As a result, the client user can reach the destination more intuitively.

[0149] Description of symbols

[0150] 1: system; 9: network; 10: computer; 12: management server device; 13: route search server device; 14: translation server device; 15: speech synthesis server device; 16: speech recognition server device; 101: processor; 102: memory; 103: communication interface; 104: touch screen; 105: microphone; 106: speaker; 107: camera; 108: GNSS unit; 109: orientation sensor; 110: storage unit; 111: Receiving unit; 112: display indication unit; 113: voice indication unit; 114: operation acceptance unit; 115: selection image acquisition unit; 116: sending unit; 117: dynamic image acquisition unit; 118: voice acquisition unit; 119: position acquisition unit; 1101: display device; 1102: voice device; 1103: operating device; 1104: photographing device; 1105: orientation measuring device; 1106: sound collecting device; 1107: position measuring device.

Claims

1. A terminal device, comprising: a receiving unit for receiving dynamic image data transmitted in real time from a device of a communication partner; a storage unit that stores a portion of the dynamic image data received by the receiving unit corresponding to at least a prescribed amount in the most recent past; an operation accepting unit for accepting a user's operation on an image displayed by the display device; a display instruction unit that causes the display device to display the dynamic image represented by the dynamic image data received by the receiving unit in real time, and causes the display device to display an image designated by the operation performed by the operation receiving unit among images constituting past dynamic images represented by the dynamic image data stored in the storage unit; a selection image acquisition unit that acquires image recognition data that recognizes an image of a selection object among a plurality of images constituting a dynamic image represented by the dynamic image data received by the receiving unit; as well as a sending unit configured to send the image recognition data acquired by the selected image acquisition unit or the image data representing the image recognized by the image recognition data to the device of the communication object, The selection image acquisition unit acquires image recognition data for identifying an image being displayed by the display device when the operation acceptance unit accepts an operation by the user on a dynamic image displayed by the display device as image recognition data of an image to be selected or image recognition data of an image candidate for the selection object, When the operation accepting unit accepts an operation by the user to select an image displayed on the display device and designated by the operation performed by the operation accepting unit, the selected image acquisition unit acquires image recognition data of the selected image as image recognition data of an image of a selection object or image recognition data of an image of a candidate for a selection object.

2. A terminal device, comprising: a receiving unit for receiving dynamic image data transmitted in real time from a device of a communication partner; an operation accepting unit for accepting a user's operation on an image displayed by the display device; a display instruction unit that causes the display device to display a dynamic image represented by the dynamic image data received by the receiving unit; a selection image acquisition unit that acquires image recognition data that recognizes an image of a selection object among a plurality of images constituting a dynamic image represented by the dynamic image data received by the receiving unit; as well as a sending unit configured to send the image recognition data acquired by the selected image acquisition unit or the image data representing the image recognized by the image recognition data to the device of the communication object, The selection image acquisition unit acquires image recognition data for identifying an image being displayed by the display device when the operation acceptance unit accepts an operation by the user on a dynamic image displayed by the display device as image recognition data of an image to be selected or image recognition data of an image candidate for the selection object, When the selection image acquisition unit acquires the image recognition data of the candidate image of the selection object, the display instruction unit causes the display device to display the image recognized based on the image recognition data as the candidate image of the selection object. When the operation accepting unit accepts an operation of the user selecting an arbitrary image from one or more candidate images of a selection object displayed on the display device, the selection image acquiring unit acquires image recognition data of the selected image as image recognition data of the selection object.

3. A terminal device, comprising: a receiving unit for receiving dynamic image data transmitted in real time from a device of a communication partner; an operation accepting unit for accepting a user's operation on an image displayed by the display device; a display instruction unit that causes the display device to display a dynamic image represented by the dynamic image data received by the receiving unit; a selection image acquisition unit that acquires image recognition data that recognizes an image of a selection object among a plurality of images constituting a dynamic image represented by the dynamic image data received by the receiving unit; as well as a sending unit configured to send the image recognition data acquired by the selected image acquisition unit or the image data representing the image recognized by the image recognition data to the device of the communication object, The display instruction unit causes the display device to display a virtual operation button. the operation accepting unit accepting an operation by the user to drag and drop an image of a selection object recognized based on the image recognition data acquired by the selection image acquiring unit onto the operation button; The transmitting unit transmits data indicating an area within the image touched by the user when performing the drag and drop operation to the device of the communication partner.

4. A terminal device, comprising: a receiving unit for receiving dynamic image data transmitted in real time from a device of a communication partner; an operation accepting unit for accepting a user's operation on an image displayed by the display device; a display instruction unit that causes the display device to display a dynamic image represented by the dynamic image data received by the receiving unit; a selection image acquisition unit that acquires image recognition data that recognizes an image of a selection object among a plurality of images constituting a dynamic image represented by the dynamic image data received by the receiving unit; as well as a sending unit configured to send the image recognition data acquired by the selected image acquisition unit or the image data representing the image recognized by the image recognition data to the device of the communication object, The display instruction unit causes the display device to display a virtual operation button. When the operation accepting unit receives an operation of dragging and dropping the image selected by the user onto the operation button, the sending unit matches message identification data that identifies the message corresponding to the operation button or message data representing the message with image identification data acquired by the selected image acquiring unit or image data representing the image identified by the image identification data and sends the matched pairs to the device of the communication object.

5. The terminal device according to claim 4, wherein: The message data is text data representing a sentence written in a language used by a user of the communication partner's device, or voice data representing a sound speaking the sentence.

6. The terminal device according to claim 4, wherein: The message data is data indicating the status of the user.

7. The terminal device according to claim 4, wherein: The message data is action instruction data for instructing the device of the communication partner to perform a physical action.

8. The terminal device according to any one of claims 1 to 4, wherein: The terminal device includes an image recognition unit that recognizes an item captured in an image of a selected object recognized based on the image recognition data acquired by the selected image acquisition unit. The display instruction unit causes the display device to display an image in which a display object indicating a recognition result of the image recognition unit is added to the image of the selection object recognized based on the image recognition data acquired by the selection image acquisition unit.

9. A terminal device, comprising: a dynamic image acquisition unit that acquires dynamic image data representing a dynamic image captured by the imaging device; a sending unit, which sends the dynamic image data acquired by the dynamic image acquisition unit to a device of a communication object in real time; a storage unit that stores a portion of the dynamic image data acquired by the dynamic image acquisition unit corresponding to at least a prescribed amount in the recent past; a receiving unit that receives image recognition data for identifying an image, area data indicating an area within the image recognized based on the image recognition data, and message recognition data for identifying a message or message data indicating the message, transmitted from the device of the communication partner; as well as A display indication unit causes a display device to display an image of a display object representing the area indicated by the area data added to an image identified according to the image recognition data received by the receiving unit among a plurality of images constituting the dynamic image represented by the dynamic image data stored by the storage unit, and a message identified according to the message recognition data or a message represented by the message data.

10. The terminal device according to claim 9, wherein: The terminal device has a translation unit that, when acquiring text data representing a sentence in a language other than the language used by a user of the device, acquires text data representing a sentence translated from the text data into the language used by the user, When the message identified by the message identification data or the message indicated by the message data indicates a sentence in a language other than the language used by the user of the device, the display instruction unit causes the display device to display a sentence translated from the sentence indicated by the text data acquired by the translation unit.

11. The terminal device according to claim 9, wherein: The terminal device comprises: an interpreting unit that, when acquiring text data representing a sentence in a language other than a language used by a user of the device, acquires voice data representing a sound of a person speaking a sentence translated from the text data into the language used by the user; and A voice instruction unit, which causes a voice device to emit a sound of a sentence translated from the voice data obtained by the interpretation unit when a message identified by the message identification data or a message indicated by the message data indicates a sentence in a language other than the language used by the user of the device.

12. A terminal device, comprising: a dynamic image acquisition unit that acquires dynamic image data representing a dynamic image captured by the imaging device; a sending unit, which sends the dynamic image data acquired by the dynamic image acquisition unit to a device of a communication object in real time; a storage unit that stores a portion of the dynamic image data acquired by the dynamic image acquisition unit corresponding to at least a prescribed amount in the recent past; a receiving unit for receiving image recognition data for identifying an image transmitted from a device of the communication object; a display instruction unit that causes a display device to display an image recognized based on the image recognition data received by the receiving unit, among a plurality of images constituting a dynamic image represented by the dynamic image data stored by the storage unit; a position acquisition unit that acquires position data representing the position of the device; as well as a map acquisition unit that acquires map data representing a map of an area including a location of the own device, The display instruction unit causes the display device to display an image in which a display object indicating a position indicated by the position data corresponding to a shooting timing of an image recognized based on the image recognition data received by the receiving unit is added to the map indicated by the map data.

13. The terminal device according to claim 12, wherein: The moving image acquisition unit acquires moving image data and shooting direction data indicating shooting directions of each of a plurality of images constituting the moving image represented by the moving image data. The display instruction unit causes the display device to display an image in which a display object indicating a shooting direction indicated by the shooting direction data corresponding to a shooting timing of an image recognized based on the image recognition data received by the receiving unit is added to the map indicated by the map data.

14. A terminal device, comprising: a dynamic image acquisition unit that acquires dynamic image data representing a dynamic image captured by the imaging device; a sending unit, which sends the dynamic image data acquired by the dynamic image acquisition unit to a device of a communication object in real time; a storage unit that stores a portion of the dynamic image data acquired by the dynamic image acquisition unit corresponding to at least a prescribed amount in the recent past; a receiving unit for receiving image recognition data for identifying an image transmitted from a device of the communication object; a display instruction unit that causes a display device to display an image recognized based on the image recognition data received by the receiving unit, among a plurality of images constituting a dynamic image represented by the dynamic image data stored by the storage unit; a position acquisition unit that acquires position data representing the position of the device; as well as a moving path acquisition unit that acquires moving path data representing a moving path from the position of the device indicated by the position data last acquired by the position acquisition unit to the position indicated by the position data corresponding to the shooting timing of the image recognized based on the image recognition data received by the receiving unit, The display instruction unit causes the display device to display a display object indicating the movement path indicated by the movement path data.

15. A terminal device comprising: a dynamic image acquisition unit that acquires dynamic image data representing a dynamic image captured by the imaging device; a sending unit, which sends the dynamic image data acquired by the dynamic image acquisition unit to a device of a communication object in real time; a storage unit that stores a portion of the dynamic image data acquired by the dynamic image acquisition unit corresponding to at least a prescribed amount in the recent past; a receiving unit for receiving image recognition data for identifying an image transmitted from a device of the communication object; a display instruction unit that causes a display device to display an image recognized based on the image recognition data received by the receiving unit, among a plurality of images constituting a dynamic image represented by the dynamic image data stored by the storage unit; a position acquisition unit that acquires position data representing the position of the device; a movement path acquisition unit that acquires movement path data representing a movement path from the position of the device indicated by the position data last acquired by the position acquisition unit to the position indicated by the position data corresponding to the shooting timing of the image recognized based on the image recognition data received by the receiving unit; as well as A mobile unit moves according to the movement path indicated by the movement path data.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor of a computer, the functions of the receiving unit, the storage unit, the operation accepting unit, the display instruction unit, the selected image acquiring unit, and the sending unit of the terminal device according to claim 1 are realized.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor of a computer, the functions of the receiving unit, the operation accepting unit, the display instruction unit, the selected image acquiring unit and the sending unit of the terminal device according to any one of claims 2 to 4 are realized.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor of a computer, the functions of the dynamic image acquisition unit, the transmission unit, the storage unit, the reception unit, and the display instruction unit of the terminal device according to claim 9 are realized.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor of a computer, the functions of the dynamic image acquisition unit, the sending unit, the storage unit, the receiving unit, the display instruction unit, the position acquisition unit and the map acquisition unit of the terminal device described in claim 12 are realized.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor of a computer, the functions of the dynamic image acquisition unit, the sending unit, the storage unit, the receiving unit, the display instruction unit, the position acquisition unit and the movement path acquisition unit of the terminal device described in claim 14 are realized.

Images