Information processing method, information processing device, and program

Abstract

[Problem] To more accurately reflect the intention of an instruction by a user. [Solution] An information processing method executed by a computer including: receiving an image; recognizing the utterance of a user; acquiring the state of the user on the basis of the result of recognition of the utterance; acquiring instruction information indicating an instruction by the user on the basis of the state of the user; and generating a display image in which the instruction information is reflected on the image.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program.

[0002] In recent years, technologies for sharing user instructions have been developed. For example, Patent Literature 1 discloses a technology for sharing user instructions by superimposing a target location identified on an image displayed on a display device by a user operating a pointing device such as a mouse on an image captured by a camera.

[0003] Japanese Patent Application Publication No. 05-151339

[0004] Here, the user may give instructions while speaking. However, according to the technology disclosed in Patent Document 1, the target location is identified without taking into consideration the user's speech.

[0005] Therefore, the present disclosure proposes a new and improved technique that can more accurately reflect the intention of a user's instruction.

[0006] According to the present disclosure, there is provided an information processing method executed by a computer, which includes receiving an image, recognizing a user's speech, acquiring the user's state based on the recognition result of the speech, acquiring instruction information indicating an instruction by the user based on the user's state, and generating a display image that reflects the instruction information in the image.

[0007] In addition, according to another aspect of the present disclosure, in order to solve the above problem, there is provided an information processing device comprising: a communication unit that receives an image; a recognition unit that recognizes a user's speech; a status acquisition unit that acquires the user's state based on the recognition result of the speech; an instruction information acquisition unit that acquires instruction information indicating an instruction by the user based on the user's state; and an image generation unit that generates a display image that reflects the instruction information in the image.

[0008] In addition, according to another aspect of the present disclosure, in order to solve the above problem, a program is provided that causes a computer to operate as a communication unit that receives an image, a recognition unit that recognizes a user's speech, a status acquisition unit that acquires the user's state based on the recognition result of the speech, an instruction information acquisition unit that acquires instruction information indicating instructions by the user based on the user's state, and an image generation unit that generates a display image that reflects the instruction information in the image.

[0009] FIG. 1 is a diagram for explaining an overview of a first configuration example of an information processing system according to an embodiment of the present disclosure. FIG. 1 is a block diagram showing an example of a functional configuration of an information processing system 1A according to the first configuration example according to an embodiment of the present disclosure. FIG. 1 is a diagram showing an example of a display image generated by an image synthesis unit 170A. FIG. 2 is a diagram for explaining an overview of a second configuration example of an information processing system according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing an example of a functional configuration of an information processing system 1B according to the second configuration example according to an embodiment of the present disclosure. FIG. 3 is a diagram for explaining an example in which a point in real space is acquired as supervisor instruction information by a point pointing recognition unit 163B. FIG. 4 is a diagram showing an example of a display image generated by an image synthesis unit 270B. FIG. 5 is a diagram for explaining an example in which an inspection supervisor I indicates, as an instruction, a point or area not included in an image captured by a moving body 30B. FIG. 6 is a diagram showing an example of a display image generated by an image synthesis unit 270B when a direction in which a point or area exists in real space is indicated as supervisor instruction information. FIG. 7 is a diagram for explaining an overview of a third configuration example of an information processing system according to an embodiment of the present disclosure. FIG. 8 is a block diagram showing an example of a functional configuration of an information processing system 1C according to the third configuration example according to an embodiment of the present disclosure. FIG. 10 is a diagram for explaining an overview of a fourth configuration example of an information processing system according to an embodiment of the present disclosure. FIG. 11 is a block diagram showing an example of the functional configuration of an information processing system 1D according to the fourth configuration example according to an embodiment of the present disclosure. FIG. 12 is a diagram for explaining an overview of a fifth configuration example of an information processing system according to an embodiment of the present disclosure. FIG. 13 is a block diagram showing an example of the functional configuration of an information processing system 1E according to the fifth configuration example according to an embodiment of the present disclosure. FIG. 14 is a flowchart showing a flow of operations executed by an information processing device according to this embodiment. FIG. 15 is a diagram showing an example of a legend for instruction marks shown in different formats depending on reliability. FIG. 16 is a block diagram showing an example of an information processing device 90.

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] The explanation will be given in the following order: 1. Overview 2. Configuration example 3. Operation processing example 4. Modification 5. Hardware configuration example 6. Supplementary information

[0012] <<1. Overview>> First, an overview of the present disclosure will be described. The present disclosure describes a technology for generating a display image that reflects instruction information indicating an instruction by a user, based on a user state acquired based on a recognition result of the user's utterance.

[0013] The technology disclosed herein may be applied, for example, when inspecting a structure using a mobile object such as a drone, an unmanned ground vehicle, or a robot, which is an aerial vehicle equipped with an imaging device.

[0014] In inspections using a mobile object, the mobile object is generally operated by a person with the necessary skills to operate the mobile object. The person operates the mobile object while communicating with an inspection manager in charge of the inspection.

[0015] The operator and the inspection manager communicate with each other instructions regarding the inspection target, the location of the inspection target, and the destination of the mobile body, which they intend to target. For example, the inspection manager communicates to the operator an instruction regarding the location of the inspection target, such as a structure to be inspected by the mobile body, or an inspection point or inspection range within the structure. The operator then operates the mobile body so that an image of the inspection target corresponding to the inspection manager's instructions is captured, or so that the mobile body moves to a destination corresponding to the instructions, thereby performing the inspection work.

[0016] The operator and the inspection manager communicate their instructions to each other by speech while checking images such as FPV (First Person View) images (still images or video images) captured by the mobile object, which are displayed on the screens of their respective transmitters. For example, the inspection manager may give instructions to the operator by the name of a building that will be imaged by the mobile object. The operator may also give instructions to the inspection manager by the name of a building to which the mobile object is moving. Speech is an effective means of communication because it allows the target to be identified by a proper noun. Speech is particularly convenient for operators who control a mobile object by operating a controller with their fingers, as it allows communication without using their fingers.

[0017] However, when instructions are communicated between the operator and the inspection supervisor only through speech, it can be difficult to accurately convey the instructions to the other party. In this case, for example, it is assumed that the operator and the inspection supervisor will indicate the inspection target using demonstrative language while pointing, etc. However, since the operator is operating a controller, it can be difficult for the operator to point. Furthermore, since the operator is visually inspecting the vehicle body and checking telemetry information related to operation, it can be difficult for the operator to confirm that the inspection supervisor is pointing.

[0018] In addition, to accurately convey instructions, it is expected that the operator and the inspection manager will both be viewing the same screen and using instructions by using directives or pointing at the inspection target on the screen. For example, the inspection target manager may say, "Please move closer to this part." However, because the operator is visually inspecting the vehicle itself and checking telemetry information related to operation, it may be difficult for the operator to see the screen of the transmitter in the inspector's hand.

[0019] In addition, in order to accurately convey instructions, it is also assumed that the inspection manager may point to the inspection target itself, or the pilot may point to an object that they recognize as an inspection target for confirmation, and communicate the instructions using demonstrative language. However, depending on the angle at which the inspection manager and the pilot view the object and the distance between them, it may be difficult for them to identify each other's instructions.

[0020] Therefore, this disclosure proposes a technology for acquiring instruction information indicating the instructions of a communicator by acquiring the state of the communicator based on the speech of the communicator (inspector or operator). In this disclosure, by reflecting such instruction information in an image, it is possible to generate a display image that more accurately reflects the intention of the instructions of each communicator.

[0021] <<2. Configuration Examples>> Various system configurations are possible for the technology according to the present disclosure, the overview of which has been described above. From here, each system configuration example according to an embodiment of the present disclosure will be described in detail.

[0022] 2-1. First Configuration Example First, a first configuration example of an information processing system according to an embodiment of the present disclosure will be described with reference to FIGS.

[0023] 1 is a diagram illustrating an overview of a first configuration example of an information processing system according to an embodiment of the present disclosure. As shown in FIG. 1, an information processing system 1A according to an embodiment of the present disclosure includes a pilot transmitter 10A, an inspection supervisor transmitter 20A, and a mobile body 30A. The pilot transmitter 10A and the inspection supervisor transmitter 20A are examples of an information processing device according to this embodiment.

[0024] 1, an inspection of an inspection object Ob is performed by operating a mobile object 30A while an operator P using an operator transmitter 10A and an inspection supervisor I using an inspection supervisor transmitter 20A communicate with each other. The operator P and the inspection supervisor I are examples of users according to this embodiment.

[0025] The operator transmitter 10A, the inspection supervisor transmitter 20A, and the mobile unit 30A are each connected for communication to transmit and receive various data.

[0026] The operator transmitter 10A is a transmitter used by the operator P and is a controller for the moving body 30A. The operator transmitter 10A may be a dedicated controller for operating the moving body 30A, or may be a smartphone or tablet terminal or the like on which an application including a function for operating the moving body 30A is installed.

[0027] The operator P controls the moving body 30A by operating the operator transmitter 10A. The operator transmitter 10A also receives, from the moving body 30A, an image of the inspection object Ob captured by the moving body 30A. The operator transmitter 10A displays to the operator P a display image including the captured image received from the moving body 30A.

[0028] Furthermore, the operator transmitter 10A acquires audio including the speech of the inspection supervisor I from the inspection supervisor transmitter 20A. The operator transmitter 10A outputs the audio received from the inspection supervisor transmitter 20A to the operator P.

[0029] The operator transmitter 10A also has various sensors. For example, the operator transmitter 10A has a microphone. The operator transmitter 10A recognizes the speech of the operator P picked up by the microphone.

[0030] The operator transmitter 10A acquires the state of the operator P from the detection results of various sensors that are detected based on the recognition results of the speech of the operator P. The state of the operator P may be, for example, the direction of the operator P's pointing finger, the direction of the line of sight, the direction of the face, the direction of the body, or the contact position of a part of the body (such as a finger or palm).

[0031] The operator transmitter 10A acquires operator instruction information, which is instruction information indicating instructions from the operator P, based on the state of the operator P. The operator instruction information may be, for example, information about a point or area indicated by the direction in which the operator P or a part of the body of the operator P is facing.

[0032] More specifically, the operator instruction information may be information indicating a point or area that the operator P is pointing at, a point or area in the line of sight of the operator P, a point or area in the direction of the operator P's body or face, or a point or area corresponding to the contact position of the operator P. The point or area indicated by the operator instruction information may be, for example, the position of the inspection object Ob to which the operator P intends to move the mobile body 30A.

[0033] The operator transmitter 10A transmits such operator instruction information and audio information acquired by the microphone to the inspection supervisor transmitter 20A. For example, in the example shown in Fig. 1, information indicating a point ahead of the line of sight of the operator P on the display screen displayed on the operator transmitter 10A may be transmitted to the inspection supervisor transmitter 20A as operator instruction information.

[0034] The inspection supervisor transmitter 20A is a transmitter used by the inspection supervisor I. The inspection supervisor transmitter 20A may be a transmitter in which the operation function of the mobile body 30A is turned off from the operator transmitter 10A. The inspection supervisor transmitter 20A may also be a smartphone, a tablet terminal, or the like. The inspection supervisor transmitter 20A receives, from the mobile body 30A, an image of the inspection object Ob captured by the mobile body 30A. The image received from the mobile body 30A is displayed to the inspection supervisor I.

[0035] Furthermore, the inspection supervisor transmitter 20A acquires audio, including speech from the operator transmitter 10A, from the operator transmitter 10A. The inspection supervisor transmitter 20A outputs the audio received from the operator transmitter 10A to the inspection supervisor I.

[0036] The inspection supervisor transmitter 20A also has various sensors. For example, the inspection supervisor transmitter 20A has a microphone. The inspection supervisor transmitter 20A recognizes the speech of the inspection supervisor I picked up by the microphone.

[0037] The various sensors included in the inspection supervisor transmitter 20A may be the same as the various sensors included in the operator transmitter 10A. The inspection supervisor transmitter 20A acquires the state of the inspection supervisor I, which is similar to the state of the operator P, based on the detection results detected by the various sensors based on the recognition results of the speech of the inspection supervisor I.

[0038] The inspection supervisor transmitter 20A acquires supervisor instruction information indicating the instructions of the inspection supervisor I based on the state of the inspection supervisor I. The supervisor instruction information is substantially similar to the pilot instruction information. The point or area indicated by the instruction information may be, for example, the position of the inspection object Ob that the inspection supervisor I is to inspect.

[0039] The inspection supervisor transmitter 20A transmits supervisor instruction information and audio information acquired by the microphone to the operator transmitter 10A. For example, in the example shown in Fig. 1, information indicating a point in the line of sight of the inspection supervisor I on the display screen displayed on the inspection supervisor transmitter 20A may be transmitted to the operator transmitter 10A as supervisor instruction information.

[0040] The operator transmitter 10A and the inspection supervisor transmitter 20A generate and display a display image that reflects the instruction information received from the operator transmitter 10A and the inspection supervisor transmitter 20A on the image received from the mobile body 30. This enables the operator P and the inspection supervisor I to accurately understand each other's instructions. Details of the display image will be described later.

[0041] The mobile body 30A is a drone that moves through the air. The mobile body 30A has a camera 310A. The camera 310A is an imaging device that captures an image of an inspection object Ob. The mobile body 30A moves to various positions, angles, and altitudes in response to control signals received from the operator transmitter 10A, and captures images of the inspection object Ob from various positions and orientations using the camera 310A.

[0042] Next, a functional configuration example of each device included in the information processing system 1A will be described in detail with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the information processing system 1A according to a first configuration example according to an embodiment of the present disclosure.

[0043] First, an example of the functional configuration of the operator transmitter 10A will be described. As shown in Fig. 2, the operator transmitter 10A has a communication unit 110A, a sensor unit 120A, a display unit 130A, an audio output unit 140A, an operation unit 150A, an instruction analysis unit 160A, and an image synthesis unit 170A.

[0044] (Communication Unit 110A) The communication unit 110A is a communication interface for transmitting and receiving data between the inspection supervisor transmitter 20A and the mobile body 30. The communication unit 110A may transmit and receive data between the inspection supervisor transmitter 20A and the mobile body 30 via a communication network such as the Internet. The communication unit 110A may be, for example, a communication card for a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication unit 110A may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0045] The communication unit 110A receives, for example, supervisor instruction information and the voice of the inspection supervisor I from the inspection supervisor transmitter 20A. The communication unit 110A also transmits to the inspection supervisor transmitter 20A operator instruction information analyzed by an instruction analysis unit 160 (described later) and the voice of the operator P acquired by a microphone 122A (described later). Note that the voice of the operator P does not need to be transmitted to the inspection supervisor transmitter 20A when the operator P and the inspection supervisor I are within a distance where they can converse with each other.

[0046] The communication unit 110A also receives images captured by the moving body 30A from the moving body 30A. The communication unit 110A transmits a control signal to the moving body 30 in response to an operation for controlling the moving body 30 input to the operation unit 150 described below.

[0047] (Sensor Unit 120A) The sensor unit 120A includes various sensors that detect the operator P. The sensor unit 120A includes a camera 121A, a microphone 122A, and a touch sensor 123A. The detection results of the sensor unit 120A are output to the instruction analysis unit 160A.

[0048] The camera 121A is an example of an image sensor that captures an image of the operator P. The camera 121A may capture an image of the entire body of the operator P, or may capture an image of a part of the body of the operator P. The part of the body of the operator P may be, for example, a finger, a face, or an eye.

[0049] The camera 121A may include a plurality of cameras that capture images of the operator P at different angles or magnifications. For example, the camera 121A may include a plurality of cameras that capture images of different parts of the operator P's body.

[0050] The camera 121A may capture an image of an object held by the operator P. The object held by the operator P may be, for example, a pointing stick. The camera 121A may also capture an image of laser light emitted by a laser pointer held by the operator P.

[0051] The microphone 122A is a microphone that captures the voice of the operator P. The microphone 122A outputs the captured voice to the communication unit 110A.

[0052] The touch sensor 123A is a sensor that detects contact by the operator P. The touch sensor 123A may be, for example, a sensor that detects the contact position of the operator P's finger or palm on the display unit 130A described below. Note that, although an example in which the touch sensor 123A detects the contact position of the operator P's finger will be described below, the camera 121 may also detect contact by an object held by the operator P. The object held by the operator P may be, for example, a touch pen.

[0053] Note that, although an example in which sensor unit 120A includes camera 121A, microphone 122A, and touch sensor 123A has been described here, sensor unit 120A may include at least one of a variety of sensors that detect operator P, such as microphone 122, camera 121A, touch sensor 123A, and a distance sensor, acceleration sensor, or infrared sensor.

[0054] The sensor unit 120A may be configured separately from the operator transmitter 10A, in which case the sensor unit 120A may be a motion sensor or the like worn by the operator P. The sensor unit 120A may also be a pointing device such as a mouse that allows the operator P to specify the position of the captured image displayed on the display unit 130A.

[0055] (Display unit 130A) The display unit 130A displays a display image including an image captured by the moving object 30A and received by the communication unit 110A. More specifically, the display unit 130A displays a display image that reflects the supervisor instruction information acquired by the communication unit 110A on the image captured by the moving object 30A and that is generated by the image synthesis unit 170A.

[0056] The display unit 130A may be a display panel such as a liquid crystal display (LCD), an organic electroluminescence (EL) display, etc. The display unit 130A may be configured to include a touch sensor 123A.

[0057] (Audio Output Unit 140A) The audio output unit 140A is an output device such as a speaker or headphones that outputs the audio acquired by the communication unit 110A. The operator P can recognize the instructions of the inspection supervisor I by listening in real time to the audio of the inspection supervisor I output from the audio output unit 140A.

[0058] (Operation Unit 150A) The operation unit 150A accepts operations for operating the moving body 30. The operation unit 150A may be configured with, for example, a joystick or buttons. The operation unit 150A may also be a touch panel or the like that is configured integrally with the display unit 130A.

[0059] (Instruction analysis unit 160A) The instruction analysis unit 160A analyzes the state of the operator P based on the detection results of the sensor unit 120A. The instruction analysis unit 160A has a voice recognition unit 161A, a gaze recognition unit 162A, a pointing recognition unit 163A, a touch recognition unit 164A, and a face recognition unit 165A.

[0060] The voice recognition unit 161A recognizes the voice detected by the microphone 122A. More specifically, the voice recognition unit 161A recognizes the speech of the operator P detected by the microphone 122A.

[0061] The voice recognition unit 161A may, for example, recognize whether a demonstrative word is included in the speech of the pilot P. A demonstrative word is a word that indicates a place, a direction, or an object, such as "that," "that," "this," "over there," or "that way."

[0062] Furthermore, the voice recognition unit 161A may recognize whether or not a word representing a direction or an object is included in the speech of the pilot P. The word representing the object may be the name of the object or the name of a part of the object.

[0063] More specifically, the word representing the object may be the name of the inspection object Ob of the inspection target, a word representing an object beside the inspection object Ob of the inspection target, or the name of a part of the inspection object Ob (for example, a support pillar, etc.). Furthermore, the word representing the direction may be a word such as left / right, up / down, front or back. The object may also be represented by a number, for example, the second support pillar from the left.

[0064] The gaze recognition unit 162A, the pointing recognition unit 163A, the touch recognition unit 164A, and the face recognition unit 165A acquire the state of the operator P based on the recognition result of the utterance by the voice recognition unit 161A. More specifically, the gaze recognition unit 162A, the pointing recognition unit 163A, the touch recognition unit 164A, and the face recognition unit 165A may acquire the state of the operator P when the voice recognition unit 161A recognizes that the utterance by the operator P includes a demonstrative word or when it recognizes that the utterance by the operator P includes a word indicating a direction or an object. The state of the operator P is acquired based on at least any one of the detection results of an object held or worn by the operator P, or the position or posture of the operator P's body or a part of the operator P's body.

[0065] Based on the detection result by the camera 121A, the gaze recognition unit 162A recognizes the gaze direction of the operator P as the state of the operator P. Existing technology may be used for detecting the gaze direction by the gaze recognition unit 162A, and for example, the gaze recognition unit 162A may acquire the gaze direction based on the relative position of the pupil of the operator P with respect to the camera 121A.

[0066] The gaze recognition unit 162A may then acquire, as the pilot instruction information, the position of a point on the display image displayed on the display unit 130A, which is indicated by the direction of the gaze of the pilot P. The gaze recognition unit 162A may acquire, as the pilot instruction information, the directional vector of the gaze direction and the position of the contact point with the display unit 130A, which are specified from the gaze direction.

[0067] In addition, if multiple gaze directions can be obtained as the pilot P's gaze moves, the gaze recognition unit 162A may acquire the position of an area on the display image displayed on the display unit 130A indicated by the pilot P's multiple gaze directions as pilot instruction information.

[0068] Based on the detection result by the camera 121A, the pointing recognition unit 163A recognizes the pointing direction of the operator P as the state of the operator P. Any existing technology may be used for the detection of the pointing direction by the pointing recognition unit 163A, and for example, the pointing recognition unit 163A may obtain the pointing direction based on the relative positions of any two points on the finger of the operator P captured at different angles by each of the two cameras included in the camera 121A. As another example, the pointing recognition unit 163A may obtain the pointing direction based on acceleration information obtained from an acceleration sensor worn by the operator P on a part of the body such as a finger or arm.

[0069] The pointing recognition unit 163A may acquire, as the operator instruction information, the position of a point on the display image displayed on the display unit 130A, which is indicated by the pointing direction of the operator P. The gaze recognition unit 162A may acquire, as the operator instruction information, the position of the contact point between the pointing direction vector and the display unit 130A, which is specified from the pointing direction.

[0070] In addition, if multiple pointing directions can be obtained by the movement of the operator P's finger, the pointing recognition unit 163A may obtain the position of an area on the display image displayed on the display unit 130A indicated by the multiple pointing directions of the operator P as operator instruction information.

[0071] Based on the detection result by the touch sensor 123A, the touch recognition unit 164A recognizes the contact position of the finger of the operator P on the display unit 130 as the state of the operator P. Then, the pointing recognition unit 163A may acquire the position of a point on the display image displayed on the display unit 130A, which is indicated by the contact position of the finger of the operator P, as operator instruction information.

[0072] In addition, if multiple contact positions can be obtained by the movement of the operator P's fingers, the touch recognition unit 164A may obtain the position of an area on the display image displayed on the display unit 130A, which is indicated by the multiple contact positions of the operator P, as operator instruction information.

[0073] Based on the detection result by the camera 121A, the face recognition unit 165A recognizes the direction of the face of the operator P as the state of the operator P. The face recognition unit 165A may use existing technology to detect the direction of the face, and for example, the face recognition unit 165A may acquire the gaze direction based on the relative positions of facial features such as the tip of the nose, eyes, mouth, or ears of the operator P with respect to the camera 121A.

[0074] The face recognition unit 165A may then acquire, as the pilot instruction information, the position of a point on the display image displayed on the display unit 130A, which is indicated by the orientation of the face of the pilot P. The face recognition unit 165A may acquire, as the pilot instruction information, the directional vector of the orientation of the face, which is identified from the orientation of the face, and the position of the contact point with the display unit 130A.

[0075] In addition, if multiple facial directions can be obtained as the pilot P's face moves, the face recognition unit 165A may acquire the position of an area on the display image displayed on the display unit 130A indicated by the multiple facial directions of the pilot P as pilot instruction information.

[0076] The above explains how the state of the pilot P and pilot instruction information based on the state of the pilot P are acquired by each component of the instruction analysis unit 160A, but the state of the pilot P and the pilot instruction information based on the state of the pilot P are not limited to the examples described above.

[0077] For example, the state of the operator P may be the orientation of the operator P's body, which is acquired based on the detection results by the camera 121A. Furthermore, the operator instruction information based on the state of the operator P may be the position of a point or area in the display image displayed on the display unit 130A, which is acquired based on the orientation of the body. Furthermore, when the sensor unit 120A detects an object held or worn by the user, the instruction analysis unit 160A may acquire, as the operator instruction information, a point or area indicated by the position or orientation of the object.

[0078] Furthermore, each of the gaze recognition unit 162A, the pointing recognition unit 163A, the touch recognition unit 164A, and the face recognition unit 165A may acquire pilot instruction information according to the content of the utterance recognized by the voice recognition unit 161A. For example, if the utterance includes a word indicating a direction or an object, the point or area corresponding to the pilot instruction information on the display image displayed on the display unit 130A may be corrected so that it does not contradict the direction or object included in the utterance. Here, the direction included in the utterance may indicate a direction as seen from the inspection supervisor I or a direction as seen from the mobile unit 30A. In this case, the voice recognition unit 161A may recognize from the utterance whether the direction included in the utterance is a direction as seen from the inspection supervisor I or a direction as seen from the mobile unit 30A, and the point or area corresponding to the pilot instruction information may be identified based on the recognition result.

[0079] The operator instruction information acquired by each component of the instruction analysis unit 160A is output to the image synthesis unit 170A, and is also output from the communication unit 110A to the inspection supervisor transmitter 20A.

[0080] Here, it is assumed that the operator instruction information is acquired by more than one of the gaze recognition unit 162A, the pointing recognition unit 163A, the touch recognition unit 164A, and the face recognition unit 165A. In this case, the instruction analysis unit 160A may determine the operator instruction information to output based on which detection result the operator instruction information was acquired based on: an object held or worn by the operator P, or a detection result of the position or posture of the body or part of the body of the operator P.

[0081] A priority order may be set in advance for each detection result. In this case, the instruction analysis unit 160A may determine the pilot instruction information corresponding to the detection result with the highest priority as the pilot instruction information to be output. Furthermore, each recognition unit (the gaze recognition unit 162A, the pointing recognition unit 163A, the touch recognition unit 164A, and the face recognition unit 165A) may perform recognition in the order of the detection results with the highest priority. Then, when pilot instruction information is obtained by any recognition unit, the pilot instruction information may be output without performing recognition by other recognition units corresponding to detection results with lower priorities.

[0082] For example, the priority may be set to, in descending order, the contact position of the finger of the operator P, the pointing direction of the operator P, the gaze direction of the operator P, the facial direction of the operator P, and the body direction of the operator P. This is because contact and pointing are intentional behaviors by the operator P, and therefore it is preferable that they be given priority over the gaze direction, facial direction, and body direction, which may be unconscious behaviors.

[0083] Note that if the gaze recognition unit 162A, the pointing recognition unit 163A, the touch recognition unit 164A, and the face recognition unit 165A do not acquire operator instruction information, the instruction analysis unit 160A does not need to output the operator instruction information. This may occur, for example, when the camera 121A does not capture an image of the operator P or a specific body part of the operator P, or when the touch sensor 123A does not detect contact. In such a case, a notification that the instruction location corresponding to the utterance is unknown may be displayed on the display unit 130A.

[0084] (Image composition unit 170A) The image composition unit 170A generates a display image that reflects the supervisor instruction information that the communication unit 110A acquired from the inspection supervisor transmitter 20A on the captured image that the communication unit 110A acquired from the mobile body 30A. The display image generated by the image composition unit 170A is displayed on the display unit 130A. The image composition unit 170A may reflect the supervisor instruction information by displaying a supervisor instruction mark that indicates, by means of a shape or color, a point or area that corresponds to the supervisor instruction information.

[0085] Fig. 3 is a diagram showing an example of a display image generated by the image synthesis unit 170A. Fig. 3 shows the pilot transmitter 10A including a display unit 130A that displays the display image generated by the image synthesis unit 170A.

[0086] The display image displayed on the display unit 130A includes an image of the inspection object Ob captured by the mobile body 30A and a supervisor instruction mark IM1, which is a graphic that indicates supervisor instruction information. Here, the supervisor instruction mark IM1 indicates, as supervisor instruction information, the position of a point on the display image displayed on the display unit 130A, which is indicated by the line of sight of the inspection supervisor I in FIG. 1. By visually recognizing the supervisor instruction mark IM1, the operator P can accurately understand the instructions of the inspection supervisor I.

[0087] For example, if the supervisor instruction mark IM1 is displayed based on the recognition of an utterance by the inspection supervisor I that includes a demonstrative phrase, such as "Come over here and let me see," the operator P can recognize that the object corresponding to "here" is the object located at the position of the supervisor instruction mark IM1 on the displayed image. In this case, the inspection supervisor I can operate the mobile unit 30A to approach the object, thereby enabling the inspection work to proceed smoothly.

[0088] Although Figure 3 shows an example in which a point corresponding to pilot instruction information is represented by a supervisor instruction mark IM1, an area corresponding to pilot instruction information may also be represented by a supervisor instruction mark IM1. In this case, the supervisor instruction mark IM1 may be a polygon that encloses the area, or the entire area may be represented by a gradation or particle representation, etc.

[0089] Furthermore, the supervisor instruction mark IM1 may display the supervisor instruction information in the form of different shapes or colors depending on whether the supervisor instruction information is acquired based on the detection result of an object held or worn by the inspection supervisor I, or the detection result of the position or posture of the inspection supervisor I's body or a part of his body. Note that different colors include the same color with different shades. For example, different colors may be represented by different shades of color on a grayscale.

[0090] For example, the supervisor instruction mark IM1 may display the supervisor instruction information using an illustration that resembles a part of the body of the inspection supervisor I. As a more specific example, if the supervisor instruction information is acquired based on the detection result of the line of sight of the inspection supervisor I, the supervisor instruction mark IM1 may display the supervisor instruction information using an illustration that resembles an eye.

[0091] Furthermore, the image synthesis unit 170A may further reflect the pilot instruction information acquired from the instruction analysis unit 160A in the display image as a pilot instruction mark. Similar to the supervisor instruction mark, the pilot instruction mark indicates a point or area corresponding to the pilot instruction information by a shape or color.

[0092] 3 shows an example in which the display image displayed on display unit 130A includes a pilot instruction mark PM1, which is a graphic that indicates pilot instruction information. Here, pilot instruction mark PM1 indicates, as pilot instruction information, the position of a point on the display image displayed on display unit 230A, which is indicated by the line of sight of pilot P in FIG. 1. It is preferable that supervisor instruction mark IM1 and pilot instruction mark PM1 are displayed in different colors or graphics.

[0093] The operator instruction information is also displayed on the inspection supervisor transmitter 20A, so that it can be understood by the inspection supervisor I. Here, by visually checking the operator instruction mark PM1, the operator P can confirm the position or area that the inspection supervisor I has understood as his / her instruction. If the operator instruction information differs from the instruction that the operator P intended, the operator P can inform the inspection supervisor I of this, or can point to the indicated position again to make the operator instruction information match the instruction that the operator intended.

[0094] An example of the functional configuration of the operator transmitter 10A has been described above. Next, an example of the functional configuration of the inspection supervisor transmitter 20A will be described with reference to FIG.

[0095] The functional configuration of the inspection supervisor transmitter 20A is substantially the same as the functional configuration of the operator transmitter 10A, except that it does not include a configuration corresponding to the operation unit 150A.

[0096] For example, the inspection supervisor transmitter 20A detects the inspection supervisor I using the camera 221A and touch sensor 223A based on the recognition result of the speech of the inspection supervisor I detected by the microphone 222A. Based on the detection result, the inspection supervisor transmitter 20A recognizes the state of the inspection supervisor I using the instruction analysis unit 260A and acquires supervisor instruction information.

[0097] Then, the inspection supervisor transmitter 20A generates a display image using the image synthesis unit 270A that reflects the operator instruction information obtained by the communication unit 210A and the supervisor instruction information obtained by the instruction analysis unit 260A on the image captured by the mobile body 30A obtained by the communication unit 210A.

[0098] The generated display image is displayed on the display unit 230A. The display image displayed on the display unit 230A may be the same as the display image displayed on the display unit 130A shown in Fig. 3. This allows the inspection supervisor I to accurately understand the intention of the operator P.

[0099] The above describes an example of the functional configuration of the inspection supervisor transmitter 20A. The example of the functional configuration of the mobile unit 30A is as described with reference to FIG.

[0100] Although the above description has been given of an example in which the mobile body 30A is composed of one unit, the information processing system 1A may include a plurality of mobile bodies 30A. The mobile body 30A may also be equipped with a plurality of cameras 310A. In this case, the operator P and the inspection supervisor I may determine which of the plurality of captured images to display on the operator transmitter 10A and the inspection supervisor transmitter 20A based on the operation of the operator transmitter 10A and the inspection supervisor transmitter 20A, respectively.

[0101] When display images including different captured images are displayed on the operator transmitter 10A and the inspection supervisor transmitter 20A, a point or area of ​​an object such as an inspection object Ob in real space may be identified based on a point or area corresponding to supervisor instruction information identified on the display image displayed on the display unit 230A of the inspection supervisor transmitter 20A. The point or area of ​​the object may be identified based on the attitude and position of each moving body 30A.

[0102] The display unit 130A of the operator transmitter 10A may display an image in which a point or area on the display image displayed on the display unit 130A that corresponds to a point or area of ​​an object such as the inspection object Ob in real space is reflected as supervisor instruction information. Also, the display unit 230A of the inspection supervisor transmitter 20A may display an image in which the operator instruction information is reflected in a similar manner.

[0103] 2-2. Second Configuration Example Next, a second configuration example of the information processing system according to an embodiment of the present disclosure will be described with reference to FIGS.

[0104] 4 is a diagram illustrating an overview of a second configuration example of an information processing system according to an embodiment of the present disclosure. As shown in FIG. 4, an information processing system 1B according to an embodiment of the present disclosure includes an operator transmitter 10B, an inspection supervisor transmitter 20B, and a mobile body 30B. In the example shown in FIG. 4, an inspection of an inspection object Ob is performed by operating the mobile body 30B while an operator P using the operator transmitter 10B and an inspection supervisor I using the inspection supervisor transmitter 20B communicate with each other.

[0105] In the first configuration example of this embodiment, an example was described in which instruction information indicating instructions from the operator P and the inspection supervisor I is obtained based on the direction of gaze, face or body orientation, and contact position, etc., toward each display unit 130A and display unit 230A of the operator transmitter 10A and the inspection supervisor transmitter 20A.

[0106] However, it is also possible that the operator P and the inspection supervisor I may direct their gaze or point at a point or area in real space as an instruction target. For example, Figure 4 shows an example in which the inspection supervisor I points at a part of an inspection target object Ob in real space as an instruction target.

[0107] Therefore, the operator transmitter 10B and the inspection supervisor transmitter 20B according to the second configuration example of this embodiment further acquire, as instruction information, points or areas in real space indicated by the direction of the line of sight and the orientation of the face or body of the operator P and the inspection supervisor I, respectively. The operator transmitter 10B and the inspection supervisor transmitter 20B are examples of information processing devices according to this embodiment.

[0108] Next, a functional configuration example of each device included in the information processing system 1B will be described in detail with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the functional configuration of the information processing system 1B according to a second configuration example according to an embodiment of the present disclosure.

[0109] Here, of the configurations provided in the operator transmitter 10B, the inspection supervisor transmitter 20B, and the mobile body 30B, the configurations having functions different from the configurations provided in the operator transmitter 10A, the inspection supervisor transmitter 20A, and the mobile body 30A described using Fig. 2 will be mainly described, and a description of the functions common to the operator transmitter 10A, the inspection supervisor transmitter 20A, and the mobile body 30A will be omitted. Also, here, the function of the inspection supervisor transmitter 20B will be mainly described, and a description of the function of the operator transmitter 10B will be omitted because it is substantially similar to the function of the inspection supervisor transmitter 20B.

[0110] In addition to having the same configuration as the mobile body 30A, the mobile body 30B further includes a position and orientation sensor 320B that detects the position and orientation of the mobile body 30B. The position and orientation sensor 320B is a sensor that detects the position and orientation of the mobile body 30B. The position and orientation sensor 320B may detect the position of the mobile body 30B using, for example, a Global Navigation Satellite System (GNSS) sensor such as a Global Positioning System (GPS) sensor. The position of the mobile body 30B may be represented by, for example, a three-dimensional position (longitude, latitude, and altitude).

[0111] The position and orientation sensor 320B may detect the orientation of the moving body 30B using an IMU (Inertial Measurement Unit) including an angular velocity sensor and an acceleration sensor. The orientation of the moving body 30B may be represented by, for example, a three-dimensional orientation angle.

[0112] The position and attitude of the moving body 30B detected by the position and attitude sensor 320B are output to the operator transmitter 10B and the inspection supervisor transmitter 20B.

[0113] The inspection supervisor transmitter 20B has the same configuration as the inspection supervisor transmitter 20A, and further includes a position and orientation recognition unit 280B. Also, the sensor unit 220B of the inspection supervisor transmitter 20B has the same configuration as the sensor unit 220A of the inspection supervisor transmitter 20A, and further includes a position and orientation sensor 224B.

[0114] The position and orientation sensor 224B is a sensor that detects the position and orientation of the inspection supervisor transmitter 20B. More specifically, the position and orientation sensor 224B detects the position and orientation of the camera 221B as the position and orientation of the inspection supervisor transmitter 20B. Note that if the camera 221B includes multiple cameras, the position and orientation sensor 224B detects the position and orientation of each of the cameras.

[0115] The position and orientation sensor 224B may detect the position of the inspection supervisor transmitter 20B by, for example, a GNSS sensor such as a GPS sensor. The position of the inspection supervisor transmitter 20B may be represented by, for example, a three-dimensional position (e.g., longitude, latitude, and altitude).

[0116] The position and orientation sensor 224B may detect the orientation of the inspection supervisor transmitter 20B using an IMU including an angular velocity sensor and an acceleration sensor. The orientation of the inspection supervisor transmitter 20B may be represented by, for example, a three-dimensional orientation angle.

[0117] The position and orientation recognition unit 280B recognizes the position and orientation of the inspection supervisor I or a part of the body of the inspection supervisor I based on the detection results of the position and orientation of the inspection supervisor transmitter 20B by the position and orientation sensor 224B and the image of the inspection supervisor I captured by the camera 221B. The part of the body of the inspection supervisor I may be, for example, the eyes, fingers, or face of the inspection supervisor I.

[0118] More specifically, the position and orientation recognition unit 280B may recognize the position and orientation of the inspection supervisor I or a part of the body of the inspection supervisor I captured by the camera 221B from the detection results of the position and orientation of the inspection supervisor transmitter 20B (camera 221B) by the position and orientation sensor 224B. The position and orientation of the inspection supervisor I or a part of the body of the inspection supervisor I is an example of the state of the inspection supervisor I. The position and orientation recognition unit 280B may recognize a direction vector representing the orientation of the inspection supervisor I or a part of the body of the inspection supervisor I based on the position and orientation of the inspection supervisor I or a part of the body of the inspection supervisor I.

[0119] The position and posture recognition unit 280B outputs a direction vector representing the orientation of the recognized inspection supervisor I or part of the body of the inspection supervisor I to the instruction analysis unit 160B.

[0120] The gaze recognition unit 162B, the pointing recognition unit 163B, and the face recognition unit 165B each acquire points or areas in real space as supervisor instruction information based on a direction vector representing the orientation of the inspection supervisor I or part of the inspection supervisor I's body acquired from the position and posture recognition unit 180B.

[0121] As an example, an example in which a point in real space is acquired as supervisor instruction information by the pointing recognition unit 163B will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining an example in which a point in real space is acquired as supervisor instruction information by the pointing recognition unit 163B.

[0122] First, the position and orientation sensor 224B detects the position Lc (lat_cam, lon_cam, alt_cam) and orientation Ec (ex_cam, ey_cam, ez_cam) of the inspection supervisor transmitter 20B in real space.

[0123] Then, the position and orientation recognition unit 280B recognizes the direction vector representing the pointing direction of the inspection supervisor I from the recognition results of the position and orientation of the inspection supervisor I's finger based on the detection results by the position and orientation sensor 224B.

[0124] The pointing recognition unit 163B recognizes, as supervisor instruction information, the position Lt (lat_target, lon_target, alt_target) of the point where the inspection object Ob included in the captured image captured by the camera 221B or the mobile body 30B intersects with the direction vector determined by the position and orientation recognition unit 280B. Note that, when the position and orientation recognition unit 280B recognizes multiple direction vectors as a result of the movement of the finger of the inspection supervisor I, the pointing recognition unit 163B may acquire, as supervisor instruction information, the position of the area indicated by the multiple pointing directions of the operator P.

[0125] The supervisor instruction information thus acquired is output to the image synthesis unit 270B in the same manner as the supervisor instruction information in the first configuration example of this embodiment, and is also output from the communication unit 210B to the operator transmitter 10B.

[0126] When the pilot instruction information received from the communication unit 210B is a point or area in real space, the image synthesis unit 270B identifies the position of the point or area indicated by the pilot instruction information on the image captured by the moving body 30B based on the position and attitude of the moving body 30B received from the moving body 30B by the communication unit 210B.

[0127] In addition, when the supervisor instruction information acquired from the instruction analysis unit 260B is a point or area in real space, the image synthesis unit 270B identifies the position of the point or area indicated by the supervisor instruction information on the image captured by the mobile body 30B based on the position and attitude of the mobile body 30B received from the mobile body 30B by the communication unit 210B.

[0128] In addition, when the pilot instruction information received from the communication unit 210B is a point or area in real space, the image synthesis unit 270B identifies the position of the point or area indicated by the pilot instruction information on the image captured by the moving body 30B based on the position and attitude of the moving body 30B received from the moving body 30B by the communication unit 210B.

[0129] 6, the image composition unit 270B can identify a position Lt corresponding to the operator instruction information in the image captured by the moving body 30B based on the position Ld (lat_drone, lon_drone, alt_drone) and attitude Ed (ex_drone, ey_drone, ez_drone) of the moving body 30B. The same coordinate axes as those set for the position Lc and attitude Ec of the inspection supervisor transmitter 20B are set for the position Ld and attitude Ed of the moving body 30B.

[0130] Fig. 7 is a diagram showing an example of a display image generated by the image synthesis unit 270B. Fig. 7 shows an inspection supervisor transmitter 20B including a display unit 230B that displays the display image generated by the image synthesis unit 270B.

[0131] The display image displayed on the display unit 230B includes an image of the inspection object Ob captured by the mobile body 30B and a supervisor instruction mark IM2 which is a graphic that indicates supervisor instruction information. Here, the supervisor instruction mark IM2 is an illustration of a finger that indicates the position of a point in real space indicated by the pointing direction of the inspection supervisor I in Figure 4 as supervisor instruction information.

[0132] 7 shows an example in which the display image displayed on display unit 230B includes a pilot instruction mark PM2, which is a graphic that indicates pilot instruction information. Here, pilot instruction mark PM2 indicates, as pilot instruction information, the position of a point on the display image displayed on display unit 130B, which is indicated by the line of sight of pilot P in FIG.

[0133] Here, when the operator P and the inspection supervisor I indicate a point or area in real space as an instruction, the point or area may not be included in the image captured by the mobile body 30B. Figure 8 is a diagram for explaining an example in which the inspection supervisor I indicates a point or area that is not included in the image captured by the mobile body 30B as an instruction.

[0134] 8, a camera 310B of a moving body 30B captures an image of a non-inspection object Obm. Meanwhile, an inspection supervisor I indicates a part of an inspection object Ob as an instruction.

[0135] In such a case, the image composition unit 170B may indicate the direction in which a point or area exists in real space using a graphic or color as supervisor instruction information. Fig. 9 is a diagram showing an example of a display image generated by the image composition unit 270B when the direction in which a point or area exists in real space is indicated as supervisor instruction information.

[0136] FIG. 9 shows an operator transmitter 10B including a display unit 130B that displays a display image generated by an image synthesis unit 170B.

[0137] The display image displayed on display unit 130B includes an image of inspection object Obm captured by mobile body 30B and a supervisor instruction mark IM3 which is a graphic that indicates supervisor instruction information. Here, supervisor instruction mark IM3 is an arrow graphic that indicates the direction of a point in real space indicated by the pointing direction of inspection supervisor I in Figure 8 as supervisor instruction information.

[0138] 9 shows an example in which the display image displayed on display unit 130B includes a pilot instruction mark PM3, which is a graphic that indicates pilot instruction information. Here, pilot instruction mark PM3 indicates, as pilot instruction information, the position of a point on the display image displayed on display unit 130B, which is indicated by the line of sight of pilot P in FIG.

[0139] According to the second configuration example of this embodiment described above, the instructions given by the operator P and the inspection supervisor I in the real space are reflected in the display image as operator instruction information and supervisor instruction information, respectively, making each instruction easier to understand, thereby enabling the inspection work to proceed smoothly.

[0140] 2-3. Third Configuration Example Next, a third configuration example of the information processing system according to an embodiment of the present disclosure will be described with reference to FIGS. 10 and 11. FIG.

[0141] 10 is a diagram illustrating an overview of a third configuration example of an information processing system according to an embodiment of the present disclosure. As shown in Fig. 10, an information processing system 1C according to an embodiment of the present disclosure includes a pilot transmitter 10C, a moving body 30C, a display device 40, and a sensing device 50. The pilot transmitter 10C is an example of an information processing device according to this embodiment.

[0142] In the first and second configuration examples according to the present embodiment described so far, an example has been described in which the inspection supervisor I is detected by the inspection supervisor transmitter 20 (20A, 20B). Also, up to this point, an example has been described in which the inspection supervisor transmitter 20 indicates operator instruction information and supervisor instruction information to the inspection supervisor I.

[0143] In a third configuration example of this embodiment, instead of the inspection supervisor transmitter 20, an inspection supervisor I is detected by a sensing device 50, and operator instruction information and supervisor instruction information are displayed to the inspection supervisor I by a display device 40.

[0144] An example of the functional configuration of each device included in the information processing system 1C will be described in detail with reference to Fig. 11. Fig. 11 is a block diagram showing an example of the functional configuration of the information processing system 1C according to the third configuration example according to an embodiment of the present disclosure.

[0145] Here, the following mainly describes the configuration of the pilot transmitter 10C that has functions different from those of the pilot transmitter 10B described using Fig. 5, and omits a description of the configuration that is common to the pilot transmitter 10B. Also, the mobile unit 30C has the same functions as the mobile unit 30B, so a description thereof will be omitted.

[0146] The sensing device 50 is a variety of sensors that detect the inspection supervisor I. The sensing device 50 may be, for example, a smartphone equipped with various sensors. The sensing device 50 includes a camera 510, a microphone 520, and a position and orientation sensor 530. Note that the camera 510, the microphone 520, and the position and orientation sensor 530 included in the sensing device 50 may be configured separately.

[0147] The camera 510 is a camera that captures an image of the inspection supervisor I, similar to the camera 221A in the first configuration example of this embodiment described with reference to Fig. 2. The microphone 520 is a microphone that captures the voice of the inspection supervisor I, similar to the microphone 222A in the first configuration example of this embodiment described with reference to Fig. 2. The position and orientation sensor 530 is a sensor that detects the position and orientation of the sensing device 50, similar to the position and orientation sensor 224B in the second configuration example of this embodiment described with reference to Fig. 5.

[0148] The sensing device 50 outputs the detection results obtained by the camera 510, microphone 520, and position and orientation sensor 530 to the operator transmitter 10C.

[0149] The position and attitude recognition unit 180C of the pilot transmitter 10C further recognizes the position and attitude of the inspection supervisor I or part of the inspection supervisor I's body based on the detection results of the position and attitude of the sensing device 50 output from the sensing device 50 and the captured image of the inspection supervisor I.

[0150] Furthermore, the instruction analysis unit 160C further analyzes the state of the inspection supervisor I based on the detection results output from the sensing device 50. Then, the instruction analysis unit 160C acquires supervisor instruction information based on the state of the inspection supervisor I. The instruction analysis unit 160C outputs the supervisor instruction information in addition to the operator instruction information to the image synthesis unit 170C.

[0151] The image synthesis unit 170C generates a display image that reflects the pilot instruction information and supervisor instruction information acquired from the instruction analysis unit 160C. The image synthesis unit 170C outputs the display image to the display device 40 by controlling the communication unit 110C.

[0152] The display device 40 is a display device such as a liquid crystal display (LCD) device or an organic light emitting diode (OLED) device that displays various information to the inspection supervisor I. The display device 40 displays the display image received from the operator transmitter 10C to the inspection supervisor I. This allows the inspection supervisor I to accurately understand the instructions of the operator P. The display image displayed on the display device 40 may be the same as the display image displayed on the display unit 230B shown in FIG. 7 when the inspection supervisor I, operator P, and inspection object Ob are arranged as shown in FIG. 10.

[0153] 10 and 11 show an example in which the information processing system 1C is equipped with only one sensing device 50, the information processing system 1C may be equipped with multiple sensing devices 50. This allows the operator transmitter 10C to acquire the status of the inspection supervisor I more accurately.

[0154] 2-4. Fourth Configuration Example Next, a fourth configuration example of the information processing system according to an embodiment of the present disclosure will be described with reference to FIGS. 12 and 13. FIG.

[0155] 12 is a diagram illustrating an overview of a fourth configuration example of an information processing system according to an embodiment of the present disclosure. As shown in FIG. 12, an information processing system 1D according to an embodiment of the present disclosure includes a pilot transmitter 10D and a moving body 30D. The pilot transmitter 10D is an example of an information processing device according to this embodiment.

[0156] 12, in the fourth configuration example according to this embodiment, both the inspection supervisor I and the operator P check the display image displayed on the operator transmitter 10D. In the fourth configuration example according to this embodiment, an example will be described in which both the inspection supervisor I and the operator P are detected by the operator transmitter 10D, and operator instruction information and supervisor instruction information are displayed to the inspection supervisor I and the operator P.

[0157] Next, a detailed description will be given of an example of the functional configuration of each device included in the information processing system 1D with reference to Fig. 13. Fig. 13 is a block diagram illustrating an example of the functional configuration of the information processing system 1D according to a fourth configuration example according to an embodiment of the present disclosure.

[0158] Here, the following mainly describes the configuration of the pilot transmitter 10D that has functions different from those of the pilot transmitter 10B described using Fig. 5, and omits a description of the configuration that is common to the pilot transmitter 10B. Also, the mobile unit 30D has the same functions as the mobile unit 30B described using Fig. 5, so a description thereof will be omitted.

[0159] The sensor unit 120D of the operator transmitter 10D uses various sensors to detect the operator P and the inspection supervisor I. In addition, the position and attitude recognition unit 180D recognizes the positions and attitudes of the operator P and the inspection supervisor I.

[0160] Furthermore, the instruction analysis unit 160D detects the status of the operator P and the inspection supervisor I based on the detection results of the operator P and the inspection supervisor I by the sensor unit 120D. Here, the face recognition unit 165D compares pre-registered facial images of the operator P and the inspection supervisor I with the image captured by the camera 121D to determine whether each detection result is the detected operator P or the inspection supervisor I. Each component of the instruction analysis unit 160D detects the status of the operator P and the inspection supervisor I based on the determination results.

[0161] Then, the instruction analysis unit 160D acquires operator instruction information and supervisor instruction information based on the respective states of the operator P and the inspection supervisor I. The instruction analysis unit 160D outputs the operator instruction information and supervisor instruction information to the image synthesis unit 170D.

[0162] The image synthesis unit 170D generates a display image that reflects the pilot instruction information and supervisor instruction information acquired from the instruction analysis unit 160D, and outputs the generated display image to the display unit 130D.

[0163] The display unit 130D displays the display image acquired from the instruction analysis unit 160D to the operator P and the inspection supervisor I. This enables the operator P and the inspection supervisor I to accurately understand their respective instructions. The display image displayed on the display unit 130D may be the same as the display image displayed on the display unit 130A shown in FIG. 3 when the inspection supervisor I, the operator P, and the inspection object Ob are arranged as shown in FIG. 12.

[0164] 2-5. Fifth Configuration Example Next, a fifth configuration example of the information processing system according to an embodiment of the present disclosure will be described with reference to FIGS. 14 and 15. FIG.

[0165] 14 is a diagram illustrating an overview of a fifth configuration example of an information processing system according to an embodiment of the present disclosure. As shown in FIG. 14, an information processing system 1E according to an embodiment of the present disclosure includes an operator transmitter 10E and a moving object 30E.

[0166] In the fifth configuration example according to this embodiment, the mobile body 30E detects the inspection supervisor I. The mobile body 30E also recognizes the status of the inspection supervisor I. The mobile body 30E then outputs supervisor instruction information acquired based on the recognition result of the status of the inspection supervisor I to the operator transmitter 10E.

[0167] Next, a functional configuration example of each device included in the information processing system 1E will be described in detail with reference to Fig. 15. Fig. 15 is a block diagram illustrating an example of the functional configuration of the information processing system 1E according to a fifth configuration example according to an embodiment of the present disclosure.

[0168] Here, the components of the moving body 30E will be mainly described. The operator transmitter 10E has the same functions as the operator transmitter 10B, so a description thereof will be omitted. As shown in FIG. 15 , the moving body 30E includes a communication unit 330E, a sensor unit 340E, a position and orientation recognition unit 350E, and an instruction analysis unit 360E.

[0169] The communication unit 330E is connected to the operator transmitter 10E for communication, and transmits to the operator transmitter 10E images captured by a camera 310E (described later) and supervisor instruction information acquired by an instruction analysis unit 360E.

[0170] The sensor unit 340E is a group of sensors that detect the inspection supervisor I. The sensor unit 340E has a camera 310E, a microphone 341E, and a position and orientation sensor 320E.

[0171] The camera 310E is a camera that captures an image of the inspection supervisor I. The camera 310E also captures an image of the inspection object Ob. The camera 310E may be provided as a separate camera for capturing an image of the inspection supervisor I and a separate camera for capturing an image of the inspection object Ob.

[0172] The microphone 341E is a microphone that captures the voice of the inspection supervisor I. The position and orientation sensor 320E is a sensor that recognizes the position and orientation of the moving body 30E.

[0173] The position and orientation recognition unit 350E recognizes the position and orientation of the inspection supervisor I, similar to the position and orientation recognition unit 280B according to the second configuration example according to this embodiment.

[0174] Similar to the instruction analysis unit 260B according to the second configuration example of this embodiment, the instruction analysis unit 360E recognizes the state of the inspection supervisor I based on the detection results of the inspection supervisor I by the sensor unit 340E and the position and posture of the inspection supervisor I recognized by the position and posture recognition unit 350E. Then, the instruction analysis unit 360E acquires supervisor instruction information based on the detection results of the state of the inspection supervisor I. Then, the instruction analysis unit 360E controls the communication unit 330E to output the acquired supervisor instruction information to the operator transmitter 10E.

[0175] According to the fifth configuration example of this embodiment, supervisor instruction information is shown to the operator P based on the detection results of the inspection supervisor I by the mobile body 30E, allowing the operator P to accurately understand the instructions of the inspection supervisor I.

[0176] In the fifth configuration example according to this embodiment, an example has been described in which the inspection supervisor I is detected by the mobile body 30E, but the operator P may also be detected by the mobile body 30E.

[0177] <<3. Operation Processing Example>> Above, each configuration example according to this embodiment has been described. Next, an operation example of the information processing device according to this embodiment will be described with reference to FIG. 16. FIG. 16 is a flowchart showing the flow of the operation executed by the information processing device according to this embodiment. The flowchart shown in FIG. 16 can be applied to the operator transmitter 10A, inspection supervisor transmitter 20A, operator transmitter 10B, inspection supervisor transmitter 20B, operator transmitter 10C, operator transmitter 10D, and operator transmitter 10E described so far. Here, an operation example by operator transmitter 10A will be described as an example.

[0178] First, the voice recognition unit 161A determines whether or not the operator P has spoken using the microphone 122A (S101). If the voice recognition unit 161A determines that the operator P has not spoken (S101 / NO), the process proceeds to S114.

[0179] When the voice recognition unit 161A determines that the pilot P has spoken (S101 / YES), it further determines whether the speech contains a demonstrative word or a word representing a direction or an object (S102). When the voice recognition unit 161A determines that the speech does not contain a demonstrative word or a word representing a direction or an object (S102 / NO), the process proceeds to S114.

[0180] If the voice recognition unit 161A determines that the utterance includes a demonstrative word or a word representing a direction or an object (S102 / YES), the pointing recognition unit 163A determines whether pointing is detected in the image of the operator P captured by the camera 121A (S103). If pointing is detected in the image of the operator P captured by the camera 121A (S103 / YES), the pointing recognition unit 163A recognizes the direction vector of the pointing from the captured image (S104). Then, based on the recognized direction vector, the pointing recognition unit 163A outputs operator instruction information, which is instruction information indicating the point or area being pointed at, to the image synthesis unit 170A and the inspection supervisor transmitter 20A (S105).

[0181] If the pointing recognition unit 163A does not detect pointing from the captured image of the operator P (S103 / NO), the gaze recognition unit 162A determines whether a gaze is detected from the captured image of the operator P (S106). If a gaze is detected from the captured image of the operator P (S106 / YES), the gaze recognition unit 162A recognizes a directional vector of the gaze direction from the captured image (S107). Then, based on the recognized directional vector, the gaze recognition unit 162A outputs operator instruction information, which is instruction information indicating the point or area at the end of the gaze, to the image synthesis unit 170A and the inspection supervisor transmitter 20A (S108).

[0182] If the gaze recognition unit 162A does not detect a gaze from the captured image of the operator P (S106 / NO), the face recognition unit 165A determines whether a face is detected from the captured image of the operator P (S109). If a face is detected from the captured image of the operator P (S109 / YES), the face recognition unit 165A recognizes a directional vector of the face's orientation from the captured image (S110). Then, based on the recognized directional vector, the face recognition unit 165A outputs operator instruction information, which is instruction information indicating a point or area ahead of the face, to the image synthesis unit 170A and the inspection supervisor transmitter 20A (S111).

[0183] The image synthesis unit 170A then generates a display image in which the acquired operator instruction information is reflected in the captured image acquired from the moving body 30A, and controls the display on the display unit 130A (S112).

[0184] On the other hand, if the gaze recognition unit 162A does not detect the gaze of the pilot P from the captured image (S109 / NO), the image synthesis unit 170A generates a display image including a message notifying the user that the instruction point corresponding to the utterance is unknown, and controls the display on the display unit 130A (S113).

[0185] Next, the image composition unit 170A determines whether other instruction information, in this case supervisor instruction information, has been acquired from the inspection supervisor transmitter 20A by the communication unit 110 (S114). If the image composition unit 170A has acquired the supervisor instruction information from the inspection supervisor transmitter 20A by the communication unit 110 (S114 / YES), the image composition unit 170A generates a display image that reflects the supervisor instruction information in the captured image acquired from the mobile object 30A, and controls the display on the display unit 130A (S115). If the communication unit 110 has not acquired the supervisor instruction information from the inspection supervisor transmitter 20A (S114 / NO), the process proceeds to S116.

[0186] If the operator P does not perform an end operation (S116 / NO), the operator transmitter 10A repeats the processes of S101 to S115 until an end operation is performed by the operator P. If the operator P performs an end operation (S116 / YES), the operator transmitter 10A ends the process. The end operation may be, for example, an operation in which the operator P presses an end button displayed on the display unit 130A.

[0187] <<4. Modifications>> The above has described an example in which, when supervisor instruction information and pilot instruction information are acquired, the supervisor instruction information and pilot instruction information are displayed using supervisor instruction marks and pilot instruction marks (hereinafter simply referred to as "instruction marks"). Such supervisor instruction information and pilot instruction information may be displayed in different formats depending on their respective reliability levels.

[0188] The reliability may be calculated, for example, based on the distance in real space from the moving body 30 (30A-30E) to the point or area indicated by the supervisor instruction information and the pilot instruction information. The reliability may be calculated so that it is higher the closer the point or area is to the moving body 30. This is because it is expected that the closer the point or area is to the moving body 30, the higher the image resolution of the point or area. The reliability may also be calculated based on the degree of agreement between each piece of instruction information (supervisor instruction information or pilot instruction information) acquired based on the contact position, pointing direction, gaze direction, and facial orientation, respectively. The higher the degree of agreement between each piece of instruction information, the higher the reliability may be calculated.

[0189] 17 is a diagram showing an example of a legend for indication marks that are displayed in different formats depending on the reliability. For example, as shown in indication mark legend MG1, the indication marks may be displayed in different colors depending on the reliability. Furthermore, as shown in indication mark legend MG2, the indication marks may be displayed at different blinking rates depending on the reliability. Furthermore, as shown in indication mark legend MG3, the indication marks may be displayed in different color intensities and color ranges depending on the reliability.

[0190] <<5. Hardware Configuration Example>> Each embodiment of the present disclosure has been described above. The information processing described above is realized by a combination of software and hardware. Below, a hardware configuration example that can be applied to the pilot transmitter 10 (10A-10E), the inspection supervisor transmitter 20 (20A and 20B), and the mobile object 30 (30A-30E) will be described.

[0191] Fig. 18 is a block diagram showing an example of an information processing device 90. Note that the hardware configuration example of the information processing device 90 described below is merely an example of the hardware configuration of the operator transmitter 10, the inspection supervisor transmitter 20, and the mobile body 30. Therefore, the operator transmitter 10, the inspection supervisor transmitter 20, and the mobile body 30 do not necessarily have to have all of the hardware configuration shown in Fig. 18 . Furthermore, the operator transmitter 10, the inspection supervisor transmitter 20, and the mobile body 30 may not have part of the hardware configuration shown in Fig. 18 .

[0192] 18 , the information processing device 90 includes a CPU 901, a ROM (Read Only Memory) 903, and a RAM 905. The information processing device 90 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Instead of or in addition to the CPU 901, the information processing device 90 may include a processing circuit such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an ASIC (Application Specific Integrated Circuit).

[0193] The CPU 901 functions as an arithmetic processing device and control device, and controls all or part of the operations within the information processing device 90 in accordance with various programs recorded in the ROM 903, RAM 905, storage device 919, or removable recording medium 927. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 temporarily stores programs used in the execution of the CPU 901 and / or parameters that change as appropriate during the execution. The CPU 901, ROM 903, and RAM 905 are interconnected by a host bus 907, which is composed of an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0194] The CPU 901, working in cooperation with the ROM 903, RAM 905, and software, can realize the functions of, for example, the instruction analysis unit 160 (160A to 160E), the instruction analysis unit 260 (260A and 160B), and the instruction analysis unit 360E.

[0195] The input device 915 is a device operated by a user, such as a button. The input device 915 may include a mouse, a keyboard, a touch panel, a switch, a lever, or the like. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device 929 such as a mobile phone that supports operation of the information processing device 90. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the signal to the CPU 901. The user operates the input device 915 to input various data and instruct processing operations to the information processing device 90.

[0196] The input device 915 may also include an imaging device and a sensor. The imaging device is a device that captures real space and generates a captured image using an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and various components such as a lens for controlling the formation of a subject image on the imaging element. The imaging device may capture still images or may capture moving images.

[0197] The sensors are various types of sensors, such as a distance measurement sensor, an acceleration sensor, a gyro sensor, a geomagnetic sensor, a vibration sensor, a light sensor, a sound sensor, etc. The sensors acquire information about the state of the information processing device 90 itself, such as the attitude of the housing of the information processing device 90, or information about the surrounding environment of the information processing device 90, such as the brightness or noise around the information processing device 90. The sensors may also include a Global Positioning System (GPS) sensor that receives GPS signals to measure the latitude, longitude, and altitude of the device.

[0198] The output device 917 is configured with a device capable of visually or audibly notifying the user of acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, or the like. The output device 917 outputs the results obtained by the processing of the information processing device 90 as video such as text or images, or as sound such as voice or audio. The output device 917 may also include a lighting device that brightens the surrounding area.

[0199] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 90. The storage device 919 is configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs or various data executed by the CPU 901, as well as various data acquired from the outside.

[0200] The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 90. The drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905. The drive 921 also writes information to the attached removable recording medium 927.

[0201] The connection port 923 is a port for directly connecting a device to the information processing device 90. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, or the like. The connection port 923 may also be an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, or the like. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 90 and the external device 929.

[0202] The communication device 925 is a communication interface configured with, for example, a communication device for connecting to a local network or a communication network with a wireless communication base station. The communication device 925 may be, for example, a communication card for a wired or wireless LAN, Bluetooth, Wi-Fi, or WUSB (Wireless USB). The communication device 925 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 925 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The local network or communication network with the base station connected to the communication device 925 is a wired or wireless network, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0203] <<6. Supplementary Information>> Although preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0204] For example, in the above embodiment, an example has been described in which the operator instruction information and supervisor instruction information are presented to the operator P and the inspection supervisor I by display images, but the operator instruction information and supervisor instruction information may also be presented by outputting sound or by providing a tactile sensation such as vibration. For example, when the points or areas indicated by the operator instruction information and the supervisor instruction information are different, sound or vibration may be output, or the direction of the point or area indicated by the supervisor instruction information relative to the point or area indicated by the operator instruction information may be output to the operator P by sound.

[0205] In the above embodiment, an example has been described in which the pilot instruction information and the supervisor instruction information are displayed as a two-dimensional display image, but the pilot instruction information and the supervisor instruction information may also be displayed as a three-dimensional display image. It is anticipated that the depths of the points or areas indicated by the pilot instruction information and the supervisor instruction information may differ in the two-dimensional display image, and such a configuration makes it possible to more clearly express instructions. Note that when a two-dimensional display image is displayed, if the depths of the points or areas indicated by the pilot instruction information and the supervisor instruction information differ, information regarding the depths may be displayed or output as sound.

[0206] Furthermore, in the above embodiment, an example has been described in which an image of the inspection object Ob is captured by a moving body, but the image of the inspection object Ob may also be obtained by a fixed camera or the like.

[0207] It is also possible to create a computer program for causing hardware such as a CPU, ROM, and RAM built into the above-mentioned operator transmitter 10, inspection supervisor transmitter 20, or mobile body 30 to perform the functions of the operator transmitter 10, inspection supervisor transmitter 20, or mobile body 30. A computer-readable storage medium storing the computer program is also provided.

[0208] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0209] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing method executed by a computer, comprising: receiving an image; recognizing a user's utterance; acquiring a state of the user based on a recognition result of the utterance; acquiring instruction information indicating an instruction by the user based on the user's state; and generating a display image that reflects the instruction information. (2) The information processing method described in (1), wherein the recognition result includes a recognition result as to whether or not a demonstrative term is included in the utterance by the user. (3) The information processing method described in (1) or (2), wherein the recognition result includes a recognition result as to whether or not a word indicating an object or a direction is included in the utterance by the user. (4) The information processing method described in any of (1) to (3), wherein the state of the user is acquired based on at least one of a detection result of an object held or worn by the user, or a position or posture of the user's body or a part of the user's body. (5) The information processing method according to (4), wherein the state of the user includes the user's pointing direction, line of sight direction, facial orientation, body orientation, or finger contact position. (6) The information processing method according to (4) or (5), wherein the instruction information is information about a point or area indicated by the direction in which the user or a part of the user's body is facing, or the position or posture of an object held or worn by the user. (7) The information processing method according to (4), wherein the object held or worn by the user is a pointing device. (8) The information processing method according to any of (4) to (7), wherein the state of the user is acquired based on a detection result by a sensor. (9) The information processing method according to (8), wherein the sensor includes at least one of an image sensor, a distance measurement sensor, an acceleration sensor, an infrared sensor, and a touch sensor.(10) The information processing method according to any one of (1) to (9), wherein the user's state includes a plurality of detection results among detection results of the position or posture of an object held or worn by the user, or the user's body or a part of the user's body, and the instruction information reflected in the image is determined based on a priority order preset for the plurality of detection results. (11) The information processing method according to (10), wherein generating the display image includes reflecting the instruction information in a different format depending on a detection result used to acquire the instruction information among the plurality of detection results acquired as the user's state. (12) The information processing method according to any one of (1) to (11), wherein the instruction information is reflected in the display image by a shape or color, and a point or area corresponding to the instruction information is reflected in the display image. (13) The information processing method according to any one of (1) to (12), wherein the image is captured by an imaging device mounted on a moving object. (14) The information processing method according to (13), wherein the moving object includes an air vehicle. (15) The information processing method according to any one of (1) to (14), wherein the image is a captured image of a building. (16) The information processing method according to (13) or (14), wherein the display image is displayed on a controller for operating the moving object. (17) The information processing method according to (13), (14), or (16), wherein the state of the user is acquired based on a captured image of the user captured by an imaging device mounted on the moving object. (18) The information processing method according to any one of (1) to (17), wherein generating the display image includes reflecting each of instruction information corresponding to a plurality of users. (19) An information processing device comprising: a communication unit that receives an image; a recognition unit that recognizes a user's utterance; a state acquisition unit that acquires the user's state based on a recognition result of the utterance; an instruction information acquisition unit that acquires instruction information indicative of an instruction by the user based on the user's state; and an image generation unit that generates a display image in which the instruction information is reflected in the image.(20) A program that causes a computer to operate as: a communication unit that receives an image; a recognition unit that recognizes a user's speech; a status acquisition unit that acquires the user's status based on the recognition result of the speech; an instruction information acquisition unit that acquires instruction information indicating an instruction by the user based on the user's status; and an image generation unit that generates a display image that reflects the instruction information in the image.

[0210] 1A to 1E Information processing system 10A to 10E Operator transmitter 110A to 110E Communication unit 120A to 120E Sensor unit 130A to 130E Display unit 160A to 160E Instruction analysis unit 170A to 170E Image synthesis unit 180B to 180E Position and orientation recognition unit 20A, 20B Inspection supervisor transmitter 210A, 210B Communication unit 220A, 220B Sensor unit 230A, 230B Display unit 270A, 270B Image synthesis unit 280B Position and orientation recognition unit 30A to 30E Mobile body 40 Display device 50 Sensing device

Claims

1. An information processing method executed by a computer, comprising: receiving an image; recognizing a user's speech; acquiring the user's state based on the recognition result of the speech; acquiring instruction information indicating an instruction by the user based on the user's state; and generating a display image that reflects the instruction information in the image.

2. The information processing method according to claim 1, wherein the recognition result includes a recognition result as to whether or not a demonstrative pronoun is included in the utterance by the user.

3. The information processing method according to claim 1, wherein the recognition result includes a recognition result as to whether or not the user's utterance includes a word representing an object or a direction.

4. The information processing method according to claim 1, wherein the state of the user is acquired based on at least one of the detection results of the position or posture of an object held or worn by the user, or the user's body or a part of the body.

5. The information processing method according to claim 4, wherein the state of the user includes the user's pointing direction, line of sight direction, facial orientation, body orientation, or finger contact position.

6. The information processing method of claim 4, wherein the instruction information is information about a point or area indicated by the direction in which the user or a part of the user's body is facing, or the position or posture of an object held or worn by the user.

7. The information processing method according to claim 4, wherein the object held or worn by the user is a pointing device.

8. The information processing method according to claim 4, wherein the user's state is acquired based on a detection result by a sensor.

9. The information processing method according to claim 8, wherein the sensor includes at least one of an image sensor, a distance measurement sensor, an acceleration sensor, an infrared sensor, and a touch sensor.

10. The information processing method of claim 1, wherein the user's state includes a plurality of detection results of the position or posture of an object held or worn by the user, or the user's body or part of the body, and the instruction information reflected in the image is determined based on a predetermined priority for the plurality of detection results.

11. An information processing method as described in claim 10, wherein generating the display image includes reflecting the instruction information in different formats depending on which of the multiple detection results obtained as the user's state was used to obtain the instruction information.

12. The information processing method according to claim 1, wherein the instruction information is a point or area that corresponds to the instruction information and is reflected on the displayed image by a shape or color.

13. The information processing method according to claim 1, wherein the image is captured by an imaging device mounted on a moving object.

14. The information processing method according to claim 13, wherein the moving object includes an air vehicle.

15. The information processing method according to claim 1, wherein the image is a captured image of a building.

16. The information processing method according to claim 13, wherein the display image is displayed on a controller for operating the moving object.

17. The information processing method according to claim 13, wherein the state of the user is acquired based on an image of the user captured by an imaging device mounted on the moving object.

18. The information processing method according to claim 1, wherein generating the display image includes reflecting each of instruction information corresponding to a plurality of users.

19. An information processing device comprising: a communication unit that receives an image; a recognition unit that recognizes a user's speech; a status acquisition unit that acquires the user's state based on the recognition result of the speech; an instruction information acquisition unit that acquires instruction information indicating an instruction by the user based on the user's state; and an image generation unit that generates a display image that reflects the instruction information in the image.

20. A program that causes a computer to operate as: a communication unit that receives images; a recognition unit that recognizes user utterances; a status acquisition unit that acquires the user's status based on the recognition results of the utterances; an instruction information acquisition unit that acquires instruction information indicating instructions given by the user based on the user's status; and an image generation unit that generates a display image that reflects the instruction information in the image.

Images