Information processing method, information processing program, and information processing device

Abstract

An information processing device 10 includes a detection unit 15B, an identification unit 15C, and an output control unit 15D. The detection unit 15B detects a person included in the captured video of the surrounding of a mobile body 20. The identification unit 15C identifies a guide member for guiding the mobile body 20 among the detected persons who are included in the captured video. The output control unit 15D outputs a display screen in which the identified guide member and included in the captured video is highlighted to an operator terminal 30 of an operator who remotely controls the mobile body 20.

Description

Information processing method, information processing program, and information processing device

[0001] An embodiment of the present invention relates to an information processing method, an information processing program, and an information processing device.

[0002] In recent years, systems have become known in which an operator remotely controls an autonomously moving object, such as an autonomous vehicle, in response to a request from the moving object, etc. In such systems, the moving object may be remotely controlled in accordance with the guidance of a guide present in the real world.

[0003] When remote control is performed in accordance with the guidance of a guide, the operator performs remote control according to the actions of the guide included in the captured video around the moving object. However, it is difficult for the operator to immediately recognize which person in the captured video is the guide, and there are cases where the operator is unable to provide appropriate remote control support.

[0004] Japanese Patent Application Laid-Open No. 2022-173769

[0005] The present invention has been made in view of the above, and has an object to provide an information processing method, an information processing program, and an information processing device that can provide suitable remote control support.

[0006] The information processing method of the embodiment is an information processing method executed by an information processing device, and includes a detection step of detecting a person included in a captured image around a moving body, an identification step of identifying a guide who guides the moving body from among the detected people included in the captured image, and an output control step of outputting a display screen highlighting the identified guide who is included in the captured image to an operator terminal of an operator who remotely controls the moving body.

[0007] FIG. 1 is an explanatory diagram of an example of an information processing system according to an embodiment. FIG. 2 is a functional block diagram of an example of an information processing system according to an embodiment. FIG. 3A is an explanatory diagram of an example of a first identification method. FIG. 3B is an explanatory diagram of an example of the first identification method. FIG. 4A is an explanatory diagram of an example of a second identification method. FIG. 4B is an explanatory diagram of an example of the second identification method. FIG. 5A is an explanatory diagram of an example of a third identification method. FIG. 5B is an explanatory diagram of an example of the third identification method. FIG. 6A is a schematic diagram of an example of a display screen in which a guide is emphasized. FIG. 6B is a schematic diagram of an example of a display screen in which a guide is emphasized. FIG. 6C is a schematic diagram of an example of a display screen in which a guide is emphasized. FIG. 6D is a schematic diagram of an example of a display screen in which a guide is emphasized. FIG. 6E is a schematic diagram of an example of a display screen in which a guide is emphasized. FIG. 7A is a schematic diagram of an example of a display screen in which an error message is displayed. FIG. 7B is a schematic diagram of an example of a display screen in which an error message is displayed. FIG. 8 is a sequence diagram showing an example of the flow of information processing executed in the information processing system according to this embodiment. 9 is a flowchart showing an example of the flow of information processing executed by the information processing apparatus of this embodiment. FIG. 10 is a hardware configuration diagram.

[0008] Hereinafter, embodiments of an information processing method, an information processing program, and an information processing device will be described in detail with reference to the accompanying drawings.

[0009] FIG. 1 is an explanatory diagram of an example of an information processing system 1 according to this embodiment.

[0010] The information processing system 1 includes an information processing device 10, one or more mobile objects 20, and one or more operator terminals 30. The mobile objects 20 and the information processing device 10 are communicatively connected via a network NW or the like. The information processing device 10 and the operator terminal 30 are communicatively connected via a network NW or the like.

[0011] The network NW that communicatively connects the information processing device 10 and the mobile object 20 and the network NW that communicatively connects the information processing device 10 and the operator terminal 30 may be the same network or different networks.

[0012] The information processing device 10 is an information processing device that executes management of the mobile object 20 and the operator terminal 30. The information processing device 10 is a dedicated or general-purpose computer.

[0013] The mobile object 20 is an autonomously movable object that is remotely controlled by an operator OP. The information processing system 1 includes one or more mobile objects 20.

[0014] The remote control refers to a task related to the remote control of the mobile object 20 by the operator OP. The remote control includes at least one of remote operation of the mobile object 20 and remote monitoring of the mobile object 20.

[0015] The moving body 20 is, for example, an autonomously moving vehicle, a robot having a movement function and moving autonomously, etc. An autonomously moving vehicle may be called a self-driving vehicle. In this embodiment, a form in which the moving body 20 is a vehicle capable of autonomous movement will be described as an example.

[0016] The operator terminal 30 is an information processing device for remotely controlling the mobile object 20. The operator terminal 30 is operated by an operator OP who is in charge of remotely controlling the mobile object 20.

[0017] In this embodiment, the mobile body 20 is located at an airport in real space S, and an operator OP remotely controls one mobile body 20 under the guidance of a guide G to a destination position T on an airplane AP at the airport.

[0018] The airplane AP is an example of an object having a destination position T. The destination position T is a destination to which the moving body 20 is moved under the guidance of a guide G, or a destination where various tasks are performed by the moving body 20. In this embodiment, a case where the destination position T is the boarding door of the airplane AP will be described as an example. Note that the object may be anything that exists in the real space S, and is not limited to the airplane AP. Furthermore, the destination position T may be any destination of the moving body 20 in the real space S, and is not limited to the boarding door of the airplane AP.

[0019] The guide G is a person who guides the moving object 20. One or more guides G exist in the real space S. In this embodiment, a description will be given assuming a scene in which a plurality of guides G exist near an airplane AP. These plurality of guides G differ from one another in at least one of the scene in which they guide the moving object 20, the timing at which they guide the moving object 20, and the moving object 20 to be guided. For this reason, the description will be given assuming a case in which one guide G guides one moving object 20 at the timing of a certain guidance period in a certain scene.

[0020] The guide G may carry a guide terminal 40. The guide terminal 40 is connected to the information processing device 10 via the network NW so as to be able to communicate with the information processing device 10.

[0021] The guide terminal 40 is an information processing device carried by the guide G. The guide terminal 40 may be any information processing device that is capable of transmitting at least the position information of the guide G in the real space S to the information processing device 10. The guide terminal 40 is, for example, a smartphone, a tablet terminal, or the like.

[0022] FIG. 2 is a functional block diagram of an example of the information processing system 1 according to the present embodiment.

[0023] The mobile object 20 includes a communication unit 21, an input unit 22, an output unit 23, a sensor 24, a camera 25, a drive unit 26, a storage unit 27, and a control unit 28. The communication unit 21, the input unit 22, the output unit 23, the sensor 24, the camera 25, the drive unit 26, the storage unit 27, and the control unit 28 are communicatively connected via a bus or the like.

[0024] The communication unit 21 communicates with the information processing device 10 via a network NW or the like. The input unit 22 accepts various operations by a user. The output unit 23 outputs various information. The output unit 23 may be a display that displays various information, a speaker that outputs various sounds, a light that lights up or flashes, etc.

[0025] The sensor 24 detects the state of the moving body 20. The sensor 24 is, for example, an internal sensor and an external sensor.

[0026] The internal sensors are sensors that obtain observation information about the internal environment of the mobile object 20. The observation information includes information such as the position, speed, acceleration, steering angle, steering direction, accelerator depression angle, and vibration of the mobile object 20. Specifically, the internal sensors include at least a position sensor (e.g., a Global Navigation Satellite System (GNSS) or a Global Positioning System (GPS)). The internal sensors also include, for example, an inertial measurement unit (IMU), an acceleration sensor, a speed sensor, a speed sensor such as a gyro sensor, a rotary encoder, a steering angle sensor, a steering direction sensor, an accelerator depression angle detection sensor, and a vibration sensor.

[0027] The external sensor is a sensor that obtains distance information, which indicates the distance from the mobile object 20 to an object in the vicinity of the mobile object 20.

[0028] The external sensor is, for example, a distance sensor. The distance sensor is, for example, a millimeter-wave radar, a laser sensor, a distance image sensor, etc. The laser sensor is, for example, a two-dimensional LiDAR (Laser Imaging Detection and Ranging) sensor or a three-dimensional LiDAR sensor installed parallel to a horizontal plane.

[0029] The sensor 24 sequentially detects the state of the moving body 20 in chronological order, and sequentially outputs the detection results to the control unit 28 .

[0030] The camera 25 captures images of the surroundings of the moving body 20 to obtain captured image data. In this embodiment, the captured image data will be referred to as captured image. The moving body 20 may be provided with a plurality of cameras 25 each with a different capturing direction.

[0031] The driving unit 26 is a dedicated control device for devices mounted on the mobile body 20 and driving the devices related to the movement of the mobile body 20. The driving unit 26 is, for example, an ECU (Electronic Control Unit) that controls the engine, a motor driver that controls the motor, etc. The driving unit 26 controls the driving of the devices under the control of the control unit 28.

[0032] The storage unit 27 stores various types of data. The storage unit 27 is, for example, a random access memory (RAM), a semiconductor memory element such as a flash memory, a hard disk, an optical disk, etc. The storage unit 27 may be a storage device provided outside the mobile object 20. The storage unit 27 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a local area network (LAN), the Internet, etc.

[0033] The control unit 28 is a computer that executes information processing in the mobile object 20 .

[0034] The control unit 28 includes a communication control unit 28A and a drive control unit 28B. The communication control unit 28A and the drive control unit 28B are realized by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC, i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0035] The communication control unit 28A controls communication with the information processing device 10. For example, the communication control unit 28A transmits mobile object information to the information processing device 10 via the communication unit 21.

[0036] The mobile object information is information related to the traveling environment of the mobile object 20. The mobile object information includes, for example, a mobile object ID (IDentification), the detection results of the sensor 24, the video captured by the camera 25, and the camera coordinate system information of the camera 25. The mobile object ID is identification information of the mobile object 20.

[0037] The camera coordinate system information is information representing a camera coordinate system in which the center position of the lens of the camera 25 in the real space S is the origin and the Z-axis direction is the principal axis direction of the lens of the camera 25. The camera coordinate system information is represented, for example, by a coordinate system indicating the center position of the lens of the camera 25 in the real space S and the attitude (orientation) of the camera 25. The camera coordinate system information of the camera 25 mounted on the moving body 20 may be detected, for example, by the sensor 24, or may be input by a user operating the input unit 22.

[0038] The drive control unit 28B controls the drive unit 26 to autonomously move the mobile body 20. The drive control unit 28B determines the situation around the mobile body 20 based on the detection results of the sensor 24, etc., and controls the accelerator amount, brake amount, steering angle, etc., to control the drive unit 26 so that the mobile body 20 moves autonomously.

[0039] Furthermore, the drive control unit 28B executes remote driving based on remote operation information received from the operator terminal 30 via the information processing device 10. Remote driving means running and driving in accordance with remote operation by the operator terminal 30. During remote driving, the drive control unit 28B controls the accelerator amount, brake amount, steering angle, etc. based on the remote operation information received from the operator terminal 30, and controls the drive unit 26 so that the mobile body 20 runs in accordance with the operation instructions from the operator terminal 30. Furthermore, the drive control unit 28B controls the drive unit 26 based on the remote operation information, thereby controlling the drive unit 26 so that the mobile body 20 drives in accordance with the operation instructions from the operator terminal 30.

[0040] Next, the operator terminal 30 will be described.

[0041] The operator terminal 30 includes a communication unit 31, an input unit 32, an output unit 33, a storage unit 34, and a control unit 35. The communication unit 31, the input unit 32, the output unit 33, the storage unit 34, and the control unit 35 are communicatively connected via a bus or the like.

[0042] The communication unit 31 communicates with the information processing device 10 via a network NW or the like. The input unit 32 accepts various operations by an operator OP. For example, the input unit 32 is an input device such as a touch panel or a keyboard, a microphone for collecting sound, etc. The output unit 33 outputs various information. The output unit 33 is a display for displaying various information, a speaker for outputting various sounds, etc.

[0043] The storage unit 34 stores various types of data. The storage unit 34 is, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, etc. The storage unit 34 may be a storage device provided outside the operator terminal 30. The storage unit 34 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN, the Internet, etc.

[0044] The control unit 35 is a computer that executes information processing in the operator terminal 30 .

[0045] The control unit 35 includes a communication control unit 35A, an output control unit 35B, and a remote control control unit 35C. The communication control unit 35A, the output control unit 35B, and the remote control control unit 35C are implemented by one or more processors. For example, each of the above units may be implemented by a processor such as a CPU executing a program, i.e., by software. Each of the above units may be implemented by a processor such as a dedicated IC, i.e., by hardware. Each of the above units may be implemented by a combination of software and hardware. When multiple processors are used, each processor may implement one of the units, or two or more of the units. Furthermore, at least one of the above units included in the control unit 35 may be installed in an external information processing device communicatively connected to the operator terminal 30 via a network NW or the like.

[0046] The communication control unit 35A controls communication with the information processing device 10 .

[0047] The output control unit 35B displays the display screen received from the information processing device 10 via the communication control unit 35A and the communication unit 31 on the display of the output unit 33. The display screen includes a captured image of the surroundings of the mobile object 20 captured by the mobile object 20. The operator OP can operate the input unit 32 while visually checking the display screen displayed on the output unit 33, thereby enabling remote control such as remote monitoring and remote operation of the mobile object 20 (described in detail later).

[0048] The remote control control unit 35C remotely controls the mobile body 20 in response to operation of the input unit 32 by the operator OP. For example, the remote control control unit 35C transmits remote operation information corresponding to an operation instruction for the mobile body 20 input in response to operation of the input unit 32 by the operator OP, together with the mobile body ID of the mobile body 20, to the information processing device 10. The information processing device 10 transmits the remote operation information to the mobile body 20 identified by the mobile body ID. The mobile body 20 controls the drive unit 26 based on the remote operation information, thereby moving and driving the mobile body 20 in response to remote control by the operator OP. Therefore, the mobile body 20 is remotely controlled by the operator OP operating the operator terminal 30.

[0049] Next, a description will be given of the guide terminal 40. The guide terminal 40 is an information processing device carried by the guide G.

[0050] The guide terminal 40 includes a communication unit 41, an input unit 42, an output unit 43, a sensor 44, a storage unit 45, and a control unit 46. The communication unit 41, the input unit 42, the output unit 43, the sensor 44, the storage unit 45, and the control unit 46 are communicatively connected via a bus or the like.

[0051] The communication unit 41 communicates with the information processing device 10 via a network NW or the like. The input unit 42 accepts various operations by the guide G. For example, the input unit 42 is an input device such as a touch panel or a keyboard, a microphone that collects sound, or the like. The output unit 43 outputs various information. The output unit 43 is a display that displays various information, a speaker that outputs various sounds, or the like.

[0052] The sensor 44 is a sensor that detects the state of the guide terminal 40. In this embodiment, the sensor 44 includes at least a position sensor (GNSS, GPS, etc.). The storage unit 45 stores various data. The storage unit 45 is, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, an optical disk, etc. The storage unit 45 may be a storage device provided outside the guide terminal 40. The storage unit 45 may also be a storage medium that stores or temporarily stores programs and various information downloaded via a LAN, the Internet, etc.

[0053] The control unit 46 is a computer that executes information processing in the guide terminal 40. The control unit 46 includes a communication control unit 46A. The communication control unit 46A is implemented by one or more processors. For example, the communication control unit 46A may be implemented by causing a processor such as a CPU to execute a program, i.e., by software.

[0054] The communication control unit 46A controls communication with the information processing device 10. In this embodiment, the communication control unit 46A transmits the position information of the guide terminal 40 detected by the sensor 44 and the guide ID of the guide G operating the guide terminal 40 to the information processing device 10. The guide ID is identification information of the guide G.

[0055] Next, the information processing device 10 will be described.

[0056] The information processing device 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15. The communication unit 11, the input unit 12, the output unit 13, the storage unit 14, and the control unit 15 are communicatively connected via a bus or the like.

[0057] The communication unit 11 communicates with the mobile object 20, the operator terminal 30, the guide terminal 40, etc. via a network NW, etc. The input unit 12 accepts various operations by a user. The output unit 13 outputs various information. The output unit 13 is a display that displays various information, a speaker that outputs various sounds, etc.

[0058] The storage unit 14 stores various types of data. The storage unit 14 is, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like. The storage unit 14 may be a storage device provided outside the information processing device 10. The storage unit 14 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN, the Internet, or the like. At least a portion of the information stored in the storage unit 14 may also be stored in a storage device such as an external server device communicably connected to the information processing device 10.

[0059] The control unit 15 is a computer that executes information processing in the information processing device 10 .

[0060] The control unit 15 includes an acquisition unit 15A, a detection unit 15B, an identification unit 15C, and an output control unit 15D. The acquisition unit 15A, the detection unit 15B, the identification unit 15C, and the output control unit 15D are implemented by one or more processors. For example, each of the above units may be implemented by a processor such as a CPU executing a program, i.e., software. Each of the above units may be implemented by a processor such as a dedicated IC, i.e., hardware. Each of the above units may be implemented by a combination of software and hardware. When multiple processors are used, each processor may implement one of the units, or two or more of the units. Furthermore, at least one of the above units included in the information processing device 10 may be installed in an external information processing device communicatively connected to the information processing device 10 via a network NW or the like.

[0061] The acquisition unit 15A acquires various types of information from each of the moving object 20, the operator terminal 30, and the guide terminal 40 via the communication unit 11.

[0062] For example, when the acquisition unit 15A receives remote operation information from the operator terminal 30, the acquisition unit 15A transmits the remote operation information to the mobile object 20 identified by the mobile object ID included in the remote operation information.

[0063] The acquisition unit 15A also receives mobile object information from the mobile object 20. The acquisition unit 15A can also acquire the location information of the guide terminal 40 and the guide ID from the guide terminal 40. The acquisition unit 15A may also be able to acquire the location information of the airplane AP and the identification information of the airplane AP from the airplane AP, an external server device, or the like.

[0064] The detection unit 15B detects people included in the captured video around the mobile body 20. The detection unit 15B reads the captured video captured by the camera 25 of the mobile body 20, which is included in the mobile body information received from the mobile body 20. The detection unit 15B then analyzes the captured video using a known method that uses AI (Artificial Intelligence) learned by deep learning or the like, thereby detecting people included in the captured video.

[0065] The identification unit 15C identifies, from among the people detected in the photographed video, the guide G who guides the moving body 20. That is, from among the people included in the photographed video detected by the detection unit 15B, the identification unit 15C identifies the guide G who guides the moving body 20 on which the camera 25 that captured the photographed video is mounted.

[0066] In this embodiment, the identification unit 15C uses any one of the first to third identification methods to identify, from among the people included in the captured video, the guide G who guides the mobile body 20 equipped with the camera 25 that captured the captured video.

[0067] First, the first specifying method will be described.

[0068] 3A and 3B are diagrams illustrating an example of the first identification method.

[0069] 3A is an explanatory diagram of an example of the positional relationship between a destination position T in real space S and the camera 25 of the moving body 20. For example, assume a scene in real space S in which a plurality of guides G (e.g., guide G1, guide G2) are present near an airplane AP having a destination position T, and the moving body 20 is located near the airplane AP. Also assume a case in which guide G1 of the plurality of guides G is the guide G guiding the moving body 20. As described above, the control unit 28 of the moving body 20 transmits moving body information to the information processing device 10, the moving body ID of the moving body 20, the video captured by the camera 25 of the moving body 20, the detection result of the sensor 24 including the position information of the moving body 20, and the camera coordinate system information of the camera 25.

[0070] 3B is a schematic diagram of an example of a captured image F captured by the camera 25 of the moving object 20 shown in FIG. For example, assume a scene in which the detection unit 15B analyzes the captured image F and detects two persons P, a person P1 and a person P2, appearing in the captured image F.

[0071] In the first identification method, the identification unit 15C identifies second destination position information LI2 on the display screen 50 of the destination position T based on first destination position information LI1 in the real space S of the destination position T of the moving object 20. Then, the identification unit 15C identifies, as the guide G, a person P who is present within a predetermined range on the display screen 50 from the position represented by the second destination position information LI2, among the detected people P included in the captured video F.

[0072] Specifically, the identification unit 15C first acquires first destination position information LI1. The first destination position information LI1 is information that indicates the location of the destination position T of the moving object 20 in the real space S, as shown in FIG. 3A.

[0073] For example, the identification unit 15C acquires airplane information including position information of the airplane AP in the real space S and model information of the airplane AP from the airplane AP, etc., via the network NW and the communication unit 11. Furthermore, for each model of airplane identified by the model information of the airplane AP, the identification unit 15C associates in advance relative positional relationship information representing a relative positional relationship between the position represented by the position information of the airplane AP of that model and a destination position T, such as a boarding / disembarking door, provided on the airplane AP of that model, and stores the information in the storage unit 14. The identification unit 15C then uses the relative positional relationship information corresponding to the model information of the airplane AP included in the airplane information received from the airplane AP to derive first destination position information LI1, which is the location in the real space S of the destination position T provided on the airplane AP, from the position information of the airplane AP included in the airplane information. Through this derivation process, the identification unit 15C acquires first destination position information LI1 of the destination position T in the real space S.

[0074] Furthermore, the identification unit 15C may acquire the position information as the first destination position information LI1 by receiving the position information from a detection device that detects the position information of the destination position T in the real space S, which is provided on the airplane AP. The detection device may be any device that can detect the current position information of the destination position T of the airplane AP. The detection device may be, for example, a GPS device provided at the destination position T of the airplane AP, but is not limited to this.

[0075] Furthermore, the identification unit 15C may acquire the first destination position information LI1 by identifying position information representing a destination position T in the real space S of the autonomous traveling of the moving body 20. For example, the identification unit 15C may previously associate a moving body ID that identifies the moving body 20 with first destination position information LI1 in the real space S of the destination position T of the moving body 20 identified by the moving body ID, and store these in the storage unit 14. The identification unit 15C may acquire the first destination position information LI1 by reading, from the storage unit 14, the first destination position information LI1 that corresponds to the moving body ID included in the moving body information received from the moving body 20. Furthermore, the identification unit 15C may acquire the first destination position information LI1 by receiving, from the moving body 20, the first destination position information LI1 of the destination position T in the real space S of the autonomous traveling of the moving body 20.

[0076] The identification unit 15C then identifies second destination position information LI2 by coordinate-transforming the first destination position information LI1 of the destination position T in the real space S into the screen coordinate system of the display screen 50. The second destination position information LI2 is position information of the destination position T in the screen coordinate system of the display screen 50.

[0077] The display screen 50 is a screen displayed on a display included in the output unit 33 of the operator terminal 30. Coordinates on the display screen 50 are expressed by the screen coordinates of the display. In other words, the display screen 50 is a two-dimensional screen based on a screen coordinate system with the upper left corner of the screen of the display of the operator terminal 30 as the reference point.

[0078] The identification unit 15C uses camera coordinate system information of the camera 25 mounted on the mobile body 20, which is included in the mobile body information received from the mobile body 20, and pre-stored screen coordinate system information representing the screen coordinate system of the display of the operator terminal 30, to calculate a transformation matrix that converts the camera coordinate system represented by the camera coordinate system information into the screen coordinate system represented by the screen coordinate system information. The identification unit 15C may calculate this transformation matrix using a known calculation method.

[0079] The identification unit 15C then performs coordinate transformation of the first destination position information LI1 representing the position of the destination position T in the real space S into the camera coordinate system represented by the camera coordinate system information using a known method, and then transforms the first destination position information LI1 into the screen coordinate system using the calculated transformation matrix. Through these transformation processes, the identification unit 15C performs coordinate transformation of the first destination position information LI1 in the real space S of the destination position T into second destination position information LI2 represented in the screen coordinate system of the display of the operator terminal 30. In other words, through these transformation processes, the identification unit 15C performs coordinate transformation of the first destination position information LI1 in the real space S of the destination position T into second destination position information LI2 on the display screen 50.

[0080] Through these conversion processes, for example, the identification unit 15C identifies second destination position information LI2 by converting the coordinates of first destination position information LI1 of the destination position T in the real space S into a position on the captured image F displayed on the display screen 50, as shown in Figure 3B.

[0081] Then, the identification unit 15C identifies, from among the people P detected by the detection unit 15B and included in the captured video F, the people P who exist within a predetermined range on the display screen 50 from the position represented by the second destination position information LI2 on the display screen 50, as the guide G. The predetermined range may be set in advance as a range on the display screen 50 in which, from among the people P present around the destination position T, the guide G who guides the moving body 20 to the destination position T may exist. Furthermore, the predetermined range may be changeable as appropriate by a user's operation instruction via the input unit 12, etc.

[0082] In this embodiment, the identification unit 15C identifies, among the detected persons P included in the photographed video F, the person P who is present within a predetermined range on the display screen 50 from the position represented by the second destination position information LI2 and who is closest to the position, as the guide G of the moving body 20. Therefore, in the example shown in FIG. 3B , among the detected persons P1 and P2 included in the photographed video F, the person P1 who is closest to the position represented by the second destination position information LI2 of the destination position T on the display screen 50 is identified as the guide G who will guide the moving body 20 equipped with the camera 25 that captured the photographed video F. In other words, the identification unit 15C identifies the guide G1 who is present in a position closer to the destination position T as the guide G who will guide the moving body 20.

[0083] Next, the second specific method will be described.

[0084] 4A and 4B are diagrams illustrating an example of the second identification method.

[0085] 4A is an explanatory diagram of an example of the positional relationship between a destination position T in real space S, a guide G, and the camera 25 of the moving body 20. For example, assume a scene in real space S in which multiple guides G (e.g., guide G1, guide G2) are present near an airplane AP having a destination position T, and the moving body 20 is located near the airplane AP. Also assume a case in which guide G1 of the multiple guides G is the guide G guiding the moving body 20. As described above, the control unit 28 of the moving body 20 transmits moving body information to the information processing device 10, the moving body ID of the moving body 20, the video captured by the camera 25 of the moving body 20, the detection result of the sensor 24 including the position information of the moving body 20, and the camera coordinate system information of the camera 25.

[0086] 4B is a schematic diagram of an example of a captured image F captured by the camera 25 of the moving body 20 shown in FIG. For example, assume a scene in which the detection unit 15B analyzes the captured image F and detects two persons P, a person P1 and a person P2, appearing in the captured image F.

[0087] In the second identification method, the identification unit 15C identifies second guide position information GI2 of the guide G on the display screen 50 based on first guide position information GI1 of the guide G in the real space S. Then, the identification unit 15C identifies, as the guide G, a person P that exists within a predetermined range on the display screen 50 from the position represented by the second guide position information GI2, among the detected people P included in the captured video F.

[0088] In detail, first, the identification unit 15C acquires first guide position information GI1. The first guide position information GI1 is position information in the real space S of a guide G who guides the moving body 20 equipped with the camera 25 that captured the captured video F, as shown in FIG.

[0089] For example, the identification unit 15C receives position information of the guide terminal 40 from the guide terminal 40 carried by the guide G who guides the moving body 20, and acquires the received position information of the guide terminal 40 as first guide position information GI1 of the guide G. 4A and 4B show an example of a scene in which the first guide position information GI1 is acquired from the guide terminal 40 carried by the guide G1 who guides the moving body 20, among a plurality of guides G1 and G2 in the real space S.

[0090] The identification unit 15C then coordinate-transforms the first destination position information LI1 of the guide G1 in the real space S into the screen coordinate system of the display screen 50, thereby identifying second destination position information LI2 of the guide G1 on the display screen 50. The identification unit 15C performs a coordinate transformation process similar to that of the first identification method to transform the first guide position information GI1 into the camera coordinate system and then into the screen coordinate system, thereby coordinate-transforming the first guide position information GI1 in the real space S into second guide position information GI2 in the screen coordinate system. That is, the identification unit 15C coordinate-transforms the first guide position information GI1, which represents the position of the guide G of the moving body 20 in the real space S, into the camera coordinate system represented by the camera coordinate system information by a known method, and then converts it into the screen coordinate system using the transformation matrix calculated in the same manner as in the first identification method. Through these conversion processes, the identification unit 15C converts the coordinates of the first guide position information GI1 of the guide G of the moving body 20 in the real space S into the second guide position information GI2 expressed in the screen coordinate system of the display of the operator terminal 30. That is, through these conversion processes, the identification unit 15C converts the coordinates of the first guide position information GI1 of the guide G of the moving body 20 in the real space S into the second guide position information GI2 on the display screen 50.

[0091] Through these conversion processes, for example, the identification unit 15C identifies second guide position information GI2 by converting the coordinates of the first guide position information GI1 of the guide G1 in the real space S into the position on the captured image F displayed on the display screen 50, as shown in Figure 4B.

[0092] The identification unit 15C then identifies, from among the people P detected by the detection unit 15B and included in the captured video F, the people P who are present within a predetermined range on the display screen 50 from the position represented by the second guide position information GI2 on the display screen 50, as the guide G of the moving object 20. The predetermined range may be set in advance to determine that the person P is present at a position that coincides with the position represented by the second guide position information GI2. The predetermined range may also be changeable as appropriate by the user's operation instruction via the input unit 12, etc.

[0093] In the present embodiment, the identification unit 15C identifies, among the detected persons P included in the photographed video F, the person P who is present within a predetermined range on the display screen 50 from the position represented by the second guide position information GI2 and is closest to the position, as the guide G of the moving body 20. Therefore, in the example shown in Fig. 4B , among the detected persons P1 and P2 included in the photographed video F, the identification unit 15C identifies the person P1 who is closest to the position represented by the second guide position information GI2 on the display screen 50, as the guide G who will guide the moving body 20 on which the camera 25 that captured the photographed video F is mounted.

[0094] Next, a third specific method will be described.

[0095] 5A and 5B are diagrams illustrating an example of the third identification method.

[0096] 5A is an explanatory diagram of an example of the positional relationship between a destination position T, a guide G, the camera 25 of the moving body 20, and a sign M in the real space S. For example, assume a scene in which a plurality of guides G (e.g., guide G1, guide G2) are present near an airplane AP having a destination position T in the real space S, and the moving body 20 is located near the airplane AP. Also assume a scene in which the airplane AP is stopped at a position indicated by a sign M provided on a road surface in the real space S. The sign M is a mark indicating the stopping position of the airplane AP in the real space S, and is shown, for example, on the road surface in the real space S. If the real space S is an airport, the sign M indicating the stopping position is provided, for example, on a parking area or a runway called an apron or ramp.

[0097] As described above, the control unit 28 of the mobile body 20 transmits mobile body information including the mobile body ID of the mobile body 20, the captured image F captured by the camera 25 of the mobile body 20, the detection results of the sensor 24 including the position information of the mobile body 20, and the camera coordinate system information of the camera 25 to the information processing device 10.

[0098] Fig. 5B is a schematic diagram of an example of a captured image F captured by the camera 25 of the moving body 20 shown in Fig. 5A. For example, assume a scene in which the detection unit 15B analyzes the captured image F and detects two persons P, a person P1 and a person P2, appearing in the captured image F.

[0099] In the third identification method, the identification unit 15C detects a sign M that indicates a stopping position of a movable object, including the destination position T of the moving body 20, included in the captured video F. As described above, in this embodiment, a case in which the object is an airplane AP will be described as an example. The identification unit 15C then identifies second destination position information LI2 on the display screen 50 for the destination position T based on the sign M and the identification information of the airplane AP. The identification unit 15C then identifies, among the detected people P included in the captured video F, a person P that is present within a predetermined range on the display screen 50 from the position indicated by the second destination position information LI2 as a guide G.

[0100] In detail, first, the identification unit 15C detects a sign M included in the photographed video F that indicates a stopping position of the airplane AP, which includes the destination position T of the moving object 20. The identification unit 15C analyzes the photographed video F using a known method that uses AI or the like learned by deep learning or the like, thereby detecting the sign M and identification information of the airplane AP included in the photographed video F. In this embodiment, an example will be described in which model information of the airplane AP is used as the identification information of the airplane AP. The identification unit 15C may detect the model information of the airplane AP by acquiring airplane information including the model information from the airplane AP.

[0101] The identification unit 15C associates, for each aircraft type identified by the aircraft model information of the airplane AP, relative position information that indicates the relative position of the destination position T of the airplane AP with the sign M when the airplane AP of that aircraft model stops at the stopping position indicated by the sign M in advance and stores the associated information in the storage unit 14. Then, using the position of the sign M shown in the photographed video F and the relative position information associated with the aircraft model information of the airplane AP, the identification unit 15C identifies, on the display screen 50 displaying the photographed video F, the relative position indicated by the relative position information with respect to the position of the sign M included in the photographed video F, as second destination position information LI2 of the destination position T on the display screen 50.

[0102] Then, the identification unit 15C identifies, among the people P detected by the detection unit 15B and included in the captured video F, the people P who exist within a predetermined range on the display screen 50 from the position represented by the second destination position information LI2 on the display screen 50, as the guide G of the moving body 20. The predetermined range may be set in advance as a range on the display screen 50 within which, among the people P present around the destination position T, the guide G who guides the moving body 20 to the destination position T may exist. Furthermore, the predetermined range may be changeable as appropriate by a user's operation instruction via the input unit 12, etc.

[0103] In the present embodiment, the identification unit 15C identifies, among the detected persons P included in the photographed video F, the person P who is present within a predetermined range on the display screen 50 from the position represented by the second destination position information LI2 and who is closest to the position, as the guide G of the moving body 20. Therefore, in the example shown in FIG. 5B , among the detected persons P1 and P2 included in the photographed video F, the person P1 who is closest to the position represented by the second destination position information LI2 of the destination position T on the display screen 50 is identified as the guide G who will guide the moving body 20 equipped with the camera 25 that captured the photographed video F. In other words, the identification unit 15C identifies the guide G1 who is present in a position closer to the destination position T derived from the sign M as the guide G who will guide the moving body 20.

[0104] Returning to FIG. 2, the description will continue.

[0105] The output control unit 15D outputs a display screen 50 highlighting the identified guide G included in the captured video F to the operator terminal 30 of the operator OP who remotely controls the moving body 20.

[0106] 6A to 6E are schematic diagrams of an example of a display screen 50 that highlights a guide G. 6A to 6E show an example in which a person P1, among multiple people P appearing in a photographed video F, is identified as the guide G who guides the moving object 20 equipped with the camera 25 that captured the photographed video F.

[0107] FIG. 6A is a schematic diagram of an example of a display screen 50A. The display screen 50A is an example of the display screen 50. FIG. 6A shows an example of a configuration in which the output control unit 15D highlights an identified guide G appearing in the photographed video F by superimposing a frame image 52 surrounding the identified guide G on the photographed video F. For example, the output control unit 15D generates a display screen 50A in which a frame image 52 surrounding an area in which a person P identified as the guide G appears among multiple detected people P included in the photographed video F is superimposed on the photographed video F. The output control unit 15D may also display, on the display screen 50A, the mobile object ID of the mobile object 20 equipped with the camera 25 that captured the photographed video F as information representing the mobile object 20 to be remotely controlled in accordance with the guidance of the guide G.

[0108] Then, the output control unit 15D outputs the display screen 50A to the operator terminal 30. The output control unit 35B of the operator terminal 30 receives the display screen 50A and displays the display screen 50A on the display of the output unit 33 of the operator terminal 30. Therefore, the operator OP can easily recognize the guide G highlighted by the frame image 52, among the multiple people P included in the display screen 50A, as the guide G of the mobile object 20 identified by the mobile object ID displayed on the display screen 50A.

[0109] FIG. 6B is a schematic diagram of an example of a display screen 50B. The display screen 50B is an example of the display screen 50. FIG. 6B shows an example of a configuration in which the output control unit 15D superimposes a frame image 52 surrounding an identified guide G appearing in the photographed video F on the photographed video F, and also masks people P other than the guide G to increase the transparency of the people P, thereby highlighting the guide G. For example, the output control unit 15D superimposes a frame image 52 surrounding an area in which a person P identified as a guide G appears among multiple detected people P included in the photographed video F on the photographed video F. Furthermore, the output control unit 15D generates a display screen 50B in which, among the detected people P included in the photographed video F, people P other than the person P identified as the guide G are masked to increase the transparency of the people P. The output control unit 15D may generate a display screen 50B in which the frame image 52 is not displayed, and the persons P other than the person P identified as the guide G are masked to increase the transparency of the persons P. The output control unit 15D may further display, on the display screen 50B, the mobile object ID of the mobile object 20 equipped with the camera 25 that captured the captured video F, as information representing the mobile object 20 to be remotely controlled in accordance with the guidance of the guide G.

[0110] Then, the output control unit 15D outputs the display screen 50B to the operator terminal 30. The output control unit 35B of the operator terminal 30 receives the display screen 50B and displays the display screen 50B on the display of the output unit 33 of the operator terminal 30. Therefore, the operator OP can easily recognize the guide G, who is highlighted by the frame image 52 or masking of other people P, among the multiple people P included in the display screen 50B, as the guide G of the mobile object 20 identified by the mobile object ID displayed on the display screen 50B.

[0111] FIG. 6C is a schematic diagram of an example of a display screen 50C. The display screen 50C is an example of the display screen 50. FIG. 6C illustrates an example in which the output control unit 15D highlights the guide G by superimposing an arrow image 54 indicating the direction in which the guide G is located on the captured image F. For example, the output control unit 15D superimposes a frame image 52 surrounding an area in which a person P identified as the guide G appears among multiple detected people P included in the captured image F on the captured image F. The output control unit 15D also masks the detected people P included in the captured image F other than the person P identified as the guide G, thereby generating a display screen 50C in which the transparency of the person P is increased. Furthermore, when at least a portion of the area of ​​the guide G appearing in the captured image F is outside the display screen 50C, the output control unit 15D generates a display screen 50C in which an arrow image 54 indicating the direction in which the guide G is located is superimposed on the captured image F. The case where at least a portion of the area of ​​the guide G is outside the screen of the display screen 50C includes both a case where a portion of the area of ​​the guide G is outside the screen of the display screen 50C and a case where the entire area of ​​the guide G is outside the screen of the display screen 50C. Fig. 6C shows an example in which a portion of the area of ​​the guide G is outside the screen of the display screen 50C. However, as described above, the entire area of ​​the guide G may be outside the screen of the display screen 50C.

[0112] The output control unit 15D then outputs the display screen 50C to the operator terminal 30. The output control unit 35B of the operator terminal 30 receives the display screen 50C and displays the display screen 50C on the display of the output unit 33 of the operator terminal 30. This allows the operator OP to easily recognize the guide G, who is highlighted by the frame image 52, transparency adjustment, arrow image 54, etc., among the multiple people P included in the display screen 50B, as the guide G of the mobile unit 20 identified by the mobile unit ID displayed on the display screen 50C. Furthermore, when at least a portion of the guide G is outside the frame of the display screen 50C, the operator OP can remotely operate the mobile unit 20 in accordance with the arrow image 54, thereby remotely operating the mobile unit 20 so that the guide G is displayed more centrally on the display screen 50C.

[0113] Fig. 6D is a schematic diagram of an example of a display screen 50D. The display screen 50D is an example of the display screen 50. Fig. 6D shows an example of a display screen 50D including a front screen 60F and an enlarged screen 60B. The front screen 60F is a captured image F captured by a front camera of the cameras 25 that captures an image in front of the moving object 20. The enlarged screen 60B is a screen in which a portion of the captured image F is enlarged.

[0114] For example, the output control unit 15D generates a front screen 60F in which a frame image 52 surrounding a specified guide G who appears in a captured image F taken by the forward camera is superimposed on the captured image F. The output control unit 15D also generates an enlarged screen 60B in which an area including the guide G in the captured image F is enlarged. The output control unit 15D may also display, on the display screen 50D, the mobile object ID of the mobile object 20 equipped with the camera 25 that captured the captured image F, as information representing the mobile object 20 to be remotely controlled in accordance with the guidance of the guide G.

[0115] In this way, the output control unit 15D may generate a display screen 50D that further includes an enlarged screen 60B that enlarges the area in the captured image F in which the identified guide G appears.

[0116] The output control unit 15D then outputs the display screen 50D to the operator terminal 30. The output control unit 35B of the operator terminal 30, which has received the display screen 50D, displays the display screen 50D on the display of the output unit 33 of the operator terminal 30. Therefore, the operator OP can easily identify the guide G appearing in the image F captured by the forward camera by visually recognizing the guide G highlighted by the frame image 52 on the front screen 60F included in the display screen 50D. Furthermore, the operator OP can easily identify the guide G and check the actions of the guide G by checking the enlarged screen 60B included in the display screen 50D.

[0117] FIG. 6E is a schematic diagram of an example of a display screen 50E. The display screen 50E is an example of the display screen 50. The display screen 50E includes screens (left side screen 60L, front screen 60F, right side screen 60R) of the captured images F (captured images FL, FF, FR) captured by the front camera capturing the area in front of the moving body 20, the left camera capturing the area to the left, and the right camera capturing the area to the right of the moving body 20, respectively, among the cameras 25. The display screen 50E also includes an enlarged screen 60B. FIG. 6E illustrates an example in which the output control unit 15D generates the display screen 50E by highlighting the person P identified as the guide G with a frame image 52 and including an enlarged screen 60B enlarging the area in which the guide G is captured. In this way, the output control unit 15D may generate the display screen 50E further including an enlarged screen 60B enlarging the area in which the identified guide G is captured in the captured images F. In addition, the output control unit 15D may further display on the display screen 50E the mobile body ID of the mobile body 20 equipped with the camera 25 that captured the captured image F as information representing the mobile body 20 to be remotely controlled in accordance with the guidance of the guide G.

[0118] The output control unit 15D then outputs the display screen 50E to the operator terminal 30. The output control unit 35B of the operator terminal 30, which has received the display screen 50E, displays the display screen 50E on the display of the output unit 33 of the operator terminal 30. Therefore, the operator OP can easily identify the guide G appearing in the image F captured by the forward camera by visually recognizing the guide G highlighted by the frame image 52 on the front screen 60F included in the display screen 50E. Furthermore, the operator OP can easily identify the guide G and check the actions of the guide G by checking the enlarged screen 60B included in the display screen 50E.

[0119] The operator OP remotely controls the moving object 20 by operating the input unit 32 in accordance with guidance content represented by the highlighted actions of the guide G included in the display screen 50 displayed on the display of the output unit 33. Specifically, the operator OP operates the input unit 32 in accordance with the highlighted actions of the guide G included in the display screen 50. The remote control control unit 35C of the operator terminal 30 transmits remote operation information corresponding to operation instructions for the moving object 20 input in response to the operator OP's operation of the input unit 32 to the information processing device 10, together with the moving object ID of the moving object 20. The information processing device 10 transmits the remote operation information to the moving object 20 identified by the moving object ID. The moving object 20 controls the drive unit 26 based on the remote operation information to move and drive the moving object 20 in accordance with the remote control by the operator OP. Therefore, when the operator OP operating the operator terminal 30 operates the input unit 32 in accordance with the guidance of the guide G highlighted on the display screen 50, the mobile body 20 is remotely controlled in accordance with the guidance of the guide G, moves closer to the target position T, and various driving processes for the mobile body 20 to the target position T are performed.

[0120] In the above-described identification method, the identification unit 15C identifies, among the detected persons P included in the captured image F, the person P that is within a predetermined range on the display screen 50 from the position represented by the second destination position information LI2 or the second guide position information GI2 and is closest to that position, as the guide G of the moving body 20.

[0121] However, the identification unit 15C may identify, among the detected persons P included in the captured video F, the person P who is closest to the moving body 20 as the guide G of the moving body 20. For example, the identification unit 15C identifies the planned movement route of the moving body 20 from the moving body information of the moving body 20, etc. Then, the identification unit 15C may identify, as the guide G of the moving body 20, the person P who is closest to the moving body 20 among the persons P who are on the identified planned movement route or downstream in the movement direction of the moving body 20 on the planned movement route.

[0122] Furthermore, when the next guide G to be followed is captured in the captured image F, the output control unit 15D may output to the operator terminal 30 a display screen 50 in which information indicating that the guide G is the next guide G to be followed is superimposed on the captured image F. Furthermore, when the position in real space of the next guide G to be followed is identified, the output control unit 15D may further output to the operator terminal 30 information indicating the distance the moving body 20 has to travel before the next guide G to be followed is present.

[0123] Furthermore, when the identification unit 15C receives remote control request information requesting remote control from a certain moving body 20, it may identify the moving body 20 and the guide G previously associated with the moving body 20. In this case, the identification unit 15C identifies the second guide position information GI2 of the identified guide G on the display screen 50 in the same manner as described above. The output control unit 15D may output the display screen 50 highlighting the identified guide G included in the captured video F to the operator terminal 30 of the operator OP who remotely controls the moving body 20 in the same manner as described above.

[0124] Furthermore, the identification unit 15C may receive, from the guide terminal 40 of the guide G, moving body identification information of the moving body 20 that the guide G has designated as the moving body 20 to be guided. When the identification unit 15C receives remote control request information including the moving body identification information from the guide terminal 40, the identification unit 15C may identify the guide G who designated the moving body 20 as the guide G for the moving body 20. Then, the identification unit 15C may identify second guide position information GI2 on the display screen 50 of the identified guide G in the same manner as described above. The output control unit 15D may output the display screen 50 highlighting the identified guide G included in the captured video F to the operator terminal 30 of the operator OP who remotely controls the moving body 20 in the same manner as described above.

[0125] Furthermore, when the guide G wishes to end guidance, the guide G may input the end of guidance by operating the guide terminal 40. The guide terminal 40, which has received the input to end guidance, notifies the information processing device 10 of guidance end information, which indicates the end of guidance. When the output control unit 15D of the information processing device 10 receives the guidance end information, it may notify the operator OP of the end of guidance by outputting the guidance end information to the operator terminal 30 of the operator OP who remotely controls the mobile body 20 guided by the guide G.

[0126] There may also be cases where the guide G is replaced. In this case, the guide G operates the guide terminal 40 to input the change of guide G. The guide terminal 40, which has received the input of the change of guide G, notifies the information processing device 10 of guide change information indicating the change of guide G. When the output control unit 15D of the information processing device 10 receives the guide change information, it outputs the guide change information to the operator terminal 30 of the operator OP who remotely controls the mobile body 20 guided by the guide G, thereby notifying the operator OP of the change of guide G.

[0127] The identification unit 15C may re-identify the new guide G after the replacement in the same manner as described above. Alternatively, the previous guide G may designate the new guide G, and the guide terminal 40 may transmit the guide identification information of the designated guide G to the information processing device 10, thereby identifying the guide G identified by the received guide identification information as the new guide G. Alternatively, the new guide G may designate the moving body 20 to be guided by the new guide G, and the guide terminal 40 may transmit the guide identification information of the new guide G and the moving body identification information of the moving body 20 to the information processing device 10. The identification unit 15C may identify the guide G identified by the received guide identification information as the guide G of the moving body 20 identified by the received moving body identification information.

[0128] Furthermore, the guide G may operate the guide terminal 40 to input mobile object abnormality information indicating that a mobile object 20 not intended by the guide G is being targeted for guidance. The guide terminal 40 transmits the received mobile object abnormality information to the information processing device 10. Upon receiving the mobile object abnormality information, the output control unit 15D of the information processing device 10 may notify the operator OP of the information indicating that a mobile object 20 not intended by the guide G is being targeted for guidance by outputting the information to the operator terminal 30 of the operator OP. Furthermore, in response to an operation instruction from the guide G on the guide terminal 40, the identification unit 15C of the information processing device 10 may identify the original guide G for the mobile object 20 as the guide G.

[0129] Furthermore, the identification unit 15C may identify a person P wearing a light-emitting item among the people P captured in the captured video F as the guide G. The item may be any item that includes a part that becomes light-emitting when light is emitted and becomes unlit when the light is turned off. The item may be, but is not limited to, clothing worn on the body of the person P, a hat such as a helmet worn on the head of the person P, etc.

[0130] Furthermore, the information processing device 10 and the item may be communicatively connected. Then, the output control unit 15D may transmit a light-emitting instruction signal to the item worn by the identified guide G. The item that receives the light-emitting instruction signal is switched from an off state to an emitting state, for example. In this case, the item worn by the person P identified as the guide G can be made to emit light.

[0131] Furthermore, the information processing device 10 may transmit light-emitting instruction information for the item to the guide terminal 40 of the guide G. The control unit 46 of the guide terminal 40 outputs the received light-emitting instruction information to the output unit 43. When the guide G confirms the light-emitting instruction information output to the output unit 43, the guide G may switch the item worn by the guide G to a light-emitting state, thereby causing the worn item to emit light.

[0132] In addition, when the output control unit 15D displays a display screen 50 highlighting the guide G on the operator terminal 30 and the operator OP starts remote control of the mobile body 20 by operating the input unit 32 in accordance with the actions of the highlighted guide G, the following processing may be further performed.

[0133] For example, the output control unit 15D may cause the operator terminal 30 to display a display screen 50 that further includes information prompting the operator OP to perform remote control in accordance with the instructions of the guide G. Furthermore, the remote control control unit 35C of the operator terminal 30 may receive an input from the operator OP indicating that the operator OP approves the display.

[0134] The output control unit 15D of the information processing device 10 may identify guidance content by the identified guide G by analyzing the gestures of the identified guide G included in the captured video F, and determine whether the operator OP is performing remote operation in accordance with the guidance content. The determination of whether the operator OP is performing remote operation in accordance with the guidance content may be performed by at least one of the operation of the input unit 32 by the operator OP and detection of the line of sight of the operator OP. The detection of the line of sight of the operator OP may be performed by configuring the operator terminal 30 to include an imaging device and analyzing the captured video in which the operator OP is captured by the imaging device using a known image analysis method.

[0135] When the output control unit 15D determines that the operator OP is remotely operating the vehicle in accordance with the guidance instructions of the guide G, the output control unit 15D may transmit a message indicating that the remote operation is being performed in accordance with the guidance instructions of the guide G to the guide terminal 40 of the guide G. By checking the message, the guide G can easily confirm that the mobile object 20 is being correctly remotely controlled by the guide G, and can guide the mobile object 20 with peace of mind.

[0136] Furthermore, when the output control unit 15D determines that the operator OP is remotely operating the vehicle in accordance with the guidance provided by the guide G, it may also send a message indicating that the remote operation is being performed in accordance with the guidance provided by the guide G to the guide terminal 40 of a person P other than the guide G (for example, the guide terminal 40 that is the terminal of a person P present in the vicinity of the mobile body 20). By checking this message, not only the guide G but also people P present in the vicinity of the mobile body 20 other than the guide G can easily understand that the mobile body 20 is being correctly remotely controlled by guidance from the guide G.

[0137] Furthermore, when the guide terminal 40 of the guide G receives approval of the message through the guide G's operation of the input unit 42 of the guide terminal 40, the guide terminal 40 may transmit approval information indicating the approval to the information processing device 10. The approval of the message may be received by the guide G through any of a touch operation of the input unit 42, a voice input, and a gesture input on the input unit 42. The output control unit 15D of the information processing device 10 may output the approval information to the output unit 33 of the operator terminal 30 of the operator OP. In this case, the output control unit 15D can notify the operator OP that the guide G recognizes that the mobile object 20 is being remotely controlled in accordance with his / her guidance. This allows the operator OP to perform remote control with peace of mind.

[0138] Furthermore, when the output control unit 15D determines that the operator OP is performing remote operation in accordance with the guidance content by the guide G, the output control unit 15D may transmit a message indicating that the remote operation is being performed in accordance with the guidance content by the guide G to the mobile body 20 being guided by the guide G. Furthermore, the message may further include guide identification information for identifying the guide G.

[0139] The control unit 28 of the mobile object 20 outputs the received message to the output unit 23. For example, the control unit 28 of the mobile object 20 outputs to the output unit 23, by voice or image, information indicating that remote operation is being performed in accordance with the guidance content by the guide G, as indicated in the message, and the guide identification information of the guide G.

[0140] In this case, the guide G and the person P other than the guide G who are present around the moving body 20 can easily understand which guide G is guiding the moving body 20 and that the moving body 20 is operating in accordance with the guidance of the guide G.

[0141] Furthermore, the voice of the guide G, such as speech, may be collected by a microphone included in the input unit 42 of the guide terminal 40 of the guide G, and transmitted from the guide terminal 40 to the information processing device 10. The output control unit 15D of the information processing device 10 may output the voice of the guide G received from the guide terminal 40 to the output unit 33 of the operator terminal 30. In this case, the operator OP can remotely control the moving object 20 in accordance with the guidance given by the voice instructions of the guide G in addition to the gestures of the guide G.

[0142] Furthermore, the voice of the operator OP, such as speech, may be collected by a microphone included in the input unit 32 of the operator terminal 30 of the operator OP, and transmitted from the operator terminal 30 to the information processing device 10. The output control unit 15D of the information processing device 10 may output the voice of the operator OP received from the operator terminal 30 to the output unit 43 of the guide terminal 40. In this case, it becomes possible to transmit the voice of the operator OP to the guide G, and it becomes possible for the guide G and the operator OP to communicate with each other while providing guidance.

[0143] On the other hand, when the output control unit 15D determines that the operator OP has not performed remote operation in accordance with the guidance content by the guide G, the output control unit 15D may further output an error message to the display screen 50. Furthermore, when the output control unit 15D determines that the operator OP has not performed remote operation in accordance with the guidance content by the guide G during at least a part of the period of remote control by the operator OP in accordance with the guidance of the guide G, the output control unit 15D may further output an error message to the display screen 50.

[0144] 7A and 7B are schematic diagrams of an example of a display screen 50 on which an error message is displayed.

[0145] 7A is a schematic diagram of an example of a display screen 50E. The display screen 50E is an example of the display screen 50. For example, the output control unit 15D identifies the direction to be guided by the guide G by analyzing the gestures of the guide G. If the output control unit 15D determines that the moving direction of the mobile object 20 represented by the transition of the position information included in the mobile object information received from the mobile object 20 differs from the direction to be guided by the guide G, the output control unit 15D displays a display screen 50E on the display of the output unit 33 of the operator terminal 30, in which an error message 56 indicating a guidance error is superimposed on the captured image F.

[0146] Therefore, the output control unit 15D can provide the operator OP with a message indicating that the moving body 20 is being remotely operated in a direction different from the direction guided by the guide G.

[0147] 7B is a schematic diagram of an example of a display screen 50F. The display screen 50F is an example of the display screen 50. For example, the output control unit 15D analyzes the gestures of the guide G to identify the guidance content represented by the gestures of the guide G. Then, the output control unit 15D analyzes the detection results of the sensor 24 included in the mobile object information received from the mobile object 20 to determine whether the drive represented by the identified guidance content is being executed. Then, if the output control unit 15D determines that the mobile object 20 is being driven in a manner different from the guidance content, the output control unit 15D displays a display screen 50F on the display of the output unit 33 of the operator terminal 30, in which an error message 56 indicating an operation error is superimposed on the captured image F.

[0148] Therefore, the output control unit 15D can provide the operator OP with a message indicating that the mobile object 20 is being remotely operated in a manner different from the guidance content given by the guide G.

[0149] Furthermore, when it is determined that the operator OP is not performing remote operation in accordance with the guidance content by the guide G, the information processing device 10 may perform control not to transmit to the mobile body 20 remote operation information corresponding to the operation instruction for the mobile body 20 input in response to the operation of the input unit 32 by the operator OP, which is received from the operator terminal 30. That is, in this case, the information processing device 10 may perform control not to transmit to the mobile body 20 the remote operation information for remote control input in response to the operation of the operator OP. In addition, in this case, the operator terminal 30 may perform control not to transmit to the information processing device 10 the remote operation information for remote control input in response to the operation of the operator OP.

[0150] Furthermore, when it is determined that the operator OP is not performing remote operation in accordance with the guidance content by the guide G, the information processing device 10 may transmit to the operator terminal 30 inquiry information indicating that the remote operation is different from the guidance content by the guide G and inquiring whether or not to continue the operation, before transmitting to the mobile body 20 remote operation information corresponding to the operation instruction for the mobile body 20 input in response to the operation of the input unit 32 by the operator OP, which information processing device 10 received from the operator terminal 30. Then, when the information processing device 10 receives an instruction to continue the operation by the operation instruction of the input unit 32 by the operator OP, line of sight detection, or the like, it may transmit the remote operation information received from the operator terminal 30 to the mobile body 20.

[0151] Furthermore, the output control unit 15D of the information processing device 10 may analyze the gestures of the identified guide G included in the captured video F to identify guidance content by the guide G, and output a display screen 50 including the identified guidance content to the output unit 33 of the operator terminal 30. The identified guidance content may be, for example, but is not limited to, stop, run, turn right, turn left, or a combination of two or more of these. A known method may be used to identify guidance content through gesture analysis. The period during which the guidance content is output to the output unit 33 may be a period that includes at least a period during which the operator OP is remotely controlling the mobile object 20. The period during which the guidance content is output to the output unit 33 may include at least one of a period before and a period after it is determined that the operator OP is not remotely operating the mobile object 20 in accordance with the guidance content by the guide G.

[0152] Next, an example of the flow of information processing executed by the information processing system 1 of this embodiment will be described.

[0153] FIG. 8 is a sequence diagram showing an example of the flow of information processing executed by the information processing system 1 of this embodiment.

[0154] The communication control unit 28A of the mobile object 20 transmits mobile object information including the captured image F captured by the camera 25 mounted on the mobile object 20 to the information processing device 10 (step S100).

[0155] The detection unit 15B of the information processing device 10 detects a person P included in the photographed image F of the periphery of the moving object 20 received from the moving object 20 (step S102). The identification unit 15C of the information processing device 10 identifies a guide G guiding the moving object 20 from among the people P detected in step S102 and included in the photographed image F (step S104). The output control unit 15D of the information processing device 10 generates a display screen 50 that highlights the guide G included in the photographed image F and identified in step S104 (step S106), and transmits the display screen 50 to the operator terminal 30 (step S108).

[0156] The output control unit 35B of the operator terminal 30 displays the display screen 50 received from the information processing device 10 on the display of the output unit 33 (step S110). Therefore, by visually checking the display screen 50, the operator OP can easily confirm that the highlighted person P included in the display screen 50 is the guide G of the moving object 20.

[0157] The operator OP remotely controls the moving object 20 by operating the input unit 32 in accordance with the highlighted actions of the guide G included in the display screen 50 displayed on the display of the output unit 33 of the operator terminal 30. Specifically, the operator OP operates the input unit 32 in accordance with the highlighted actions of the guide G included in the display screen 50. The remote control control unit 35C of the operator terminal 30 transmits remote operation information corresponding to operation instructions for the moving object 20 input in response to the operator OP's operation of the input unit 32 to the information processing device 10 together with the moving object ID of the moving object 20 (step S112). The information processing device 10 transmits the remote operation information to the moving object 20 identified by the moving object ID (step S114). The moving object 20 controls the drive unit 26 based on the remote operation information to move and drive the moving object 20 in accordance with the remote control by the operator OP. Therefore, the operator OP operating the operator terminal 30 operates the input unit 32 in accordance with the guidance of the guide G highlighted on the display screen 50, whereby the mobile body 20 is remotely controlled in accordance with the guidance of the guide G, and moves closer to the destination position T, and various drive processes for the mobile body 20 to the destination position T are executed, etc. Then, this sequence ends.

[0158] Next, an example of the flow of information processing executed by the information processing device 10 will be described.

[0159] FIG. 9 is a flowchart showing an example of the flow of information processing executed by the information processing device 10 of this embodiment.

[0160] The acquisition unit 15A acquires, from the moving object 20, moving object information including a captured image F of the periphery of the moving object 20 (step S200).

[0161] The detection unit 15B detects the person P included in the photographed video F acquired in step S200 (step S202).

[0162] The identification unit 15C identifies the guide G who guides the moving object 20 from among the people P detected in step S202 and included in the captured video F (step S204).

[0163] The output control unit 15D generates a display screen 50 that highlights the guide G identified in step S204 and included in the captured image F (step S206), and outputs the display screen 50 to the operator terminal 30 of the operator OP who remotely controls the moving object 20 (step S208). Then, this routine ends.

[0164] As described above, the information processing device 10 of this embodiment includes a detection unit 15B, an identification unit 15C, and an output control unit 15D. The detection unit 15B detects a person P included in the photographed video F around the moving object 20. The identification unit 15C identifies a guide G who guides the moving object 20 from among the detected people P included in the photographed video F. The output control unit 15D outputs a display screen 50 highlighting the identified guide G included in the photographed video F to the operator terminal 30 of the operator OP who remotely controls the moving object 20.

[0165] Here, there are cases where the operator OP remotely controls the moving body 20 in accordance with guidance from a guide G present in the real space S. In such cases, the operator OP performs remote control in accordance with the actions of the guide G included in the photographed image F of the surroundings of the moving body 20. However, with the conventional technology, it may be difficult for the operator OP to immediately recognize which person P captured in the photographed image F is the guide G. For example, in a situation where multiple people P are captured in the photographed image F or where the guide G is captured in a small size in the photographed image F, it may be difficult for the operator OP to identify which person P is the guide G of the moving body 20. In other words, with the conventional technology, there are cases where suitable remote control support is not provided.

[0166] On the other hand, in the information processing device 10 of this embodiment, the identification unit 15C identifies a guide G who guides the moving body 20 from among the detected people P included in the photographed video F. The output control unit 15D outputs a display screen 50 that highlights the identified guide G included in the photographed video F to the operator terminal 30 of the operator OP who remotely controls the moving body 20. Therefore, the information processing device 10 of this embodiment can output the display screen 50 that highlights the guide G who guides the moving body 20 to the operator terminal 30 of the operator OP who remotely controls the moving body 20. Therefore, the operator OP can easily identify the highlighted guide G included in the display screen 50, and can easily confirm which person P is the guide G of the moving body 20.

[0167] Therefore, the information processing device 10 of this embodiment can provide suitable remote control support.

[0168] Next, an example of the hardware configuration of the information processing device 10, the moving body 20, the operator terminal 30, and the guide terminal 40 of the above embodiment will be described.

[0169] FIG. 10 is a diagram illustrating an example of the hardware configuration of the information processing device 10, the mobile object 20, the operator terminal 30, and the guide terminal 40 according to the above embodiment.

[0170] The information processing device 10, the mobile body 20, the operator terminal 30, and the guide terminal 40 of the above embodiment have a hardware configuration that utilizes a normal computer, in which a CPU (Central Processing Unit) 80, a ROM (Read Only Memory) 82, a RAM (Random Access Memory) 84, an I / F 86, etc. are interconnected by a bus 88.

[0171] The CPU 80 is a computing device that controls the information processing device 10, the mobile object 20, the operator terminal 30, and the guide terminal 40 of the above embodiment. The ROM 82 stores programs and the like that realize information processing by the CPU 80. The RAM 84 stores data necessary for various processes by the CPU 80. The I / F 86 is an interface that is connected to the storage unit, input unit, output unit, sensor, communication unit, etc., and is used to send and receive data.

[0172] In the information processing device 10, the mobile body 20, the operator terminal 30, and the guide terminal 40 of the above embodiments, the CPU 80 reads a program from the ROM 82 onto the RAM 84 and executes it, thereby realizing each of the above functional units on the computer.

[0173] The programs for executing the above processes executed by the information processing device 10, the mobile body 20, the operator terminal 30, and the guide terminal 40 of the above embodiment may be stored in a hard disk drive (HDD). Also, the programs for executing the above processes executed by the information processing device 10, the mobile body 20, the operator terminal 30, and the guide terminal 40 of the above embodiment may be provided by being pre-installed in the ROM 82.

[0174] The programs for executing the above processes executed by the information processing device 10, the mobile body 20, the operator terminal 30, and the guide terminal 40 of the above embodiments may be stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD) and provided as a computer program product. The programs for executing the above information processes executed by the information processing device 10, the mobile body 20, the operator terminal 30, and the guide terminal 40 of the above embodiments may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. The programs for executing the above information processes executed by the information processing device 10, the mobile body 20, the operator terminal 30, and the guide terminal 40 of the above embodiments may be provided or distributed via a network such as the Internet.

[0175] Although the above describes an embodiment, the embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0176] Note that the present technology can also be configured as follows. (1) An information processing method executed by an information processing device, including: a detection step of detecting a person included in a captured video around a moving object; an identification step of identifying a guide who guides the moving object from among the detected people included in the captured video; and an output control step of outputting a display screen highlighting the identified guide included in the captured video to an operator terminal of an operator who remotely controls the moving object. (2) The identification step includes: identifying second destination position information on the display screen of the destination position of the moving object based on first destination position information in real space of the destination position of the moving object; and identifying, as the guide, a person who is present on the display screen within a predetermined range from a position represented by the second destination position information from among the detected people included in the captured video. (3) The information processing method according to (2), wherein the identifying step identifies, as the guide, the person who is present within a predetermined range on the display screen from the position represented by the second destination position information and is closest to the position, among the detected people included in the captured video. (4) The information processing method according to (2) or (3), wherein the first destination position information is acquired by receiving from a detection device that detects position information of the destination position in real space, or by reading position information representing the destination position in the real space of autonomous traveling of the mobile body. (5) The information processing method according to (1), wherein the identifying step identifies second guide position information of the guide on the display screen based on first guide position information of the guide in real space, and identifies, as the guide, the person who is present within a predetermined range on the display screen from the position represented by the second guide position information, among the detected people included in the captured video. (6) The information processing method described in (5), wherein the identification step identifies, among the detected persons included in the captured video, the person who is within a predetermined range on the display screen from the position represented by the second guide position information and is closest to the position, as the guide.(7) The information processing method according to (1), wherein the specifying step detects signs included in the captured video that indicate stopping positions of movable objects including the destination position of the moving body, specifies second destination position information on the display screen of the destination position based on identification information of the signs and the objects, and specifies, as the guide, a person who is present within a predetermined range on the display screen from the position represented by the second destination position information among the detected people included in the captured video. (8) The information processing method according to (7), wherein the specifying step specifies, as the guide, a person who is present within a predetermined range on the display screen from the position represented by the second destination position information and is closest to the position among the detected people included in the captured video. (9) An information processing program for causing a computer to execute the following steps: a detection step of detecting a person included in a captured image of the periphery of a moving object, an identification step of identifying a guide who guides the moving object from among the detected people included in the captured image, and an output control step of outputting a display screen highlighting the identified guide who is included in the captured image to an operator terminal of an operator who remotely controls the moving object. (10) An information processing device comprising: an output control unit that receives a display screen highlighting the guide who guides the moving object from among people included in a captured image of the periphery of the moving object, and outputs the received display screen to an output unit.

[0177] REFERENCE SIGNS LIST 1 Information processing system 10 Information processing device 15B Detection unit 15C Identification unit 15D Output control unit 20 Mobile body 30 Operator terminal 35A Communication control unit 35B Output control unit 40 Guide terminal

Claims

1. An information processing method executed by an information processing device, comprising: a detection step of detecting a person included in a captured image of the periphery of a moving object; an identification step of identifying a guide who guides the moving object from among the detected people included in the captured image; and an output control step of outputting a display screen highlighting the identified guide included in the captured image to an operator terminal of an operator who remotely controls the moving object.

2. The information processing method of claim 1, wherein the identification step identifies second destination position information on the display screen of the destination position based on first destination position information in real space of the destination position of the moving body, and identifies, among the detected people included in the captured video, a person who is present within a predetermined range on the display screen from the position represented by the second destination position information as the guide.

3. The information processing method of claim 2, wherein the identification step identifies, among the detected persons included in the captured video, the person that is within a predetermined range on the display screen from the position represented by the second destination position information and is closest to said position, as the guide.

4. The information processing method according to claim 2, wherein the first destination position information is obtained by receiving from a detection device that detects position information of the destination position in real space, or by reading position information representing the destination position in real space of the autonomous driving of the mobile body.

5. The information processing method of claim 1, wherein the identification step identifies second guide position information of the guide on the display screen based on first guide position information of the guide in real space, and identifies, among the detected people included in the captured video, a person that exists within a predetermined range on the display screen from the position represented by the second guide position information as the guide.

6. The information processing method described in claim 5, wherein the identification step identifies, among the detected persons included in the captured video, the person who is within a predetermined range on the display screen from the position represented by the second guide position information and is closest to said position, as the guide.

7. The information processing method of claim 1, wherein the identification step detects a sign included in the captured image indicating a stopping position of a movable object including the destination position of the moving body, identifies second destination position information on the display screen of the destination position based on identification information of the sign and the object, and identifies, among the detected people included in the captured image, a person who is present within a predetermined range on the display screen from the position indicated by the second destination position information as the guide.

8. The information processing method described in claim 7, wherein the identification step identifies, among the detected persons included in the captured video, the person who is within a predetermined range on the display screen from the position represented by the second destination position information and is closest to said position, as the guide.

9. An information processing program for causing a computer to execute the following steps: a detection step for detecting a person included in a photographed image around a moving object; an identification step for identifying a guide who guides the moving object from among the detected people included in the photographed image; and an output control step for outputting a display screen highlighting the identified guide included in the photographed image to an operator terminal of an operator who remotely controls the moving object.

10. An information processing device comprising: an output control unit that receives a display screen that highlights a guide who guides a moving object among people included in a captured image of the area around the moving object, and outputs the received display screen to an output unit.

Images