Information management device, display control method, and program

Abstract

To provide users with information about local features that are of interest to them. [Solution] The information management device manages information provided to a first device mounted on a mobile vehicle carrying a crew member and a second device used by a user at a location different from the mobile vehicle. The information management device comprises a display control unit, a body motion information acquisition unit, a detection unit, and a detailed information acquisition unit. The display control unit causes the second device to display an image based on an image captured by an imaging unit of the first device. The body motion information acquisition unit acquires body motion information indicating the user's body movements. The detection unit detects features of interest to the user in the image based on the body motion information acquired by the body motion information acquisition unit. The detailed information acquisition unit acquires detailed information of the features detected by the detection unit. The display control unit displays the features and the detailed information in association.

Description

Technical Field

[0001] The present invention relates to an information management device, a display control method, and a program.

Background Art

[0002] Conventionally, research has been underway on communicating between a device mounted on a moving object such as a vehicle and a device used in a location different from the moving object to share an image of the scenery outside the vehicle (see, for example, Patent Document 1 below). Further, it is known that a head-mounted display (HMD) displays an image of the real space and displays tourist information corresponding to buildings in the real space (see, for example, Patent Document 2 below).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, there is a problem that information on features of interest to a user in a location different from the moving object may not be provided to the user.

[0005] One object of the present application is to provide information on features of interest to a user to the user in order to solve the above problems.

Means for Solving the Problems

[0006] The information management device, display control method, and program according to this invention employ the following configuration. (1): An information management device according to one aspect of the present invention is an information management device for managing information provided to a first device mounted on a mobile vehicle on which a crew member is riding, and a second device used by a user at a location different from the mobile vehicle, comprising: a display control unit that causes the second device to display an image based on an image captured by an imaging unit of the first device; a body movement information acquisition unit that acquires body movement information indicating the body movements of the user; a detection unit that detects a feature of interest to the user in the image based on the body movement information acquired by the body movement information acquisition unit; and a detailed information acquisition unit that acquires detailed information of the feature detected by the detection unit, wherein the display control unit displays the feature and the detailed information in association. It is an information management device.

[0007] (2) In the embodiment of (1) above, the body movement information includes gaze information indicating the direction of the user's gaze, and the detection unit detects the object in the direction of the gaze for a predetermined time or longer based on the gaze information.

[0008] (3) In the embodiment of (1) or (2) above, the motion information includes behavior information indicating the user's behavior, and the detection unit detects the object indicated by the behavior based on the behavior information.

[0009] (4) In the embodiment of (1) or (2) above, the motion information includes voice information indicating the voice spoken by the user, and the detection unit detects the feature whose name was spoken by the user based on the voice information.

[0010] (5) A display control method according to one aspect of the present invention is a display control method in which an information management device manages information provided to a first device mounted on a mobile vehicle on which a crew member is riding and a second device used by a user at a location different from the mobile vehicle, and the information management device performs a process including: a first display control step of causing the second device to display an image based on an image captured by an imaging unit provided in the first device; a body movement information acquisition step of acquiring body movement information indicating the body movements of the user; a detection step of detecting a feature of interest to the user in the image based on the body movement information acquired in the body movement information acquisition step; a detailed information acquisition step of acquiring detailed information of the feature detected in the detection step; and a second display control step of displaying the feature and the detailed information in association.

[0011] (6): A program according to one aspect of the present invention is a program that causes a computer in an information management device, which manages information provided to a first device mounted on a mobile vehicle on which a crew member is riding and a second device used by a user at a location different from the mobile vehicle, to function as a display control unit that causes the second device to display an image based on an image captured by an imaging unit of the first device, a motion information acquisition unit that acquires motion information indicating the body movements of the user, a detection unit that detects a feature of interest to the user in the image based on the motion information acquired by the motion information acquisition unit, and a detailed information acquisition unit that acquires detailed information of the feature detected by the detection unit, and the display control unit displays the feature and the detailed information in association. [Effects of the Invention]

[0012] According to the embodiments described in (1) to (6) above, information on features of interest to the user can be provided to that user. [Brief explanation of the drawing]

[0013] [Figure 1] This diagram shows the operating environment of the information processing system and management server. [Figure 2A]This is a diagram showing an example of the content of user data. [Figure 2B] This is a diagram showing an example of the content of user data. [Figure 3] This is a diagram showing a configuration example of the first device. [Figure 4] This is a diagram showing an example of the arrangement of a part of the first device in the mobile body M. [Figure 5] This is a diagram showing a configuration example of the second device. [Figure 6] This is a diagram showing an example of the angular field of view corresponding to the pointing direction. [Figure 7] This is a diagram showing an example of the functional configuration of the first control device and the second control device. [Figure 8] This is a sequence diagram showing an example of the processing executed by the information processing system. [Figure 9] This is a flowchart showing an example of the display processing of POI information performed by the management server. [Figure 10] This is a flowchart showing an example of the deletion processing of POI information performed by the management server. [Figure 11] This is a diagram showing an example of the display of POI information based on the line-of-sight information. [Figure 12] This is a diagram showing an example of the display of POI information based on the behavior information.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, referring to the drawings, embodiments of the information management device, display control method, and program of the present invention will be described. In the following, an information processing system to which the information management device is applied will be described. The information processing system includes a first device, a second device, and an information management device. The first device is mounted on, for example, a mobile body on which a passenger rides. The second device is used by a user at a location different from the mobile body. The information management device manages the first device and the second device. The mobile body is, for example, a vehicle. However, the mobile body may be any mobile body (for example, a ship, an aircraft) as long as a passenger can ride on it. Also, the passenger is mainly the driver of the mobile body, but may be a passenger other than the driver.

[0015] Between the first device and the second device, a state is achieved where a telephone call is made by transmitting the voice collected by the microphone to the other side and playing it back by the speaker. Further, by displaying a part of the image captured by the camera unit of the first device on the second device, MR (Mixed Reality) is provided to the second device side. As a result, the user of the second device can obtain a feeling (virtual riding experience) as if he / she is virtually riding on the moving body while being in a place different from the moving body. Also, the passenger can obtain a feeling as if the user is actually riding together on the moving body by conversing with the user via the first device. Hereinafter, as described above, the case where the user has a virtual experience as if actually riding on the moving body may be referred to as "virtually riding". Also, in the information processing system, the first device and the second device do not necessarily have a one-to-one relationship, and a one-to-many relationship is also possible. Also, the first device may be plural, that is, in the information processing system, a plurality of second devices may be connected and operate for each of the plurality of first devices. In this case, for example, one passenger can communicate with a plurality of users simultaneously or in sequence.

[0016] <Basic Configuration> Figure 1 shows the operating environment of the information processing system 1 and the management server 300. The information processing system 1 includes a first device 100 and a second device 200. The first device 100 is a mobile device mounted on a mobile vehicle M on which a crew member P is riding. The second device 200 is a user device used by a user U at a location different from the mobile vehicle M (although it is not excluded that it may be in a nearby location). The first device 100, the second device 200, and the management server 300 each communicate with each other via a network NW. The network NW includes at least one of the following: the Internet, WAN (Wide Area Network), LAN (Local Area Network), mobile communication network, cellular network, etc. The management server 300 is an example of an "information management device". Note that the management server 300 may be implemented as a server device or storage device incorporated into a cloud computing system. In this case, the functions of the management server 300 may be implemented by multiple server devices and storage devices in the cloud computing system. Furthermore, the first device 100 mounted on the mobile body M may be realized by multiple units.

[0017] The management server 300 manages the information provided to the first device 100 and the second device 200, and manages communication between them. As shown in Figure 1, the management server 300 includes, for example, a communication device 310, a matching processing unit 320, a provisioning unit 330, a charge management unit 335, a mobile object display control unit 343, a motion information acquisition unit 345, a detection unit 347, a detailed information acquisition unit 349, and a storage unit 350. The matching processing unit 320, the provisioning unit 330, the charge management unit 335, the mobile object display control unit 343, the motion information acquisition unit 345, the detection unit 347, and the detailed information acquisition unit 349 are implemented, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or by the collaboration of software and hardware. The program may be stored in advance on a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device with a non-transient storage medium), or it may be stored on a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is inserted into a drive device. Alternatively, the program may be stored on a management server 300 and updated as needed when connected to a network NW.

[0018] The communication device 310 is a communication interface for connecting to the network NW. Communication between the communication device 310 and the first device 100, and communication between the communication device 310 and the second device 200, are conducted, for example, according to TCP / IP (Transmission Control Protocol / Internet Protocol).

[0019] The matching processing unit 320 is implemented by a processor, such as a CPU, executing a program (set of instructions) stored in a storage medium. For example, when the communication device 310 receives a matching request from user U via the second device 200, or from crew member P via the first device 100, the matching processing unit 320 refers to user data 360 to perform matching between user U and crew member P, and uses the communication device 310 to send the communication identification information of crew member P's first device 100 to the matched user U's second device 200, and sends the communication identification information of user U's second device 200 to the matched crew member P's first device 100. Between the first device 100 and the second device 200, which have received these, more real-time communication is possible, for example, using UDP (User Datagram Protocol).

[0020] The provisioning unit 330 generates information to be provided to the first device 100 and the second device 200, respectively, and transmits the generated information to the target devices. The provisioning unit 330 also generates information such as the processing results by the matching processing unit 320 and the fee information (settlement information) managed by the fee management unit 335, and provides the generated information to the target devices.

[0021] The fare management unit 335 manages the fares to be charged to user U based on the information provided to user U, and the fares to be charged to crew member P based on the information provided to crew member P of the mobile vehicle M. The fare management unit 335 may also manage, for example, the compensation paid to user U and crew member P based on the information provided by user U and crew member P. The fare management unit 335 may also handle settlement processing for user U and crew member P.

[0022] The storage unit 350 may be implemented using the various storage devices mentioned above, or an SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The storage unit 350 can store, for example, user data 360, a provided information DB 362, programs, and various other information. The user data 360 and the provided information DB 362 are not limited to being provided on the management server 300, but may also be provided on other devices. In this case, the management server 300 can access the other device and obtain information related to the user data 360 and the provided information DB 362 from that device.

[0023] The moving object display control unit 343, the body movement information acquisition unit 345, the detection unit 347, and the detailed information acquisition unit 349 will be described later.

[0024] Figure 2 shows an example of the contents of user data 360. User data 360 includes a crew list 360A and a user list 360B. Crew list 360A includes, for example, crew ID, communication identification information, user ID of the target user U, mobile device information, and availability information. Crew ID is the identification information of crew member P of mobile device M. Communication identification information is information that identifies the first device 100, such as an IP address. User ID is the identification information of user U to be matched. Mobile device information includes, for example, information on equipment mounted on mobile device M (mounted equipment information) and vehicle class information indicating the size and external shape of mobile device M. Mobile device information may also include information on the current location, destination, and surrounding conditions of mobile device M (for example, driving on a road along the coast) transmitted from mobile device M at predetermined intervals.

[0025] The availability information indicates various types of information that can (or cannot) be provided to user U. This availability information is set by crew member P. It may be set for each piece of equipment on the mobile device M, or for each user U. Examples of availability information include "allow / deny providing images," "allow / deny providing audio," "allow / deny providing interior images," "allow / deny providing exterior images," "allow / deny providing crew images," and "allow / deny using navigation devices." The availability information may also include a fee (service fee) for enabling the provision of these services.

[0026] The user list 360B includes, for example, a user ID, communication identification information (such as an IP address), the crew ID of the crew member P to be matched, and user information. The user ID is the identification information of user U. The communication identification information is information that identifies the second device 200, such as an IP address. The crew ID is the identification information of the crew member P to be matched. The user information is information about user U, and may include, for example, information about physical characteristics (e.g., height and sitting height) or information that can predict physical characteristics (e.g., age). User data 360 may be generated in any manner, not limited to the manner shown in Figure 2A, as long as it includes this information.

[0027] The provided information DB362 stores various types of information to be provided to user U or crew member P. This information includes, for example, map information, POI (Point of Interest) information, and computer-generated images (e.g., computer graphics images of people, marks, symbols, icons, etc.). POI information, for example, is information about various shops, theme parks, and landmarks at specific locations, and may be included in the map information. Audio information may also be included in this information.

[0028] Figure 2B shows an example of POI information stored in the provided information DB362. POI information (an example of detailed information) includes items such as location information, feature name, rating, reviews, and basic information. Location information indicates the location of the feature on the map. Feature name indicates the name of the feature. Features include everything that appears on the map, such as shops, facilities, buildings, structures, ruins, mountains and rivers, the sea, and tourist attractions.

[0029] The rating is a score calculated based on the ratings of each user who has used the property in question. Reviews are the impressions of users who have used the property. Basic information includes information such as the property's address, contact information, and business hours. In addition, basic information includes all information about the building (for example, in the case of a building, it includes information such as the year it was built, the number of floors, the construction company, the management company, and the usage fee).

[0030] Figure 3 shows an example configuration of the first device 100. The first device 100 includes, for example, a first communication device 110, a first microphone 120, an external sensor 125, a camera unit 130, a first speaker 140, a user display device 150, an HMI (Human-machine Interface) 160, and a first control device 170. The first control device 170 is connected to a controlled device 190 mounted on a mobile body M.

[0031] The first communication device 110 is a communication interface for communicating with the communication device 310 of the management server 300 and the second communication device 210 of the second device 200 via the network NW.

[0032] The first microphone 120 is installed inside or outside the mobile vehicle M and is capable of collecting voices emitted by the occupant P and voices from outside the mobile vehicle M. In the following, voice information acquired by the microphone installed inside the mobile vehicle M may be referred to as "indoor voice information". The voice collected by the first microphone 120 is transmitted, for example, via the first control device 170 to the management server 300 or the second device 200 by the first communication device 110. In the case of a configuration without a microphone installed outside the mobile vehicle M, the indoor voice information may be processed based on driving information (vehicle speed, acceleration / deceleration, road vibration, etc.) and the surrounding driving environment to generate pseudo-outdoor voice information. It is also possible to record the positional relationship of the speaker relative to the mobile vehicle M (whether the speaker is inside or outside the vehicle), and processing may be performed on the collected voice according to that positional relationship.

[0033] The external sensor 125 detects the position of objects around the moving object M. The external sensor 125 is, for example, a radar device, a LIDAR (Light Detection and Ranging) sensor, or various other proximity sensors. The radar device emits radio waves such as millimeter waves around the moving object M and detects the radio waves reflected by objects (reflected waves) to detect at least the position (distance and direction) of the objects. The radar device may also detect the position and velocity of objects using the FM-CW (Frequency Modulated Continuous Wave) method. The LIDAR sensor irradiates light (or electromagnetic waves with a wavelength close to light) around the moving object M, measures the scattered light, and detects the distance to the target based on the time from emission to reception. The irradiated light is, for example, pulsed laser light. The radar device and LIDAR sensor can be attached to any part of the moving object M. Alternatively, the external sensor 125 may detect surrounding objects using images captured by the outdoor camera 134 of the camera unit 130.

[0034] The camera unit 130 includes, for example, an indoor camera 132 and an outdoor camera 134. The first speaker 140 outputs the voice spoken by user U, which is acquired via the first communication device 110. Details of the camera unit 130 and the arrangement of the first speaker 140 will be explained later with reference to Figure 4.

[0035] The user display device 150 virtually displays user U as if the user U were present inside the mobile device M. For example, the user display device 150 may display a hologram, or it may display user U in a part of the mobile device M corresponding to a mirror or window.

[0036] HMI160 consists of a touch panel and a voice response device (agent device), etc. HMI160 receives various instructions from crew member P to the first device 100 and provides various information to crew member P.

[0037] The first control device 170 includes, for example, a processor such as a CPU and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the first device 100 by executing the set of instructions.

[0038] The controlled device 190 is, for example, an in-vehicle device such as a navigation device mounted on the mobile vehicle M that guides the driver to a destination, or a driver assistance device that controls either or both of the steering and / or speed of the mobile vehicle M to assist the driver P. The controlled device 190 includes, for example, a seat drive device that can adjust the position (front, back, left, and right), orientation, and height of the seat. When the camera unit 130 of the first device 100 is attached to the seat and the image is viewed by the second device 200, the impact on the image can be suppressed by prohibiting seat movement. Even if seat movement is permitted, processing such as angle of view conversion may be performed to ensure that the image is not affected when the seat is moved. Furthermore, if the user U requests to view an image outside the current angle of view of the camera unit 130, the seat drive device may be controlled as a request from the second device 200.

[0039] Figure 4 shows an example of the arrangement of part of the first device 100 in the mobile vehicle M. The interior camera 132 is attached, for example, to the neck pillow of the passenger seat S2 (an example of a "designated seat") via attachment 132A, and is positioned slightly away from the backrest of the passenger seat S2 towards the direction of travel of the mobile vehicle M. The interior camera 132 has a wide-angle lens and is capable of imaging the area represented by the hatched area 132B in the figure. The interior camera 132 can photograph not only the interior of the mobile vehicle M but also the exterior through the windows. In the following description, the passenger seat S2 is assumed to be a designated seat, but the designated seat may be another seat such as a rear seat.

[0040] The outdoor camera 134 includes, for example, a plurality of sub-outdoor cameras 134-1 to 134-4. By combining the images captured by the plurality of sub-outdoor cameras 134-1 to 134-4, an image such as a panoramic image capturing the outside of the mobile body M can be obtained. The outdoor camera 134 may also include (or in addition to) these a wide-angle camera mounted on the roof of the mobile body M. An indoor camera 132 capable of capturing the area behind the passenger seat S2 may also be added. The image captured by the camera unit 130 may be generated as a 360-degree panoramic image. The 360-degree panoramic image may be obtained, for example, by combining images captured by one or more indoor cameras 132 by the first control device 170, or by appropriately combining images captured by indoor cameras 132 and images captured by outdoor cameras 134.

[0041] The first speaker 140 outputs the voice of user U. This voice is acquired via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, sub-first speaker 140-1 is located in the center of the instrument panel. Sub-first speaker 140-2 is located at the left end of the instrument panel. Sub-first speaker 140-3 is located at the right end of the instrument panel. Sub-first speaker 140-4 is located at the bottom of the left door. Sub-first speaker 140-5 is located at the bottom of the right door.

[0042] When the first control device 170 outputs the voice of user U to the first speaker 140, it localizes the sound image so that the voice sounds as if it were coming from the passenger seat S2 to the occupant P seated in the driver's seat S1. This sound image localization is achieved, for example, by outputting sound at roughly the same volume from sub-first speakers 140-2 and 140-4, and turning off the other sub-first speakers. Note that sound image localization is not limited to volume adjustment; it may also be achieved by shifting the phase (timing) of the sound output by each sub-first speaker 140. For example, to localize the sound image so that it sounds as if it is coming from the left, the timing of outputting sound from the left sub-first speaker should be slightly earlier than the timing of outputting the same sound from the right sub-first speaker.

[0043] Furthermore, when the first control device 170 outputs the voice of user U to the first speaker 140, it may direct the sound image to the occupant P so that the voice is heard from a position corresponding to the height of user U's head on the passenger seat S2, and then output the voice emitted by user U to the first speaker 140. In this case, the first speaker 140 needs to have multiple child first speakers 140-k (where k is a number of natural numbers) at different heights.

[0044] Figure 5 shows an example configuration of the second device 200. The second device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a mobile image display device 250, an HMI 260, and a second control device 270. The detection device 230 includes a direction detection device 232 and a motion sensor 236.

[0045] The second communication device 210 is a communication interface for communicating with the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0046] The second microphone 220 collects the voice emitted by user U. The voice collected by the second microphone 220 is transmitted, for example, via the second control device 270 to the management server 300 and the first communication device 110 by the second communication device 210.

[0047] The direction detection device 232 is a device for detecting the direction of direction. The direction of direction is the orientation of the user U's face or gaze, or an orientation based on both. The direction of direction is an angle that includes a vertical component. However, the direction of direction may also be an angle in the horizontal plane, i.e., an angle that does not have a vertical component. The direction detection device 232 may include a physical sensor (e.g., an acceleration sensor, a gyroscope, etc.) attached to the VR goggles described later, or it may be an infrared sensor that detects multiple positions of the user U's head, or a camera that images the user U's head. In any case, the second control device 270 calculates the direction of direction based on the information input from the direction detection device 232. Since various technologies for this are publicly known, a detailed explanation will be omitted. The information indicating the direction of direction detected by the direction detection device 232 is transmitted to the management server 300 by the second communication device 210.

[0048] The motion sensor 236 is a device for recognizing gesture operations performed by user U. For example, a camera that captures images of user U's upper body is used as the motion sensor 236. In this case, the second control device extracts characteristic points of user U's body (fingertips, wrists, elbows, etc.) from the images captured by the camera and recognizes user U's gesture operations based on the movement of these characteristic points. Information indicating the gesture operations recognized by the motion sensor 236 is transmitted to the management server 300 by the second communication device 210.

[0049] The second speaker 240 outputs various sounds. For example, the second speaker 240 outputs the voice emitted by occupant P. This voice is acquired via the second communication device 210. The second speaker 240 has a function, for example, to change the direction from which the sound is heard. The second control device 270 instructs the user U to output sound from the second speaker so that the sound is heard from the position of occupant P as seen from the passenger seat S2. The second speaker 240 includes multiple child second speakers 240-n (where n is a natural number). The second speaker 240 performs sound localization by the second control device 270 adjusting the volume of each child second speaker 240-n. If headphones are attached to the VR goggles, sound localization may be performed by utilizing the function of the headphones.

[0050] The mobile image display device 250 displays the image corresponding to the direction of direction as seen from the passenger seat, from among the images captured by the camera unit 130 (which may be images that have undergone stitching processing, and will be referred to as mobile images below).

[0051] Figure 6 shows an example of a field of view corresponding to the direction of direction. In this example, the VR goggles 255 include a physical sensor as a direction detection device 232 and a moving image display device 250. The second control device 270, for example, uses the center of the user U's head or the center of the VR goggles 255 as Ω, and detects the direction the VR goggles 255 are facing as the direction of direction φ, using a pre-calibrated direction as the reference direction. Since various methods for this function are already publicly known, a detailed explanation is omitted.

[0052] The moving object image display device 250 displays a field of view image A2, which is within an angular range of plus or minus α centered on the directional direction φ, from the moving object image A1 (which has an angle of approximately 240 degrees in the figure, but the field of view may be expanded by the merging process), toward the user U. In the example in Figure 6, the directional direction φ for movement is shown to the left and right of the user U's head, but it is also possible to perform processing to extract an image corresponding to the directional direction for vertical movement of the head.

[0053] HMI260 consists of a touch panel, voice response device (agent device), and switches. HMI260 receives various instructions from crew member P to the second device 200.

[0054] The second control device 270 includes, for example, a processor such as a CPU and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the second device 200 by executing the set of instructions. Note that all the functions of the second device 200 shown in Figure 5 may be integrated into the VR goggles.

[0055] <Functional configuration of the first control device 170 and the second control device 270> The functional configurations of the first control device 170 and the second control device 270 are described below. In the example shown below, the first control device 170 and the second control device 270 are mainly responsible for generating and transmitting information, while the management server 300 manages the matching of simulated passengers P and users U.

[0056] Figure 7 shows an example of the functional configuration of the first control device 170 and the second control device 270. In Figure 7, the first control device 170 includes a matching request / acceptance unit 171, an audio output control unit 172, an image transmission unit 173, and an onboard device linkage unit 174. The second control device 270 includes a matching request / acceptance unit 271, an audio output control unit 272, a direction detection unit 273, a gesture input detection unit 275, and an image editing unit 276. These functional units are realized, for example, by a processor such as a CPU executing a program (set of instructions). Some or all of these components may be realized by hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or by the cooperation of software and hardware.

[0057] The matching request / acceptance unit 171 uses the HMI 160 to receive input of a matching request (simulated boarding request) from the occupant P and transmits it to the management server 300, and also uses the HMI 160 to receive input of acceptance of a matching request received from the management server 300 and transmits it to the management server 300. The matching request / acceptance unit 171 controls the first communication device 110 to communicate with the second device 200 of user U, with whom a match has been made. In addition, the matching request / acceptance unit 171 may, for example, accept the setting of information provision permission information indicating whether or not to allow the occupant P to provide information when accepting a matching request obtained from the management server 300.

[0058] The audio output control unit 172 controls the first speaker 140.

[0059] After communication with the second device 200 begins, the image transmission unit 173 uses the first communication device 110 to transmit the mobile image A1 to the second device 200. The mobile image A1 may include, for example, an indoor image captured by the indoor camera 132 and / or an outdoor image captured by the outdoor camera 134.

[0060] The onboard equipment linkage unit 174 controls the controlled equipment 190 based on instruction signals input from the second device 200. The onboard equipment linkage unit 174 may also perform control if the user U has been authorized by the crew member P to control the controlled equipment 190.

[0061] The matching request / acceptance unit 271 uses the HMI 260 to receive matching requests (simulated ride requests) from user U and transmits them to the management server 300, and also uses the HMI 260 to receive acceptance input for matching requests received from the management server 300 and transmits it to the management server 300. The matching request / acceptance unit 271 controls the second communication device 210 to communicate with the first device 100 of the crew member P with whom a match has been made. The matching request / acceptance unit 271 may also, for example, present to user U what information will be provided based on the matching result (request result) obtained from the management server 300, allow user U to confirm, and then accept instructions from user U on whether or not to proceed with the simulated ride.

[0062] The audio output control unit 272 controls the second speaker 240.

[0063] The direction detection unit 273 detects the direction φ based on the output of the direction detection device 232. The gesture input detection unit 275 detects the gesture input of user U based on the output of the motion sensor 236.

[0064] The image editing unit 276 performs a process to extract, for example, a field of view image A2 corresponding to the directional direction φ as seen from the passenger seat S2, from the moving object image A1 (Figure 6). The field of view image A2 extracted by the image editing unit 276 is sequentially transmitted to the management server 300 by the second communication device 210.

[0065] <Configuration of the mobile display control unit 343, etc., provided by the management server 300> Next, we will describe the mobile object display control unit 343, the body movement information acquisition unit 345, the detection unit 347, and the detailed information acquisition unit 349, which are included in the management server 300 shown in Figure 1. Note that, although details will be described later, these functional units can also be provided in the second device 200.

[0066] <Mobile object display control unit 343> The mobile object display control unit 343 (an example of a display control unit) controls the image to be displayed on the second device 200. Based on the captured image taken by the camera unit 130 (an example of an imaging unit) of the first device 100, the mobile object display control unit 343 causes the second device 200 to display the field of view image A2. Specifically, the mobile object display control unit 343 causes the field of view image A2, which has been cropped by the image editing unit 276, to be displayed on the mobile object image display device 250. The field of view image A2 may be an image corresponding to the directional direction φ as seen from the height indicated by the height information of the user U's head. For example, the field of view image A2 may be an indoor image or an outdoor image that can be switched according to the distance from the reference position of the user U's head (for example, upward movement or lateral movement).

[0067] <Body movement information acquisition unit 345> The body movement information acquisition unit 345 acquires body movement information of user U transmitted by the second device 200 via the network NW. The body movement information includes gaze information, behavior information, and voice information. Gaze information is information indicating the direction of the gaze detected by the direction detection device 232. Behavior information is information indicating the behavior of user U. Behavior information includes information indicating the fingertips recognized by the motion sensor 236 and information indicating the direction of the face detected by the direction detection device 232. Voice information is voice information collected by the second microphone 220.

[0068] <Detection unit 347> The detection unit 347 detects a feature on the ground (hereinafter referred to as the "target feature") that the user U focuses on in the field of view image A2 based on the body movement information acquired by the body movement information acquisition unit 345. Specifically, the detection unit 347 detects, based on the line-of-sight information, a feature on the ground on which the line of sight has been directed for a predetermined time (for example, 3 seconds) or longer as the target feature. In addition, the detection unit 347 detects, based on the behavior information, a feature on the ground at the position indicated by the behavior (finger tip or face direction) as the target feature. Further, the detection unit 347 detects, based on the voice information, a feature on the ground with the name uttered by the user U as the target feature. Note that the detection unit 347 determines whether or not the feature on the ground with the name uttered by the user U exists within the field of view image A2, and detects the target feature when the result of the determination is affirmative. In other words, the detection unit 347 does not detect the target feature when the feature on the ground with the name uttered by the user U does not exist within the field of view image A2.

[0069] <Detailed information acquisition unit 349> The detailed information acquisition unit 349 acquires POI information (an example of detailed information) of the target feature detected by the detection unit 347. Specifically, the detailed information acquisition unit 349 acquires the POI information corresponding to the target feature from the POI information stored in the provided information DB 362.

[0070] <Display of POI information> The mobile body display control unit 343 associates and displays the POI information acquired by the detailed information acquisition unit 349 and the target feature in the field of view image A2. Specifically, the mobile body display control unit 343 superimposes and displays the POI information on the target feature on the field of view image A2. The displayed POI information may be output as voice by the second device 200 (second speaker 240).

[0071] The POI information is pop-up displayed (two-dimensional display) near the target object in the field-of-view image A2. The pop-up display does not cover the entire field-of-view image A2, but is displayed small around the target object. Therefore, not all of the POI information may fit within the display area of the pop-up display. In this case, scrolling may be accepted from the user U, and all of the POI information may be made viewable according to the scrolling. Also, the POI information is not limited to being displayed as a still image, and may be displayed as a moving image. Also, the POI information may be displayed as a moving image only when not all of it fits within the pop-up display.

[0072] Also, the POI information is not limited to two-dimensional display, and may be three-dimensional display. The three-dimensional display may be such that different information is displayed on each of a plurality of pages like a book, and by accepting a page-turning operation from the user U, the information on other pages may be displayed. Also, the three-dimensional display may be a polyhedron (e.g., a die) on which different information is displayed on each face, and by accepting a selection operation of the display face from the user U, the information on other faces may be displayed.

[0073] <Deletion of POI Information> Also, the mobile object display control unit 343 deletes the displayed POI information when a predetermined deletion condition is satisfied. The satisfaction of the deletion condition is, for example, after a predetermined time (e.g., 3 seconds), the target object moves out of the line of sight, or the target object is no longer included (disappears) in the field-of-view image A2 due to the travel of the mobile object M. Note that the satisfaction of the deletion condition may be, instead of or in addition to such conditions, an input of a deletion instruction from the user U via the HMI 260 or the like. Thereby, the POI information can be deleted based on the operation of the user U. In particular, when the user U is only looking straight ahead, the POI information of the object in the front continues to be displayed, which becomes unnecessary information for the user U who wants to gaze blankly at the scenery in the front. In such a case, the POI information can be deleted based on the intention of the user U.

[0074] <Contents of POI Information> The display content of the POI information is an item of the POI information stored in the provided information DB 362. That is, the display content of the POI information is the name of the feature, evaluation, reviews, basic information (address, contact information, business hours, usage fee), and the like. Note that the POI information may include, for example, in the case of a store (such as a gas station) that offers products whose prices vary according to the date and time, the price (unit price) of the product, the average value within a predetermined period, and the price trend. Also, the POI information of a supermarket handling special sale items may include the special sale items and the prices of the special sale items.

[0075] <When multiple features overlap> In addition, it is assumed that multiple features exist overlapping at the destination that the user U is paying attention to. In this case, the mobile body display control unit 343 may display candidates for the POI information for each of the multiple overlapping features. For example, assume that there is a park in the front, a tall building behind it, and an even taller Mount Fuji further behind it. In this case, the mobile body display control unit 343 causes the mobile body image display device 250 to display the candidates for the POI information and a pointer (selection button) for selecting the candidates at positions corresponding to the heights of the respective features. The user U can select any one of the candidates by operating the pointer displayed on the mobile body image display device 250. For example, the user U can select the feature (Mount Fuji) existing in the back. Thereby, the mobile body display control unit 343 can display the POI information for any one of the features even when multiple features exist overlapping.

[0076] Note that when multiple features exist overlapping, the mobile body display control unit 343 may display the POI information for the closest feature, or may display the POI information for the most popular feature among the overlapping features. Note that the most popular feature may be, for example, the feature that has been detected as the most frequently noticed feature from past history.

[0077] <Upper limit of the number of displayed POI information> The mobile object display control unit 343 may set an upper limit (for example, two) on the number of POI information items to be displayed within the field of view image A2, and display POI information within that limit. For example, a point of interest that user U is fixated on, a point of interest that user U is calling out, and a point of interest that user U is pointing to may be detected simultaneously. In such cases, a priority order may be set in advance for which point of interest's POI information should be displayed preferentially. This prevents the display area of ​​the field of view image A2 from becoming narrowed due to the simultaneous display of a large number of POI items.

[0078] Let's assume that the display condition for POI information is set to a predetermined gaze time (e.g., 3 seconds), and the deletion condition is set to the same time (3 seconds). In this case, even if POI information for a particular feature (e.g., a building) is displayed when user U gazes at that feature (building), the POI information for the building will be deleted when user U gazes at another feature (e.g., a park) for a predetermined time (the building will be out of the line of sight for the predetermined time). In this way, by setting the predetermined time for the POI information display condition and the predetermined time for the POI information deletion condition to be the same, it is possible to suppress the detection of multiple points of interest based on user U's gaze, that is, it is possible to suppress the display of multiple POI information for multiple points of interest based on user U's gaze.

[0079] <Other examples of motion information: pulse rate> From the perspective that the user U gets excited (when getting excited by seeing features of interest), the body movement information may include information on the user U's pulse. The sensor for detecting the pulse may be provided in the second device 200, or may be another device connectable to the second device 200, such as a wearable terminal. In this case, the detection unit 347 may detect the feature of interest based on the pulse. Specifically, when the pulse is greater than or equal to the threshold value, for example, the detection unit 347 may detect the most popular feature in the visual field image A2 as the feature of interest. Thereby, when the user U looking at the visual field image A2 gets excited, or rather, when a feature of interest to the user U is displayed in the visual field image A2, the POI information can be displayed for the said feature.

[0080] In addition, the detection unit 347 may detect the feature of interest based on both the pulse and the body movement information. For example, when there is a feature towards which the line of sight has been directed for a predetermined time (e.g., 3 seconds) or more and the pulse is greater than or equal to the threshold value, the detection unit 347 may detect the said feature as the feature of interest. Thereby, the features of interest to the user U can be detected with higher accuracy.

[0081] <On / Off Switching of POI Information Display> Regarding the display of the POI information, the user U may be able to switch (set) it on and off. This switching may be performed before the pseudo boarding or during the pseudo boarding. Thereby, for the user U who does not need the POI information, the POI information can be prevented from being displayed.

[0082] <Regarding Post - factum Browsing> The management server 300 may store the POI information displayed during the virtual ride. Other devices accessible to the second device 200 or the management server 300 may later make the POI information viewable. Also, during the display of the POI information, the user U may be accepted to make an on-site check (selection), and only the checked ones may be made viewable later. By enabling such post-facto viewing, the user U can view again the features that he or she actually wants to visit or wants to review again during the virtual ride.

[0083] <In case of bad weather> In case of bad weather, tall features (mountains or towers) that should be visible in the distance may not be visible. In this case, the mobile display control unit 343 may display the feature on the image A and also display the POI information corresponding to the feature.

[0084] <Fee related to the use of POI information> The fee management unit 335 may charge for the service related to the presentation of the POI information and perform processing related to settlement. The service may be a fee according to the number of displayed POI information or a flat fee. Also, the payer of the fee is the user U, but may include the passenger P instead of or in addition to the user U.

[0085] <Processing related to virtual ride> Next, in the embodiment, a series of processes executed by the information processing system 1 when the user U takes a virtual ride on the mobile body M (the passenger P makes the user U take a virtual ride) will be specifically described. FIG. 8 is a sequence diagram showing an example of the processes executed by the information processing system 1. In the example of FIG. 8, the processes in the first device 100 mounted on the mobile body M in which the passenger P is on board, the second device 200 used by the user U, and the management server 300 are shown. Also, it is assumed that the user U and the passenger P are pre-registered in the management server 300 and registered in the user data 360 in order to receive the services in the information processing system 1.

[0086] In the example shown in Figure 8, the second device 200 generates a request for a simulated ride (simulated ride request) (step S100) and sends the generated simulated ride request to the management server 300 (step S102). The simulated ride request may include, for example, information specifying a passenger P in advance, or information specifying the current location, destination, or surrounding conditions of the moving vehicle M. More specific examples of simulated ride requests include requests such as "I want to simulate a ride on a vehicle driven by Mr. A" or "I want to simulate a ride on a vehicle traveling on a road in Tokyo." These are examples of the matching requests described above.

[0087] The matching processing unit 320 of the management server 300 receives a simulated experience request from the second device 200 and performs matching processing based on the user data 360 (step S104). Then, if there is one or more matched mobile bodies M, the provisioning unit 330 generates inquiry information to ask whether or not to allow all mobile bodies M to simulate riding (step S106), and transmits the generated information to the first device 100 of the matched mobile bodies M (step S108).

[0088] The first device 100 displays the inquiry information sent from the management server 300 on a settings screen (step S110). This settings screen includes information about user U who made a simulated ride request, selection buttons for information that crew member P provides to user U (such as whether or not there is a call, whether or not to display interior images, etc.), and a button indicating whether or not user U is allowed to ride the vehicle.

[0089] The first device 100 receives the availability setting of user U from the setting screen (step S112) and transmits the received availability setting information to the management server 300 (step S114). The provision unit 330 of the management server 300 generates a request result (response result) based on the availability setting information obtained from the first device 100 (step S116) and transmits information indicating the generated request result to the second device 200 (step S118).

[0090] The second device 200 receives the request result from the management server 300 and displays information indicating the received request result on the results screen (step S120). The results screen includes information about the passenger P who authorized the simulated ride request, a list of information authorized by passenger P, the fare (service usage fee) based on the authorized information, and a switch to accept or reject the request for simulated ride.

[0091] The second device 200 receives a selection instruction from user U on the results screen (step S122) and transmits the received instruction information to the management server 300 (step S124). In the following description, it is assumed that user U has given an instruction to virtually board the mobile body M of occupant P. Based on the instruction from the second device 200, the management server 300 manages the communication status between the first device 100 and the second device (step S126) and transmits information to the first device 100 and the second device 200 indicating that the virtual boarding will be performed (steps S128, S130). If user U chooses not to virtually board during the processing in steps S126 to S130, the provision unit 330 of the management server 300 transmits information to the first device 100 to notify user U.

[0092] Next, the first device 100 and the second device 200 perform virtual boarding through the management server 300 or directly (step S132). When the virtual boarding is completed, information indicating that the communication has ended is transmitted to the management server 300 (steps S134, S136). In the process of step S132, the second device 200 may notify the passenger P from the user U before the information from the first device 100 is provided to the second device 200. As a result, the user U can permit communication at their own timing and start providing information such as calls and video distribution. Next, the charge management unit 335 of the management server 300 performs settlement processing for the charges related to the use of each of the first device 100 and the second device 200. Specifically, the charge management unit 335 varies the charges for information provision based on at least one of the type of the passenger P, user U, and mobile body M providing the information, the type of information that can be provided to the user U, and the amount of information. For example, when the passenger P or the user U is a famous person, the usage fee is set higher than the normal fee, or when the mobile body M is a new car or a rare vehicle (rare car), the fee is also set higher than the normal fee. In addition, the charge management unit 335 performs settlement processing corresponding to the set charges. As a result, the processing of this sequence ends.

[0093] <Display control processing performed by the management server 300 in virtual boarding> Next, with reference to FIGS. 9 and 10, the display control processing performed by the management server 300 in the execution of virtual boarding (step S132 in FIG. 8) will be described. The processing shown in FIGS. 9 and 10 is executed at the control cycle of the management server 300 during the execution of virtual boarding.

[0094] <Display processing of POI information> Figure 9 is a flowchart illustrating an example of the POI information display processing performed by the management server 300. In Figure 9, the management server 300 determines whether the gaze destination obtained from the gaze information on the field of view image A2 is the same for a predetermined time (e.g., 3 seconds) (step S901). If the gaze destination is not the same for the predetermined time (step S901: NO), the management server 300 proceeds to step S903. On the other hand, if the gaze destination is the same for the predetermined time (step S901: YES), the management server 300 determines whether there is a feature at the gaze destination on the field of view image A2 (step S902).

[0095] If there is a feature in the line of sight (step S902: YES), the management server 300 proceeds to step S907 and detects the feature as a feature of interest. On the other hand, if there is no feature in the line of sight (step S902: NO), the management server 300 determines whether or not there is behavior based on the behavior information (step S903). If there is no behavior (step S903: NO), the management server 300 proceeds to step S905. On the other hand, if there is behavior (change in the direction of fingertips or face) (step S903: YES), the management server 300 determines whether or not there is a feature in the direction indicated by the behavior (step S904).

[0096] If there is a feature at the location indicated by the behavior (step S904: YES), the management server 300 proceeds to step S907 and detects the feature as a feature of interest. On the other hand, if there is no feature at the location indicated by the behavior (step S904: NO), the management server 300 determines, based on the voice information, whether or not user U has uttered the name of a feature (step S905). If user U has not uttered the name of a feature (step S905: NO), the management server 300 returns to step S901.

[0097] On the other hand, when the user U utters a feature name (step S905: YES), the management server 300 determines whether the feature with the name uttered by the user U exists within the field of view image A2 (step S906). If the feature with the name uttered by the user U does not exist within the image A (step S906: NO), the management server 300 returns to the process of step S901. On the other hand, if the feature with the name uttered by the user U exists within the field of view image A2 (step S906: YES), the management server 300 detects the feature as the target feature (step S907).

[0098] Then, the management server 300 acquires the POI information corresponding to the target feature from the POI information stored in the provision information DB362 (step S908), associates the acquired POI information with the target feature, and causes the second device 200 (mobile body image display device 250) to display it (step S909). Next, the management server 300 determines whether the execution of the pseudo boarding has ended (step S910). If the execution of the pseudo boarding has not ended (step S910: NO), the management server 300 returns to the process of step S901. On the other hand, if the execution of the pseudo boarding has ended (step S910: YES), the management server 300 ends the series of processes shown in FIG. 9.

[0099] <POI information deletion process> FIG. 10 is a flowchart showing an example of the POI information deletion process performed by the management server 300. In FIG. 10, the management server 300 determines whether the POI information is being displayed on the second device 200 (step S1001). If the POI information is not being displayed (step S1001: NO), the management server 300 proceeds to the process of step S1005. On the other hand, if the POI information is being displayed (step S1001: YES), the management server 300 determines whether the target feature has moved out of the line of sight for a predetermined time (step S1002).

[0100] If the line of sight moves away from the point of interest after a predetermined time (step S1002: YES), the management server 300 proceeds to step S1004 and deletes the POI information. On the other hand, if the line of sight does not move away from the point of interest after a predetermined time (step S1002: NO), the management server 300 determines whether or not the point of interest is included in the field of view image A2 (step S1003). If the point of interest is included in the field of view image A2 (step S1003: YES), the management server 300 returns to step S1002 and repeats the processing of steps S1002 and S1003.

[0101] On the other hand, if the point of interest is no longer included in the view image A2 (step S1003: NO), the management server 300 deletes the POI information (step S1004). Then, the management server 300 determines whether the execution of the simulated ride has finished (step S1005). If the execution of the simulated ride has not finished (step S1005: NO), the management server 300 returns to the process in step S1001. On the other hand, if the execution of the simulated ride has finished (step S1005: YES), the management server 300 terminates the series of processes shown in Figure 10.

[0102] <Information provided during simulated ride> Next, we will explain some examples of the information provided to user U during a simulated ride. The following examples are the information provided in step S132 shown in Figure 9, and the output manner of the information provided to the second device 200 differs depending on, for example, the actions of user U, and the type or amount of information that can be provided to user U. In the following examples, it is assumed that occupant P is authorized to provide interior images, exterior images, and telephone (voice) communication.

[0103] <Example of POI information based on eye-tracking data> Figure 11 shows an example of displaying POI information based on gaze information. In the example in Figure 11, a user U is seated on a chair CH in a location other than the mobile vehicle M (for example, a room in one's home), and an example of an interior image IM30 provided by a second device 200 (integrated with VR goggles in the example in Figure 11) used by user U is shown. The state of user U shown in the example in Figure 11 (seated on chair CH) represents the initial state immediately after the simulated ride is performed, and the first device 100 and the second device 200 recognize user U's position (reference posture) at this time as the reference position. The reference position is the position where user U is assumed to be seated in the passenger seat S2. The reference position may be corrected so that the height of the interior camera 132 matches user U's sitting height by the first control device 170 driving the seat drive device based on the user U's height or sitting height information registered in the user information of user data 360 (so-called zero-point correction). Furthermore, the reference position may be corrected based on correction instructions from user U who has viewed the indoor image IM30. The reference position may also include orientation information when user U is seated in chair CH and facing forward.

[0104] For example, when the user U (second device 200) is seated in chair CH and facing any direction from their reference position, an image captured by the interior camera 132 of the first device 100 is provided as an interior image IM30, corresponding to that direction. This allows user U to obtain an interior image of any direction as seen from the passenger seat S2. In addition, audio information of the occupant P's voice and sounds inside and outside the mobile vehicle M, acquired by the first microphone 120, is provided to user U. By using the interior image IM30 and audio information to converse with occupant P, user U can enjoy a conversation as if they were actually riding in the mobile vehicle M.

[0105] In the diagram, user U is assumed to be fixated on building 1100, which is visible in front of them. If the user's gaze is directed towards building 1100 for a predetermined period of time, POI information 1110 is superimposed on the interior image IM30. POI information 1110 includes various information about building 1100, such as its name, rating, reviews, business hours, year of construction, number of floors, and management company. This allows user U to grasp detailed information about building 1100.

[0106] <Example of POI information based on behavioral data> Figure 12 shows an example of displaying POI information based on behavioral information. Compared to the example in Figure 11, Figure 12 shows an example where an outdoor image of the side of the moving object M is provided instead of an indoor image.

[0107] In the example shown in Figure 12, the outdoor image IM50 is displayed when the orientation of user U's face (second device 200) moves laterally by a distance D2 from the reference position. In this case, the first device 100 determines whether the distance moved D2 is greater than or equal to a threshold, and if it is, switches from displaying the indoor image IM30 from the indoor camera 132 to displaying the outdoor image IM50 from the outdoor camera 134. In the example in Figure 12, since the distance D2 is greater than or equal to the threshold, the outdoor image IM50 (more specifically, the image of the front, as the user is facing slightly forward) captured from the side of the moving vehicle M in the direction of movement is displayed. This allows user U to have a simulated ride experience as if they were sticking their head out from the side of the moving vehicle M (for example, through a side window) and looking at the surrounding scenery. Note that the above threshold may be a fixed value or may be set variably according to the vehicle class information of the moving vehicle M.

[0108] Furthermore, in the illustration, user U is assumed to be facing park 1200, which is visible to the side. When the face is facing park 1200, POI information 1210 is superimposed on the outdoor image IM50. POI information 1210 includes various information such as the name, rating, reviews, opening hours, and usage fees of park 1200. This allows user U to grasp detailed information about park 1200.

[0109] Furthermore, the image containing the POI information described above may be provided to the first device 100. This allows the crew member P to more accurately understand what image user U is currently viewing and to engage in more appropriate conversation.

[0110] In another embodiment, the first device 100 may generate an image of user U seated in the passenger seat S2 (or rear seat) and display the generated image on the rearview mirror display device or window. In this case, the first control device 170 generates a person image of user U or a character corresponding to user U seated in the passenger seat S2 and displays the generated image on the rearview mirror display device. In this case, the posture of the person image may be changed according to the actions of user U obtained from the second device 200. The first control device 170 may also control the first speaker 140 to output sound from near the passenger seat S2 when the person image is displayed on the rearview mirror display device 152. This sound is, for example, sound emitted by user U to the second microphone 220, transmitted by the second communication device 210 and acquired via the first communication device 110. The first control device 170, for example, outputs sound to the first speaker 140 that has been localized so that the sound image is heard by the occupant P in the driver's seat from the passenger seat S2.

[0111] <Variation> In the information processing system 1 described above, at least some of the functions of the first device 100 and the second device 200 may be provided on the management server 300, and at least some of the functions of the management server 300 may be provided on the first device 100 or the second device 200.

[0112] For example, the mobile object display control unit 343, the motion information acquisition unit 345, the detection unit 347, and the detailed information acquisition unit 349 may all be provided in the second device 200. The following provides supplementary information on a configuration in which the second device 200 includes each of these functional units. The mobile object display control unit 343 in the second device 200 acquires captured images from the camera unit 130 in the first device 100 and displays the field of view image A2 on the mobile object image display device 250 based on the captured images. The motion information acquisition unit 345 acquires motion information from the direction detection device 232, the motion sensor 236, and the second microphone 220, respectively. The detection unit 347 detects a point of interest in the field of view image A2 based on the motion information. The detailed information acquisition unit 349 acquires POI information for the point of interest by querying the management server 300. The mobile object display control unit 343 displays the point of interest and the POI information in association. Thus, by having the second device 200 equipped with each functional unit, it is possible to display POI information of the point of interest on the mobile image display device 250.

[0113] Furthermore, the functions of the image editing unit 276 described above may also be provided by the management server 300. In other words, the management server 300 may be equipped with all functions related to image editing and image display control.

[0114] <Summary> According to the information processing system 1 configured as described above, based on body movement information, it detects points of interest in the field of view image A2, associates the detected points of interest's POI information with the points of interest, and displays them on the second communication device 210. This makes it possible to provide user U with POI information of points of interest that are of interest to user U during a simulated ride. [Explanation of Symbols]

[0115] 1. Information Processing System 100 1st device 110 First communication device 120 First microphone 125 External Sensors 130 Camera Unit 132 Indoor Cameras 134 Outdoor Camera 140 First speaker 150 User display device 160 HMI 170 First control device 190 Controlled Equipment 200 2nd device 210 Second communication device 220 Second microphone 230 detection device 232 Directional detection device 236 Motion Sensors 240 Second speaker 250 Mobile Image Display Device 260 HMI 270 Second control device 300 Management Servers 310 Communication equipment 315 Acquisition Department 320 Matching Processing Unit 330 Provision Department 335 Billing Department 343 Mobile Display Control Unit 345 Body movement information acquisition unit 347 Detection unit 349 Detailed information acquisition section 350 Storage section M Mobile object U User

Claims

1. An information management device that manages information provided to a first device mounted on a mobile vehicle carrying a crew, and a second device used by a user at a location different from the mobile vehicle, A display control unit that causes the second device to display an image based on an image captured by the imaging unit of the first device, A body motion information acquisition unit acquires body motion information indicating the body movements of the user, A detection unit detects a feature in the image that the user is interested in, based on the motion information acquired by the motion information acquisition unit, A detailed information acquisition unit acquires detailed information about the features detected by the detection unit, Equipped with, The display control unit displays the feature and the detailed information in association. Information management device.

2. The aforementioned body movement information includes gaze information indicating the direction of the user's gaze, The detection unit detects the object in the line of sight for a predetermined period of time or longer based on the line of sight information. The information management device according to claim 1.

3. The aforementioned body movement information includes behavioral information indicating the user's behavior, The detection unit detects the feature indicated by the behavior based on the behavior information. The information management device according to claim 1 or 2.

4. The aforementioned body movement information includes audio information indicating the voice spoken by the user, The detection unit detects the feature whose name was uttered by the user based on the voice information. The information management device according to claim 1 or 2.

5. An information management device that manages information provided to a first device mounted on a mobile vehicle carrying crew members and a second device used by a user at a location different from the said mobile vehicle, A first display control step involves causing the second device to display an image based on an image captured by the imaging unit of the first device, A motion information acquisition step which acquires motion information indicating the body movements of the user, A detection step in which, based on the motion information acquired in the motion information acquisition step, detects a feature in the image that the user is interested in, A detailed information acquisition step is to acquire detailed information of the features detected in the above detection step, A second display control step that displays the aforementioned feature and the aforementioned detailed information in association, A display control method that performs processing including the following.

6. The computer of the information management device that manages information provided to the first device installed on a mobile vehicle carrying the crew and the second device used by the user at a location different from the mobile vehicle, A display control unit causes the second device to display an image based on an image captured by the imaging unit of the first device. A motion information acquisition unit that acquires motion information indicating the body movements of the user. A detection unit detects a feature in the image that the user is interested in, based on the motion information acquired by the motion information acquisition unit. Detailed information acquisition unit that acquires detailed information of the feature detected by the detection unit, To make it function as, The display control unit displays the feature and the detailed information in association. program.

Images