Information processing system

Abstract

To enable users who are in a different location from the moving vehicle to experience a more appropriate simulated ride. [Solution] The information management system according to this embodiment includes a first device mounted on a mobile body, a plurality of second devices each used by a plurality of users in a location different from the mobile body, an information acquisition unit that acquires predetermined information transmitted by the first device, a provision unit that provides the predetermined information acquired by the information acquisition unit to the plurality of second devices, and a plurality of presentation units that present the predetermined information provided to the plurality of second devices by the provision unit. When an operation input is made by the first user to one of the plurality of second devices, the presentation units of the second devices other than the one second device present the output result based on the operation input made by the one second device.

Description

Technical Field

[0001] The present invention relates to an information processing system.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions such as the elderly, disabled, and children among transportation participants. In particular, research and development have been focused on further improving traffic safety and convenience through the development of means of transportation for vulnerable road users in order to achieve this. In relation to this, there is a known system configured to externally store video by storing video data in a time series in an external server (filing device) so that it can be reproduced at any time (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, when performing a pseudo-experience (hereinafter referred to as "pseudo-riding") as if riding in a moving body using a device used in a place different from the moving body, it is a problem that a burden is imposed on a user who performs pseudo-riding in a place different from the moving body.

[0005] The present application aims to enable a user in a place different from the moving body to perform a more appropriate pseudo-riding in order to solve the above problems. And, by extension, it contributes to the development of a sustainable transportation system and the support of good connections between urban areas and local areas including urban peripheries.

Means for Solving the Problems

[0006] The information processing system according to this invention employs the following configuration. (1) An information processing system according to one aspect of the present invention comprises: a first device mounted on a mobile body; a plurality of second devices each used by a plurality of users at a location different from the mobile body; an information acquisition unit that acquires predetermined information transmitted by the first device; a provision unit that provides the predetermined information acquired by the information acquisition unit to the plurality of second devices; and a plurality of presentation units that present the predetermined information provided to the plurality of second devices by the provision unit, wherein when an operation input is made by the first user to one of the plurality of second devices, the presentation units of the second devices other than the one second device present an output result based on the operation input made by the one second device.

[0007] (2): In the embodiment of (1) above, the second device used by the first user further comprises a user information acquisition unit that acquires user information of the second device user, and the second device used by the first user further comprises a creation unit that creates an output signal for presenting the output result based on the user information acquired by the user information acquisition unit.

[0008] (3) In the embodiment of (1) above, the second device includes a gaze detection unit that detects the direction of the user's gaze, and the presentation unit presents the output result based on the direction of the user's gaze detected by the gaze detection unit.

[0009] (4): In the embodiment of (1) above, the second device includes a POI (Point of Interest) acquisition unit that acquires POI information, and the POI acquisition unit acquires POI information based on the position information of the first device when the presentation unit presents the output result, and the presentation unit presents the POI information acquired by the POI acquisition unit together with the output result.

[0010] (5) In the embodiment of (3) above, the second device includes a POI (Point of Interest) acquisition unit that acquires POI information, the POI acquisition unit acquires POI information based on the user's gaze direction detected by the gaze detection unit, and the presentation unit determines a position to present the output result based on the POI information acquired by the POI acquisition unit.

[0011] (6) In the embodiment of (1) above, the second device further comprises a POI acquisition unit that acquires POI (Point of Interest) information, and a selection unit that, when a plurality of POI pieces of information are acquired by the POI acquisition unit, selects the POI piece of information closest to the moving object from the plurality of POI pieces of information, and the presentation unit presents the POI piece of information selected by the selection unit.

[0012] (7) In the embodiment of (1) above, the display unit changes the display position of the output result based on the speed of the moving body.

[0013] (8) The embodiment of (1) above further comprises a moving body behavior detection unit that detects whether the moving body has turned right or left, and the presentation unit changes the content of the output result presentation when the moving body behavior detection unit detects that the moving body has turned right or left after presenting the output result.

[0014] (9) In the embodiment of (1) above, when an operation input is made by the first user to one of the multiple second devices, the first device is further provided with a creation unit that creates notification information including an output result based on the operation input made to one of the second devices.

[0015] (10): In the embodiment of (1) above, the presenting unit further comprises a voice information acquisition unit that acquires voice information based on the user's voice, and the presenting unit presents output results based on the operation input and the voice information.

[0016] (11): In the aspect of (4) above, it further includes a voice information acquisition unit that acquires voice information based on the voice of the user, and the presentation unit presents the POI information together with the output result and the voice information acquired by the voice information acquisition unit in relation to the POI information.

Effect of the Invention

[0017] According to the aspects of (1) to (11) above, it is possible to make a user in a location different from the moving body perform a more appropriate pseudo boarding.

Brief Description of the Drawings

[0018] [Figure 1] It is a diagram showing the usage environment and the like of the information processing system 1 and the management server 300. [Figure 2] It is a diagram showing an example of the content of the user data 360. [Figure 3] It is a configuration diagram of the first device 100. [Figure 4] It is a diagram showing an example of the arrangement of a part of the first device 100 in the moving body M. [Figure 5] It is a configuration diagram of the second device 200. [Figure 6] It is a diagram for explaining an image corresponding to the pointing direction. [Figure 7] It is a diagram showing a first example of the functional configuration of the first control device 170 and the second control device 270. [Figure 8] It is a diagram showing a second example of the functional configuration of the first control device 170 and the second control device 270. [Figure 9] It is a sequence diagram showing an example of the process executed by the information processing system 1. [Figure 10] It is a diagram showing an example of an image IM10 indicating inquiry information. [Figure 11] It is a diagram showing an example of an image IM20 indicating a request result. [Figure 12A] It is a sequence diagram showing a first example of the process executed by the information processing system 1. [Figure 12B]This is a sequence diagram showing a second example of processing performed by information processing system 1. [Figure 12C] This is a sequence diagram showing a third example of processing performed by information processing system 1. [Figure 13A] This is a diagram illustrating an example of the information provided. [Figure 13B] This is a diagram illustrating an example of the information provided. [Figure 13C] This is a diagram illustrating an example of the information provided. [Figure 14] This figure shows a third example of the functional configuration of the first control device 170 and the second control device 270. [Figure 15A] This is a sequence diagram showing a fourth example of processing performed by information processing system 1. [Figure 15B] This is a sequence diagram showing the fifth example of processing performed by information processing system 1. [Figure 16] This is a diagram illustrating an example of the information provided. [Modes for carrying out the invention]

[0019] The embodiments of the information processing system of the present invention will be described below with reference to the drawings. In the following description, an information processing system to which the information processing device is applied will be described. The information processing system includes, for example, a first device mounted on a mobile vehicle on which an occupant is riding, a second device used by a user at a location different from the mobile vehicle, and the information processing device. The mobile vehicle is, for example, a vehicle, but it can be any mobile vehicle (e.g., a ship, a flying object) that an occupant can ride. The occupant is mainly the driver of the mobile vehicle, but it can also be an occupant other than the driver.

[0020] Between the first and second devices, audio collected by a microphone is transmitted to the other side and played back through a speaker, creating a state similar to a telephone call. Furthermore, a portion of the image captured by the camera unit of the first device is displayed on the second device, providing the second device with MR (Mixed Reality). This allows the user of the second device to experience the sensation of being on board the mobile vehicle (simulated ride experience) even though they are in a different location from the mobile vehicle, and the crew can converse with the user experiencing the simulated ride on the mobile vehicle via the first device, giving them the feeling that the user is actually riding on the mobile vehicle with them. Alternatively, a portion of the image captured by the camera unit of the first device may be displayed on the second device, providing the second device with VR (Virtual Reality). This allows the user of the second device to experience the sensation of being on board the mobile vehicle (simulated ride experience) even though they are in a different location from the mobile vehicle, and the crew can converse with the user experiencing the simulated ride on the mobile vehicle via the first device, giving them the feeling that the user is actually riding on the mobile vehicle with them. Furthermore, users can fully immerse themselves in the world of the video and use the application. In the following, as mentioned above, the experience of a user having the simulated feeling of actually riding in a moving vehicle may be referred to as "simulated riding." The relationship between the first and second devices does not need to be one-to-one; the information processing system can operate by matching one of several first devices with several second devices in a one-to-many relationship. In the latter case, for example, one passenger can communicate with multiple users simultaneously or sequentially.

[0021] <Basic configuration> Figure 1 shows the operating environment of the information processing system 1 and the management server 300. The information processing system 1 includes a first device (mobile device) 100 mounted on a mobile vehicle M on which crew member P is riding, and a second device (user device) 200-1 used by user U1 and a second device (user device) 200-2 used by user U2 at a location different from the mobile vehicle M (although it is not ruled out that it may be a nearby location). The first device 100, the second device 200-1, the second device 200-2, and the management server 300 each communicate with each other via a network NW. The information processing system 1 may or may not include the management server 300. Hereinafter, any user among user U1 and user U2 will be referred to as user U. Also, any second device among the second device 200-1 and the second device 200-2 will be referred to as second device 200. The network NW includes at least one of the following: the Internet, WAN (Wide Area Network), LAN (Local Area Network), mobile communication network, cellular network, etc. The management server 300 is an example of an "information processing device." The management server 300 may be implemented as a server device or storage device incorporated into a cloud computing system. In this case, the functions of the management server 300 may be implemented by multiple server devices and storage devices in the cloud computing system. Furthermore, the first device 100 mounted on the mobile device M may be implemented by multiple units.

[0022] The management server 300 manages the information provided to the first device 100 and the second device 200, and manages their communication. The management server 300 includes, for example, a communication device 310, an information acquisition unit 315, a matching processing unit 320, a provisioning unit 330, a billing management unit 340, and a storage unit 350. The information acquisition unit 315, the matching processing unit 320, the provisioning unit 330, and the billing management unit 340 are implemented, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software).

[0023] Some or all of these components may be implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or by the collaboration of software and hardware. The program may be stored in advance on a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device with a non-transient storage medium), or it may be stored on a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is inserted into a drive device. Alternatively, the program may be stored on a management server 300 and updated as needed when connected to a network NW.

[0024] The communication device 310 is a communication interface for connecting to the network NW. Communication between the communication device 310 and the first device 100, and communication between the communication device 310 and the second device 200, are conducted, for example, according to TCP / IP (Transmission Control Protocol / Internet Protocol).

[0025] The information acquisition unit 315 acquires various types of information transmitted via the network NW by the first device 100, the second device 200, and other external devices. These types of information may include video data, driver voice data, and vehicle data (such as the display contents of the METER and IVI (in-vehicle infotainment) systems).

[0026] The matching processing unit 320 is implemented by a processor, such as a CPU, executing a program (set of instructions) stored in a storage medium. For example, when the communication device 310 receives a matching request from user U via the second device 200, or from crew member P via the first device 100, the matching processing unit 320 refers to user data 360 to perform matching between user U and crew member P, and uses the communication device 310 to send the communication identification information of crew member P's first device 100 to the second device 200 of the matched user U, and sends the communication identification information of user U's second device 200 to the first device 100 of the matched crew member P. After receiving these, the first device 100 and the second device 200 can perform more real-time communication, such as one based on UDP (User Datagram Protocol).

[0027] The provisioning unit 330 generates information to be provided to the first device 100 and the second device 200, respectively, based on the various information acquired by the information acquisition unit 315, and transmits the generated information to the target devices. The provisioning unit 330 also generates information such as the processing results by the matching processing unit 320 and the fee information (settlement information) managed by the fee management unit 340, and provides the generated information to the target devices.

[0028] The fare management unit 340 manages the fares to be charged to user U based on the information provided to user U, and the fares to be charged to crew member P based on the information provided to crew member P of the mobile vehicle M. The fare management unit 340 may also manage, for example, the compensation paid to user U and crew member P based on the information provided by user U and crew member P. The fare management unit 340 may also handle settlement processing for user U and crew member P.

[0029] The storage unit 350 may be implemented using the various storage devices mentioned above, or an SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 350 can store, for example, user data 360, provided information DB 362, programs, and various other information.

[0030] Figure 2 shows an example of the contents of user data 360. User data 360 includes a crew list 360A and a user list 360B. Crew list 360A includes, for example, the crew ID, which is the identification information of crew member P of mobile vehicle M, their communication identification information (IP address, etc.), the user ID, which is the identification information of user U to be matched, the mobile vehicle information on which the crew member is riding, and the availability information set by the crew member, all of which are associated with each other. Mobile vehicle information includes, for example, information on equipment installed on mobile vehicle M (installed equipment information) and vehicle class information indicating the size and external shape of mobile vehicle M. Mobile vehicle information may also include information on the current location, destination, and surrounding conditions of mobile vehicle M (for example, driving on a road along the coast) that is transmitted from mobile vehicle M at predetermined intervals.

[0031] The user list 360B associates, for example, a user ID, its communication identification information (such as an IP address), the crew member P to be matched, and user information with each other. User information may include information about physical size (e.g., height and sitting height) and information that can predict physical size (e.g., age). Availability information is information that the mobile unit M can or cannot provide, and is set, for example, by the crew member P. Availability information may be set for each piece of equipment installed on the mobile unit M, or for each user U. Examples of availability information include, but are not limited to, "Provision of images is permitted," "Provision of audio is not permitted," "Provision of interior images is permitted, but provision of exterior images is not permitted," "Provision of crew member images is not permitted," and "Use of navigation devices is not permitted." Availability information may also include a fee for enabling the provision of services (service provision fee). User data 360 may be generated in any manner, not limited to the manner shown in Figure 2, as long as it includes this information.

[0032] The provided information DB362 stores various types of information to be provided to user U or crew P. This information includes, for example, map information, POI (Point of Interest) information, and computer-generated images (e.g., computer graphics images of people, marks, symbols, icons, etc.). POI information, for example, is information about various shops, theme parks, and landmarks at specific locations, and may be included in the map information. Audio information may also be included. Furthermore, the provided information DB362 may include advertising information. This advertising information may include, for example, advertisements related to the mobile vehicle M, advertisements related to user U or crew P, and advertisements related to the products and services of shops. The inserted advertising information is managed separately from indoor and outdoor information, and when archived and distributed later, it may differ from the inserted advertising information distributed in real time (for example, closed shops may be updated, and the featured menus may be updated). Advertising information may include, for example, videos and audio.

[0033] Figure 3 is a configuration diagram of the first device 100. The first device 100 includes, for example, a first communication device 110, a first microphone 120, an external sensor 125, a camera unit 130, a first speaker 140, a user display device 150, an HMI (Human-machine Interface) 160, and a first control device 170. The first control device 170 is connected to a controlled device 190 mounted on a mobile body M.

[0034] The first communication device 110 is a communication interface for communicating via the network NW with the communication device 310 of the management server 300 and the second communication device 210 of the second device 200, which will be described later.

[0035] The first microphone 120 collects at least the voice emitted by the occupant P. The first microphone 120 may be installed inside the vehicle M and have the sensitivity to collect sounds from outside the vehicle M, or it may include a microphone installed inside the vehicle M and a microphone installed outside the vehicle M. Hereinafter, the voice information acquired by the microphone installed inside the vehicle may be referred to as "indoor voice information". The voice collected by the first microphone 120 is transmitted, for example, via the first control device 170 to the management server 300 or the second device 200 by the first communication device 110. If it is not possible to set up a microphone installed outside the vehicle M, the indoor voice information may be processed based on driving information (vehicle speed, acceleration / deceleration, road vibration, etc.) and the surrounding driving environment to generate pseudo-outdoor voice information. It is also possible to record the positional relationship of the speaker relative to the vehicle M (whether the speaker is inside or outside the vehicle), and processing may be performed on the collected voice according to that positional relationship.

[0036] The external sensor 125 detects the position of objects around the moving object M. The external sensor 125 is, for example, a radar device, a LIDAR (Light Detection and Ranging) sensor, or various other proximity sensors. The radar device emits radio waves such as millimeter waves around the moving object M and detects the radio waves reflected by objects (reflected waves) to detect at least the position (distance and direction) of the objects. The radar device may also detect the position and velocity of objects using the FM-CW (Frequency Modulated Continuous Wave) method. The LIDAR sensor irradiates light (or electromagnetic waves with a wavelength close to light) around the moving object M, measures the scattered light, and detects the distance to the target based on the time from emission to reception. The irradiated light is, for example, pulsed laser light. The radar device and LIDAR sensor can be attached to any part of the moving object M. Alternatively, the external sensor 125 may detect surrounding objects using images captured by the outdoor camera 134 of the camera unit 130.

[0037] The camera unit 130 includes, for example, an indoor camera 132 and an outdoor camera 134. The first speaker 140 outputs the voice spoken by user U, which is acquired via the first communication device 110. Details of the camera unit 130 and the arrangement of the first speaker 140 will be explained later with reference to Figure 4.

[0038] The user display device 150 virtually displays user U as if the user U were present inside the mobile device M. For example, the user display device 150 may display a hologram or display user U in a part of the mobile device M corresponding to a mirror or window.

[0039] HMI160 consists of a touch panel and a voice response device (agent device), etc. HMI160 receives various instructions from crew member P to the first device 100 and provides various information to crew member P.

[0040] The first control device 170 includes, for example, a processor such as a CPU, and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the first device 100 by executing the set of instructions.

[0041] The controlled device 190 is, for example, an in-vehicle device such as a navigation system mounted on the mobile vehicle M that guides the driver to a destination, or a driver assistance system that controls either or both the steering and / or speed of the mobile vehicle M to assist the driver P. The controlled device 190 includes, for example, a seat drive system that can adjust the position (front, back, left, and right), orientation, and height of the seat. When viewing the image using the second device 200, if the camera unit 130 of the first device 100 is attached to the seat, the impact on the image can be suppressed by prohibiting seat movement. Even if seat movement is permitted, processing such as angle of view conversion may be performed to ensure that the image is not affected when the seat is moved. Furthermore, if user U requests to view images outside the current angle of view of the camera unit 130, the seat drive system may be controlled as a request from the second device 200.

[0042] Figure 4 shows an example of the arrangement of part of the first device 100 in the mobile vehicle M. The interior camera 132 is attached, for example, to the neck pillow of the passenger seat S2 (an example of a "designated seat") via attachment 132A, and is positioned slightly away from the backrest of the passenger seat S2 towards the direction of travel of the mobile vehicle M. The interior camera 132 has a wide-angle lens and is capable of imaging the area represented by the hatched area 132B in the figure. The interior camera 132 can photograph not only the interior of the mobile vehicle M but also the exterior through the windows. In the following description, the passenger seat S2 is assumed to be a designated seat, but the designated seat may be another seat such as a rear seat.

[0043] The outdoor camera 134 includes, for example, a plurality of sub-outdoor cameras 134-1 to 134-4. By combining the images captured by the plurality of sub-outdoor cameras 134-1 to 134-4, an image such as a panoramic image capturing the outside of the mobile body M is obtained. The outdoor camera 134 may also include (or in addition to) these a wide-angle camera mounted on the roof of the mobile body M. An indoor camera 132 capable of capturing the area behind the passenger seat S2 may be added, and the mobile body image described later may be generated as a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 by the first control device 170, or it may be generated as a 360-degree panoramic image by appropriately combining images captured by indoor cameras 132 and images captured by outdoor cameras 134.

[0044] The first speaker 140 outputs the voice of user U acquired via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, sub-first speaker 140-1 is located in the center of the instrument panel, sub-first speaker 140-2 is located at the left end of the instrument panel, sub-first speaker 140-3 is located at the right end of the instrument panel, sub-first speaker 140-4 is located at the bottom of the left door, and sub-first speaker 140-5 is located at the bottom of the right door. When the first control device 170 outputs the voice of user U to the first speaker 140, for example, it outputs sound at a similar volume from sub-first speakers 140-2 and 140-4, and turns off the other sub-first speakers, thereby localizing the sound image so that the voice sounds to the passenger seat S2 for the occupant P seated in the driver's seat S1. Furthermore, the method of sound image localization is not limited to volume adjustment, but may also be performed by shifting the phase of the sound output by each sub-first speaker. For example, to localize the sound image so that it sounds like it is coming from the left, the timing of outputting sound from the left sub-first speaker should be slightly earlier than the timing of outputting the same sound from the right sub-first speaker.

[0045] Furthermore, when the first control device 170 outputs the voice of user U to the first speaker 140, it may direct the sound image to the occupant P so that the voice is heard from a position corresponding to the height of user U's head on the passenger seat S2, and then output the voice emitted by user U to the first speaker 140. In this case, the first speaker 140 needs to have multiple child first speakers 140-k (where k is a number of natural numbers) at different heights.

[0046] Figure 5 is a configuration diagram of the second device 200. The second device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a mobile image display device 250, an HMI 260, and a second control device 270. The detection device 230 includes, for example, a direction detection device 232, a head position detection device 234, a motion sensor 236, and a gaze detection device 238.

[0047] The second communication device 210 is a communication interface for communicating with the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0048] The second microphone 220 collects the voice emitted by user U. The voice collected by the second microphone 220 is transmitted to the first communication device 110 by the second communication device 210 via the second control device 270, for example.

[0049] The direction detection device 232 is a device for detecting the direction of direction. The direction of direction is the orientation of the user U's face or gaze, or an orientation based on both. Hereinafter, the direction of direction will be assumed to be an angle in the horizontal plane, that is, an angle without a vertical component, but the direction of direction may also be an angle that includes a vertical component. The direction detection device 232 may include a physical sensor (e.g., an accelerometer, a gyroscope, etc.) attached to the VR goggles described later, or it may be an infrared sensor that detects multiple positions of the user U's head, or a camera that images the user U's head. In any case, the second control device 270 calculates the direction of direction based on the information input from the direction detection device 232. Since various technologies for this are publicly known, a detailed explanation will be omitted.

[0050] The head position detection device 234 is a device for detecting the position (height) of the user U's head. For example, one or more infrared sensors or optical sensors installed around the chair where the user U is seated can be used as the head position detection device 234. In this case, the second control device 270 detects the position of the user U's head based on the presence or absence of detection signals from one or more infrared sensors or optical sensors. Alternatively, the head position detection device 234 may be an accelerometer attached to the VR goggles. In this case, the second control device 270 detects the position of the user U's head by integrating the output of the accelerometer minus the acceleration due to gravity. The head position information obtained in this way is provided to the second control device 270 as height information. The user's head position may also be obtained based on the user U's operation of the HMI 260. For example, the user U may input their height numerically into the HMI 260, or they may input their height using a dial switch included in the HMI 260. In these cases, the head position, i.e., height information, is calculated from the height. Furthermore, user U may input discrete values ​​such as body size: large / medium / small into the HMI260 instead of continuous values. In this case, height information will be obtained based on the information indicating body size. Alternatively, the user's head height may be obtained simply based on the average adult body size (which may be separated by gender) without specifically obtaining the user's head height.

[0051] The motion sensor 236 is a device for recognizing gestures made by user U. For example, a camera that captures images of user U's upper body is used as the motion sensor 236. In this case, the second control device extracts characteristic points of user U's body (fingertips, wrists, elbows, etc.) from the images captured by the camera and recognizes user U's gestures based on the movement of these characteristic points.

[0052] The gaze detection device 238 is a device for detecting gaze. Gaze refers to the direction of the user U's gaze. The gaze detection device 238 may also be a camera that captures gaze images and is attached to the VR goggles described later. The second control device 270 derives the gaze based on the information input from the gaze detection device 238. Since various technologies for this are publicly known, a detailed explanation will be omitted.

[0053] The second speaker 240 outputs the voice emitted by occupant P, which is acquired via the second communication device 210. The second speaker 240 has a function, for example, to change the direction from which the voice is heard. The second control device 270 causes the second speaker to output voice so that the voice is heard by the user U from the position of occupant P as seen from the passenger seat S2. The second speaker 240 may include multiple child second speakers 240-n (where n is a number of natural numbers), and sound localization may be achieved by the second control device 270 adjusting the volume of each child second speaker 240-n, or if headphones are attached to the VR goggles, sound localization may be achieved by utilizing the function of the headphones.

[0054] The mobile image display device 250 displays an image from the camera unit 130 (which may be an image that has undergone the aforementioned stitching process, and will be referred to as a mobile image below) that corresponds to the direction of orientation as seen from the passenger seat. Figure 6 is a diagram illustrating the image corresponding to the direction of orientation. In this example, the VR goggles 255 include a direction detection device 232, a physical sensor as a head position detection device 234, and the mobile image display device 250. The second control device 270, for example, uses the center of the user U's head or the center of the VR goggles 255 as Ω, and detects the direction the VR goggles 255 are facing as the direction of orientation φ, using a pre-calibrated direction as the reference direction. Since various methods for this function are already publicly known, a detailed explanation is omitted.

[0055] The mobile object image display device 250 displays image A2, which is part of the mobile object image A1 (which has an angle of approximately 240 degrees in the figure, but the field of view may be expanded by the merging process as described above), towards the user U.

[0056] Returning to Figure 5, the HMI260 is a touch panel, voice response device (agent device), or the switches mentioned above. The HMI260 receives various instructions from user U to the second device 200.

[0057] The second control device 270 includes, for example, a processor such as a CPU, and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the second device 200 by executing the set of instructions. Note that all the functions of the second device 200 shown in Figure 5 may be integrated into the VR goggles.

[0058] <Functional configuration of the first control device 170 and the second control device 270> The functional configurations of the first control device 170 and the second control device 270 will be described below. In the first and second examples shown below, the first control device 170 and the second control device 270 are mainly responsible for generating and transmitting information, while the management server 300 manages the matching of simulated passengers P and users U.

[0059] [Example 1] Figure 7 shows a first example of the functional configuration of the first control device 170 and the second control device 270. In this first example, the first control device 170 includes a matching request / acceptance unit 171, an audio output control unit 172, an image transmission unit 173, and an onboard device linkage unit 174. The second control device 270 includes a matching request / acceptance unit 271, an audio output control unit 272, a direction detection unit 273, a head position detection unit 274, a gesture input detection unit 275, an image editing unit 276, a presentation unit 277, a reception unit 278, a user information acquisition unit 279, a gaze detection unit 280, a creation unit 281, and a moving object behavior detection unit 282. These functional units are realized, for example, by a processor such as a CPU executing a program (set of instructions). Some or all of these components may be implemented by hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or by the collaboration of software and hardware.

[0060] The matching request / acceptance unit 171 uses the HMI 160 to receive input of a matching request (simulated boarding request) from the occupant P and transmits it to the management server 300, and also uses the HMI 160 to receive input of acceptance of a matching request received from the management server 300 and transmits it to the management server 300. The matching request / acceptance unit 171 controls the first communication device 110 to communicate with the second device 200 of user U, with whom a match has been made. In addition, the matching request / acceptance unit 171 may, for example, accept the setting of information provision permission information indicating whether or not to allow the occupant P to provide information when accepting a matching request obtained from the management server 300. As described above, the audio output control unit 172 controls the first speaker 140.

[0061] After communication with the second device 200 begins, the image transmission unit 173 uses the first communication device 110 to transmit the mobile image A1 to the second device 200. The mobile image A1 may include, for example, an indoor image captured by the indoor camera 132 and / or an outdoor image captured by the outdoor camera 134.

[0062] The onboard equipment linkage unit 174 controls the controlled equipment 190 based on instruction signals input from the second device 200. The onboard equipment linkage unit 174 acquires operating information of the mobile unit M from the controlled equipment 190 and transmits it to the second device 200 from the first communication device 110. The onboard equipment linkage unit 174 may also perform control if the crew member P has authorized user U to control the controlled equipment 190.

[0063] The matching request / acceptance unit 271 uses the HMI 260 to receive matching requests (simulated ride requests) from user U and transmits them to the management server 300, and also uses the HMI 260 to receive acceptance input for matching requests received from the management server 300 and transmits it to the management server 300. The matching request / acceptance unit 271 controls the second communication device 210 to communicate with the first device 100 of the crew member P with whom a match has been made. The matching request / acceptance unit 271 may also, for example, present to user U what information will be provided based on the matching result (request result) obtained from the management server 300, allow user U to confirm, and then accept instructions from user U on whether or not to proceed with the simulated ride. As described above, the audio output control unit 272 controls the second speaker 240.

[0064] The direction detection unit 273 detects the direction φ based on the output of the direction detection device 232. The head position detection unit 274 detects the height of the user U's head based on the output of the head position detection device 234. The head position may be expressed as three-dimensional coordinates, or the head height may simply be detected as the head position. The gesture input detection unit 275 detects the gesture input of user U based on the output of the motion sensor 236.

[0065] The image editing unit 276 performs a process to extract, for example, image A2 from the moving object image A1, which corresponds to the direction of direction φ as seen from the passenger seat S2 (Figure 6). In the example in Figure 6, the direction of direction φ for movement is shown to the left and right of the user U's head, but the process may also be performed to extract an image corresponding to the direction of direction of movement in the vertical direction of the head. The display unit 277 displays (presents) the image A2 extracted by the image editing unit 276 on the mobile image display device 250. At this time, the image editing unit 276 may also display on the mobile image display device 250 an image corresponding to the directional direction φ as viewed from the height indicated by the height information of the user U's head. For example, the image editing unit 276 switches between displaying an indoor image or an outdoor image depending on the distance from the reference position of the user U's head (for example, the amount of upward movement or the amount of lateral movement). The display unit 277 also acquires an output signal from another second device 200 from the second communication device 210 and displays an icon on the mobile image display device 250 based on the position information contained in the acquired output signal.

[0066] The reception unit 278 accepts user input for the image A2 presented on the mobile image display device 250 by the presentation unit 277. The user information acquisition unit 279 acquires user information of users who use other second devices 200 among the multiple second devices 200. An example of user information is information for communicating with each user of the other second devices 200. The gaze detection unit 280 detects the direction of gaze based on the output of the gaze detection device 238. The display unit 277 identifies the position on the image (video) displayed on the mobile image display device 250 and displays an icon at the identified position. For example, when a user inputs an operation to the HMI 260, the display unit 277 identifies the position on the image (video) displayed on the mobile image display device 250 instructed by the operation input and displays an icon at the identified position. The creation unit 281 creates an output signal containing the position information identified by the display unit 277 and transmits it from the second communication device 210 to another second device 200. For example, when a user is viewing an image (video), they use the HMI 260 they are holding to aim at an arbitrary position on the image (video) and press an input button to perform an operation input. Specifically, the user uses the HMI 260 they are holding to aim at a symbol displayed on the image (video) at an arbitrary position and presses an input button to perform an operation input.

[0067] Here, the display unit 277 may identify a display area on the mobile image display device 250 based on the position targeted by the HMI 260, and display an icon in the identified display area. For example, the display unit 277 may display an icon in the center of the display area. Furthermore, the creation unit 281 may create notification information including the created output signal and transmit it from the second communication device 210 to the first device 100. This allows the occupant P to be notified of the result of the user's reaction, enabling the occupant P to understand the user's emotions. Here, it may be possible to instruct whether or not to notify the occupant P of the result of the user's reaction through an operation input to the HMI 260. This prevents notifications that would interfere with the occupant P's driving.

[0068] Furthermore, for example, the display unit 277 identifies a position on the moving object image display device 250 based on the gaze direction detected by the gaze detection unit 280, and displays an icon on the object displayed at the identified position. The creation unit 281 creates an output signal containing the position information identified by the display unit 277 and transmits it from the second communication device 210 to another second device 200.

[0069] Here, the presentation unit 277 may identify a display area on the moving object image display device 250 based on the gaze direction detected by the gaze detection unit 280, and display an icon on the object displayed in the identified display area. For example, the presentation unit 277 may display an icon in the center of the display area. An example of a display area is a bounding box. In this case, the creation unit 281 creates an output signal containing information indicating the display area identified by the presentation unit 277 and transmits it to the other second device 200. As long as the moving object M (vehicle) is traveling along the road, the shape and color of the object will not change significantly. Unless there are right or left turns, the recognized object will appear to be cut off from the screen after approaching the vehicle. The display unit 277 may change the position of the specified position or display area based on the movement of the moving object M. The display unit 277 may also continue to display the icon in the same position based on the positional relationship between the specified position and the display area.

[0070] The driving information of the mobile body M transmitted by the first device 100 is received by the second communication device 210. The mobile body behavior detection unit 282 acquires the driving information of the mobile body M from the second communication device 210 and detects whether the mobile body M has turned right or left based on the acquired driving information of the mobile body M. The display unit 277 changes the content of the output result if, after displaying the icon, the moving object behavior detection unit 282 detects that the moving object M has turned right or left. For example, the display unit 277 displays the icon differently than when the moving object M has not turned left or right. For example, the display unit 277 displays the icon with a dashed line. For example, the display unit 277 determines whether the icon continues to be displayed at a specific position based on the moving image displayed on the moving object image display device 250, and if it determines that the icon has stopped being displayed, it displays the icon with a dashed line.

[0071] [Example 2] Figure 8 shows a second example of the functional configuration of the first control device 170 and the second control device 270. Compared to the first example in Figure 7, the first control device 170 includes an image editing unit 175, while the second control device 270 does not include an image editing unit 276 but includes a directional transmission unit 283. As the other components basically have the same functions as in the first example, further explanation is omitted. The direction transmitting unit 283 transmits the direction φ detected by the direction detection unit 273 to the first device 100 using the second communication device 210. The image editing unit 175 performs the process of extracting image A2 from the moving image A1 that corresponds to the directional direction φ (transmitted from the second device 200) as seen from the passenger seat (Figure 6). At this time, the image editing unit 175 may also obtain the height information of the user U's head from the second device 200 and perform the process of extracting image A2 that corresponds to the directional direction φ as seen from the height indicated by the height information.

[0072] In the second example, the image transmission unit 173 uses the first communication device 110 to transmit the image A2 extracted by the image editing unit 175 to the second device 200. The display unit 277 then displays the image A2 transmitted from the first device 100 on the mobile image display device 250.

[0073] <Processing related to simulated rides> Next, in this embodiment, a series of processes performed by the information processing system 1 when user U virtually boards the mobile vehicle M (or when crew member P virtually boards user U) will be specifically described. Figure 9 is a sequence diagram showing an example of the processes performed by the information processing system 1. In the example in Figure 9, the processes in the first device 100 mounted on the mobile vehicle M while crew member P is on board, the second device 200 used by user U, and the management server 300 are shown. Furthermore, it is assumed that user U and crew member P are registered in advance with the management server 300 and registered in user data 360 in order to receive services from the information processing system 1.

[0074] In the example shown in Figure 9, the second device 200 generates a request for a simulated ride (simulated ride request) (step S100) and sends the generated simulated ride request to the management server 300 (step S102). The simulated ride request may include, for example, information specifying a passenger P in advance, or information specifying the current location, destination, or surrounding conditions of the moving vehicle M. More specific examples of simulated ride requests include requests such as "I want to simulate ride in a vehicle driven by person A" or "I want to simulate ride in a vehicle traveling on a road along the coast." These are examples of the matching requests described above.

[0075] The matching processing unit 320 of the management server 300 receives a simulated experience request from the second device 200 and performs matching processing based on the user data 360 (step S104). Then, if there is one or more matched mobile bodies M, the provisioning unit 330 generates inquiry information to ask whether or not to allow all mobile bodies M to simulate riding (step S106), and transmits the generated information to the first device 100 of the matched mobile bodies M (step S108).

[0076] The first device 100 displays the inquiry information sent from the management server 300 on the screen (step S110). Figure 10 is a diagram showing an example of image IM10 showing the inquiry information. Note that the display content and layout of image IM10 are not limited to this. The same applies to the descriptions of other images that follow. Image IM10 includes, for example, a user information display area AR11, a availability selection area AR12, and a switch display area AR13 as an inquiry screen. The user information display area AR11 displays information about user U who has made a simulated boarding request. The information about user U may be a user ID, or other information that allows the crew member P to identify user U.

[0077] The information provision selection area AR12 displays images for selecting the content of the information that occupant P will provide to user U, for each type of information. In the example in Figure 10, a screen is displayed for selecting whether or not to allow occupant P to provide information regarding interior images, exterior images, phone calls (voice), and navigation information (information provided by the navigation device). Phone calls (voice) is an example of "interior voice information." For example, if occupant P is willing to provide phone calls and images to user U, but does not want user U to know the specific destination, occupant P can allow the provision of phone calls, interior images, and exterior images, but refuse to provide navigation information. This allows occupant P to head to the destination secretly from user U, even though user U could easily find out the destination from the navigation information if user U were actually riding in the vehicle M. If navigation information is not provided during a simulated ride, user U can be surprised and experience a unique surprise that only a simulated ride can offer. Also, for example, if user U is not very familiar with the occupant P, refusing to provide phone calls (voice) allows the driver to drive without conversation. The information to be selected for provision is not limited to the example in Figure 10, and may include, for example, information to select whether or not user U can operate the navigation system or other controlled devices 190, or information to select whether or not to provide images including occupant P. The availability of the information to be provided, selected by the provision availability selection area AR12, may be changed during the simulated ride. The switch display area AR13 includes, for example, icon IC11 and icon IC12. Icon IC11 is a switch that accepts permission for occupant P to allow user U to simulate ride. Conversely, icon IC12 is a switch that accepts permission for occupant P to refuse user U's simulated ride.

[0078] Returning to Figure 9, the first device 100 receives the user U's availability setting via the image IM10 shown in Figure 10 (step S112) and transmits the received availability setting information to the management server 300 (step S114). Note that the processing in steps S110 to S112 may be displayed on the screen when the mobile body M stops in order to ensure safety during operation, or instead of (or in addition to) a screen display, the system may notify the user that a simulated ride request has been made or accept the availability setting using voice. The provision unit 330 of the management server 300 generates a request result (response result) based on the availability setting information obtained from the first device 100 (step S116) and transmits information indicating the generated request result to the second device 200 (step S118).

[0079] The second device 200 receives the request result from the management server 300 and displays information indicating the received request result on the screen (step S120). Figure 11 shows an example of an image IM20 showing the request result. Image IM20 includes, for example, a passenger information display area AR21, a provided information list display area AR22, and a switch display area AR23 as a request result screen. This image IM20 may be generated for each passenger P (mobile body M) who is permitted to simulate riding, and an image showing a list of request results may also be displayed.

[0080] The passenger information display area AR21 displays information about passenger P who has authorized the simulated ride request. The information about passenger P may be a passenger ID, or other information that allows user U to identify passenger P. The provided information list display area AR22 displays a list of information authorized by passenger P. The provided information list display area AR22 may also display the fee (service usage fee) based on the authorized information. The switch display area AR23 includes, for example, icons IC21 to IC23. Icon IC21 is a switch that accepts a simulated ride on passenger P's mobile body M, as displayed in the passenger information display area AR21, using the provided information displayed in the provided information list display area AR22. Conversely, icon IC22 is a switch that accepts not to perform a simulated ride. Icon IC23 is a switch that accepts searching for other passengers (mobile bodies). When searching for other passengers, users may select from other passengers included in the request results, or they may input other request conditions and have the management server 300 perform the matching process again.

[0081] Returning to Figure 9, the second device 200 receives a selection instruction from user U via the image IM20 shown in Figure 11 (step S122) and transmits the received instruction information to the management server 300 (step S124). In the following description, it will be assumed that user U has given an instruction to virtually board the mobile body M of occupant P. Based on the instruction from the second device 200, the management server 300 manages the communication status between the first device 100 and the second device (step S126) and transmits information to the first device 100 and the second device 200 indicating that a virtual boarding will be performed (steps S128, S130). If user U chooses not to virtually board during the processing in steps S126 to S130, the provision unit 330 of the management server 300 transmits information to the first device 100 to notify user U.

[0082] Next, the first device 100 and the second device 200 communicate with each other to perform a simulated ride (step S132), and when the simulated ride is completed, they send information to the management server 300 indicating that communication has ended (steps S134, S136). In the process of step S132, the second device 200 may notify the passenger P from the user U before the information from the first device 100 is provided to the second device 200. This allows the user U to authorize communication at their own timing and start providing information such as phone calls or video streaming. Next, the fare management unit 340 of the management server 300 performs settlement processing for the charges related to the use of each service of the first device 100 and the second device 200. Specifically, the fare management unit 340 determines different charges for information provision based on at least one of the following: the type of passenger P, user U, and mobile body M that provide the information, the type of information that can be provided to the user U, and the amount of information. For example, if the passenger P or user U is a celebrity, the fare may be set higher than the normal fare. Similarly, if the vehicle M is a new car or a rare vehicle, the fare may also be set higher than the normal fare. The fare management unit 340 then performs the settlement process corresponding to the set fare. This completes the processing of this sequence. Note that the processing in step S112 shown in Figure 9 (service availability setting process) may be set in advance by the passenger P for each registered user U.

[0083] <Processing related to icon display> Next, in an embodiment, a series of processes performed by the information processing system 1 when user U1 and user U2 virtually ride on the mobile body M1, and user U1 makes an operation input when image A is displayed on the mobile body image display device 250 of the second device 200-1, thereby causing an icon to be displayed on the mobile body image display device 250 of the second device 200-2, will be specifically described. Figure 12A is a sequence diagram showing a first example of processing performed by the information processing system 1. The first device 100, the second devices 200-1 and 200-2, and the management server 300 communicate to perform a simulated ride (step S200). In Figure 12A, the first device 100 has already acquired user information for communication with the second device 200.

[0084] Each of the second device 200-1 and the second device 200-2 presents predetermined information, such as image A2, to the mobile image display device 250 (step S202). The presentation units 277 of each of the second device 200-1 and the second device 200-2 cause the image A2 extracted by the image editing unit 276 to be displayed (presented) on the mobile image display device 250. The second device 200-1 receives an operation input from user U-1 (step S204), creates an output signal based on the received operation input, and transmits it to the second device 200-2 (step S206). The operation input may be, for example, a tap operation performed by user U using HMI 260 on at least a portion of the area of ​​the moving image display device 250. For example, user U taps a predetermined location, such as a place to be viewed from a remote location, included in the moving image display device 250. HMI 260 acquires the input of the tap operation performed by user U. Based on the tap operation acquired by HMI 260, the creation unit 281 creates an output signal including the tapped location information and transmits it to the second device 200-2.

[0085] The display unit 277 of the second device 200-2 displays an icon on the mobile image display device 250 based on the output signal from the second device 200-1 (step S208). For example, the display unit 277 of the second device 200-2 acquires the output signal from the second device 200-1 from the second control device 270 and acquires the position information contained in the acquired output signal. Based on the acquired position information, the display unit 277 of the second device 200-2 displays an icon on image A displayed on the mobile image display device 250.

[0086] Figure 12B is a sequence diagram showing a second example of processing performed by the information processing system 1. The first device 100, the second devices 200-1 and 200-2, and the management server 300 communicate to perform a simulated ride (step S300).

[0087] Each of the second device 200-1 and the second device 200-2 presents predetermined information, such as image A2, to the mobile image display device 250 (step S302). The presentation units 277 of each of the second device 200-1 and the second device 200-2 cause the image A2 extracted by the image editing unit 276 to be displayed (presented) on the mobile image display device 250.

[0088] The second device 200-1 receives operation input from user U-1 (step S304), acquires user information of user U-2 from the second device 200-2 (step S306), creates an output signal based on the received operation input and user information of user U-2, and transmits it to the second device 200-2 (step S308). The operation input may be, for example, a tap operation performed by user U using the HMI 260 on at least a portion of the area of ​​the moving image display device 250. For example, user U taps a predetermined location, such as a place to be viewed from a remote location, included in the moving image display device 250. The HMI 260 acquires the input of the tap operation performed by user U. The user information acquisition unit 279 acquires user information of user U2 who is using the second device 200-2. An example of user information of user U2 is information for communication with the second device 200-2. The creation unit 281 creates an output signal including the tapped location information based on the tap operation acquired by the HMI 260 and the user information of user U2, and transmits it to the second device 200-2. Step S310 is the same as step S208 in Figure 12A, so its explanation is omitted.

[0089] Figure 12C is a sequence diagram showing a third example of processing performed by the information processing system 1. The first device 100, the second devices 200-1 and 200-2, and the management server 300 communicate to perform a simulated ride (step S400).

[0090] Each of the second device 200-1 and the second device 200-2 presents predetermined information, such as image A2, to the mobile image display device 250 (step S402). The presentation units 277 of each of the second device 200-1 and the second device 200-2 cause the image A2 extracted by the image editing unit 276 to be displayed (presented) on the mobile image display device 250.

[0091] The second device 200-1 receives operation input from user U-1 (step S404), detects the direction of user U-1's gaze (step S406), creates an output signal based on the received operation input and the direction of user U-1's gaze, and transmits it to the second device 200-2 (step S408). The operation input may be, for example, a button press operation or gesture input performed by user U using the HMI 260 on at least a portion of the area of ​​the moving image display device 250. For example, user U performs a button press operation or gesture input while fixating on a predetermined location, such as a place to be viewed from a distance, included in the moving image display device 250. The gaze detection unit 280 detects the direction of user U1's gaze and acquires predetermined location information. The HMI 260 acquires the button press operation or gesture input performed by user U. The creation unit 281 creates an output signal including position information based on predetermined position information acquired by the gaze detection unit 280 and button press operations or gesture inputs acquired by the HMI 260, and transmits it to the second device 200-2. Step S410 is the same as step S208 in Figure 12A, so its explanation is omitted. Figures 12A to 12C illustrate an example where the information processing system 1 has two second devices 200, but the same applies to cases where the information processing system 1 has three or more second devices 200.

[0092] <Information provided during simulated ride> Next, we will explain the information provided to user U during the simulated ride. The following examples show the information provided in the process of step S208 shown in Figure 12A, the process of step S310 shown in Figure 12B, and the process of step S410 shown in Figure 12C. For example, the output manner of the information provided to the second device 200 differs depending on the actions of user U and the type or amount of information that can be provided to user U. In the following examples, it is assumed that occupant P is authorized to provide interior images, exterior images, and telephone (voice).

[0093] <Example of provided information: Interior image> Figure 13A is a diagram illustrating an example of the information provided. In the example in Figure 13A, a user U is seated on a chair CH in a location other than the mobile vehicle M (for example, a room in one's home), and an example of an image IM30 provided by the second device 200 (integrated with VR goggles in the example in Figure 13A) used by the user U is shown. Here, as an example, the case where the second device 200 is integrated with VR goggles is described, but it may also be a head-mounted display or the like. The image IM30 includes a heart-shaped icon IC25. Note that the state of user U shown in the example in Figure 13A (seated on chair CH) represents the initial state immediately after the simulated ride is performed, and the first device 100 and the second device 200 recognize the position of user U at this time (reference posture) as the reference position. The reference position is the position where user U is assumed to be seated in the passenger seat S2. The reference position may be corrected by the first control device 170 driving the seat drive device based on the user U's height or sitting height information registered in the user information of user data 360, so that the height of the indoor camera 132 matches the user U's sitting height (so-called zero-point correction). The reference position may also be corrected by correction instructions from user U who has viewed the image IM30. The reference position may also include orientation information when user U is seated in the chair CH and facing forward.

[0094] For example, when the user U (second device 200) is seated in chair CH and facing any direction from their reference position, an image captured by the interior camera 132 of the first device 100 is provided as an interior image IM30, corresponding to that direction. This allows user U to obtain an interior image of any direction as seen from the passenger seat S2. In addition, audio information of the occupant P's voice and sounds inside and outside the mobile vehicle M, acquired by the first microphone 120, is provided to user U. By using the image IM30 and audio information to converse with occupant P, user U can enjoy a conversation as if they were actually riding in the mobile vehicle M.

[0095] In the information processing system 1, it was explained that user U can view any direction from the passenger seat S2. However, restrictions may be placed on the directions that user U can view, for example, through arrangements during matching or settings for whether or not occupant P can provide information. For example, occupant P may want to provide the scenery in the direction of travel of the mobile vehicle M, or the scenery opposite the driver's seat S1, but not their own image. This is intended to address situations where occupant P and user U are not family or friends, and the user wants to confirm the driving feel of the mobile vehicle M or view a desired city landscape. In this case, the matching processing unit 320 or the provisioning unit 330 of the management server 300 sets such restrictions based on the matching results, and the first control device 170 or the second control device 270 masks the angle range that is not to be viewed, or corrects the direction of view φ so that it does not point in the restricted direction, according to the settings. Furthermore, since information regarding such restrictions concerns the privacy of occupant P, it may also be set on the first device 100 side. Furthermore, any information that has been refused may be provided after the consent of crew member P, provided that user U pays an additional fee.

[0096] Furthermore, the mobile image display device 250 may replace the portion of the image captured by the camera unit 130 that shows a predetermined item inside the room of the mobile object M with a computer-generated image (CG image) and display it. The area OB1 shown in Figure 13A is a display device that performs navigation display and the like, and is an example of a "predetermined item". If the display screen of the display device is displayed as is, the image may be blurry or visibility may be reduced due to light reflection. For this reason, the mobile image display device 250 may acquire data for configuring the display screen of the display device, or image data drawn by computer processing in the mobile object M, from the first device 100, and display an image redrawn from the data by computer processing, or the acquired image data, embedded in the image (edited version) captured by the camera unit 130. In this case, the location of the item inside the room of the mobile body M, which is the predetermined item, is shared in advance between the first device 100 and the second device 200. The display unit 277 determines whether the predetermined item is included in the image to be displayed on the mobile body image display device 250, for example, based on the direction of orientation φ, and if it is determined that it is included, it replaces the image as described above. The "predetermined item" may also be the head or face of the occupant P. In that case, the CG image of an avatar or the like may be changed according to the display of the occupant P. At least a part of the above-mentioned image data (CG image), etc., may be provided from the information database of the management server 300.

[0097] Figure 13B is a diagram illustrating an example of the information provided. In the second communication device 210, the display unit 277 changes the display position of the icon based on the speed of the moving object M. For example, the display unit 277 recognizes the position from which the icon was output and displays it so that it flows backward according to the speed of the moving object M. For example, the display unit 277 obtains information indicating the speed of the moving object M from the first device 100 and changes the position from icon IC25-1 to icon IC25-2 based on the acquired speed of the moving object M. By changing the position from icon IC25-1 to icon IC25-2 and providing a dynamic visual effect, a more realistic and engaging experience can be provided to the user U. By adjusting the movement of icon IC25 according to the speed of the vehicle, visual consistency can be maintained for the user U.

[0098] Figure 13C is a diagram illustrating an example of the information provided. In the second device 200, if the mobile object behavior detection unit 282 detects that the mobile object M has made a left turn or a right turn after a tap operation by the user U, the display unit 277 displays an icon differently than when the mobile object M has made a left turn or a right turn. For example, the display unit 277 displays the icon with a dashed line. For example, the display unit 277 determines, based on the moving image displayed on the mobile object image display device 250, whether at least a portion of the moving image that was tapped by the user U is still being displayed, and if it determines that it is no longer being displayed, it displays the icon with a dashed line. Here, the display unit 277 may determine whether the display area (bounding box) specified on the moving image display device 250 based on the gaze direction detected by the gaze detection unit 280 is still being displayed, and if it determines that it is no longer being displayed, it may display the icon with a dashed line. Furthermore, the display unit 277 may display the icon with a dashed line and also display information indicating the object that was displayed. In Figure 13C, the icon IC26 is displayed with a dashed line, and the information indicating the object that was displayed (tree, building, utility pole, sea, etc.) is displayed as "A good tree was visible."

[0099] In the embodiment described above, the second device 200 may be configured to accept an operation to cancel an operation input from user U after receiving an operation input from user U. For example, while an icon is displayed in at least a portion of the area of ​​the moving image display device 250, user U may perform a predetermined cancel operation input using the HMI 260. For example, user U may aim the HMI 260 at the icon displayed on the moving image display device 250 and perform the cancel operation. Alternatively, user U may perform the cancel operation while gazing at the icon displayed on the moving image display device 250. The gaze detection unit 280 detects the direction of user U1's gaze and acquires the icon's position information. The HMI 260 acquires the cancel operation performed by user U. The creation unit 281 creates a cancel signal including position information based on the icon's position information acquired by the gaze detection unit 280 and the cancel operation input acquired by the HMI 260, and transmits it to the other second device. The display unit 277 of the other device deletes the icon based on the cancellation signal from the second device 200. For example, the display unit 277 of the other second device receives the cancellation signal from the second device 200 from the second control device 270 and obtains the position information included in the obtained output signal. Based on the obtained position information, the display unit 277 of the other second device deletes the icon displayed on image A presented on the mobile image display device 250. In the embodiments described above, some of the second devices 200 may include a display unit capable of displaying images. In that case, the second device 200 may be a screen (such as a television) installed at home, in addition to VR goggles. A second device 200 such as a screen may or may not have a reception unit for receiving operation inputs. Even if it does not have a reception unit, it may display the results of operation inputs received by the reception unit of another second device 200.

[0100] <Variation> In the information processing system 1 described above, when the second device 200 receives an operation input from user U for image A2, it detects an arbitrary position in image A based on the operation input and acquires POI information for the detected position. The second device 200 displays the acquired POI information and icon on the mobile image display device 250. The second device 200 creates an output signal containing the arbitrary position information and POI information of image A detected based on the operation input and transmits it to another second device 200. The second device 200 displays the icon and POI information on the mobile image display device 250 based on the position information and POI information contained in the output signal from the other second device 200.

[0101] Figure 14 shows a third example of the functional configuration of the first control device 170 and the second control device 270. Compared to the first example in Figure 7, the second control device 270 differs in that it includes a POI acquisition unit 284, a selection unit 285, and a voice information acquisition unit 286. The other components basically have the same functions as in the first example, so a further explanation is omitted. The POI acquisition unit 284, when it receives user input regarding image A2 presented on the mobile image display device 250 by the presentation unit 277, identifies a position on image A2 based on the input and acquires POI information based on the information indicating the identified position. Alternatively, it may acquire information indicating the gaze position of user U from the gaze detection unit 280 and acquire POI information based on the acquired information indicating the gaze position of user U. For example, the POI acquisition unit 284 may acquire POI information from the management server 300. The presentation unit 277 acquires POI information from the POI acquisition unit 284 and displays (presents) image A2 on the mobile image display device 250, displaying the POI information in addition to the icon.

[0102] The selection unit 285 selects the POI information to be displayed along with the icon when multiple POI pieces of information are acquired by the POI acquisition unit 284. For example, the selection unit 285 may select the POI information closest to the foreground (closest to the moving object M) from among the multiple POI pieces of information. In this case, the presentation unit 277 acquires the POI information from the selection unit 285 and displays (presents) image A2 on the moving object image display device 250, displaying the POI information in addition to the icon. Here, the selection unit 285 may select any POI information from among the multiple POI pieces of information. The voice information acquisition unit 286 acquires voice information based on the voice collected by the second microphone 220.

[0103] The display unit 277 identifies the position on the image (video) displayed on the mobile image display device 250 as instructed by the operation input, and outputs sound based on the sound information acquired by the sound information acquisition unit 286 when displaying an icon at the identified position. The creation unit 281 creates an output signal including the position information identified by the display unit 277 and the sound information acquired by the sound information acquisition unit 286, and transmits it from the second communication device 210 to another second device 200. Here, when the presentation unit 277 displays (presents) image A2 on the mobile image display device 250, it may also display POI information in addition to the icon, and play audio based on audio information acquired by the audio information acquisition unit 286 in relation to the POI information. The creation unit 281 may create an output signal including the location information identified by the presentation unit 277, the POI information acquired by the POI acquisition unit 284, and the audio information acquired by the audio information acquisition unit 286, and transmit it from the second communication device 210 to another second device 200.

[0104] Figure 15A is a sequence diagram showing a fourth example of processing performed by the information processing system 1. The first device 100, the second devices 200-1 and 200-2, and the management server 300 communicate to perform a simulated ride (step S500). Each of the second device 200-1 and the second device 200-2 presents predetermined information, such as image A2, to the mobile image display device 250 (step S502). The presentation units 277 of each of the second device 200-1 and the second device 200-2 cause the image A2 extracted by the image editing unit 276 to be displayed (presented) on the mobile image display device 250.

[0105] The second device 200-1 receives operation input from user U-1 (step S504), detects the direction of user U-1's gaze (step S506), and acquires POI information based on the position of user U-1's gaze (step S508). The operation input may be, for example, a button press operation or gesture input performed by user U using the HMI 260 on at least a portion of the area of ​​the moving image display device 250. For example, user U performs a button press operation or gesture input while gazing at a predetermined position, such as a place to be viewed from a distance, included in the moving image display device 250. The gaze detection unit 280 detects the direction of user U1's gaze and acquires predetermined position information. The POI acquisition unit 284 acquires information indicating the position of user U's gaze from the gaze detection unit 280 and acquires POI information based on the acquired information indicating the position of user U's gaze.

[0106] The second device 200-1 determines whether or not multiple POI pieces of information have been acquired (step S510). If multiple POI pieces of information have been acquired, the second device 200-1 selects the POI pieces of information to be displayed with an icon from among the multiple POI pieces of information (step S512), creates an output signal including information indicating the gaze position and the POI pieces of information, and transmits it to the second device 200-2 (step S514). The selection unit 285 selects the POI pieces of information to be displayed with an icon when multiple POI pieces of information have been acquired by the POI acquisition unit 284. For example, the selection unit 285 selects the POI pieces of information that are closest to the viewer (closest to the moving object M) from among the multiple POI pieces of information. The creation unit 281 creates an output signal including POI pieces of information and position information based on predetermined position information acquired by the gaze detection unit 280 and button press operations or gesture inputs acquired by the HMI 260, and transmits it to the second device 200-2.

[0107] The display unit 277 of the second device 200-2 displays icons and POI information on the mobile image display device 250 based on the output signal from the second device 200-1 (step S516). For example, the display unit 277 of the second device 200-2 acquires the output signal from the second device 200-1 from the second communication device 210 and acquires the location information and POI information included in the acquired output result. Based on the acquired location information, the display unit 277 of the second device 200-2 displays icons and POI information on image A displayed on the mobile image display device 250.

[0108] Figure 15B is a sequence diagram showing a fifth example of processing performed by the information processing system 1. Steps S600 to S606 and S610 to S614 can be explained by applying steps S500 to S512 in Figure 15A. In step S608, the voice information acquisition unit 286 of the second device 200-1 acquires voice information from the voice emitted by user U. In step S616, an output signal is created that includes information indicating the gaze position, POI information, and audio information acquired when the POI information was obtained, and this signal is transmitted to the second device 200-2. In step S618, the display unit 277 of the second device 200-2 displays icons and POI information on the mobile image display device 250 based on the output signal from the second device 200-1, and outputs sound from the second speaker 240 based on the sound information. For example, the display unit 277 of the second device 200-2 acquires the output signal from the second device 200-1 from the second communication device 210, and acquires location information, POI information, and sound information included in the acquired output result. Based on the acquired location information, the display unit 277 of the second device 200-2 displays icons and POI information on image A displayed on the mobile image display device 250, and outputs sound from the second speaker 240 based on the sound information.

[0109] Figure 16 is a diagram illustrating an example of the information provided. When a moving image is displayed on the mobile image display device 250, user U can aim at an arbitrary position by operating the HMI 260 and perform button presses or gesture inputs. Alternatively, user U may perform button presses or gesture inputs while fixating on a predetermined position included in the moving image displayed on the mobile image display device 250. The gaze detection unit 280 detects the direction of user U's gaze and acquires predetermined position information. The gaze detection unit 280 may convert the acquired predetermined position information into three-dimensional position information based on the map data and the position information of the mobile object M. In this case, the creation unit 281 creates an output signal that includes three-dimensional position information instead of predetermined position information. In the example shown in Figure 16, the icon IC27 is displayed on Tokyo Tower TT, which user U was fixating on, and POI information and the comment "You've liked Tokyo Tower" are displayed for the POI information.

[0110] <Summary> According to the information processing system 1 configured as described above, the system comprises a first device 100 mounted on a mobile body M, a plurality of second devices 200 each used by a plurality of users U at locations different from the mobile body M, an information acquisition unit 315 that acquires predetermined information transmitted by the first device 100, a provision unit 330 that provides the predetermined information acquired by the information acquisition unit 315 to the plurality of second devices 200, and a plurality of presentation units 277 that present the predetermined information provided to the plurality of second devices 200 by the provision unit 330. When an operation input is made by the first user U1 to one of the plurality of second devices 200, the presentation units 277 of the second devices 200 other than the first second device 200 present the output result based on the operation input made by the first second device 200. The convenience is improved because users of the second device 200 can smoothly communicate with each other what other users are interested in and what they think. An example of the second device 200 is a head-mounted display. Examples of predetermined information may include video data, image data, vehicle position data, map data of the surrounding area centered on the driving position, polygon data created as a 3D model of the area around the driving position, and video from an in-vehicle camera or a mounted smartphone.

[0111] The information processing system 1 includes a user information acquisition unit 279 that acquires user information of the user of the second device 200. The second device 200, used by the first user, further includes a creation unit 281 that creates an output signal for presenting output results based on the user information acquired by the user information acquisition unit 279. The second device can obtain user information from information provided by other second devices from the first device, and can communicate based on the obtained user information. The second device 200 can transmit signals without communicating with the first device (in-vehicle device), thus suppressing an unnecessary increase in processing load. Here, the second device may also present information via the first device 100 without obtaining information from other second devices.

[0112] In the information processing system 1, the second device includes a gaze detection unit 280 that detects the direction of the user's gaze. The presentation unit 277 presents output results based on the direction of the user's gaze detected by the gaze detection unit 280. By determining the position where output results are displayed based on the user's gaze, it is possible to improve ease of operation and intuitiveness by displaying the results in the desired location. The gaze detection unit 280 accurately tracks the head movements and position of user U by utilizing, for example, a 6DoF (6 Degrees of Freedom) sensor mounted on the second device 200. A camera with gaze detection capabilities may be used instead of, or in conjunction with, the 6DoF sensor. The display unit 277 may determine the user's position based on the gaze direction detected by the gaze detection unit 280 and display the output result at the determined position. Alternatively, the display unit 277 may determine the display area to which the user's gaze is directed based on the gaze direction detected by the gaze detection unit 280 and display the output result in the determined display area.

[0113] In the information processing system 1, the second device includes a POI acquisition unit 284 that acquires POI information. The POI acquisition unit 284 acquires POI information based on the location information of the first device 100 when the presentation unit 277 presents the output results. The presentation unit 277 presents the POI information acquired by the POI acquisition unit 284 along with the output results. By combining visual feedback with POI (Point of Interest) information, it becomes possible to provide users with more relevant information, improving their user experience. POI information includes information related to places and objects that users find visually interesting. It also generally includes information about popular facilities, scenic spots, and distinctive areas.

[0114] In the information processing system 1, the second device includes a POI acquisition unit 284 that acquires POI information. The POI acquisition unit 284 acquires POI information based on the user's gaze direction detected by the gaze detection unit 280. The presentation unit 277 determines the position for presenting the output result based on the POI information acquired by the POI acquisition unit 284. By identifying POI information based on the user's gaze and outputting results in the appropriate location, accuracy and relevance can be improved.

[0115] In the information processing system 1, the second device further includes a POI acquisition unit 284 that acquires POI information, and a selection unit 285 that, when multiple POI pieces of information are acquired by the POI acquisition unit 284, selects the POI information closest to the moving object from among the multiple POI pieces of information. The presentation unit 277 presents the POI information selected by the selection unit 285 to multiple second devices. When multiple POI (Point of Interest) pieces of information exist, the system can provide the user with the most relevant information, thereby preventing information overload and improving the user experience. The selection unit 285 may also provide an interface for the user to select the POI information they are interested in by displaying a POI selection screen.

[0116] In the information processing system 1, the display unit 277 changes the display position of the output result based on the speed of the moving object. By changing the display position of the output results based on the speed of the moving object, dynamic visual effects can be provided, resulting in a more realistic and engaging experience for the user.

[0117] The information processing system 1 further includes a mobile object behavior detection unit 282 that detects whether a mobile object M has turned right or left. After presenting the output result, the presentation unit 277 changes the content of the presented output result if the mobile object behavior detection unit 282 detects that the mobile object M has turned right or left. By changing the displayed output when the mobile object M turns right or left, it becomes possible to inform the user that there was something they were interested in before turning right or left, thus improving convenience.

[0118] The information processing system 1 further includes a creation unit 281 that, when an operation input is made by a first user to one of the multiple second devices, creates notification information including the output result for the first device based on the operation input made to the first second device. Since the first device can obtain notification information from the second device, it can notify the first user of their emotions based on the obtained notification information.

[0119] The information processing system 1 further includes a voice information acquisition unit 286 that acquires voice information based on the user's voice. The presentation unit 277 presents the output result based on the operation input and the voice information. By combining visual feedback with audio, it becomes possible to provide users with more relevant information, thereby improving their user experience.

[0120] The information processing system 1 further includes a voice information acquisition unit 286 that acquires voice information based on the user's voice. The presentation unit 277 presents the POI information and the voice information acquired by the voice information acquisition unit 286 in relation to the POI information, along with the output result. By combining visual feedback with POI (Point of Interest) information and audio, it becomes possible to provide users with more relevant information, thereby improving their user experience.

[0121] <Usage example> Information processing system 1 can be used in the following ways:

[0122] (A) A configuration in which the crew member P and user U are in a relationship such as family or friends, and the virtual drive is provided to user U. User U can converse with the crew member P about the scenery around the mobile vehicle M while viewing images.

[0123] (B) A configuration in which the crew member P is a general user and the user U is a provider of services such as directions and driving instruction. The user U can provide directions to places that are difficult to understand using a navigation device or that are not shown on a map, or provide instruction on driving operations, while observing the scenery around the mobile vehicle M.

[0124] (C) A configuration in which a crew member P is a celebrity and a user U is a regular user, and a commercially based virtual drive is provided to user U. In this case, multiple users U may be associated with one crew member P simultaneously, and for example, the transmission of voice from user U may be turned off. In addition, user U may be able to have a conversation with crew member P for a predetermined period of time by paying an additional fee.

[0125] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0126] 1. Information Processing System 100 1st device 110 First communication device 120 First microphone 125 External Sensors 130 Camera Unit 132 Indoor Cameras 134 Outdoor Camera 140 First speaker 150 User display device 160 HMI 170 First control device 190 Controlled Equipment 200 2nd device 210 Second communication device 220 Second microphone 230 detection device 232 Directional detection device 234 Head position detection device 236 Motion Sensors 238 Eye-tracking device 240 Second speaker 250 Mobile Image Display Device 260 HMI 270 Second control device 300 Management Servers 310 Communication equipment 315 Information Acquisition Department 320 Matching Processing Unit 330, 330A supply department 340 Billing Department 350 Storage section M Mobile object

Claims

1. The first device mounted on the mobile vehicle, Multiple second devices, each used by multiple users in locations different from the aforementioned mobile body, An information acquisition unit that acquires predetermined information transmitted by the first device, A providing unit that provides the predetermined information acquired by the information acquisition unit to a plurality of the second devices, A plurality of display units that display the predetermined information provided to a plurality of the second devices by the providing unit, Equipped with, When an operation input is made by the first user to one of the multiple second devices, the display unit of the second devices other than the first second device displays the output result based on the operation input made to the first second device. Information processing system.

2. User information acquisition unit that acquires user information of the user of the second device Equipped with, The second device used by the first user is a creation unit that creates an output signal for presenting the output result based on the user information acquired by the user information acquisition unit. The information processing system according to claim 1, further comprising:

3. The second device is Eye-line detection unit that detects the direction of the user's gaze. Equipped with, The information processing system according to claim 1, wherein the display unit displays output results based on the user's gaze direction detected by the gaze detection unit.

4. The second device is POI (Point of Interest) acquisition unit Equipped with, The POI acquisition unit acquires POI information based on the position information of the first device when the presentation unit presents the output result. The information processing system according to claim 1, wherein the presentation unit presents the POI information acquired by the POI acquisition unit together with the output result.

5. The second device is POI (Point of Interest) acquisition unit Equipped with, The POI acquisition unit acquires POI information based on the user's gaze direction detected by the gaze detection unit. The information processing system according to claim 3, wherein the presentation unit determines a position to present the output result based on the POI information acquired by the POI acquisition unit.

6. The second device is A POI (Point of Interest) acquisition unit that acquires POI information, When multiple POI pieces of information are acquired by the POI acquisition unit, a selection unit selects the POI piece of information closest to the moving object from among the multiple POI pieces of information. Furthermore, The information processing system according to claim 1, wherein the presentation unit presents the POI information selected by the selection unit to a second device other than one of the second devices.

7. The information processing system according to claim 1, wherein the display unit changes the display position of the output result based on the speed of the moving body.

8. A moving body behavior detection unit that detects that the moving body has turned right or left. Furthermore, The information processing system according to claim 1, wherein the display unit changes the content of the display of the output result when the moving body behavior detection unit detects that the moving body has turned right or left after displaying the output result.

9. When an operation input is made by the first user to one of the multiple second devices, a creation unit creates notification information including the output result for the first device based on the operation input made to the one of the second devices. The information processing system according to claim 1, further comprising:

10. Voice information acquisition unit that acquires voice information based on the user's voice. Furthermore, The information processing system according to claim 1, wherein the display unit displays output results based on the operation input and the voice information.

11. Voice information acquisition unit that acquires voice information based on the user's voice. Furthermore, The information processing system according to claim 4, wherein the presentation unit presents the POI information and the voice information acquired by the voice information acquisition unit in relation to the POI information together with the output result.

Images