Information processing device, information processing system, and information processing method

Abstract

To enable users who are in a different location from the moving vehicle to experience a more appropriate simulated ride. [Solution] The information management device according to the embodiment processes information provided from a first device mounted on a mobile body to a second device used by a user at a location different from the mobile body. The information processing device comprises: an acquisition unit that acquires information including video data transmitted by the first device; a provision unit that provides video data to the second device based on the information acquired by the acquisition unit; a reception unit that receives a first operation performed on a portion of the video data presented to the user based on the video data data provided to the second device by the provision unit; and a brightness change unit that changes the brightness of a portion of the video data based on the first operation received by the reception unit.

Description

Technical Field

[0003] ,

[0005] , , , , , , , , , , , , ,

[0004] , , , , ,

[0006] ,

[0001] The present invention relates to an information processing apparatus, an information processing system, and an information processing method.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transport system that takes into account people in vulnerable positions such as the elderly, disabled, and children among traffic participants. In particular, research and development focusing on further improving traffic safety and convenience through the development of means of transportation for vulnerable road users are being carried out towards this realization. In this regard, a technique is known that can automatically adjust the brightness of a projected image according to the ambient brightness (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, when performing a pseudo experience (hereinafter referred to as "pseudo riding") as if riding in a moving body using a device used in a place different from the moving body, it is a problem that a burden is imposed on a user who performs pseudo riding in a place different from the moving body.

[0005] The present application aims to allow a user in a place different from the moving body to perform a more appropriate pseudo riding in order to solve the above problems. And, by extension, it contributes to the development of a sustainable transport system and the support of good connections between local areas including urban areas and the outskirts of cities.

Means for Solving the Problems

[0006] The information processing device, information processing system, and information processing method according to this invention employ the following configuration. (1) An information processing device according to one aspect of the present invention is an information processing device that processes information provided from a first device mounted on a mobile body to a second device used by a user at a location different from the mobile body, and comprises: an acquisition unit that acquires information including moving image data transmitted by the first device; a provision unit that provides moving image data to the second device based on the information acquired by the acquisition unit; a reception unit that receives a first operation performed on a portion of the moving image data presented to the user based on the moving image data data provided to the second device by the provision unit; and a brightness changing unit that changes the brightness of the portion of the moving image data based on the first operation received by the reception unit.

[0007] (2) In the embodiment of (1) above, the first apparatus comprises an imaging device, and the brightness changing unit creates control information that causes the imaging device to change the brightness based on the first operation received by the reception unit.

[0008] (3) In the embodiment of (1) above, the receiving unit notifies the mobile body that the first operation to change the brightness has been performed based on the first operation it has received.

[0009] (4) In the embodiment of (2) above, the imaging device adjusts the brightness of a portion of the area based on the ambient brightness.

[0010] (5) In the embodiment of (1) above, the device further comprises a detection unit for detecting a gaze region of the moving image that the user is fixated on, and the brightness changing unit changes the brightness of the gaze region of the moving image based on the gaze region detected by the detection unit.

[0011] (6) The embodiment of (1) above further comprises a voice recognition unit that detects and performs voice recognition, wherein the brightness changing unit adjusts the brightness of the relevant POI (Point of Interest) information when the voice recognized by the voice recognition unit is related to POI (Point of Interest) information around the moving object.

[0012] (7): In the embodiment of (1) above, the device further comprises a detection unit for detecting a gaze region of the video image that the user is looking at, the reception unit receives a second operation when a portion of the video image is being gazed at for a predetermined time or longer by the detection unit, and the brightness change unit changes the brightness of the portion of the video image based on the second operation received by the reception unit.

[0013] (8) In the embodiment of (7) above, the second operation includes blinking, gestures, and speech.

[0014] (9) In the embodiment of (7) above, the number of actions included in the second operation is greater than the number of actions included in the first operation.

[0015] (10): In the embodiment of (1) above, the brightness adjustment unit grants priority for brightness adjustment to the second device that is first connected to the information processing device among the plurality of second devices.

[0016] (11): In the embodiment of (1) above, the brightness changing unit notifies each of the competing second devices when the reception unit receives conflicting first operations from multiple second devices.

[0017] (12): In the embodiment of (1) above, the brightness changing unit, when the first operation from a plurality of second devices is in conflict, performs a process to detect POI information around a portion of the video image where the first operation was performed by each of the users of the conflicting plurality of second devices, and adjusts the brightness of the area where the POI information was detected or the brightness of the POI information if POI information can be detected.

[0018] (13): In the embodiment of (1) above, the brightness changing unit, when the reception unit detects a conflict in the first operation from multiple second devices, performs a process to detect POI information around a portion of the video image where the first operation was performed by each of the users of the multiple second devices that were in conflict, and when multiple POI pieces of information can be detected, it notifies each of the users of the multiple second devices that were in conflict of the first operation of an overview of the multiple POI pieces of information.

[0019] (14): An information processing system according to one aspect of the present invention is an information processing system comprising: a first device mounted on a mobile body; a second device used by a user at a location different from the mobile body; an acquisition unit that acquires information including motion image data transmitted by the first device; a provision unit that provides motion image data to the second device based on the information acquired by the acquisition unit; a reception unit that receives a first operation performed on a portion of the motion image data presented to the user based on the motion image data data provided to the second device by the provision unit; and a brightness change unit that changes the brightness of the portion of the motion image data based on the first operation received by the reception unit.

[0020] (15): An information processing method according to one aspect of the present invention is an information processing device that processes information provided from a first device mounted on a mobile body to a second device used by a user at a location different from the mobile body, the device acquiring information including video data transmitted by the first device, providing video data to the second device based on the acquired information, receiving a first operation performed on a portion of the video data presented to the user based on the video data data provided to the second device, and changing the brightness of the portion of the video data based on the received first operation. [Effects of the Invention]

[0021] According to the aspects (1) to (15) above, it is possible to make a user who is in a location different from the moving body perform a more appropriate pseudo-riding.

Brief Description of Drawings

[0022] [Figure 1] It is a diagram showing the usage environment etc. of the information processing system 1 and the management server 300. [Figure 2] It is a diagram showing an example of the content of the user data 360. [Figure 3] It is a configuration diagram of the first device 100. [Figure 4] It is a diagram showing an example of the arrangement of a part of the first device 100 in the moving body M. [Figure 5] It is a configuration diagram of the second device 200. [Figure 6] It is a diagram for explaining an image corresponding to the pointing direction. [Figure 7] It is a diagram showing a first example of the functional configuration of the first control device 170 and the second control device 270. [Figure 8] It is a diagram showing a second example of the functional configuration of the first control device 170 and the second control device 270. [Figure 9] It is a sequence diagram showing an example of the processing executed by the information processing system 1. [Figure 10] It is a diagram showing an example of an image IM10 indicating inquiry information. [Figure 11] It is a diagram showing an example of an image IM20 indicating a request result. [[ID=ID=40]] [Figure 12A] It is a sequence diagram showing a first example of the processing executed by the information processing system 一. [Figure 12B] It is a sequence diagram showing a second example of the processing executed by the information processing system 1. [[ID=ID=46]] [Figure 12C] It is a sequence diagram showing a third example of the processing executed by the information processing system 1. [Figure 12D] It is a sequence diagram showing a fourth example of the processing executed by the information processing system 1. [Figure 13]This is a diagram illustrating an example of the information provided. [Modes for carrying out the invention]

[0023] Embodiments of the information processing device, information processing system, and information processing method of the present invention will be described below with reference to the drawings. In the following description, an information processing system to which the information processing device is applied will be described. The information processing system includes, for example, a first device mounted on a mobile vehicle on which an occupant is riding, a second device used by a user at a location different from the mobile vehicle, and the information processing device. The mobile vehicle is, for example, a vehicle, but any mobile vehicle (e.g., a ship, a flying object) that can accommodate an occupant is acceptable. The occupant is mainly the driver of the mobile vehicle, but may be an occupant other than the driver.

[0024] Between the first and second devices, audio collected by a microphone is transmitted to the other side and played back through a speaker, creating a state similar to a telephone call. Furthermore, a portion of the image captured by the camera unit of the first device is displayed on the second device, providing the second device with MR (Mixed Reality). This allows the user of the second device to experience the sensation of being on board the mobile vehicle (simulated ride experience) even though they are in a different location from the mobile vehicle, and the crew can converse with the user experiencing the simulated ride on the mobile vehicle via the first device, giving them the feeling that the user is actually riding on the mobile vehicle with them. Alternatively, a portion of the image captured by the camera unit of the first device may be displayed on the second device, providing the second device with VR (Virtual Reality). This allows the user of the second device to experience the sensation of being on board the mobile vehicle (simulated ride experience) even though they are in a different location from the mobile vehicle, and the crew can converse with the user experiencing the simulated ride on the mobile vehicle via the first device, giving them the feeling that the user is actually riding on the mobile vehicle with them. Furthermore, users can fully immerse themselves in the world of the video and use the application. In the following, as mentioned above, the experience of a user having the simulated feeling of actually riding in a moving vehicle may be referred to as "simulated riding." The relationship between the first and second devices does not need to be one-to-one; the information processing system can operate by matching one of several first devices with several second devices in a one-to-many relationship. In the latter case, for example, one passenger can communicate with multiple users simultaneously or sequentially.

[0025] <Basic configuration> Figure 1 shows the operating environment of the information processing system 1 and the management server 300. The information processing system 1 includes a first device (mobile device) 100 mounted on a mobile vehicle M on which a crew member P is riding, and a second device (user device) 200 used by a user U at a location different from the mobile vehicle M (although it is not ruled out that it may be a nearby location). The first device 100, the second device 200, and the management server 300 each communicate with each other via a network NW. The information processing system 1 may or may not include the management server 300.

[0026] The network NW includes at least one of the following: the Internet, WAN (Wide Area Network), LAN (Local Area Network), mobile communication network, cellular network, etc. The management server 300 is an example of an "information processing device." The management server 300 may be implemented as a server device or storage device incorporated into a cloud computing system. In this case, the functions of the management server 300 may be implemented by multiple server devices and storage devices in the cloud computing system. Furthermore, the first device 100 mounted on the mobile device M may be implemented by multiple units.

[0027] The management server 300 manages the information provided to the first device 100 and the second device 200, and manages their communication. The management server 300 includes, for example, a communication device 310, an acquisition unit 315, a matching processing unit 320, a provisioning unit 330, a reception unit 332, a brightness change unit 334, a charge management unit 340, and a storage unit 350. The acquisition unit 315, the matching processing unit 320, the provisioning unit 330, the reception unit 332, the brightness change unit 334, and the charge management unit 340 are implemented, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software).

[0028] Some or all of these components may be implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or by the collaboration of software and hardware. The program may be stored in advance on a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device with a non-transient storage medium), or it may be stored on a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is inserted into a drive device. Alternatively, the program may be stored on a management server 300 and updated as needed when connected to a network NW.

[0029] The communication device 310 is a communication interface for connecting to the network NW. Communication between the communication device 310 and the first device 100, and communication between the communication device 310 and the second device 200, are conducted, for example, according to TCP / IP (Transmission Control Protocol / Internet Protocol).

[0030] The acquisition unit 315 acquires various types of information transmitted via the network NW by the first device 100, the second device 200, and other external devices. These types of information may include video data, driver voice data, and vehicle data (such as the display contents of the METER and IVI (in-vehicle infotainment) systems).

[0031] The matching processing unit 320 is implemented by a processor, such as a CPU, executing a program (set of instructions) stored in a storage medium. For example, when the communication device 310 receives a matching request from user U via the second device 200, or from crew member P via the first device 100, the matching processing unit 320 refers to user data 360 to perform matching between user U and crew member P. Using the communication device 310, it transmits the communication identification information of crew member P's first device 100 to the second device 200 of the matched user U, and transmits the communication identification information of user U's second device 200 to the first device 100 of the matched crew member P. After receiving these, the first device 100 and the second device 200 can perform more real-time communication, such as one based on UDP (User Datagram Protocol).

[0032] The providing unit 330 generates information to be provided to the first device 100 and the second device 200 based on the various information acquired by the acquisition unit 315, and transmits the generated information to the target devices. The providing unit 330 also generates information indicating the processing results by the matching processing unit 320 and the fee information (settlement information) managed by the fee management unit 340, and provides the generated information to the target devices.

[0033] The reception unit 332 receives first operation information and second operation information transmitted from the second device 200. The first and second operation information are performed on a portion of the video image presented to the user U based on the video image data provided to the second device 200 by the provision unit 330. The reception unit 332 also receives text information transmitted from the second device 200. The reception unit 332 also receives information indicating the gaze area transmitted from the second device 200.

[0034] The brightness change unit 334 changes the brightness of a portion of the video based on the first operation information received by the reception unit 332. For example, based on the first operation information, the brightness change unit 334 creates control information to change the brightness of the camera among the cameras included in the camera unit 130 that is capturing a portion of the video. Based on the second operation information received by the reception unit 332, the brightness change unit 334 changes the brightness of a portion of the video. For example, based on the second operation information, the brightness change unit 334 creates control information to change the brightness of the camera among the cameras included in the camera unit 130 that is capturing a gaze area that user U has been looking at for a predetermined time or longer.

[0035] The brightness adjustment unit 334 determines, based on the character information received by the reception unit 332, whether or not the POI (Point of Interest) information registered in the provided information DB 362 stored in the storage unit 350 contains POI information related to the character information. If the POI information registered in the provided information DB 362 contains POI information related to the character information, the brightness adjustment unit 334 identifies the camera among the cameras included in the camera unit 130 that is capturing the relevant POI information from the information (moving image data) that the providing unit 330 provides to the second device 200, and creates control information to change the brightness of the identified camera.

[0036] The brightness adjustment unit 334 changes the brightness of the gaze area in the moving image based on the information indicating the gaze area received by the reception unit 332. For example, based on the information indicating the gaze area, the brightness adjustment unit 334 creates control information to change the brightness of the camera among the cameras included in the camera unit 130 that is capturing the gaze area that the user U is looking at. The brightness adjustment unit 334 transmits the created control information to the first device 100.

[0037] The fare management unit 340 manages the fares to be charged to user U based on the information provided to user U, and the fares to be charged to crew member P based on the information provided to crew member P of the mobile vehicle M. The fare management unit 340 may also manage, for example, the compensation paid to user U and crew member P based on the information provided by user U and crew member P. The fare management unit 340 may also handle settlement processing for user U and crew member P.

[0038] The storage unit 350 may be implemented using the various storage devices mentioned above, or an SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 350 can store, for example, user data 360, provided information DB 362, programs, and various other information.

[0039] Figure 2 shows an example of the contents of user data 360. User data 360 includes a crew list 360A and a user list 360B. Crew list 360A includes, for example, the crew ID, which is the identification information of crew member P of mobile vehicle M, their communication identification information (IP address, etc.), the user ID, which is the identification information of user U to be matched, the mobile vehicle information on which the crew member is riding, and the availability information set by the crew member, all of which are associated with each other. Mobile vehicle information includes, for example, information on equipment installed on mobile vehicle M (installed equipment information) and vehicle class information indicating the size and external shape of mobile vehicle M. Mobile vehicle information may also include information on the current location, destination, and surrounding conditions of mobile vehicle M (for example, driving on a road along the coast) that is transmitted from mobile vehicle M at predetermined intervals.

[0040] The user list 360B associates, for example, a user ID, its communication identification information (such as an IP address), the crew member P to be matched, and user information with each other. User information may include information about physical size (e.g., height and sitting height) and information that can predict physical size (e.g., age). Availability information is information that the mobile unit M can or cannot provide, and is set, for example, by the crew member P. Availability information may be set for each piece of equipment installed on the mobile unit M, or for each user U. Examples of availability information include, but are not limited to, "Provision of images is permitted," "Provision of audio is not permitted," "Provision of interior images is permitted, but provision of exterior images is not permitted," "Provision of crew member images is not permitted," and "Use of navigation devices is not permitted." Availability information may also include a fee for enabling the provision of services (service provision fee). User data 360 may be generated in any manner, not limited to the manner shown in Figure 2, as long as it includes this information.

[0041] The provided information DB362 stores various types of information to be provided to user U or crew P. This information includes, for example, map information, POI (Point of Interest) information, and computer-generated images (e.g., computer graphics images of people, marks, symbols, icons, etc.). POI information, for example, is information about various shops, theme parks, and landmarks at specific locations, and may be included in the map information. Audio information may also be included. Furthermore, the provided information DB362 may include advertising information. This advertising information may include, for example, advertisements related to the mobile vehicle M, advertisements related to user U or crew P, and advertisements related to the products and services of shops. The inserted advertising information is managed separately from indoor and outdoor information, and when archived and distributed later, it may differ from the inserted advertising information distributed in real time (for example, closed shops may be updated, and the featured menus may be updated). Advertising information may include, for example, videos and audio.

[0042] Figure 3 is a configuration diagram of the first device 100. The first device 100 includes, for example, a first communication device 110, a first microphone 120, an external sensor 125, a camera unit 130, a first speaker 140, a user display device 150, an HMI (Human-machine Interface) 160, and a first control device 170. The first control device 170 is connected to a controlled device 190 mounted on a mobile body M.

[0043] The first communication device 110 is a communication interface for communicating via the network NW with the communication device 310 of the management server 300 and the second communication device 210 of the second device 200, which will be described later.

[0044] The first microphone 120 collects at least the voice emitted by the occupant P. The first microphone 120 may be installed inside the vehicle M and have the sensitivity to collect sounds from outside the vehicle M, or it may include a microphone installed inside the vehicle M and a microphone installed outside the vehicle M. Hereinafter, the voice information acquired by the microphone installed inside the vehicle may be referred to as "indoor voice information". The voice collected by the first microphone 120 is transmitted, for example, via the first control device 170 to the management server 300 or the second device 200 by the first communication device 110. If it is not possible to set up a microphone installed outside the vehicle M, the indoor voice information may be processed based on driving information (vehicle speed, acceleration / deceleration, road vibration, etc.) and the surrounding driving environment to generate pseudo-outdoor voice information. It is also possible to record the positional relationship of the speaker relative to the vehicle M (whether the speaker is inside or outside the vehicle), and processing may be performed on the collected voice according to that positional relationship.

[0045] The external sensor 125 detects the position of objects around the moving object M. The external sensor 125 is, for example, a radar device, a LIDAR (Light Detection and Ranging) sensor, or various other proximity sensors. The radar device emits radio waves such as millimeter waves around the moving object M and detects the radio waves reflected by objects (reflected waves) to detect at least the position (distance and direction) of the objects. The radar device may also detect the position and velocity of objects using the FM-CW (Frequency Modulated Continuous Wave) method. The LIDAR sensor irradiates light (or electromagnetic waves with a wavelength close to light) around the moving object M, measures the scattered light, and detects the distance to the target based on the time from emission to reception. The irradiated light is, for example, pulsed laser light. The radar device and LIDAR sensor can be attached to any part of the moving object M. Alternatively, the external sensor 125 may detect surrounding objects using images captured by the outdoor camera 134 of the camera unit 130.

[0046] The camera unit 130 is an example of an imaging device, and includes, for example, an indoor camera 132 and an outdoor camera 134. The camera unit 130 acquires control information obtained from the management server 300 via the first communication device 110 and adjusts the brightness based on the acquired control information. The first speaker 140 outputs the voice spoken by user U, which is acquired via the first communication device 110. Details of the camera unit 130 and the arrangement of the first speaker 140 will be explained later with reference to Figure 4.

[0047] The user display device 150 virtually displays user U as if the user U were present inside the mobile device M. For example, the user display device 150 may display a hologram or display user U in a part of the mobile device M corresponding to a mirror or window.

[0048] HMI160 consists of a touch panel and a voice response device (agent device), etc. HMI160 receives various instructions from crew member P to the first device 100 and provides various information to crew member P.

[0049] The first control device 170 includes, for example, a processor such as a CPU, and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the first device 100 by executing the set of instructions.

[0050] The controlled device 190 is, for example, an in-vehicle device such as a navigation system mounted on the mobile vehicle M that guides the driver to a destination, or a driver assistance system that controls either or both the steering and / or speed of the mobile vehicle M to assist the driver P. The controlled device 190 includes, for example, a seat drive system that can adjust the position (front, back, left, and right), orientation, and height of the seat. When viewing the image using the second device 200, if the camera unit 130 of the first device 100 is attached to the seat, the impact on the image can be suppressed by prohibiting seat movement. Even if seat movement is permitted, processing such as angle of view conversion may be performed to ensure that the image is not affected when the seat is moved. Furthermore, if user U requests to view images outside the current angle of view of the camera unit 130, the seat drive system may be controlled as a request from the second device 200.

[0051] Figure 4 shows an example of the arrangement of part of the first device 100 in the mobile vehicle M. The interior camera 132 is attached, for example, to the neck pillow of the passenger seat S2 (an example of a "designated seat") via attachment 132A, and is positioned slightly away from the backrest of the passenger seat S2 towards the direction of travel of the mobile vehicle M. The interior camera 132 has a wide-angle lens and is capable of imaging the area represented by the hatched area 132B in the figure. The interior camera 132 can photograph not only the interior of the mobile vehicle M but also the exterior through the windows. In the following description, the passenger seat S2 is assumed to be a designated seat, but the designated seat may be another seat such as a rear seat.

[0052] The outdoor camera 134 includes, for example, a plurality of sub-outdoor cameras 134-1 to 134-4. By combining the images captured by the plurality of sub-outdoor cameras 134-1 to 134-4, an image such as a panoramic image capturing the outside of the mobile body M is obtained. The outdoor camera 134 may also include (or in addition to) these a wide-angle camera mounted on the roof of the mobile body M. An indoor camera 132 capable of capturing the area behind the passenger seat S2 may be added, and the mobile body image described later may be generated as a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 by the first control device 170, or it may be generated as a 360-degree panoramic image by appropriately combining images captured by indoor cameras 132 and images captured by outdoor cameras 134.

[0053] The first speaker 140 outputs the voice of user U acquired via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, sub-first speaker 140-1 is located in the center of the instrument panel, sub-first speaker 140-2 is located at the left end of the instrument panel, sub-first speaker 140-3 is located at the right end of the instrument panel, sub-first speaker 140-4 is located at the bottom of the left door, and sub-first speaker 140-5 is located at the bottom of the right door.

[0054] When the first control device 170 outputs the voice of user U to the first speaker 140, for example, it outputs sound at a similar volume from sub-first speakers 140-2 and 140-4, and turns off the other sub-first speakers, thereby localizing the sound image so that the voice sounds to the passenger seat S2 for the occupant P seated in the driver's seat S1. Furthermore, the method of sound image localization is not limited to volume adjustment, but may also be performed by shifting the phase of the sound output by each sub-first speaker. For example, to localize the sound image so that it sounds like it is coming from the left, the timing of outputting sound from the left sub-first speaker should be slightly earlier than the timing of outputting the same sound from the right sub-first speaker.

[0055] Furthermore, when the first control device 170 outputs the voice of user U to the first speaker 140, it may direct the sound image to the occupant P so that the voice is heard from a position corresponding to the height of user U's head on the passenger seat S2, and then output the voice emitted by user U to the first speaker 140. In this case, the first speaker 140 needs to have multiple child first speakers 140-k (where k is a number of natural numbers) at different heights.

[0056] Figure 5 is a configuration diagram of the second device 200. The second device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a mobile image display device 250, an HMI 260, and a second control device 270. The detection device 230 includes, for example, a direction detection device 232, a gaze area detection device 233, a head position detection device 234, and a motion sensor 236.

[0057] The second communication device 210 is a communication interface for communicating with the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0058] The second microphone 220 collects the voice emitted by user U. The voice collected by the second microphone 220 is transmitted to the first communication device 110 by the second communication device 210 via the second control device 270, for example.

[0059] The direction detection device 232 is a device for detecting the direction of direction. The direction of direction is the orientation of the user U's face or gaze, or an orientation based on both. Hereinafter, the direction of direction will be assumed to be an angle in the horizontal plane, that is, an angle without a vertical component, but the direction of direction may also be an angle that includes a vertical component. The direction detection device 232 may include a physical sensor (e.g., an accelerometer, a gyroscope, etc.) attached to the VR goggles described later, or it may be an infrared sensor that detects multiple positions of the user U's head, or a camera that images the user U's head. In any case, the second control device 270 calculates the direction of direction based on the information input from the direction detection device 232. Since various technologies for this are publicly known, a detailed explanation will be omitted.

[0060] The gaze area detection device 233 is a device for detecting the gaze area. The gaze area is the area that user U is looking at carefully. The gaze area detection device 233 may be a camera that captures the line of sight and is attached to the VR goggles described later. The second control device 270 derives the gaze area based on the information input from the gaze area detection device 233. Since various technologies for this are publicly known, a detailed explanation will be omitted.

[0061] The head position detection device 234 is a device for detecting the position (height) of the user U's head. For example, one or more infrared sensors or optical sensors installed around the chair where the user U is seated can be used as the head position detection device 234. In this case, the second control device 270 detects the position of the user U's head based on the presence or absence of detection signals from one or more infrared sensors or optical sensors. Alternatively, the head position detection device 234 may be an accelerometer attached to VR goggles (head-mounted display (HMD), etc.). In this case, the second control device 270 detects the position of the user U's head by integrating the output of the accelerometer minus the acceleration due to gravity. The head position information obtained in this way is provided to the second control device 270 as height information.

[0062] The user's head position may be obtained based on the user U's actions on the HMI260. For example, the user U may input their height numerically into the HMI260, or they may input their height using the dial switch included in the HMI260. In these cases, the head position, i.e., height information, is calculated from the height. Alternatively, the user U may input discrete values ​​such as body size: large / medium / small, rather than continuous values, into the HMI260. In this case, height information is obtained based on information indicating body size. Furthermore, the user's head height may not be specifically obtained, and the user U's head height may be simply obtained based on the body size of a typical adult (which may be separated by gender).

[0063] The motion sensor 236 is a device for recognizing blinking and gestures performed by user U. For example, a camera that captures images of user U's upper body is used as the motion sensor 236. In this case, the second control device extracts feature points of user U's face (eyes, nose, mouth) and feature points of the body (fingertips, wrists, elbows, etc.) from the image captured by the camera, and recognizes user U's blinking and gestures based on the movement of these feature points.

[0064] The second speaker 240 outputs the voice emitted by occupant P, which is acquired via the second communication device 210. The second speaker 240 has a function, for example, to change the direction from which the voice is heard. The second control device 270 causes the second speaker to output voice so that the voice is heard by the user U from the position of occupant P as seen from the passenger seat S2. The second speaker 240 may include multiple child second speakers 240-n (where n is a number of natural numbers), and sound localization may be achieved by the second control device 270 adjusting the volume of each child second speaker 240-n, or if headphones are attached to the VR goggles, sound localization may be achieved by utilizing the function of the headphones.

[0065] The mobile image display device 250 displays an image from the camera unit 130 (which may be an image that has undergone the aforementioned stitching process, and will be referred to as a mobile image below) that corresponds to the direction of orientation as seen from the passenger seat. Figure 6 is a diagram illustrating the image corresponding to the direction of orientation. In this example, the VR goggles 255 include a direction detection device 232, a physical sensor as a head position detection device 234, and the mobile image display device 250. The second control device 270, for example, uses the center of the user U's head or the center of the VR goggles 255 as Ω, and detects the direction the VR goggles 255 are facing as the direction of orientation φ, using a pre-calibrated direction as the reference direction. Since various methods for this function are already publicly known, a detailed explanation is omitted.

[0066] The mobile object image display device 250 displays image A2, which is part of the mobile object image A1 (which has an angle of approximately 240 degrees in the figure, but the field of view may be expanded by the merging process as described above), towards the user U.

[0067] HMI260 is a touch panel, voice response device (agent device), or the switches mentioned above. HMI260 receives various instructions from user U to the second device 200.

[0068] The second control device 270 includes, for example, a processor such as a CPU, and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the second device 200 by executing the set of instructions. Note that all the functions of the second device 200 shown in Figure 5 may be integrated into the VR goggles.

[0069] <Functional configuration of the first control device 170 and the second control device 270> The functional configurations of the first control device 170 and the second control device 270 will be described below. In the first and second examples shown below, the first control device 170 and the second control device 270 are mainly responsible for generating and transmitting information, while the management server 300 manages the matching of simulated passengers P and users U.

[0070] [Example 1] Figure 7 shows a first example of the functional configuration of the first control device 170 and the second control device 270. In the first example, the first control device 170 includes a matching request / acceptance unit 171, an audio output control unit 172, an image transmission unit 173, an onboard device linkage unit 174, and a brightness adjustment unit 175. The second control device 270 includes a matching request / acceptance unit 271, an audio output control unit 272, a direction detection unit 273, a head position detection unit 274, a gesture input detection unit 275, an image editing unit 276, a mobile object image display control unit 277, an audio recognition unit 279, and an operation information transmission unit 280. These functional units are realized, for example, by a processor such as a CPU executing a program (set of instructions). Some or all of these components may be realized by hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or by the cooperation of software and hardware.

[0071] The matching request / acceptance unit 171 uses the HMI 160 to receive input of a matching request (simulated boarding request) from the occupant P and transmits it to the management server 300, and also uses the HMI 160 to receive input of acceptance of a matching request received from the management server 300 and transmits it to the management server 300. The matching request / acceptance unit 171 controls the first communication device 110 to communicate with the second device 200 of user U, with whom a match has been made. In addition, the matching request / acceptance unit 171 may, for example, accept the setting of information provision permission information indicating whether or not to allow the occupant P to provide information when accepting a matching request obtained from the management server 300.

[0072] As described above, the audio output control unit 172 controls the first speaker 140. After communication with the second device 200 begins, the image transmission unit 173 uses the first communication device 110 to transmit the mobile image A1 to the second device 200. The mobile image A1 may include, for example, an indoor image captured by the indoor camera 132 and / or an outdoor image captured by the outdoor camera 134.

[0073] The onboard equipment linkage unit 174 controls the controlled equipment 190 based on instruction signals input from the second device 200. The onboard equipment linkage unit 174 may also perform control if the user U has been authorized by the crew member P to control the controlled equipment 190. The brightness adjustment unit 175 adjusts the brightness of the moving object image captured by the camera unit 130 based on control information to change the brightness received from the management server 300.

[0074] The matching request / acceptance unit 271 uses the HMI 260 to receive matching requests (simulated ride requests) from user U and transmits them to the management server 300, and also uses the HMI 260 to receive acceptance input for matching requests received from the management server 300 and transmits it to the management server 300. The matching request / acceptance unit 271 controls the second communication device 210 to communicate with the first device 100 of the crew member P with whom a match has been made. The matching request / acceptance unit 271 may also, for example, present to user U what information will be provided based on the matching result (request result) obtained from the management server 300, allow user U to confirm, and then accept instructions from user U on whether or not to proceed with the simulated ride.

[0075] As described above, the audio output control unit 272 controls the second speaker 240. The direction detection unit 273 detects the direction φ based on the output of the direction detection device 232. The head position detection unit 274 detects the height of the user U's head based on the output of the head position detection device 234. The head position may be expressed as three-dimensional coordinates, or the head height may simply be detected as the head position. The gesture input detection unit 275 detects the user U's blinking and gesture input based on the output of the motion sensor 236.

[0076] The image editing unit 276 performs a process to extract, for example, image A2 from the moving object image A1, which corresponds to the direction of direction φ as seen from the passenger seat S2 (Figure 6). In the example in Figure 6, the direction of direction φ for movement is shown to the left and right of the user U's head, but the process may also be performed to extract an image corresponding to the direction of direction of movement in the vertical direction of the head.

[0077] The mobile image display control unit 277 displays the image A2 extracted by the image editing unit 276 on the mobile image display device 250. At this time, the image editing unit 276 may also display on the mobile image display device 250 an image corresponding to the direction of direction φ as viewed from the height indicated by the height information of the user U's head. For example, the image editing unit 276 switches between displaying an indoor image or an outdoor image depending on the distance from the reference position of the user U's head (for example, the amount of upward movement or the amount of lateral movement). The gaze area detection unit 278 derives the gaze area based on the output of the gaze area detection device 233. For example, the gaze area detection unit 278 may define the gaze area as a certain range from the center of the point where the direction of the user U's gaze intersects with the moving image displayed on the moving image display device 250. The speech recognition unit 279 acquires the voice (utterance) collected by the second microphone 220, recognizes the acquired voice, and converts it into text information.

[0078] The operation information transmission unit 280 receives input of a first operation performed by user U using the HMI 260 on at least a portion of the area of ​​the moving image displayed on the mobile image display device 250, and transmits it to the management server 300. The operation information transmission unit 280 receives character information converted by the speech recognition unit 279 and transmits the received character information to the management server 300. The operation information transmission unit 280 receives information indicating a gaze area derived by the gaze area detection unit 278 and transmits the received information indicating a gaze area to the management server 300. The operation information transmission unit 280 receives information indicating a gaze area derived by the gaze area detection unit 278, determines that user U has been gazing at the gaze area for a predetermined time or longer based on the received information indicating a gaze area, and receives information indicating at least one of blinking and / or a gesture operation recognized by the gesture input detection unit 275, and transmits it to the management server 300.

[0079] [Example 2] Figure 8 shows a second example of the functional configuration of the first control device 170 and the second control device 270. Compared to the first example in Figure 7, the first control device 170 includes an image editing unit 176, while the second control device 270 does not include an image editing unit 276 but includes a directional transmission unit 281. As the other components basically have the same functions as in the first example, further explanation is omitted. The direction transmitting unit 281 transmits the direction φ detected by the direction detection unit 273 to the first device 100 using the second communication device 210. The image editing unit 176 processes image A2 from the moving image A1 to extract image A2 corresponding to the directional direction φ (transmitted from the second device 200) as seen from the passenger seat (Figure 6). At this time, the image editing unit 176 may also obtain the height information of the user U's head from the second device 200 and process image A2 corresponding to the directional direction φ as seen from the height indicated by the height information.

[0080] In the second example, the image transmission unit 173 uses the first communication device 110 to transmit the image A2 extracted by the image editing unit 176 to the second device 200. The mobile image display control unit 277 then displays the image A2 transmitted from the first device 100 on the mobile image display device 250.

[0081] <Processing related to simulated rides> Next, in this embodiment, a series of processes performed by the information processing system 1 when user U virtually boards the mobile vehicle M (or when crew member P virtually boards user U) will be specifically described. Figure 9 is a sequence diagram showing an example of the processes performed by the information processing system 1. In the example in Figure 9, the processes in the first device 100 mounted on the mobile vehicle M while crew member P is on board, the second device 200 used by user U, and the management server 300 are shown. Furthermore, it is assumed that user U and crew member P are registered in advance with the management server 300 and registered in user data 360 in order to receive services from the information processing system 1.

[0082] In the example shown in Figure 9, the second device 200 generates a request for a simulated ride (simulated ride request) (step S100) and sends the generated simulated ride request to the management server 300 (step S102). The simulated ride request may include, for example, information specifying a passenger P in advance, or information specifying the current location, destination, or surrounding conditions of the moving vehicle M. More specific examples of simulated ride requests include requests such as "I want to simulate ride in a vehicle driven by person A" or "I want to simulate ride in a vehicle traveling on a road along the coast." These are examples of the matching requests described above.

[0083] The matching processing unit 320 of the management server 300 receives a simulated experience request from the second device 200 and performs matching processing based on the user data 360 (step S104). Then, if there is one or more matched mobile bodies M, the provisioning unit 330 generates inquiry information to ask whether or not to allow all mobile bodies M to simulate riding (step S106), and transmits the generated information to the first device 100 of the matched mobile bodies M (step S108).

[0084] The first device 100 displays the inquiry information sent from the management server 300 on the screen (step S110). Figure 10 is a diagram showing an example of image IM10 showing the inquiry information. Note that the display content and layout of image IM10 are not limited to this. The same applies to the descriptions of other images that follow. Image IM10 includes, for example, a user information display area AR11, a availability selection area AR12, and a switch display area AR13 as an inquiry screen. The user information display area AR11 displays information about user U who has made a simulated boarding request. The information about user U may be a user ID, or other information that allows the crew member P to identify user U.

[0085] The information provision selection area AR12 displays images for selecting the content of the information that occupant P will provide to user U, for each type of information. In the example in Figure 10, a screen is displayed for selecting whether or not to allow occupant P to provide information regarding interior images, exterior images, phone calls (voice), and navigation information (information provided by the navigation system). Phone calls (voice) is an example of "interior voice information." For example, if occupant P is willing to provide phone calls and images to user U, but does not want user U to know the specific destination, occupant P can allow the provision of phone calls, interior images, and exterior images, and refuse to provide navigation information.

[0086] This means that, for example, if user U is actually riding in the vehicle M, the destination can be easily revealed through navigation information. However, by not providing navigation information during a simulated ride, the vehicle can head to the destination without user U knowing, providing user U with a unique surprise experience specific to simulated rides. Furthermore, for example, with user U, with whom the driver is not very close, the driver can refuse to provide voice communication, allowing the driver to operate the vehicle without conversation.

[0087] The information to be selected for provision is not limited to the example in Figure 10, and may include, for example, information to select whether or not user U can operate the navigation system or other controlled devices 190, or information to select whether or not to provide images including occupant P. The availability of the information to be provided, selected by the provision availability selection area AR12, may be changed during the simulated ride. The switch display area AR13 includes, for example, icon IC11 and icon IC12. Icon IC11 is a switch that accepts permission for occupant P to allow user U to simulate ride. Conversely, icon IC12 is a switch that accepts permission for occupant P to refuse user U's simulated ride.

[0088] Returning to Figure 9, the first device 100 receives the user U's availability setting via the image IM10 shown in Figure 10 (step S112) and transmits the received availability setting information to the management server 300 (step S114). Note that the processing in steps S110 to S112 may be displayed on the screen when the mobile body M stops in order to ensure safety during operation, or instead of (or in addition to) a screen display, the system may notify the user that a simulated ride request has been made or accept the availability setting using voice. The provision unit 330 of the management server 300 generates a request result (response result) based on the availability setting information obtained from the first device 100 (step S116) and transmits information indicating the generated request result to the second device 200 (step S118).

[0089] The second device 200 receives the request result from the management server 300 and displays information indicating the received request result on the screen (step S120). Figure 11 shows an example of an image IM20 showing the request result. Image IM20 includes, for example, a passenger information display area AR21, a provided information list display area AR22, and a switch display area AR23 as a request result screen. This image IM20 may be generated for each passenger P (mobile body M) who is permitted to simulate riding, and an image showing a list of request results may also be displayed.

[0090] The passenger information display area AR21 displays information about passenger P who has authorized the simulated ride request. The information about passenger P may be a passenger ID, or other information that allows user U to identify passenger P. The provided information list display area AR22 displays a list of information authorized by passenger P. The provided information list display area AR22 may also display the fee (service usage fee) based on the authorized information. The switch display area AR23 includes, for example, icons IC21 to IC23. Icon IC21 is a switch that accepts a simulated ride on passenger P's mobile body M, as displayed in the passenger information display area AR21, using the provided information displayed in the provided information list display area AR22. Conversely, icon IC22 is a switch that accepts not to perform a simulated ride. Icon IC23 is a switch that accepts searching for other passengers (mobile bodies). When searching for other passengers, users may select from other passengers included in the request results, or they may input other request conditions and have the management server 300 perform the matching process again.

[0091] Returning to Figure 9, the second device 200 receives a selection instruction from user U via the image IM20 shown in Figure 11 (step S122) and transmits the received instruction information to the management server 300 (step S124). In the following description, it will be assumed that user U has given an instruction to virtually board the mobile body M of occupant P. Based on the instruction from the second device 200, the management server 300 manages the communication status between the first device 100 and the second device (step S126) and transmits information to the first device 100 and the second device 200 indicating that a virtual boarding will be performed (steps S128, S130). If user U chooses not to virtually board during the processing in steps S126 to S130, the provision unit 330 of the management server 300 transmits information to the first device 100 to notify user U.

[0092] Next, the first device 100 and the second device 200 communicate with each other to perform a simulated ride (step S132), and when the simulated ride is completed, they send information to the management server 300 indicating that communication has ended (steps S134, S136). In the process of step S132, the second device 200 may notify the passenger P from the user U before the information from the first device 100 is provided to the second device 200. This allows the user U to authorize communication at their own timing and start providing information such as phone calls or video streaming. Next, the fare management unit 340 of the management server 300 performs settlement processing for the charges related to the use of each service of the first device 100 and the second device 200. Specifically, the fare management unit 340 determines different charges for information provision based on at least one of the following: the type of passenger P, user U, and mobile body M that provide the information, the type of information that can be provided to the user U, and the amount of information. For example, if the passenger P or user U is a celebrity, the fare may be set higher than the normal fare, or if the vehicle M is a new car or a rare vehicle, the fare may be set higher than the normal fare. The fare management unit 340 then performs the settlement process corresponding to the set fare. This completes the processing of this sequence. Note that the processing in step S112 shown in Figure 9 (service availability setting process) may be set in advance by the passenger P for each registered user U.

[0093] <Processing related to brightness adjustment> Next, in an embodiment, a series of processes performed by the information processing system 1 when a user U virtually rides on a mobile vehicle M and adjusts the brightness based on the user U's request will be specifically described. Figure 12A is a sequence diagram showing a first example of the processes performed by the information processing system 1. The first device 100 and the second device 200 communicate with each other to perform the virtual ride.

[0094] In the example shown in Figure 12A, the second device 200 receives input for a first operation from user U (step S200) and transmits first operation information to the management server 300 based on the received first operation (step S202). The first operation may be, for example, a tap operation performed by user U using the HMI 260 on at least a portion of the area of ​​the moving image displayed on the mobile image display device 250. For example, user U taps any location included in the moving image displayed on the mobile image display device 250. The location may be a location that user U wants to view from a remote location. The HMI 260 acquires input for the tap operation performed by user U. The operation information transmission unit 280 receives the tap operation acquired by the HMI 260, creates first operation information including information indicating the tapped area, and transmits the created first operation information to the management server 300. An example of information indicating an area is the coordinates of at least a portion of the outline of the area on the mobile image display device 250.

[0095] Next, the brightness change unit 334 of the management server 300 creates control information to change the brightness of a portion of the video based on the first operation information from the second device 200 (step S204), and transmits the created control information to the first device 100 (step S206). For example, the reception unit 332 receives and accepts first operation information from the second device 200 via the communication device 310. The brightness change unit 334 receives the first operation information from the reception unit 332 and, based on the received first operation information, identifies the camera that is capturing a portion of the video image in which the first operation was performed. Here, the brightness change unit 334 may store camera video image area related information that associates information indicating multiple cameras included in the camera unit 130 with information indicating the video image area being captured by each of the multiple cameras. For example, the area of ​​the windshield may be divided into three parts and set as left, center, and right areas, and the video image areas being captured by each of the multiple cameras may be set, or it may be divided into six parts and set. In this case, the brightness change unit 334 receives information indicating the camera associated with the information indicating a portion of the video image in which the first operation was performed, and, based on the received camera information, creates control information to change the brightness of that camera.

[0096] Next, the camera unit 130 of the first device 100 adjusts the brightness based on control information from the management server 300 (step S208). For example, either the indoor camera 132 or the outdoor camera 134 acquires control information from the first control device 170 and adjusts the brightness by adjusting the EV value (Exposure Value) based on the acquired control information. Subsequently, either the indoor camera 132 or the outdoor camera 134 may adjust the brightness based on the ambient brightness. This allows for dynamic brightness adjustment so that the point instructed by the user U is always easily visible, even if the external environment changes due to driving and the ambient brightness changes.

[0097] Figure 12B is a sequence diagram showing a second example of processing performed by the information processing system 1. The first device 100 and the second device 200 communicate with each other to perform a simulated ride. In the example shown in Figure 12B, the second device 200 collects the voice (utterances) of user U, performs speech recognition on the collected voice (step S300), and transmits it to the management server 300 (step S302). For example, the second microphone 220 collects the voice (utterances) of user U. The speech recognition unit 279 acquires the voice (utterances) collected by the second microphone 220, recognizes the acquired voice, and converts it into text information. The operation information transmission unit 280 receives the text information converted by the speech recognition unit 279 and transmits the received text information to the management server 300.

[0098] Next, the brightness change unit 334 of the management server 300 determines whether or not there is POI information related to the character information based on the character information from the second device 200 (step S304). If there is POI information, it creates control information to change the brightness of the area containing that POI information included in a part of the video (step S306) and transmits it to the first device 100 (step S308). For example, the reception unit 332 receives and accepts the character information from the second device 200 via the communication device 310. The brightness change unit 334 receives the character information from the reception unit 332 and determines whether or not there is POI information related to the character information among the POI information registered in the provided information DB 362 stored in the storage unit 350. If there is POI information related to the character information among the POI information registered in the provided information DB 362, the brightness change unit 334 identifies the camera among the cameras included in the camera unit 130 that is capturing the related POI information from the video image data provided by the provision unit 330 to the second device 200, and creates control information to change the brightness of the identified camera. The brightness adjustment unit 334 terminates if the POI information registered in the provided information DB362 does not contain any POI information related to character information.

[0099] Step S310 is the same as step S208 in Figure 12A, so its explanation is omitted. Subsequently, either the indoor camera 132 or the outdoor camera 134 may be configured to adjust its brightness based on the ambient light. This allows for dynamic brightness adjustment so that POI information related to text information is always easily visible, even if the external environment changes due to driving and the ambient light changes.

[0100] Figure 12C is a sequence diagram showing a third example of processing performed by the information processing system 1. The first device 100, the second device 200-1, and the second device 200-2 communicate with each other to perform a simulated ride. In the example shown in Figure 12C, the second device 200-1 detects the gaze area that user U1 is looking at (hereinafter referred to as the "first gaze area") (step S400) and transmits information indicating the detected first gaze area to the management server 300 (step S402). The second device 200-2 detects the gaze area that user U2 is looking at (hereinafter referred to as the "second gaze area") (step S404) and transmits information indicating the detected second gaze area to the management server 300 (step S406). For example, in the second device 200-1, the gaze area detection unit 278 derives the first gaze area that user U1 is looking at. The operation information transmission unit 280 receives the information indicating the first gaze area derived by the gaze area detection unit 278 and transmits the received information indicating the first gaze area to the management server 300. In the second device 200-2, the gaze area detection unit 278 derives the second gaze area that user U2 is looking at. The operation information transmission unit 280 receives the information indicating the second gaze area derived by the gaze area detection unit 278 and transmits the received information indicating the second gaze area to the management server 300.

[0101] Next, the brightness change unit 334 of the management server 300 determines whether the gaze areas are the same based on the information indicating the first gaze area from the second device 200-1 and the information indicating the second gaze area from the second device 200-2 (step S408). If they are the same, it creates control information to change the brightness of a portion of the video (step S410) and transmits the created control information to the first device 100 (step S412). For example, the reception unit 332 receives and accepts the information indicating the first gaze area from the second device 200-1 and the information indicating the second gaze area from the communication device 310. The brightness change unit 334 receives the information indicating the first gaze area and the information indicating the second gaze area from the reception unit 332 and determines whether the gaze areas are the same. Here, "the gaze areas are the same" includes the fact that at least a portion of the first gaze area and the second gaze area are the same. If the gaze areas are not the same, the process terminates. If the gaze area is the same, the brightness change unit 334 creates control information to change the brightness of the camera among the cameras included in the camera unit 130 that is capturing the gaze area that the user U is looking at. Here, the brightness change unit 334 may store camera motion image area related information that associates information indicating multiple cameras included in the camera unit 130 with information indicating the area of ​​motion images captured by each of the multiple cameras. In this case, the brightness change unit 334 obtains information indicating the camera associated with the information indicating the gaze area that the user U is looking at, based on the camera motion image area related information, and creates control information to change the brightness of that camera based on the obtained information indicating the camera.

[0102] Step S414 is the same as step S208 in Figure 12A, so its explanation is omitted. Subsequently, either the indoor camera 132 or the outdoor camera 134 may be configured to adjust its brightness based on the ambient light. This allows for dynamic brightness adjustment, for example, when the external environment changes due to driving and the ambient light changes, so that the area of ​​focus that user U is looking at is always easily visible.

[0103] Figure 12D is a sequence diagram showing a fourth example of processing performed by the information processing system 1. The first device 100 and the second device 200 communicate with each other to perform a simulated ride. In the example shown in Figure 12D, the second device 200 detects the gaze area that user U is looking at (step S500) and determines whether a predetermined time has elapsed while user U has been looking at the gaze area (step S502). If the predetermined time has not elapsed (step S502: NO), the device returns to step S500. If the predetermined time has elapsed (step S502: YES), the device determines whether at least one of user U's blinking and / or gesture has been detected (step S504). For example, the gaze area detection unit 278 derives the gaze area that user U is looking at. The operation information transmission unit 280 obtains information indicating the gaze area from the gaze area detection device 233 and determines whether user U has been looking at the gaze area for a predetermined time or longer based on the obtained information indicating the gaze area. The motion sensor 236 recognizes blinking and gesture operations performed by user U. The operation information transmission unit 280 obtains the recognition results of user U's blinking and gesture operations from the motion sensor 236 and determines whether it has received information indicating at least one of blinking and / or a gesture operation.

[0104] Next, the operation information transmission unit 280 of the second device 200 receives input for a second operation when it receives information indicating at least one of blinking and / or a gesture operation (step S506) and transmits it to the management server 300 (step S508). An example of a second operation is blinking at regular intervals, closing the eyes for more than one second, and saying "adjust".

[0105] The brightness change unit 334 of the management server 300 creates control information to change the brightness of a portion of the video based on the second operation information from the second device 200 (step S510), and transmits the created control information to the first device 100 (step S512). For example, the reception unit 332 receives and accepts second operation information from the second device 200 via the communication device 310. The brightness change unit 334 receives the second operation information from the reception unit 332, obtains information indicating the gaze area based on the obtained second operation information, and creates control information to change the brightness of the camera among the cameras included in the camera unit 130 that is imaging the gaze area that the user U is fixated on. Here, the brightness change unit 334 may store camera motion image area related information that associates information indicating multiple cameras included in the camera unit 130 with information indicating the area of ​​the motion image being captured by each of the multiple cameras. In this case, the brightness change unit 334 obtains information indicating the camera associated with the information indicating the gaze area that the user U is fixated on based on the camera motion image area related information, and creates control information to change the brightness of that camera based on the obtained camera information.

[0106] Step S514 is the same as step S208 in Figure 12A, so its explanation is omitted. Subsequently, either the indoor camera 132 or the outdoor camera 134 may be configured to adjust its brightness based on the ambient light. This allows for dynamic brightness adjustment so that the area of ​​focus that the user U is looking at remains easily visible, even if the external environment changes due to driving and the ambient light changes. In Figure 12D, the case where there are two second devices 200 is described as an example, but this can also be applied to cases with three or more devices.

[0107] <Information provided during simulated ride> Next, we will explain the information provided to user U during a simulated ride. Figure 13 is a diagram illustrating an example of the information provided. The following example shows the information provided in the process of step S208 shown in Figure 12A, step S310 shown in Figure 12B, step S414 shown in Figure 12C, and step S514 shown in Figure 12D, and includes a brightness change area BAA in which the brightness of a part of the moving image V is changed. Depending on the actions of user U and the type or amount of information that can be provided to user U, the output manner of the information provided to the second device 200 will differ. In the following example, it is assumed that occupant P is authorized to provide interior images, exterior images, and telephone (voice). In the embodiment described above, the brightness adjustment unit 334 may, when it obtains either the first operation information or the second operation information from a plurality of second devices, grant priority for brightness adjustment to the second device connected to the management server 300. This allows brightness adjustment to be performed without having to perform a process to determine which second device to grant priority to, even if there is a conflict between the first operation information and the second operation information from a plurality of second devices.

[0108] In the embodiment described above, the brightness changing unit 334 may also notify each of the multiple second devices that there is a conflict when the information indicating a portion of the video image on which the first operation has been performed, included in the first operation information from the multiple second devices, conflicts. This allows each of the users U of the multiple second devices that have been notified of the conflict to communicate with each other and decide which user U's first operation information to adopt.

[0109] In the embodiment described above, when first operation information from multiple second devices conflicts, the brightness adjustment unit 334 may determine whether or not POI information related to the periphery of each of the multiple regions of a part of the video is included, based on information indicating a part of the video included in each of the multiple conflicting first operation information. For example, the brightness adjustment unit 334 obtains POI information from the provided information DB 362 of the storage unit 350. If POI information related to the periphery of one of the multiple regions of a part of the video is included, the brightness adjustment unit 334 may adjust the brightness of the region where the POI information was detected or the brightness of the POI information itself. Here, the brightness changing unit 334 may, when the first operation information contains information related to each of the multiple areas surrounding a part of the moving image, notify each of the multiple second devices that have conflicted about the first operation information of the summary of the multiple POI information. This allows each of the users U of the multiple second devices that have been notified of the summary of the multiple POI information to communicate with each other and decide which user U's first operation information to adopt.

[0110] <Summary> According to the information processing system 1 configured as described above, the information processing device processes information provided from a first device 100 mounted on a mobile body to a second device 200 used by a user U at a location different from the mobile body. The information processing device includes an acquisition unit 315 that acquires information including video data transmitted by the first device 100, a provision unit 330 that provides video data to the second device 200 based on the information acquired by the acquisition unit 315, a reception unit 332 that receives a first operation performed on a portion of the video data presented to the user U based on the video data data provided to the second device 200 by the provision unit 330, and a brightness change unit 334 that changes the brightness of a portion of the video data based on the first operation received by the reception unit 332. Convenience can be improved because the second device 200 can remotely process the video data transmitted by the first device 100 to make it easier to view. Convenience can be improved compared to when brightness can only be changed from the first device 100, because the brightness can be changed from the second device 200, which receives and views the video data, rather than from the first device 100, which distributes the video data. Since video with adjusted brightness can be distributed, overexposure and underexposure can be prevented in areas that are important to view.

[0111] In the information processing device, the first device 100 includes an imaging device. The brightness changing unit 334 creates control information to change the brightness of the imaging device based on the first operation received by the reception unit 332. Since control information can be created to change the brightness of the imaging device based on the first operation, the brightness of the imaging device provided in the first device 100 can be controlled.

[0112] In the information processing device, the reception unit 332 notifies the mobile device that a first operation to change the brightness has been performed based on the first operation it has received. A mobile device that has been notified that a first operation to change brightness has been performed can provide a topic of conversation by notifying its occupants that the first operation to change brightness has been performed. This facilitates communication among occupants. For example, if the first operation to change brightness is performed on a place with a beautiful view, it can provide a topic of conversation about that place.

[0113] In an information processing device, the imaging device adjusts the brightness of a portion of an area based on the ambient brightness. Even when the external environment changes due to the movement of the moving object and the ambient brightness changes, the brightness can be dynamically adjusted so that the location indicated by the user U is always easily visible.

[0114] The information processing device further includes a detection unit that detects the gaze region of a moving image that the user U is fixated on. The brightness changing unit 334 changes the brightness of the gaze region of the moving image based on the gaze region detected by the detection unit. Since the second device 200 can remotely process the video data transmitted by the first device 100 to make it easier to view, without requiring any manual operation, convenience can be further improved compared to cases where brightness is changed manually.

[0115] The information processing device further includes a speech recognition unit 279 that detects and performs speech recognition. The brightness adjustment unit 334 adjusts the brightness of the associated POI information if the speech recognized by the speech recognition unit 279 is related to POI information around the moving object. The second device 200 can remotely process the video data transmitted by the first device 100 to make it easier to view, without requiring any operation by the second device 200. Furthermore, it can adjust the brightness of POI information around moving objects related to audio, thus further improving convenience compared to cases where brightness is changed manually.

[0116] The information processing device further includes a detection unit that detects the gaze area of ​​a video image that the user U is fixated on. The reception unit 332 receives a second operation when the detection unit detects that a portion of the video image has been gazed on for a predetermined time or longer. The brightness change unit 334 changes the brightness of a portion of the video image based on the second operation received by the reception unit 332. The second operation includes blinking, gestures, and speech. For this reason, the number of actions included in the second operation is greater than the number of actions included in the first operation. Processing to make the video data transmitted by the first device 100 easier to view can be performed remotely when a portion of the video data is viewed for a predetermined period of time or longer and the second operation is performed.

[0117] In the information processing device, the brightness adjustment unit 334 grants priority for brightness adjustment to the second device 200 that is first connected to the information processing device among a plurality of second devices 200. Since the second device 200, which is given priority by being the first to connect, can remotely perform processing to make the video data transmitted by the first device 100 easier to view, convenience can be improved.

[0118] In the information processing device, the brightness changing unit 334 notifies each of the competing second devices 200 when the reception unit 332 detects a conflict in the first operation from multiple second devices 200. When multiple users U of the second devices are notified of a conflict, they can communicate with each other to decide which user U's first operation information to adopt. Therefore, even if the first operation information from multiple second devices conflicts, the second device 200 can remotely process the video data transmitted by the first device 100 to make it easier to view, thereby improving convenience.

[0119] In the information processing device, the brightness changing unit 334, when a first operation from multiple second devices 200 conflicts, performs a process to detect POI information around a portion of the video image where the first operation was performed by each of the conflicting users U of the multiple second devices 200, and adjusts the brightness of the region where the POI information was detected or the brightness of the POI information if POI information can be detected. Since the brightness of the area where POI information is detected or the POI information itself can be adjusted around a portion of the video image where the first operation was performed, processing can be performed to make the video image data transmitted by the first device 100 even easier to view.

[0120] In the information processing device, when the reception unit 332 detects conflicting first operations from multiple second devices 200, the brightness changing unit 334 processes the detection of POI information around a portion of the video image where the first operation was performed by each of the conflicting users U of the multiple second devices 200, and if multiple POI information can be detected, it notifies each of the users U of the multiple second devices 200 whose first operations conflicted with an overview of the multiple POI information. By detecting POI information around a portion of a video image where a first operation was performed by each of the competing second devices (user U), and notifying each of the users U of the multiple second devices (user U) of the multiple POI information summaries, convenience can be improved.

[0121] <Usage example> Information processing system 1 can be used in the following ways:

[0122] (A) A configuration in which the crew member P and user U are in a relationship such as family or friends, and the virtual drive is provided to user U. User U can converse with the crew member P about the scenery around the mobile vehicle M while viewing images.

[0123] (B) A configuration in which the crew member P is a general user and the user U is a provider of services such as directions and driving instruction. The user U can provide directions to places that are difficult to understand using a navigation device or that are not shown on a map, or provide instruction on driving operations, while observing the scenery around the mobile vehicle M.

[0124] (C) A configuration in which a crew member P is a celebrity and a user U is a regular user, and a commercially based virtual drive is provided to user U. In this case, multiple users U may be associated with one crew member P simultaneously, and for example, the transmission of voice from user U may be turned off. In addition, user U may be able to have a conversation with crew member P for a predetermined period of time by paying an additional fee.

[0125] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0126] 1. Information Processing System 100 1st device 110 First communication device 120 First microphone 125 External Sensors 130 Camera Unit 132 Indoor Cameras 134 Outdoor Camera 140 First speaker 150 User display device 160 HMI 170 First control device 190 Controlled Equipment 200 2nd device 210 Second communication device 220 Second microphone 230 detection device 232 Directional detection device 233 Gaze Area Detection Device 234 Head position detection device 236 Motion Sensors 240 Second speaker 250 Mobile Image Display Device 260 HMI 270 Second control device 300 Management Servers 310 Communication equipment 315 Acquisition Department 320 Matching Processing Unit 330 Provision Department 332 Reception Department 334 Brightness adjustment section 340 Billing Department 350 Storage section M Mobile object

Claims

1. An information processing device that processes information provided from a first device mounted on a mobile body to a second device used by a user at a location different from the mobile body, An acquisition unit that acquires information including video data transmitted by the first device, Based on the information acquired by the acquisition unit, a providing unit provides motion image data to the second device, A receiving unit that receives a first operation performed on a portion of the video image presented to the user based on the video image data provided to the second device by the providing unit, A brightness changing unit that changes the brightness of a portion of the video image based on the first operation received by the reception unit, An information processing device equipped with the following features.

2. The first apparatus includes an imaging device, The information processing apparatus according to claim 1, wherein the brightness changing unit creates control information that causes the imaging device to change the brightness based on the first operation received by the reception unit.

3. The information processing apparatus according to claim 1, wherein the receiving unit notifies the mobile body that a first operation to change brightness has been performed based on the first operation it has received.

4. The imaging device adjusts the brightness of a portion of the area based on the ambient brightness, as described in claim 2.

5. Detection unit for detecting the gaze area of ​​the video image that the user is fixated on. Furthermore, The information processing apparatus according to claim 1, wherein the brightness changing unit changes the brightness of the gaze region of the moving image based on the gaze region detected by the detection unit.

6. Speech recognition unit that detects and performs speech recognition. Furthermore, The information processing apparatus according to claim 1, wherein the brightness changing unit adjusts the brightness of the relevant POI (Point of Interest) information when the voice recognized by the voice recognition unit is related to POI (Point of Interest) information around the moving object.

7. Detection unit for detecting the gaze area of ​​the video image that the user is fixated on. Furthermore, The reception unit receives a second operation when the detection unit has observed a portion of the video image for a predetermined period of time or longer. The information processing apparatus according to claim 1, wherein the brightness changing unit changes the brightness of a portion of the moving image based on the second operation received by the reception unit.

8. The information processing apparatus according to claim 7, wherein the second operation includes blinking, gestures, and speech.

9. The information processing apparatus according to claim 7, wherein the number of actions included in the second operation is greater than the number of actions included in the first operation.

10. The information processing apparatus according to claim 1, wherein the brightness adjustment unit grants priority for brightness adjustment to the second device among a plurality of second devices that is first connected to the information processing apparatus.

11. The information processing apparatus according to claim 1, wherein the brightness changing unit notifies each of the multiple second devices that are in conflict when the first operation from multiple second devices is in conflict with the reception unit.

12. The brightness changing unit, in the event of a conflict between multiple second devices, performs a process to detect POI information around a portion of the video image where the first operation was performed by each of the users of the multiple second devices that are in conflict, and adjusts the brightness of the region where the POI information is detected or the brightness of the POI information, according to claim 1.

13. The brightness changing unit, when the reception unit detects a conflict in the first operation from a plurality of second devices, performs a process to detect POI information around a portion of the video image where the first operation was performed by each of the users of the conflicting plurality of second devices, and when multiple POI pieces of information can be detected, notifies each of the users of the plurality of second devices whose first operations conflicted of an overview of the multiple POI pieces of information, as described in claim 1.

14. The first device mounted on the mobile vehicle, A second device used by a user at a location different from the aforementioned mobile body, An acquisition unit that acquires information including video data transmitted by the first device, Based on the information acquired by the acquisition unit, a providing unit provides motion image data to the second device, A receiving unit that receives a first operation performed on a portion of the video image presented to the user based on the video image data provided to the second device by the providing unit, A brightness changing unit that changes the brightness of a portion of the video image based on the first operation received by the reception unit, An information processing system equipped with the following features.

15. An information processing device that processes information provided from a first device mounted on a mobile body to a second device used by a user at a location different from the mobile body, The first device acquires information including video data transmitted by the first device, Based on the acquired information, the second device is provided with motion image data. The second device receives a first operation performed on a portion of the video image presented to the user based on the video image data provided to the second device, An information processing method for changing the brightness of a portion of the moving image based on the first operation that has been received.

Images