Information management apparatus, information management method, and program

Abstract

To enhance presence to be given to both an occupant of a moving body and a user at a place different from the moving body.SOLUTION: An information management device according to an embodiment is an information management device that manages information provided to a first device mounted on a moving body in which an occupant boards and a second device used by a user at a place different from the moving body, the information management device including an acquisition unit that acquires information from the first device and the second device, and a provision unit that provides information for each of the first device and the second device based on the acquired information, in which the acquisition unit acquires an indoor image and an outdoor image of the moving body by the first device, the second device acquires motion information of the user, and the providing unit switches between the indoor image and the outdoor image and provides the image to the second device when the motion of the user is a predetermined motion, the providing unit sets a reference position based on a reference posture of the user, switches between the indoor image and the outdoor image and provides the image when a movement distance from the set reference position is equal to or greater than a threshold value, and the threshold value is set according to a movement direction of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information management device, an information management method, and a program. [Background technology]

[0002] Research has been conducted on communication between a device mounted on a moving object such as a vehicle and a device used in a location other than the moving object, and sharing images of scenery outside the vehicle, etc. (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-94958 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies have had the problem that both the passengers of the vehicle and users in locations other than the vehicle may not be able to feel a sufficient sense of realism.

[0005] In order to solve the above-mentioned problems, one of the objects of the present application is to provide an information management device, an information management method, and a program that can enhance the sense of realism given to both passengers of a moving vehicle and users in locations other than the moving vehicle, thereby improving user visibility and passenger safety, and ultimately further improving traffic safety and contributing to the development of a sustainable transportation system. [Means for solving the problem]

[0006] The information management device, the information management method, and the program according to the present invention employ the following configuration. (1): An information management device according to one embodiment of the present invention is an information management device that manages information provided to a first device mounted on a mobile body on which a passenger rides and a second device used by a user at a location different from the mobile body, and includes an acquisition unit that acquires information from the first device and the second device, and a provision unit that provides information for each of the first device and the second device based on the information acquired by the acquisition unit, wherein the acquisition unit acquires indoor images and outdoor images of the mobile body using the first device and acquires movement information of the user using the second device, and the provision unit switches between the indoor image and the outdoor image and provides them to the second device when the user's movement is a predetermined movement, and the provision unit sets a reference position based on the user's reference posture, and when the movement distance from the set reference position is equal to or greater than a threshold, switches from the indoor image to the outdoor image and provides them, and the threshold is set according to the direction of movement of the user.

[0007] (2): Another aspect of the present invention provides an information management device that manages information provided to a first device mounted on a mobile body on which a passenger rides and a second device used by a user at a location different from the mobile body, and includes an acquisition unit that acquires information from the first device and the second device, and a provision unit that provides information for each of the first device and the second device based on the information acquired by the acquisition unit. The acquisition unit acquires indoor and outdoor images of the mobile body using the first device, and acquires movement information of the user using the second device. When the user's movement is a predetermined movement, the provision unit switches between the indoor image and the outdoor image and provides them to the second device. The provision unit sets a reference position based on the user's reference posture, and when the distance traveled from the set reference position is equal to or greater than a threshold, the provision unit switches from the indoor image to the outdoor image and provides it. When the distance traveled from the set reference position is equal to or greater than a threshold, the provision unit switches to the outdoor image viewed from a direction corresponding to the user's movement and provides it. This information management device is.

[0008] (3): In the above aspect (1) or (2), the providing unit switches to and provides the outdoor image when the height of the user obtained from the second device becomes higher than a predetermined distance from the reference position.

[0009] (4): In the above aspect (3), the height of the user is the height of the user's head.

[0010] (5): In the above-mentioned aspects (1) or (2), the reference posture of the user is the posture when the user is seated on a chair, and the distance traveled from the reference position is the distance traveled by the user standing up.

[0011] (6): In the above aspect (5), when the user stands up and the position of the user's head moves forward, the providing unit corrects the outdoor image so that it does not move forward.

[0012] (7): In the above aspect (1) or (2), when the distance traveled in the lateral direction from the set reference position is equal to or greater than a threshold, the providing unit switches to and provides an outdoor image captured from the side of the moving body corresponding to the direction of movement.

[0013] (8): An information management method according to one embodiment of the present invention is an information management method in which a computer of an information management device that manages information provided to a first device mounted on a mobile body on which an occupant rides and a second device used by a user at a location different from the mobile body acquires information from the first device and the second device, provides information for each of the first device and the second device based on the acquired information, acquires indoor images and outdoor images of the mobile body using the first device, acquires movement information of the user using the second device, and if the user's movement is a predetermined movement, switches between the indoor image and the outdoor image and provides them to the second device, sets a reference position based on the user's reference posture, and if the movement distance from the set reference position is equal to or greater than a threshold, switches from the indoor image to the outdoor image and provides them, the threshold being set according to the direction of movement of the user.

[0014] (9): Another aspect of the present invention is an information management method in which a computer of an information management device that manages information provided to a first device mounted on a mobile body on which an occupant rides and a second device used by a user at a location different from the mobile body acquires information from the first device and the second device, provides information for each of the first device and the second device based on the acquired information, acquires indoor images and outdoor images of the mobile body using the first device, acquires movement information of the user using the second device, and if the user's movement is a predetermined movement, switches between the indoor image and the outdoor image and provides them to the second device, sets a reference position based on the user's reference posture, and if the distance moved from the set reference position is equal to or greater than a threshold, switches from the indoor image to the outdoor image and provides it, and if the distance moved from the set reference position is equal to or greater than a threshold, switches to the outdoor image viewed from a direction corresponding to the user's movement direction and provides it.

[0015] (10): A program according to one embodiment of the present invention causes a computer of an information management device that manages information provided to a first device mounted on a mobile body on which an occupant rides and a second device used by a user at a location different from the mobile body to acquire information from the first device and the second device, provide information for each of the first device and the second device based on the acquired information, acquire indoor and outdoor images of the mobile body using the first device, acquire movement information of the user using the second device, switch between the indoor image and the outdoor image and provide them to the second device when the user's movement is a predetermined movement, set a reference position based on the user's reference posture, and switch from the indoor image to the outdoor image and provide them when the distance moved from the set reference position is equal to or greater than a threshold, the threshold being set according to the direction of movement of the user.

[0016] (11): Another aspect of the present invention is a program that causes a computer of an information management device that manages information provided to a first device mounted on a mobile body on which an occupant rides and a second device used by a user at a location different from the mobile body to acquire information from the first device and the second device, and provides information for each of the first device and the second device based on the acquired information, causes the first device to acquire indoor and outdoor images of the mobile body, and the second device to acquire movement information of the user, and when the user's movement is a predetermined movement, causes the second device to switch between the indoor image and the outdoor image and provide them, sets a reference position based on the user's reference posture, and when the distance moved from the set reference position is equal to or greater than a threshold, causes the computer to switch from the indoor image to the outdoor image and provide it, and when the distance moved from the set reference position is equal to or greater than a threshold, causes the computer to switch to the outdoor image viewed from a direction corresponding to the user's movement and provide it. [Effects of the Invention]

[0017] According to the above aspects (1) to (11), it is possible to enhance the sense of realism given to both the passengers of the moving body and the users who are in a location other than the moving body. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a diagram showing the usage environment of an information processing system 1 and a management server 300. FIG. [Figure 2] FIG. 10 is a diagram showing an example of the contents of user data 360. [Figure 3] FIG. 2 is a configuration diagram of a first device 100. [Figure 4] 1 is a diagram showing an example of the arrangement of a part of a first device 100 in a moving body M. FIG. [Figure 5] FIG. 2 is a diagram illustrating the configuration of a second device 200. [Figure 6] FIG. 10 is a diagram for explaining an image corresponding to a pointing direction. [Figure 7] 2 is a diagram illustrating a first example of the functional configuration of a first control device 170 and a second control device 270. FIG. [Figure 8] FIG. 10 is a diagram illustrating a second example of the functional configuration of the first control device 170 and the second control device 270. [Figure 9] 3 is a sequence diagram showing an example of processing executed by the information processing system 1. FIG. [Figure 10] FIG. 10 is a diagram showing an example of an image IM10 showing inquiry information. [Figure 11] FIG. 10 is a diagram showing an example of an image IM20 showing a request result. [Figure 12] FIG. 10 is a diagram illustrating an example of first provided information. [Figure 13] FIG. 10 is a diagram illustrating an example of second provided information. [Figure 14] FIG. 10 is a diagram illustrating an example of third provided information. [Figure 15] FIG. 10 is a diagram illustrating an example of a functional configuration of a management server 300A in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, with reference to the drawings, an embodiment of an information management device, an information management method, and a program of the present invention will be described. The following describes an information processing system to which the information management device is applied. The information processing system includes, for example, a first device mounted on a mobile body on which a crew member rides, a second device used by a user at a location different from the mobile body, and the information management device. The mobile body is, for example, a vehicle, but may be any mobile body (for example, a ship or an aircraft) that can accommodate a crew member. Furthermore, the crew member is primarily the driver of the mobile body, but may also be a crew member other than the driver.

[0020] Between the first device and the second device, audio collected by a microphone is transmitted to the other party and played back through a speaker, creating a situation similar to a telephone conversation. Furthermore, a portion of an image captured by the camera unit of the first device is displayed on the second device, providing MR (Mixed Reality) on the second device. This allows the user of the second device to experience a virtual sensation of being aboard a moving object (a simulated ride experience) even while in a different location from the moving object. By conversing with the user experiencing the simulated ride on the moving object via the first device, the passenger can experience a sensation of actually riding the moving object together with the user. Hereinafter, as described above, the virtual sensation of a user actually riding on a moving object may be referred to as "simulated riding." The first device and the second device do not necessarily have to have a one-to-one relationship; one of multiple first devices may be matched with multiple second devices in a one-to-many relationship, allowing the system to function as an information processing system. In the latter case, for example, one passenger can communicate with multiple users simultaneously or sequentially.

[0021] <Basic configuration> FIG. 1 illustrates a usage environment of an information processing system 1 and a management server 300. The information processing system 1 includes a first device (mobile device) 100 mounted on a mobile device M carrying an occupant P, and a second device (user device) 200 used by a user U at a location different from the mobile device M (although a location nearby by chance is not excluded). The first device 100, the second device 200, and the management server 300 communicate with each other via a network NW. The information processing system 1 may or may not include the management server 300. The network NW includes, for example, at least one of the Internet, a wide area network (WAN), a local area network (LAN), a mobile communication network, a cellular network, and the like. The management server 300 is an example of an "information management device." The management server 300 may be implemented as a server device or a storage device incorporated in a cloud computing system. In this case, the functions of the management server 300 may be implemented by multiple server devices and storage devices in the cloud computing system. Furthermore, the first device 100 mounted on the moving object M may be realized by a plurality of units.

[0022] Management server 300 manages information provided to first device 100 and second device 200, and manages communication between them. Management server 300 includes, for example, communication device 310, acquisition unit 315, matching processing unit 320, provision unit 330, fee management unit 340, and storage unit 350. Acquisition unit 315, matching processing unit 320, provision unit 330, and fee management unit 340 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as an HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM and installed by inserting the storage medium into a drive device. Also, the program may be stored in the management server 300 and updated as needed by connecting to the network NW.

[0023] Communication device 310 is a communication interface for connecting to network NW. Note that communication between communication device 310 and first device 100, and communication between communication device 310 and second device 200 are performed in accordance with, for example, TCP / IP (Transmission Control Protocol / Internet Protocol).

[0024] Acquisition unit 315 acquires various types of information transmitted by first device 100, second device 200, and other external devices via network NW.

[0025] The matching processing unit 320 is realized by a processor such as a CPU executing a program (a group of instructions) stored in a storage medium. For example, when the communication device 310 receives a matching request from the user U via the second device 200 or from the occupant P via the first device 100, the matching processing unit 320 matches the matching user U with the occupant P by referring to the user data 360, and transmits the communication identification information of the first device 100 of the occupant P to the second device 200 of the matched user U using the communication device 310, and transmits the communication identification information of the second device 200 of the user U to the first device 100 of the matched occupant P. Between the first device 100 and the second device 200 that receive these, communication with higher real-time performance based on, for example, UDP (User Datagram Protocol) can be performed.

[0026] Providing unit 330 generates information to be provided to each of first device 100 and second device 200 based on various information acquired by acquiring unit 315, and transmits the generated information to the target device. Providing unit 330 also generates information indicating the processing results by matching processing unit 320 and fee information (adjustment information) managed by fee management unit 340, and provides the generated information to the target device.

[0027] The fee management unit 340 manages the fee to be charged to the user U in accordance with the information provided to the user U, and the fee to be charged to the occupant P of the moving body M in accordance with the information provided to the occupant P. The fee management unit 340 may also manage, for example, the compensation to be paid to the user U or the occupant P in accordance with the information provided by the user U or the occupant P. The fee management unit 340 may also perform processing related to the settlement of accounts for the user U and the occupant P.

[0028] The storage unit 350 may be realized by the various storage devices described above, or a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), a random access memory (RAM), etc. The storage unit 350 stores, for example, user data 360, a provided information DB 362, programs, and various other information.

[0029] FIG. 2 is a diagram showing an example of the contents of the user data 360. The user data 360 includes an occupant list 360A and a user list 360B. The occupant list 360A associates, for example, an occupant ID, which is identification information of an occupant P of a moving object M, the occupant's communication identification information (such as an IP address), a user ID, which is identification information of a user U to be matched, information about the moving object the occupant is riding in, and information about whether or not the occupant is available, set by the occupant. The moving object information includes, for example, information about devices mounted on the moving object M (mounted device information) and vehicle class information indicating the size and external shape of the moving object M. The moving object information may also include information about the current location, destination, and surrounding conditions of the moving object M (for example, traveling on a seaside road), which are transmitted from the moving object M at predetermined intervals. The user list 360B associates, for example, a user ID, the communication identification information (such as an IP address), the occupant P to be matched, and user information. The user information may include information about physical build (e.g., height and sitting height) and information that can predict physical build (e.g., age). The provision availability information is information that can or cannot be provided by the moving body M, and is set, for example, by the occupant P. The provision availability information may be set for each device installed in the moving body M, or may be set for each user U. Examples of the provision availability information include, but are not limited to, "provision of images is permitted," "provision of audio is not permitted," "provision of interior images is permitted but provision of exterior images is not permitted," "provision of images of occupants is not permitted," and "use of navigation device is not permitted." The provision availability information may also include a fee for enabling provision (service provision fee). The user data 360 may be generated in any manner, not limited to the manner shown in FIG. 2, as long as it includes this information.

[0030] The provided information DB 362 stores various types of information to be provided to the user U or the occupant P. The various types of information include, for example, map information, point of interest (POI) information, and computer-generated images (for example, CG (Computer Graphics) images of people, etc., and images of marks, symbols, icons, etc.). POI information is, for example, information about various stores, theme parks, features, etc. at each location, and may be included in the map information. The various types of information may also include audio information. The provided information DB 362 may also include advertising information. The advertising information may include, for example, advertisements related to the moving object M, advertisements related to the user U or the occupant P, and advertisements related to products and services of stores. The inserted advertising information is managed separately from indoor and outdoor information, etc., and may differ from the inserted advertising information when distributed in real time when it is later archived and distributed (for example, information about closed stores is replaced with the latest information, the menu items introduced are the latest, etc.). The advertising information is, for example, video or audio.

[0031] 3 is a configuration diagram of first device 100. First device 100 includes, for example, first communication device 110, first microphone 120, external sensor 125, camera unit 130, first speaker 140, user display device 150, HMI (Human Machine Interface) 160, and first control device 170. First control device 170 is connected to control target device 190 mounted on moving object M.

[0032] The first communication device 110 is a communication interface for communicating with a communication device 310 of the management server 300 and a second communication device 210 of the second device 200 (described later) via a network NW.

[0033] The first microphone 120 collects at least the voice of the occupant P. The first microphone 120 may be installed inside the vehicle M and have a sensitivity that allows it to collect voices outside the vehicle M, or it may include a microphone installed inside the vehicle M and a microphone installed outside the vehicle M. Hereinafter, voice information acquired by a microphone installed inside the vehicle may be referred to as "indoor voice information." The voice collected by the first microphone 120 is transmitted to the management server 300 or the second device 200 by the first communication device 110 via the first control device 170, for example. Furthermore, if it is not possible to set a microphone installed outside the vehicle M, indoor voice information may be processed to generate pseudo-outdoor voice information based on driving information (vehicle speed, acceleration / deceleration, road vibration, etc.) and the surrounding driving environment. Furthermore, the positional relationship of the speaker relative to the vehicle M (whether the speaker is inside or outside the vehicle) may be recorded, and the collected voice may be processed according to that positional relationship.

[0034] The external sensor 125 detects the position of an object around the moving body M. The external sensor 125 is, for example, a radar device, a LIDAR (Light Detection and Ranging) sensor, or any other type of proximity sensor. The radar device emits radio waves such as millimeter waves around the moving body M and detects radio waves (reflected waves) reflected by the object to detect at least the object's position (distance and direction). The radar device may detect the position and speed of an object using an FM-CW (Frequency Modulated Continuous Wave) method. The LIDAR sensor irradiates light (or electromagnetic waves with a wavelength similar to light) around the moving body M, measures the scattered light, and detects the distance to the object based on the time between light emission and light reception. The irradiated light is, for example, pulsed laser light. The radar device or LIDAR sensor is attached to any location on the moving body M. The external sensor 125 may also detect surrounding objects using images captured by the outdoor camera 134 of the camera unit 130.

[0035] The camera unit 130 includes, for example, an indoor camera 132 and an outdoor camera 134. The first speaker 140 outputs the voice uttered by the user U, which is acquired via the first communication device 110. Details of the arrangement of the camera unit 130 and the first speaker 140 will be described later with reference to FIG.

[0036] The user display device 150 virtually displays the user U as if the user U were present inside the room of the moving object M. For example, the user display device 150 makes a hologram appear, or displays the user U on a part of the moving object M that corresponds to a mirror or window.

[0037] The HMI 160 is a touch panel, a voice response device (agent device), etc. The HMI 160 receives various instructions from the occupant P to the first device 100 and provides various information to the occupant P.

[0038] The first control device 170 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (a set of instructions), and the processor controls each part of the first device 100 by executing the set of instructions.

[0039] The control target device 190 is, for example, an in-vehicle device such as a navigation device mounted on the moving body M that provides route guidance to a destination, or a driving assistance device that controls one or both of the steering and speed of the moving body M to assist the occupant P in driving. The control target device 190 includes, for example, a seat drive device that can adjust the seat position (front / back / left / right), orientation, and height. When viewing video using the second device 200 and the camera unit 130 of the first device 100 is attached to a seat, the influence on the video can be suppressed by prohibiting seat movement. Even if seat movement is permitted, processing such as field of view conversion may be performed to prevent the video from being affected by seat movement. Furthermore, if the user U requests to view video outside the field of view of the current camera unit 130, the seat drive device may be controlled as a request from the second device 200.

[0040] FIG. 4 is a diagram showing an example of the arrangement of a portion of the first device 100 in a moving object M. The indoor camera 132 is attached, for example, via an attachment 132A to the neck pillow of the passenger seat S2 (an example of a "predetermined seat"), and is provided at a position slightly separated from the backrest of the passenger seat S2 in the traveling direction of the moving object M. The indoor camera 132 has a wide-angle lens and is capable of capturing an image of the range indicated by the hatched area 132B in the figure. The indoor camera 132 is capable of capturing not only the interior of the moving object M but also the exterior through the window. In the following description, the passenger seat S2 is considered to be the predetermined seat, but the predetermined seat may be another seat such as a rear seat.

[0041] The outdoor camera 134 includes, for example, a plurality of subsidiary outdoor cameras 134-1 to 134-4. By combining images captured by the plurality of subsidiary outdoor cameras 134-1 to 134-4, an image such as a panoramic image capturing the outside of the moving object M can be obtained. Instead of (or in addition to) these, the outdoor camera 134 may include a wide-angle camera provided on the roof of the moving object M. Note that a camera capable of capturing an image of the area behind the passenger seat S2 may be added as the indoor camera 132, and the moving object image described below may be generated as a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 by the first control device 170, or may be generated as a 360-degree panoramic image by appropriately combining images captured by the indoor camera 132 and the outdoor camera 134.

[0042] The first speaker 140 outputs the voice of the user U acquired via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, the sub-first speaker 140-1 is located in the center of the instrument panel, the sub-first speaker 140-2 is located at the left end of the instrument panel, the sub-first speaker 140-3 is located at the right end of the instrument panel, the sub-first speaker 140-4 is located at the bottom of the left door, and the sub-first speaker 140-5 is located at the bottom of the right door. When the first control device 170 outputs the voice of the user U to the first speaker 140, for example, the first control device 170 outputs the voice from the sub-first speaker 140-2 and the sub-first speaker 140-4 at approximately the same volume and turns off the other sub-first speakers, thereby localizing the sound image so that the voice can be heard from the passenger seat S2 by the occupant P seated in the driver's seat S1. Furthermore, the method of localizing the sound image is not limited to adjusting the volume, and may be performed by shifting the phase of the sound output from each sub-first speaker. For example, to localize the sound image so that the voice is heard from the left side, the timing of outputting the sound from the left sub-first speaker may be slightly earlier than the timing of outputting the same sound from the right sub-first speaker.

[0043] Furthermore, when the first control device 170 outputs the voice of the user U to the first speaker 140, it may localize a sound image so that the voice can be heard by the occupant P from a position at a height corresponding to the height of the head of the user U sitting on the passenger seat S2, and output the voice emitted by the user U to the first speaker 140. In this case, the first speaker 140 needs to have multiple sub-first speakers 140-k (k is a multiple natural number) at different heights.

[0044] 5 is a configuration diagram of second device 200. Second device 200 includes, for example, second communication device 210, second microphone 220, detection device 230, second speaker 240, mobile image display device 250, HMI 260, and second control device 270. Detection device 230 includes, for example, orientation direction detection device 232, head position detection device 234, and motion sensor 236.

[0045] The second communication device 210 is a communication interface for communicating with the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0046] The second microphone 220 collects the voice uttered by the user U. The voice collected by the second microphone 220 is transmitted to the first communication device 110 by the second communication device 210 via the second control device 270, for example.

[0047] The orientation direction detection device 232 is a device for detecting the orientation direction. The orientation direction is a direction based on the direction of the user U's face or line of sight, or both. In the following, the orientation direction is assumed to be an angle in a horizontal plane, that is, an angle that does not have an up-down component. However, the orientation direction may also be an angle that includes an up-down component. The orientation direction detection device 232 may include a physical sensor (e.g., an acceleration sensor, a gyro sensor, etc.) attached to the VR goggles described below, an infrared sensor that detects multiple positions of the user U's head, or a camera that captures an image of the user U's head. In either case, the second control device 270 calculates the orientation direction based on information input from the orientation direction detection device 232. Since various techniques for this are publicly known, detailed description will be omitted.

[0048] The head position detection device 234 is a device for detecting the position (height) of the head of the user U. For example, one or more infrared sensors or optical sensors provided around the chair on which the user U sits may be used as the head position detection device 234. In this case, the second control device 270 detects the position of the head of the user U based on the presence or absence of a detection signal from the one or more infrared sensors or optical sensors. The head position detection device 234 may also be an acceleration sensor attached to the VR goggles. In this case, the second control device 270 detects the position of the head of the user U by integrating the output of the acceleration sensor minus the gravitational acceleration. The head position information thus acquired is provided to the second control device 270 as height information. The user's head position may also be acquired based on the user U's operation of the HMI 260. For example, the user U may input their height numerically into the HMI 260, or may input their height using a dial switch included in the HMI 260. In these cases, the head position, i.e., height information, is calculated from the height. Furthermore, the user U may input a discrete value such as physique: large / medium / small into the HMI 260 instead of a continuous value. In this case, height information is acquired based on information indicating the physique. Furthermore, the head height of the user U may be acquired simply based on the physique of an average adult (which may be separated by gender) without acquiring the head height of the user specifically.

[0049] The motion sensor 236 is a device for recognizing gesture operations performed by the user U. For example, a camera capturing an image of the upper body of the user U is used as the motion sensor 236. In this case, the second control device extracts feature points (fingertips, wrists, elbows, etc.) of the user U's body from the image captured by the camera, and recognizes the gesture operations of the user U based on the movements of the feature points.

[0050] The second speaker 240 outputs the voice uttered by the occupant P, which is acquired via the second communication device 210. The second speaker 240 has a function of changing the direction from which the voice is heard, for example. The second control device 270 causes the second speaker to output the voice so that the user U can hear the voice from the position of the occupant P as seen from the passenger seat S2. The second speaker 240 includes a plurality of sub-second speakers 240-n (n is a plurality of natural numbers), and sound image localization may be performed by the second control device 270 adjusting the volume of each sub-second speaker 240-n, or, if headphones are attached to the VR goggles, the sound image localization may be performed using the function of the headphones.

[0051] The mobile object image display device 250 displays an image corresponding to the orientation direction as seen from the passenger seat among the images captured by the camera unit 130 (which may be images that have undergone the aforementioned combining process, and are hereinafter referred to as mobile object images). FIG. 6 is a diagram for explaining an image corresponding to the orientation direction. In the example of this figure, the VR goggles 255 include an orientation direction detection device 232, a physical sensor serving as a head position detection device 234, and the mobile object image display device 250. The second control device 270, for example, determines the head center of the user U or the center of the VR goggles 255 as Ω, and detects the direction in which the VR goggles 255 are facing as the orientation direction φ using a pre-calibrated direction as the reference direction. Various techniques for this function are already known, so detailed description will be omitted.

[0052] The moving body image display device 250 displays, toward the user U, an image A2 of the moving body image A1 (which has an angle of about 240 degrees in the figure, but the angle of view may be expanded by combining processing as described above) within an angle range of plus or minus α centered on the directional direction φ.

[0053] The HMI 260 is a touch panel, a voice response device (agent device), the above-mentioned switch, etc. The HMI 260 receives various instructions from the occupant P to the second device 200.

[0054] The second control device 270 includes, for example, a processor such as a CPU and a storage medium connected to the processor and storing a program (a group of instructions), and the processor executes the group of instructions to control each unit of the second device 200. Note that the second device 200 may be configured such that all of the functions shown in Fig. 5 are integrated into the VR goggles.

[0055] <Functional Configuration of First Control Device 170 and Second Control Device 270> The following describes the functional configurations of the first control device 170 and the second control device 270. Note that in the first and second examples shown below, the generation and transmission of information provided between the first control device 170 and the second control device 270 is mainly performed, and the management server 300 manages the matching of a pseudo-ride between an occupant P and a user U. [Example 1] FIG. 7 is a diagram showing a first example of the functional configuration of the first control device 170 and the second control device 270. In the first example, the first control device 170 includes a matching request / acceptance unit 171, an audio output control unit 172, an image transmission unit 173, and an on-board device cooperation unit 174. The second control device 270 includes a matching request / acceptance unit 271, an audio output control unit 272, an orientation direction detection unit 273, a head position detection unit 274, a gesture input detection unit 275, an image editing unit 276, and a moving object image display control unit 277. These functional units are realized, for example, by a processor such as a CPU executing a program (a group of instructions). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware.

[0056] The matching request / acceptance unit 171 accepts input of a matching request (pseudo ride request) from the occupant P using the HMI 160 and transmits it to the management server 300, and also accepts input of acceptance of a matching request received from the management server 300 using the HMI 160 and transmits it to the management server 300. The matching request / acceptance unit 171 controls the first communication device 110 so that the second device 200 of the user U with whom a match has been established becomes the communication partner. Furthermore, the matching request / acceptance unit 171 may, for example, accept a setting of provision permission information from the occupant P, which indicates whether or not to permit the provision of information, when accepting a matching request obtained from the management server 300.

[0057] The audio output control unit 172 controls the first speaker 140 as described above.

[0058] After communication with second device 200 is initiated, image transmission unit 173 transmits moving object image A1 to second device 200 using first communication device 110. Moving object image A1 may include, for example, an indoor image captured by indoor camera 132 and / or an outdoor image captured by outdoor camera 134.

[0059] The on-board device cooperation unit 174 controls the control-target device 190 based on the instruction signal input from the second device 200. Note that the on-board device cooperation unit 174 may perform control when the occupant P permits the user U to control the control-target device 190.

[0060] The matching request / acceptance unit 271 accepts input of a matching request (pseudo-ride request) from the user U using the HMI 260 and transmits it to the management server 300, and also accepts input of acceptance of a matching request received from the management server 300 using the HMI 260 and transmits it to the management server 300. The matching request / acceptance unit 271 controls the second communication device 210 so that the first device 100 of the occupant P with whom a match has been established is set as the communication partner. Furthermore, the matching request / acceptance unit 271 may, for example, present to the user U what information will be provided based on the matching result (request result) obtained from the management server 300 so that the user U can confirm it, and then accept an instruction from the user U as to whether or not to take a pseudo-ride.

[0061] The audio output control unit 272 controls the second speaker 240 as described above.

[0062] The orientation direction detection unit 273 detects the orientation direction φ based on the output of the orientation direction detection device 232. The head position detection unit 274 detects the height of the head of the user U based on the output of the head position detection device 234. The head position may be expressed as three-dimensional coordinates, or the head height may simply be detected as the head position. The gesture input detection unit 275 detects a gesture input by the user U based on the output of the motion sensor 236.

[0063] The image editing unit 276 performs processing to cut out, for example, an image A2 corresponding to the orientation direction φ as seen from the passenger seat S2 from the moving object image A1 (FIG. 6). Note that in the example of FIG. 6, the orientation direction φ corresponding to the left-right movement of the head of the user U is shown, but processing to cut out an image corresponding to the orientation direction as seen from the up-down movement of the head may also be performed. The moving object image display control unit 277 causes the moving object image display device 250 to display the image A2 cut out by the image editing unit 276. At this time, the image editing unit 276 may cause the moving object image display device 250 to display an image corresponding to the orientation direction φ as seen from the height indicated by the height information of the head of the user U. For example, the image editing unit 276 switches between displaying an indoor image or an outdoor image depending on the distance of the user U's head from a reference position (for example, the amount of upward movement or the amount of lateral movement).

[0064] [Example 2] Fig. 8 is a diagram showing a second example of the functional configuration of first control device 170 and second control device 270. Compared to the first example in Fig. 7, it differs in that first control device 170 includes image editing unit 175, and second control device 270 does not include image editing unit 276 but includes directivity direction transmission unit 278. The other components basically have the same functions as in the first example, so repeated explanations will be omitted.

[0065] Directivity direction transmitter 278 transmits directivity direction φ detected by directivity direction detector 273 to first device 100 using second communication device 210.

[0066] The image editing unit 175 performs a process of cutting out an image A2 corresponding to the orientation direction φ (transmitted from the second device 200) as seen from the passenger seat from the moving object image A1 (FIG. 6). At this time, the image editing unit 175 may acquire height information of the head of the user U from the second device 200, and perform a process of cutting out an image A2 corresponding to the orientation direction φ as seen from the height indicated by the height information.

[0067] In the second example, image transmission unit 173 transmits image A2 cut out by image editing unit 175 to second device 200 using first communication device 110. Then, moving object image display control unit 277 causes moving object image display device 250 to display image A2 transmitted from first device 100.

[0068] <Processing related to pseudo-ride> Next, in the embodiment, a series of processes executed by the information processing system 1 when a user U takes a pseudo-ride on a moving body M (a passenger P allows the user U to take a pseudo-ride) will be specifically described. FIG. 9 is a sequence diagram showing an example of the processes executed by the information processing system 1. The example of FIG. 9 shows the first device 100 mounted on the moving body M on which the passenger P is riding, the second device 200 used by the user U, and the processing in the management server 300. It is also assumed that the user U and the passenger P are registered in advance in the management server 300 and registered in the user data 360 in order to receive services provided by the information processing system 1.

[0069] In the example of FIG. 9, the second device 200 generates a request for a simulated ride (a simulated ride request) (step S100) and transmits the generated simulated ride request to the management server 300 (step S102). The simulated ride request may include, for example, information specifying a passenger P in advance, or may include information specifying the current location or destination of a moving vehicle M while it is moving, or the surrounding conditions. More specific examples of a simulated ride request include requests such as "I would like to take a simulated ride in a moving vehicle driven by Mr. A" or "I would like to take a simulated ride in a moving vehicle traveling on a road along the sea." These are examples of the above-mentioned matching requests.

[0070] The matching processing unit 320 of the management server 300 receives the simulated experience request from the second device 200 and performs matching processing based on the user data 360 (step S104). Then, when there is one or more matched moving bodies M, the providing unit 330 generates inquiry information inquiring of all moving bodies M as to whether or not to allow the user to ride in the simulated experience (step S106), and transmits the generated information to the first device 100 of the matched moving bodies M (step S108).

[0071] The first device 100 displays the inquiry information transmitted from the management server 300 on the screen (step S110). FIG. 10 is a diagram showing an example of an image IM10 showing the inquiry information. Note that the display content and layout, etc., included in the image IM10 are not limited to this. The same applies to the subsequent descriptions of other images. The image IM10 includes, as an inquiry screen, for example, a user information display area AR11, a provision / non-provision selection area AR12, and a switch display area AR13. The user information display area AR11 displays information about the user U who has made a pseudo-ride request. The information about the user U may be a user ID or other information that allows the occupant P to identify the user U.

[0072] The provision permission selection area AR12 displays an image for selecting the content of information that the occupant P should provide to the user U for each piece of information. In the example of FIG. 10, a screen is displayed for selecting whether to allow the provision of information to the occupant P for interior images, exterior images, phone calls (voice), and navigation information (information provided by a navigation device). Phone calls (voice) are an example of "interior voice information." For example, if the occupant P is willing to allow phone calls and image provision to the user U but does not want the user U to know the specific destination, the occupant P may allow phone calls and provision of interior and exterior images, but refuse provision of navigation information. This allows, for example, when the user U is actually riding in the vehicle M, the destination can easily be revealed from the navigation information. However, by not providing navigation information during a simulated ride, the user U can head to the destination without the user U knowing, giving the user U a surprise experience unique to the simulated ride. Also, for example, if the user U is not very close to the user U, refusing to provide phone calls (voice) allows the user U to drive without having to talk to them. The information for selecting whether to provide is not limited to the example in FIG. 10 and may include, for example, information for selecting whether to allow the user U to operate the navigation system or other control target devices 190, or information for selecting whether to provide an image including the occupant P. The information for providing selected in the provision selection area AR12 may be changed during the simulated ride. The switch display area AR13 includes, for example, an icon IC11 and an icon IC12. The icon IC11 is a switch that accepts permission from the occupant P to allow the user U to simulate a ride. Conversely, the icon IC12 is a switch that accepts the occupant P to reject the user U's simulated ride.

[0073] Returning to FIG. 9, the first device 100 accepts the provision permission setting of the user U using the image IM10 shown in FIG. 10 or the like (step S112) and transmits the accepted provision permission setting information to the management server 300 (step S114). Note that the processing of steps S110 to S112 may be displayed on a screen when the moving body M stops to ensure safety during driving, and instead of (or in addition to) the screen display, the user may be notified of the pseudo-ride request or the provision permission setting may be accepted by voice. The providing unit 330 of the management server 300 generates a request result (answer result) based on the provision permission setting information obtained from the first device 100 (step S116) and transmits information indicating the generated request result to the second device 200 (step S118).

[0074] The second device 200 receives the request result from the management server 300 and displays information indicating the received request result on the screen (step S120). FIG. 11 is a diagram showing an example of an image IM20 indicating the request result. The image IM20 includes, as a request result screen, for example, an occupant information display area AR21, a provided information list display area AR22, and a switch display area AR23. This image IM20 may be generated for each occupant P (moving object M) who is permitted to pseudo-ride, and an image indicating a list of the request results may also be displayed.

[0075] The occupant information display area AR21 displays information about the occupant P who has approved the pseudo-ride request. The information about the occupant P may be the occupant ID or other information that allows the user U to identify the occupant P. The provided information list display area AR22 displays a list of information approved by the occupant P. The provided information list display area AR22 may also display a fee (service usage fee) based on the approved information. The switch display area AR23 includes, for example, icons IC21 to IC23. Icon IC21 is a switch that accepts pseudo-riding on the mobile body M of the occupant P displayed in the occupant information display area AR21 using the provided information displayed in the provided information list display area AR22. Conversely, icon IC22 is a switch that accepts not pseudo-riding. Icon IC23 is a switch that accepts searching for other occupants (mobile bodies). When searching for other occupants, the user may select from other occupants included in the request results, or may input other request conditions to have the management server 300 perform the matching process again.

[0076] Returning to FIG. 9, the second device 200 receives a selection instruction from the user U via the image IM20 shown in FIG. 11 (step S122) and transmits the received instruction information to the management server 300 (step S124). The following description will be given assuming that the user U has received an instruction to simulate a ride on the moving body M of the occupant P. The management server 300 manages the communication status between the first device 100 and the second device based on the instruction from the second device 200 (step S126) and transmits information indicating that the simulated ride will be executed to the first device 100 and the second device 200 (steps S128 and S130). Note that if the user U selects not to simulate a ride in the processing of steps S126 to S130, the providing unit 330 of the management server 300 notifies the user U by transmitting information indicating that to the first device 100.

[0077] Next, the first device 100 and the second device 200 communicate with each other to execute a simulated ride (step S132). When the simulated ride ends, the first device 100 and the second device 200 transmit information indicating the end of the communication to the management server 300 (steps S134 and S136). In the process of step S132, the second device 200 may notify the occupant P from the user U before the information from the first device 100 is provided to the second device 200. This allows the user U to permit communication at their own timing and start providing information such as a phone call or video streaming. Next, the fee management unit 340 of the management server 300 performs a settlement process for the fees related to the use of the services of the first device 100 and the second device 200. Specifically, the fee management unit 340 varies the fees for providing information based on at least one of the types of the occupant P, user U, and mobile object M to which information is provided, the type of information that can be provided to the user U, and the amount of information. For example, if the occupant P or user U is a celebrity, the usage fee may be set higher than the normal fee, or if the moving object M is a new car or a rare vehicle (rare car), the fee may be set higher than the normal fee. The fee management unit 340 then performs a settlement process corresponding to the set fee. This completes the processing of this sequence. Note that the processing of step S112 shown in FIG. 9 (provision availability setting process) may be set in advance by the occupant P for each registered user U.

[0078] <Information provided during the simulated ride> Next, several examples of information provided to the user U during a simulated ride will be described. The following examples are information provided in the processing of step S132 shown in Fig. 9, and the output manner of the information provided to the second device 200 varies depending on, for example, the actions of the user U and the type or amount of information that can be provided to the user U. Note that in the following examples, it is assumed that the occupant P has permitted the provision of interior images, exterior images, and calls (voice).

[0079] <Example of the first provided information: indoor image> FIG. 12 is a diagram illustrating an example of the first provided information. The example of FIG. 12 illustrates a user U seated in a chair CH in a location other than the moving object M (e.g., a room in one's home) and an example of an image IM30 provided from the second device 200 (integrated with VR goggles in the example of FIG. 12) used by the user U. The state of the user U (seated in the chair CH) illustrated in the example of FIG. 12 represents an initial state immediately after the virtual ride is performed, and the first device 100 and the second device 200 recognize the position of the user U at this time (reference posture) as a reference position. The reference position is a position where the user U is assumed to be seated in the passenger seat S2. The reference position may be corrected by the first control device 170 to match the height of the indoor camera 132 to the sitting height of the user U by driving a seat drive device based on physique information, such as height or sitting height information, of the user U registered in the user information of the user data 360 (so-called zero-point correction). The reference position may be corrected based on a correction instruction from the user U who views the image IM30. The reference position may also include orientation information when the user U is seated on the chair CH and facing forward.

[0080] For example, when the user U (second device 200) is seated in a chair CH and facing in any direction from the reference position, an image captured by the interior camera 132 of the first device 100 according to that direction is provided as the interior image IM30. This allows the user U to obtain an interior image in any direction as seen from the passenger seat S2. In addition, audio information of the voice of the occupant P and sounds inside and outside the moving object M, acquired by the first microphone 120, is provided to the user U. By using the image IM30 and audio information to converse with the occupant P, the user U can enjoy a conversation as if he or she were actually riding in the moving object M.

[0081] In the information processing system 1, the user U can view any direction from the passenger seat S2. However, the user U may be restricted in the viewable direction, for example, by a matching agreement or a setting by the occupant P to allow or disallow viewing. For example, the occupant P may request that the view in the direction of travel of the moving object M or the view on the opposite side of the driver's seat S1 be provided, but that the occupant P not display his or her own image. This is intended to address the needs of the occupant P and the user U, who are not family or friends, but who want to confirm the driving feel of the moving object M or view a desired cityscape. In this case, the matching processing unit 320 or the providing unit 330 of the management server 300 sets such a restriction based on the matching result, and the first control device 170 or the second control device 270 masks the angle range that is not to be viewed or corrects the orientation direction φ so that it does not point in the restricted direction, depending on the setting. Furthermore, information regarding such restrictions may be set by the first device 100, since it concerns the privacy of the occupant P. Furthermore, the information that has been refused to be provided may be provided after the occupant P gives his / her consent, provided that the user U pays an additional fee.

[0082] Furthermore, the mobile object image display device 250 may replace a portion of an image captured by the camera unit 130 that shows a predetermined object inside the mobile object M with an image (CG image) rendered by computer processing and display the image. The area OB1 shown in FIG. 12 is a display device that performs navigation display, etc., and is an example of a "predetermined object." If an image captured on the display screen of the display device is displayed as is, the image may be blurred or visibility may be reduced due to light reflection. For this reason, the mobile object image display device 250 may acquire data for configuring the display screen of the display device or image data rendered by computer processing in the mobile object M from the first device 100, and then display an image re-rendered from the data by computer processing, or the acquired image data, by embedding it in an image (edited version) captured by the camera unit 130. In this case, the position of the predetermined item in the interior of the moving object M is shared in advance between the first device 100 and the second device 200, and the moving object image display control unit 277 determines whether the predetermined item is included in the image to be displayed on the moving object image display device 250, for example, based on the orientation direction φ, and performs image replacement as described above if it is determined that the predetermined item is included. Furthermore, the "predetermined item" may be the head or face of the occupant P. In this case, a CG image such as an avatar may be changed depending on the display of the occupant P. Note that at least a portion of the image data (CG image) described above may be provided from the provided information DB of the management server 300.

[0083] <Example of second provided information: Outdoor image (overhead image)> FIG. 13 is a diagram illustrating an example of the second provided information. The example in FIG. 13 shows an image IM40 provided when the height of the head of the user U (second device 200) is increased by a distance D1 from a reference position. In this case, the first device 100 determines whether the moved distance D1 is equal to or greater than a threshold, and if it is equal to or greater than the threshold, switches the display from the indoor image captured by the indoor camera 132 to the outdoor image captured by the outdoor camera 134. In the example in FIG. 13, since the distance D1 is equal to or greater than the threshold, the outdoor image captured from above the moving object M is displayed. This allows the user U to have a simulated riding experience that is not possible in an actual ride, such as poking his / her head out from above the moving object M and looking at the surrounding scenery.

[0084] The threshold value described above may be a fixed value, or may be variably set depending on the vehicle class of the moving object M. For example, based on distance D1 and vehicle class information included in the moving object information stored in user data 360, first device 100 switches to an exterior image when distance D1 from the reference position is equal to or greater than the vehicle class (external shape of the vehicle). As a result, for example, when the user U is simulated riding in a large vehicle such as a bus, the image remains as an interior image even if the user U stands up in the same way (moves by distance D1 shown in FIG. 13), so that more realistic information is provided, which makes it less likely that the user U will feel uncomfortable. The threshold value described above may also be set taking into account the physique of user U.

[0085] When user U stands up from chair CH, due to the structure of the human body, the position of the head moves further forward than when sitting. However, since user U thinks that he or she is simply standing up straight, moving the outdoor image forward in accordance with the forward movement of the head when standing up may cause discomfort to user U. Therefore, first device 100 may perform correction processing so that the outdoor image does not move forward when the position of user U's head moves forward a predetermined distance as a result of standing up.

[0086] In the second example of information to be provided, an image IM42 showing navigation information or an image IM44 showing POI information (e.g., store information) corresponding to a nearby building may be superimposed on the image IM40, thereby providing more detailed information.

[0087] Furthermore, when switching from an indoor image to an outdoor image, the first device 100 may perform correction to change the loudness (volume) of sounds provided to the second device 200 depending on the type of sound. For example, when switching from an indoor image to an outdoor image, the first device 100 adjusts each sound based on characteristic information, such as the frequency of the sound, contained in the indoor audio information acquired by the first microphone 120 installed in the vehicle. For example, the first device 100 performs effect processing to increase the volume of sounds (e.g., noise) emitted from outside the vehicle and / or decrease the volume of sounds (e.g., the voice of the occupant P or indoor music) emitted from inside the vehicle and contained in the indoor audio information. This allows the user U to feel as if he or she is actually sticking his or her head out of the vehicle through the image IM40 and the audio. Furthermore, when an outdoor image is being provided and the user U is seated in the chair CH again (when the distance D1 from the reference position becomes less than the threshold value), the first device 100 switches to an indoor image and performs audio effect processing to reverse the volume of the sound from the above (or returns to the original audio obtained by the first microphone 120) and provides it to the second device 200.

[0088] <Example of third provided information: Outdoor image (side image)> Fig. 14 is a diagram for explaining an example of the third provided information. In comparison with the example of Fig. 13, Fig. 14 shows an example in which an outdoor image to the side of a moving object M is provided instead of an outdoor image above the moving object M. Therefore, Fig. 14 will be mainly described focusing on the device points in Fig. 13.

[0089] The example of FIG. 14 shows an image IM50 provided when the position of the head of the user U (second device 200) moves a distance D2 laterally from a reference position. In this case, the first device 100 determines whether the moved distance D2 is equal to or greater than a threshold, and if so, switches the display from the indoor image captured by the indoor camera 132 to the outdoor image captured by the outdoor camera 134. In the example of FIG. 14, because the distance D2 is equal to or greater than the threshold, an outdoor image captured from the side of the moving object M in the direction of movement (more specifically, an image of the front, since the moving object M is facing slightly forward) is displayed. This allows the user U to have a simulated riding experience, as if he or she were sticking his or her head out the side of the moving object M (e.g., from a side window) and looking at the surrounding scenery. The threshold may be a fixed value or may be variably set according to the vehicle class information of the moving object M.

[0090] Note that when the user U moves sideways, the position of the head may shift, as in the case of a standing-up motion, so the first device 100 may perform correction so that there is no shift in the image when the shift amount is within a predetermined range.

[0091] Furthermore, in the third example of information to be provided, image IM40 may display information (e.g., image IM52 or a numerical value indicating the distance) indicating the distance to an obstacle OBJ, such as a curb, detected by external sensor 125. In this case, first device 100 acquires information from external sensor 125 that recognizes objects around moving body M, and when providing an exterior image to second device 200, provides image IM52 indicating the distance between the object (obstacle OBJ) detected by external sensor 125 and moving body M. This allows user U to view, for example, exterior image IM50 of a blind spot near the shoulder of the road that occupant P cannot see directly from the driver's seat, and inform occupant P of the content of the image by conversation, thereby reducing contact with obstacle OBJ and enabling more appropriate driving.

[0092] Furthermore, instead of (or in addition to) the information indicating the distance, when the distance between the moving body M and the object (obstacle OBJ) is within a predetermined distance, the first device 100 may provide the second device 200 with information (for example, alert information) indicating that the moving body M and the object are approaching. This can prompt the user U to check the surroundings using the outside-of-vehicle image, and can allow the occupant P to more accurately grasp the area near the blind spot.

[0093] The provided information described above can provide images and sounds of a simulated ride in accordance with a predetermined action of the user U. In the above example, the images (indoor image, outdoor image) and sounds (with or without effect processing) are switched in accordance with the user's actions, but the images and sounds may be switched in accordance with a mechanical switch (operation unit) such as a switching button provided in the second device 200, a gesture action of the user U, an operation unit or gesture action provided in the first device 100, or the like, regardless of the action of the user U.

[0094] The above-described first to third provided images may be provided to the first device 100. This allows the occupant P to more accurately grasp what image the user U is currently looking at, and allows for more appropriate conversation.

[0095] Furthermore, in the embodiment, the first device 100 may generate an image in which the user U appears to be sitting in the passenger seat S2 (or a rear seat) and display the generated image on the rearview mirror display device or a window. In this case, the first control device 170 generates a person image in which the user U or a character corresponding to the user U appears to be sitting in the passenger seat S2 and displays the generated image on the rearview mirror display device. In this case, the posture of the person image may be changed according to the movement of the user U obtained from the second device 200. Furthermore, when the first control device 170 is displaying a person image on the rearview mirror display device 152, the first control device 170 may control the first speaker 140 to output audio from near the passenger seat S2. This audio is, for example, audio uttered by the user U into the second microphone 220, transmitted by the second communication device 210, and acquired via the first communication device 110. For example, the first control device 170 causes the first speaker 140 to output a sound image localized so that the sound can be heard from the passenger seat S2 for the occupant P in the driver's seat.

[0096] <Modification> In the above-described information processing system 1, at least some of the functions of the first device 100 and the second device 200 may be provided in the management server 300, and at least some of the functions of the management server 300 may be provided in the first device 100 or the second device 200.

[0097] For example, the functions of the image editing unit 276 or (image editing unit 175) and the functions of the moving object image display control unit 277 may be provided on the management server side. FIG. 15 is a diagram illustrating an example of the functional configuration of a management server 300A in a modified example. In the example of FIG. 15, the management server 300A includes, for example, a communication device 310, an acquisition unit 315, a matching processing unit 320, a provision unit 330A, a fee management unit 340, and a storage unit 350. Compared to the management server 300 shown in FIG. 1, the management server 300A includes a provision unit 330A instead of the provision unit 330. Therefore, the following description will mainly focus on the function of the provision unit 330A, and descriptions of the other components will be omitted. In addition, in the modified example shown in FIG. 15, for example, information acquired by the first device 100 and the second device 200 is acquired by the acquisition unit 315 of the management server 300A, and various information corresponding to each device is provided by the provision unit 330.

[0098] For example, based on the provision availability information set by the occupant P, the providing unit 330A provides the second device 200 with information that is set as available from among the indoor image, the outdoor image, and the indoor audio information. For example, the providing unit 330A has the same functions as the providing unit 330, and further includes an image editing unit 332 and a moving object image display control unit 334. The image editing unit 332 performs the same processing as the image editing unit 276 (or the image editing unit 175) described above. For example, the image editing unit 332 acquires images captured by the first device 100, and based on the motion information (head position information) of the user U detected by the detection device 230 of the second device 200, switches between the indoor image and the outdoor image depending on whether or not a predetermined motion has been performed, and provides the images to the second device 200. In this case, the image editing unit 332 may generate images in which other information (e.g., navigation information or POI information) is superimposed on the indoor image or the outdoor image.

[0099] Furthermore, the moving object image display control unit 334 provides the images generated by the image editing unit 332 to the second device 200, and controls the switching of images and sounds based on the actions of the user U. For example, the moving object image display control unit 334 may acquire indoor sound information from the first device 100, and perform the above-mentioned sound effect processing or the like in accordance with the switching of images, and provide sounds corresponding to the images provided to the second device 200.

[0100] Furthermore, when providing an outdoor image to the second device 200, the providing unit 330A may provide information regarding the distance between an object (e.g., an obstacle) detected by the external sensor 125 and the moving body M. Furthermore, when the distance between the moving body M and the object is within a predetermined distance, the providing unit 330A may provide the second device 200 with information indicating that the distance between the moving body M and the object is approaching. Furthermore, when the user U requests to acquire information from the first device 100, the providing unit 330A may provide the occupant P with information indicating the request and acquire provision availability information set by the occupant P. Furthermore, the providing unit 330A may provide the second device 200 with information to allow the user U to confirm information regarding the information provided by the first device 100. Furthermore, the providing unit 330A may change the output mode of information (images or sounds) to be provided to the second device 200 depending on the type or amount of information that the user U has set as available for provision, or may provide advertising information according to the type or amount of information to the second device 200. Furthermore, the providing unit 330A may notify the occupant P from the user U before the information acquired from the first device 100 is provided to the second device 200.

[0101] According to the above-described modification, the management server 300A can centrally manage the images and sounds to be provided. This allows for more accurate understanding of the type, amount of data, and duration of the information to be provided, thereby enabling appropriate management of service fees and the like. Therefore, for example, the providing unit 330A may provide the first device 100 and the second device 200 with advertising information corresponding to the type or amount of information to be provided from among the advertising information registered in the provided information DB 362, and cause the fee management unit 340 to adjust the fee, such as by reducing the fee for the simulated ride service or paying a fee, depending on the number of times or duration of viewing of the advertisement by the occupant P or the user U. The advertising information may also be information about the moving object M, or, if the occupant P or the user U is a celebrity, information about the celebrity.

[0102] Furthermore, as shown in the modified example, image editing, display control, and the like are performed on the management server 300A side, so that the processing load on the first device 100 and the second device 200 can be reduced.

[0103] <Summary> The information processing system 1 configured as described above can enhance the sense of realism provided to both the occupant P of the moving body M and the user U who is in a location different from the moving body M. Furthermore, according to the embodiment, the visibility of the user and the safety of the occupant are improved, which in turn can further improve traffic safety and contribute to the development of a sustainable transportation system. In the embodiment, an image corresponding to the user U's orientation direction φ as seen from the passenger seat is displayed to the user U, allowing the user U to view a landscape as if he or she were sitting in the passenger seat S2 and looking around. Furthermore, the first speaker 140 outputs the voice of the user U by localizing a sound image so that the voice can be heard from the passenger seat S2 for the occupant P, allowing the occupant P to converse with the user U as if the user U were sitting in the passenger seat S2. Furthermore, the second speaker 240 outputs the voice of the occupant P by localizing the sound image so that the voice can be heard from the position of the occupant P as seen from the passenger seat S2, so that the user U can have a conversation with the occupant P as if he or she were sitting in the passenger seat S2. The user U can also control the information that can be provided by the simulated ride. Therefore, the experience level and experience mode of the simulated ride by the occupant P can be controlled by the user U's settings.

[0104] <Usage example> The information processing system 1 can be used in the following manner.

[0105] (A) A mode in which the occupant P and the user U are in a relationship such as family or friends, and a virtual drive is provided to the user U. The user U can talk with the occupant P about the scenery around the moving body M while looking at the image.

[0106] (B) A mode in which the passenger P is a general user and the user U is a provider of route guidance services, driving instruction services, etc. The user U can provide route guidance or driving instruction in places that are difficult to find using a navigation device or that are not on a map while viewing the scenery around the moving object M.

[0107] (C) A mode in which the passenger P is a celebrity and the user U is a general user, and a commercial virtual drive is provided to the user U. In this case, multiple users U may be associated with one passenger P at the same time, and, for example, audio forwarding from the user U's side may be set to off. Also, the user U may be able to talk to the passenger P for a specified period of time by paying an additional fee.

[0108] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0109] 1. Information Processing Systems 100 1st device 110 First communication device 120 First Microphone 125 External Sensor 130 Camera Unit 132 Indoor Camera 134 Outdoor Camera 140 1st Speaker 150 User display device 160 HMI 170 First control device 190 Controlled Equipment 200 2nd device 210 Second communication device 220 Second Microphone 230 Detection Device 232 Directional direction detection device 234 Head position detection device 236 Motion Sensor 240 Second Speaker 250 Mobile image display device 260 HMI 270 Second control device 300, 300A Management Server 310 Communication Equipment 315 Acquisition Department 320 Matching processing section 330, 330A supply department 340 Fee Management Department 350 Storage section M Mobile object

Claims

1. An information management device that manages information provided to a first device mounted on a vehicle on which a passenger rides and a second device used by a user at a location different from the vehicle, an acquisition unit that acquires information from the first device and the second device; a providing unit that provides information for each of the first device and the second device based on the information acquired by the acquiring unit, the acquisition unit acquires an indoor image and an outdoor image of the moving object using the first device, and acquires motion information of the user using the second device; the providing unit, when the user's movement is a predetermined movement, switches between the indoor image and the outdoor image and provides the images to the second device; the providing unit sets a reference position based on the reference posture of the user, and when a moving distance from the set reference position is equal to or greater than a threshold, switches from the indoor image to the outdoor image and provides the outdoor image; The threshold value is set according to the direction of movement of the user. Information management device.

2. An information management device that manages information provided to a first device mounted on a vehicle on which a passenger rides and a second device used by a user at a location different from the vehicle, an acquisition unit that acquires information from the first device and the second device; a providing unit that provides information for each of the first device and the second device based on the information acquired by the acquiring unit, the acquisition unit acquires an indoor image and an outdoor image of the moving object using the first device, and acquires motion information of the user using the second device; the providing unit, when the user's movement is a predetermined movement, switches between the indoor image and the outdoor image and provides the images to the second device; the providing unit sets a reference position based on the reference posture of the user, and when a moving distance from the set reference position is equal to or greater than a threshold, switches from the indoor image to the outdoor image and provides the outdoor image; the providing unit switches to the outdoor image viewed from a direction corresponding to the moving direction of the user when the moving distance from the set reference position is equal to or greater than a threshold value, and provides the outdoor image. Information management device.

3. the providing unit switches to the outdoor image and provides the outdoor image when the height of the user acquired from the second device becomes higher than a predetermined distance from the reference position.

3. The information management device according to claim 1.

4. The height of the user is the height of the user's head. The information management device according to claim 3 .

5. The reference posture of the user is the posture of the user when seated on a chair, The movement distance from the reference position is a movement distance due to the standing-up motion of the user.

3. The information management device according to claim 1.

6. the providing unit corrects the outdoor image so that the head position of the user does not move forward when the user stands up. The information management device according to claim 5 .

7. When a moving distance in a lateral direction from the set reference position is equal to or greater than a threshold, the providing unit switches to and provides an outdoor image captured from a side of the moving object corresponding to the moving direction.

3. The information management device according to claim 1.

8. a computer of an information management device that manages information provided to a first device mounted on a vehicle on which a passenger rides and a second device used by a user at a location different from the vehicle, obtaining information from the first device and the second device; providing information for each of the first device and the second device based on the acquired information; The first device acquires an indoor image and an outdoor image of the moving object, and the second device acquires motion information of the user; When the user's action is a predetermined action, switching between the indoor image and the outdoor image and providing the switching between the indoor image and the outdoor image to the second device; setting a reference position based on the reference posture of the user, and when a moving distance from the set reference position is equal to or greater than a threshold, switching from the indoor image to the outdoor image and providing the image; The threshold value is set according to the direction of movement of the user. Information management method.

9. a computer of an information management device that manages information provided to a first device mounted on a vehicle on which a passenger rides and a second device used by a user at a location different from the vehicle, obtaining information from the first device and the second device; providing information for each of the first device and the second device based on the acquired information; The first device acquires an indoor image and an outdoor image of the moving object, and the second device acquires motion information of the user; When the user's action is a predetermined action, switching between the indoor image and the outdoor image and providing the switching between the indoor image and the outdoor image to the second device; setting a reference position based on the reference posture of the user, and when a moving distance from the set reference position is equal to or greater than a threshold, switching from the indoor image to the outdoor image and providing the image; When the moving distance from the set reference position is equal to or greater than a threshold, the image is switched to the outdoor image viewed from a direction corresponding to the moving direction of the user and is provided. Information management method.

10. A computer of an information management device that manages information provided to a first device mounted on a vehicle on which a passenger rides and a second device used by a user at a location different from the vehicle, obtaining information from the first device and the second device; causing information to be provided for each of the first device and the second device based on the acquired information; acquiring an indoor image and an outdoor image of the moving object by the first device, and acquiring motion information of the user by the second device; When the user's action is a predetermined action, the second device switches between the indoor image and the outdoor image and provides the images; a reference position is set based on the reference posture of the user, and when a moving distance from the set reference position is equal to or greater than a threshold, the indoor image is switched to the outdoor image and the image is provided; The threshold value is set according to the direction of movement of the user. program.

11. A computer of an information management device that manages information provided to a first device mounted on a vehicle on which a passenger rides and a second device used by a user at a location different from the vehicle, obtaining information from the first device and the second device; causing information to be provided for each of the first device and the second device based on the acquired information; acquiring an indoor image and an outdoor image of the moving object by the first device, and acquiring motion information of the user by the second device; When the user's action is a predetermined action, the second device switches between the indoor image and the outdoor image and provides the images; a reference position is set based on the reference posture of the user, and when a moving distance from the set reference position is equal to or greater than a threshold, the indoor image is switched to the outdoor image and the image is provided; When the moving distance from the set reference position is equal to or greater than a threshold, the image is switched to the outdoor image viewed from a direction corresponding to the moving direction of the user and is provided. program.

Images