Information management device, information management method, and program

Abstract

To enhance the sense of presence given to users who are in a different location from the moving object. [Solution] The information management device according to the embodiment is an information management device that manages information acquired from a first device mounted on a mobile vehicle carrying a crew member and provided to a second device used by a user at a location different from the mobile vehicle, comprising: an acquisition unit that acquires information from the first device and the second device; and a provision unit that provides information to each of the second devices based on the information acquired by the acquisition unit, wherein the acquisition unit acquires a first sound at a first position of the mobile vehicle and a second sound at a second position different from the first position from the first device, acquires user operation information from the second device, and provides the second device with a sound whose volume ratio of the first sound and the second sound is adjusted according to the user's operation.

Description

Technical Field

[0006] , , , ,

[0001] The present invention relates to an information management apparatus, an information management method, and a program.

Background Art

[0002] Conventionally, research has been underway on communicating between a device mounted on a moving body such as a vehicle and a device used in a location separate from the moving body to share an image of the scenery outside the vehicle or the like. For example, Patent Document 1 discloses a technique for outputting information in response to external factors outside the device. <{0000010}>

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, although sound is output according to the distance to an object in the user's line of sight or the state of the object, it does not output information according to the user's actions. For this reason, there has been a problem that for a user in a location different from the moving body, a sense of presence may not be sufficiently felt.

[0005] One of the objects of the present application is to provide an information management apparatus, an information management method, and a program that can enhance the sense of presence given to a user in a location different from the moving body in order to solve the above problems. And it improves the visibility of the user and the safety of the passengers, and further contributes to the development of a sustainable transport system by further improving the safety of traffic.

Means for Solving the Problems

[0006] The information management apparatus, information management method, and program according to this invention adopt the following configuration. (1) An information management device according to one aspect of the present invention is an information management device that manages information acquired from a first device mounted on a mobile vehicle carrying a crew member and provided to a second device used by a user at a location different from the mobile vehicle, comprising: an acquisition unit that acquires information from the first device and the second device; and a provision unit that provides information to each of the second devices based on the information acquired by the acquisition unit, wherein the acquisition unit acquires a first sound at a first position of the mobile vehicle and a second sound at a second position different from the first position from the first device, acquires user operation information from the second device, and provides the second device with a sound whose volume ratio of the first sound and the second sound is adjusted according to the user's operation.

[0007] (2) In the embodiment of (1) above, the first sound is a sound collected inside the room of the mobile body, and the second sound is a sound collected outside the room of the mobile body. The providing unit provides the second device with a sound whose volume is lower than that of the first sound when the user is in a first posture corresponding to being inside the room of the mobile body, and provides the second device with a sound whose volume is higher than that of the first sound when the user moves from the first posture to a second posture corresponding to being outside the room of the mobile body.

[0008] (3) In the embodiment of (2) above, the first posture is a seated posture, and the second posture is a standing posture.

[0009] (4) In the embodiment of (2) above, the first posture is a seated posture facing the direction of travel of the moving body, and the second posture is a seated posture facing a direction perpendicular to the direction of travel of the moving body.

[0010] (5) In the embodiment of (2) above, the first posture is a posture in which the moving body is viewed in the direction of travel, and the second posture is a posture in which the moving body is viewed in a direction perpendicular to the direction of travel.

[0011] (6): In the embodiment of (1) above, the first sound is a sound inside the room of the moving body, The second sound is the sound from outside the mobile body, and the providing unit provides the second device with a sound that is quieter than the first sound when the window of the mobile body is closed, and provides the second device with a sound that is quieter than the first sound when the user opens the window of the mobile body.

[0012] (7) In the embodiment of (2) above, the acquisition unit acquires indoor and outdoor images of the moving object using the first device, the providing unit provides the indoor image to the second device in accordance with the user being in the first posture, and provides the outdoor image to the second device in accordance with the user performing an action to change from the first posture to the second posture.

[0013] (8) In the embodiment of (1) above, the providing unit provides the second device with an image showing the volume ratio of the first sound and the second sound.

[0014] (9): An information management method according to one aspect of the present invention is an information management method in which a computer of an information management device manages information acquired from a first device mounted on a mobile vehicle on which a crew member is riding, and provided to a second device used by a user at a location different from the mobile vehicle, acquires information from the first device and the second device, provides information to each of the second devices based on the acquired information, acquires a first sound at a first position of the mobile vehicle and a second sound at a second position different from the first position from the first device, acquires user operation information from the second device, and provides the second device with a sound whose volume ratio of the first sound and the second sound is adjusted according to the user's operation.

[0015] (10): The program according to one aspect of the present invention causes a computer of an information management device that manages information provided from a first device mounted on a moving body on which an occupant rides to a second device used by a user at a location different from the moving body to acquire information from the first device and the second device, provide information for each of the second devices based on the acquired information, acquire a first sound at a first position of the moving body and a second sound at a second position different from the first position by the first device, acquire operation information of the user by the second device, and provide a sound obtained by adjusting a volume ratio of the first sound and the second sound according to the operation of the user to the second device.

Effect of the Invention

[0016] According to the aspects (1) to (10) above, it is possible to enhance the sense of presence given to a user at a location different from the moving body.

Brief Description of the Drawings

[0017] [Figure 1] It is a diagram showing the usage environment etc. of the information processing system 1 and the management server 300. [Figure 2] It is a diagram showing an example of the content of the user data 360. [Figure 3] It is a configuration diagram of the first device 100. [Figure 4] It is a diagram showing an example of the arrangement of a part of the first device 100 in the moving body M. [Figure 5] It is a configuration diagram of the second device 200. [Figure 6] It is a diagram for explaining an image corresponding to the pointing direction. [Figure 7] It is a diagram showing a first example of the functional configuration of the first control device 170 and the second control device 270. [Figure 8] It is a diagram showing a second example of the functional configuration of the first control device 170 and the second control device 270. [Figure 9] It is a sequence diagram showing an example of the processing executed by the information processing system 1. [Figure 10]This is a diagram for explaining an example of the first provided information. [Figure 11] This is a diagram for explaining an example of the second provided information. [Figure 12] This is a diagram for explaining an example of the third provided information. [Figure 13] This is a diagram showing an example of the functional configuration of the management server 300A in a modified example.

Embodiments for Carrying Out the Invention

[0018] Hereinafter, embodiments of an information management apparatus, an information management method, and a program according to the present invention will be described with reference to the drawings. In the following, an information processing system to which the information management apparatus is applied will be described. The information processing system includes, for example, a first device mounted on a moving body on which an occupant rides, a second device used by a user at a location different from the moving body, and an information management apparatus. The moving body is, for example, a vehicle, but any moving body (for example, a ship, an aircraft) on which an occupant can ride may be used. The occupant is mainly the driver of the moving body, but may be an occupant other than the driver.

[0019] Between the first and second devices, sounds collected by microphones are transmitted to the other side and played back through speakers, creating a situation similar to a telephone call. The sounds here include the voices of the occupants collected by the first microphone unit of the first device, the sounds inside the first vehicle, the sounds outside the first vehicle, and the voices of the users collected by the second microphone of the second device. Furthermore, between the first and second devices, a portion of the image captured by the camera unit of the first device is displayed by the second device. This provides the second device with MR (Mixed Reality). Therefore, users of the second device can experience the sensation of being on board the vehicle (simulated ride experience) even though they are in a different location from the vehicle, and occupants can converse with users who are experiencing a simulated ride on the vehicle via the first device, giving them the feeling that the users are actually riding on the vehicle with them. Hereafter, as described above, the simulated experience of a user being on board the vehicle may be referred to as "simulated riding." The relationship between the first and second devices does not need to be one-to-one; the information processing system can function by matching one of multiple first devices with multiple second devices in a one-to-many relationship. In the latter case, for example, one crew member can communicate with multiple users simultaneously or sequentially.

[0020] <Basic configuration> Figure 1 shows the operating environment of the information processing system 1 and the management server 300. The information processing system 1 includes a first device (mobile device) 100 mounted on a mobile vehicle M on which a crew member P is riding, and a second device (user device) 200 used by a user U at a location different from the mobile vehicle M (although it is not excluded that the location may be nearby). The first device 100, the second device 200, and the management server 300 each communicate with each other via a network NW. The information processing system 1 may or may not include the management server 300. The network NW includes at least one of the following: the Internet, WAN (Wide Area Network), LAN (Local Area Network), mobile communication network, cellular network, etc. The management server 300 is an example of an "information management device". The management server 300 may be implemented as a server device or storage device incorporated into a cloud computing system. In this case, the functions of the management server 300 may be implemented by multiple server devices and storage devices in the cloud computing system. Furthermore, the first device 100 mounted on the mobile body M may be realized by multiple units.

[0021] The management server 300 manages the information provided to the first device 100 and the second device 200, and manages their communication. The management server 300 includes, for example, a communication device 310, an acquisition unit 315, a matching processing unit 320, a provisioning unit 330, a billing management unit 340, and a storage unit 350. The acquisition unit 315, the matching processing unit 320, the provisioning unit 330, and the billing management unit 340 are implemented, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be implemented by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or by the cooperation of software and hardware. The program may be stored in advance on a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device equipped with a non-transient storage medium), or it may be stored on a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is inserted into a drive device. Alternatively, the program may be stored on the management server 300 and updated as needed when connected to the network NW.

[0022] The communication device 310 is a communication interface for connecting to the network NW. Communication between the communication device 310 and the first device 100, and communication between the communication device 310 and the second device 200, are conducted, for example, according to TCP / IP (Transmission Control Protocol / Internet Protocol).

[0023] The acquisition unit 315 acquires various types of information transmitted via the network NW by the first device 100, the second device 200, and other external devices. These types of information may include video data, driver voice data, and vehicle data (such as the display contents of the METER and IVI (in-vehicle infotainment) systems).

[0024] The matching processing unit 320 is implemented by a processor, such as a CPU, executing a program (set of instructions) stored in a storage medium. For example, when the communication device 310 receives a matching request from user U via the second device 200, or from crew member P via the first device 100, the matching processing unit 320 refers to user data 360 to perform matching between user U and crew member P, and uses the communication device 310 to send the communication identification information of crew member P's first device 100 to the second device 200 of the matched user U, and sends the communication identification information of user U's second device 200 to the first device 100 of the matched crew member P. After receiving these, the first device 100 and the second device 200 can perform more real-time communication, such as one based on UDP (User Datagram Protocol).

[0025] The providing unit 330 generates information to be provided to the first device 100 and the second device 200 based on the information acquired by the acquisition unit 315 and various other information, and transmits the generated information to the target devices. The providing unit 330 also generates information indicating the processing results by the matching processing unit 320 and the fee information (settlement information) managed by the fee management unit 340, and provides the generated information to the target devices.

[0026] The fare management unit 340 manages the fares to be charged to user U based on the information provided to user U, and the fares to be charged to crew member P based on the information provided to crew member P of the mobile vehicle M. The fare management unit 340 may also manage, for example, the compensation paid to user U and crew member P based on the information provided by user U and crew member P. The fare management unit 340 may also handle settlement processing for user U and crew member P.

[0027] The storage unit 350 may be implemented using the various storage devices mentioned above, or an SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 350 can store, for example, user data 360, provided information DB 362, programs, and various other information.

[0028] Figure 2 shows an example of the contents of user data 360. User data 360 includes a crew list 360A and a user list 360B. Crew list 360A includes, for example, the crew ID, which is the identification information of crew member P of mobile vehicle M, their communication identification information (IP address, etc.), user ID, which is the identification information of user U to be matched, mobile vehicle information on which the crew member is riding, and the availability information set by the crew member, all of which are associated with each other. Mobile vehicle information includes, for example, information on equipment installed on mobile vehicle M (installed equipment information) and vehicle class information indicating the size and external shape of mobile vehicle M. Mobile vehicle information may also include information on the current location, destination, and surrounding conditions of mobile vehicle M (for example, driving on a road along the coast), which are transmitted from mobile vehicle M at predetermined intervals. User list 360B includes, for example, the user ID, their communication identification information (IP address, etc.), the crew member P to be matched, and user information, all of which are associated with each other. User information may include information about physical characteristics (e.g., height and sitting height) and information that can predict physical characteristics (e.g., age). Availability information is information that the mobile device M can or cannot provide, and is set, for example, by the crew member P. Availability information may be set for each piece of equipment on the mobile device M, or for each user U. Examples of information regarding the availability of services include, but are not limited to, "Provision of images is permitted," "Provision of audio is not permitted, but provision of in-car and external sounds is permitted," "Provision of interior images is permitted, but provision of exterior images is not permitted," "Provision of occupant images is not permitted," and "Use of navigation devices is not permitted." Furthermore, the information regarding availability may include a fee for enabling the provision of services (service provision fee). User data 360 may be generated in any manner, not limited to the manner shown in Figure 2, as long as it includes this information.

[0029] The provided information DB362 stores various types of information to be provided to user U or crew member P. This information includes, for example, map information, POI (Point of Interest) information, and computer-generated images (e.g., computer graphics images of people, marks, symbols, icons, etc.). POI information, for example, is information about various shops, theme parks, and landmarks at specific locations, and may be included in the map information. Sound information may also be included in the provided information. Furthermore, the provided information DB362 may include advertising information. This advertising information may include, for example, advertisements related to the mobile vehicle M, advertisements related to user U or crew member P, and advertisements related to the products and services of shops. The inserted advertising information is managed separately from indoor and outdoor information, and when archived and distributed later, it may differ from the inserted advertising information distributed in real time (for example, closed shops may be updated, and the featured menus may be updated). Advertising information may include, for example, videos and audio.

[0030] Figure 3 is a configuration diagram of the first device 100. The first device 100 includes, for example, a first communication device 110, a first microphone unit 120, an external sensor 125, a camera unit 130, a first speaker 140, a user display device 150, an HMI (Human-machine Interface) 160, and a first control device 170. The first control device 170 is connected to a controlled device 190 mounted on a mobile body M.

[0031] The first communication device 110 is a communication interface for communicating via the network NW with the communication device 310 of the management server 300 and the second communication device 210 of the second device 200, which will be described later.

[0032] The first microphone unit 120 includes, for example, an indoor microphone 122 and an outdoor microphone 124. The indoor microphone 122 is installed inside the vehicle M and collects in-vehicle sounds. Here, the interior is an example of a "first location". Also, in-vehicle sounds are an example of a "first sound". In-vehicle sounds include sounds originating from the vehicle M, such as the sound of turn signals, wipers, engine noise, motor noise, sounds emitted from the car audio system, and voices spoken by the occupant P. The indoor microphone 122 may include a non-voice microphone that mainly collects sounds originating from the vehicle M and a voice microphone that mainly collects voices spoken by the occupant P. In this case, the non-voice microphone that mainly collects sounds originating from the vehicle M is installed on the instrument panel or the like. The voice microphone that mainly collects voices spoken by the occupant P is installed near the driver's seat S1 or is attached to the occupant P, such as a lavalier microphone.

[0033] The outdoor microphone 124 is installed outside the mobile vehicle M and collects external sounds. Here, "outside" is an example of a "second location." Also, external sounds are an example of a "second sound." External sounds may include wind noise associated with the movement of the mobile vehicle M, sounds of other vehicles other than the mobile vehicle M, siren sounds, horn sounds, brake sounds, sounds emitted by audible traffic signals, chimes and broadcast sounds emitted from speakers installed outdoors such as disaster prevention administrative radio broadcasts, and noise including street speeches and conversations between passersby. In addition, sounds emitted outside the vehicle in a specific area, for example, when driving near a train station or railway tracks, train sounds, horn sounds, crossing gate alarm sounds, station announcements and departure melodies may be included in external sounds. When driving near event facilities such as concert venues, cheers and applause from the venue may be included in external sounds. Furthermore, depending on the weather conditions while the mobile vehicle M is in motion, sounds such as rain, wind, and thunder may be included in the external sounds. Hereinafter, sound information acquired by the indoor microphone 122 may be referred to as "indoor sound information." Similarly, sound information acquired by the outdoor microphone 124 may be referred to as "outdoor sound information."

[0034] The external sensor 125 detects the position of objects around the moving object M. The external sensor 125 is, for example, a radar device, a LIDAR (Light Detection and Ranging) sensor, or various other proximity sensors. The radar device emits radio waves such as millimeter waves around the moving object M and detects the radio waves reflected by objects (reflected waves) to detect at least the position (distance and direction) of the objects. The radar device may also detect the position and velocity of objects using the FM-CW (Frequency Modulated Continuous Wave) method. The LIDAR sensor irradiates light (or electromagnetic waves with a wavelength close to light) around the moving object M, measures the scattered light, and detects the distance to the target based on the time from emission to reception. The irradiated light is, for example, pulsed laser light. The radar device and LIDAR sensor can be attached to any part of the moving object M. Alternatively, the external sensor 125 may detect surrounding objects using images captured by the outdoor camera 134 of the camera unit 130.

[0035] The camera unit 130 includes, for example, an indoor camera 132 and an outdoor camera 134. The first speaker 140 outputs the voice spoken by user U, which is acquired via the first communication device 110. Details of the camera unit 130 and the arrangement of the first speaker 140 will be explained later with reference to Figure 4.

[0036] The user display device 150 virtually displays user U as if the user U were present inside the mobile device M. For example, the user display device 150 may display a hologram or display user U in a part of the mobile device M corresponding to a mirror or window.

[0037] HMI160 consists of a touch panel and a voice response device (agent device), etc. HMI160 receives various instructions from crew member P to the first device 100 and provides various information to crew member P.

[0038] The first control device 170 includes, for example, a processor such as a CPU and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the first device 100 by executing the set of instructions.

[0039] The controlled device 190 is, for example, an in-vehicle device such as a navigation system mounted on the mobile vehicle M that guides the driver to a destination, or a driver assistance system that controls either or both the steering and / or speed of the mobile vehicle M to assist the driver P. The controlled device 190 includes, for example, a seat drive system that can adjust the position (front, back, left, and right), orientation, and height of the seat. When viewing the image using the second device 200, if the camera unit 130 of the first device 100 is attached to the seat, the impact on the image can be suppressed by prohibiting seat movement. Even if seat movement is permitted, processing such as angle of view conversion may be performed to ensure that the image is not affected when the seat is moved. Furthermore, if user U requests to view images outside the current angle of view of the camera unit 130, the seat drive system may be controlled as a request from the second device 200.

[0040] Figure 4 shows an example of the arrangement of some of the first devices 100 in the mobile vehicle M. The indoor microphone 122 is provided, for example, in the lower center of the instrument panel. The outdoor microphone 124 is provided, for example, in the upper center of the outside of the back door. Alternatively, the outdoor microphone 124 may be provided on the roof of the mobile vehicle M. Although this example illustrates the case where the outdoor microphone 124 is provided on the mobile vehicle M, it is not limited to this. The outdoor microphone 124 only needs to be able to collect sounds from outside the vehicle and does not necessarily need to be attached to the mobile vehicle M.

[0041] The interior camera 132 is attached, for example, to the neck pillow of the passenger seat S2 (an example of a "designated seat") via attachment 132A, and is positioned slightly away from the backrest of the passenger seat S2 towards the direction of travel of the mobile body M. The interior camera 132 has a wide-angle lens and is capable of imaging the area represented by the hatched region 132B in the figure. The interior camera 132 can photograph not only the interior of the mobile body M but also the exterior through the windows. In the following description, the passenger seat S2 is assumed to be the designated seat, but the designated seat may be another seat such as the rear seat.

[0042] The outdoor camera 134 includes, for example, a plurality of sub-outdoor cameras 134-1 to 134-4. By combining the images captured by the plurality of sub-outdoor cameras 134-1 to 134-4, an image such as a panoramic image capturing the outside of the mobile body M is obtained. The outdoor camera 134 may also include (or in addition to) these a wide-angle camera mounted on the roof of the mobile body M. An indoor camera 132 capable of capturing the area behind the passenger seat S2 may be added, and the mobile body image described later may be generated as a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 by the first control device 170, or it may be generated as a 360-degree panoramic image by appropriately combining images captured by indoor cameras 132 and images captured by outdoor cameras 134. The outdoor camera 134 may also be a 360-degree camera. Similarly, the indoor camera 132 may also be a 360-degree camera.

[0043] Furthermore, multiple indoor microphones 122 may be provided inside the mobile unit M. For example, an indoor microphone 122 may be provided at each location where an indoor camera 132 is installed. Also, multiple outdoor microphones 124 may be provided outside the mobile unit M. For example, an outdoor microphone 124 may be provided at each location where an outdoor camera 134 is installed. In this case, for example, a camera with an integrated microphone can be used.

[0044] The first speaker 140 outputs the voice of user U acquired via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, sub-first speaker 140-1 is located in the center of the instrument panel, sub-first speaker 140-2 is located at the left end of the instrument panel, sub-first speaker 140-3 is located at the right end of the instrument panel, sub-first speaker 140-4 is located at the bottom of the left door, and sub-first speaker 140-5 is located at the bottom of the right door. When the first control device 170 outputs the voice of user U to the first speaker 140, for example, it outputs sound at a similar volume from sub-first speakers 140-2 and 140-4, and turns off the other sub-first speakers, thereby localizing the sound image so that the voice sounds to the passenger seat S2 for the occupant P seated in the driver's seat S1. Furthermore, the method of sound image localization is not limited to volume adjustment, but may also be performed by shifting the phase of the sound output by each sub-first speaker. For example, to localize the sound image so that it sounds like it is coming from the left, the timing of outputting sound from the left sub-first speaker should be slightly earlier than the timing of outputting the same sound from the right sub-first speaker.

[0045] Furthermore, when the first control device 170 outputs the voice of user U to the first speaker 140, it may direct the sound image to the occupant P so that the voice is heard from a position corresponding to the height of user U's head on the passenger seat S2, and then output the voice emitted by user U to the first speaker 140. In this case, the first speaker 140 needs to have multiple child first speakers 140-k (where k is a number of natural numbers) at different heights.

[0046] Figure 5 is a configuration diagram of the second device 200. The second device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a mobile image display device 250, an HMI 260, and a second control device 270. The detection device 230 includes, for example, a direction detection device 232, a head position detection device 234, and a motion sensor 236.

[0047] The second communication device 210 is a communication interface for communicating with the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0048] The second microphone 220 collects voices emitted by user U and sounds generated in the space where user U is located (for example, the sound of a user standing up when they move from a seated position). The voices collected by the second microphone 220 are transmitted to the first communication device 110 by the second communication device 210 via the second control device 270, for example.

[0049] The direction detection device 232 is a device for detecting the direction of direction. The direction of direction is the orientation of the user U's face or gaze, or an orientation based on both. Hereinafter, the direction of direction will be assumed to be an angle in the horizontal plane, that is, an angle without a vertical component, but the direction of direction may also be an angle that includes a vertical component. The direction detection device 232 may include a physical sensor (e.g., an accelerometer, a gyroscope, etc.) attached to the VR goggles described later, or it may be an infrared sensor that detects multiple positions of the user U's head, or a camera that images the user U's head. In any case, the second control device 270 calculates the direction of direction based on the information input from the direction detection device 232. Since various technologies for this are publicly known, a detailed explanation will be omitted.

[0050] The head position detection device 234 is a device for detecting the position (height) of the user U's head. For example, one or more infrared sensors or optical sensors installed around the chair where the user U is seated can be used as the head position detection device 234. In this case, the second control device 270 detects the position of the user U's head based on the presence or absence of detection signals from one or more infrared sensors or optical sensors. Alternatively, the head position detection device 234 may be an accelerometer attached to the VR goggles. In this case, the second control device 270 detects the position of the user U's head by integrating the output of the accelerometer minus the acceleration due to gravity. The head position information obtained in this way is provided to the second control device 270 as height information. The user's head position may also be obtained based on the user U's operation of the HMI 260. For example, the user U may input their height numerically into the HMI 260, or they may input their height using a dial switch included in the HMI 260. In these cases, the head position, i.e., height information, is calculated from the height. Furthermore, user U may input discrete values ​​such as body size: large / medium / small into the HMI260 instead of continuous values. In this case, height information will be obtained based on the information indicating body size. Alternatively, the user's head height may be obtained simply based on the average adult body size (which may be separated by gender) without specifically obtaining the user's head height.

[0051] The motion sensor 236 is a device for recognizing blinking and gestures performed by user U. For example, a camera that captures images of user U's upper body is used as the motion sensor 236. In this case, the second control device extracts feature points of user U's face (eyes, nose, mouth) and feature points of the body (fingertips, wrists, elbows, etc.) from the image captured by the camera, and recognizes user U's blinking, eye movements, and gestures based on the movement of these feature points.

[0052] The second speaker 240 outputs a sound which is a mix of external and internal sounds collected by the first microphone unit 120 of the mobile body M, acquired via the second communication device 210. The second speaker 240 may have a function to change the direction from which the sound is heard. When the second control device 270 is mainly outputting the voice of the occupant P to the user U, it may cause the second speaker to output the voice so that the voice is heard from the position of occupant P as seen from the passenger seat S2. The second speaker 240 may include a plurality of child second speakers 240-n (where n is a plurality of natural numbers), and sound localization may be achieved by the second control device 270 adjusting the volume of each child second speaker 240-n, or if headphones are attached to the VR goggles, sound localization may be achieved by utilizing the function of the headphones.

[0053] The mobile image display device 250 displays an image from the camera unit 130 (which may be an image that has undergone the aforementioned stitching process, and will be referred to as a mobile image below) that corresponds to the direction of orientation as seen from the passenger seat. Figure 6 is a diagram illustrating the image corresponding to the direction of orientation. In this example, the VR goggles 255 include a direction detection device 232, a physical sensor as a head position detection device 234, and the mobile image display device 250. The second control device 270, for example, uses the center of the user U's head or the center of the VR goggles 255 as Ω, and detects the direction the VR goggles 255 are facing as the direction of orientation φ, using a pre-calibrated direction as the reference direction. Since various methods for this function are already publicly known, a detailed explanation is omitted.

[0054] The mobile object image display device 250 displays image A2, which is part of the mobile object image A1 (which has an angle of approximately 240 degrees in the figure, but the field of view may be expanded by the merging process as described above), towards the user U.

[0055] HMI260 is a touch panel, voice response device (agent device), or the switches mentioned above. HMI260 receives various instructions from user U to the second device 200.

[0056] The second control device 270 includes, for example, a processor such as a CPU and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the second device 200 by executing the set of instructions. Note that all the functions of the second device 200 shown in Figure 5 may be integrated into the VR goggles.

[0057] <Functional configuration of the first control device 170 and the second control device 270> The functional configurations of the first control device 170 and the second control device 270 will be described below. In the first and second examples shown below, the first control device 170 and the second control device 270 are mainly responsible for generating and transmitting information, while the management server 300 manages the matching of simulated rides between passenger P and user U. [Example 1] Figure 7 shows a first example of the functional configuration of the first control device 170 and the second control device 270. In the first example, the first control device 170 includes a matching request / acceptance unit 171, an audio output control unit 172, an image transmission unit 173, and an onboard device linkage unit 174. The second control device 270 includes a matching request / acceptance unit 271, an in-vehicle / out-of-vehicle sound output control unit 272, a direction detection unit 273, a head position detection unit 274, a gesture input detection unit 275, an image editing unit 276, and a mobile object image display control unit 277. These functional units are realized, for example, by a processor such as a CPU executing a program (set of instructions). Some or all of these components may be realized by hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or by the cooperation of software and hardware.

[0058] The matching request / acceptance unit 171 uses the HMI 160 to receive input of a matching request (simulated boarding request) from the occupant P and transmits it to the management server 300, and also uses the HMI 160 to receive input of acceptance of a matching request received from the management server 300 and transmits it to the management server 300. The matching request / acceptance unit 171 controls the first communication device 110 to communicate with the second device 200 of user U, with whom a match has been made. In addition, the matching request / acceptance unit 171 may, for example, accept the setting of information provision permission information indicating whether or not to allow the occupant P to provide information when accepting a matching request obtained from the management server 300.

[0059] As described above, the audio output control unit 172 controls the first speaker 140.

[0060] After communication with the second device 200 begins, the image transmission unit 173 uses the first communication device 110 to transmit the mobile image A1 to the second device 200. The mobile image A1 may include, for example, an indoor image captured by the indoor camera 132 and / or an outdoor image captured by the outdoor camera 134.

[0061] The onboard equipment linkage unit 174 controls the controlled equipment 190 based on instruction signals input from the second device 200. The onboard equipment linkage unit 174 may also perform control if the user U has been authorized by the crew member P to control the controlled equipment 190.

[0062] After communication with the second device 200 begins, the sound transmission unit 176 transmits sound information collected by the first microphone unit 120, namely in-vehicle sound information and external sound information, to the second device 200.

[0063] The matching request / acceptance unit 271 uses the HMI 260 to receive matching requests (simulated ride requests) from user U and transmits them to the management server 300, and also uses the HMI 260 to receive acceptance input for matching requests received from the management server 300 and transmits it to the management server 300. The matching request / acceptance unit 271 controls the second communication device 210 to communicate with the first device 100 of the crew member P with whom a match has been made. The matching request / acceptance unit 271 may also, for example, present to user U what information will be provided based on the matching result (request result) obtained from the management server 300, allow user U to confirm, and then accept instructions from user U on whether or not to proceed with the simulated ride.

[0064] The direction detection unit 273 detects the direction φ based on the output of the direction detection device 232. The head position detection unit 274 detects the height of the user U's head based on the output of the head position detection device 234. The head position may be expressed as three-dimensional coordinates, or the height of the head may simply be detected as the head position. The gesture input detection unit 275 detects the user U's blinking, eye movements, gesture operations, etc., based on the output of the motion sensor 236.

[0065] As described above, the in-vehicle / out-of-vehicle sound output control unit 272 controls the second speaker 240. The in-vehicle / out-of-vehicle sound output control unit 272 causes the second speaker 240 to output sound based on sound information transmitted to the second device 200 via the first communication device 110 by the sound transmission unit 176 of the first device 100. The in-vehicle / out-of-vehicle sound output control unit 272 adjusts the volume ratio of in-vehicle sound to out-of-vehicle sound according to the actions of user U, and outputs the sound with the adjusted volume ratio from the second speaker 240. For example, depending on whether user U is in a first posture corresponding to being inside the vehicle M, the in-vehicle / out-of-vehicle sound output control unit 272 outputs a sound from the second speaker 240 in which the volume of the out-of-vehicle sound is lower than the volume of the in-vehicle sound. The first posture is a posture corresponding to being in the passenger seat S2, for example, a seated posture. Furthermore, the in-vehicle / out-of-vehicle sound output control unit 272 outputs a sound from the second speaker 240 that is louder than the in-vehicle sound, depending on whether the user U is in a second posture corresponding to being outside the vehicle M. The first posture corresponds to being outside the vehicle M or having part of one's body outside the vehicle M, such as standing up or turning to the side. The second posture may also include actions corresponding to opening the front window of the vehicle M or opening the front door. The user's actions can be identified using all or part of the detection results from the direction detection unit 273, the head position detection unit 274, and the gesture input detection unit 275.

[0066] Furthermore, depending on the user's physical characteristics, it may be difficult for them to maintain a seated position, stand up, or turn to the side, either in whole or in part. In such cases, it is desirable to provide a realistic simulated riding experience. As a countermeasure, actions corresponding to a first posture and actions corresponding to a second posture may be pre-set according to the user U. For example, for a user who has difficulty maintaining a seated position, a lying position, such as a lateral or side-lying position, may be set as the first posture. For a user who can sit but has difficulty standing up, the action of turning their head upwards while seated may be set as the second posture. For a user who has difficulty turning their head upwards or to the side, the action of turning their gaze upwards or to the side may be set as the second posture. The actions corresponding to the first posture and actions corresponding to the second posture according to the user U can be pre-stored, for example, in the user list 360B of the user data 360.

[0067] Furthermore, depending on the settings configured by the occupant P to allow or disallow the provision of their voice, it may not be permitted to provide the occupant P's voice to the user U. In such cases, the in-vehicle / out-of-vehicle sound output control unit 272 generates silent in-vehicle sound by removing the voice from the in-vehicle sound and outputs the generated silent in-vehicle sound from the second speaker 240. For example, the in-vehicle / out-of-vehicle sound output control unit 272 extracts voice components and non-voice components (turn signal sound, wiper sound, engine sound, motor sound, etc.) based on characteristic information such as the frequency of the sound contained in the in-vehicle sound information, and adjusts each extracted component. Alternatively, if the in-vehicle microphone 122 consists of a non-voice microphone and a voice microphone, the in-vehicle / out-of-vehicle sound output control unit 2727 removes the sound (voice) component collected by the voice microphone from the sound collected by the non-voice microphone. Existing technologies, such as effect processing and noise cancellation technology, can be used as methods for removing sound components.

[0068] The image editing unit 276 performs a process to extract image A2 from the moving object image A1, for example, which corresponds to the direction of direction φ as viewed from the passenger seat S2 (Figure 6). In the example in Figure 6, the direction of direction φ for movement is shown to the left and right of the user U's head, but the process may also be performed to extract an image corresponding to the direction of direction of movement in the vertical direction of the head. The moving object image display control unit 277 displays image A2 extracted by the image editing unit 276 on the moving object image display device 250. At this time, the image editing unit 276 may also display on the moving object image display device 250 an image corresponding to the direction of direction φ as viewed from the height indicated by the height information of the user U's head. For example, the image editing unit 276 switches between displaying an indoor image or an outdoor image depending on the distance from the reference position of the user U's head (for example, the amount of upward movement or the amount of lateral movement).

[0069] [Example 2] Figure 8 shows a second example of the functional configuration of the first control device 170 and the second control device 270. Compared to the first example in Figure 7, the first control device 170 differs in that it includes an image editing unit 175 and an in-vehicle / out-of-vehicle sound output control unit 177. The second control device 270 differs in that it does not include an image editing unit 276 but includes a directional transmission unit 278. The second control device 270 also differs in that it does not include an in-vehicle / out-of-vehicle sound output control unit 272. As other components basically have the same functions as in the first example, further explanation is omitted.

[0070] The direction transmission unit 278 transmits the direction φ detected by the direction detection unit 273 to the first device 100 using the second communication device 210. The direction transmission unit 278 may also transmit the user's actions detected by all or part of the direction detection unit 273, the head position detection unit 274, and the gesture input detection unit 275 to the first device 100 using the second communication device 210.

[0071] The image editing unit 175 performs the process of extracting image A2 from the moving image A1 that corresponds to the directional direction φ (transmitted from the second device 200) as seen from the passenger seat (Figure 6). At this time, the image editing unit 175 may also obtain the height information of the user U's head from the second device 200 and perform the process of extracting image A2 that corresponds to the directional direction φ as seen from the height indicated by the height information.

[0072] In the second example, the image transmission unit 173 uses the first communication device 110 to transmit the image A2 extracted by the image editing unit 175 to the second device 200. The mobile image display control unit 277 then displays the image A2 transmitted from the first device 100 on the mobile image display device 250.

[0073] The in-vehicle / out-of-vehicle sound output control unit 177 adjusts the volume ratio of in-vehicle sound to out-of-vehicle sound according to the user's actions (transmitted from the second device 200).

[0074] Furthermore, if providing the voice of occupant P to the user is not permitted, the in-vehicle / out-of-vehicle sound output control unit 177 generates silent in-vehicle sound by removing the voice from the in-vehicle sound. The method for generating silent in-vehicle sound can be the same as that used by the in-vehicle / out-of-vehicle sound output control unit 272. By generating silent in-vehicle sound, the in-vehicle / out-of-vehicle sound output control unit 177 can control the transmission of unauthorized information (in this case, the voice of occupant P) to the second device 200.

[0075] In the second example, the sound transmission unit 176 uses the first communication device 110 to transmit sound information to the second device 200, in which the volume ratio of in-vehicle sound to out-of-vehicle sound has been adjusted by the in-vehicle / out-of-vehicle sound output control unit 177. The second control device 270 then outputs sound from the second speaker 240 based on the sound information with the adjusted volume ratio of in-vehicle sound to out-of-vehicle sound transmitted from the first device 100.

[0076] Furthermore, some of the functions of the in-vehicle / out-of-vehicle sound output control unit 272 (for example, the function of generating silent in-vehicle sound) may be provided in the first device 100 as the in-vehicle / out-of-vehicle sound output control unit 177. In this case, the first device 100 generates silent in-vehicle sound from the in-vehicle sound information collected by the indoor microphone 122 and transmits the generated silent in-vehicle sound and the external sound information collected by the outdoor microphone 124 to the second device 200. The second device 200 mixes the silent in-vehicle sound and external sound information received from the first device 100 and outputs a sound from the second speaker 240 in which the volume ratio of in-vehicle sound to external sound changes according to the actions of the user U.

[0077] <Processing related to simulated rides> Next, in this embodiment, a series of processes performed by the information processing system 1 when user U virtually boards the mobile vehicle M (or when crew member P virtually boards user U) will be specifically described. Figure 9 is a sequence diagram showing an example of the processes performed by the information processing system 1. In the example in Figure 9, the processes in the first device 100 mounted on the mobile vehicle M while crew member P is on board, the second device 200 used by user U, and the management server 300 are shown. Furthermore, it is assumed that user U and crew member P are registered in advance with the management server 300 and registered in user data 360 in order to receive services from the information processing system 1.

[0078] In the example shown in Figure 9, the second device 200 generates a request for a simulated ride (simulated ride request) (step S100) and sends the generated simulated ride request to the management server 300 (step S102). The simulated ride request may include, for example, information specifying a passenger P in advance, or information specifying the current location, destination, or surrounding conditions of the moving vehicle M. More specific examples of simulated ride requests include requests such as "I want to simulate ride in a vehicle driven by person A" or "I want to simulate ride in a vehicle traveling on a road along the coast." These are examples of the matching requests described above.

[0079] The matching processing unit 320 of the management server 300 receives a simulated experience request from the second device 200 and performs matching processing based on the user data 360 (step S104). Then, if there is one or more matched mobile bodies M, the provisioning unit 330 generates inquiry information to ask whether or not to allow all mobile bodies M to simulate riding (step S106), and transmits the generated information to the first device 100 of the matched mobile bodies M (step S108).

[0080] The first device 100 displays the inquiry information sent from the management server 300 on the screen (step S110). This screen displays a message such as "A simulated ride request has been received. Do you allow it?", along with user information about user U who made the simulated ride request, a selection image for selecting the content of the information that the occupant P will provide to user U, and an acceptance button to confirm whether or not to allow user U to ride in the simulated vehicle by the occupant P. The selection image displays information that the occupant P can choose whether or not to provide to user U, such as phone calls (voice), in-car sounds, exterior sounds, interior images, exterior images, and navigation information (information provided by the navigation system). For example, the occupant P may refuse to provide phone calls (voice) to user U, with whom they are not very familiar, but allow the provision of in-car and exterior sounds. This allows the driver to operate the vehicle without conversation.

[0081] The first device 100 accepts the user U's request for availability by displaying an image of the inquiry information described above (step S112), and transmits the accepted availability setting information to the management server 300 (step S114). Note that the processing in steps S110 to S112 may be displayed on the screen when the mobile body M stops in order to ensure safety during operation, or instead of (or in addition to) displaying on the screen, the system may notify the user that a simulated ride request has been made using voice, or accept the availability setting. The provision unit 330 of the management server 300 generates a request result (response result) based on the availability setting information obtained from the first device 100 (step S116), and transmits information indicating the generated request result to the second device 200 (step S118).

[0082] The second device 200 receives the request result from the management server 300 and displays information indicating the received request result on the screen (step S120). This screen displays a guidance message such as "Simulated ride has been permitted. Do you wish to board?", along with a list image showing a list of provided information and a confirmation button to indicate whether to proceed with the simulated ride or to search for another vehicle. This image may be generated for each passenger P (mobile body M) who has been permitted to ride in a simulated ride, and an additional image showing a list of request results may also be displayed.

[0083] The second device 200 receives the selection instruction from the user U as described above (step S122) and transmits the received instruction information to the management server 300 (step S124). In the following description, it is assumed that user U has given an instruction to virtually board the mobile body M of occupant P. Based on the instruction from the second device 200, the management server 300 manages the communication status between the first device 100 and the second device (step S126) and transmits information to the first device 100 and the second device 200 indicating that the virtual boarding will be performed (steps S128, S130). If user U chooses not to virtually board during the processing in steps S126 to S130, the provision unit 330 of the management server 300 transmits information to the first device 100 to notify user U of this decision.

[0084] Next, the first device 100 and the second device 200 communicate with each other to perform a simulated ride (step S132), and when the simulated ride is completed, they send information to the management server 300 indicating that communication has ended (steps S134, S136). The fare management unit 340 of the management server 300 performs settlement processing for the charges related to the use of the services of the first device 100 and the second device 200 (step S138). This completes the processing of this sequence. Note that the processing in step S112 shown in Figure 9 (service availability setting processing) may be set in advance by the passenger P for each registered user U.

[0085] <Information provided during simulated ride> Next, we will explain some examples of the information provided to user U during a simulated ride. The following examples are the information provided in the process of step S132 shown in Figure 9, and the output manner of the information provided to the second device 200 differs depending on, for example, the actions of user U and the type or amount of information that can be provided to user U. In the following examples, it is assumed that occupant P is authorized to provide interior images, exterior images, interior sounds, exterior sounds, and telephone (voice).

[0086] <Example of information provided in the first category: Information provided in a seated position> Figure 10 is a diagram illustrating an example of the first type of information provided. The example in Figure 10 shows a user U seated on a chair CH in a location other than the mobile vehicle M (for example, a room in one's home), and an example of an image IM30 provided by the second device 200 (integrated with VR goggles in the example in Figure 10) used by the user U. The state of user U shown in the example in Figure 10 (seated on chair CH) represents the initial state immediately after the simulated ride is performed, and the first device 100 and the second device 200 recognize the position of user U at this time (reference posture) as the reference position. The reference position is the position where user U is assumed to be seated in the passenger seat S2. The reference position may be corrected so that the height of the interior camera 132 matches the sitting height of user U by the first control device 170 driving the seat drive device based on the user U's height or sitting height information registered in the user information of user data 360 (so-called zero-point correction). Furthermore, the reference position may be corrected based on correction instructions from user U who has viewed image IM30. The reference position may also include orientation information for when user U is seated in chair CH and facing forward.

[0087] For example, when the user U (second device 200) is seated in chair CH and facing any direction from their reference position, the image captured by the interior camera 132 of the first device 100 is provided as the interior image IM30 according to that direction. This allows the user U to obtain an interior image of any direction as seen from the passenger seat S2.

[0088] Furthermore, sound information of in-vehicle sounds, including the voice of occupant P acquired by the first microphone unit 120, and external sounds is provided to user U. The in-vehicle / external sound output control unit 272 mixes the in-vehicle sound information and external sound information so that the volume of in-vehicle sounds is louder than the volume of external sounds when user U is in the reference position, that is, seated in the chair CH as shown in Figure 10. For example, the in-vehicle / external sound output control unit 272 mixes the in-vehicle sound information and external sound information at a volume ratio such that the in-vehicle sounds are heard loudly and the external sounds are heard faintly. The in-vehicle / external sound output control unit 272 outputs the sound information of the mixed sounds from the second speaker 240. By sitting in the chair CH, user U can hear the voice of occupant P and the sound of the turn signal as in-vehicle sounds, and by listening carefully, can also hear the faint external sounds mixed in. This allows the user to have an experience as if they were actually riding in the mobile vehicle M and having a conversation with occupant P.

[0089] Furthermore, while the information processing system 1 was described as allowing user U to view any direction from the passenger seat S2, restrictions may be placed on the directions that user U can view, for example, through arrangements during matching or settings for whether or not occupant P can provide information. For example, occupant P may want to provide the scenery in the direction of travel of the mobile vehicle M, or the scenery opposite the driver's seat S1, but not their own image. This is intended to address situations where occupant P and user U are not family or friends, and the user wants to confirm the driving feel of the mobile vehicle M or view a desired city landscape. In this case, the matching processing unit 320 or the provisioning unit 330 of the management server 300 sets such restrictions based on the matching results, and the first control device 170 or the second control device 270 masks the angle range that is not to be viewed, or corrects the direction of view φ so that it does not point in the restricted direction, according to the settings. Also, since information regarding such restrictions concerns the privacy of occupant P, it may be set on the first device 100 side. Furthermore, any information that has been refused may be provided after the consent of crew member P, provided that user U pays an additional fee.

[0090] Furthermore, the mobile image display device 250 may replace the portion of the image captured by the camera unit 130 that shows a predetermined item inside the room of the mobile object M with a computer-generated image (CG image) and display it. The region OB1 shown in Figure 10 is a display device that performs navigation display and the like, and is an example of a "predetermined item". If the display screen of the display device is displayed as is, the image may be blurry or visibility may be reduced due to light reflection. For this reason, the mobile image display device 250 may acquire data for configuring the display screen of the display device, or image data drawn by computer processing in the mobile object M, from the first device 100, and display an image redrawn from the data by computer processing, or the acquired image data, embedded in the image (edited version) captured by the camera unit 130. In this case, the location of the item inside the room of the mobile body M, which is the predetermined item, is shared in advance between the first device 100 and the second device 200. The mobile body image display control unit 277 determines whether the predetermined item is included in the image to be displayed on the mobile body image display device 250, for example, based on the direction of orientation φ, and if it is determined that it is included, it replaces the image as described above. The "predetermined item" may also be the head or face of the occupant P. In that case, the CG image of an avatar or the like may be changed according to the display of the occupant P. At least a part of the above-mentioned image data (CG image), etc., may be provided from the information database of the management server 300.

[0091] Furthermore, the in-vehicle / out-of-vehicle sound output control unit 272 may also display the volume ratio of in-vehicle sound to out-of-vehicle sound output from the second speaker 240 on the mobile image display device 250. In the region OB2 shown in Figure 10, an image (CG image) is displayed as a pie chart showing the volume ratio of in-vehicle sound to out-of-vehicle sound under a title such as "In-vehicle / out-of-vehicle sound balance". By displaying this image, the user can understand that the volume ratio of in-vehicle sound to out-of-vehicle sound has been adjusted and can grasp that volume ratio. In addition, depending on the user and the driving conditions of the mobile vehicle M, the user may want to adjust the volume ratio of in-vehicle sound to out-of-vehicle sound themselves. To meet such needs, the user may be allowed to fine-tune the volume ratio. In this case, for example, adjustment buttons for adjusting the volume ratio may be provided in region OB2. For example, two operation buttons such as "Increase in-vehicle sound ratio" and "Decrease in-vehicle sound ratio" may be displayed. The user U can perform an operation by pressing one of the two operation buttons via the HMI 260, for example. The in-vehicle / out-of-vehicle sound output control unit 272 outputs sound from the second speaker 240 with the volume ratio of in-vehicle sound and out-of-vehicle sound adjusted according to the operation input to the HMI 260.

[0092] Furthermore, the in-vehicle / out-of-vehicle sound output control unit 272 may be configured to switch between displaying or hiding the image shown in area OB2, depending on user operations or other factors.

[0093] <Example of the second type of information provided: Information provided in the standing position> Figure 11 is a diagram illustrating an example of the second type of information provided. In the example in Figure 11, image IM40 is shown, which is provided when the height of user U's head (second device 200) rises by a distance D1 from the reference position. In this case, the first device 100 determines whether the distance traveled D1 is greater than or equal to a threshold, and if it is, switches from displaying an indoor image from the indoor camera 132 to displaying an outdoor image from the outdoor camera 134. In the example in Figure 11, since the distance D1 is greater than or equal to the threshold, an outdoor image captured from the top of the moving object M is displayed. As a result, user U can obtain a unique simulated ride experience that would not be possible in an actual ride, such as sticking their head out from above the moving object M and looking at the surrounding scenery.

[0094] Furthermore, if the user U's head (second device 200) rises by a distance D1 from the reference position, the volume ratio of in-vehicle sound to external sound provided to the second device 200 is changed. When the user U's head rises by a distance D1 from the reference position and the user U stands up from the seat CH, the in-vehicle sound output control unit 272 mixes the in-vehicle sound information and external sound information so that the volume of in-vehicle sound is lower than the volume of external sound. For example, the in-vehicle sound output control unit 272 mixes the in-vehicle sound information and external sound information at a volume ratio such that the in-vehicle sound is barely audible and the external sound is loud. The in-vehicle sound output control unit 272 outputs the mixed sound information from the second speaker 240. When user U stands up from the seat CH, external sounds such as wind noise, the sound of other vehicles driving, and sirens become louder and clearer. Also, if the user listens carefully, they can still faintly hear the sounds inside the vehicle. This allows users to experience a unique simulated ride that would be impossible in an actual ride, such as sticking their head out from the top of the moving vehicle M and listening to sounds from outside.

[0095] The threshold value described above may be a fixed value or may be set variably according to the vehicle class of the mobile object M. For example, the first device 100 switches to an exterior image when the distance D1 from the reference position is greater than or equal to the vehicle class (external shape of the vehicle), based on the distance D1 and the vehicle class information included in the mobile object information stored in the user data 360. As a result, for example, when simulating riding in a large vehicle such as a bus, even if the user U stands up in the same way (moving only the distance D1 shown in Figure 11), the image remains as an interior image, providing more realistic information and making it less likely to cause discomfort to the user U. Furthermore, the threshold value described above may be set taking into consideration the physique of the user U.

[0096] Furthermore, when user U stands up from chair CH, the position of the head moves forward due to the structure of the human body compared to when seated. However, since user U believes that they are simply standing straight up, moving the outdoor image forward in accordance with the forward movement of the head when standing up may cause user U to feel uncomfortable. Therefore, the first device 100 may perform a correction process so as not to move the outdoor image forward when user U's head position moves forward by a predetermined distance due to standing up.

[0097] Furthermore, in the second example of information provided, image IM40 may have image IM42 showing navigation information and image IM44 showing POI information corresponding to surrounding buildings (for example, information about shops) superimposed on image IM40. This allows for the provision of more detailed information.

[0098] Furthermore, the region OB2 shown in Figure 11 displays an image (CG image) that shows the volume ratio of in-vehicle sound to external sound as a pie chart, similar to Figure 10. This figure shows that the volume ratio has been adjusted so that the volume of external sound is greater than the volume of in-vehicle sound. Similar to Figure 10, the in-vehicle / external sound output control unit 272 may switch the image displayed in region OB2 to either a hidden state or a displayed state depending on user operation.

[0099] <Example of third type of information provided: Information provided in a side-lying position> Figure 12 is a diagram illustrating an example of the third type of information provided. In Figure 12, compared to the example in Figure 11, an example is shown in which an outdoor image from the side of the moving object M is provided, rather than an outdoor image from above. Therefore, Figure 12 will mainly be explained focusing on the differences from Figure 11.

[0100] In the example shown in Figure 12, image IM50 is displayed when the position of user U's head (second device 200) moves laterally by a distance D2 from the reference position. In this case, the first device 100 determines whether the distance moved D2 is greater than or equal to a threshold, and if it is greater than or equal to the threshold, it switches from displaying the indoor image from the indoor camera 132 to displaying the outdoor image from the outdoor camera 134. In the example in Figure 12, since the distance D2 is greater than or equal to the threshold, the outdoor image captured from the side of the moving body M in the direction of movement (more specifically, the image of the front because the user is facing slightly forward) is displayed. This allows user U to have a simulated ride experience as if they were sticking their head out from the side of the moving body M (for example, through a side window) and looking at the surrounding scenery. Note that the above threshold may be a fixed value or may be set variably according to the vehicle class information of the moving body M.

[0101] Furthermore, if the position of user U's head (second device 200) moves laterally by a distance D2 from the reference position, the volume ratio of in-vehicle sound to out-of-vehicle sound provided to the second device 200 is changed. The in-vehicle / out-of-vehicle sound output control unit 272 mixes the in-vehicle sound information and out-of-vehicle sound information so that the volume of in-vehicle sound becomes lower than the volume of out-of-vehicle sound when user U's head moves leftward by a distance D2 from the reference position and the user U assumes a posture facing left, as if looking out from the passenger seat S2. On the other hand, when user U's head moves rightward by a distance D2 from the reference position and the user U assumes a posture facing right, as if looking at occupant P in the driver's seat S1 from the passenger seat S2, the in-vehicle sound information and out-of-vehicle sound information are mixed so that the volume of in-vehicle sound (especially occupant P's voice) becomes higher than the volume of out-of-vehicle sound. For example, if the indoor microphone 122 consists of a non-speech microphone and a speech microphone, the in-vehicle / out-of-vehicle sound output control unit 272 mixes the in-vehicle sound information and the out-of-vehicle sound information so that the volume of the sound collected by the outdoor microphone 124 and the non-speech microphone is reduced, and the volume of the sound collected by the speech microphone is increased. The in-vehicle / out-of-vehicle sound output control unit 272 outputs the mixed sound from the second speaker 240.

[0102] Note that the arrangement of the driver's seat S1 and passenger seat S2 may differ depending on the mobile vehicle M. The in-vehicle / out-of-vehicle sound output control unit 272 determines, according to the arrangement of the driver's seat S1 and passenger seat S2 in the mobile vehicle M, whether the movement of the user's head laterally from the reference position corresponds to looking outside or to looking at occupant P in the driver's seat S1. The arrangement of the driver's seat S1 and passenger seat S2 in the mobile vehicle M is stored in the occupant list 360A in advance, for example. The second device 200 refers to the management server 300 storage unit 350 and obtains the arrangement of the driver's seat S1 and passenger seat S2 in the mobile vehicle M for occupant P that matches user U, which is stored in the occupant list 360A. As a result, the in-vehicle / out-of-vehicle sound output control unit 272 can obtain the arrangement of the driver's seat S1 and passenger seat S2. The in-vehicle and out-of-vehicle sound output control unit 272 can change the volume ratio of in-vehicle sound to out-of-vehicle sound according to the user's actions, based on the acquired arrangement of the driver's seat S1 and passenger seat S2.

[0103] Furthermore, when simulating riding in a large vehicle such as a bus, a user U turning sideways may not correspond to looking outside or to the occupant P, but rather to looking inside the vehicle. In such cases, the in-vehicle / out-of-vehicle sound output control unit 272 changes the volume ratio of each component of the in-vehicle sound from the current ratio. For example, the in-vehicle / out-of-vehicle sound output control unit 272 increases (or decreases) the volume ratio of non-voice components relative to the volume of voice components of the in-vehicle sound. Alternatively, if multiple indoor microphones 122 are arranged laterally, the in-vehicle / out-of-vehicle sound output control unit 272 may change the volume ratio so that the volume of sound collected by the indoor microphone 122 positioned in the opposite direction to the direction the user U is facing is lower than the volume of sound collected by the indoor microphone 122 positioned in the direction the user U is facing. By changing the volume ratio, the user can experience changes in the surrounding sound corresponding to their own movements, and can experience a more realistic simulated ride.

[0104] Furthermore, if the mobile unit M is equipped with multiple indoor microphones 122, the in-vehicle / out-of-vehicle sound output control unit 272 may adjust the volume ratio of the sound collected from each of the multiple indoor microphones 122 according to the user's directional direction φ. For example, the in-vehicle / out-of-vehicle sound output control unit 272 calculates the straight-line distance between the line corresponding to the user's directional direction φ and each of the multiple indoor microphones 122, and adjusts the volume ratio of the sound collected from each of the multiple indoor microphones 122 according to the calculated distance. The in-vehicle / out-of-vehicle sound output control unit 272 adjusts the volume ratio such that the volume ratio increases as the distance to the line corresponding to the directional direction φ decreases.

[0105] Similarly, when the mobile unit M is equipped with multiple outdoor microphones 124, the volume ratio of the sound collected from each of the multiple indoor microphones 122 can be adjusted according to the user's directional direction φ. The in-vehicle / out-of-vehicle sound output control unit 272 adjusts the volume ratio so that the volume ratio of the sound collected by the outdoor microphone 124 located at a position with a small linear distance to the line corresponding to the directional direction φ is greater than the volume ratio of the sound collected by the outdoor microphone 124 located at a position with a large linear distance.

[0106] Furthermore, since the position of the user U's head may shift during lateral movement, similar to the standing-up motion, the first device 100 may perform correction to ensure that there is no image shift when the shift is within a predetermined amount.

[0107] Furthermore, the in-vehicle / out-of-vehicle sound output control unit 272 may also display an image (CG image) on the mobile image display device 250 that corresponds to the volume ratio of the in-vehicle sound to the out-of-vehicle sound output from the second speaker 240. In the regions OB3 and OB4 shown in Figure 12, images of "ripples" that schematically represent how sound waves propagate are shown as image images indicating the magnitude of the out-of-vehicle sound. In this figure, the radius of the ripple displayed in region OB3 is shown to be larger than the radius of the ripple displayed in region OB4, indicating that the volume of the out-of-vehicle sound is adjusted to be louder than the volume of the in-vehicle sound. The in-vehicle / out-of-vehicle sound output control unit 272 may also switch between displaying or hiding the images shown in regions OB3 and OB4 depending on user operation or other factors.

[0108] Furthermore, in the third example of provided information, the image IM50 may display information indicating the distance to an obstacle OBJ such as a curb detected by the external sensor 125 (for example, image IM52 or a numerical value indicating the distance). In this case, the first device 100 acquires information from the external sensor 125, which recognizes objects around the moving body M, and provides the second device 200 with an image IM52 showing the distance between the object (obstacle OBJ) detected by the external sensor 125 and the moving body M when providing the outdoor image. This allows, for example, the user U to see the outdoor image IM50 of the roadside area, which is a blind spot that the occupant P cannot see directly from the driver's seat, and notify the occupant P of its contents through conversation, thereby suppressing contact with the obstacle OBJ and enabling more appropriate driving.

[0109] Furthermore, the first device 100 may, in lieu of (or in addition to) distance information, provide the second device 200 with information indicating that the distance between the moving body M and the object (obstacle OBJ) is approaching (for example, alert information) when the distance between the moving body M and the object (obstacle OBJ) is within a predetermined distance. This allows the user U to check the surroundings using external images and to more accurately grasp the area near the occupant P's blind spot.

[0110] Furthermore, the system may be configured to amplify and output sounds (operational sounds) generated in conjunction with the actions of user U or crew member P.

[0111] For example, the second microphone 220 amplifies the rise sound (an example of an operating sound) collected when the user U's head (second device 200) rises by a distance D1 from the reference position, and transmits the amplified sound (amplified sound) to the first communication device 110. This makes it easier for the crew member P to recognize that user U has stood up. Alternatively, the second microphone 220 amplifies the lateral sound (an example of an operating sound) collected when the user U's head (second device 200) moves laterally by a distance D2 from the reference position, and transmits the amplified sound (amplified sound) to the first device 100. This makes it easier for the crew member P to recognize that user U has turned to the side.

[0112] Furthermore, the indoor microphone 122 amplifies the sound of the movement collected when it detects the movement of occupant P, and transmits the amplified sound to the second device 200. This makes it easier for user U to understand that occupant P has performed some kind of action. Here, the actions of occupant P include, for example, occupant P readjusting their position in the driver's seat, rising to stand up, turning around, stopping the mobile vehicle M and disembarking from the mobile vehicle M. These actions of occupant P can be detected by performing image processing (for example, object recognition) on the image captured by the indoor camera 132.

[0113] Based on the information provided as described above, images and sounds of a simulated ride can be provided in response to the user U's predetermined actions. In the example above, images (indoor images, outdoor images) were switched and the volume ratio of sounds (in-car sounds, outside sounds, voice) was changed according to the user's movements, but this is not limited to this. Images may be switched or the volume ratio of sounds may be changed independently of the user U's actions, in response to mechanical switches (operating parts) such as toggle buttons provided on the second device 200, gestures of the user U, or operating parts and gestures provided on the first device 100.

[0114] Furthermore, the first to third provided images described above may be provided to the first device 100. This allows the crew member P to more accurately understand what image user U is currently viewing and to engage in more appropriate conversation.

[0115] In another embodiment, the first device 100 may generate an image of user U seated in the passenger seat S2 (or rear seat) and display the generated image on the rearview mirror display device or window. In this case, the first control device 170 generates a person image of user U or a character corresponding to user U seated in the passenger seat S2 and displays the generated image on the rearview mirror display device. In this case, the posture of the person image may be changed according to the actions of user U obtained from the second device 200. The first control device 170 may also control the first speaker 140 to output sound from near the passenger seat S2 when the person image is displayed on the rearview mirror display device 152. This sound is, for example, sound emitted by user U to the second microphone 220, transmitted by the second communication device 210 and acquired via the first communication device 110. The first control device 170, for example, outputs sound to the first speaker 140 that has been localized so that the sound image is heard by the occupant P in the driver's seat from the passenger seat S2.

[0116] <Example 1> In the information processing system 1 described above, at least some of the functions of the first device 100 and the second device 200 may be provided on the management server 300, and at least some of the functions of the management server 300 may be provided on the first device 100 or the second device 200.

[0117] For example, the functions of the image editing unit 276 (or image editing unit 175) and the mobile image display control unit 277 described above may be provided on the management server 300. Also, the functions of the in-vehicle / out-of-vehicle sound output control unit 272 (or in-vehicle / out-of-vehicle sound output control unit 177) may be provided on the management server 300.

[0118] Figure 15 shows an example of the functional configuration of the management server 300A in a modified example. In the example in Figure 15, the management server 300A includes, for example, a communication device 310, an acquisition unit 315, a matching processing unit 320, a provision unit 330A, a billing management unit 340, and a storage unit 350. Compared to the management server 300 shown in Figure 1, the management server 300A has a provision unit 330A instead of a provision unit 330. Therefore, the following explanation will mainly focus on the functions of the provision unit 330A, and the explanation of other components will be omitted. Also, in the modified example shown in Figure 15, for example, information acquired by the first device 100 and the second device 200 is acquired by the acquisition unit 315 of the management server 300A, and various information corresponding to each device is provided by the provision unit 330.

[0119] For example, the providing unit 330A provides the second device 200 with information that has been set as available from among interior images, exterior images, interior sounds, and exterior sounds, based on the availability information set by the occupant P. For example, the providing unit 330A has the same functions as the providing unit 330, and also includes an image editing unit 332, a mobile image display control unit 334, and an interior / exterior sound output control unit 336.

[0120] The image editing unit 332 performs the same processing as the image editing unit 276 (or image editing unit 175) described above. For example, the image editing unit 332 acquires an image captured by the first device 100 and, based on the motion information (head position information) of user U detected by the detection device 230 of the second device 200, switches between an indoor image and an outdoor image depending on whether a predetermined action has been performed and provides them to the second device 200. In this case, the image editing unit 332 may generate an image by superimposing other information (for example, navigation information or POI information) onto the indoor or outdoor image.

[0121] The in-vehicle / out-of-vehicle sound output control unit 336 performs the same processing as the in-vehicle / out-of-vehicle sound output control unit 272 (or in-vehicle / out-of-vehicle sound output control unit 177) described above. For example, the in-vehicle / out-of-vehicle sound output control unit 336 acquires the in-vehicle sound and out-of-vehicle sound collected by the first device 100, and based on the operation information (head position information) of user U detected by the detection device 230 of the second device 200, it changes the volume ratio of the in-vehicle sound and out-of-vehicle sound according to a predetermined operation and provides it to the second device 200.

[0122] The mobile image display control unit 334 provides images generated by the image editing unit 332 to the second device 200, and controls image switching and the volume ratio of in-vehicle sound to external sound based on the actions of user U. For example, the mobile image display control unit 334 outputs a sound from the second speaker 240 that changes the volume ratio of in-vehicle sound to external sound in accordance with the image switching.

[0123] Furthermore, when providing an outdoor image to the second device 200, the providing unit 330A may provide information regarding the distance between the moving body M and an object (e.g., an obstacle) detected by the external sensor 125. The providing unit 330A may also provide the second device 200 with information indicating that the moving body M and the object are approaching, if the distance between the moving body M and the object is within a predetermined distance. Additionally, when a user U requests information from the first device 100, the providing unit 330A may provide the occupant with information indicating that a request has been made and obtain information regarding the availability of the information set by the occupant P. Furthermore, the providing unit 330A may provide the second device 200 with information to allow user U to confirm the information provided by the first device 100. Furthermore, the information provision unit 330A may vary the output manner of the information (images and audio) provided to the second device 200, or provide advertising information to the second device 200 according to the type or amount of information available as set by the user U. The information provision unit 330A may also notify the crew member P from the user U before the information acquired from the first device 100 is provided to the second device 200.

[0124] According to the modified configuration described above, the images and audio provided can be centrally managed by the management server 300A. This allows for a more accurate understanding of the type, amount, and duration of information provided, and enables appropriate management of service fees, etc. Therefore, for example, the provision unit 330A may provide the first device 100 and the second device 200 with advertising information from the advertising information registered in the provision information DB 362, according to the type or amount of information provided, and the fare management unit 340 may adjust the fare, such as reducing the usage fee for the simulated ride service or paying a fee, according to the number of times or duration of viewing of the advertisement by the passenger P or user U. The advertising information may also include information about the mobile body M, or, if the passenger P or user U is a celebrity, information about that person.

[0125] Furthermore, as shown in the modified example, image editing and display control can be performed on the management server 300A side, thereby reducing the processing load on the first device 100 and the second device 200.

[0126] <Summary> The information processing system 1 configured as described above can enhance the sense of presence given to the user U, who is in a different location from the mobile vehicle M. Furthermore, according to the embodiment, it is possible to improve the visibility of the user and the safety of the occupants, thereby further improving traffic safety and contributing to the development of a sustainable transportation system. In the embodiment, the user U is output a sound in which the volume ratio of in-vehicle sound to external sound is adjusted according to the user's movements, so the user U can hear the sounds associated with the movement of the mobile vehicle M as if they were in the passenger seat S2. In addition, an image corresponding to the user's direction of orientation φ as seen from the passenger seat is displayed, so the user U can see the surrounding scenery in conjunction with the sounds associated with the movement of the mobile vehicle M. Furthermore, an image showing the volume ratio of in-vehicle sound to external sound is displayed on the mobile vehicle image display device 250, so the user U can grasp the volume ratio. In addition, the user U can pre-set a first posture corresponding to being inside the vehicle and a second posture corresponding to being outside the vehicle. Therefore, even users who have difficulty standing up due to physical characteristics can hear sounds that are adjusted according to their movements, both inside and outside the vehicle, thereby enhancing the sense of realism in the simulated ride.

[0127] <Usage example> Information processing system 1 can be used in the following ways:

[0128] (A) A configuration in which the occupant P and user U are in a relationship such as family or friends, and the virtual drive is provided to user U. User U can hear sounds inside and outside the vehicle related to the movement of the mobile vehicle M and converse with the occupant P.

[0129] (B) A configuration in which the occupant P is a general user and the user U is a provider of services such as directions and driving instruction. The user U can listen to the sounds around the mobile vehicle M and provide instruction on driving operations based on the timing of using the turn signal and the driving conditions of other vehicles.

[0130] (C) A configuration in which a crew member P is a celebrity and a user U is a regular user, and a commercially based virtual drive is provided to user U. In this case, multiple users U may be associated with one crew member P simultaneously, and for example, the transmission of voice from user U may be turned off. In addition, user U may be able to have a conversation with crew member P for a predetermined period of time by paying an additional fee.

[0131] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0132] 1. Information Processing System 100 1st device 110 First communication device 120 First Microphone Unit 122 Indoor microphone 124 Outdoor microphone 125 External Sensors 130 Camera Unit 132 Indoor Cameras 134 Outdoor Camera 140 First speaker 150 User display device 160 HMI 170 First control device 190 Controlled Equipment 200 2nd device 210 Second communication device 220 Second microphone 230 detection device 232 Directional detection device 234 Head position detection device 236 Motion Sensors 240 Second speaker 250 Mobile Image Display Device 260 HMI 270 Second control device 300, 300A Management Server 310 Communication equipment 315 Acquisition Department 320 Matching Processing Unit 330, 330A supply department 340 Billing Department 350 Storage section M Mobile object

Claims

1. An information management device that manages information acquired from a first device mounted on a mobile vehicle carrying a crew, and provided to a second device used by a user at a location different from the mobile vehicle, An acquisition unit that acquires information from the first device and the second device, The system includes a providing unit that provides information to each of the second devices based on the information acquired by the acquisition unit, The acquisition unit acquires a first sound at a first position of the moving body and a second sound at a second position different from the first position using the first device, and acquires the user's operation information using the second device. The providing unit provides the second device with a sound whose volume ratio between the first sound and the second sound is adjusted according to the user's actions. Information management device.

2. The first sound is a sound collected inside the room of the mobile device. The second sound is a sound collected outside the mobile body. The aforementioned supply unit is, Depending on whether the user is in a first posture corresponding to being inside the room of the mobile body, the second device is provided with a sound whose volume is lower than that of the first sound. In response to the user performing an action from the first posture to a second posture corresponding to being outside the room of the mobile body, the second device is provided with a sound whose volume is greater than that of the first sound. The information management device according to claim 1.

3. The first posture is a seated posture. The second posture is an upright posture. The information management device according to claim 2.

4. The first posture is a seated position facing the direction of travel of the moving body. The second posture is a seated position in which the person faces a direction perpendicular to the direction of movement of the moving body. The information management device according to claim 2.

5. The first posture is a posture in which the moving body is viewed in the direction of travel. The second posture is one in which the moving body is viewed in a direction perpendicular to the direction of movement. The information management device according to claim 2.

6. The first sound is a sound inside the room of the mobile body. The second sound is an external sound from the mobile body. The aforementioned supply unit is, When the window of the moving body is closed, the second device is provided with a sound whose volume is lower than that of the first sound. In response to the user performing the action of opening the window of the mobile body, the second device is provided with a sound that is louder than the first sound. The information management device according to claim 1.

7. The acquisition unit acquires indoor and outdoor images of the moving object using the first device. The aforementioned supply unit is, The second device is provided with an indoor image depending on whether the user is in the first posture. The outdoor image is provided to the second device in response to the user performing an action from the first posture to the second posture. The information management device according to claim 2.

8. The providing unit provides the second device with an image showing the volume ratio of the first sound and the second sound. The information management device according to claim 1.

9. The computer of the information management device manages information acquired from a first device mounted on a mobile vehicle carrying the crew, and provided to a second device used by a user at a location different from the said mobile vehicle. Information is obtained from the first device and the second device, Based on the acquired information, information is provided for each of the second devices. The first device acquires a first sound at a first position of the moving body and a second sound at a second position different from the first position, and the second device acquires the user's operation information. The second device is provided with a sound whose volume ratio between the first sound and the second sound is adjusted according to the user's actions. Information management method.

10. The computer of the information management device manages information acquired from a first device mounted on a mobile vehicle carrying the crew, and provided to a second device used by a user at a location different from the said mobile vehicle. Information is acquired from the first device and the second device. Based on the acquired information, information is provided for each of the second devices. The first device acquires a first sound at a first position of the moving body and a second sound at a second position different from the first position, and the second device acquires the user's operation information. The second device is provided with a sound whose volume ratio between the first sound and the second sound is adjusted according to the user's actions. program.

Images