Information processing system

Abstract

The invention provides an information processing system which enables a user in a place different from a moving body to perform more appropriate simulated riding. An information management system includes a first device mounted on a moving body, a plurality of second devices used by a plurality of users at a location different from the moving body, an information acquisition unit, a provision unit, and a plurality of presentation units. The information acquisition unit acquires predetermined information transmitted by the first device. The provision unit provides the predetermined information acquired by the information acquisition unit to the plurality of second devices. The presentation unit presents the prescribed information provided to the plurality of second devices by the provision unit. When an operation input by a first user is performed by one of the plurality of second devices, a presentation unit of a second device other than the one second device presents an output result on the basis of the operation input performed by the one second device.

Description

Technical Field

[0001] This invention relates to an information processing system. Background Technology

[0002] In recent years, efforts have intensified to provide sustainable transportation systems that also take into account vulnerable groups among transport participants, particularly the elderly, people with disabilities, and children. To achieve this, research and development are particularly focused on further improving the safety and convenience of transportation through the development of mobility facilities geared towards vulnerable transport users.

[0003] In this regard, a system is known that is configured to store images externally by storing image data in a time sequence on an external server (archiving device) and to be able to play the images at any time (for example, see Patent Document 1 (Japanese Patent Application Publication No. 2003-335197)). Summary of the Invention

[0004] [The problem the invention aims to solve]

[0005] However, when using devices in locations other than the mobile vehicle to provide a simulated experience as if riding in the mobile vehicle (hereinafter referred to as "simulated ride"), it is a problem that this can burden users who are taking simulated rides in locations different from the mobile vehicle.

[0006] This application aims to address the aforementioned issues by providing users in locations different from the actual vehicles they are in with a more appropriate simulated ride. This, in turn, contributes to the development of sustainable transportation systems and supports better connections between urban areas and surrounding regions.

[0007] [Solution to the problem]

[0008] The information processing system involved in this invention adopts the following structure.

[0009] (1): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: a first device mounted on a mobile body; a plurality of second devices, which are used by a plurality of users in different locations from the mobile body; an information acquisition unit that acquires predetermined information sent by the first device; a provision unit that provides the predetermined information acquired by the information acquisition unit to the plurality of second devices; and a plurality of prompting units that prompt the predetermined information provided by the provisioning unit to the plurality of second devices, wherein when one of the plurality of second devices performs an operation input based on the first user, the prompting unit of the second device other than the second device prompts the output result based on the operation input performed by the second device.

[0010] (2): Based on the above (1) scheme, the information processing system may also have a user information acquisition unit that acquires user information of the user of the second device, and the second device used by the first user may also have a production unit that produces an output signal for prompting the output result based on the user information acquired by the user information acquisition unit.

[0011] (3): Based on the above (1) scheme, the second device may also have a gaze detection unit that detects the user's gaze direction, and the prompting unit may also prompt and output the result based on the gaze direction of the user detected by the gaze detection unit.

[0012] (4): Based on the above (1) scheme, the second device may also have a POI acquisition unit that obtains POI information, i.e., point of interest information. The POI acquisition unit may also obtain POI information based on the location information of the first device when the prompting unit prompts the output result. The prompting unit may also prompt the POI information obtained by the POI acquisition unit together with the output result.

[0013] (5): Based on the above (3) scheme, the second device may also have a POI acquisition unit that obtains POI information, i.e., point of interest information. The POI acquisition unit may also obtain POI information based on the user's gaze direction detected by the gaze detection unit. The prompting unit may also determine the position of the prompt output result based on the POI information obtained by the POI acquisition unit.

[0014] (6): Based on the above (1) scheme, the second device may also include: a POI acquisition unit that acquires POI information, i.e., point of interest information; and a selection unit that, when the POI acquisition unit acquires multiple POI information, selects the POI information closest to the mobile body from the multiple POI information, and the prompting unit may also prompt the POI information selected by the selection unit.

[0015] (7): Based on the above (1) scheme, the prompting unit can also change the prompting position of the output result based on the speed of the moving body.

[0016] (8): Based on the above (1) scheme, the information processing system may also have a mobile body behavior detection unit that detects that the mobile body has turned right or left. The prompting unit may also change the prompting content of the output result when the mobile body behavior detection unit detects that the mobile body has turned right or left after prompting the output result.

[0017] (9): Based on the above (1) scheme, the information processing system may also include a production unit, which, when a first user performs an operation input by one of the multiple second devices, produces a notification message containing the output result for the first device based on the operation input performed by one of the second devices.

[0018] (10): Based on the above (1) scheme, the information processing system may also include a sound information acquisition unit that acquires sound information based on the user's voice, and the prompting unit may also prompt the output result based on the operation input and the sound information.

[0019] (11): Based on the above (4) scheme, the information processing system may also include a voice information acquisition unit that acquires voice information based on the user's voice, and the prompting unit may also prompt the POI information and the voice information acquired by the voice information acquisition unit in association with the POI information together with the output result.

[0020] [Invention Effects]

[0021] According to the above schemes (1) to (11), users in locations different from the moving vehicle can more appropriately simulate riding a vehicle. Attached Figure Description

[0022] Figure 1 This is a diagram showing the usage environment of the information processing system 1 and the management server 300.

[0023] Figure 2 This is a diagram representing an example of the content of user data 360.

[0024] Figure 3 This is a structural diagram of the first device 100.

[0025] Figure 4 This is a diagram showing an example of the configuration of a portion of the first device 100 in the mobile body M.

[0026] Figure 5 This is a structural diagram of the second device 200.

[0027] Figure 6 It is a diagram used to illustrate the image corresponding to the pointing direction.

[0028] Figure 7 This is a diagram illustrating a first example of the functional structure of the first control device 170 and the second control device 270.

[0029] Figure 8 This is a second example of a diagram showing the functional structure of the first control device 170 and the second control device 270.

[0030] Figure 9 This is a sequence diagram representing an example of the processing performed by information processing system 1.

[0031] Figure 10 This is an example of an image IM10 representing a query.

[0032] Figure 11 This is an example of an image IM20 showing the result of a request.

[0033] Figure 12A This is a sequence diagram representing the first example of the processing performed by information processing system 1.

[0034] Figure 12B This is a sequence diagram representing the second example of the processing performed by information processing system 1.

[0035] Figure 12C This is a sequence diagram representing the third example of the processing performed by information processing system 1.

[0036] Figure 13A It is a diagram used to illustrate examples of information provided.

[0037] Figure 13B It is a diagram used to illustrate examples of information provided.

[0038] Figure 13C It is a diagram used to illustrate examples of information provided.

[0039] Figure 14 This is a third example of a diagram showing the functional structure of the first control device 170 and the second control device 270.

[0040] Figure 15A This is a sequence diagram representing the fourth example of the processing performed by information processing system 1.

[0041] Figure 15B This is a sequence diagram representing the fifth example of the processing performed by information processing system 1.

[0042] Figure 16 It is a diagram used to illustrate examples of information provided.

[0043] [Explanation of Labels in the Attached Image]

[0044] 1. Information Processing System

[0045] 100 First Device

[0046] 110 First Communication Device

[0047] 120 First Microphone

[0048] 125 External Sensors

[0049] 130 camera unit

[0050] 132 Indoor Camera

[0051] 134 Outdoor Camera

[0052] 140 First loudspeaker

[0053] 150 User display devices

[0054] 160 HMI

[0055] 170 First Control Device

[0056] 190 Controlled Object Equipment

[0057] 200 Second Device

[0058] 210 Second Communication Device

[0059] 220 Second Microphone

[0060] 230 Detection device

[0061] 232 Direction Detection Device

[0062] 234 Head position detection device

[0063] 236 Motion Sensors

[0064] 238 Line of sight detection device

[0065] 240 Second speaker

[0066] 250 Moving Body Image Display Device

[0067] 260 HMI

[0068] 270 Second Control Device

[0069] 300 Management Server

[0070] 310 Communication device

[0071] 315 Information Acquisition Department

[0072] 320 Matching Processing Unit

[0073] 330, 330A Supply Department

[0074] 340 Expense Management Department

[0075] 350 Storage Unit

[0076] M is a moving body. Detailed Implementation

[0077] Hereinafter, embodiments of the information processing system of the present invention will be described with reference to the accompanying drawings. It should be noted that the following description applies to information processing systems using information processing devices. An information processing system may include, for example, a first device mounted on a mobile body carried by occupants, a second device used by a user at a location different from the mobile body, and an information processing device. A mobile body refers to, for example, a vehicle, but can be any mobile body that occupants can ride on (e.g., a ship, an aircraft). Occupants are primarily the driver of the mobile body, but can also be occupants other than the driver.

[0078] The first and second devices operate in a state where a telephone conversation is conducted by transmitting sound collected by a microphone to the other party and playing it through a speaker. Furthermore, a portion of the image captured by the camera unit of the first device is displayed on the second device, thus providing MR (Mixed Reality) to the second device. Therefore, the user of the second device, even though in a different location from the mobile device, can experience the feeling of actually riding in the mobile device (simulated ride experience). The user, through the first device, engages in a conversation with the simulated user of the mobile device, thus gaining the feeling that the user is actually riding in the mobile device together. Alternatively, a portion of the image captured by the camera unit of the first device can be displayed on the second device, thus providing VR (Virtual Reality) to the second device. Therefore, the user of the second device, even though in a different location from the mobile device, can experience the feeling of actually riding in the mobile device (simulated ride experience). Furthermore, the user, through the first device, engages in a conversation with the simulated user of the mobile device, thus gaining the feeling that the user is actually riding in the mobile device together. Moreover, the user can be fully immersed in the world of images while using the application. As mentioned above, the simulated experience where the user feels as if they are actually riding in a moving vehicle is sometimes referred to as "simulated ride." The first and second devices do not need to be in a one-to-one relationship; rather, one of multiple first devices can be matched with multiple second devices in a one-to-many relationship, thereby functioning as an information processing system. In the latter case, for example, it is possible for one passenger to communicate simultaneously with multiple users or for one passenger to communicate sequentially with multiple users.

[0079] <Basic Structure>

[0080] Figure 1This diagram illustrates the usage environment of the information processing system 1 and the management server 300. The information processing system 1 includes a first device (mobile device) 100 mounted on a mobile body M carried by passenger P, a second device (user device) 200-1 used by user U1 at a different location from the mobile body M (excluding a coincidentally nearby location), and a second device (user device) 200-2 used by user U2. The first device 100, second device 200-1, second device 200-2, and management server 300 communicate with each other via network NW. The information processing system 1 may or may not include the management server 300. Hereinafter, either user U1 or user U2 will be referred to as user U. Either the second device 200-1 or the second device 200-2 will be referred to as second device 200.

[0081] Network NW includes, for example, at least one of the following: Internet, WAN (Wide Area Network), LAN (Local Area Network), mobile communication network, cellular network, etc. Management server 300 is an example of an "information processing device." It should be noted that management server 300 can also be implemented by server devices and storage devices assembled in a cloud computing system. In this case, the functions of management server 300 can also be implemented by multiple server devices and storage devices in the cloud computing system. The first device 100 mounted on the mobile body M can also be implemented by multiple units.

[0082] The management server 300 manages the information provided to the first device 100 and the second device 200 respectively, and manages the communication between the first device 100 and the second device 200. The management server 300 includes, for example, a communication device 310, an information acquisition unit 315, a matching processing unit 320, a provisioning unit 330, a cost management unit 340, and a storage unit 350. The information acquisition unit 315, the matching processing unit 320, the provisioning unit 330, and the cost management unit 340 are implemented, for example, by executing programs (software) using a hardware processor such as a CPU (Central Processing Unit).

[0083] Some or all of these components can be implemented using hardware (including the circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or through the coordinated use of software and hardware. The program can be pre-stored on storage devices such as HDD (Hard Disk Drive) and flash memory (storage devices with non-transitory storage media), or on removable storage media such as DVDs and CD-ROMs (non-transitory storage media), and installed via the storage media mounted on a drive. The program can also be stored on the management server 300 and updated appropriately via a connection to the network NW.

[0084] The communication device 310 is a communication interface for connecting to a network NW. It should be noted that the communication between the communication device 310 and the first device 100, as well as the communication between the communication device 310 and the second device 200, is performed, for example, using TCP / IP (Transmission Control Protocol / Internet Protocol).

[0085] The information acquisition unit 315 acquires various information sent by the first device 100, the second device 200, and other external devices via the network NW. This information may include dynamic image data, driver's voice data, and vehicle data (such as METER (instrument panel) and IVI (in-vehicle infotainment) system display content).

[0086] The matching processing unit 320 is implemented, for example, by executing a program (command group) stored in a storage medium using a processor such as a CPU. For example, when the communication device 310 receives a matching request from user U via second device 200 or from passenger P via first device 100, the matching processing unit 320 performs a matching between user U and passenger P by referring to user data 360. Furthermore, the matching processing unit 320 uses the communication device 310 to send communication identification information of passenger P's first device 100 to the second device 200 of the successfully matched user U, and also sends communication identification information of user U's second device 200 to the first device 100 of the successfully matched passenger P. Upon receiving this information, the first device 100 and the second device 200 can perform high-real-time communication, for example, using UDP (User Datagram Protocol).

[0087] The providing unit 330 generates information to be provided to the first device 100 and the second device 200 respectively based on various information obtained by the information acquisition unit 315, and sends the generated information to the target device. The providing unit 330 generates information representing the processing results processed by the matching processing unit 320, the cost information (accounting information) managed by the cost management unit 340, etc., and provides the generated information to the target device.

[0088] The Fee Management Department 340 manages the fees charged to user U based on information provided to user U, and the fees charged to passenger P based on information provided to passenger P of the mobile vehicle M. The Fee Management Department 340 may also manage, for example, the remuneration paid to user U and passenger P based on information provided from user U and passenger P. The Fee Management Department 340 may also perform processing related to the accounting of user U and passenger P.

[0089] The storage unit 350 can be implemented using various storage devices described above, or SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The storage unit 350 stores, for example, user data 360, information DB362, programs, and other various types of information.

[0090] Figure 2 This diagram illustrates an example of the content of user data 360. User data 360 includes a passenger list 360A and a user list 360B. In passenger list 360A, for example, the passenger ID of passenger P of mobile body M, their communication identification information (IP address, etc.), the user ID of user U who is the matching target, the mobile body information of the passenger, and the passenger's settings regarding whether or not they can provide information are mutually correlated. The mobile body information includes, for example, equipment information mounted on mobile body M (equipment information), and vehicle specification information indicating the size and shape of mobile body M. The mobile body information may also include information transmitted from mobile body M at predetermined intervals related to the current location, destination, and surrounding conditions of mobile body M (e.g., while traveling on a coastal road).

[0091] In the user list 360B, a correspondence is established between user ID, their communication identification information (IP address, etc.), the passenger P who is matched, and user information. User information may include physical-related information (e.g., height, sitting height) and information that can predict physical characteristics (e.g., age). The "can be provided" information is information that the mobile body M can or cannot provide, for example, set by the passenger P. The "can be provided" information can also be set according to the equipment carried by the mobile body M or according to the user U. Examples of "can be provided" information include, but are not limited to, "permitted image provision," "disallowed sound provision," "permitted indoor image provision but not outdoor image provision," "disallowed passenger image provision," and "disallowed use of navigation device." The "can be provided" information may also include the cost (service provision fee) for making the provision possible. User data 360 is not limited to this as long as it contains this information. Figure 2 The shape shown can also be generated in any shape.

[0092] The information provided in DB362 stores various information provided to the user U or passenger P. This information includes, for example, map information, POI (Point of Interest) information, and computer-generated images (e.g., CG images of people, symbols, icons, etc.). POI information, such as information related to various shops, theme parks, and landmarks at each location, can be included in the map information. Audio information may also be included. The information provided in DB362 may include advertising information. This advertising information may include, for example, advertisements related to the mobile vehicle M, advertisements related to the user U or passenger P, and other advertisements related to the goods and services of shops. It should be noted that inserted advertising information can be managed separately from indoor and outdoor information, and will differ from the inserted advertising information during subsequent archiving and distribution (e.g., closed shops become the latest shops, and introduced menus become the latest menus). Advertising information may include, for example, animated images and audio.

[0093] Figure 3 This is a structural diagram of the first device 100. The first device 100 includes, for example, a first communication device 110, a first microphone 120, an external sensor 125, a camera unit 130, a first speaker 140, a user display device 150, an HMI (Human Machine Interface) 160, and a first control device 170. The first control device 170 is connected to a control object device 190 mounted on the mobile body M.

[0094] The first communication device 110 is a communication interface for communicating with the communication device 310 of the management server 300 and the second communication device 210 of the second device 200 (described later) via a network NW.

[0095] The first microphone 120 collects at least the sound emitted by the occupant P. The first microphone 120 may also be installed indoors within the mobile body M and has the sensitivity to also collect external sounds from the mobile body M. The first microphone 120 may also include a microphone installed indoors within the mobile body M and a microphone installed outside the mobile body M. Hereinafter, the sound information obtained by the microphone installed indoors will sometimes be referred to as "indoor sound information." The sound collected by the first microphone 120 is transmitted, for example, via the first control device 170 to the management server 300 or the second device 200 by the first communication device 110. In cases where a microphone cannot be installed outside the mobile body M, the indoor sound information can be processed to simulate outdoor sound information based on driving information (vehicle speed, acceleration / deceleration, road vibration, etc.) and the surrounding driving environment. Alternatively, the speaker's position relative to the mobile body M (whether the speaker is indoors or outdoors) can be recorded, and the collected sound can be processed according to this positional relationship.

[0096] An external sensor 125 detects the position of objects around the moving body M. The external sensor 125 can be, for example, a radar device, a LIDAR (Light Detection and Ranging) sensor, or various other proximity sensors. The radar device emits millimeter-wave or other electromagnetic waves around the moving body M and detects the reflected waves (reflected waves) to at least detect the object's position (distance and orientation). The radar device can also detect the object's position and velocity using FM-CW (Frequency Modulated Continuous Wave). The LIDAR sensor illuminates the moving body M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light, detecting the distance to the object based on the time from emission to reception. The illuminated light can be, for example, a pulsed laser. The radar device or LIDAR sensor can be mounted at any location on the moving body M. The external sensor 125 can also detect surrounding objects using images captured by the outdoor camera 134 of the camera unit 130.

[0097] Camera unit 130 includes, for example, an indoor camera 132 and an outdoor camera 134. First speaker 140 outputs sound emitted by user U, obtained via first communication device 110. Details regarding the configuration of camera unit 130 and first speaker 140 will be provided later. Figure 4 To illustrate.

[0098] The user display device 150 displays the user U as if the user were present inside the moving body M. For example, the user display device 150 displays a hologram showing the user U in a portion corresponding to the reflector or window of the moving body M.

[0099] HMI160 includes a touch panel, a voice response device (smart agent device), etc. HMI160 receives various instructions from occupant P to the first device 100 and provides various information to occupant P.

[0100] The first control device 170 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (command group), and controls various parts of the first device 100 by executing the command group through the processor.

[0101] The control device 190 is, for example, a navigation device mounted on the mobile body M that guides the user to a destination, or a driving support device that controls one or both of the steering and speed of the mobile body M to support the occupant P's driving. The control device 190 includes, for example, a seat drive mechanism capable of adjusting the seat's position (forward, backward, left, right), orientation, and height. When viewing images via the second device 200, if the camera unit 130 of the first device 100 is mounted on the seat, the impact on the image can be suppressed by preventing seat movement. Even when seat movement is permitted, processing such as changing the viewing angle can be performed to avoid affecting the image when the seat has moved. If the user U wishes to view images outside the viewing angle of the current camera unit 130, the seat drive mechanism can be controlled as a request from the second device 200.

[0102] Figure 4 This diagram illustrates a configuration example of a portion of the first device 100 in the mobile body M. An interior camera 132 is mounted, for example, with the aid of accessory 132A, to the neck pillow of the co-driver's seat S2 (an example of a "prescribed seat"), and is positioned slightly offset from the backrest of the co-driver's seat S2 towards the direction of travel of the mobile body M. The interior camera 132 has a wide-angle lens, capable of capturing the area indicated by the shaded region 132B in the diagram. The interior camera 132 can capture images not only of the interior of the mobile body M but also of the exterior through the windows. In the following description, the co-driver's seat S2 is assumed to be a prescribed seat, but the prescribed seat could also be a rear seat or other seats.

[0103] The outdoor camera 134 may include, for example, multiple sub-outdoor cameras 134-1 to 134-4. By combining the images captured by the multiple sub-outdoor cameras 134-1 to 134-4, a panoramic image of the exterior of the moving body M can be obtained. The outdoor camera 134 may also include, instead of these (or in addition to), a wide-angle camera mounted on the roof of the moving body M.

[0104] It should be noted that an additional camera capable of capturing images from behind the co-pilot's seat S2 can be added to the indoor camera 132. The moving body image described later can be generated into a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 through the first control device 170, or by appropriately combining images captured by indoor cameras 132 with images captured by outdoor cameras 134.

[0105] The first speaker 140 outputs the sound of the user U obtained via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, sub-first speaker 140-1 is located in the center of the dashboard. Sub-first speaker 140-2 is located at the left end of the dashboard. Sub-first speaker 140-3 is located at the right end of the dashboard. Sub-first speaker 140-4 is located at the lower part of the left door. Sub-first speaker 140-5 is located at the lower part of the right door. When the first speaker 140 outputs the sound of the user U, the first control device 170, for example, outputs the sound from sub-first speakers 140-2 and 140-4 at the same volume level, and shuts off the other sub-first speakers, thereby performing sound image localization for the occupant P seated in the driver's seat S1 so that the sound can be heard from the front passenger seat S2. The sound image localization method is not limited to volume adjustment; it can also be performed by staggering the phase of the sounds output by each sub-first speaker. For example, in the case of sound image localization in a manner where the sound is heard from the left, the timing of the sound output from the left sub-first speaker can be slightly earlier than the timing of the same sound output from the right sub-first speaker.

[0106] When the first control device 170 outputs the user U's voice through the first speaker 140, it can also perform acoustic imaging localization so that the occupant P can hear the sound from a position on the co-pilot's seat S2 at a height corresponding to the height of the user U's head, and then output the user U's voice through the first speaker 140. In this case, the first speaker 140 needs to have multiple sub-first speakers 140-k (k is a plurality of natural numbers) with different heights.

[0107] Figure 5This is a structural diagram of the second device 200. The second device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a moving body image display device 250, an HMI 260, and a second control device 270. The detection device 230 includes, for example, a pointing direction detection device 232, a head position detection device 234, a motion sensor 236, and a gaze detection device 238.

[0108] The second communication device 210 is a communication interface for communicating with the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0109] The second microphone 220 collects the sound emitted by the user U. The sound collected by the second microphone 220 is transmitted to the first communication device 110 via the second communication device 210, for example, through the second control device 270.

[0110] The pointing direction detection device 232 is a device for detecting the pointing direction. The pointing direction refers to the orientation of the user U's face or line of sight, or an orientation derived from both. Hereinafter, the pointing direction is defined as an angle in the horizontal plane, i.e., an angle without a vertical component, but the pointing direction can also be an angle that includes a vertical component. The pointing direction detection device 232 may also include physical sensors (e.g., accelerometer, gyroscope, etc.) mounted on the VR glasses described later, or infrared sensors that detect multiple positions of the user U's head, or a camera that captures images of the user U's head. In any case, the second control device 270 calculates the pointing direction based on information input from the pointing direction detection device 232. Various techniques are known in this regard, therefore detailed descriptions are omitted.

[0111] The head position detection device 234 is a device for detecting the position (height) of the user U's head. For example, one or more infrared sensors or optical sensors positioned around the chair where the user U sits are used as the head position detection device 234. In this case, the second control device 270 detects the position of the user U's head based on the presence or absence of detection signals detected by one or more infrared or optical sensors. The head position detection device 234 can also be an accelerometer sensor mounted on the VR glasses. In this case, the second control device 270 detects the position of the user U's head by integrating the component obtained by subtracting gravitational acceleration from the output of the accelerometer sensor. The head position information obtained in this way is provided to the second control device 270 as height information. The user's head position can also be obtained based on the user U's operation of the HMI 260. For example, the user U can input their height digitally into the HMI 260, or input their height using a DIP switch included in the HMI 260. In these cases, the head position, i.e., the height information, is calculated based on the height. User U can also input discrete values ​​into the HMI260, such as body size (large / medium / small), instead of continuous values. In this case, height information is obtained based on the body size information. Alternatively, the user's head height can be easily obtained based on the typical adult body size (which may be gender-specific) without specifically obtaining the user's head height.

[0112] Motion sensor 236 is a device for recognizing the posture operations performed by user U. For example, a camera that captures images of the upper body of user U is used as motion sensor 236. In this case, the second control device extracts feature points (fingertips, wrists, elbows, etc.) of user U's body based on the images captured by the camera, and recognizes the posture operations of user U based on the movement of the feature points.

[0113] The gaze detection device 238 is a device for detecting gaze. Gaze refers to the direction of the user U's gaze. The gaze detection device 238 can also be a camera installed in the VR glasses described later to capture the gaze. The second control device 270 derives the gaze based on information input from the gaze detection device 238. Various techniques related to this are well known, therefore detailed descriptions are omitted.

[0114] The second speaker 240 outputs the sound emitted by occupant P, obtained via the second communication device 210. The second speaker 240, for example, has a function to change the direction of the sound. The second control device 270 causes the second speaker to output sound so that the user U hears the sound from the position of occupant P as viewed from the co-pilot's seat S2. The second speaker 240 may also include multiple sub-second speakers 240-n (n being multiple natural numbers). In this case, the second control device 270 can also perform sound image localization by adjusting the volume of each sub-second speaker 240-n; if headphones are included with the VR glasses, the headphones can also be used for sound image localization.

[0115] The moving body image display device 250 displays an image captured by the camera unit 130 (or an image that has undergone the aforementioned combination processing, hereinafter referred to as a moving body image) that corresponds to the direction of view from the co-driver's seat. Figure 6 This is a diagram illustrating the image corresponding to the pointing direction. In the example shown, the VR glasses 255 include a physical sensor serving as a pointing direction detection device 232 and a head position detection device 234, as well as a moving body image display device 250. The second control device 270, for example, sets the center of the user U's head or the center of the VR glasses 255 as Ω, and uses a pre-calibrated direction as a reference direction to detect the direction in which the VR glasses 255 is facing as the pointing direction φ. Various methods are known regarding the functions involved, therefore detailed descriptions are omitted.

[0116] The moving body image display device 250 displays an image A2 of the moving body image A1 (which has an angle of about 240 degrees in the figure, but the screen angle can also be expanded by combining processing as described above) with the positive and negative α angle range centered on the pointing direction φ, facing the user U.

[0117] return Figure 5 HMI260 is a touch panel, a voice-responsive device (smart device), or a switch as described above. HMI260 accepts various instructions from user U to the second device 200.

[0118] The second control device 270 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (command set), and controls various parts of the second device 200 by executing the command set through the processor. It should be noted that the second device 200 may also be... Figure 5 All the functions shown are integrated into the VR glasses.

[0119] <Functional Structure of First Control Device 170 and Second Control Device 270>

[0120] The functional structures of the first control device 170 and the second control device 270 are described below. It should be noted that the first and second examples shown below mainly involve the generation and transmission of information provided between the first control device 170 and the second control device 270, while the management server 300 manages the matching of simulated rides between passenger P and user U.

[0121] [First example]

[0122] Figure 7 This diagram illustrates a first example of the functional structure of the first control device 170 and the second control device 270. In this first example, the first control device 170 includes a matching request / acceptance unit 171, a voice output control unit 172, an image transmission unit 173, and a device cooperation unit 174. The second control device 270 includes a matching request / acceptance unit 271, a voice output control unit 272, a pointing direction detection unit 273, a head position detection unit 274, a posture input detection unit 275, an image editing unit 276, a prompting unit 277, a receiving unit 278, a user information acquisition unit 279, a gaze detection unit 280, a production unit 281, and a moving body behavior detection unit 282. These functional units are implemented, for example, by executing programs (command groups) using a processor such as a CPU. Some or all of these components can also be implemented using hardware (including circuitry) such as LSI, ASIC, FPGA, and GPU, or through the coordinated use of software and hardware.

[0123] The matching request / approval unit 171 uses HMI 160 to accept input of a matching request (simulated ride request) from passenger P and sends it to management server 300. It also uses HMI 160 to accept input of approval for the matching request received from management server 300 and sends it to management server 300. The matching request / approval unit 171 controls the first communication device 110 to make the second device 200 of the user U, which enables the matching, a communication partner. For example, the matching request / approval unit 171 can also, upon approval of a matching request received from management server 300, receive settings from passenger P indicating whether or not information is permitted to be provided.

[0124] The sound output control unit 172 controls the first speaker 140 as described above.

[0125] After initiating communication with the second device 200, the image transmitting unit 173 transmits a moving body image A1 to the second device 200 using the first communication device 110. The moving body image A1 may include, for example, an indoor image captured by the indoor camera 132 and / or an outdoor image captured by the outdoor camera 134.

[0126] The onboard equipment coordination unit 174 controls the controlled object device 190 based on the instruction signal input from the second device 200. The onboard equipment coordination unit 174 obtains driving information of the mobile body M from the controlled object device 190 and transmits it from the first communication device 110 to the second device 200. It should be noted that the onboard equipment coordination unit 174 can also perform control of the controlled object device 190 if the user U has authorized the user P to do so.

[0127] The matching request / approval unit 271 uses HMI 260 to accept input of a matching request (simulated ride request) from user U and sends it to management server 300. It also uses HMI 260 to accept input of approval for the matching request received from management server 300 and sends it to management server 300. The matching request / approval unit 271 controls the second communication device 210 to make the first device 100 of the matched passenger P a communication partner. For example, the matching request / approval unit 271 can also, after prompting user U based on the matching result (request result) obtained from management server 300 to allow user U to confirm what information to provide, receive an instruction from user U on whether to simulate a ride.

[0128] The sound output control unit 272 controls the second speaker 240 as described above.

[0129] The pointing direction detection unit 273 detects the pointing direction φ based on the output of the pointing direction detection device 232. The head position detection unit 274 detects the height of the user U's head based on the output of the head position detection device 234. The head position can be represented as three-dimensional coordinates, or only the head height can be detected as the head position.

[0130] The gesture input detection unit 275 detects the gesture input of the user U based on the output of the motion sensor 236.

[0131] Image editing unit 276, for example, performs processing to extract image A2 from moving body image A1, corresponding to the pointing direction φ when viewed from the co-driver's seat S2. Figure 6 It should be noted that, in Figure 6 In the example, the pointing direction φ for left and right movements toward the user U's head is shown, but it is also possible to perform image cropping corresponding to the pointing direction for up and down movements toward the head.

[0132] The prompting unit 277 causes the mobile image display device 250 to display (prompt) the image A2 captured by the image editing unit 276. At this time, the image editing unit 276 can also cause the mobile image display device 250 to display an image corresponding to the pointing direction φ when viewed from the height indicated by the user U's head height information. For example, the image editing unit 276 switches between displaying an indoor image and an outdoor image based on the distance of the user U's head from the reference position (e.g., the amount of upward or lateral movement). The prompting unit 277 obtains an output signal from the second communication device 210 from another second device 200 and displays an icon on the mobile image display device 250 based on the position information contained in the obtained output signal.

[0133] The receiving unit 278 accepts operation input from the user on the image A2 displayed on the moving image display device 250 by the prompting unit 277.

[0134] The user information acquisition unit 279 acquires user information of users of other second devices 200 among the plurality of second devices 200. One example of user information is information used for communication with users of each of the other second devices 200.

[0135] The gaze detection unit 280 detects the gaze direction based on the output of the gaze detection device 238. The prompting unit 277 determines the position on the image (video) displayed on the mobile image display device 250 and displays an icon at the determined position. For example, when the user performs operation input on the HMI 260, the prompting unit 277 determines the position on the image (video) displayed on the mobile image display device 250 indicated by the operation input and displays an icon at the determined position. The production unit 281 produces an output signal including the position information determined by the prompting unit 277 and sends the output signal from the second communication device 210 to other second devices 200. For example, when the user is watching or listening to an image (video), they aim at any position on the image (video) on the HMI 260 they are holding and press the input button to perform operation input. Specifically, the user aligns the mark displayed on the image (video) on the HMI 260 they are holding with any position and presses the input button to perform operation input.

[0136] Here, the prompting unit 277 can also determine the display area on the moving image display device 250 based on the position aimed by the HMI 260. An icon can also be displayed within the determined display area. For example, the prompting unit 277 can display the icon at the center of the display area.

[0137] The production unit 281 can also generate notification information including the generated output signal and send this notification information from the second communication device 210 to the first device 100. This allows the user's reaction to be communicated to the occupant P, enabling P to understand the user's feelings. Alternatively, the user's reaction can be communicated to the occupant P via operation input to the HMI 260. This avoids notifications that could hinder P's driving.

[0138] For example, the prompting unit 277 determines the position on the moving object image display device 250 based on the gaze direction detected by the gaze detection unit 280, and displays an icon on the object (object) displayed at the determined position. The production unit 281 generates an output signal including the position information determined by the prompting unit 277, and sends the output signal from the second communication device 210 to other second devices 200.

[0139] Here, the prompting unit 277 may also determine the display area on the moving object image display device 250 based on the viewing direction detected by the viewing direction detection unit 280, and display an icon on the object (object) displayed in the determined display area. For example, the prompting unit 277 may also display an icon in the center of the display area. An example of the display area is a bounding box. In this case, the production unit 281 produces an output signal including information representing the display area determined by the prompting unit 277, and sends the output signal to the other second device 200.

[0140] If the moving object M (vehicle) is traveling along the path, the shape and color of the object will not change significantly. If there is no right or left turn, the detected object will appear to disappear from the screen after approaching the vehicle. The prompt unit 277 can also change the determined position or the position of the display area based on the movement of the moving object M. The prompt unit 277 can also keep the icon continuously displayed in the same position based on the positional relationship between the determined position and the display area.

[0141] The driving information of the mobile body M transmitted by the first device 100 is received by the second communication device 210. The mobile body behavior detection unit 282 obtains the driving information of the mobile body M from the second communication device 210, and detects whether the mobile body M has made a right turn or a left turn based on the obtained driving information of the mobile body M.

[0142] When the motion detection unit 282 detects that the mobile body M has turned right or left after the prompt icon, the prompt unit 277 changes the prompt content of the output result. For example, the prompt unit 277 displays the icon in a different way than when the mobile body M has not turned left or right. For example, the prompt unit 277 displays the icon with a dashed line. For example, the prompt unit 277 determines whether the icon is continuously displayed at the position determined based on the dynamic image displayed on the mobile body image display device 250, and displays the icon with a dashed line when it is determined that it will no longer be displayed.

[0143] [Second example]

[0144] Figure 8 This is a second example of a diagram illustrating the functional structure of the first control device 170 and the second control device 270. (Compared to...) Figure 7 Compared to the first example, the difference lies in that the first control device 170 includes an image editing unit 175, while the second control device 270 does not include an image editing unit 276 but includes a direction sending unit 283. The other constituent elements have essentially the same functions as in the first example, therefore further explanation is omitted.

[0145] The pointing direction transmitting unit 283 uses the second communication device 210 to transmit the pointing direction φ detected by the pointing direction detecting unit 273 to the first device 100.

[0146] The image editing unit 175 performs processing to extract image A2 from the moving body image A1, which is viewed from the co-pilot's seat and corresponds to the pointing direction φ (the pointing direction sent from the second device 200). Figure 6 At this time, the image editing unit 175 can also obtain the height information of the user U's head from the second device 200 in advance, and perform processing to extract the image A2 corresponding to the pointing direction φ from the height shown in the height information.

[0147] In the second example, the image transmitting unit 173 uses the first communication device 110 to send the image A2, which was captured by the image editing unit 175, to the second device 200. Furthermore, the prompting unit 277 causes the moving image display device 250 to display the image A2 sent from the first device 100.

[0148] <Processing related to simulated rides>

[0149] Next, in the implementation method, a series of processes performed by the information processing system 1 will be specifically described in the case where user U simulates riding in a moving body M (passenger P allows user U to simulate riding in a moving body M). Figure 9 This is a sequence diagram representing an example of the processing performed by information processing system 1. Figure 9In the example, a first device 100 mounted on a mobile body M carried by passenger P, a second device 200 used by user U, and processing in a management server 300 are shown. User U and passenger P register in advance with the management server 300 and with user data 360 in order to receive services provided in the information processing system 1.

[0150] exist Figure 9 In the example, the second device 200 generates a request for simulating a ride (simulated ride request) (step S100) and sends the generated simulated ride request to the management server 300 (step S102). The simulated ride request may include, for example, information pre-specified about the passenger P, or information specifying the current location, destination, or surrounding conditions of the moving vehicle M. More specific examples of simulated ride requests include requests such as "I want to simulate a ride on a vehicle driven by Ms. / Mr. A" or "I want to simulate a ride on a vehicle traveling on a coastal road." These are examples of the matching requests described above.

[0151] The matching processing unit 320 of the management server 300 receives a simulation experience request from the second device 200 and performs matching processing based on user data 360 (step S104). Furthermore, if there is more than one successfully matched mobile body M, the providing unit 330 generates inquiry information for all mobile bodies M asking whether to allow the simulation ride (step S106) and sends the generated information to the first device 100 of the successfully matched mobile body M (step S108).

[0152] The first device 100 displays the query information sent from the management server 300 on the screen (step S110). Figure 10 This is an example of an image IM10 representing an inquiry. It should be noted that the display content, layout, and other display formats included in image IM10 are not limited to this. The same applies to the descriptions of other images thereafter. In image IM10, the inquiry screen includes, for example, a user information display area AR11, an area AR12 indicating whether an option is available, and a switch display area AR13. The user information display area AR11 displays information related to the user U who has made a simulated ride request. The information related to user U may be the user ID or other information that passenger P can use to identify user U.

[0153] The AR12 area displays an image of the content used by occupant P to select information from the information provided to user U. Figure 10In the example, regarding indoor images, outdoor images, communication (sound), and navigation information (information provided from the navigation device), a screen is displayed showing whether to allow the provision of information to passenger P. Communication (sound) is an example of "indoor sound information." For example, passenger P can also communicate and provide images to user U, but in cases where they do not want the other party to know the destination, they can allow communication, indoor images, and outdoor images, but refuse to provide navigation information. Thus, if user U is actually riding in a mobile vehicle M, they can simply know the destination based on the navigation information, but if the navigation information is not provided during the simulated ride, they can keep the destination confidential from user U and allow user U to have the unique and surprising experience of simulated rides. For example, for a less close user U, by refusing to provide communication (sound), they can drive without talking. It should be noted that the information that can be provided is not limited to... Figure 10 Examples may include, for instance, information regarding the selectability of navigation operations performed by user U, operations to other control object devices 190, and information regarding whether to provide an image including occupant P. It should be noted that the selectability of the information provided, as selected by the selectability area AR12, can also be changed during the simulated ride. The switch display area AR13 includes, for example, icons IC11 and IC12. Icon IC11 is a switch that accepts occupant P's permission for user U to ride the simulated ride. Conversely, icon IC12 is a switch that accepts occupant P's refusal to accept user U's simulated ride.

[0154] return Figure 9 The first device 100 accepts based on the above. Figure 10 The image IM10 shown is used to determine whether the user U can provide the setting (step S112), and the received setting information is sent to the management server 300 (step S114). It should be noted that, in order to ensure driving safety, the processing in steps S110 to S112 can also be performed when the moving body M is stopped, or the simulated ride request can be notified by sound instead of the screen display (or in addition to the screen display), and the setting can be accepted. The provisioning unit 330 of the management server 300 generates a request result (response result) based on the setting information obtained from the first device 100 (step S116), and sends information indicating the generated request result to the second device 200 (step S118).

[0155] The second device 200 receives the request result from the management server 300 and displays the received information representing the request result on the screen (step S120). Figure 11This diagram illustrates an example of an image IM20 representing the result of a request. The image IM20, as a screen displaying the request result, includes, for example, an occupant information display area AR21, an information overview display area AR22, and a switch display area AR23. This image IM20 can also be generated based on a licensed occupant P (moving body M) simulating a vehicle ride, and can further display an image representing an overview of the request result.

[0156] The passenger information display area AR21 displays information related to passenger P who has granted permission for a simulated ride request. This information may include the passenger ID or other information that allows user U to identify passenger P. The information granted by passenger P is displayed in a summary view in the information provision overview display area AR22. The fee (service usage fee) based on the granted information may also be displayed in the information provision overview display area AR22. The switch display area AR23 includes icons IC21-IC23, for example. Icon IC21 is a switch for accepting the information provided in the information provision overview display area AR22 to simulate a ride with the mobile body M of passenger P, which is displayed in the passenger information display area AR21. Conversely, icon IC22 is a switch for accepting no simulated ride. Icon IC23 is a switch for accepting the search for other passengers (mobile bodies). When searching for other passengers, users can select from other passengers included in the request results, or enter other request conditions to cause the management server 300 to perform matching processing again.

[0157] return Figure 9 The second device 200 accepts based on the above. Figure 11 The image IM20 shown receives a selection instruction from user U (step S122) and sends the received instruction information to management server 300 (step S124). Hereinafter, it will be explained that user U has given an instruction to simulate a ride on the mobile body M of passenger P. Management server 300 manages the communication status between first device 100 and second device based on the instruction from second device 200 (step S126), and sends information indicating the execution of the simulated ride to both first device 100 and second device 200 (steps S128, S130). It should be noted that during the processing of steps S126 to S130, if user U selects not to simulate a ride, the providing unit 330 of management server 300 sends information indicating this intention to first device 100 and notifies user U.

[0158] Next, the first device 100 and the second device 200 communicate with each other to perform a simulated ride (step S132). When the simulated ride ends, a message indicating the end of communication is sent to the management server 300 (steps S134, S136). It should be noted that during the processing in step S132, the second device 200 can also notify the passenger P from the user U before providing information from the first device 100 to the second device 200. Thus, the user U can grant communication at their own time to start a call, receive dynamic images, and other information. Next, the cost management unit 340 of the management server 300 performs cost calculation processing related to the service utilization of the first device 100 and the second device 200. Specifically, the cost management unit 340 differentiates the costs related to information provision based on at least one of the following: the type of passenger P providing information, the type of user U, the type of mobile vehicle M, the type of information that can be provided to the user U, and the amount of information. For example, if passenger P and user U are celebrities, the usage fee will be set higher than usual; similarly, if the vehicle M is a new or rare car, the usage fee will be set higher than usual. The fee management department 340 will perform the corresponding accounting processing. This concludes the processing for this sequence. It should be noted that regarding... Figure 9 The processing of step S112 shown (whether setting processing can be provided) can also be preset by the occupant P according to each registered user U.

[0159] <Processing related to icon display>

[0160] Next, in the specific implementation, user U1 and user U2 simulate riding in a vehicle M1. When user U1 inputs an operation when image A is displayed on the mobile image display device 250 of the second device 200-1, a series of processes are performed by the information processing system 1 when the mobile image display device 250 of the second device 200-2 displays an icon.

[0161] Figure 12A This is a sequence diagram representing the first example of the processing performed by the information processing system 1. The first device 100, the second device 200-1, the second device 200-2, and the management server 300 communicate to perform a simulated ride (step S200). Figure 12A In this process, the first device 100 has obtained user information for communicating with the second device 200.

[0162] The second devices 200-1 and 200-2 each present the specified information, such as image A2, to the mobile image display device 250 (step S202). The prompting units 277 of the second devices 200-1 and 200-2 cause the image A2 captured by the image editing unit 276 to be displayed (prompted) on the mobile image display device 250.

[0163] The second device 200-1 receives operation input from the user U-1 (step S204), generates an output signal based on the received operation input, and sends the output signal to the second device 200-2 (step S206). The operation input could be, for example, a click operation performed by the user U using the HMI 260 on at least a portion of the area of ​​the moving image displayed on the mobile image display device 250. For example, the user U clicks on a designated location, such as a place they wish to view from a remote location, within the moving image displayed on the mobile image display device 250. The HMI 260 receives the click operation input from the user U. The generation unit 281 generates an output signal containing location information of the clicked location based on the click operation received by the HMI 260, and sends the output signal to the second device 200-2.

[0164] The prompting unit 277 of the second device 200-2 displays an icon on the moving image display device 250 based on the output signal from the second device 200-1 (step S208). For example, the prompting unit 277 of the second device 200-2 obtains the output signal from the second control device 270 from the second device 200-1, and obtains the position information contained in the obtained output signal. The prompting unit 277 of the second device 200-2 displays an icon for image A displayed on the moving image display device 250 based on the obtained position information.

[0165] Figure 12B This is a sequence diagram representing a second example of the processing performed by the information processing system 1. The first device 100, the second device 200-1, the second device 200-2, and the management server 300 communicate to perform a simulated ride (step S300).

[0166] The second devices 200-1 and 200-2 each present the specified information, such as image A2, to the mobile image display device 250 (step S302). The prompting units 277 of the second devices 200-1 and 200-2 cause the image A2 captured by the image editing unit 276 to be displayed (prompted) on the mobile image display device 250.

[0167] The second device 200-1 receives operation input from user U-1 (step S304), obtains user information of user U-2 from the second device 200-2 (step S306), generates an output signal based on the received operation input and user information of user U-2, and sends the output signal to the second device 200-2 (step S308). The operation input may be, for example, a click operation performed by user U using HMI 260 on at least a portion of the area of ​​the moving image displayed on the mobile image display device 250. For example, user U clicks on a designated location, such as a place they wish to view from a remote location, within the moving image displayed on the mobile image display device 250. HMI 260 obtains the click operation input performed by user U. User information acquisition unit 279 obtains user information of user U2 using the second device 200-2. One example of user information for user U2 is information used for communication with the second device 200-2. The production unit 281 generates an output signal containing the location information of the click operation based on the click operation obtained by the HMI260 and the user information of the user U2, and sends the output signal to the second device 200-2.

[0168] Step S310 and Figure 12A Step S208 is the same, so the explanation is omitted.

[0169] Figure 12C This is a sequence diagram representing the third example of the processing performed by the information processing system 1. The first device 100, the second device 200-1, the second device 200-2, and the management server 300 communicate to perform a simulated ride (step S400).

[0170] The second devices 200-1 and 200-2 each present the specified information, such as image A2, to the mobile image display device 250 (step S402). The prompting units 277 of the second devices 200-1 and 200-2 cause the image A2 captured by the image editing unit 276 to be displayed (prompted) on the mobile image display device 250.

[0171] The second device 200-1 receives operation input from the user U-1 (step S404), detects the user U-1's gaze direction (step S406), generates an output signal based on the received operation input and the user U-1's gaze direction, and sends the output signal to the second device 200-2 (step S408). The operation input can be, for example, a button press or gesture input performed by the user U using the HMI 260 on at least a portion of the moving image displayed on the mobile image display device 250. For example, the user U presses a button while looking at a predetermined location, such as a place they want to view from a remote location, included in the moving image displayed on the mobile image display device 250. The gaze detection unit 280 detects the user U-1's gaze direction and obtains predetermined location information. The HMI 260 obtains the button press or gesture input performed by the user U. The production unit 281 generates an output signal containing position information based on the specified position information obtained by the line of sight detection unit 280 and the operation and posture input of the pressed button obtained by the HMI 260, and sends the output signal to the second device 200-2.

[0172] Step S410 and Figure 12A Step S208 is the same, so the explanation is omitted.

[0173] exist Figures 12A to 12C The example described is of an information processing system 1 having two second devices 200, but the information processing system 1 can also be applied to a case with three or more second devices 200.

[0174] <Information provided during the simulated ride>

[0175] Next, we will explain the information provided to user U during the simulated ride. The following example is... Figure 12A The processing in step S208 shown, Figure 12B The processing of step S310 shown and Figure 12C The information provided in step S410 shown. For example, the output format of the information provided to the second device 200 varies depending on the action performed by the user U, the type of information that can be provided to the user U, or the amount of information. It should be noted that, in the following example, it is assumed that the occupant P has authorized the provision of indoor images, outdoor images, and audio communication.

[0176] <Example of information provided: Indoor image>

[0177] Figure 13A This is a diagram used to illustrate examples of provided information. In Figure 13AIn the example shown, a user U is depicted sitting in a chair CH located outside the mobile body M (e.g., in a room of his own home), and a second device 200 utilized by the user U (in... Figure 13A The example provided is an example of image IM30 for a VR glasses all-in-one device. Here, as an example, the case where the second device 200 is a VR glasses all-in-one device is illustrated, but it could also be a head-mounted display, etc. Image IM30 contains a heart-shaped icon IC25. It should be noted that... Figure 13A The example shown depicts the user U's state (sitting in chair CH), representing the initial state immediately after the simulated ride. The first device 100 and the second device 200 identify the user U's current position (reference posture) as the reference position. The reference position is assumed to be the user U sitting in the front passenger seat S2. It should be noted that the reference position can also be corrected by the first control device 170 using physical information such as the user U's height or sitting height registered in the user information of user data 360, which drives the seat drive to adjust the height of the indoor camera 132 to suit the user U's sitting height (so-called zero-point correction). The reference position can be corrected using correction instructions from the user U viewing image IM30. The reference position may include orientation information when the user U is sitting in chair CH facing forward.

[0178] For example, while seated in chair CH, an image captured by the interior camera 132 of the first device 100 in any direction from the user U's (second device 200) reference position is provided as an interior image IM30. Thus, the user U can obtain an interior image from any direction as seen from the passenger seat S2. Sound information, including the voice of the occupant P, and sounds inside and outside the vehicle M, acquired by the first microphone 120, is provided to the user U. By using the image IM30 and the sound information, the user U can converse with the occupant P, enjoying a conversation as if actually riding in the vehicle M.

[0179] It should be noted that while information processing system 1 describes a scenario where user U can visually identify any direction from the co-driver's seat S2, restrictions can also be placed on the directions that user U can visually identify, depending on factors such as the agreement made during matching or whether the provision of passenger P is possible. For example, passenger P may sometimes urgently desire to see the scenery in the direction of travel of the vehicle M or the scenery on the opposite side from the driver's seat S1, but without wanting to display their own figure. This is because it is assumed that passenger P and user U are not family or friends, but rather that passenger P is responding to a need to confirm the driving feel of the vehicle M or to visually identify the desired street scenery. In this case, the matching processing unit 320 or the provisioning unit 330 of the management server 300 sets such restrictions based on the matching results. Furthermore, the first control device 170 or the second control device 270 may, depending on the settings, block the angle range that prevents visual identification or correct the direction by preventing the direction φ from being directed. Information related to such restrictions concerns the privacy of passenger P, and therefore can also be set on the first device 100 side. Furthermore, regarding information that has been refused, it can be provided by user U after paying an additional fee and passenger P's consent.

[0180] The moving body image display device 250 can also replace the portion of a specified object in the interior of the moving body M in the image captured by the camera unit 130 with an image (CG image) drawn by computer processing and display it. Figure 13A The area OB1 shown is a display device for navigation display, etc., and is an example of a "prescribed item". When directly displaying an image captured from the display screen of the display device, the image is sometimes blurry and visual recognition is reduced due to light reflection. Therefore, the mobile body image display device 250 can also obtain data from the first device 100 for constructing the display screen of the display device involved, or image data drawn by computer processing in the mobile body M, and display the image drawn by computer processing based on the data or the obtained image data embedded in the image captured by the camera unit 130 (an edited image). In this case, the positions of items in the interior of the mobile body M, which is the prescribed item, are shared in advance between the first device 100 and the second device 200. The prompting unit 277 determines whether the image displayed by the mobile body image display device 250 includes the prescribed item based on, for example, the pointing direction φ. If it is determined that the prescribed item is included, the image is replaced as described above. The "prescribed item" can also be the head or face of the occupant P. In this case, the CG image such as the head can also be changed according to the display of the occupant P. It should be noted that at least a portion of the aforementioned image data (CG images), etc., can be provided from the information database provided by the management server 300.

[0181] Figure 13B This diagram illustrates an example of providing information. In the second communication device 210, the prompting unit 277 changes the prompting position of the icon based on the speed of the moving body M. For example, the prompting unit 277 identifies the position of the output icon and displays it in a manner that it flows backward according to the speed of the moving body M. For example, the prompting unit 277 obtains information indicating the moving speed of the moving body M from the first device 100 and changes the position from icon IC25-1 to icon IC25-2 based on the obtained speed of the moving body M. By changing the position from icon IC25-1 to icon IC25-2, a dynamic visual effect is provided, enabling a more realistic and attractive experience for the user U. By adjusting the movement of icon IC25 according to the vehicle's speed, visual consistency can be maintained for the user U.

[0182] Figure 13C This is a diagram used to illustrate an example of providing information. In the second device 200, if the mobile body behavior detection unit 282 detects that the mobile body M has turned left or right after the user U performs a click operation, the prompting unit 277 displays an icon in a manner different from the case where the mobile body M has not turned left or right. For example, the prompting unit 277 displays the icon with a dashed line. For example, the prompting unit 277 determines whether at least a portion of the dynamic image that the user U clicked is still being displayed based on the dynamic image displayed on the mobile body image display device 250, and displays the icon with a dashed line if it determines that it will no longer be displayed.

[0183] Here, the prompting unit 277 may also determine whether the defined display area (boundary box) is continuously displayed on the moving body image display device 250 based on the viewing direction detected by the viewing detection unit 280, and display an icon with a dashed line if it is determined that the display is no longer displayed.

[0184] Furthermore, the prompt section 277 can also display icons using dashed lines, and simultaneously display information indicating the objects that have been displayed. Figure 13C In the middle, the icon IC26 is displayed with a dashed line, and "Good trees seen" is displayed as information indicating objects that have been seen (trees, buildings, telephone poles, the sea, etc.).

[0185] In the aforementioned embodiments, the second device 200 may also be configured to accept a cancellation operation after receiving an operation input from the user U. For example, the user U may use the HMI 260 to perform a predetermined cancellation operation input while an icon is displayed in at least a portion of the dynamic image displayed on the mobile image display device 250. For example, the user U may aim at the icon displayed on the mobile image display device 250 through the HMI 260 and perform the cancellation operation. The user U may also perform the cancellation operation while looking at the icon displayed on the mobile image display device 250. The gaze detection unit 280 detects the gaze direction of the user U1 and obtains the position information of the icon. The HMI 260 obtains the cancellation operation performed by the user U. The generation unit 281 generates a cancellation signal containing position information based on the position information of the icon obtained by the gaze detection unit 280 and the cancellation operation input obtained by the HMI 260, and sends it to other second devices.

[0186] The prompting unit 277 of the other device deletes the icon based on the cancellation signal from the second device 200. For example, the prompting unit 277 of the other second device obtains the cancellation signal from the second control device 270 from the second device 200 and obtains the position information contained in the obtained output signal. Based on the obtained position information, the prompting unit 277 of the other second device deletes the icon displayed in image A of the moving body image display device 250.

[0187] In the aforementioned embodiments, the plurality of second devices 200 may also include a device comprising a display unit capable of displaying images. In this case, the second device 200 may be a screen (such as a television) placed in one's own home, in addition to VR glasses. The second device 200 such as a screen may or may not have a receiving unit for receiving operation input. It is also possible to provide a prompt regarding the result of operation input received by the receiving unit of another second device 200, even if it does not have a receiving unit.

[0188] <Variation Example>

[0189] In the aforementioned information processing system 1, when the second device 200 receives an operation input from the user U regarding image A2, it detects any position of image A based on the operation input and obtains the POI information of the detected position. The second device 200 displays the obtained POI information and icon on the mobile image display device 250. The second device 200 generates an output signal including the arbitrary position information and POI information of image A detected based on the operation input and sends it to other second devices 200. The second device 200 displays the icon and POI information on the mobile image display device 250 based on the position information and POI information contained in the output signals from the other second devices 200.

[0190] Figure 14 This is a third example of a diagram illustrating the functional structure of the first control device 170 and the second control device 270. (Compared to...) Figure 7 The difference between the second control device 270 and the first example is that the second control device 270 includes a POI acquisition unit 284, a selection unit 285, and a voice information acquisition unit 286. Regarding the other components, they basically have the same functions as the first example, so further explanation is omitted.

[0191] When the POI acquisition unit 284 receives an operation input from the user in response to the image A2 prompted to the mobile image display device 250 by the prompting unit 277, it determines the position on the image A2 based on the operation input and acquires POI information based on information indicating the determined position. Alternatively, it can acquire information indicating the user's gaze position from the gaze detection unit 280 and acquire POI information based on that information. For example, the POI acquisition unit 284 can also acquire POI information from the management server 300. When the prompting unit 277 acquires POI information from the POI acquisition unit 284 and displays the image A2 on the mobile image display device 250, it also displays the POI information in addition to the icon.

[0192] When the selection unit 285 has acquired multiple POI information from the POI acquisition unit 284, it selects the POI information that is displayed along with its icon. For example, the selection unit 285 may also select the POI information that is closest to the moving body M from among the multiple POI information. In this case, when the prompting unit 277 acquires the POI information from the selection unit 285 and displays the image A2 on the moving body image display device 250, it also displays the POI information in addition to the icon. Here, the selection unit 285 may also select any POI information from among the multiple POI information.

[0193] The sound information acquisition unit 286 acquires sound information based on the sound collected by the second microphone 220.

[0194] The prompting unit 277 determines the position on the image (video) displayed on the moving body image display device 250 indicated by the operation input, and outputs sound based on the sound information acquired by the sound information acquisition unit 286 when an icon is displayed at the determined position. The production unit 281 produces an output signal including the position information determined by the prompting unit 277 and the sound information acquired by the sound information acquisition unit 286, and sends it from the second communication device 210 to other second devices 200.

[0195] Here, when displaying image A2 on the mobile image display device 250, the prompting unit 277 may also display POI information in addition to the icon, and associate it with the POI information to emit a sound based on the sound information acquired by the sound information acquisition unit 286. The production unit 281 may also produce an output signal that includes the location information determined by the prompting unit 277, the POI information acquired by the POI acquisition unit 284, and the sound information acquired by the sound information acquisition unit 286, and transmit it from the second communication device 210 to other second devices 200.

[0196] Figure 15A This is a sequence diagram representing the fourth example of the processing performed by the information processing system 1. The first device 100, the second device 200-1, the second device 200-2, and the management server 300 communicate to perform a simulated ride (step S500).

[0197] The second devices 200-1 and 200-2 each present the specified information, such as image A2, to the mobile image display device 250 (step S502). The prompting units 277 of the second devices 200-1 and 200-2 display (prompt) the image A2 captured by the image editing unit 276 on the mobile image display device 250.

[0198] The second device 200-1 receives operation input from the user U-1 (step S504), detects the user U-1's gaze direction (step S506), and obtains POI information based on the user U-1's gaze position (step S508). The operation input may be, for example, a button press or gesture input performed by the user U using the HMI 260 on at least a portion of the dynamic image displayed on the mobile image display device 250. For example, the user U presses a button while gazing at a predetermined location, such as a place they wish to view from a remote location, included in the dynamic image displayed on the mobile image display device 250. The gaze detection unit 280 detects the user U-1's gaze direction and obtains predetermined position information. The POI acquisition unit 284 obtains information indicating the user U-1's gaze position from the gaze detection unit 280 and obtains POI information based on the obtained information indicating the user U-1's gaze position.

[0199] The second device 200-1 determines whether there are multiple POI information pieces (step S510). If multiple POI information pieces are obtained, the second device 200-1 selects the POI information that is displayed along with an icon from the multiple POI information pieces (step S512), and generates an output signal including information indicating the gaze position and the POI information, and sends it to the second device 200-2 (step S514). The selection unit 285 selects the POI information that is displayed along with an icon when multiple POI information pieces are obtained by the proposed POI acquisition unit 284. For example, the selection unit 285 selects the POI information that is closest to the moving body M from the multiple POI information pieces. The generation unit 281 generates an output signal including POI information and position information based on the predetermined position information obtained by the gaze detection unit 280 and the operation and posture input of the pressed button obtained by the HMI 260, and sends it to the second device 200-2.

[0200] The prompting unit 277 of the second device 200-2 displays an icon and POI information on the mobile image display device 250 based on the output signal from the second device 200-1 (step S516). For example, the prompting unit 277 of the second device 200-2 obtains the output signal from the second communication device 210 from the second device 200-1, and obtains the location information and POI information contained in the obtained output result. Based on the obtained location information, the prompting unit 277 of the second device 200-2 displays the icon and POI information for image A displayed on the mobile image display device 250.

[0201] Figure 15B This is a sequence diagram representing the fifth example of the processing performed by information processing system 1. Steps S600 to S606 and S610 to S614 are applicable. Figure 15ASteps S500 to S512 are therefore omitted from the description.

[0202] In step S608, the sound information acquisition unit 286 of the second device 200-1 acquires sound information based on the sound emitted by the user U.

[0203] In step S616, an output signal is generated that includes information indicating the gaze position, POI information, and sound information obtained when the POI information is obtained, and is sent to the second device 200-2.

[0204] In step S618, the prompting unit 277 of the second device 200-2 displays an icon and POI information on the mobile image display device 250 based on the output signal from the second device 200-1, and outputs sound from the second speaker 240 based on the sound information. For example, the prompting unit 277 of the second device 200-2 obtains the output signal from the second communication device 210 from the second device 200-1, and obtains the location information, POI information, and sound information contained in the obtained output result. Based on the obtained location information, the prompting unit 277 of the second device 200-2 displays the icon and POI information to the image A displayed on the mobile image display device 250, and outputs sound from the second speaker 240 based on the sound information.

[0205] Figure 16 This is a diagram used to illustrate an example of providing information. When a moving image is displayed on the mobile image display device 250, the user U aims at any position by operating the HMI 260 and performs button presses or gesture input. The user U can also perform button presses or gesture input while looking at a predetermined position contained in the moving image displayed on the mobile image display device 250. The gaze detection unit 280 detects the user U's gaze direction and obtains predetermined position information. The gaze detection unit 280 can also transform the predetermined position information into three-dimensional position information based on the obtained predetermined position information, map data, and the position information of the mobile body M. In this case, the production unit 281 generates an output signal that includes three-dimensional position information instead of the predetermined position information. Figure 16 In the example shown, the icon IC27 is displayed on Tokyo Tower TT, which user U has viewed, and "Thumbs up for Tokyo Tower" is displayed as POI information and a comment on the POI information.

[0206] <Summary>

[0207] The information processing system 1 configured as described above includes: a first device 100 mounted on a mobile body M; a plurality of second devices 200 used by a plurality of users U in locations different from the mobile body M; an information acquisition unit 315 that acquires predetermined information sent by the first device 100; a providing unit 330 that provides the predetermined information acquired by the information acquisition unit 315 to the plurality of second devices 200; and a plurality of prompting units 277 that prompt the predetermined information provided to the plurality of second devices 200 by the providing unit 330. When one of the plurality of second devices 200 performs an operation input based on the first user U1, the prompting units 277 of the second devices 200 other than the first user U1 prompt the output result based on the operation input performed by the first user U1.

[0208] Among users of the second device 200, it is possible to smoothly communicate what other users are interested in and what they are thinking, thus improving convenience. An example of the second device 200 is a head-mounted display. Examples of the specified information could also be animation data, image data, vehicle location data, map data of the surrounding area centered on the driving location, polygonal data of the surrounding area created using a 3D model, images from an in-vehicle camera, or images from an installed smartphone.

[0209] The information processing system 1 includes a user information acquisition unit 279 that acquires user information of the user of the second device 200. The second device 200 used by the first user also includes a production unit 281 that produces an output signal for indicating the output result based on the user information acquired by the user information acquisition unit 279.

[0210] The second device can obtain user information based on information from other second devices received from the first device, and therefore can communicate based on the obtained user information. Since the second device 200 can transmit signals without communicating with the first device (vehicle-mounted device), it can suppress unnecessary increases in processing load. Alternatively, the second device may not obtain information from other second devices, but instead provide information via the first device 100.

[0211] In information processing system 1, the second device includes a gaze detection unit 280, which detects the user's gaze direction. The prompting unit 277 prompts and outputs results based on the user's gaze direction detected by the gaze detection unit 280.

[0212] By determining the location of the prompt output based on the user's line of sight, the ease of use and intuitiveness regarding the desired location can be improved.

[0213] The gaze detection unit 280 utilizes a 6DoF (6 Degrees of Freedom) sensor, such as that mounted on the second device 200, to accurately track the movement and position of the user U's head. Alternatively, a camera with gaze detection capabilities can be used in place of or in conjunction with a 6DoF sensor. The prompting unit 277 can also determine the position based on the user's gaze direction detected by the gaze detection unit 280 and output a prompt at the determined position. The prompting unit 277 can also determine the display area towards which the gaze is directed based on the user's gaze direction detected by the gaze detection unit 280 and output a prompt within the determined display area.

[0214] In the information processing system 1, the second device includes a POI acquisition unit 284 for acquiring POI information. When the prompting unit 277 displays the output result, the POI acquisition unit 284 acquires the POI information based on the location information of the first device 100. The prompting unit 277 displays the POI information acquired by the POI acquisition unit 284 along with the output result.

[0215] By combining visual feedback with Points of Interest (POI) information, more appropriate information can be provided to users, thus improving the user experience. POI information includes places that users are visually interested in and information associated with objects. POI information typically includes popular facilities, tourist attractions, and distinctive areas.

[0216] In information processing system 1, the second device includes a POI acquisition unit 284 for acquiring POI information. The POI acquisition unit 284 acquires POI information based on the user's gaze direction detected by the gaze detection unit 280. The prompting unit 277 determines the location of the prompt output result based on the POI information acquired by the POI acquisition unit 284.

[0217] It can determine POI information based on the user's line of sight and output the results in the appropriate location, thus improving accuracy and relevance.

[0218] In the information processing system 1, the second device further includes: a POI acquisition unit 284 for acquiring POI information; and a selection unit 285, which selects the POI information closest to the moving body from the multiple POI information when the POI acquisition unit 284 has acquired multiple POI information. A prompting unit 277 prompts the multiple second devices with the POI information selected by the selection unit 285.

[0219] When multiple POIs exist, the system can provide the user with the most relevant information, thus preventing information overload and improving the user experience. The selection unit 285 can also provide an interface for users to select POIs of interest by displaying a POI selection screen.

[0220] In the information processing system 1, the prompting unit 277 changes the prompting position of the output result based on the speed of the moving body.

[0221] By changing the position of the output prompt based on the speed of the moving object, a dynamic visual effect can be provided, thus offering users a more realistic and engaging experience.

[0222] The information processing system 1 also includes a motion behavior detection unit 282 that detects whether the moving body M has turned right or left. When the motion behavior detection unit 282 detects that the moving body M has turned right or left after the prompt outputs a result, the prompt content of the output result is changed by the prompt unit 277.

[0223] By changing the prompt content of the output result when the moving body M turns right or left, it is possible to convey to the user that there was a place of interest before making the right or left turn, thus improving convenience.

[0224] The information processing system 1 also includes a production unit 281, which, when a first user performs an operation input by one of a plurality of second devices, produces a notification message containing the output result to the first device based on the operation input performed by one of the second devices.

[0225] The first device can obtain notification information from the second device, and therefore can notify the first user of their feelings based on the obtained notification information.

[0226] The information processing system 1 also includes a voice information acquisition unit 286 that acquires voice information based on the user's voice. A prompting unit 277 prompts for output results based on the operation input and the voice information.

[0227] It can combine visual feedback with sound, thus providing users with more appropriate information and improving the user experience.

[0228] The information processing system 1 also includes a voice information acquisition unit 286 that acquires voice information based on the user's voice. The prompting unit 277 prompts the POI information along with the output result, and prompts the voice information acquired by the voice information acquisition unit 286 in association with the POI information.

[0229] It can combine visual feedback with POI information and sound, thus providing users with more appropriate information and improving the user experience.

[0230] <Example of use>

[0231] Information processing system 1 can be used in the following form.

[0232] (A) The occupant P and the user U are family members, friends, etc., and the occupant P provides the user U with a virtual driving mode. The user U can talk to the occupant P while watching images and talking about the scenery around the mobile body M.

[0233] (B) Passenger P is a general user, and user U is a provider of road guidance services, driving instruction services, etc. User U can provide road guidance and driving operation instructions in places that are not easily known in navigation devices or are not displayed on maps while viewing the scenery around the mobile body M.

[0234] (C) Passenger P is a celebrity, and user U is a general user. A business-based virtual driving experience is provided to user U. In this case, a correspondence is established simultaneously between one passenger P and multiple users U. For example, the transmission of voice from user U can be turned off. Users U can also pay an additional fee to have a scheduled conversation with passenger P.

[0235] The above describes specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way. Various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. An information processing system, wherein the information processing system comprises: a first device mounted on a mobile body; a plurality of second devices used by a plurality of users at different places from the mobile body; an information acquisition section that acquires prescribed information transmitted by the first device; a provision section that provides the prescribed information acquired by the information acquisition section to the plurality of second devices; and a plurality of prompting sections that prompt the prescribed information provided to the plurality of second devices by the provision section, in a case where an operation input by a first user is performed by one of the second devices, the prompting section of a second device other than the one prompts an output result based on the operation input by the one.

2. The information processing system according to claim 1, wherein the information processing system comprises a user information acquisition section that acquires user information of a user of the second device, the second device used by the first user further comprises a production section that produces an output signal for prompting the output result based on the user information acquired by the user information acquisition section.

3. The information processing system according to claim 1, wherein the second device comprises a line-of-sight sensing section that senses a line-of-sight direction of a user, the prompting section prompts an output result based on the line-of-sight direction of the user sensed by the line-of-sight sensing section.

4. The information processing system according to claim 1, wherein the second device comprises a POI acquisition section that acquires POI information, the POI acquisition section acquires POI information based on position information of the first device when the prompting section prompts an output result, the prompting section prompts the POI information acquired by the POI acquisition section together with the output result.

5. The information processing system according to claim 3, wherein the second device comprises a POI acquisition section that acquires POI information, the POI acquisition section acquires POI information based on a line-of-sight direction of a user sensed by the line-of-sight sensing section, the prompting section finds a position for prompting an output result based on the POI information acquired by the POI acquisition section.

6. The information processing system according to claim 1, wherein the second device further comprises: a POI acquisition section that acquires POI information; and a selection section that selects, in a case where a plurality of POI information is acquired by the POI acquisition section, POI information closest to the mobile body from the plurality of POI information, the prompting section prompts the POI information selected by the selection section to a second device other than the one.

7. The information processing system according to claim 1, wherein the prompting section changes a prompting position of an output result based on a speed of the mobile body.

8. The information processing system according to claim 1, wherein the information processing system further comprises a mobile body behavior sensing section that senses a case where the mobile body makes a right turn or a left turn, the prompting section prompts an output result based on the case sensed by the mobile body behavior sensing section. The prompting section changes the content of the output result when the mobile body is detected by the mobile body behavior detection section to have made a right turn or a left turn after the output result is prompted.

9. The information processing system according to claim 1, wherein The information processing system further includes a production section that produces notification information including an output result for the first device based on an operation input by one of the second devices when the operation input by the one second device is made.

10. The information processing system according to claim 1, wherein The information processing system further includes a sound information acquisition section that acquires sound information based on the user's voice, The prompting section prompts an output result based on the operation input and the sound information.

11. The information processing system according to claim 4, wherein The information processing system further includes a sound information acquisition section that acquires sound information based on the user's voice, The prompting section prompts the POI information and the sound information associated with the POI information acquired by the sound information acquisition section together with the output result.

Images