Information Processing Systems

Abstract

To provide an information processing system capable of enhancing a sense of presence given to both an occupant of a mobile object and a user who is in a different location from the mobile object.SOLUTION: An information processing system includes: a first device that is mounted on a mobile object boarded by an occupant; and a second device that is used by a user at a location different from the mobile object. The first device includes: a first speaker configured to output a voice uttered by the user; and a camera unit that is provided on a predetermined seat of the mobile object and has one or more cameras including at least an indoor camera capable of capturing an image of an interior of the mobile object viewed from the predetermined seat. The second device includes: a detection device for detecting an orientation direction, which is a face orientation or gaze direction of the user; and a display device configured to display an image corresponding to the orientation direction viewed from the predetermined seat among images captured by the camera unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system. [Background technology]

[0002] Research has been conducted on communication between a device mounted on a moving object such as a vehicle and a device used in a location other than the moving object, thereby sharing images of the scenery outside the vehicle (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-94958 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, there are cases where both the passengers of the vehicle and users in locations other than the vehicle are unable to feel a sufficient sense of realism.

[0005] The present invention has been made in consideration of these circumstances, and one of its objectives is to provide an information processing system that can enhance the sense of realism given to both the occupants of a moving body and users who are in a location other than the moving body. [Means for solving the problem]

[0006] The information processing system according to the present invention employs the following configuration. (1): An information processing system according to one embodiment of the present invention comprises a first device mounted on a vehicle in which a passenger rides, and a second device used by a user at a location different from the vehicle, wherein the first device comprises a first communication device that communicates with a second communication device of the second device, a first speaker that outputs voice uttered by the user acquired via the first communication device, and a camera unit that is installed on a predetermined seat of the vehicle and has one or more cameras including at least an indoor camera that can capture images of the interior of the vehicle as seen from the predetermined seat, wherein the second device comprises the second communication device that communicates with the first communication device, a second microphone that collects voice uttered by the user, a detection device that detects the direction of orientation of the user, and a display device that displays images captured by the camera unit that correspond to the direction of orientation as seen from the predetermined seat, and the second communication device transmits the voice collected by the second microphone to the first communication device.

[0007] (2): In the above aspect (1), the first speaker outputs the voice uttered by the user by localizing a sound image so that the voice can be heard by the occupant from the specified seat.

[0008] (3): In the above aspect (2), the first speaker includes a plurality of sub-first speakers arranged at different positions from each other, and the first device further includes a first control device that adjusts the volume and / or phase difference of the plurality of sub-first speakers to localize a sound image so that the sound can be heard by the occupant from the specified seat.

[0009] (4): In the above aspect (3), the second device further acquires height information indicating the height of the user's head, and the first control device localizes a sound image so that the sound is heard by the occupant from a position at a height corresponding to the height of the user's head on the specified seat, and outputs the sound emitted by the user to the first speaker.

[0010] (5): In the above aspect (1), the second device further acquires height information indicating the height of the user's head, and the display device displays an image corresponding to the direction of orientation as seen from the height indicated by the height information above the specified seat.

[0011] (6): In the above aspect (1), the second communication device transmits information on the orientation direction to the first communication device, and the first device further has a first control device that controls the first communication device to selectively transmit to the second communication device, from among the images captured by the camera unit, an image corresponding to the orientation direction obtained via the first communication device, and the display device of the second device displays an image corresponding to the orientation direction as seen from the specified seat, obtained via the second communication device.

[0012] (7): In the above aspect (1), the first communication device transmits images captured by the camera unit to the second communication device, and the second device further has a second control device that selectively displays on the display device, from among the images captured by the camera unit, an image that corresponds to the orientation direction.

[0013] (8): In the above aspect (1), the first device further has a first microphone that collects at least the voice uttered by the occupant, the second device further has a second speaker that outputs the voice uttered by the occupant acquired via the second communication device, and the first communication device transmits the voice collected by the first microphone to the second communication device.

[0014] (9): In the above aspect (8), the second speaker outputs the sound emitted by the occupant by localizing the sound image so that the sound can be heard by the user from the position of the occupant as seen from the specified seat.

[0015] (10): In the above aspect (1), the display device is a display device of VR (Virtual Reality) goggles, and the detection device includes a physical sensor attached to the VR goggles.

[0016] (11) In the above aspect (1), a mode in which the display device can display an image in a limited angle range can be executed.

[0017] (12) In the aspect (1) above, the moving body is a vehicle, and the predetermined seat is a passenger seat.

[0018] (13): In the above aspect (1), the display device replaces the portion of the image captured by the camera unit that shows a specified item inside the room of the moving body with an image drawn by computer processing and displays it. [Effects of the Invention]

[0019] According to aspects (1) to (13), it is possible to enhance the sense of realism given to both the passengers of the moving body and the users who are in a location other than the moving body. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a diagram showing the usage environment of an information processing system 1 and a management server 300. FIG. [Figure 2] FIG. 10 is a diagram showing an example of the contents of user data 360. [Figure 3] FIG. 2 is a configuration diagram of a first device 100. [Figure 4] 1 is a diagram showing an example of the arrangement of a part of a first device 100 in a moving body M. FIG. [Figure 5] FIG. 2 is a diagram illustrating the configuration of a second device 200. [Figure 6] FIG. 10 is a diagram for explaining an image corresponding to a pointing direction. [Figure 7] 2 is a diagram illustrating a first example of the functional configuration of a first control device 170 and a second control device 270. FIG. [Figure 8] FIG. 10 is a diagram illustrating a second example of the functional configuration of the first control device 170 and the second control device 270. [Figure 9] FIG. 10 is a diagram illustrating an example of displaying a replacement image. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an embodiment of an information processing system of the present invention will be described with reference to the drawings. The information processing system includes a first device mounted on a mobile body on which a passenger rides, and a second device used by a user at a location different from the mobile body. The mobile body is, for example, a vehicle, but it can be any mobile body that can accommodate a passenger. Furthermore, the passenger is usually the driver of the mobile body, but it can also be a passenger other than the driver.

[0022] Between the first device and the second device, audio collected by a microphone is transmitted to the other party and played back by a speaker, creating a state similar to a telephone conversation. Furthermore, a portion of an image captured by the camera unit of the first device is displayed on the second device, providing MR (Mixed Reality) on the second device side. The first device and the second device do not need to be in a one-to-one relationship; one of multiple first devices may be matched with multiple second devices in a one-to-many relationship, thereby operating as an information processing system. In the latter case, for example, one occupant can communicate with multiple users simultaneously or sequentially.

[0023] <Basic configuration> FIG. 1 is a diagram showing the usage environment of an information processing system 1 and a management server 300. The information processing system 1 includes a first device (mobile device) 100 mounted on a mobile body M, and a second device (user device) 200 used by a user U at a location different from the mobile body M (although it is not excluded that the second device 200 may happen to be close by). The first device 100, the second device 200, and the management server 300 communicate with each other via a network NW. The information processing system 1 may or may not include the management server 300.

[0024] The management server 300 includes, for example, a communication device 310, a matching processing unit 320, and a storage unit 350. The storage unit 350 stores user data 360.

[0025] Communication device 310 is a communication interface for connecting to network NW. Note that communication between communication device 310 and first device 100, and communication between communication device 310 and second device 200 are performed in accordance with, for example, TCP / IP (Transmission Control Protocol / Internet Protocol).

[0026] The matching processing unit 320 is realized by a processor such as a CPU (Central Processing Unit) executing a program (a group of instructions) stored in a storage medium. The storage unit 350 is a RAM (Random Access Memory), an HDD (Hard Disk Drive), a flash memory, or the like.

[0027] 2 is a diagram showing an example of the contents of the user data 360. The user data 360 includes, for example, an occupant list 360A in which an occupant ID, which is identification information of an occupant P, its communication identification information (such as an IP address), and a user ID, which is identification information of a user U to be matched, are associated with each other, and a user list 360B in which a user ID, its communication identification information (such as an IP address), and an occupant P to be matched are associated with each other. The user data 360 may be generated in any manner, not limited to the manner shown in FIG. 2, as long as it includes this information.

[0028] When the communication device 310 receives a matching request from the user U via the second device 200 or from the occupant P via the first device 100, the matching processing unit 320 matches the matching user U with the occupant P by referring to the user data 360, and transmits the communication identification information of the first device 100 of the occupant P to the second device 200 of the matched user U, and transmits the communication identification information of the second device 200 of the user U to the first device 100 of the matched occupant P, using the communication device 310. Between the first device 100 and the second device 200 that receive these, communication with higher real-time performance is performed, for example, in accordance with UDP (User Datagram Protocol).

[0029] 3 is a configuration diagram of first device 100. First device 100 includes, for example, first communication device 110, first microphone 120, camera unit 130, first speaker 140, user display device 150, HMI (Human Machine Interface) 160, and first control device 170. First control device 170 is connected to control target device 190 mounted on moving object M.

[0030] The first communication device 110 is a communication interface for communicating with the communication device 310 of the management server 300 and the second communication device 210 of the second device 200 via the network NW.

[0031] The first microphone 120 collects at least the voice uttered by the occupant P. The first microphone 120 may be provided inside the interior of the moving body M and may have a sensitivity that allows it to collect voices outside the moving body M, or may include a microphone provided inside the interior of the moving body M and a microphone provided outside the moving body M. The voice collected by the first microphone 120 is transmitted to the second communication device 210 by the first communication device 110 via the first control device 170, for example.

[0032] The camera unit 130 includes at least an indoor camera 132, and may also include an outdoor camera 134. The first speaker 140 outputs the voice uttered by the user U, which is acquired via the first communication device 110. Details of the arrangement of the camera unit 130 and the first speaker 140 will be described later with reference to FIG. 4.

[0033] The user display device 150 virtually displays the user U as if the user U were present inside the room of the moving object M. For example, the user display device 150 makes a hologram appear, or displays the user U on a part of the moving object M that corresponds to a mirror or window.

[0034] The HMI 160 is a touch panel, a voice response device (agent device), etc. The HMI 160 receives various instructions from the occupant P to the first device 100.

[0035] The first control device 170 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (a set of instructions), and the processor controls each part of the first device 100 by executing the set of instructions.

[0036] The control target device 190 is, for example, a navigation device or a driving assistance device mounted on the moving body M.

[0037] FIG. 4 is a diagram showing an example of the arrangement of a portion of the first device 100 in a moving object M. The indoor camera 132 is attached, for example, via an attachment 132A to the neck pillow of the passenger seat S2 (an example of a "predetermined seat"), and is provided at a position slightly separated from the backrest of the passenger seat S2 in the traveling direction of the moving object M. The indoor camera 132 has a wide-angle lens and is capable of capturing an image of the range indicated by the hatched area 132B in the figure. The indoor camera 132 is capable of capturing not only the interior of the moving object M but also the exterior through the window. In the following description, the passenger seat S2 is considered to be the predetermined seat, but the predetermined seat may be another seat such as a rear seat.

[0038] The outdoor camera 134 includes, for example, a plurality of subsidiary outdoor cameras 134-1 to 134-4. By combining images captured by the plurality of subsidiary outdoor cameras 134-1 to 134-4, an image such as a panoramic image capturing the outside of the moving object M can be obtained. Instead of (or in addition to) these, the outdoor camera 134 may include a wide-angle camera provided on the roof of the moving object M. Note that a camera capable of capturing an image of the area behind the passenger seat S2 may be added as the indoor camera 132, and the moving object image described below may be generated as a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 by the first control device 170, or may be generated as a 360-degree panoramic image by appropriately combining images captured by the indoor camera 132 and the outdoor camera 134.

[0039] The first speaker 140 outputs the voice of the user U acquired via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, the sub-first speaker 140-1 is located in the center of the instrument panel, the sub-first speaker 140-2 is located at the left end of the instrument panel, the sub-first speaker 140-3 is located at the right end of the instrument panel, the sub-first speaker 140-4 is located at the bottom of the left door, and the sub-first speaker 140-5 is located at the bottom of the right door. When the first control device 170 outputs the voice of the user U to the first speaker 140, for example, the first control device 170 outputs the voice from the sub-first speaker 140-2 and the sub-first speaker 140-4 at approximately the same volume and turns off the other sub-first speakers, thereby localizing the sound image so that the voice can be heard from the passenger seat S2 by the occupant P seated in the driver's seat S1. Furthermore, the method of localizing the sound image is not limited to adjusting the volume, and may be performed by shifting the phase of the sound output from each sub-first speaker. For example, to localize the sound image so that the voice is heard from the left side, the timing of outputting the sound from the left sub-first speaker may be slightly earlier than the timing of outputting the same sound from the right sub-first speaker.

[0040] Furthermore, when the first control device 170 outputs the voice of the user U to the first speaker 140, it may localize a sound image so that the voice can be heard by the occupant P from a position at a height corresponding to the height of the head of the user U sitting on the passenger seat S2, and output the voice emitted by the user U to the first speaker 140. In this case, the first speaker 140 needs to have multiple sub-first speakers 140-k (k is a multiple natural number) at different heights.

[0041] 5 is a configuration diagram of second device 200. Second device 200 includes, for example, second communication device 210, second microphone 220, detection device 230, second speaker 240, mobile image display device 250, HMI 260, and second control device 270. Detection device 230 includes, for example, orientation direction detection device 232, head position detection device 234, and motion sensor 236.

[0042] The second communication device 210 is a communication interface for communicating with the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0043] The second microphone 220 collects the voice uttered by the user U. The voice collected by the second microphone 220 is transmitted to the first communication device 110 by the second communication device 210 via the second control device 270, for example.

[0044] The orientation direction detection device 232 is a device for detecting the orientation direction. The orientation direction is a direction based on the facial orientation or gaze direction of the user U, or both. Alternatively, the orientation direction may be a direction indicated by the movement of the user U's arm or fingers, such as tilting the terminal device or swiping the screen. Hereinafter, the orientation direction is assumed to be an angle within a horizontal plane, i.e., an angle without an up-down component. However, the orientation direction may also be an angle including an up-down component. The orientation direction detection device 232 may include a physical sensor (e.g., an acceleration sensor, a gyro sensor, etc.) attached to the VR goggles described below, an infrared sensor that detects multiple positions of the user U's head, or a camera that captures an image of the user U's head. In either case, the second control device 270 calculates the orientation direction based on information input from the orientation direction detection device 232. Various techniques for this are publicly known, so detailed description will be omitted.

[0045] The head position detection device 234 is a device for detecting the position (height) of the head of the user U. For example, one or more infrared sensors or optical sensors provided around the chair on which the user U sits may be used as the head position detection device 234. In this case, the second control device 270 detects the position of the head of the user U based on the presence or absence of a detection signal from the one or more infrared sensors or optical sensors. The head position detection device 234 may also be an acceleration sensor attached to the VR goggles. In this case, the second control device 270 detects the position of the head of the user U by integrating the output of the acceleration sensor minus the gravitational acceleration. The head position information thus acquired is provided to the second control device 270 as height information. The user's head position may also be acquired based on the user U's operation of the HMI 260. For example, the user U may input their height numerically into the HMI 260, or may input their height using a dial switch included in the HMI 260. In these cases, the head position, i.e., height information, is calculated from the height. Furthermore, the user U may input a discrete value such as physique: large / medium / small into the HMI 260 instead of a continuous value. In this case, height information is acquired based on information indicating the physique. Furthermore, the head height of the user U may be acquired simply based on the physique of an average adult (which may be separated by gender) without acquiring the head height of the user specifically.

[0046] The motion sensor 236 is a device for recognizing gesture operations performed by the user U. For example, a camera capturing an image of the upper body of the user U is used as the motion sensor 236. In this case, the second control device extracts feature points (fingertips, wrists, elbows, etc.) of the user U's body from the image captured by the camera, and recognizes the gesture operations of the user U based on the movements of the feature points.

[0047] The second speaker 240 outputs the voice uttered by the occupant P, which is acquired via the second communication device 210. The second speaker 240 has a function of changing the direction from which the voice is heard, for example. The second control device 270 causes the second speaker to output the voice so that the user U can hear the voice from the position of the occupant P as seen from the passenger seat S2. The second speaker 240 includes a plurality of sub-second speakers 240-n (n is a plurality of natural numbers), and sound image localization may be performed by the second control device 270 adjusting the volume of each sub-second speaker 240-n, or, if headphones are attached to the VR goggles, the sound image localization may be performed using the function of the headphones.

[0048] The mobile object image display device 250 displays an image corresponding to the orientation direction as seen from the passenger seat among the images captured by the camera unit 130 (which may be images that have undergone the aforementioned combining process, and will be referred to as a mobile object image hereinafter). FIG. 6 is a diagram for explaining an image corresponding to the orientation direction. In the example of this figure, the VR goggles 255 include an orientation direction detection device 232, a physical sensor serving as a head position detection device 234, and the mobile object image display device 250. The second control device 270 detects the direction in which the VR goggles 255 are facing as the orientation direction φ, using a pre-calibrated direction as a reference direction. Various techniques for this function are already known, so detailed description will be omitted.

[0049] The moving body image display device 250 displays, toward the user U, an image A2 of the moving body image A1 (which has an angle of about 240 degrees in the figure, but the angle of view may be expanded by combining processing as described above) within an angle range of plus or minus α centered on the directional direction φ.

[0050] The HMI 260 is a touch panel, a voice response device (agent device), the above-mentioned switch, etc. The HMI 260 receives various instructions from the occupant P to the second device 200.

[0051] The second control device 270 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (a set of instructions), and the processor controls each part of the second device 200 by executing the set of instructions.

[0052] <Functional configuration> The functional configurations of the first control device 170 and the second control device 270 will be described below. [Example 1] FIG. 7 is a diagram illustrating a first example of the functional configuration of the first control device 170 and the second control device 270. In the first example, the first control device 170 includes a matching request / acceptance unit 171, an audio output control unit 172, an image transmission unit 173, and an on-board device cooperation unit 174. The second control device 270 includes a matching request / acceptance unit 271, an audio output control unit 272, an orientation direction detection unit 273, a head position detection unit 274, a gesture input detection unit 275, an image editing unit 276, and a moving object image display control unit 277. These functional units are realized, for example, by a processor such as a CPU executing a program (a group of instructions). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware.

[0053] The matching request / acceptance unit 171 accepts input of a matching request from the occupant P using the HMI 160 and transmits it to the management server 300, and also accepts input of an acceptance of a matching request received from the management server 300 using the HMI 160 and transmits it to the management server 300. The matching request / acceptance unit 171 controls the first communication device 110 so that the second device 200 of the user U with whom a match has been established becomes the communication partner.

[0054] The audio output control unit 172 controls the first speaker 140 as described above.

[0055] The image transmission unit 173 uses the first communication device 110 to transmit the moving object image A1 to the second device 200.

[0056] The on-board device cooperation unit 174 controls the control target device 190 based on the instruction signal input from the second device 200.

[0057] The matching request / acceptance unit 271 accepts input of a matching request from the user U using the HMI 260 and transmits it to the management server 300, and also accepts input of acceptance of a matching request received from the management server 300 using the HMI 260 and transmits it to the management server 300. The matching request / acceptance unit 271 controls the second communication device 210 so that the first device 100 of the occupant P with whom a match has been established becomes the communication partner.

[0058] The audio output control unit 272 controls the second speaker 240 as described above.

[0059] The orientation direction detection unit 273 detects the orientation direction φ based on the output of the orientation direction detection device 232. The head position detection unit 274 detects the height of the head of the user U based on the output of the head position detection device 234. The head position may be expressed as three-dimensional coordinates, or the head height may simply be detected as the head position. The gesture input detection unit 275 detects a gesture input by the user U based on the output of the motion sensor 236.

[0060] The image editing unit 276 performs processing to cut out an image A2 corresponding to the directional direction φ as seen from the passenger seat from the moving object image A1 (FIG. 6). The moving object image display control unit 277 causes the moving object image display device 250 to display the image A2 cut out by the image editing unit 276. At this time, the image editing unit 276 may cause the moving object image display device 250 to display an image corresponding to the directional direction φ as seen from the height indicated by the height information of the head of the user U.

[0061] [Example 2] Fig. 8 is a diagram showing a second example of the functional configuration of first control device 170 and second control device 270. Compared to the first example in Fig. 7, it differs in that first control device 170 includes image editing unit 175, and second control device 270 does not include image editing unit 276 but includes directivity direction transmission unit 278. The other components basically have the same functions as in the first example, so repeated explanations will be omitted.

[0062] Directivity direction transmitter 278 transmits directivity direction φ detected by directivity direction detector 273 to first device 100 using second communication device 210.

[0063] The image editing unit 175 performs a process of cutting out an image A2 corresponding to the orientation direction φ (transmitted from the second device 200) as seen from the passenger seat from the moving object image A1 (FIG. 6). At this time, the image editing unit 175 may acquire height information of the head of the user U from the second device 200, and perform a process of cutting out an image A2 corresponding to the orientation direction φ as seen from the height indicated by the height information.

[0064] In the second example, image transmission unit 173 transmits image A2 cut out by image editing unit 175 to second device 200 using first communication device 110. Then, moving object image display control unit 277 causes moving object image display device 250 to display image A2 transmitted from first device 100.

[0065] <Other> In the information processing system 1, the user U can view any direction from the passenger seat S2. However, the user U may have restrictions on the direction of view, for example, as determined during matching. For example, the occupant P may wish to view the scenery in the direction of travel of the moving object M or the scenery on the opposite side of the driver's seat S1, but may not want to see his or her own image. This is intended to address the needs of the occupant P and the user U, who are not family or friends, but who want to check the driving feel of the moving object M or view a desired cityscape. In this case, such restrictions are set when the matching processing unit 320 of the management server 300 performs matching processing, and the first control device 170 or the second control device 270 masks the angle range that should not be viewed or corrects the orientation direction φ so that it does not point in the restricted direction, depending on the settings. Furthermore, information regarding such restrictions may be set by the first device 100, since it concerns the privacy of the occupant P.

[0066] Furthermore, the mobile object image display device 250 may replace a portion of an image captured by the camera unit 130 that shows a predetermined object inside the mobile object M with an image (CG image) rendered by computer processing and display the replaced image. FIG. 9 is a diagram showing an example of displaying a replaced image. In the figure, OB is a display device that performs navigation display, etc., and is an example of a "predetermined object." If an image captured on the display screen of the display device is displayed as is, the image may become blurred or visibility may be reduced due to light reflection. For this reason, the mobile object image display device 250 may acquire data for configuring the display screen of the display device or image data rendered by computer processing in the mobile object M from the first device, and then display an image re-rendered from the data by computer processing, or the acquired image data, by embedding it in an image (edited version) captured by the camera unit 130. In this case, the position of the predetermined object in the interior of the moving object M is shared in advance between the first device 100 and the second device, and the moving object image display control unit 277 determines whether the predetermined object is included in the image to be displayed on the moving object image display device 250, for example, based on the orientation direction φ, and performs image replacement as described above if it is determined that the predetermined object is included. Furthermore, the "predetermined object" may be the head or face of the occupant P. In that case, a CG image such as an avatar may be changed according to the display of the occupant P.

[0067] <Summary> The information processing system 1 configured as described above can enhance the sense of realism provided to both the occupant P of the moving body M and the user U who is in a location different from the moving body M. Since an image corresponding to the user U's orientation direction φ as seen from the passenger seat is displayed to the user U, the user U can view the scenery as if he or she were sitting in the passenger seat S2 and looking around. Furthermore, the first speaker 140 outputs the voice of the user U by localizing a sound image so that the voice can be heard from the passenger seat S2, allowing the occupant P to have a conversation with the user U as if the user U were sitting in the passenger seat S2. Furthermore, the second speaker 240 outputs the voice of the occupant P by localizing a sound image so that the voice can be heard from the position of the occupant P as seen from the passenger seat S2, allowing the user U to have a conversation with the occupant P as if he or she were sitting in the passenger seat S2.

[0068] <Usage example> The information processing system 1 can be used in the following manner.

[0069] (A) A mode in which the occupant P and the user U are in a relationship such as family or friends, and a virtual drive is provided to the user U. The user U can talk with the occupant P about the scenery around the moving body M while looking at the image.

[0070] (B) A mode in which the passenger P is a general user and the user U is a provider of route guidance services, driving instruction services, etc. The user U can provide route guidance or driving instruction in places that are difficult to find using a navigation device or that are not on a map while viewing the scenery around the moving object M.

[0071] (C) A mode in which the passenger P is a celebrity and the user U is a general user, and a commercial virtual drive is provided to the user U. In this case, multiple users U are simultaneously associated with one passenger P, and, for example, audio transfer from the user U's side may be set to off.

[0072] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0073] 1. Information Processing Systems 100 1st device 110 First communication device 120 First Microphone 130 Camera Unit 132 Indoor Camera 134 Outdoor Camera 140 1st Speaker 150 User display device 160 HMI 170 First control device 190 Controlled Equipment 200 2nd device 210 Second communication device 220 Second Microphone 230 Detection Device 232 Directional direction detection device 234 Head position detection device 236 Motion Sensor 240 Second Speaker 250 Mobile image display device 260 HMI 270 Second control device 300 Management Server M Mobile object

Claims

1. a first device mounted on a vehicle on which a passenger boards; a second device used by a user at a location different from the mobile object; An information processing system comprising: The first device is a first communication device for communicating with a second communication device of the second device; a first speaker that outputs a voice uttered by the user and acquired via the first communication device; a camera unit having one or more cameras, the camera unit being provided at a height position of the head of the occupant in a predetermined seat of the vehicle, the camera unit including at least an interior camera capable of capturing an image of the inside of the vehicle as seen from the predetermined seat; and The second device is the second communication device communicating with the first communication device; a second microphone for collecting the voice uttered by the user; a detection device for detecting a direction of orientation of the user; a display device that displays an image corresponding to the orientation direction as seen from the predetermined seat among the images captured by the camera unit; and and the second communication device transmits the sound collected by the second microphone to the first communication device; The second device further acquires height information indicating a height of the user's head, the display device displays an image corresponding to the orientation direction as seen from a height indicated by the height information above the predetermined seat. Information processing system.

2. The camera unit is provided on a neck pillow on the predetermined seat. The information processing system according to claim 1 .

3. The first speaker outputs the voice uttered by the user by localizing a sound image so that the voice can be heard by the occupant from the predetermined seat.

2. The information processing system according to claim 1.

4. the first speaker includes a plurality of sub-first speakers arranged at different positions from each other, The first device further includes a first control device that adjusts volumes and / or phase differences of the plurality of sub first speakers to localize a sound image so that the passenger can hear the sound from the predetermined seat.

4. The information processing system according to claim 3.

5. The second device further acquires height information indicating a height of the user's head, the first control device localizes a sound image so that the sound can be heard by the occupant from a height position indicated by the height information above the predetermined seat, and outputs the sound emitted by the user to the first speaker; 5. The information processing system according to claim 4.

6. the second communication device transmits information of the orientation direction to the first communication device; the first device further includes a first control device that controls the first communication device to selectively transmit, to the second communication device, an image corresponding to the orientation direction acquired via the first communication device, from among images captured by the camera unit; a display device of the second device displays an image corresponding to the orientation direction as viewed from the predetermined seat, the image being acquired via the second communication device; 2. The information processing system according to claim 1.

7. the first communication device transmits an image captured by the camera unit to the second communication device; The second device further includes a second control device that selectively displays, on the display device, an image corresponding to the orientation direction from among images captured by the camera unit.

2. The information processing system according to claim 1.

8. the first device further includes a first microphone that collects at least a voice uttered by the occupant; the second device further includes a second speaker that outputs a voice uttered by the occupant and acquired via the second communication device; The first communication device transmits the sound collected by the first microphone to the second communication device.

2. The information processing system according to claim 1.

9. the second speaker outputs the sound emitted by the occupant by localizing a sound image so that the sound can be heard from the position of the occupant as seen from the predetermined seat to the user.

9. The information processing system according to claim 8.

10. the display device is a display device of VR (Virtual Reality) goggles, The detection device includes a physical sensor attached to the VR goggles.

2. The information processing system according to claim 1.

11. The display device is capable of executing a mode in which the angle range in which the display can be performed is limited.

2. The information processing system according to claim 1.

12. the moving body is a vehicle, The predetermined seat is the passenger seat.

2. The information processing system according to claim 1.

13. the display device replaces a portion of the image captured by the camera unit that shows a predetermined item in the room of the moving body with an image drawn by computer processing and displays the image.

2. The information processing system according to claim 1.

Images