Information processing system

Abstract

To provide an information processing system capable of increasing presence to be imparted to both crew members of a moving body and users in a different place from the moving body.SOLUTION: The present invention relates to an information processing system that comprises a first device mounted on a moving body that a crew member rides on, and a second device used by a user in a different place from the moving body, where the first device has a first communication device which communicates with a second communication device of the second device and a camera unit provided on the moving body, and the second device has a second communication device which communicates with the first communication device, a detection device for detecting the user, and a display device which displays an image that the camera unit captures.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system.

Background Art

[0002] Conventionally, research has been underway on communication between a device mounted on a moving body such as a vehicle and a device used at a location separate from the moving body to share an image of the scenery outside the vehicle or the like (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, there were cases where the sense of presence was not sufficiently felt for both the passengers of the moving body and the users in a location different from the moving body.

[0005] The present invention has been made in consideration of such circumstances, and one of its objects is to provide an information processing system capable of enhancing the sense of presence given to both the passengers of the moving body and the users in a location different from the moving body.

Means for Solving the Problems

[0006] The information processing system according to this invention employs the following configuration. (1) An information processing system according to one aspect of the present invention is an information processing system comprising: a first device mounted on a mobile vehicle on which a crew member is riding; and a second device used by a user at a location different from the mobile vehicle, wherein the first device comprises a first communication device that communicates with a second communication device of the second device, and a camera unit provided on the mobile vehicle; and the second device comprises a second communication device that communicates with the first communication device, a detection device for detecting the user, and a display device for displaying an image captured by the camera unit.

[0007] (2): An information processing system according to one aspect of the present invention is an information processing system comprising: a first device mounted on a mobile vehicle on which a crew member is riding; and a second device used by a user at a location different from the mobile vehicle, wherein the first device comprises: a first communication device that communicates with a second communication device of the second device; and a camera unit provided on the mobile vehicle and having one or more cameras, including at least an indoor camera capable of imaging the inside of the mobile vehicle; the second device comprises: a second communication device that communicates with the first communication device; a detection device for detecting the user's direction of orientation and gestures; and a display device that displays an image from among the images captured by the camera unit that corresponds to the user's direction of orientation; the second device determines, based on the detected direction of orientation and gestures, the operating unit of the mobile vehicle that the user is pointing at and the content of the operation to the operating unit from among the images; and the second communication device transmits the determined content of the operation to the operating unit to the first communication device.

[0008] (3): An information processing system according to one aspect of the present invention is an information processing system comprising: a first device mounted on a mobile vehicle on which a crew member is riding; and a second device used by a user at a location different from the mobile vehicle, wherein the first device comprises: a first communication device that communicates with a second communication device of the second device; and a camera unit having one or more cameras, including at least an interior camera provided on a predetermined seat of the mobile vehicle and capable of capturing images of the inside of the mobile vehicle as seen from the predetermined seat; the second device comprises: a second communication device that communicates with the first communication device; a detection device for detecting the direction of the user's gaze or gesture; and a display device that displays an image from among the images captured by the camera unit that corresponds to the direction of the user's gaze as seen from the predetermined seat; the second device determines, based on the detected direction of the user's gaze or gesture, the operating unit of the mobile vehicle that the user is pointing at and the content of the operation to the operating unit from among the images; and the second communication device transmits the determined content of the operation to the operating unit to the first communication device.

[0009] (4): In the embodiment of (2) or (3) above, the display device changes the position of the operation unit in response to the gesture.

[0010] (5) In the embodiment of (2) or (3) above, the display device changes the size of the operation section in response to the gesture.

[0011] (6) In the embodiment of (2) or (3) above, the display device highlights the operation unit that the user can operate from among the plurality of operation units.

[0012] (7) In the embodiment of (2) or (3) above, the second device determines which of the images the user is permitted to operate the operating unit of the mobile body, based on the user's permission to operate the operating unit of the mobile body.

[0013] (8) In the embodiment of (2) or (3) above, the first device further comprises a display that displays an image indicating the operating unit that the user is pointing to.

[0014] (9) In the embodiment of (2) or (3) above, the second communication device transmits the information of the direction of direction to the first communication device, and the first device further includes a first control device that controls the first communication device to selectively transmit to the second communication device an image corresponding to the direction of direction acquired via the first communication device from among the images captured by the camera unit, and the display device of the second device displays an image corresponding to the direction of direction as seen from the predetermined seat, which is acquired via the second communication device.

[0015] (10): In the embodiment of (2) or (3) above, the first communication device transmits images captured by the camera unit to the second communication device, and the second device further includes a second control device that causes the camera unit to selectively display images corresponding to the direction of orientation on the display device.

[0016] (11): In the embodiment of (2) or (3) above, the display device is a display device for VR (Virtual Reality) goggles, and the detection device includes a physical sensor attached to the VR goggles.

[0017] (12): In the embodiment of (9) above, the moving body is a vehicle and the predetermined seat is a passenger seat. [Effects of the Invention]

[0018] According to the embodiments described in (1) to (12), the sense of presence can be enhanced for both the occupants of the moving vehicle and the users who are in a different location from the moving vehicle. [Brief explanation of the drawing]

[0019] [Figure 1] This diagram shows the operating environment of information processing system 1 and management server 300. [Figure 2] This is a diagram showing an example of the content of user data 360. [Figure 3] This is a configuration diagram of the first device 100. [Figure 4] This is a diagram showing an example of the arrangement of a part of the first device 100 in the moving body M. [Figure 5] This is a configuration diagram of the second device 200. [Figure 6] This is a diagram for explaining an image corresponding to the pointing direction. [Figure 7] This is a diagram showing a first example of the functional configuration of the first control device 170 and the second control device 270. [Figure 8] This is a diagram showing a second example of the functional configuration of the first control device 170 and the second control device 270. [Figure 9] This is a diagram showing an example of an image corresponding to the pointing direction. [Figure 10] This is a diagram showing an example of the content of user data 360C that defines the operation authority of the moving body M. [Figure 11] This is a diagram showing an example of an image indicating an operation area for which the user U has operation authority. [Figure 12] This is a diagram showing an example of a method of superimposing and displaying an instruction image H representing the gesture of the user U on the operation area. [Figure 13] This is a diagram showing an example of a method of changing the display mode of the operation area to be displayed for the user U according to the gesture of the user U.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, referring to the drawings, embodiments of the information processing system of the present invention will be described. The information processing system includes a first device mounted on a moving body on which an occupant rides, and a second device used by a user at a location different from the moving body. The moving body is, for example, a vehicle, but any moving body on which an occupant can ride may be used. Also, the occupant is mainly the driver of the moving body, but may be an occupant other than the driver.

[0021] Between the first and second devices, audio collected by a microphone is transmitted to the other side and played back through a speaker, creating a situation similar to a telephone call. Furthermore, a portion of the image captured by the camera unit of the first device is displayed by the second device, providing the second device with Mixed Reality (MR). The first and second devices do not need to have a one-to-one relationship; multiple first devices can be matched with multiple second devices in a one-to-many relationship, and the system can operate as an information processing system. In the latter case, for example, one occupant can communicate with multiple users simultaneously or sequentially.

[0022] <Basic configuration> Figure 1 shows the operating environment of the information processing system 1 and the management server 300. The information processing system 1 includes a first device (mobile device) 100 mounted on a mobile device M, and a second device (user device) 200 used by user U at a location different from the mobile device M (although it is not ruled out that it may be in a nearby location). The first device 100, the second device 200, and the management server 300 each communicate with each other via a network NW. The information processing system 1 may or may not include the management server 300.

[0023] The management server 300 includes, for example, a communication device 310, a matching processing unit 320, and a storage unit 350. User data 360 is stored in the storage unit 350.

[0024] The communication device 310 is a communication interface for connecting to the network NW. Communication between the communication device 310 and the first device 100, and communication between the communication device 310 and the second device 200, are conducted, for example, according to TCP / IP (Transmission Control Protocol / Internet Protocol).

[0025] The matching processing unit 320 is implemented by a processor, such as a CPU (Central Processing Unit), executing a program (set of instructions) stored in a storage medium. The storage unit 350 is a RAM (Random Access Memory), HDD (Hard Disk Drive), flash memory, etc.

[0026] Figure 2 shows an example of the contents of user data 360. User data 360 includes, for example, a crew list 360A in which the crew ID, which is the identification information of crew member P, their communication identification information (IP address, etc.), and the user ID, which is the identification information of user U to be matched, are associated with each other, and a user list 360B in which the user ID, their communication identification information (IP address, etc.), and the crew member P to be matched are associated with each other. User data 360 may be generated in any manner, not limited to the manner shown in Figure 2, as long as it includes this information.

[0027] When the communication device 310 receives a matching request from user U via the second device 200, or from crew member P via the first device 100, the matching processing unit 320 refers to the user data 360 to perform matching between user U and crew member P. The communication processing unit 320 then uses the communication device 310 to transmit the communication identification information of crew member P's first device 100 to the second device 200 of the matched user U, and the communication identification information of user U's second device 200 to the first device 100 of the matched crew member P. After receiving these, the first device 100 and the second device 200 communicate in real-time, for example, using UDP (User Datagram Protocol).

[0028] Figure 3 is a configuration diagram of the first device 100. The first device 100 includes, for example, a first communication device 110, a first microphone 120, a camera unit 130, a first speaker 140, a user display device 150, an HMI (Human-machine Interface) 160, and a first control device 170. The first control device 170 is connected to a controlled device 190 mounted on a mobile body M.

[0029] The first communication device 110 is a communication interface for communicating with the communication device 310 of the management server 300 and the second communication device 210 of the second device 200 via the network NW.

[0030] The first microphone 120 collects at least the voice emitted by the crew member P. The first microphone 120 may be installed inside the cabin of the mobile vehicle M and have the sensitivity to collect sounds from outside the mobile vehicle M, or it may include a microphone installed inside the cabin of the mobile vehicle M and a microphone installed outside the mobile vehicle M. The voice collected by the first microphone 120 is transmitted to the second communication device 210 by the first communication device 110 via, for example, the first control device 170.

[0031] The camera unit 130 includes at least an indoor camera 132 and may also include an outdoor camera 134. The first speaker 140 outputs audio spoken by user U, which is acquired via the first communication device 110. Details of the camera unit 130 and the arrangement of the first speaker 140 will be explained later with reference to Figure 4.

[0032] The user display device 150 virtually displays user U as if the user U were present inside the mobile device M. For example, the user display device 150 may display a hologram or display user U in a part of the mobile device M corresponding to a mirror or window.

[0033] HMI160 consists of a touch panel and a voice response device (agent device), etc. HMI160 receives various instructions from crew member P to the first device 100.

[0034] The first control device 170 includes, for example, a processor such as a CPU, and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the first device 100 by executing the set of instructions.

[0035] The controlled device 190 is, for example, a navigation system or a driver assistance system mounted on the mobile body M.

[0036] Figure 4 shows an example of the arrangement of part of the first device 100 in the mobile vehicle M. The interior camera 132 is attached, for example, to the neck pillow of the passenger seat S2 (an example of a "designated seat") via attachment 132A, and is positioned slightly away from the backrest of the passenger seat S2 towards the direction of travel of the mobile vehicle M. The interior camera 132 has a wide-angle lens and is capable of imaging the area represented by the hatched area 132B in the figure. The interior camera 132 can photograph not only the interior of the mobile vehicle M but also the exterior through the windows. In the following description, the passenger seat S2 is assumed to be a designated seat, but the designated seat may be another seat such as a rear seat.

[0037] The outdoor camera 134 includes, for example, a plurality of sub-outdoor cameras 134-1 to 134-4. By combining the images captured by the plurality of sub-outdoor cameras 134-1 to 134-4, an image such as a panoramic image capturing the outside of the mobile body M is obtained. The outdoor camera 134 may also include (or in addition to) these a wide-angle camera mounted on the roof of the mobile body M. An indoor camera 132 capable of capturing the area behind the passenger seat S2 may be added, and the mobile body image described later may be generated as a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 by the first control device 170, or it may be generated as a 360-degree panoramic image by appropriately combining images captured by indoor cameras 132 and images captured by outdoor cameras 134.

[0038] The first speaker 140 outputs the voice of user U acquired via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, sub-first speaker 140-1 is located in the center of the instrument panel, sub-first speaker 140-2 is located at the left end of the instrument panel, sub-first speaker 140-3 is located at the right end of the instrument panel, sub-first speaker 140-4 is located at the bottom of the left door, and sub-first speaker 140-5 is located at the bottom of the right door. When the first control device 170 outputs the voice of user U to the first speaker 140, for example, it outputs sound at a similar volume from sub-first speakers 140-2 and 140-4, and turns off the other sub-first speakers, thereby localizing the sound image so that the voice sounds to the passenger seat S2 for the occupant P seated in the driver's seat S1. Furthermore, the method of sound image localization is not limited to volume adjustment, but may also be performed by shifting the phase of the sound output by each sub-first speaker. For example, to localize the sound image so that it sounds like it is coming from the left, the timing of outputting sound from the left sub-first speaker should be slightly earlier than the timing of outputting the same sound from the right sub-first speaker.

[0039] Furthermore, when the first control device 170 outputs the voice of user U to the first speaker 140, it may direct the sound image to the occupant P so that the voice is heard from a position corresponding to the height of user U's head on the passenger seat S2, and then output the voice emitted by user U to the first speaker 140. In this case, the first speaker 140 needs to have multiple child first speakers 140-k (where k is a number of natural numbers) at different heights.

[0040] Figure 5 is a configuration diagram of the second device 200. The second device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a mobile image display device 250, an HMI 260, and a second control device 270. The detection device 230 includes, for example, a direction detection device 232, a head position detection device 234, and a motion sensor 236.

[0041] The second communication device 210 is a communication interface for communicating with the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0042] The second microphone 220 collects the voice emitted by user U. The voice collected by the second microphone 220 is transmitted to the first communication device 110 by the second communication device 210 via the second control device 270, for example.

[0043] The direction detection device 232 is a device for detecting the direction of direction. The direction of direction is the direction of the user U's face or gaze, or a direction based on both. Alternatively, it may be a direction indicated by the movement of the user's arms or fingers, such as tilting the terminal device used by the user U or swiping the screen. Hereinafter, the direction of direction will be assumed to be an angle in the horizontal plane, that is, an angle without a vertical component, but the direction of direction may also be an angle that includes a vertical component. The direction detection device 232 may include a physical sensor attached to the VR goggles described later (e.g., an accelerometer, a gyroscope, etc.), an infrared sensor that detects multiple positions of the user U's head, or a camera that images the user U's head. In any case, the second control device 270 calculates the direction of direction based on the information input from the direction detection device 232. Since various technologies for this are publicly known, a detailed explanation will be omitted.

[0044] The head position detection device 234 is a device for detecting the position (height) of the user U's head. For example, one or more infrared sensors or optical sensors installed around the chair where the user U is seated can be used as the head position detection device 234. In this case, the second control device 270 detects the position of the user U's head based on the presence or absence of detection signals from one or more infrared sensors or optical sensors. Alternatively, the head position detection device 234 may be an accelerometer attached to the VR goggles. In this case, the second control device 270 detects the position of the user U's head by integrating the output of the accelerometer minus the acceleration due to gravity. The head position information obtained in this way is provided to the second control device 270 as height information. The user's head position may also be obtained based on the user U's operation of the HMI 260. For example, the user U may input their height numerically into the HMI 260, or they may input their height using a dial switch included in the HMI 260. In these cases, the head position, i.e., height information, is calculated from the height. Furthermore, user U may input discrete values ​​such as body size: large / medium / small into the HMI260 instead of continuous values. In this case, height information will be obtained based on the information indicating body size. Alternatively, the user's head height may be obtained simply based on the average adult body size (which may be separated by gender) without specifically obtaining the user's head height.

[0045] The motion sensor 236 is a device for recognizing gestures made by user U. For example, a camera that captures images of user U's upper body is used as the motion sensor 236. In this case, the second control device extracts characteristic points of user U's body (fingertips, wrists, elbows, etc.) from the images captured by the camera and recognizes user U's gestures based on the movement of these characteristic points.

[0046] The second speaker 240 outputs the voice emitted by occupant P, which is acquired via the second communication device 210. The second speaker 240 has a function, for example, to change the direction from which the voice is heard. The second control device 270 causes the second speaker to output voice so that the voice is heard by the user U from the position of occupant P as seen from the passenger seat S2. The second speaker 240 may include multiple child second speakers 240-n (where n is a number of natural numbers), and sound localization may be achieved by the second control device 270 adjusting the volume of each child second speaker 240-n, or if headphones are attached to the VR goggles, sound localization may be achieved by utilizing the function of the headphones.

[0047] The mobile image display device 250 displays the image corresponding to the direction of orientation as seen from the passenger seat, from among the images captured by the camera unit 130 (which may be the image after the aforementioned stitching process, and will be referred to as the mobile image below). Figure 6 is a diagram illustrating the image corresponding to the direction of orientation. In this example, the VR goggles 255 include a physical sensor as a direction detection device 232 and a head position detection device 234, and the mobile image display device 250. The second control device 270 detects the direction the VR goggles 255 are facing as the direction of orientation φ, using a pre-calibrated direction as the reference direction. Since various methods for this function are already publicly known, a detailed explanation will be omitted.

[0048] The mobile object image display device 250 displays image A2, which is part of the mobile object image A1 (which has an angle of approximately 240 degrees in the figure, but the field of view may be expanded by the merging process as described above), towards the user U.

[0049] HMI260 consists of a touch panel, a voice response device (agent device), or the switches mentioned above. HMI260 receives various instructions from crew member P to the second device 200.

[0050] The second control device 270 includes, for example, a processor such as a CPU, and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the second device 200 by executing the set of instructions.

[0051] <Functional Configuration> The functional configurations of the first control device 170 and the second control device 270 will be described below. [Example 1] Figure 7 shows a first example of the functional configuration of the first control device 170 and the second control device 270. In the first example, the first control device 170 includes a matching request / acceptance unit 171, an audio output control unit 172, an image transmission unit 173, and an onboard device linkage unit 174. The second control device 270 includes a matching request / acceptance unit 271, an audio output control unit 272, a direction detection unit 273, a head position detection unit 274, a gesture input detection unit 275, an image editing unit 276, and a mobile object image display control unit 277. These functional units are realized, for example, by a processor such as a CPU executing a program (set of instructions). Some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or by the cooperation of software and hardware.

[0052] The matching request / acceptance unit 171 uses the HMI 160 to receive matching requests from crew member P and transmits them to the management server 300, and also uses the HMI 160 to receive acceptance inputs for matching requests received from the management server 300 and transmits them to the management server 300. The matching request / acceptance unit 171 controls the first communication device 110 to communicate with the second device 200 of user U, with whom a match has been made.

[0053] As described above, the audio output control unit 172 controls the first speaker 140.

[0054] The image transmission unit 173 transmits the mobile image A1 to the second device 200 using the first communication device 110.

[0055] The mounted device coordination unit 174 controls the controlled device 190 based on instruction signals input from the second device 200.

[0056] The matching request / acceptance unit 271 uses the HMI 260 to receive matching requests from user U and transmits them to the management server 300, and also uses the HMI 260 to receive acceptance inputs for matching requests received from the management server 300 and transmits them to the management server 300. The matching request / acceptance unit 271 controls the second communication device 210 to communicate with the first device 100 of crew member P for whom a match has been made.

[0057] As described above, the audio output control unit 272 controls the second speaker 240.

[0058] The direction detection unit 273 detects the direction φ based on the output of the direction detection device 232. The head position detection unit 274 detects the height of the user U's head based on the output of the head position detection device 234. The head position may be expressed as three-dimensional coordinates, or the height of the head may simply be detected as the head position. The gesture input detection unit 275 detects the gesture input of the user U based on the output of the motion sensor 236.

[0059] The image editing unit 276 processes the image A2 corresponding to the directional direction φ as viewed from the passenger seat from the moving image A1 (Figure 6). The moving image display control unit 277 displays the image A2 extracted by the image editing unit 276 on the moving image display device 250. At this time, the image editing unit 276 may also display on the moving image display device 250 an image corresponding to the directional direction φ as viewed from the height indicated by the height information of the user U's head.

[0060] [Example 2] Figure 8 shows a second example of the functional configuration of the first control device 170 and the second control device 270. Compared to the first example in Figure 7, the first control device 170 includes an image editing unit 175, while the second control device 270 does not include an image editing unit 276 but includes a directional transmission unit 278. As the other components basically have the same functions as in the first example, further explanation is omitted.

[0061] The direction transmitting unit 278 transmits the direction φ detected by the direction detection unit 273 to the first device 100 using the second communication device 210.

[0062] The image editing unit 175 performs the process of extracting image A2 from the moving image A1 that corresponds to the directional direction φ (transmitted from the second device 200) as seen from the passenger seat (Figure 6). At this time, the image editing unit 175 may also obtain the height information of the user U's head from the second device 200 and perform the process of extracting image A2 that corresponds to the directional direction φ as seen from the height indicated by the height information.

[0063] In the second example, the image transmission unit 173 uses the first communication device 110 to transmit the image A2 extracted by the image editing unit 175 to the second device 200. The mobile image display control unit 277 then displays the image A2 transmitted from the first device 100 on the mobile image display device 250.

[0064] <Gesture-based operation> Figure 9 shows an example of an image corresponding to the direction of direction of direction. The image shown in Figure 9 is an image captured by the camera unit 130 that corresponds to the direction of direction as seen from the passenger seat (hereinafter sometimes referred to as the "direction image"), and is displayed to the user U by the mobile image display device 250. In the direction image of Figure 9, the operation area OA is the same (or simplified) operation area displayed on the HMI 160 of the mobile body M, and is displayed to the user U, enabling the user U to remotely control the HMI 160. As will be explained in detail below, the content of the operation performed by the user U on the operation area OA is transmitted to the first device 100 by the second communication device 210, and the first device 100 causes the mobile body M to execute the content of the operation. As shown in Figure 9, the operation area OA includes, for example, multiple operation areas such as operation areas OA1 to OA6.

[0065] First, the second device 200 identifies an operation area from among the operation areas OA1 to OA6 of the directional image that corresponds to the direction of user U's directional orientation φ detected by the direction of orientation detection unit 273. For example, if user U makes a gesture of pointing their finger towards operation area OA1 indicating "search nearby," the mobile image display device 250 superimposes information about stores / facilities located within a predetermined distance from the mobile body M onto the directional image, displaying it as a map, for example. Simultaneously, the second communication device 210 transmits this operation information to the first device 100, and the first device 100, upon receiving this operation information, displays the information about stores / facilities located within a predetermined distance from the mobile body M on the mobile body M's HMI 160. Figure 9 shows a scene where, as a result of user U making a gesture for "search nearby," information about stores / facilities around the mobile body M is displayed as a map. This allows the occupant P to check information about the area around the mobile body M without having to operate the HMI 160 themselves, even while driving the mobile body M.

[0066] Furthermore, in the scenario shown in Figure 9, the second device 200 identifies a point on the map that corresponds to the direction of user U's gaze φ detected by the direction detection unit 273. For example, in Figure 9, points P1 to P3 are displayed on the map, and the direction detection unit 273 identifies point P1 as the point corresponding to the direction of user U's gaze φ. With point P1 identified, if the gesture input detection unit 275 detects a gesture by user U operating point P1, the second device 200 executes the operation corresponding to point P1 that corresponds to the detected gesture. For example, in Figure 9, if user U makes a gesture pointing towards point P1, the mobile image display device 250 displays detailed information of the store corresponding to point P1 (e.g., menu information) superimposed on the directional image.

[0067] Furthermore, for example, if user U makes a gesture pointing their finger towards the operation area OA2 indicating "Audio" with their direction of direction φ, the audio output control unit 272 causes the second speaker 240 to output sound. Simultaneously, the second communication device 210 transmits the operation information to the first device 100, and the first device 100, upon receiving the operation information, causes the audio equipment of the mobile vehicle M to play. For example, user U selects a song to play by operating the operation area OA2 with a gesture, and the audio output control unit 272 causes the second speaker 240 to play the selected song. Simultaneously, the second communication device 210 transmits information indicating the selected song to the first device 100, and the first device 100 causes the audio equipment of the mobile vehicle M to play the song. As a result, occupant P can listen to music without operating the HMI 160 themselves, even while driving the mobile vehicle M.

[0068] Furthermore, for example, if user U makes a gesture pointing their finger towards the operation area OA3 indicating "Settings" with their direction of view φ, the mobile image display device 250 displays a screen related to changing the settings of the mobile body M superimposed on the direction of view image. User U changes the settings of the mobile body M by making further gestures on the screen related to changing the settings of the mobile body M. At the same time, the second communication device 210 transmits this operation information to the first device 100, and the first device 100 changes the settings of the mobile body M in response to receiving this operation information. For example, user U changes the air conditioning settings of the mobile body M by operating the operation area OA3 with a gesture, and the second communication device 210 transmits information indicating the changed air conditioning settings to the first device 100. The first device 100 changes the air conditioning settings of the mobile body M based on the information indicating the changed air conditioning settings. As a result, the occupant P can change the air conditioning settings of the mobile body M without operating the HMI 160 themselves, even while driving the mobile body M.

[0069] Furthermore, for example, if user U makes a gesture pointing their finger towards the operation area OA4 indicating "Navigation" with their direction of direction φ, the mobile image display device 250 overlays a screen related to changing the navigation settings of the mobile body M onto the direction image. User U changes the navigation settings of the mobile body M by making further gestures on the screen related to changing the navigation settings of the mobile body M. At the same time, the second communication device 210 transmits this operation information to the first device 100, and the first device 100 changes the navigation settings of the mobile body M in response to receiving this operation information. For example, user U changes the destination of the mobile body M by operating the operation area OA4 with a gesture, and the second communication device 210 transmits information indicating the changed destination to the first device 100. The first device 100 changes the destination of the mobile body M based on the information indicating the changed destination. As a result, occupant P can change the destination of the mobile body M without operating the HMI 160 themselves, even while driving the mobile body M.

[0070] Furthermore, for example, if user U makes a gesture of pointing their finger towards the operation area OA5 that indicates the hazard lights, the second communication device 210 transmits the operation information to the first device 100, and the first device 100, upon receiving the operation information, flashes the hazard lights of the mobile vehicle M. As a result, even while driving the mobile vehicle M, occupant P can flash the hazard lights to alert surrounding vehicles of the mobile vehicle M without having to operate the HMI 160 themselves.

[0071] The above-mentioned operation areas OA1 to OA5 are merely examples, and operation areas related to arbitrary control of the mobile body M may be displayed on the mobile body image display device 250, allowing the user U to operate them by gesture. For example, the operation areas may include an operation area for opening and closing the windows of the mobile body M, an operation area for operating the wipers of the mobile body M, and an operation area for selecting a store menu displayed by selecting a location and making a payment.

[0072] <Operation Permissions> Thus, in this embodiment, the mobile object image display device 250 overlays the operation area for user U to remotely control the mobile object M onto the image. However, for every combination of crew member P and user U, it is not always desirable to allow user U to operate all operation areas. More specifically, if crew member P and user U have a close relationship, it is preferable to allow user U to operate more operation areas, while if crew member P and user U have a distant relationship, it is preferable to allow user U to operate fewer operation areas. Therefore, in this embodiment, the management server 300 stores user data 360C in the storage unit 350 in advance, which defines the scope of user U's operation area (i.e., operation authority) for each combination of crew member P and user U. When crew member P and user U are matched, the management server 300 refers to the user data 360C and allows user U to operate the operation areas within the scope of their operation authority.

[0073] Figure 10 shows an example of the contents of user data 360C that defines the operating privileges of mobile device M. User data 360C associates, for example, the operating privileges of a user corresponding to a combination of crew ID and user ID. User data 360C may, for example, be pre-set by the crew to grant operating privileges to each user. In this case, it may be time-consuming for the crew to set operating privileges for each individual user. Therefore, for example, as shown in the second and third lines of user data 360C, the crew may uniformly set operating privileges to grant to all users (i.e., "ALL") and individually set operating privileges for users to whom they wish to grant more privileges. Conversely, the crew may uniformly set operating privileges to grant to all users and individually set operating privileges for users to whom they wish to grant fewer privileges.

[0074] Figure 11 shows an example of an image indicating an operation area for which user U has control. The image shown in Figure 11 is displayed, for example, when user U selects point P1 in the image shown in Figure 9 using a gesture. In Figure 11, the newly displayed operation area OA includes operation areas OA1 to OA5, and operation area OA6 for selecting a menu.

[0075] The mobile image display device 250 may improve the convenience of user U by explicitly displaying only the operation areas to which user U has been granted operation authority. For example, as shown in Figure 11, if user U only has the authority to operate the audio equipment of the mobile device M, the mobile image display device 250 will highlight only operation area OA2, which corresponds to the operation of the audio equipment. Alternatively, for example, the mobile image display device 250 may display only operation area OA2, which corresponds to the operation of the audio equipment, and hide the other operation areas. This improves the convenience of user U, who selects and operates operation areas.

[0076] <Display Gestures> Furthermore, the mobile image display device 250 may superimpose an instruction image H representing the gesture of user U detected by the gesture input detection unit 275 onto the operation area. Figure 12 shows an example of a method for superimposing an instruction image H representing the gesture of user U onto the operation area. Figure 12 shows the situation in the image shown in Figure 11 where user U has selected one of the displayed menu items.

[0077] In this case, the mobile image display device 250 superimposes an instruction image H representing the gesture of user U onto the menu item selected by user U by gesture, and the second communication device 210 transmits instruction information to the first device 100 indicating that user U has selected the menu item. Upon receiving this instruction information, the first device 100 similarly displays the instruction image H representing user U's gesture on the HMI 160. This allows the occupant P to grasp the operation performed by user U in real time. In this embodiment, the mobile image display device 250 displays a hand icon as the instruction image H representing user U's gesture, but the present invention is not limited to such a configuration. For example, the mobile image display device 250 may display a shadow representing the assumed position of user U's hand, or more generally, it may display the operation area that user U is pointing to in a visible manner. The first device 100 may also have a mode that displays the image displayed on the mobile image display device 250 (the image that user U is looking at) on the HMI 160.

[0078] Furthermore, the mobile image display device 250 may change the display mode of the operation area shown to the user U in accordance with the gesture of the user U detected by the gesture input detection unit 275. Figure 13 is a diagram showing an example of a method for changing the display mode of the operation area shown to the user U in accordance with the gesture of the user U. As an example, Figure 13 shows a situation in which the user U makes a gesture to grasp and move the operation area OA displayed by the mobile image display device 250.

[0079] For example, if the gesture input detection unit 275 of the mobile image display device 250 detects that user U has made a gesture of grasping and moving the operation area OA, the device may modify the instruction image H representing user U's hand to display it as if grasping the operation area OA. In this case, the mobile image display device 250 changes the position of the operation area OA in the direction in which user U made the gesture and displays it. This allows user U to change the position of the operation area OA in the image and view the image area they wish to focus on. In this case, the device may also restrict the movement of the operation area OA in the depth direction and the rotation of the screen of the operation area OA, and change the position of the operation area OA while maintaining a constant distance between the operation area OA and user U. This prevents excessive shaking of the operation area OA and improves the user experience.

[0080] Furthermore, for example, the mobile image display device 250 may hide the operation area OA when the gesture input detection unit 275 detects that the user U has made a gesture of grasping the operation area OA and moving it with an acceleration greater than a predetermined value. In this case, for example, the mobile image display device 250 will fade out the operation area OA in the direction in which the user U made the gesture. This allows the user U to hide unnecessary operation areas OA and view the image area they wish to focus on.

[0081] Furthermore, for example, if the mobile image display device 250 detects that the gesture input detection unit 275 has made a gesture of grasping the operation area OA with both hands and widening (narrowing) it, the device may enlarge (reduce) the size of the operation area OA. This allows the user U to change the size of the operation area OA in the image and view the image area they wish to focus on.

[0082] <Other> In the information processing system 1, it was explained that user U can see any direction from the passenger seat S2. However, restrictions may be placed on the directions that user U can see, for example, by agreements made during matching. For example, occupant P may want to be provided with the scenery in the direction of travel of the mobile vehicle M, or the scenery opposite the driver's seat S1, but not their own image. This is intended to meet the needs of occupant P and user U, who are not related as family or friends, and who want to check the driving feel of the mobile vehicle M or see a desired city landscape. In this case, when the matching processing unit 320 of the management server 300 performs the matching process, such restrictions are set, and the first control device 170 or the second control device 270 masks the angle range that is not to be seen, or corrects the direction of direction φ so that it does not point in the restricted direction, according to the setting. Furthermore, since information regarding such restrictions concerns the privacy of occupant P, it may also be set on the first device 100 side.

[0083] <Summary> The information processing system 1 configured as described above can enhance the sense of presence given to both the occupant P of the mobile vehicle M and the user U who is in a different location from the mobile vehicle M. The user U is shown an image corresponding to their direction of orientation φ as seen from the passenger seat, and the user U can remotely control the movement of the mobile vehicle M by operating the control area displayed on the image with gestures. Furthermore, the image representing the user U's gestures is similarly displayed on the HMI of the mobile vehicle M, allowing the occupant P to understand the user U's operations in real time. In addition, the display mode of the control area displayed on the image can be changed by the user U's gestures, thereby increasing convenience for the user U.

[0084] <Usage example> Information processing system 1 can be used in the following ways:

[0085] (A) A configuration in which the crew member P and user U are in a relationship such as family or friends, and the virtual drive is provided to user U. User U can converse with the crew member P about the scenery around the mobile vehicle M while viewing images.

[0086] (B) A configuration in which the crew member P is a general user and the user U is a provider of services such as directions and driving instruction. The user U can provide directions to places that are difficult to understand using a navigation device or that are not shown on a map, or provide instruction on driving operations, while observing the scenery around the mobile vehicle M.

[0087] (C) A configuration in which a crew member P is a celebrity and a user U is a regular user, and a commercially based virtual drive is provided to user U. In this case, multiple users U can be associated with one crew member P simultaneously, and for example, audio transmission from user U may be set to off.

[0088] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0089] 1. Information Processing System 100 1st device 110 First communication device 120 First microphone 130 Camera Unit 132 Indoor Cameras 134 Outdoor Camera 140 First speaker 150 User display device 160 HMI 170 First control device 190 Controlled Equipment 200 2nd device 210 Second communication device 220 Second microphone 230 detection device 232 Directional detection device 234 Head position detection device 236 Motion Sensors 240 Second speaker 250 Mobile Image Display Device 260 HMI 270 Second control device 300 Management Servers M Mobile object

Claims

1. The first device is mounted on the mobile vehicle on which the crew is riding, One or more second devices used by one or more users at a location different from the aforementioned mobile body, Management server and An information processing system comprising, The first apparatus is A first communication device that communicates with the second communication device of one or more of the aforementioned second devices, A camera unit having one or more cameras, including at least an indoor camera capable of imaging the inside of the moving body, provided on the moving body, It has, The one or more second devices mentioned above are: The second communication device communicates with the first communication device, A detection device for detecting the user's direction of orientation and gestures, A display device that displays an image from among the images captured by the camera unit that corresponds to the user's direction of orientation, It has, The management server has a matching processing unit that matches the first device with the one or more second devices in response to a matching request received from the first device or the one or more second devices. If the matching described above is performed, the one or more second devices determine, based on the detected direction of orientation and the gesture, the operating part of the moving object that the user is pointing at from the image, and the content of the operation to be performed on the operating part. The second communication device transmits the determined operation content to the operation unit to the first communication device. Information processing system.

2. The display device changes the position of the operation unit and displays according to the gesture. The information processing system according to claim 1.

3. The display device changes the size of the operation section in response to the gesture. The information processing system according to claim 1.

4. The display device highlights the operation unit that the user can operate from among the plurality of operation units. The information processing system according to claim 1.

5. The second device determines, based on the user's permission to operate the operating unit of the mobile unit, which of the images the user is permitted to operate. The information processing system according to claim 1.

6. The first device further includes a display that shows an image indicating the operating unit pointed to by the user. The information processing system according to claim 1.

7. The second communication device transmits the information of the direction of direction to the first communication device. The first device further includes a first control device that controls the first communication device to selectively transmit to the second communication device images corresponding to the directional direction, from among the images captured by the camera unit, via the first communication device. The display device of the second device displays an image acquired via the second communication device that corresponds to the direction of direction as viewed from a predetermined seat. The information processing system according to claim 1.

8. The first communication device transmits the image captured by the camera unit to the second communication device. The second device further includes a second control device that selectively displays on the display device an image corresponding to the direction of direction from among the images captured by the camera unit. The information processing system according to claim 1.

9. The aforementioned display device is a display device for VR (Virtual Reality) goggles, The detection device includes a physical sensor attached to the VR goggles. The information processing system according to claim 1.

10. The aforementioned moving object is a vehicle. The aforementioned designated seat is the passenger seat. The information processing system according to claim 7.

11. The first device is mounted on the mobile vehicle on which the crew is riding, A second device used by a user at a location different from the aforementioned mobile body, An information processing system comprising, The first apparatus is A first communication device that communicates with the second communication device of the second device, A camera unit having one or more cameras, including at least an indoor camera capable of imaging the inside of the moving body, provided on the moving body, It has, The second device is The second communication device communicates with the first communication device, A detection device for detecting the user's direction of orientation and gestures, A display device that displays an image from among the images captured by the camera unit that corresponds to the user's direction of orientation, It has, Based on the detected direction of orientation and the gesture, the second device determines the operating part of the moving object that the user is pointing at from the image, and the content of the operation to be performed on the operating part. The second communication device transmits the determined operation content to the operation unit to the first communication device. The second device identifies the operation rights set for each user by referring to a table that stores the operation rights set for each user to the operation unit of the mobile body, and determines which operation unit of the mobile body in the image the user is permitted to operate, based on the operation rights set for the user to the operation unit of the mobile body. Information processing system.

12. The first device is mounted on the mobile vehicle on which the crew is riding, A second device used by a user at a location different from the aforementioned mobile body, An information processing system comprising, The first apparatus is A first communication device that communicates with the second communication device of the second device, A camera unit having one or more cameras, including at least an indoor camera capable of imaging the inside of the moving body, provided on the moving body, It has, The second device is The second communication device communicates with the first communication device, A detection device for detecting the user's direction of orientation and gestures, A display device that displays an image from among the images captured by the camera unit that corresponds to the user's direction of orientation, It has, Based on the detected direction of orientation and the gesture, the second device determines the operating part of the moving object that the user is pointing at from the image, and the content of the operation to be performed on the operating part. The second communication device transmits the determined operation content to the operation unit to the first communication device. The first device further includes a first control device that controls the first communication device to selectively transmit to the second communication device images corresponding to the directional direction, from among the images captured by the camera unit, via the first communication device. The display device of the second device displays an image corresponding to the direction of direction, which is acquired via the second communication device. The direction in which the image is displayed is limited according to the settings of the first device. Information processing system.

Images