Information processing system

The information processing system addresses the lack of presence by enabling remote control and gesture-based interaction between occupants and users of a moving object, enhancing engagement and convenience through integrated communication and display technologies.

JP2024161224AActive Publication Date: 2024-11-15HONDA MOTOR CO LTD
View PDF 18 Cites 0 Cited by

Patent Information

Application Number
JP2024152505
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-11-15
Estimated Expiration
2042-09-12

AI Technical Summary

Technical Problem

Conventional systems fail to adequately enhance the sense of presence for both occupants of a moving object and users at a different location, leading to a lack of engagement and interaction.

Method used

An information processing system comprising a first device mounted on a moving body and a second device used by a user at a different location, utilizing communication devices, cameras, detection devices, and display devices to capture and display images, detect user gestures and orientation, and allow remote operation of the moving object's controls.

Benefits of technology

Enhances the sense of presence and interaction by allowing users to remotely control the moving object through gestures and display user gestures, improving engagement and convenience for both occupants and users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024161224000001_ABST
    Figure 2024161224000001_ABST
Patent Text Reader

Abstract

To provide an information processing system capable of increasing presence to be imparted to both crew members of a moving body and users in a different place from the moving body.SOLUTION: The present invention relates to an information processing system that comprises a first device mounted on a moving body that a crew member rides on, and a second device used by a user in a different place from the moving body, where the first device has a first communication device which communicates with a second communication device of the second device and a camera unit provided on the moving body, and the second device has a second communication device which communicates with the first communication device, a detection device for detecting the user, and a display device which displays an image that the camera unit captures.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing system. [Background technology]

[0002] Conventionally, research has been conducted into communication between a device mounted on a moving object such as a vehicle and a device used in a location other than the moving object, and sharing images of the scenery outside the vehicle, etc. (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-94958 A Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, there are cases where the sense of realism is not felt sufficiently by both the passengers of the vehicle and users in locations other than the vehicle.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objectives is to provide an information processing system that can enhance the sense of realism given to both the occupants of a moving body and users who are in a location other than the moving body. [Means for solving the problem]

[0006] The information processing system according to the present invention employs the following configuration. (1): An information processing system according to one embodiment of the present invention is an information processing system comprising a first device mounted on a moving body in which an occupant rides, and a second device used by a user at a location different from the moving body, wherein the first device has a first communication device that communicates with a second communication device of the second device, and a camera unit provided on the moving body, and the second device has the second communication device that communicates with the first communication device, a detection device for detecting the user, and a display device that displays an image captured by the camera unit.

[0007] (2): An information processing system according to one embodiment of the present invention includes a first device mounted on a moving body in which an occupant rides, and a second device used by a user at a location different from the moving body, wherein the first device includes a first communication device that communicates with a second communication device of the second device, and a camera unit that is provided on the moving body and has one or more cameras including at least an indoor camera capable of capturing images of the interior of the moving body, and the second device includes the second communication device that communicates with the first communication device, a detection device for detecting the orientation direction and gesture of the user, and a display device that displays an image that corresponds to the orientation direction of the user among the images captured by the camera unit, and the second device determines, based on the detected orientation direction and gesture, an operation unit of the moving body toward which the user is pointing and the operation content for the operation unit, and the second communication device transmits the determined operation content for the operation unit to the first communication device.

[0008] (3): An information processing system according to one embodiment of the present invention includes a first device mounted on a moving body in which a passenger rides, and a second device used by a user at a location different from the moving body, wherein the first device includes a first communication device that communicates with a second communication device of the second device, and a camera unit that is provided on a predetermined seat of the moving body and has one or more cameras including at least an indoor camera that can capture images of the interior of the moving body as seen from the predetermined seat, and the second device includes the second communication device that communicates with the first communication device, a detection device for detecting a direction of orientation or a gesture of the user, and a display device that displays an image, among images captured by the camera unit, that corresponds to the direction of orientation as seen from the predetermined seat, and the second device determines, based on the detected direction of orientation or gesture, an operation unit of the moving body toward which the user is pointing and an operation content for the operation unit, and the second communication device transmits the determined operation content for the operation unit to the first communication device.

[0009] (4) In the above aspect (2) or (3), the display device changes and displays the position of the operation unit in response to the gesture.

[0010] (5) In the above aspect (2) or (3), the display device changes the size of the operation unit in response to the gesture.

[0011] (6) In the above aspect (2) or (3), the display device highlights an operation part that can be operated by the user, among the plurality of operation parts.

[0012] (7): In the above aspect (2) or (3), the second device determines, based on the operation authority for the operation unit of the moving object set for the user, which operation unit of the moving object is permitted to be operated by the user among the images.

[0013] (8) In the above aspect (2) or (3), the first device further includes a display that displays an image showing the operation unit pointed by the user.

[0014] (9): In the above aspect (2) or (3), the second communication device transmits information on the orientation direction to the first communication device, and the first device further has a first control device that controls the first communication device to selectively transmit to the second communication device an image, among images captured by the camera unit, that corresponds to the orientation direction obtained via the first communication device, and the display device of the second device displays an image, obtained via the second communication device, that corresponds to the orientation direction as seen from the specified seat.

[0015] (10): In the above aspect (2) or (3), the first communication device transmits an image captured by the camera unit to the second communication device, and the second device further has a second control device that selectively displays, on the display device, an image from among the images captured by the camera unit that corresponds to the orientation direction.

[0016] (11): In the above aspect (2) or (3), the display device is a display device of VR (Virtual Reality) goggles, and the detection device includes a physical sensor attached to the VR goggles.

[0017] (12) In the aspect of (9) above, the moving body is a vehicle, and the specified seat is a passenger seat. Effect of the Invention

[0018] According to aspects (1) to (12), the sense of realism given to both the passengers of the moving body and the user who is in a location other than the moving body can be enhanced. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram showing the usage environment of an information processing system 1 and a management server 300. [Diagram 2] FIG. 2 is a diagram showing an example of the contents of user data 360. [Diagram 3] FIG. 2 is a configuration diagram of a first device 100. [Figure 4] 2 is a diagram showing an example of the arrangement of a part of a first device 100 in a moving object M. FIG. [Diagram 5] FIG. 2 is a diagram illustrating the configuration of a second device 200. [Figure 6] FIG. 11 is a diagram for explaining an image corresponding to a pointing direction. [Figure 7] 2 is a diagram illustrating a first example of a functional configuration of a first control device 170 and a second control device 270. FIG. [Figure 8] 13 is a diagram illustrating a second example of the functional configuration of the first control device 170 and the second control device 270. FIG. [Figure 9] FIG. 13 is a diagram showing an example of an image corresponding to a pointing direction. [Figure 10] 13 is a diagram showing an example of the contents of user data 360C that defines the operating authority of a moving object M. FIG. [Figure 11] 13 is a diagram showing an example of an image showing an operation area for which a user U has operation authority. FIG. [Figure 12] 11 is a diagram showing an example of a method for superimposing an instruction image H representing a gesture of a user U on an operation area. FIG. [Figure 13] 13 is a diagram showing an example of a method for changing the display mode of an operation area displayed to a user U in response to a gesture of the user U. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an embodiment of the information processing system of the present invention will be described with reference to the drawings. The information processing system includes a first device mounted on a moving body on which a passenger boards, and a second device used by a user at a location different from the moving body. The moving body is, for example, a vehicle, but may be any moving body that a passenger can board. In addition, the passenger is usually the driver of the moving body, but may be a passenger other than the driver.

[0021] Between the first device and the second device, the voice collected by the microphone is transmitted to the other party and played back by the speaker, creating a state similar to a telephone conversation, and further, a part of an image captured by the camera unit of the first device is displayed by the second device, providing MR (Mixed Reality) on the second device side. The first device and the second device do not need to be in a one-to-one relationship, and one of the multiple first devices and multiple second devices may be matched in a one-to-many relationship to operate as an information processing system. In the latter case, for example, one passenger can communicate with multiple users simultaneously or in sequence.

[0022] <Basic configuration> 1 is a diagram showing the usage environment of an information processing system 1 and a management server 300. The information processing system 1 includes a first device (mobile device) 100 mounted on a mobile body M, and a second device (user device) 200 used by a user U at a location different from the mobile body M (a location that happens to be close is not excluded). The first device 100, the second device 200, and the management server 300 communicate with each other via a network NW. The information processing system 1 may or may not include the management server 300.

[0023] The management server 300 includes, for example, a communication device 310, a matching processing unit 320, and a storage unit 350. The storage unit 350 stores user data 360.

[0024] Communication device 310 is a communication interface for connecting to network NW. Note that communication between communication device 310 and first device 100, and communication between communication device 310 and second device 200 are performed in accordance with, for example, TCP / IP (Transmission Control Protocol / Internet Protocol).

[0025] The matching processing unit 320 is realized by a processor such as a CPU (Central Processing Unit) executing a program (a group of instructions) stored in a storage medium. The storage unit 350 is a RAM (Random Access Memory), a HDD (Hard Disk Drive), a flash memory, or the like.

[0026] Fig. 2 is a diagram showing an example of the contents of the user data 360. The user data 360 includes, for example, an occupant list 360A in which an occupant ID, which is identification information of an occupant P, its communication identification information (IP address, etc.), and a user ID, which is identification information of a user U to be matched, are associated with each other, and a user list 360B in which a user ID, its communication identification information (IP address, etc.), and an occupant P to be matched are associated with each other. The user data 360 may be generated in any manner, not limited to the manner shown in Fig. 2, as long as it includes this information.

[0027] When the communication device 310 receives a matching request from the user U via the second device 200 or from the occupant P via the first device 100, the matching processing unit 320 performs matching between the matching user U and the occupant P by referring to the user data 360, and transmits the communication identification information of the first device 100 of the occupant P to the second device 200 of the matched user U, and transmits the communication identification information of the second device 200 of the user U to the first device 100 of the matched occupant P, using the communication device 310. Between the first device 100 and the second device 200 that receive these, communication with higher real-time properties based on, for example, UDP (User Datagram Protocol) is performed.

[0028] 3 is a configuration diagram of the first device 100. The first device 100 includes, for example, a first communication device 110, a first microphone 120, a camera unit 130, a first speaker 140, a user display device 150, an HMI (Human machine Interface) 160, and a first control device 170. The first control device 170 is connected to a control target device 190 mounted on a moving object M.

[0029] The first communication device 110 is a communication interface for communicating with each of the communication device 310 of the management server 300 and the second communication device 210 of the second device 200 via the network NW.

[0030] The first microphone 120 collects at least the voice uttered by the occupant P. The first microphone 120 may be provided inside the interior of the moving body M and may have a sensitivity capable of collecting voice outside the moving body M, or may include a microphone provided inside the interior of the moving body M and a microphone provided outside the moving body M. The voice collected by the first microphone 120 is transmitted to the second communication device 210 by the first communication device 110 via the first control device 170, for example.

[0031] The camera unit 130 includes at least an indoor camera 132, and may include an outdoor camera 134. The first speaker 140 outputs a voice uttered by the user U, which is acquired via the first communication device 110. Details of the arrangement of the camera unit 130 and the first speaker 140 will be described later with reference to FIG.

[0032] The user display device 150 virtually displays the user U as if the user U were present inside the room of the moving object M. For example, the user display device 150 makes a hologram appear, or displays the user U on a part of the moving object M that corresponds to a mirror or window.

[0033] The HMI 160 is a touch panel, a voice response device (agent device), etc. The HMI 160 receives various instructions from the occupant P to the first device 100.

[0034] First control device 170 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (a set of instructions), and controls each part of first device 100 by the processor executing the set of instructions.

[0035] The control target device 190 is, for example, a navigation device or a driving support device mounted on the moving object M.

[0036] FIG. 4 is a diagram showing an example of the arrangement of a part of the first device 100 in the moving object M. The indoor camera 132 is attached to the neck pillow of the passenger seat S2 (an example of a "predetermined seat") via an attachment 132A, for example, and is provided at a position slightly separated from the backrest of the passenger seat S2 in the traveling direction of the moving object M. The indoor camera 132 has a wide-angle lens and can capture an image of a range represented by a hatched area 132B in the figure. The indoor camera 132 can capture not only the interior of the moving object M, but also the outside through the window. In the following description, the passenger seat S2 is considered to be the predetermined seat, but the predetermined seat may be another seat such as a rear seat.

[0037] The outdoor camera 134 includes, for example, a plurality of child outdoor cameras 134-1 to 134-4. By combining images captured by the plurality of child outdoor cameras 134-1 to 134-4, an image such as a panoramic image capturing the outside of the moving object M can be obtained. Instead of (or in addition to) these, the outdoor camera 134 may include a wide-angle camera provided on the roof of the moving object M. Note that a camera capable of capturing an image behind the passenger seat S2 may be added as the indoor camera 132, and the moving object image described later may be generated as a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 by the first control device 170, or may be generated as a 360-degree panoramic image by appropriately combining images captured by the indoor camera 132 and the outdoor camera 134.

[0038] The first speaker 140 outputs the voice of the user U acquired via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, the sub-first speaker 140-1 is disposed in the center of the instrument panel, the sub-first speaker 140-2 in the left end of the instrument panel, the sub-first speaker 140-3 in the right end of the instrument panel, the sub-first speaker 140-4 in the lower part of the left door, and the sub-first speaker 140-5 in the lower part of the right door. When the first control device 170 outputs the voice of the user U to the first speaker 140, for example, the first control device 170 outputs the voice from the first child speaker 140-2 and the first child speaker 140-4 at approximately the same volume and turns off the other first child speakers, thereby localizing the sound image so that the voice can be heard from the passenger seat S2 by the occupant P seated in the driver's seat S1. In addition, the method of localizing the sound image is not limited to adjusting the volume, and may be performed by shifting the phase of the sound output from each of the first child speakers. For example, when localizing the sound image so that the sound is heard from the left side, the timing of outputting the sound from the first child speaker on the left side may be slightly earlier than the timing of outputting the same sound from the first child speaker on the right side.

[0039] Furthermore, when the first control device 170 outputs the voice of the user U to the first speaker 140, the first control device 170 may localize a sound image so that the voice is heard by the occupant P from a position at a height corresponding to the height of the head of the user U sitting on the passenger seat S2, and output the voice emitted by the user U from the first speaker 140. In this case, the first speaker 140 needs to have multiple child first speakers 140-k (k is a multiple natural number) at different heights.

[0040] 5 is a configuration diagram of the second device 200. The second device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a mobile image display device 250, an HMI 260, and a second control device 270. The detection device 230 includes, for example, an orientation direction detection device 232, a head position detection device 234, and a motion sensor 236.

[0041] The second communication device 210 is a communication interface for communicating with each of the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0042] The second microphone 220 collects the voice uttered by the user U. The voice collected by the second microphone 220 is transmitted by the second communication device 210 to the first communication device 110 via the second control device 270, for example.

[0043] The orientation direction detection device 232 is a device for detecting the orientation direction. The orientation direction is a direction based on the face direction or the line of sight direction of the user U, or both of them. Alternatively, it may be a direction indicated by the movement of the arm or finger, such as the action of tilting the terminal device used by the user U or the action of swiping the screen. In the following, the orientation direction is assumed to be an angle in a horizontal plane, that is, an angle that does not have a vertical component, but the orientation direction may also be an angle that includes a vertical component. The orientation direction detection device 232 may include a physical sensor (e.g., an acceleration sensor, a gyro sensor, etc.) attached to the VR goggles described later, or may be an infrared sensor that detects multiple positions of the head of the user U, or a camera that captures the head of the user U. In either case, the second control device 270 calculates the orientation direction based on the information input from the orientation direction detection device 232. Since various technologies for this are publicly known, detailed explanations will be omitted.

[0044] The head position detection device 234 is a device for detecting the position (height) of the head of the user U. For example, one or more infrared sensors or optical sensors provided around the chair on which the user U sits are used as the head position detection device 234. In this case, the second control device 270 detects the position of the head of the user U based on the presence or absence of a detection signal by one or more infrared sensors or optical sensors. The head position detection device 234 may also be an acceleration sensor attached to the VR goggles. In this case, the second control device 270 detects the position of the head of the user U by integrating the output of the acceleration sensor minus the gravitational acceleration. The head position information thus acquired is provided to the second control device 270 as height information. The user's head position may be acquired based on the user U's operation on the HMI 260. For example, the user U may input his / her height in numbers into the HMI 260, or may input his / her height using a dial switch included in the HMI 260. In these cases, the head position, i.e., height information, is calculated from the height. Furthermore, the user U may input a discrete value such as physique: large / medium / small to the HMI 260 instead of a continuous value. In this case, height information is acquired based on information indicating the physique. Furthermore, the head height of the user U may be acquired simply based on the physique of a general adult (which may be gender-specific) without acquiring the head height of the user in particular.

[0045] The motion sensor 236 is a device for recognizing a gesture operation performed by the user U. For example, a camera capturing an image of the upper body of the user U is used as the motion sensor 236. In this case, the second control device extracts feature points (fingertips, wrists, elbows, etc.) of the user U's body from the image captured by the camera, and recognizes the gesture operation of the user U based on the movement of the feature points.

[0046] The second speaker 240 outputs the voice uttered by the occupant P acquired via the second communication device 210. The second speaker 240 has a function of changing the direction from which the voice is heard, for example. The second control device 270 causes the second speaker to output the voice so that the user U can hear the voice from the position of the occupant P as seen from the passenger seat S2. The second speaker 240 includes a plurality of child second speakers 240-n (n is a plurality of natural numbers), and the second control device 270 may adjust the volume of each child second speaker 240-n to perform sound image localization, or if headphones are attached to the VR goggles, the function of the headphones may be used to perform sound image localization.

[0047] The moving body image display device 250 displays an image corresponding to the orientation direction as seen from the passenger seat among the images captured by the camera unit 130 (which may be images that have been subjected to the above-mentioned combining process, and are hereinafter referred to as moving body images). FIG. 6 is a diagram for explaining an image corresponding to the orientation direction. In the example of this figure, the VR goggles 255 includes an orientation direction detection device 232, a physical sensor as a head position detection device 234, and the moving body image display device 250. The second control device 270 detects the direction in which the VR goggles 255 faces as the orientation direction φ, with a pre-calibrated direction as the reference direction. Various methods for this function are already known, so detailed explanations will be omitted.

[0048] The moving body image display device 250 displays, toward the user U, an image A2 within an angle range of plus or minus α centered on the directional direction φ, from the moving body image A1 (which has an angle of about 240 degrees in the figure, but the angle of view may be expanded by combining processing as described above).

[0049] The HMI 260 is a touch panel, a voice response device (agent device), the above-mentioned switch, etc. The HMI 260 receives various instructions from the occupant P to the second device 200.

[0050] The second control device 270 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (a group of instructions), and the processor controls each part of the second device 200 by executing the group of instructions.

[0051] <Functional configuration> The functional configurations of the first control device 170 and the second control device 270 will be described below. [Example 1] FIG. 7 is a diagram showing a first example of the functional configuration of the first control device 170 and the second control device 270. In the first example, the first control device 170 includes a matching request / acceptance unit 171, an audio output control unit 172, an image transmission unit 173, and an on-board device cooperation unit 174. The second control device 270 includes a matching request / acceptance unit 271, an audio output control unit 272, an orientation direction detection unit 273, a head position detection unit 274, a gesture input detection unit 275, an image editing unit 276, and a moving object image display control unit 277. These functional units are realized, for example, by a processor such as a CPU executing a program (a group of instructions). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware.

[0052] The matching request / acceptance unit 171 accepts an input of a matching request from the occupant P using the HMI 160 and transmits it to the management server 300, and also accepts an input of acceptance of a matching request received from the management server 300 using the HMI 160 and transmits it to the management server 300. The matching request / acceptance unit 171 controls the first communication device 110 to set the second device 200 of the user U with whom a match has been established as the communication partner.

[0053] The audio output control unit 172 controls the first speaker 140 as described above.

[0054] The image transmission unit 173 uses the first communication device 110 to transmit the moving object image A1 to the second device 200.

[0055] The on-board device cooperation unit 174 controls the control target device 190 based on the instruction signal input from the second device 200.

[0056] The matching request / acceptance unit 271 accepts an input of a matching request from the user U using the HMI 260 and transmits it to the management server 300, and also accepts an input of acceptance of a matching request received from the management server 300 using the HMI 260 and transmits it to the management server 300. The matching request / acceptance unit 271 controls the second communication device 210 to communicate with the first device 100 of the occupant P with whom a match has been established.

[0057] The audio output control unit 272 controls the second speaker 240 as described above.

[0058] The orientation direction detection unit 273 detects the orientation direction φ based on the output of the orientation direction detection device 232. The head position detection unit 274 detects the height of the head of the user U based on the output of the head position detection device 234. The head position may be represented as three-dimensional coordinates, or the head height may simply be detected as the head position. The gesture input detection unit 275 detects a gesture input by the user U based on the output of the motion sensor 236.

[0059] The image editing unit 276 performs processing to cut out an image A2 corresponding to the directional direction φ as seen from the passenger seat from the moving object image A1 (FIG. 6). The moving object image display control unit 277 causes the moving object image display device 250 to display the image A2 cut out by the image editing unit 276. At this time, the image editing unit 276 may cause the moving object image display device 250 to display an image corresponding to the directional direction φ as seen from the height indicated by the height information of the head of the user U.

[0060] [Example 2] Fig. 8 is a diagram showing a second example of the functional configuration of the first control device 170 and the second control device 270. Compared to the first example in Fig. 7, it differs in that the first control device 170 includes an image editing unit 175, and the second control device 270 does not include an image editing unit 276 but includes a directivity direction transmitting unit 278. The other components basically have the same functions as in the first example, so repeated explanations will be omitted.

[0061] Directivity direction transmitter 278 transmits directivity direction φ detected by directivity direction detector 273 to first device 100 using second communication device 210.

[0062] The image editing unit 175 performs a process of cutting out an image A2 corresponding to the directional direction φ (transmitted from the second device 200) seen from the passenger seat from the moving object image A1 (FIG. 6). At this time, the image editing unit 175 may obtain head height information of the user U from the second device 200 and perform a process of cutting out an image A2 corresponding to the directional direction φ seen from the height indicated by the height information.

[0063] The image transmission unit 173 in the second example transmits the image A2 cut out by the image editing unit 175 to the second device 200 using the first communication device 110. Then, the moving object image display control unit 277 causes the moving object image display device 250 to display the image A2 transmitted from the first device 100.

[0064] <Gesture control> FIG. 9 is a diagram showing an example of an image corresponding to a directional direction. The image shown in FIG. 9 is an image (hereinafter, sometimes referred to as a "directional image") displayed by the mobile body image display device 250 to the user U, which corresponds to the directional direction seen from the passenger seat among the images captured by the camera unit 130. In the directional image of FIG. 9, the operation area OA is an operation area displayed on the HMI 160 of the mobile body M displayed to the user U as it is (or simplified), and enables the user U to remotely operate the HMI 160. As will be described in detail below, the content of the operation performed by the user U on the operation area OA is transmitted to the first device 100 by the second communication device 210, and the first device 100 causes the mobile body M to execute the operation content. As shown in FIG. 9, the operation area OA includes a plurality of operation areas such as operation areas OA1 to OA6.

[0065] First, the second device 200 identifies an operation area corresponding to the directional direction φ of the user U detected by the directional direction detection unit 273 among the operation areas OA1 to OA6 of the directional image. For example, when the user U makes a gesture of pointing his / her finger toward the operation area OA1 indicating "surrounding area search", the mobile body image display device 250 superimposes store / facility information present within a predetermined distance from the mobile body M on the directional image and displays it, for example, as a map. At the same time, the second communication device 210 transmits the operation information to the first device 100, and the first device 100 displays the store / facility information present within a predetermined distance from the mobile body M on the HMI 160 of the mobile body M in response to receiving the operation information. FIG. 9 shows a scene in which the store / facility information in the vicinity of the mobile body M is displayed as a map as a result of the user U making a gesture to "surrounding area search". This allows the occupant P to check the information around the mobile body M without operating the HMI 160 himself, even while driving the mobile body M.

[0066] Furthermore, in the scene shown in FIG. 9, the second device 200 identifies a point on the map that corresponds to the orientation direction φ of the user U detected by the orientation direction detection unit 273. For example, in the case of FIG. 9, points P1 to P3 are displayed on the map, and the orientation direction detection unit 273 identifies the point P1 as the point that corresponds to the orientation direction φ of the user U. In a state in which the point P1 is identified, if the gesture input detection unit 275 detects a gesture of the user U operating the point P1, the second device 200 executes an operation corresponding to the point P1 that corresponds to the detected gesture. For example, in the case of FIG. 9, if the user U makes a gesture of pointing a finger toward the point P1, the mobile image display device 250 displays detailed information (e.g., menu information) of the store corresponding to the point P1 by superimposing it on the directional image.

[0067] Also, for example, when the user U makes a gesture of pointing his / her finger toward the operation area OA2 indicating "audio", the audio output control unit 272 causes the second speaker 240 to output audio. At the same time, the second communication device 210 transmits the operation information to the first device 100, and the first device 100 causes the audio device of the moving object M to play the operation information in response to receiving the operation information. For example, the user U operates the operation area OA2 by a gesture to select a song to be played, and the audio output control unit 272 causes the second speaker 240 to play the selected song. At the same time, the second communication device 210 transmits information indicating the selected song to the first device 100, and the first device 100 causes the audio device of the moving object M to play the song. This allows the occupant P to listen to music without operating the HMI 160 himself / herself, even while driving the moving object M.

[0068] Also, for example, when the user U makes a gesture of pointing his / her finger toward the operation area OA3 indicating "settings", the mobile object image display device 250 displays a screen related to changing the settings of the mobile object M superimposed on the directional image. The user U changes the settings of the mobile object M by making a further gesture on the screen related to changing the settings of the mobile object M. At the same time, the second communication device 210 transmits the operation information to the first device 100, and the first device 100 changes the settings of the mobile object M in response to receiving the operation information. For example, the user U changes the air conditioning settings of the mobile object M by operating the operation area OA3 with a gesture, and the second communication device 210 transmits information indicating the changed air conditioning settings to the first device 100. The first device 100 changes the air conditioning settings of the mobile object M based on the information indicating the changed air conditioning settings. This allows the occupant P to change the air conditioning settings of the mobile object M without operating the HMI 160 himself / herself, even while driving the mobile object M.

[0069] Also, for example, when the user U makes a gesture of pointing his / her finger toward the operation area OA4 indicating "NAVI", the mobile body image display device 250 displays a screen related to changing the navigation setting of the mobile body M superimposed on the directional image. The user U changes the navigation setting of the mobile body M by making a further gesture on the screen related to changing the navigation setting of the mobile body M. At the same time, the second communication device 210 transmits the operation information to the first device 100, and the first device 100 changes the navigation setting of the mobile body M in response to receiving the operation information. For example, the user U operates the operation area OA4 by a gesture to change the destination of the mobile body M, and the second communication device 210 transmits information indicating the changed destination to the first device 100. The first device 100 changes the destination of the mobile body M based on the information indicating the changed destination. This allows the occupant P to change the destination of the mobile body M without operating the HMI 160 himself / herself, even while driving the mobile body M.

[0070] Also, for example, when the user U makes a gesture of pointing his / her finger toward the operation area OA5 indicating the hazard lamps, the second communication device 210 transmits the operation information to the first device 100, and the first device 100, in response to receiving the operation information, blinks the hazard lamps of the moving object M. As a result, even while driving the moving object M, the occupant P can blink the hazard lamps to alert surrounding vehicles of the moving object M without operating the HMI 160.

[0071] The above operation areas OA1 to OA5 are merely examples, and an operation area relating to any control of the moving object M may be displayed on the moving object image display device 250 and operated by the user U through gestures. For example, the operation area may include an operation area for opening and closing the windows of the moving object M, an operation area for operating the wipers of the moving object M, an operation area for selecting a store menu displayed by selecting a location and making a payment, and the like.

[0072] <Operation Permissions> In this manner, in this embodiment, the moving body image display device 250 displays an operation area for the user U to remotely operate the moving body M in a superimposed manner on the image, but it may not be preferable to have the user U operate all operation areas for all combinations of the occupant P and the user U. More specifically, when the occupant P and the user U have a close relationship, it is preferable to have the user U operate more operation areas, whereas when the occupant P and the user U have a distant relationship, it is preferable to have the user U operate fewer operation areas. Therefore, in this embodiment, the management server 300 stores in advance in the storage unit 350 user data 360C that defines the range of the operation area (i.e., operation authority) of the user U for the combination of the occupant P and the user U, and when the occupant P and the user U are matched, the management server 300 refers to the user data 360C and has the user U operate the operation area within the range of the operation authority.

[0073] FIG. 10 is a diagram showing an example of the contents of user data 360C that defines the operation authority of the moving body M. For example, the user data 360C is obtained by associating the operation authority of the user corresponding to the user ID with a combination of an occupant ID and a user ID. For example, the occupant may set the operation authority to be granted to each user in advance in the user data 360C. In this case, it may be troublesome for the occupant to set the operation authority for each user. Therefore, for example, as shown in the second and third lines of the user data 360C, the occupant may uniformly set the operation authority to be granted to all users (i.e., "ALL") and may set the operation authority individually for the user to whom the occupant wishes to grant more operation authority. Conversely, the occupant may uniformly set the operation authority to be granted to all users and may set the operation authority individually for the user to whom the occupant wishes to grant less operation authority.

[0074] Fig. 11 is a diagram showing an example of an image showing operation areas for which a user U has operation authority. The image shown in Fig. 11 is displayed, for example, when the user U selects point P1 on the image shown in Fig. 9 by a gesture. In Fig. 11, the newly displayed operation area OA includes operation areas OA1 to OA5 and an operation area OA6 for selecting a menu.

[0075] The mobile object image display device 250 may improve the convenience of the user U by explicitly displaying only the operation area for which the user U has been granted operation authority among multiple operation areas. For example, as shown in FIG. 11, when the user U only has the authority to operate an audio device of the mobile object M, the mobile object image display device 250 highlights only the operation area OA2 corresponding to the operation of the audio device. Also, for example, the mobile object image display device 250 may display only the operation area OA2 corresponding to the operation of the audio device and hide the other operation areas. This improves the convenience of the user U who selects and operates an operation area.

[0076] <Gesture display> Furthermore, the mobile image display device 250 may superimpose an instruction image H representing a gesture of the user U detected by the gesture input detection unit 275 on the operation area. Fig. 12 is a diagram showing an example of a method for superimposing an instruction image H representing a gesture of the user U on the operation area. Fig. 12 shows a situation in which the user U selects one of the displayed menu items in the image shown in Fig. 11.

[0077] In this case, the mobile body image display device 250 superimposes an instruction image H representing the gesture of the user U on the menu item selected by the user U through a gesture, and the second communication device 210 transmits instruction information indicating that the user U has selected the menu item to the first device 100. In response to receiving the instruction information, the first device 100 similarly displays an instruction image H representing the gesture of the user U on the HMI 160. This allows the occupant P to grasp the operation by the user U in real time. In this embodiment, the mobile body image display device 250 displays a hand icon as the instruction image H representing the gesture of the user U, but the present invention is not limited to such a configuration. For example, the mobile body image display device 250 may display a shadow representing the assumed position of the user U's hand, or more generally, it is sufficient to display the operation area to which the user U is pointing in a visible form among the operation areas. The first device 100 may have a mode in which an image displayed on the mobile body image display device 250 (an image the user U is looking at) is displayed on the HMI 160.

[0078] Furthermore, the mobile body image display device 250 may change the display mode of the operation area displayed to the user U in response to a gesture of the user U detected by the gesture input detection unit 275. Fig. 13 is a diagram showing an example of a method for changing the display mode of the operation area displayed to the user U in response to a gesture of the user U. Fig. 13 shows, as an example, a situation in which the user U makes a gesture of grabbing and moving the operation area OA displayed by the mobile body image display device 250.

[0079] For example, when the gesture input detection unit 275 detects that the user U has made a gesture of grabbing and moving the operation area OA, the mobile image display device 250 may deform the instruction image H representing the hand of the user U to display the operation area OA as if it is being grabbed. In this case, the mobile image display device 250 changes the position of the operation area OA in the direction in which the user U made the gesture and displays it. This allows the user U to change the position of the operation area OA in the image and visually recognize the image area that the user wants to focus on. In this case, it is also possible to restrict the movement of the operation area OA in the depth direction or the rotation of the screen of the operation area OA, and to change the position of the operation area OA while keeping the distance between the operation area OA and the user U constant. This prevents the operation area OA from swinging excessively, improving usability.

[0080] Also, for example, the mobile object image display device 250 may hide the operation area OA when the gesture input detection unit 275 detects that the user U has performed a gesture of grabbing the operation area OA and moving it at a predetermined acceleration or higher. In this case, for example, the mobile object image display device 250 fades out the operation area OA in the direction in which the user U performed the gesture. This allows the user U to hide the unnecessary operation area OA and view the image area that he or she wants to focus on.

[0081] Also, for example, when the gesture input detection unit 275 detects that the user U has performed a gesture of grabbing the operation area OA with both hands and widening (narrowing) it, the mobile image display device 250 may enlarge (reduce) the size of the operation area OA. This allows the user U to change the size of the operation area OA in the image and visually recognize the image area that the user wants to focus on.

[0082] <Other> In the information processing system 1, the user U can view any direction from the passenger seat S2. However, for example, a restriction may be set on the direction that the user U can view, depending on an agreement made at the time of matching. For example, the passenger P may request that the scenery in the traveling direction of the moving body M or the scenery on the opposite side of the driver's seat S1 may be provided, but that the passenger P does not want to see his / her own figure. This is intended to meet the needs of the passenger P and the user U, who are not family members or friends, and who want to check the driving feeling of the moving body M or view a desired cityscape. In this case, such a restriction is set when the matching processing unit 320 of the management server 300 performs matching processing, and the first control device 170 or the second control device 270 masks the angle range that is not visible or corrects the orientation direction φ so that it does not face the restricted direction according to the setting. In addition, information regarding such a restriction may be set on the first device 100 side, since it is related to the privacy of the passenger P.

[0083] <Summary> According to the information processing system 1 configured as above, it is possible to enhance the sense of realism given to both the occupant P of the moving body M and the user U who is in a location different from the moving body M. An image corresponding to the directional direction φ of the user U as seen from the passenger seat is displayed to the user U, and the user U can remotely control the operation of the moving body M by operating the operation area displayed on the image with a gesture. Furthermore, an image showing the gesture of the user U is similarly displayed on the HMI of the moving body M, and the occupant P can grasp the operation of the user U in real time. Furthermore, by making it possible to change the display mode of the operation area displayed on the image with the gesture of the user U, it is possible to enhance convenience for the user U.

[0084] <Usage example> The information processing system 1 can be used in the following manner.

[0085] (A) A mode in which the occupant P and the user U are in a relationship such as family or friends, and a virtual drive is provided to the user U. The user U can talk with the occupant P about the scenery around the moving body M while looking at the image.

[0086] (B) A mode in which the passenger P is a general user and the user U is a provider of route guidance services, driving instruction services, etc. The user U can provide route guidance or driving instruction in places that are difficult to understand using a navigation device or that are not on a map while viewing the scenery around the moving body M.

[0087] (C) A mode in which the passenger P is a celebrity and the user U is a general user, and a commercial virtual drive is provided to the user U. In this case, multiple users U are associated with one passenger P at the same time, and, for example, audio transfer from the user U side may be set to off.

[0088] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0089] 1. Information Processing Systems 100 1st device 110 First communication device 120 First Microphone 130 Camera unit 132 Indoor Camera 134 Outdoor Camera 140 1st Speaker 150 User display device 160 HMI 170 First control device 190 Controlled Equipment 200 2nd device 210 Second communication device 220 2nd microphone 230 Detection Device 232 Directional direction detection device 234 Head Position Detection Device 236 Motion Sensor 240 2nd Speaker 250 Mobile image display device 260 HMI 270 Second control device 300 Management Server M Mobile object

Claims

1. A first device mounted on a moving body in which a passenger boards; A second device used by a user at a location different from the mobile object; An information processing system comprising: The first device is a first communication device for communicating with a second communication device of the second device; A camera unit provided on the moving body; having The second device is the second communication device communicating with the first communication device; A detection device for detecting the user; A display device that displays an image captured by the camera unit. Information processing system.

2. A first device mounted on a moving body in which a passenger boards; A second device used by a user at a location different from the mobile object; An information processing system comprising: The first device is a first communication device for communicating with a second communication device of the second device; A camera unit provided on the moving body and having one or more cameras including at least an indoor camera capable of capturing an image of the inside of the moving body; having The second device is the second communication device communicating with the first communication device; A detection device for detecting a pointing direction and a gesture of the user; a display device that displays an image corresponding to the orientation direction of the user among the images captured by the camera unit; having The second device determines, based on the detected pointing direction and the gesture, an operation unit of the moving object to which the user is pointing, and an operation content for the operation unit, from among the image; The second communication device transmits the determined operation content to the operation unit to the first communication device. Information processing system.

3. A first device mounted on a moving body in which a passenger boards; A second device used by a user at a location different from the mobile object; An information processing system comprising: The first device is a first communication device for communicating with a second communication device of the second device; a camera unit provided on a predetermined seat of the moving body and having one or more cameras including at least an indoor camera capable of capturing an image of the inside of the moving body as seen from the predetermined seat; having The second device is the second communication device communicating with the first communication device; A detection device for detecting a pointing direction or a gesture of the user; a display device that displays an image corresponding to the orientation direction as seen from the predetermined seat among the images captured by the camera unit; having The second device determines, based on the detected pointing direction or the detected gesture, an operation unit of the moving object to which the user is pointing, and an operation content for the operation unit, from among the image; The second communication device transmits the determined operation content to the operation unit to the first communication device. Information processing system.

4. the display device changes and displays the position of the operation unit in response to the gesture.

4. The information processing system according to claim 2 or 3.

5. the display device changes and displays the size of the operation unit in response to the gesture.

4. The information processing system according to claim 2 or 3.

6. the display device highlights an operation unit that can be operated by the user among the plurality of operation units; 4. The information processing system according to claim 2 or 3.

7. the second device determines, based on an operation authority for the operation unit of the moving object set for the user, an operation unit of the moving object that is permitted to be operated by the user, among the images; 4. The information processing system according to claim 2 or 3.

8. The first device further includes a display that displays an image indicating the operation unit pointed by the user.

4. The information processing system according to claim 2 or 3.

9. The second communication device transmits information of the orientation direction to the first communication device; The first device further includes a first control device configured to selectively transmit to the second communication device an image corresponding to the orientation direction acquired via the first communication device, among images captured by the camera unit; a display device of the second device displays an image corresponding to the orientation direction as seen from a predetermined seat, the image being acquired via the second communication device; 4. The information processing system according to claim 2 or 3.

10. the first communication device transmits an image captured by the camera unit to the second communication device; The second device further includes a second control device that selectively displays, on the display device, an image corresponding to the orientation direction among images captured by the camera unit.

4. The information processing system according to claim 2 or 3.

11. The display device is a display device of VR (Virtual Reality) goggles, The detection device includes a physical sensor attached to the VR goggles.

4. The information processing system according to claim 2 or 3.

12. the moving object is a vehicle, The predetermined seat is a passenger seat. The information processing system according to claim 9.

Citation Information

Patent Citations

  • Image distribution system

    JP2004128804A

  • Image drawing device

    JP2009038428A

  • Camera control method, camera control device, camera control program, and camera system

    JP2009049798A

  • Onboard device control apparatus

    JP2009234467A

  • Prompter-type operation device

    JP2009252105A