Information processing device, information processing system, and information processing method

Abstract

To improve an entertainment property.SOLUTION: An information processing device is mounted in a moving object that an occupant boards. The information processing device includes: a display device having a display portion, which displays a person image indicating a person or a character corresponding to the person, and capable of adjusting a position of the display portion so that an occupant of a back seat is shown in a mirror when an occupant of a driver's seat or a front passenger seat of the moving object visually recognizes the display portion on the assumption that the display portion is the mirror and the occupant is sitting in the back seat of the moving object; and a control device that controls a first speaker within the moving object so that a sound is output from a direction of the back seat when the display portion is allowed to display a service image including the person image in which the person or the character is sitting the back seat of the moving object. The control device controls the display of the service image on the basis of a traveling state of the moving object, a shift position of the moving object, or a driving mode set to the moving object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] Conventionally, research has been conducted into communicating between a device mounted on a moving object such as a vehicle and a device used in a location other than the moving object, and providing images of users in the other location to the user of the moving object (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-80542 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology, the user of the moving object could not feel a sufficient sense of realism, and the entertainment value within the moving object could not be said to be sufficient. Also, the visibility when using the entertainment service could not be said to be high enough.

[0005] The present invention has been made in consideration of the above circumstances, and has as its object to provide an information processing device, an information processing system, and an information processing method that can improve entertainment value. For example, it is possible to provide a highly entertaining service that can enhance the sense of realism and take into account the user's visibility. In turn, it is possible to further improve traffic safety and contribute to the development of a sustainable transportation system. [Means for solving the problem]

[0006] An information processing device, an information processing system, and an information processing method according to the present invention employ the following configuration. (1): An information processing device according to one embodiment of the present invention is an information processing device mounted on a moving body in which an occupant rides, the information processing device comprising: a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display device being capable of adjusting the position of the display unit so that when an occupant in the driver's seat or passenger seat of the moving body looks at the display unit, assuming that the display unit is a mirror and that the occupant is seated in the back seat of the moving body, the display device being capable of adjusting the position of the display unit so that the occupant in the back seat is reflected in the mirror; and a control device that controls a first speaker in the moving body to output sound from the direction of the back seat when a service image including the person image that makes it appear as if the person or character is seated in the back seat of the moving body is displayed on the display unit, the control device being an information processing device that controls the display of the service image based on the driving status of the moving body, the shift position of the moving body, or the driving mode set on the moving body.

[0007] (2): An information processing device according to another embodiment of the present invention is an information processing device mounted on a moving body in which an occupant rides, the information processing device comprising: a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display device being capable of adjusting the position of the display unit so that when an occupant in the driver's seat or passenger seat of the moving body looks at the display unit, assuming that the display unit is a mirror and that the occupant is sitting in the back seat of the moving body, the occupant in the back seat is reflected in the mirror; and a control device that controls a first speaker in the moving body to output sound from the direction of the back seat when a service image including the person image that makes the person or character appear to be sitting in the back seat of the moving body is displayed on the display unit, the control device being an information processing device that displays a service image of a first quality on the display unit while the moving body is moving, and displays a service image of a second quality, the image quality of which is superior to the first quality, on the display unit while the moving body is stopped.

[0008] (3): An information processing device according to another embodiment of the present invention is an information processing device mounted on a moving body in which an occupant rides, the information processing device comprising: a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display device being capable of adjusting the position of the display unit so that when an occupant in the driver's seat or passenger seat of the moving body looks at the display unit, assuming that the display unit is a mirror and that the occupant is sitting in the back seat of the moving body, the occupant in the back seat is reflected in the mirror; and a control device that controls a first speaker in the moving body to output sound from the direction of the back seat when the display unit is displaying a service image including the person image that makes the person or character appear to be sitting in the back seat of the moving body, the control device is an information processing device that allows the display device to display the service image including the person image when the moving body is not stopped, and displays an image captured by an imaging unit that images the rear of the moving body without allowing the display device to display the service image including the person image when the moving body is stopped.

[0009] (4): In any of the above aspects (1) to (3), a first communication device is provided which communicates with a second communication device of a second device used by a user at a location different from the mobile body including a second communication device and a second microphone, and the voice is voice uttered by the user to the second microphone and acquired via the first communication device.

[0010] (5): In the above aspect (4), the person image is an image of a user who uses the second device or an image showing a character corresponding to the user.

[0011] (6): In the aspect (5) above, the control device generates the service image by combining the person image with an image of the rear seat, and displays the generated service image on the display device.

[0012] (7): In any of the above aspects (1) to (3), the first speaker outputs the sound by positioning a sound image so that the sound can be heard from the rear seat by an occupant in the driver's seat or the passenger seat.

[0013] (8): In the above aspect (7), the first speaker includes a plurality of sub-first speakers arranged at different positions from each other, and the control device adjusts the volume and / or phase difference of the plurality of sub-first speakers to localize a sound image so that an occupant in the driver's seat or the passenger seat can hear sound from the rear seat.

[0014] (9): In any of the above aspects (1) to (3), the display unit of the display device is a rearview mirror provided to enable an occupant in the driver's seat to view the rear of the moving body, and displays an image from an imaging unit that images the rear.

[0015] (10): In the above aspect (9), the rearview mirror is provided on the front windshield or ceiling inside the vehicle cabin of the moving body, in front of the occupant in the driver's seat when the occupant is seated in the driver's seat of the moving body.

[0016] (11): In any one of the above aspects (1) to (3), the control device causes the display device to display a service image including the person image when the moving object is not stationary.

[0017] (12): In the above aspect (3), a first communication device is provided that communicates with a second communication device of a second device used by a user at a location different from the moving body including a second communication device and a second microphone, the voice being a voice uttered by the user to the second microphone acquired via the first communication device, and the control device controls the first speaker to display a service image including the person image on the display device when the moving body is not stopped, and to output the voice from the direction of the rear seat or the passenger seat, and thereafter, when the moving body stops, continues to control the first speaker to output the voice, stops displaying the service image including the person image on the display device, and causes the display device to display an image captured by the imaging unit.

[0018] (13): An information processing device according to another aspect of the present invention is an information processing device mounted on a moving body in which an occupant rides, the information processing device comprising: a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display device being capable of adjusting a position of the display unit so that when an occupant in the driver's seat or passenger seat of the moving body looks at the display unit, assuming that the display unit is a mirror and that the occupant is seated in the rear seat of the moving body, the occupant in the rear seat is reflected in the mirror; and a control device that controls a first speaker in the moving body to output sound from the direction of the rear seat when the display unit is displaying a service image including the person image as if the person or character is seated in the rear seat of the moving body, the control device allows the display device to display the service image including the person image when the shift position of the moving body is not in a parking position, and when the shift position of the moving body is in the parking position, the control device displays an image captured by an imaging unit that captures the rear of the moving body without allowing the display device to display the service image including the person image.

[0019] (14): In the aspect of (13) above, a first communication device is provided that communicates with a second communication device of a second device used by a user at a location different from the mobile body including a second communication device and a second microphone, the voice is a voice uttered by the user to the second microphone acquired via the first communication device, and the control device, when the shift position of the mobile body is a position where the mobile body can start, controls the first speaker to display a service image including the person image on the display device and to output the voice from the direction of the rear seat or the passenger seat, and then, when the shift position of the mobile body is changed from the position where the mobile body can start to a parking position, continues to control the first speaker to output the voice, and displays an image captured by the imaging unit on the display device.

[0020] (15): An information processing device according to another aspect of the present invention is an information processing device mounted on a moving body on which a passenger rides, the information processing device being a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being capable of adjusting the position of the display unit so that the passenger in the rear seat is reflected in the mirror when the passenger in the driver's seat or passenger seat of the moving body looks at the display unit assuming that the display unit is a mirror and that the passenger is seated in the rear seat of the moving body, and the information processing device being a display device that displays a service image including the person image, such that the person or character is seated in the rear seat of the moving body, on the display unit. and a control device that controls a first speaker in the moving body to output sound from the direction of the rear seat when the first speaker is displayed as a voice command, and when the moving body control device is performing driving assistance or automatic driving to control the traveling of the moving body, the control device allows the display device to display a service image including the person image, and when the moving body control device is not performing driving assistance or automatic driving but is operating the moving body to control the traveling of the moving body, the control device displays an image captured by an imaging unit that captures images of the rear of the moving body without allowing the display device to display the service image including the person image.

[0021] (16): In the aspect of (15) above, a first communication device is provided that communicates with a second communication device of a second device used by a user at a location different from the moving body including a second communication device and a second microphone, and the voice is a voice uttered by the user to the second microphone acquired via the first communication device, and the control device, when the moving body control device is performing driving assistance or automatic driving to control the traveling of the moving body, causes the display device to display a service image including the person image, and controls the first speaker to output the voice from the direction of the rear seat or the passenger seat, and thereafter, when the driving assistance or the automatic driving is ended and the moving body is controlled by the operation of the driver, continues to control the first speaker to output the voice, and causes the display device to display an image captured by the imaging unit.

[0022] (17): In any one of (1) to (3), (13), or (15) above, the moving body is a vehicle.

[0023] (18): In the aspect of (17) above, when the vehicle does not satisfy the conditions, the control device controls the display device to display a service image including the person image and the first speaker to output the audio from the direction of the rear seat, and when the vehicle satisfies the conditions, the control device controls the display device not to display the service image including the person image and the first speaker to output the audio from the direction of the passenger seat, the conditions being that the vehicle has stopped, that the shift position of the vehicle is in a parking position, or that the driver of the moving body is operating to control the traveling of the moving body without performing driving assistance or autonomous driving.

[0024] (19): An information processing system according to another embodiment of the present invention includes an information processing device according to embodiment (4) above, a user device having a second communication device that communicates with the first communication device, and a second microphone that collects voice uttered by the user, and the second communication device transmits the voice collected by the second microphone to the first communication device.

[0025] (20): In the above aspect (19), the device for the user includes a detection device for detecting the direction of orientation of the user, a camera unit having one or more cameras including at least an indoor camera installed on a designated seat of the moving body and capable of capturing images of the interior of the moving body as seen from the designated seat, and a display device for displaying images captured by the camera unit that correspond to the direction of orientation as seen from the designated seat of the moving body.

[0026] (21): In the above aspect (20), the second communication device transmits information on the orientation direction to the first communication device, the control device controls the first communication device to selectively transmit to the second communication device an image, among images captured by the camera unit, that corresponds to the orientation direction obtained via the first communication device, and the display device of the user device displays the image, obtained via the second communication device, that corresponds to the orientation direction as seen from the specified seat.

[0027] (22): In the above aspect (20), the first communication device transmits an image captured by the camera unit to the second communication device, and the user device selectively displays, on the display device, an image from the images captured by the camera unit that corresponds to the orientation direction.

[0028] (23): In the above aspect (20), the information processing device further has a first microphone that collects at least the voice uttered by the occupant, the user device further has a second speaker that outputs the voice uttered by the occupant acquired via the second communication device, and the first communication device transmits the voice collected by the first microphone to the second communication device.

[0029] (24): An information processing method according to one embodiment of the present invention is a display device having a display unit that displays a person image showing a person or a character corresponding to a person, and when a driver of a moving body in which the occupant rides looks at the display unit assuming that the display unit is a mirror and that the occupant is sitting in the back seat of the moving body, a service image including the person image in which the person or character is sitting in the back seat of the moving body is displayed on a display unit whose position can be adjusted so that the occupant is reflected in the mirror, a first speaker that outputs audio is controlled to output audio from the direction of the back seat, and the display of the service image is controlled based on the driving status of the moving body, the shift position of the moving body, or the driving mode set for the moving body.

[0030] (25): An information processing method according to one embodiment of the present invention is a display device in which a computer has a display unit that displays a person image showing a person or a character corresponding to a person, and when a driver of a moving body in which the moving body is riding looks at the display unit assuming that the display unit is a mirror and that the occupant is sitting in the back seat of the moving body, the display unit is adjustable in position so that the occupant is reflected in the mirror.When a service image including the person image in which the person or character is sitting in the back seat of the moving body is displayed, a first speaker that outputs audio is controlled to output audio from the direction of the back seat, and while the moving body is moving, a service image of a first quality is displayed on the display unit, and while the moving body is stopped, a service image of a second quality having image quality superior to the first quality is displayed on the display unit.

[0031] (26): An information processing method according to one embodiment of the present invention is a display device having a display unit in which a computer displays a person image showing a person or a character corresponding to a person, and when a driver of a moving body in which the moving body is riding views the display unit assuming that the display unit is a mirror and that the occupant is sitting in the back seat of the moving body, the display unit is adjustable in position so that the occupant is reflected in the mirror, and when a service image including the person image as if the person or character is sitting in the back seat of the moving body is displayed on the display unit, the position of which can be adjusted so that the occupant is reflected in the mirror, a first speaker that outputs audio is controlled to output audio from the direction of the back seat, and when the moving body is not stopped, the display device is permitted to display the service image including the person image, and when the moving body is stopped, an image captured by an imaging unit that captures the rear of the moving body is displayed without allowing the display device to display the service image including the person image.

[0032] (27): An information processing method according to one embodiment of the present invention is a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that the person in the rear seat is reflected in the mirror when an occupant in the driver's seat or passenger seat of the moving body views the display unit, assuming that the display unit is a mirror and that the occupant is seated in the rear seat of the moving body, and when a service image including the person image as if the person or character is seated in the rear seat of the moving body is displayed on the display unit of the display device, the display unit controls a first speaker in the moving body to output sound from the direction of the rear seat, when the shift position of the moving body is not in a parking position, allows the display device to display the service image including the person image, and when the shift position of the moving body is in the parking position, displays an image captured by an imaging unit that images the rear of the moving body without allowing the display device to display the service image including the person image.

[0033] (28): A computer of an information processing device mounted on a moving body in which a passenger rides, according to one embodiment of the present invention, is a display device having a display unit that displays a person image showing a person or a character corresponding to a person, and when a passenger in the driver's seat or passenger seat of the moving body views the display unit, assuming that the display unit is a mirror and that the passenger is sitting in the back seat of the moving body, the display unit is adjustable in position so that the passenger in the back seat is reflected in the mirror.When a service image including the person image as if the person or character is sitting in the back seat of the moving body is displayed on the display unit of the display device, the display unit controls a first speaker in the moving body to output sound from the direction of the back seat, and when a moving body control device is performing driving assistance or automatic driving to control the traveling of the moving body, the display device is permitted to display the service image including the person image, and when the moving body control device is not performing driving assistance or automatic driving and the driver of the moving body is operating to control the traveling of the moving body, the display device displays an image captured by an imaging unit that captures the rear of the moving body without permitting the display device to display the service image including the person image. Effect of the Invention

[0034] According to the aspects (1) to (28), the entertainment value can be improved. For example, the sense of realism given to the passengers of the moving body can be enhanced. In addition, the driver can be provided with information suitable for the driver depending on the situation of the moving body, thereby realizing convenience for the driver and improved entertainment value. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 is a diagram showing the usage environment of an information processing system 1 and a management server 300. [Diagram 2] FIG. 2 is a diagram showing an example of the contents of user data 360. [Diagram 3] FIG. 2 is a configuration diagram of a first device 100. [Figure 4] 2 is a diagram showing an example of the arrangement of a part of a first device 100 in a moving object M. FIG. [Diagram 5] FIG. 2 is a diagram illustrating the configuration of a second device 200. [Figure 6] FIG. 11 is a diagram for explaining an image corresponding to a pointing direction. [Figure 7] 2 is a diagram illustrating a first example of a functional configuration of a first control device 170 and a second control device 270. FIG. [Figure 8] FIG. 13 is a diagram showing an example of displaying a replacement image. [Figure 9] 13 is a diagram showing an example of a service image displayed on the display unit of the rearview mirror display device 152. FIG. [Figure 10] FIG. 13 is a diagram focusing on a service image. [Figure 11] 4 is a flowchart showing an example of the flow of processing executed by a first control device 170. [Figure 12] FIG. 13 is a diagram illustrating an example of a condition. [Figure 13] 11 is a diagram showing an example of the relationship between the transition of an image displayed on a rearview mirror display device 152 and the determination result of a state determination unit. FIG. [Figure 14] 13 is a diagram showing another example of the relationship between the transition of the image displayed on the rearview mirror display device 152 and the determination result of the state determination unit. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Hereinafter, with reference to the drawings, an embodiment of an information processing device, an information processing system, and an information processing method of the present invention will be described. The information processing system includes a first device (information processing device) mounted on a moving body on which a passenger boards, and a second device (user device) used by a user at a location different from the moving body. The moving body is, for example, a vehicle, but may be any moving body that a passenger can board. The passenger is mainly the driver of the moving body, but may be a passenger other than the driver. In the following description, the information processing device (first device 100) and the rearview mirror (rearview mirror display device 152) are described as separate entities, but this is not necessary and the information processing device and the rearview mirror may be configured as one body. That is, the rearview mirror may be equipped with the information processing device. In this case, a user who purchases a rearview mirror equipped with an information processing device as an add-on can also get the passenger experience according to the present invention.

[0037] Between the first device and the second device, the voice collected by the microphone is transmitted to the other party and played back by the speaker, creating a state similar to a telephone conversation, and further, a part of an image captured by the camera unit of the first device is displayed by the second device, providing MR (Mixed Reality) on the second device side. The first device and the second device do not need to be in a one-to-one relationship, and one of the multiple first devices and multiple second devices may be matched in a one-to-many relationship to operate as an information processing system. In the latter case, for example, one passenger can communicate with multiple users simultaneously or in sequence.

[0038] <Basic configuration> 1 is a diagram showing the usage environment of an information processing system 1 and a management server 300. The information processing system 1 includes a first device (mobile device) 100 mounted on a mobile body M, and a second device (user device) 200 used by a user U at a location different from the mobile body M (a location that happens to be close is not excluded). The first device 100, the second device 200, and the management server 300 communicate with each other via a network NW. The information processing system 1 may or may not include the management server 300.

[0039] The management server 300 includes, for example, a communication device 310, a matching processing unit 320, and a storage unit 350. The storage unit 350 stores user data 360.

[0040] Communication device 310 is a communication interface for connecting to network NW. Note that communication between communication device 310 and first device 100, and communication between communication device 310 and second device 200 are performed in accordance with, for example, TCP / IP (Transmission Control Protocol / Internet Protocol).

[0041] The matching processing unit 320 is realized by a processor such as a CPU (Central Processing Unit) executing a program (a group of instructions) stored in a storage medium. The storage unit 350 is a RAM (Random Access Memory), a HDD (Hard Disk Drive), a flash memory, or the like.

[0042] Fig. 2 is a diagram showing an example of the contents of the user data 360. The user data 360 includes, for example, an occupant list 360A in which an occupant ID, which is identification information of an occupant P, its communication identification information (IP address, etc.), and a user ID, which is identification information of a user U to be matched, are associated with each other, and a user list 360B in which a user ID, its communication identification information (IP address, etc.), and an occupant P to be matched are associated with each other. The user data 360 may be generated in any manner, not limited to the manner shown in Fig. 2, as long as it includes this information.

[0043] When the communication device 310 receives a matching request from the user U via the second device 200 or from the occupant P via the first device 100, the matching processing unit 320 refers to the user data 360 to perform matching between the matching user U and the occupant P, transmits communication identification information of the first device 100 of the occupant P to the second device 200 of the matched user U using the communication device 310, and transmits communication identification information of the second device 200 of the user U to the first device 100 of the matched occupant P using the communication device 310. Between the first device 100 and the second device 200 that receive these, communication with higher real-time quality based on, for example, UDP (User Datagram Protocol) is performed.

[0044] 3 is a configuration diagram of the first device 100. The first device 100 includes, for example, a first communication device 110, a first microphone 120, a camera unit 130, a first speaker 140, a user display device 150, a rearview mirror display device 152, an HMI (Human machine Interface) 160, and a first control device ("control device") 170. The first control device 170 is connected to a control target device 190 mounted on a moving object M and a moving object control device 192.

[0045] The first communication device 110 is a communication interface for communicating with each of the communication device 310 of the management server 300 and the second communication device 210 of the second device 200 via the network NW.

[0046] The first microphone 120 collects at least the voice uttered by the occupant P. The first microphone 120 may be provided inside the interior of the moving body M and may have a sensitivity capable of collecting voice outside the moving body M, or may include a microphone provided inside the interior of the moving body M and a microphone provided outside the moving body M. The voice collected by the first microphone 120 is transmitted to the second communication device 210 by the first communication device 110 via the first control device 170, for example.

[0047] The camera unit 130 includes at least an indoor camera 132, and may also include an outdoor camera 134. The outdoor camera 134 includes, for example, a camera that captures an image of the rear of the moving body M. The captured image is displayed on the HMI 160, the rearview mirror display device 152, or the like. This allows the occupants of the moving body to recognize the situation behind. The first speaker 140 outputs the voice uttered by the user U acquired via the first communication device 110. Details of the arrangement of the camera unit 130 and the first speaker 140 will be described later with reference to FIG. 4.

[0048] The user display device 150 virtually displays the user U as if the user U were present inside the room of the moving object M. For example, the user display device 150 makes a hologram appear or displays the user U on a part of the moving object M that corresponds to a mirror or window.

[0049] The rearview mirror display device 152 is a rearview mirror provided for a driver's seat occupant to view the rear of the moving body M. The rearview mirror display device 152 is provided, for example, in front of the driver's seat occupant when the driver sits in the driver's seat of the moving body, on the front windshield or ceiling in the vehicle interior of the moving body. The rearview mirror display device 152 has a mirror (mirror) / display surface, and in the mirror mode, the mirror / display surface functions as a mirror to reflect an object by reflecting light, and in the monitor mode, the mirror / display surface functions as a display unit that displays an image to display an image captured by a camera that captures an image of the rear of the moving body M or a service image to be described later. For example, the mirror mode and the monitor mode are switched by operating a changeover switch provided in the rearview mirror display device 152. The rearview mirror display device 152 is an example of a "display device".

[0050] In this embodiment, the rearview mirror display device 152 is described as displaying a service image including a person image. However, instead of (or in addition to) this, the service image may be displayed on another display device (another example of a "display device") provided in the vehicle cabin. This display device may be, for example, a rear seat confirmation mirror display device for a driver or passenger in the front passenger seat to confirm or view the rear seat or passengers in the rear seat. This mirror display device has a mirror / display surface, and in the mirror mode, the display surface functions as a mirror to reflect objects by reflecting light, and in the monitor mode, the display surface functions as a display unit to display images and display images from a vehicle interior camera or a service image. Also, as described above, the rearview mirror display device 152 may include a functional configuration similar to that of the first control device ("control device") 170, and control of the display of the service image and control of the first speaker 140 may be executed.

[0051] As described above, the position of the mirror / display surface of the rearview mirror display device 152 or another display device provided in the vehicle interior is adjustable. The rearview mirror display device 152 or another display device provided in the vehicle interior can adjust the position of the display unit so that, for example, when the display unit is assumed to be a mirror and an occupant is seated in the rear seat of the moving body M, an occupant in the driver's seat or passenger seat of the moving body M looks at the display unit, the occupant in the rear seat is reflected in the mirror. For example, the rearview mirror display device 152 or another display device provided in the vehicle interior is a display device capable of adjusting the position of the display unit so that, when an occupant in the driver's seat or passenger seat of the moving body M looks at the occupant in the rear seat of the moving body M through the mirror, a human image is displayed at a position in the mirror where the occupant in the rear seat is reflected. The adjustable position of the display unit means, for example, that the occupant can adjust the attitude, angle, and direction of the display unit (for example, the direction facing the driver's seat, passenger seat, or rear seat).

[0052] The HMI 160 is a touch panel, a voice response device (agent device), etc. The HMI 160 receives various instructions from the occupant P to the first device 100.

[0053] The first control device 170 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (a set of instructions), and the processor controls each part of the first device 100 by executing the set of instructions.

[0054] The control target device 190 is, for example, a navigation device or a driving support device mounted on the moving object M.

[0055] The mobile body control device 192 is a control device that controls one or both of the steering and acceleration / deceleration of the vehicle. For example, the mobile body control device 192 controls the steering of the mobile body M so that the mobile body M runs within the lane. For example, the mobile body control device 192 controls the acceleration / deceleration of the mobile body M so that the distance between the mobile body M and the preceding vehicle becomes a predetermined distance, or controls the acceleration / deceleration of the mobile body M so that the mobile body M runs while maintaining a predetermined speed. In addition, the mobile body control device 192 may perform, for example, automatic driving that controls the mobile body M without depending on the operation of the driver. For example, the driving assistance or automatic driving includes, for example, a first mode in which the driver does not operate and does not need to monitor the surroundings, a second mode in which the driver does not operate and needs to monitor the surroundings, and a third mode in which the driver operates and needs to monitor the surroundings. In addition, the mobile body control device 192 provides the first control device 170 with the state of the mobile body M, such as the speed of the mobile body M and the shift position.

[0056] FIG. 4 is a diagram showing an example of the arrangement of a part of the first device 100 in the moving object M. The indoor camera 132 is attached to the neck pillow of the passenger seat S2 (an example of a "predetermined seat") via an attachment 132A, for example, and is provided at a position slightly separated from the backrest of the passenger seat S2 in the traveling direction of the moving object M. The indoor camera 132 has a wide-angle lens and can capture an image of a range represented by a hatched area 132B in the figure. The indoor camera 132 can capture not only the interior of the moving object M, but also the outside through the window. In the following description, the passenger seat S2 is considered to be the predetermined seat, but the predetermined seat may be another seat such as a rear seat.

[0057] The outdoor camera 134 includes, for example, a plurality of child outdoor cameras 134-1 to 134-5. By synthesizing the images captured by the plurality of child outdoor cameras 134-1 to 134-5, an image such as a panoramic image capturing the outside of the moving object M can be obtained. Instead of (or in addition to) these, the outdoor camera 134 may include a wide-angle camera provided on the roof of the moving object M. Note that a camera capable of capturing an image of the rear of the passenger seat S2 may be added as the indoor camera 132, and the moving object image described later may be generated as a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 by the first control device 170, or may be generated as a 360-degree panoramic image by appropriately combining an image captured by the indoor camera 132 and an image captured by the outdoor camera 134. The child outdoor camera 134-5 is an example of an "imaging unit capturing an image of the rear of the moving object".

[0058] The first speaker 140 outputs the voice of the user U acquired via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, the sub-first speaker 140-1 is disposed at the center of the instrument panel, the sub-first speaker 140-2 at the left end of the instrument panel, the sub-first speaker 140-3 at the right end of the instrument panel, the sub-first speaker 140-4 at the bottom of the left door, and the sub-first speaker 140-5 at the bottom of the right door. The first control device 170, for example, outputs voices at approximately the same volume from the sub-first speaker 140-2 and the sub-first speaker 140-4 and turns off the other sub-first speakers, thereby localizing the sound image so that the voice can be heard from the passenger seat S2 by the occupant P seated in the driver's seat S1. The first control device 170, for example, outputs sound from the child first speaker 140-4 and the child first speaker 140-5 at approximately the same volume and turns off the other child first speakers, thereby localizing the sound image so that the occupant P seated in the driver's seat S1 can hear the sound from the rear seat. The method of localizing the sound image is not limited to adjusting the volume, and may be performed by shifting the phase of the sound output from each child first speaker. For example, when localizing the sound image so that the sound is heard from the left side, the timing of outputting the sound from the left child first speaker may be slightly earlier than the timing of outputting the same sound from the right child first speaker. For example, when localizing the sound image so that the sound is heard from the rear side, the timing of outputting the sound from the rear child first speaker may be slightly earlier than the timing of outputting the same sound from the front child first speaker.

[0059] Furthermore, when the first control device 170 outputs the voice of the user U to the first speaker 140, the first control device 170 may localize a sound image so that the voice is heard by the occupant P from a position at a height corresponding to the height of the head of the user U sitting on the passenger seat S2, and output the voice emitted by the user U from the first speaker 140. In this case, the first speaker 140 needs to have multiple child first speakers 140-k (k is a multiple natural number) at different heights.

[0060] The first control device 170 controls, for example, the rearview mirror display device 152. The first control device 170 controls, for example, an image to be displayed on the rearview mirror display device 152. Details will be described later.

[0061] 5 is a configuration diagram of the second device 200. The second device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a mobile image display device 250, an HMI 260, and a second control device 270. The detection device 230 includes, for example, an orientation direction detection device 232, a head position detection device 234, and a motion sensor 236.

[0062] The second communication device 210 is a communication interface for communicating with each of the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0063] The second microphone 220 collects the voice uttered by the user U. The voice collected by the second microphone 220 is transmitted by the second communication device 210 to the first communication device 110 via the second control device 270, for example.

[0064] The orientation direction detection device 232 is a device for detecting the orientation direction. The orientation direction is a direction based on the direction of the user U's face or the direction of the line of sight, or both. Alternatively, it may be a direction indicated by the movement of the arm or finger, such as the motion of tilting the terminal device used by the user U or the motion of swiping the screen. In the following, the orientation direction is assumed to be an angle in a horizontal plane, that is, an angle that does not have a vertical component, but the orientation direction may also be an angle that includes a vertical component. The orientation direction detection device 232 may include a physical sensor (e.g., an acceleration sensor, a gyro sensor, etc.) attached to the VR goggles described later, or may be an infrared sensor that detects multiple positions of the user U's head, or a camera that captures the user U's head. In either case, the second control device 270 calculates the orientation direction based on the information input from the orientation direction detection device 232. Since various techniques for this are publicly known, detailed explanations will be omitted.

[0065] The head position detection device 234 is a device for detecting the position (height) of the head of the user U. For example, one or more infrared sensors or optical sensors provided around the chair on which the user U sits are used as the head position detection device 234. In this case, the second control device 270 detects the position of the head of the user U based on the presence or absence of a detection signal by one or more infrared sensors or optical sensors. The head position detection device 234 may also be an acceleration sensor attached to the VR goggles. In this case, the second control device 270 detects the position of the head of the user U by integrating the output of the acceleration sensor minus the gravitational acceleration. The head position information thus acquired is provided to the second control device 270 as height information. The user's head position may be acquired based on the user U's operation on the HMI 260. For example, the user U may input his / her height in numbers into the HMI 260, or may input his / her height using a dial switch included in the HMI 260. In these cases, the head position, i.e., height information, is calculated from the height. Furthermore, the user U may input a discrete value such as physique: large / medium / small to the HMI 260 instead of a continuous value. In this case, height information is acquired based on information indicating the physique. Furthermore, the head height of the user U may be acquired simply based on the physique of a general adult (which may be gender-specific) without acquiring the head height of the user in particular.

[0066] The motion sensor 236 is a device for recognizing a gesture operation performed by the user U. For example, a camera capturing an image of the upper body of the user U is used as the motion sensor 236. In this case, the second control device extracts feature points (fingertips, wrists, elbows, etc.) of the user U's body from the image captured by the camera, and recognizes the gesture operation of the user U based on the movement of the feature points.

[0067] The second speaker 240 outputs the voice uttered by the occupant P acquired via the second communication device 210. The second speaker 240 has a function of changing the direction from which the voice is heard, for example. The second control device 270 causes the second speaker to output the voice so that the user U can hear the voice from the position of the occupant P as seen from the passenger seat S2. The second speaker 240 includes a plurality of child second speakers 240-n (n is a plurality of natural numbers), and the second control device 270 may adjust the volume of each child second speaker 240-n to perform sound image localization, or if headphones are attached to the VR goggles, the function of the headphones may be used to perform sound image localization.

[0068] The moving body image display device 250 displays an image corresponding to the orientation direction as seen from the passenger seat among the images captured by the camera unit 130 (which may be images that have been subjected to the above-mentioned combining process, and are hereinafter referred to as moving body images). FIG. 6 is a diagram for explaining an image corresponding to the orientation direction. In the example of this figure, the VR goggles 255 includes an orientation direction detection device 232, a physical sensor as a head position detection device 234, and the moving body image display device 250. The second control device 270 detects the direction in which the VR goggles 255 faces as the orientation direction φ, with a pre-calibrated direction as the reference direction. Various methods for this function are already known, so detailed explanations will be omitted.

[0069] The moving body image display device 250 displays, toward the user U, an image A2 within an angle range of plus or minus α centered on the directional direction φ, from the moving body image A1 (which has an angle of about 240 degrees in the figure, but the angle of view may be expanded by combining processing as described above).

[0070] The HMI 260 is a touch panel, a voice response device (agent device), the above-mentioned switch, etc. The HMI 260 receives various instructions from the occupant P to the second device 200.

[0071] The second control device 270 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (a group of instructions), and the processor controls each part of the second device 200 by executing the group of instructions.

[0072] <Functional configuration> The functional configuration of the first control device 170 and the second control device 270 will be described below. FIG. 7 is a diagram showing an example of the functional configuration of the first control device 170 and the second control device 270. The first control device 170 includes a matching request / acceptance unit 171, an audio output control unit 172, an image transmission unit 173, an on-board device cooperation unit 174, a display control unit 176, and a state determination unit 178. The second control device 270 includes a matching request / acceptance unit 271, an audio output control unit 272, an orientation direction detection unit 273, a head position detection unit 274, a gesture input detection unit 275, an image editing unit 276, and a moving object image display control unit 277. These functional units are realized, for example, by a processor such as a CPU executing a program (a group of instructions). Some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware.

[0073] The matching request / acceptance unit 171 accepts an input of a matching request from the occupant P using the HMI 160 and transmits it to the management server 300, and also accepts an input of acceptance of a matching request received from the management server 300 using the HMI 160 and transmits it to the management server 300. The matching request / acceptance unit 171 controls the first communication device 110 to set the second device 200 of the user U with whom a match has been established as the communication partner.

[0074] The audio output control unit 172 controls the first speaker 140 as described above.

[0075] The image transmission unit 173 uses the first communication device 110 to transmit the moving object image A1 to the second device 200.

[0076] The on-board device cooperation unit 174 controls the control target device 190 based on the instruction signal input from the second device 200.

[0077] The display control unit 176 causes the rearview mirror display device 152 to display a service image including an image of a person or character sitting in the back seat of the moving body M.

[0078] The state determination unit 178 determines whether the moving object M satisfies a condition based on the state of the moving object M (see FIG. 12 described later).

[0079] The matching request / acceptance unit 271 accepts an input of a matching request from the user U using the HMI 260 and transmits it to the management server 300, and also accepts an input of acceptance of a matching request received from the management server 300 using the HMI 260 and transmits it to the management server 300. The matching request / acceptance unit 271 controls the second communication device 210 to communicate with the first device 100 of the occupant P with whom a match has been established.

[0080] The audio output control unit 272 controls the second speaker 240 as described above.

[0081] The orientation direction detection unit 273 detects the orientation direction φ based on the output of the orientation direction detection device 232. The head position detection unit 274 detects the height of the head of the user U based on the output of the head position detection device 234. The head position may be represented as three-dimensional coordinates, or the head height may simply be detected as the head position. The gesture input detection unit 275 detects a gesture input by the user U based on the output of the motion sensor 236.

[0082] The image editing unit 276 performs processing to cut out an image A2 corresponding to the directional direction φ as seen from the passenger seat from the moving object image A1 (FIG. 6). The moving object image display control unit 277 causes the moving object image display device 250 to display the image A2 cut out by the image editing unit 276. At this time, the image editing unit 276 may cause the moving object image display device 250 to display an image corresponding to the directional direction φ as seen from the height indicated by the height information of the head of the user U.

[0083] Note that the first control device 170 may include an image editing unit having the same function as the image editing unit 276, and the second control device 270 may include a direction of orientation transmission unit without including the image editing unit 276. The direction of orientation transmission unit 278 transmits the direction of orientation φ detected by the direction of orientation detection unit 273 to the first device 100 using the second communication device 210, and the image editing unit of the first control device 170 performs a process of cutting out an image A2 corresponding to the direction of orientation φ (transmitted from the second device 200) seen from the passenger seat from the moving object image A1 (FIG. 6).

[0084] <Other> In the information processing system 1, the user U can view any direction from the passenger seat S2. However, for example, a restriction may be set on the direction that the user U can view, depending on an agreement made at the time of matching. For example, the passenger P may request that the scenery in the traveling direction of the moving body M or the scenery on the opposite side of the driver's seat S1 may be provided, but that the passenger P does not want to see his / her own figure. This is intended to meet the needs of the passenger P and the user U, who are not family members or friends, and who want to check the driving feeling of the moving body M or view a desired cityscape. In this case, such a restriction is set when the matching processing unit 320 of the management server 300 performs matching processing, and the first control device 170 or the second control device 270 masks the angle range that is not visible or corrects the orientation direction φ so that it does not face the restricted direction according to the setting. In addition, information regarding such a restriction may be set on the first device 100 side, since it is related to the privacy of the passenger P.

[0085] Furthermore, the mobile body image display device 250 may replace a portion of the image captured by the camera unit 130 that shows a predetermined object inside the room of the mobile body M with an image (CG image) drawn by computer processing and display it. FIG. 8 is a diagram showing an example of displaying a replaced image. In the figure, OB is a display device that performs navigation display and the like, and is an example of a "predetermined object." If an image captured on the display screen of the display device is displayed as is, the image may be blurred or visibility may be reduced due to light reflection. For this reason, the mobile body image display device 250 may acquire data for configuring the display screen of the display device, or image data drawn by computer processing in the mobile body M, from the first device, and may display an image redrawn from the data by computer processing, or the acquired image data, embedded in an image captured by the camera unit 130 (edited image). In this case, the position of the object in the room of the moving object M that is the predetermined object is shared in advance between the first device 100 and the second device 200, and the moving object image display control unit 277 determines whether the predetermined object is included in the image to be displayed on the moving object image display device 250 based on, for example, the orientation direction φ, and performs image replacement as described above when it is determined that the predetermined object is included. The "predetermined object" may also be the head or face of the occupant P. In that case, a CG image such as an avatar may be changed according to the display of the occupant P.

[0086] <Summary so far> According to the information processing system 1 configured as above, it is possible to enhance the sense of realism given to both the occupant P of the moving body M and the user U who is in a different place from the moving body M. Since an image corresponding to the directional direction φ of the user U as seen from the passenger seat is displayed to the user U, the user U can visually recognize the scenery as if he / she were sitting in the passenger seat S2 and looking around. Furthermore, the first speaker 140 outputs the voice of the user U by localizing the sound image so that the voice is heard from the passenger seat S2 for the occupant P, so that the occupant P can talk with the user U as if the user U is in the passenger seat S2. Furthermore, the second speaker 240 outputs the voice of the occupant P by localizing the sound image so that the voice is heard from the position of the occupant P as seen from the passenger seat S2 for the user U, so that the user U can talk with the occupant P as if he / she is in the passenger seat S2.

[0087] <Usage example> The information processing system 1 can be used in the following manner.

[0088] (A) A mode in which the occupant P and the user U are in a relationship such as family or friends, and a virtual drive is provided to the user U. The user U can talk with the occupant P about the scenery around the moving body M while looking at the image.

[0089] (B) A mode in which the passenger P is a general user and the user U is a provider of route guidance services, driving instruction services, etc. The user U can provide route guidance or driving instruction in places that are difficult to find with a navigation device or that are not on a map while viewing the scenery around the moving object M.

[0090] (C) A mode in which the passenger P is a celebrity and the user U is a general user, and a commercial virtual drive is provided to the user U. In this case, multiple users U are associated with one passenger P at the same time, and, for example, audio transfer from the user U side may be set to off.

[0091] <Processing for displaying users on rearview mirror display device> The first control device 170 may display a service image including a person image in which a person or a character corresponding to a person is seated in the back seat of a moving object on the display unit of the rearview mirror display device 152. When the service image is displayed on the display unit of the rearview mirror display device 152, the first control device 170 may control the first speaker 140 to output a sound from the direction of the back seat. The sound output from the direction of the back seat is, for example, a sound uttered by the user U to the second microphone 220, transmitted by the second communication device 210, and acquired via the first communication device 110. The first control device 170 outputs the sound to the first speaker 140 by localizing a sound image so that the sound can be heard from the back seat by an occupant in the driver's seat or the passenger seat, for example.

[0092] Fig. 9 is a diagram showing an example of a service image displayed on the display unit of the rearview mirror display device 152. As shown in Fig. 9, when a passenger visually checks the display unit of the rearview mirror display device 152, a service image including a person image is displayed on the display unit of the rearview mirror display device 152 as if the user U were actually seated in the back seat.

[0093] FIG. 10 is a diagram focusing on the service image. The service image IM2 is, for example, an image obtained by combining the person image IM1 and an image (background image) of the back seat. The person image IM1 is an image of a user U who uses the second device 200. The person image IM1 may be an image stored in advance in the storage device of the first device 100, or may be an image transmitted from the second device 200. The person image IM1 may be an image showing a character corresponding to the user U. The person image IM1 may be an image of an avatar corresponding to the user. The person image IM1 may be, for example, an image of a user after removing the VR goggles used by the user of the second device 200 by processing such as CG. The background image may be an image stored in advance in the storage device of the first device 100, or may be an image captured by the indoor camera 132. The display control unit 176 generates the service image IM2 by combining the person image IM1 with the background image so that, for example, the person in the person image IM1 is seated in the back seat in the background image. The process of combining images may be performed by the second device 200, and the first device 100 may obtain the combined image from the second device 200.

[0094] The service image may be displayed both while the moving body M is moving and while it is stopped, or, as described later, the service image may be displayed either while the moving body M is moving or while it is stopped. The quality of the service image displayed while the moving body M is moving may differ from the quality of the service image displayed while the moving body M is stopped. For example, the quality of one may be high and the quality of the other may be low. For example, the quality of the service image displayed while the moving body M is stopped may be superior to the quality of the service image displayed while the moving body M is moving. The service image having superior quality means that the service image is displayed so that it can be seen more clearly when viewed by the user. The service image having superior quality means, for example, that the frame rate, image quality, number of pixels, etc. are superior. Since the user does not keep his / her eyes on the display unit of the rearview mirror display device 152 while the moving body M is moving, and the state of wireless communication may be unstable while the moving body M is stopped, it is preferable to lower the quality of the service image while the moving body M is moving, and since the user can keep his / her eyes on the display unit of the rearview mirror display device 152 more than while the moving body M is moving, and the state of wireless communication may be stable more than while the moving body M is moving, it is preferable to control the quality as described above. Alternatively, the quality of the service image may be determined based on the quality of wireless communication. For example, when the quality of wireless communication is below a predetermined threshold, a service image of lower quality may be displayed. The quality of wireless communication is represented by an index obtained from information for evaluating the quality of wireless communication, such as radio wave strength, communication delay time, and communication speed. This makes it possible to secure communication resources for communication of audio data output from the first speaker 140 via the first communication device 110, and to maintain audio communication to a certain degree regardless of the quality of wireless communication.

[0095] (Example of processing) The first control device 170 may determine, depending on conditions, whether or not to display the service image IM2 on the rearview mirror display device 152. FIG. 11 is a flowchart showing an example of the flow of processing executed by the first control device 170.

[0096] First, the state determination unit 178 of the first control device 170 determines whether or not a condition is satisfied (step S100). Details of the condition will be described later. If the state determination unit 178 determines that the condition is satisfied, the display control unit 176 permits the service image IM2 (image of the user) to be displayed on the rearview mirror display device 152 (step S102). The permission means that the service image IM2 is displayed on the rearview mirror display device 152 when the user U or the passenger intends to display the service image IM2 on the rearview mirror display device 152. The intention means that the service image IM2 is instructed to be displayed on the rearview mirror display device 152 or a mode for displaying the service image IM2 on the rearview mirror display device 152 is selected.

[0097] If the state determination unit 178 determines that the condition is not satisfied, the display control unit 176 restricts the display of the service image IM2 (image of the user) on the rearview mirror display device 152 (step S104). Restricting means not displaying the service image IM2 on the rearview mirror display device 152, or restricting the display mode to a level lower than that when display is permitted. For example, the time for displaying the service image IM2 may be restricted. For example, the display control unit 176 may repeatedly display the rear image on the rearview mirror display device 152 for a first predetermined time, and then display the service image IM2 for a second predetermined time that is shorter than the first predetermined time. This ends the processing of one routine of this flowchart.

[0098] (About the conditions) 12 is a diagram showing an example of the condition. The condition is, for example, that the moving object M is stopped, that the shift position of the moving object M is in the parking position, that the moving object M is controlled by driving assistance or automatic driving (for example, the driving assistance or automatic driving in the first mode described above), etc. For example, when the driver does not have a high need to monitor the surroundings such as the rear, the condition is met and the display of the service image IM2 on the rearview mirror display device 152 is permitted.

[0099] (Example of image transition (part 1)) 13 is a diagram showing an example of the relationship between the transition of the image displayed on the rearview mirror display device 152 and the determination result of the state determination unit. When the above conditions are satisfied, the first control device 170 causes the rearview mirror display device 152 to display the service image IM2. When the above conditions are no longer satisfied at time T+1, the first control device 170 causes the rearview mirror display device 152 to display a rear image instead of the service image IM2. When the above conditions are satisfied at time T+2, the first control device 170 causes the rearview mirror display device 152 to display the service image IM2 instead of the rear image. Note that the sound is localized so that it can be heard from the back seat.

[0100] As described above, the first control device 170 changes the image displayed on the rearview mirror display device 152 according to the state of the moving object, so that the passenger can view an image suitable for the state of the moving object. In other words, the first control device 170 can improve entertainment value and provide information suitable for the driver.

[0101] The above-mentioned conditions may be changed (for example, may be selectively changed). For example, the conditions are that the moving body M is not stopped, that the shift position of the moving body M is a position that allows starting (not a parking position), that the moving body M is not controlled by driving assistance or automatic driving (for example, the driving assistance or automatic driving of the first mode described above) and that the driver is operating to control the traveling of the moving body. The first control device 170 displays a service image on the display unit when the moving body is moving, the shift position is a position that allows starting, or the driver is operating to control the traveling of the moving body. By setting the conditions in this way, when the moving body is moving or the driver is controlling the moving body, the service image is provided, so that the occupant can easily feel the presence of a person who is not inside the moving body. As a result, the first control device 170 can provide the occupant with an experience of moving with a person who is not inside the moving body.

[0102] (Example of image transition (part 2)) Fig. 14 is a diagram showing another example of the relationship between the transition of the image displayed on the rearview mirror display device 152 and the judgment result of the state judgment unit. In the example of Fig. 14, the control of the sound image localization is changed depending on whether or not the condition is satisfied. The following mainly describes the differences from the process explained in Fig. 13.

[0103] If the above conditions are no longer satisfied at time T+1, the first control device 170 stops localizing the sound image so that the sound can be heard from the rear seat, and starts localizing the sound image so that the sound can be heard from the passenger seat. If the above conditions are satisfied at time T+2, the first control device 170 stops localizing the sound image so that the sound can be heard from the passenger seat, and starts localizing the sound image so that the sound can be heard from the rear seat.

[0104] For example, when the above condition is not satisfied, the second device 200 is controlled so that the user U can view the scenery as if he / she were sitting in the passenger seat and looking around, and at this time, the sound is localized so that the driver in the driver's seat can hear the sound from the passenger seat. When the above condition is satisfied, an image of the user U is displayed on the rearview mirror display device 152 as if the user U were sitting in the back seat, and at this time, the sound is localized so that the driver in the driver's seat can hear the sound from the back seat. If the sound image is localized according to the position where the user is virtually sitting as described above, the passengers can feel the presence of the user U without feeling uncomfortable, and the sense of realism is enhanced. This further improves the entertainment value. Note that, even in (Specific example (part 2) of image transition), the conditions may be changed (for example, may be selectively changed). For example, the conditions include that the moving body M is not stopped, that the shift position of the moving body M is in a position that allows it to start (not a parking position), and that the moving body M is not controlled by driving assistance or autonomous driving (for example, the driving assistance or autonomous driving in the first mode described above) but is being controlled by the driver to drive the moving body.

[0105] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0106] 1. Information Processing Systems 100 1st device 110 First communication device 120 First Microphone 130 Camera unit 132 Indoor Camera 134 Outdoor Camera 140 1st Speaker 150 User display device 152 Rearview mirror display device 160 HMI 170 First control device 190 Controlled Equipment 192 Mobile Control Device 200 2nd device 210 Second communication device 220 2nd microphone 230 Detection Device 232 Directional direction detection device 234 Head Position Detection Device 236 Motion Sensor 240 2nd Speaker 250 Mobile image display device 260 HMI 270 Second control device 300 Management Server M Mobile object

Claims

1. An information processing device mounted on a moving body in which a passenger rides, a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that when the display unit is assumed to be a mirror and the person is seated in the rear seat of the moving body and an occupant in the driver's seat or passenger seat of the moving body views the display unit, the occupant in the rear seat is reflected in the mirror; a control device that controls a first speaker in the moving body so as to output sound from a direction of the rear seat when a service image including the person image in which the person or the character is seated in a rear seat of the moving body is displayed on the display unit, The control device controls the display of the service image based on a traveling state of the mobile body, a shift position of the mobile body, or a driving mode set to the mobile body. Information processing device.

2. An information processing device mounted on a moving body in which a passenger rides, a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that when the display unit is assumed to be a mirror and the person is seated in the rear seat of the moving body and an occupant in the driver's seat or passenger seat of the moving body views the display unit, the occupant in the rear seat is reflected in the mirror; a control device that controls a first speaker in the moving body so as to output sound from a direction of the rear seat when a service image including the person image in which the person or the character is seated in a rear seat of the moving body is displayed on the display unit, The control device includes: displaying a service image of a first quality on a display unit while the moving object is moving; while the moving object is stopped, displaying a service image of a second quality, the service image having a quality superior to that of the first quality, on the display unit; Information processing device.

3. An information processing device mounted on a moving body in which a passenger rides, a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that when the display unit is assumed to be a mirror and the person is seated in the rear seat of the moving body and an occupant in the driver's seat or passenger seat of the moving body views the display unit, the occupant in the rear seat is reflected in the mirror; a control device that controls a first speaker in the moving body so as to output sound from a direction of the rear seat when a service image including the person image in which the person or the character is seated in a rear seat of the moving body is displayed on the display unit, The control device includes: permitting the display device to display a service image including the person image when the moving object is not stationary; When the moving object is stopped, displaying an image captured by an imaging unit that captures an image behind the moving object is displayed without allowing the display device to display a service image including the person image. Information processing device.

4. a first communication device that communicates with a second communication device of a second device used by a user at a location different from the mobile body, the first communication device including a second communication device and a second microphone; the voice is a voice uttered by the user into the second microphone and acquired via the first communication device; The information processing device according to claim 1 .

5. The person image is an image of a user who uses the second device or an image showing a character corresponding to the user. The information processing device according to claim 4.

6. the control device generates the service image by combining the person image with the image of the rear seat, and causes the display device to display the generated service image. The information processing device according to claim 5 .

7. The first speaker outputs the sound by localizing a sound image so that the sound can be heard from the rear seat by an occupant in the driver's seat or the passenger seat.

4. The information processing device according to claim 1.

8. The first speaker includes a plurality of child first speakers arranged at different positions from each other, The control device adjusts the volumes and / or phase differences of the plurality of sub first speakers to localize a sound image so that a passenger in the driver's seat or the passenger seat can hear a sound from the rear seat. The information processing device according to claim 7.

9. The display unit of the display device is a rearview mirror provided so that an occupant in the driver's seat can visually check the rear of the vehicle, and displays an image of an imaging unit that images the rear.

4. The information processing device according to claim 1.

10. The rearview mirror is provided on a front windshield or a ceiling in a vehicle interior of the moving body in front of the occupant in the driver's seat when the occupant in the driver's seat is seated in the driver's seat of the moving body. The information processing device according to claim 9.

11. the control device causes the display device to display a service image including the person image when the moving object is not stopped; 4. The information processing device according to claim 1.

12. a first communication device that communicates with a second communication device of a second device used by a user at a location different from the mobile body, the first communication device including a second communication device and a second microphone; the voice is a voice uttered by the user into the second microphone and acquired via the first communication device; The control device includes: When the moving object is not stopped, a service image including the person image is displayed on the display device, and the first speaker is controlled to output the sound from a direction of the rear seat or the passenger seat, and then When the moving object stops, the control of the first speaker to output the sound is continued, and the display of the service image including the person image on the display device is stopped, and the image captured by the imaging unit is displayed on the display device. The information processing device according to claim 3 .

13. An information processing device mounted on a moving body in which a passenger rides, a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that when the display unit is assumed to be a mirror and the person is seated in the rear seat of the moving body and an occupant in the driver's seat or passenger seat of the moving body views the display unit, the occupant in the rear seat is reflected in the mirror; a control device that controls a first speaker in the moving body so as to output sound from a direction of the rear seat when a service image including the person image in which the person or the character is seated in a rear seat of the moving body is displayed on the display unit, The control device includes: permitting the display device to display a service image including the person image when a shift position of the moving object is not a parking position; when a shift position of the moving body is a parking position, displaying an image captured by an imaging unit that captures an image of the rear of the moving body without allowing the display device to display a service image including the person image; Information processing device.

14. a first communication device that communicates with a second communication device of a second device used by a user at a location different from the mobile body, the first communication device including a second communication device and a second microphone; the voice is a voice uttered by the user into the second microphone and acquired via the first communication device; The control device includes: When the shift position of the moving body is a position where the moving body can start, a service image including the person image is displayed on the display device, and the first speaker is controlled to output the sound from a direction of the rear seat or the passenger seat, and then When the shift position of the moving body is changed from a position where the moving body can start to a parking position, the control of the first speaker to output the sound is continued, and an image captured by the imaging unit is displayed on the display device. The information processing device according to claim 13.

15. An information processing device mounted on a moving body in which a passenger rides, a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that when the display unit is assumed to be a mirror and the person is seated in the rear seat of the moving body and an occupant in the driver's seat or passenger seat of the moving body views the display unit, the occupant in the rear seat is reflected in the mirror; a control device that controls a first speaker in the moving body so as to output sound from a direction of the rear seat when a service image including the person image in which the person or the character is seated in a rear seat of the moving body is displayed on the display unit, The control device includes: permitting the display device to display a service image including the person image when a mobile object control device executes driving assistance or automatic driving to control traveling of the mobile object; When a mobile body control device is not performing driving assistance or automatic driving and a driver of the mobile body is operating to control traveling of the mobile body, an image captured by an imaging unit that captures an image of the rear of the mobile body is displayed without allowing the display device to display a service image including the person image. Information processing device.

16. a first communication device that communicates with a second communication device of a second device used by a user at a location different from the mobile body, the first communication device including a second communication device and a second microphone; the voice is a voice uttered by the user into the second microphone and acquired via the first communication device; The control device includes: When a mobile object control device executes driving assistance or automatic driving to control the traveling of the mobile object, a service image including the person image is displayed on the display device, and the first speaker is controlled to output the sound from a direction of the rear seat or the passenger seat, and then, When the driving assistance or the automatic driving is ended and the moving body is controlled by an operation of the driver, the control of the first speaker to output the sound is continued, and an image captured by the imaging unit is displayed on the display device. The information processing device according to claim 15.

17. The moving object is a vehicle.

16. An information processing device according to claim 1, 13, or 15.

18. The control device includes: If the vehicle does not satisfy the condition, a service image including the person image is displayed on the display device, and the first speaker is controlled to output the sound from a direction of the rear seat; When the vehicle satisfies a condition, the display device is not caused to display a service image including the person image, and the first speaker is controlled to output the sound from a direction of the passenger seat; The condition is that the vehicle is stopped, the shift position of the vehicle is in a parking position, or the driver of the moving body is controlling the traveling of the moving body by operating the moving body without performing driving assistance or automatic driving. The information processing device according to claim 17.

19. An information processing device according to claim 4; the second communication device communicating with the first communication device; A user device having a second microphone for collecting the voice uttered by the user. The second communication device transmits the voice collected by the second microphone to the first communication device. Information processing system.

20. The user device comprises: A detection device for detecting a direction of orientation of the user; a camera unit provided on a predetermined seat of the moving body and having one or more cameras including at least an indoor camera capable of capturing an image of the inside of the moving body as seen from the predetermined seat; and a display device that displays an image captured by the camera unit that corresponds to the orientation direction as seen from a predetermined seat of the moving body.

20. The information processing system according to claim 19.

21. The second communication device transmits information of the orientation direction to the first communication device; the control device controls the first communication device to selectively transmit to the second communication device an image corresponding to the orientation direction acquired via the first communication device, among images captured by the camera unit; a display device of the user device displays an image corresponding to the orientation direction as seen from the specific seat, the image being acquired via the second communication device; 21. The information processing system according to claim 20.

22. the first communication device transmits an image captured by the camera unit to the second communication device; the user device selectively displays, on the display device, an image corresponding to the orientation direction among images captured by the camera unit; 21. The information processing system according to claim 20.

23. The information processing device further includes a first microphone that collects at least a voice uttered by the occupant, the user device further includes a second speaker that outputs a voice uttered by the occupant acquired via the second communication device, The first communication device transmits the sound collected by the first microphone to the second communication device.

21. The information processing system according to claim 20.

24. The computer a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that when a driver of a moving body in which a passenger is riding views the display unit while assuming that the display unit is a mirror and that the passenger is seated in a rear seat of the moving body, a service image including the person image in which the person or character is seated in the rear seat of the moving body is displayed so that the passenger is reflected in the mirror, a first speaker that outputs sound is controlled to output sound from the direction of the rear seat, Controlling display of the service image based on a traveling state of the mobile body, a shift position of the mobile body, or a driving mode set to the mobile body. Information processing methods.

25. The computer a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that when a driver of a moving body in which a passenger is riding views the display unit while assuming that the display unit is a mirror and that the passenger is seated in a rear seat of the moving body, a service image including the person image in which the person or character is seated in the rear seat of the moving body is displayed so that the passenger is reflected in the mirror, a first speaker that outputs sound is controlled to output sound from the direction of the rear seat, displaying a service image of a first quality on a display unit while the moving object is moving; while the moving object is stopped, displaying a service image of a second quality, the service image having a quality superior to that of the first quality, on the display unit; Information processing methods.

26. The computer a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that when a driver of a moving body in which a passenger is riding views the display unit while assuming that the display unit is a mirror and that the passenger is seated in a rear seat of the moving body, a service image including the person image in which the person or character is seated in the rear seat of the moving body is displayed so that the passenger is reflected in the mirror, a first speaker that outputs sound is controlled to output sound from the direction of the rear seat, permitting the display device to display a service image including the person image when the moving object is not stationary; When the moving object is stopped, displaying an image captured by an imaging unit that captures an image behind the moving object is displayed without allowing the display device to display a service image including the person image. Information processing methods.

27. A computer of an information processing device mounted on a vehicle in which a passenger boards, a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that the display unit is a mirror and a passenger in the driver's seat or passenger seat of the moving body is viewed while the display unit is being adjusted so that the passenger in the rear seat is reflected in the mirror; when a service image including the person image in which the person or character is seated in the rear seat of the moving body is displayed on the display unit, the display device controls a first speaker within the moving body to output sound from the direction of the rear seat; permitting the display device to display a service image including the person image when a shift position of the moving object is not a parking position; when a shift position of the moving body is a parking position, displaying an image captured by an imaging unit that captures an image of the rear of the moving body without allowing the display device to display a service image including the person image; Information processing methods.

28. A computer of an information processing device mounted on a vehicle in which a passenger boards, a display device having a display unit that displays a person image showing a person or a character corresponding to a person, the display unit being adjustable in position so that the display unit is a mirror and a passenger in the driver's seat or passenger seat of the moving body is viewed while the display unit is being adjusted so that the passenger in the rear seat is reflected in the mirror; when a service image including the person image in which the person or character is seated in the rear seat of the moving body is displayed on the display unit, the display device controls a first speaker within the moving body to output sound from the direction of the rear seat; permitting the display device to display a service image including the person image when a mobile object control device executes driving assistance or automatic driving to control traveling of the mobile object; When a mobile body control device is not performing driving assistance or automatic driving and a driver of the mobile body is operating to control traveling of the mobile body, an image captured by an imaging unit that captures an image of the rear of the mobile body is displayed without allowing the display device to display a service image including the person image. Information processing methods.

Images