Information processing system

Abstract

The present application provides an information processing system that enables a user at a different location from a mobile body to easily view an external view of the mobile body. The information processing system includes: a first device mounted on a mobile body for a passenger to ride; and a second device used by a user at a different location from the mobile body. The first device includes: a first communication device that communicates with a second communication device of the second device; and a camera unit that can capture an internal and external view of the mobile body. The second device includes: the second communication device that communicates with the first communication device; a display device that displays an image; and a second control device that switches a first mode and a second mode according to an operation of the user. The first mode causes the display device to display an image captured by the camera unit including a predetermined interior of the mobile body. The second mode causes the display device to display a replacement image that is an image in which a region of the predetermined interior in the image captured by the camera unit is replaced with an image of an external view of the mobile body.

Description

Technical Field

[0001] This invention relates to an information processing system. Background Technology

[0002] Previously, research was being conducted on communication and sharing of images of the scenery outside the vehicle between devices mounted on mobile bodies such as vehicles and devices used in locations different from mobile bodies (see Japanese Patent Application Publication No. 2020-94958). Summary of the Invention

[0003] In previous technologies, users in locations different from the mobile device sometimes had difficulty seeing the external scenery of the mobile device due to the influence of the interior decorations installed on the mobile device.

[0004] One of the objectives of this invention is to provide an information processing system that enables users in locations different from the mobile body to easily see the scenery outside the mobile body.

[0005] The information processing system of the first aspect of the present invention comprises: a first device mounted on a mobile body for passengers; a second device used by a user at a location different from the mobile body, wherein the first device includes: a first communication device that communicates with a second communication device of the second device; and a camera unit that includes at least an indoor camera mounted on a designated seat of the mobile body and capable of capturing images of the interior and exterior of the mobile body; the second device includes: a second communication device that communicates with the first communication device; a display device that displays images captured by the camera unit; and a second control device that switches between a first mode and a second mode according to the user's operation, wherein the first mode causes the display device to display an image captured by the camera unit that includes a designated interior of the mobile body, and the second mode causes the display device to display a replacement image, which is an image formed by replacing the area of ​​the designated interior in the image captured by the camera unit with an image of the exterior of the mobile body captured by the camera unit.

[0006] The second approach, based on the information processing system of the first approach, may also involve the second control device recommending either the first mode or the second mode based on the driving state of the moving body.

[0007] The third option, based on the information processing system of the second option, may also involve the second control device recommending either the first mode or the second mode based on images of the scenery outside the moving body captured by the camera unit.

[0008] The fourth option, based on the information processing system of the second option, may also involve the second control device recommending either the first mode or the second mode based on whether the moving body is entering the garage.

[0009] The fifth option, based on the information processing system of the second option described above, may also involve the second control device recommending either the first mode or the second mode based on whether other moving objects are approaching the moving object.

[0010] The sixth option, based on the information processing system of the first option, may also include an outdoor camera located outside the moving body, and the display device displaying the replacement image in the second mode. The replacement image is an image formed by replacing the specified interior area in the image captured by the indoor camera with an image of the exterior of the moving body captured by the outdoor camera.

[0011] The seventh embodiment, based on the information processing system of the first embodiment, may also include the following: the first device further comprises: a first speaker that outputs sound emitted by the user obtained via the first communication device; and a first microphone that collects sound emitted by the occupant; the second device further comprises: a second speaker that outputs sound emitted by the occupant obtained via the second communication device; and a second microphone that collects sound emitted by the user; the first communication device transmits the sound collected by the first microphone to the second communication device, and the second communication device transmits the sound collected by the second microphone to the first communication device.

[0012] The eighth solution, based on the information processing system of the first solution, may also include a detection device for detecting the user's pointing direction, wherein the display device displays an image from the image captured by the camera unit that corresponds to the pointing direction when viewed from the designated seat.

[0013] The ninth solution, based on the information processing system of the eighth solution, may also involve the second communication device sending the pointing direction information to the first communication device. The first device further includes a first control device that selectively sends images captured by the camera unit that correspond to the pointing direction obtained via the first communication device to the second communication device. The display device of the second device displays the image obtained via the second communication device that corresponds to the pointing direction as viewed from the designated seat.

[0014] The tenth solution, based on the information processing system of the eighth solution, may also involve the first communication device sending the image captured by the camera unit to the second communication device, and the second control device causing the display device to selectively display the image in the image captured by the camera unit that corresponds to the pointing direction.

[0015] The eleventh option, based on the information processing system of the eighth option above, may also include a display device that is a VR glasses, i.e., a virtual reality glasses display device, and a detection device that includes physical sensors installed on the VR glasses.

[0016] The twelfth option, based on the information processing system of the first option, may also be that the moving body is a vehicle, and the specified interior is installed on the pillars or roof of the vehicle.

[0017] The thirteenth option, based on the information processing system of the first option, may also involve the display device replacing the portion of the image captured by the camera unit that shows a specified item in the room containing the moving body with an image drawn by computer processing for display.

[0018] According to the first to thirteenth schemes mentioned above, users in locations different from the mobile body can easily see the scenery outside the mobile body. Attached Figure Description

[0019] Figure 1 It is a diagram illustrating the usage environment of an information processing system and a management server.

[0020] Figure 2 This is a diagram illustrating an example of the content of user data.

[0021] Figure 3 This is a structural diagram of the first device.

[0022] Figure 4 This is a diagram showing an example configuration of a portion of the first device in a moving body.

[0023] Figure 5 This is a structural diagram of the second device.

[0024] Figure 6 It is a diagram used to illustrate the image corresponding to the pointing direction.

[0025] Figure 7 This is a diagram illustrating a first example of the functional structure of the first control device and the second control device.

[0026] Figure 8 This is a second example of the functional structure of the first control device and the second control device.

[0027] Figure 9 This is an example of an image shown in normal mode.

[0028] Figure 10 This is an example of a permutation image in a permutation pattern.

[0029] Figure 11 This is a diagram illustrating an example of how a permuted image is displayed. Detailed Implementation

[0030] Hereinafter, embodiments of the information processing system of the present invention will be described with reference to the accompanying drawings. The information processing system includes: a first device mounted on a mobile body for passengers; and a second device used by a user at a location different from the mobile body. The mobile body is, for example, a vehicle, but can be any mobile body that can be used for passengers. Furthermore, the passenger is primarily the driver of the mobile body, but can also be a passenger other than the driver.

[0031] Between the first and second devices, a telephone call is conducted where sound collected by a microphone is transmitted to the other side and reproduced by a speaker. Furthermore, a portion of the image captured by the camera unit of the first device is displayed by the second device, thus providing MR (Mixed Reality) on the second device side. The first and second devices do not need to be in a one-to-one relationship; multiple first devices can be matched with multiple second devices in a one-to-many relationship, thereby operating as an information processing system. In the latter case, for example, it is possible for one occupant to communicate simultaneously with multiple users or sequentially with multiple users.

[0032] <Basic Structure>

[0033] Figure 1 This diagram illustrates the usage environment of the information processing system 1 and the management server 300. The information processing system 1 includes: a first device (mobile device) 100 mounted on a mobile body M; and a second device (user device) 200 used by a user U in a location different from the mobile body M (excluding a location that happens to be near the mobile body). The first device 100, the second device 200, and the management server 300 communicate with each other via a network NW. The information processing system 1 may or may not include the management server 300.

[0034] The management server 300 includes, for example, a communication device 310, a matching processing unit 320, and a storage unit 350. User data 360 is stored in the storage unit 350.

[0035] The communication device 310 is a communication interface for connecting to the network NW.

[0036] It should be noted that the communication between the communication device 310 and the first device 100, and the communication between the communication device 310 and the second device 200, are conducted, for example, according to TCP / IP (Transmission Control Protocol / Internet Protocol).

[0037] The matching processing unit 320 executes a program (command group) stored in the storage medium by a processor such as a CPU (Central Processing Unit). The storage unit 350 is RAM (Random Access Memory), HDD (Hard Disk Drive), flash memory, etc.

[0038] Figure 2 This diagram illustrates an example of the content of user data 360. User data 360 includes, for example, a passenger list 360A that establishes a correspondence between the passenger ID (identification information of passenger P), their communication identification information (IP address, etc.), and the user ID (identification information of user U who is the matching target); and a user list 360B that establishes a correspondence between the user ID, their communication identification information (IP address, etc.), and the passenger P who is the matching target. User data 360 can be any data containing this information, and may not be limited to... Figure 2 The scheme shown can be generated from any scheme.

[0039] When the communication device 310 receives a matching request from user U via second device 200 or from passenger P via first device 100, the matching processing unit 320 performs matching between user U and passenger P with reference to user data 360. It then uses the communication device 310 to send communication identification information of passenger P's first device 100 to the matched user U's second device 200, and also sends communication identification information of user U's second device 200 to the matched passenger P's first device 100. Upon receiving these requests, the first device 100 and second device 200, for example, perform real-time communication based on UDP (User Datagram Protocol).

[0040] Figure 3 This is a structural diagram of the first device 100. The first device 100 includes, for example, a first communication device 110, a first microphone 120, a camera unit 130, a first speaker 140, a user display device 150, an HMI (Human Machine Interface) 160, and a first control device 170. The first control device 170 is connected to the control object device 190 mounted on the mobile body M.

[0041] The first communication device 110 is a communication interface used to communicate with the communication device 310 of the management server 300 and the second communication device 210 of the second device 200 via the network NW.

[0042] The first microphone 120 collects at least the sound emitted by the occupant P. The first microphone 120 may be a microphone installed indoors in the mobile body M and having sensitivity capable of collecting sound from outside the mobile body M, or it may include a microphone installed indoors in the mobile body M and a microphone installed outside the mobile body M. The sound collected by the first microphone 120 is transmitted, for example, via the first control device 170 to the second communication device 210 through the first communication device 110.

[0043] Camera unit 130 includes at least an indoor camera 132, and may also include an outdoor camera 134. First speaker 140 outputs sound emitted by user U, acquired via first communication device 110. Details regarding the configuration of camera unit 130 and first speaker 140 will be provided later. Figure 4 Please provide an explanation.

[0044] The user display device 150 imagines the user U as if the user U were present inside the interior of the mobile body M. For example, the user display device 150 displays a holographic image or displays the user U in a portion corresponding to the rearview mirror or window of the mobile body M.

[0045] HMI160 includes a touch panel, a voice response device (smart agent device), etc. HMI160 receives various instructions from occupant P to the first device 100.

[0046] The first control device 170 includes, for example, a processor such as a CPU and a storage medium connected to the processor and storing a program (command group), and controls various parts of the first device 100 by executing the command group through the processor.

[0047] The controlled object device 190 is, for example, a navigation device or driving support device mounted on the mobile body M.

[0048] Figure 4 This diagram illustrates an example configuration of a portion of the first device 100 in the mobile vehicle M. An interior camera 132 is mounted, for example, via accessory 132A, to the neck pillow of the passenger seat S2 (an example of a "prescribed seat"), and is positioned slightly offset from the backrest of the passenger seat S2 towards the direction of travel of the mobile vehicle M. The interior camera 132 has a wide-angle lens, capable of capturing images of the area shown in the shaded region 132B in the diagram. The interior camera 132 can capture images not only of the interior of the mobile vehicle M but also of the exterior through the vehicle windows. In the following description, the passenger seat S2 is assumed to be a prescribed seat, but the prescribed seat could also be a rear seat or other seats.

[0049] The outdoor camera 134 may include, for example, multiple sub-outdoor cameras 134-1 to 134-4. By combining the images captured by the multiple sub-outdoor cameras 134-1 to 134-4, a panoramic image of the exterior of the moving body M can be obtained. The outdoor camera 134 may also replace these (or, based on these), include a wide-angle camera mounted on the roof of the moving body M.

[0050] It should be noted that, as an indoor camera 132, an additional camera capable of taking pictures of the rear of the co-driver's seat S2 can be added. The moving body image described later can be generated as a 360-degree panoramic image by combining images taken by one or more indoor cameras 132 through the first control device 170, or it can be generated as a 360-degree panoramic image by appropriately combining images taken by indoor cameras 132 with images taken by outdoor cameras 134.

[0051] The first speaker 140 outputs the sound of the user U obtained via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, sub-first speaker 140-1 is located in the center of the dashboard, sub-first speaker 140-2 is located at the left end of the dashboard, sub-first speaker 140-3 is located at the right end of the dashboard, sub-first speaker 140-4 is located at the lower part of the left door, and sub-first speaker 140-5 is located at the lower part of the right door. When the first speaker 140 outputs the sound of the user U, the first control device 170, for example, outputs sound from sub-first speakers 140-2 and 140-4 at the same volume level, while shutting off the other sub-first speakers, thereby performing sound image localization relative to the occupant P seated in the driver's seat S1 so that the sound can be heard from the front passenger seat S2. Furthermore, the sound image localization method is not limited to volume adjustment; it can also be achieved by staggering the phase of the sounds output from each sub-first speaker. For example, in the case of sound image localization in a manner that can be heard from the left, the timing of the sound output from the left sub-first speaker is slightly earlier than the timing of the same sound output from the right sub-first speaker.

[0052] Alternatively, the first control device 170 can also, when the first speaker 140 outputs the user's voice, perform sound image localization relative to the occupant P so that the sound can be heard from a position on the co-driver's seat S2 at a height corresponding to the height of the user's head, thereby causing the first speaker 140 to output the user's voice. In this case, the first speaker 140 needs to have multiple sub-first speakers 140-k (k being multiple natural numbers) with different heights.

[0053] Figure 5This is a structural diagram of the second device 200. The second device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a moving body image display device 250, an HMI 260, and a second control device 270. The detection device 230 includes, for example, a pointing direction detection device 232, a head position detection device 234, and a motion sensor 236.

[0054] The second communication device 210 is a communication interface used to communicate with the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0055] The second microphone 220 collects the sound emitted by the user U. The sound collected by the second microphone 220 is transmitted to the first communication device 110 via the second communication device 210 through the second control device 270.

[0056] The pointing direction detection device 232 is a device for detecting the pointing direction. The pointing direction is the direction of the user U's face or gaze, or the direction obtained based on both the user U's face and gaze. Alternatively, the pointing direction can also be indicated by the user U's arm or finger movements, such as tilting the terminal device or swiping the screen. Hereinafter, the pointing direction is assumed to be an angle in the horizontal plane, that is, an angle without vertical components, but the pointing direction can also be an angle that includes vertical components. The pointing direction detection device 232 may include physical sensors (such as accelerometers, gyroscopes, etc.) installed in VR (Virtual Reality) glasses described later, infrared sensors that detect multiple positions of the user U's head, or a camera that captures images of the user U's head. In any case, the second control device 270 calculates the pointing direction based on the information input from the pointing direction detection device 232. Various techniques are known in this field, so detailed descriptions are omitted.

[0057] The head position detection device 234 is a device for detecting the position (height) of the user U's head. For example, one or more infrared or optical sensors disposed around the chair where the user U sits may be used as the head position detection device 234. In this case, the second control device 270 detects the position of the user U's head based on the presence or absence of detection signals detected by the one or more infrared or optical sensors. Alternatively, the head position detection device 234 may also be an accelerometer sensor mounted on VR glasses. In this case, the second control device 270 may also detect the position of the user U's head by integrating the data obtained by subtracting the acceleration due to gravity from the output of the accelerometer. The head position information obtained in this way is provided to the second control device 270 as height information.

[0058] The user's head position can also be obtained based on the user's (U) operations on the HMI260. For example, the user (U) can input their height digitally into the HMI260, or they can input their height using a DIP switch included in the HMI260. In these cases, the head position, i.e., height information, is calculated based on the height. Alternatively, the user (U) can input a discrete value, such as body size (large / medium / small), into the HMI260 instead of a continuous value. In this case, height information is obtained based on the body size information. Alternatively, the user's head height can be obtained simply based on the typical adult body size (which may differ between male and female) without specifically obtaining the user's head height.

[0059] The motion sensor 236 is a device for recognizing the posture operations performed by the user U. For example, a camera that captures images of the user U's upper body is used as the motion sensor 236. In this case, the second control device 270 extracts feature points (fingertips, wrists, elbows, etc.) of the user U's body from the images captured by the camera, and recognizes the user U's posture operations based on the movement of these feature points.

[0060] The second speaker 240 outputs the sound emitted by occupant P, obtained via the second communication device 210. The second speaker 240, for example, has the function of changing the direction in which the sound can be heard. The second control device 270 causes the second speaker to output sound relative to the user U in a manner that allows occupant P to hear the sound from the position observed from the co-pilot's seat S2. The second speaker 240 includes a plurality of sub-second speakers 240-n (n being a plurality of natural numbers). Sound image localization can be achieved by the second control device 270 adjusting the volume of each sub-second speaker 240-n, or, in the case of VR glasses with accompanying headphones, by utilizing the function of the headphones for sound image localization.

[0061] The moving body image display device 250 displays an image from the viewpoint of the co-driver's seat that corresponds to the pointing direction of the image captured by the camera unit 130 (which may be an image that has undergone the aforementioned combination processing, hereinafter referred to as the moving body image). Figure 6 This is a diagram used to illustrate the image corresponding to the pointing direction. In the example shown in this diagram, the VR glasses 255 include physical sensors serving as a pointing direction detection device 232 and a head position detection device 234, as well as a moving body image display device 250. The second control device 270 uses a pre-calibrated direction as a reference direction to detect the direction in which the VR glasses 255 is facing as the pointing direction. Various methods are already known regarding the relevant functions, so detailed explanations will be omitted.

[0062] The moving object image display device 250 displays the moving object image A1 (which has an angle of about 240 degrees in the figure, but the field of view can also be expanded by combining processing as described above) in the direction of the pointing. The image A2, centered on the positive and negative α angular range, is displayed facing the user U.

[0063] HMI260 is a touch panel, a voice-responsive device (smart device), or a switch as described above. HMI260 accepts various instructions from user U to the second device 200.

[0064] The second control device 270 includes, for example, a processor such as a CPU and a storage medium connected to the processor and storing a program (command group), and controls various parts of the second device 200 by executing the command group through the processor.

[0065] <Functional Structure>

[0066] The functional structures of the first control device 170 and the second control device 270 are described below.

[0067] [First example]

[0068] Figure 7This diagram illustrates a first example of the functional structure of the first control device 170 and the second control device 270. In this first example, the first control device 170 includes a matching request / acceptance unit 171, a sound output control unit 172, an image transmission unit 173, and a mounting device cooperation unit 174. The second control device 270 includes a matching request / acceptance unit 271, a sound output control unit 272, a pointing direction detection unit 273, a head position detection unit 274, a posture input detection unit 275, an image editing unit 276, and a moving body image display control unit 277. These functional units are implemented, for example, by a processor such as a CPU executing a program (command group). Some or all of these components can be implemented using hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or through the coordinated use of software and hardware.

[0069] The matching request / approval unit 171 uses HMI 160 to accept input of a matching request from occupant P and sends it to management server 300, or uses HMI 160 to accept input of approval for a matching request received from management server 300 and sends it to management server 300. The matching request / approval unit 171 controls the first communication device 110 in such a way that the second device 200 of the user U, with whom a match has been established, is designated as the communication counterpart.

[0070] The sound output control unit 172 controls the first speaker 140 as described above.

[0071] The image transmitting unit 173 uses the first communication device 110 to transmit the moving body image A1 to the second device 200.

[0072] The equipment coordination unit 174 controls the controlled device 190 based on the instruction signal input from the second device 200.

[0073] The matching request / approval unit 271 uses HMI 260 to accept input of a matching request from user U and sends it to management server 300, or uses HMI 260 to accept input of approval for a matching request received from management server 300 and sends it to management server 300. The matching request / approval unit 271 controls the second communication device 210 in such a way that the first device 100 of the occupant P who has been matched is designated as the communication counterpart.

[0074] The sound output control unit 272 controls the second speaker 240 as described above.

[0075] The pointing direction detection unit 273 detects the pointing direction based on the output of the pointing direction detection device 232. The head position detection unit 274 detects the user U's head position based on the output of the head position detection device 234. The head position can be represented as three-dimensional coordinates, or only the head height can be detected as the head position. The posture input detection unit 275 detects the user U's posture input based on the output of the motion sensor 236.

[0076] Image editing unit 276 performs the process of cropping the image of the moving object A1 from the perspective of the co-pilot's seat and the direction of the pointing. Processing of the corresponding image A2 ( Figure 6 The moving body image display control unit 277 causes the moving body image display device 250 to display image A2 captured by the image editing unit 276. At this time, the image editing unit 276 can also cause the moving body image display device 250 to display the viewing height and pointing direction indicated by the height information of the user U's head. The corresponding image.

[0077] [Second Example]

[0078] Figure 8 This is a second example of a diagram illustrating the functional structure of the first control device 170 and the second control device 270. (Compared to...) Figure 7 The difference between the first example and the second example is that the first control device 170 includes an image editing unit 175, while the second control device 270 does not include an image editing unit 276 but includes a direction sending unit 278. The other components function essentially the same as in the first example, so further explanation is omitted.

[0079] The direction transmission unit 278 uses the second communication device 210 to transmit the direction detected by the direction detection unit 273 to the first device 100.

[0080] Image editing unit 175 extracts the image from the moving object image A1, showing the direction of view from the co-pilot's seat. Processing of image A2 corresponding to the pointing direction sent from the second device 200 ( Figure 6 At this time, the image editing unit 175 can also obtain the height information of the user U's head from the second device 200 in advance, and capture the angle and direction of the view from the height shown in the height information. The corresponding image A2 is processed.

[0081] In the second example, the image transmitting unit 173 uses the first communication device 110 to transmit the image A2 captured by the image editing unit 175 to the second device 200. Furthermore, the moving body image display control unit 277 causes the moving body image display device 250 to display the image A2 transmitted from the first device 100.

[0082] <Normal Mode>

[0083] Figure 9 This diagram illustrates an example of image A2 in normal mode. Normal mode (first mode) is a mode that displays image A2, including the image of the interior of the moving body M captured by camera unit 130. Image A2 is displayed on the moving body image display device 250. As previously described, image A2 is a cropped image from moving body image A1, viewed from the passenger seat and in the direction of orientation. The corresponding image.

[0084] exist Figure 9 In the example shown, image A2 includes an image of a pillar 10, which is a defined interior component, installed on the mobile body M. The pillar 10 is a column-shaped interior component that connects the body and roof of the mobile body M. Here, as an example of the pillar 10, a front pillar located at the front of the mobile body M will be described, but the pillar 10 can also be a rear pillar located at the rear of the mobile body M.

[0085] like Figure 9 As shown, since the view outside the mobile body M is blocked by the support column 10, it is difficult for the user U to see the view outside the mobile body M. Therefore, in this embodiment, the user U can select a transmission mode that allows the view outside the mobile body M to be seen through the interior of the mobile body M, such as the support column 10.

[0086] Specifically, user U selects the transmission mode icon 11 included in image A2, thereby switching from normal mode to transmission mode. For example, the posture input detection unit 275 of the second control device 270 can detect that user U has selected the transmission mode icon 11 with a finger based on the output of motion sensor 236. The second control device 270 can also switch from normal mode to transmission mode when the posture input detection unit 275 detects that the transmission mode icon 11 has been selected.

[0087] <Through the pattern>

[0088] Figure 10This diagram illustrates an example of a displacement image A3 in the transmission mode. The transmission mode (second mode) is the mode for displaying the displacement image A3, which is an image formed by replacing a specified interior area (e.g., pillar 10) in the image A2 captured by the camera unit 130 with an image of the exterior of the moving body M captured by the camera unit 130. The displacement image A3 is displayed on the moving body image display device 250.

[0089] like Figure 10 As shown, by replacing the area of ​​the support column 10 with an image of the exterior of the moving body M captured by the camera unit 130, the user U can easily see the scenery outside the moving body M. The image of the exterior of the moving body M can also be an image captured by the outdoor camera 134. In this case, the moving body image display device 250 can also display the replaced image A3 in transmission mode, which is an image formed by replacing the area of ​​the support column 10 in the image A2 captured by the indoor camera 132 with an image of the exterior of the moving body M captured by the outdoor camera 134.

[0090] For example, in the first case mentioned above, the permuted image A3 is generated by the image editing unit 276.

[0091] Specifically, the image transmitting unit 173 of the first control device 170 uses the first communication device 110 to transmit the image of the moving body A1 and an image of the exterior of the moving body M to the second device 200. The image editing unit 276 of the second control device 270 performs image editing from the moving body image A1, capturing the image from the perspective of the co-pilot's seat and the pointing direction. The corresponding image A2 is processed to generate a replacement image A3, which replaces the region of the pillar 10 in image A2 with the image of the outside of the moving body M.

[0092] On the other hand, in the second example mentioned above, the replacement image A3 is generated by the image editing unit 175. Specifically, the image editing unit 175 of the first control device 170 performs a cropping operation from the moving body image A1, showing the direction of view from the passenger seat. The image A2 corresponding to the direction sent from the second device 200 is processed to generate a replacement image A3, which replaces the area of ​​the pillar 10 in image A2 with the outside of the moving body M. The image sending unit 173 sends the replacement image A3 generated by the image editing unit 175 to the second device 200 using the first communication device 110.

[0093] It should be noted that the designated interior trim of the mobile body M is located on the vehicle's pillar 10, but it is not limited to this. For example, the designated interior trim of the mobile body M could also be the roof. In this case, the mobile body image display device 250 can also display a displacement image A3 in transmission mode. This displacement image A3 is an image formed by replacing the area of ​​the roof in image A2 captured by the indoor camera 132 with an image of the exterior of the mobile body M captured by the outdoor camera 134 located on the roof. This allows the user U to more easily see the scenery outside the mobile body M.

[0094] Alternatively, if only the indoor camera 132 can capture images of the exterior of the moving body M within the area of ​​the support pillar 10, the outdoor camera 134 may not be used. For example, the first control device 170 may pre-accumulate the images A2 continuously captured by the indoor camera 132 in a storage device, and use the accumulated past images A2 to generate an image of the exterior of the moving body M within the area of ​​the support pillar 10. In this case, the moving body image display device 250 may also display a replacement image A3 in a transmission mode, which is an image formed by replacing the area of ​​the support pillar 10 in the image A2 captured by the indoor camera 132 with an image of the exterior of the moving body M generated by the first control device 170.

[0095] Furthermore, when the user U moves their head to protrude from the window of the mobile body M, the second control device 270 can also display the image of the exterior of the mobile body M captured by the camera unit 130 on the mobile body image display device 250. Additionally, even when the user U stands up and moves their head to a position higher than the roof, the image of the exterior of the mobile body M captured by the camera unit 130 can still be displayed on the mobile body image display device 250. This makes it easier for the user U to see the scenery outside the mobile body M.

[0096] <Pattern Recommendation Processing>

[0097] Next, the mode recommendation process will be explained. The second control device 270 can also recommend either a normal mode (first mode) or a through mode (second mode) based on the driving state of the mobile body M. For example, the second control device 270 can display a message recommending either the normal mode or the through mode on the mobile body image display device 250, or it can output the message from the second speaker 240. Alternatively, the through mode can be recommended by highlighting the through mode icon 11.

[0098] For example, the second control device 270 can also recommend either a normal mode or a through-view mode based on images of the scenery outside the moving body M captured by the camera unit 130. Specifically, when the moving body M is traveling along the coast, and the user U can see the beautiful scenery, the second control device 270 can also recommend the through-view mode. In this case, the second control device 270 can also use a learned model that takes an image of the scenery outside the moving body M as input and outputs a value indicating whether it is a beautiful view to determine whether the user U can see the beautiful scenery. The learned model can be generated, for example, using AI functions such as machine learning (neural network) or deep learning that utilize teacher (positive solution) data. Thus, it is possible for the user U to easily see the beautiful scenery.

[0099] Additionally, for example, the second control device 270 can recommend either a normal mode or a through mode based on whether the mobile body M is entering the garage. Specifically, the second control device 270 can also recommend the through mode when the mobile body M is entering the garage. For example, the second control device 270 can determine whether the mobile body M is entering the garage based on instructions from the occupant P via the HMI 160, or it can determine whether the mobile body M is entering the garage based on images of the exterior of the mobile body M captured by the camera unit 130. Thus, the user U can easily monitor the status of the mobile body M entering the garage.

[0100] Additionally, for example, the second control device 270 can recommend either a normal mode or a penetration mode based on whether other moving objects are approaching the moving object M. Specifically, the second control device 270 can also recommend the penetration mode when other moving objects such as emergency vehicles (ambulances, etc.) or bicycles are approaching the moving object M. For example, the second control device 270 can also determine whether other moving objects are approaching the moving object M based on images of the exterior of the moving object M captured by the camera unit 130. Thus, the user U can easily grasp the positional relationship between the moving object M and other moving objects.

[0101] <Other Implementation Methods>

[0102] In information processing system 1, the scenario where user U can visually confirm any direction from the passenger seat S2 is described. However, it is also possible to restrict the direction that user U can visually confirm through a decision made during matching. For example, from the occupant P's side, the scenery in the direction of travel of the moving vehicle M or the scenery on the side opposite to the driver's seat S1 can be provided, but sometimes there may be requests where the occupant does not wish to display their own posture. This is because it is assumed that occupant P and user U are not family or friends, but rather respond to a need to confirm the driving feel of the moving vehicle M or to visually confirm the desired street scenery. In this case, when the matching processing unit 320 of the management server 300 performs matching processing, such a restriction is set, and the first control device 170 or the second control device 270 hides the angle range from which visual confirmation is not performed, or avoids facing the pointing direction, according to the setting. The restricted direction is corrected. Furthermore, information related to such restrictions concerns the privacy of occupant P, and therefore can also be set on one side of the first device 100.

[0103] In addition, the moving body image display device 250 can also replace the portion of a specified object in the room where the moving body M is reflected in the image captured by the camera unit 130 with an image (CG image) drawn by computer processing for display. Figure 11 This diagram illustrates an example of displaying a displacement image. In the diagram, OB is a display device for navigation displays, etc., and is an example of a "prescribed item." When an image obtained by capturing a picture of the display screen of a display device is directly displayed, the image may become blurry or visual clarity may be reduced due to light reflection. Therefore, the mobile image display device 250 can also obtain data from the first device for the display screen used to construct the related display device, or image data generated by computer processing in the mobile body M, and display an image generated by computer processing based on the data, or an image captured by the camera unit 130 (after editing) embedded with the obtained image data. In this case, the positions of items within the room of the mobile body M, which is the prescribed item, are pre-shared between the first device 100 and the second device, and the mobile image display control unit 277 displays the image based on, for example, the pointing direction. The system determines whether the image displayed by the moving body image display device 250 contains a specified item. If it determines that the item contains such an item, the image is replaced as described above. Furthermore, the "specified item" can be the head or face of the occupant P. In this case, the CG image, such as the head, can also be changed depending on the display of the occupant P.

[0104] Summary

[0105] The information processing system 1 configured as described above includes: a first device 100 mounted on a mobile body M for passenger P to ride in; and a second device 200 used by user U in a different location from the mobile body M. The first device 100 includes: a first communication device 110 that communicates with a second communication device 210 of the second device 200; and a camera unit 130 that includes at least an indoor camera 132 installed on a designated seat of the mobile body M and is capable of capturing images of the interior and exterior of the mobile body M. The second device 200 includes: a second communication device 210 that communicates with the first communication device 110; a mobile image display device 250 that displays images captured by the camera unit 130; and a second control device 270 that switches between a normal mode (first mode) and a through-view mode (second mode) according to the user's operation. In the normal mode (first mode), the mobile image display device 250 displays an image A2 captured by the camera unit 130, including a specified interior area (such as pillar 10) of the mobile body M. In the through-view mode (second mode), the mobile image display device 250 displays a replacement image A3, which is an image formed by replacing the specified interior area in image A2 captured by the camera unit 130 with an image of the exterior of the mobile body M captured by the camera unit 130. Thus, a user U located in a different place from the mobile body M can easily see the scenery outside the mobile body M.

[0106] <Example>

[0107] Information processing system 1 can be utilized in the following schemes.

[0108] (A) The occupant P and the user U are family members, friends, etc., and the virtual driving solution is provided to the user U. The user U can talk to the occupant P about the scenery around the mobile body M while watching the images.

[0109] (B) A scheme in which passenger P is a general user and user U is a provider of road guidance services, driving instruction services, etc. User U is able to receive road guidance and driving instruction in places that are difficult to understand with navigation devices or are not shown on maps while viewing the scenery around the mobile body M.

[0110] (C) A scheme in which virtual driving with a commercial tone is provided to user U, where passenger P is a celebrity and user U is a general user. In this case, there are multiple users U simultaneously for each passenger P, and for example, the transmission of sound from user U can be set to be turned off.

[0111] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. An information processing system, comprising: The first device, mounted on a mobile body for passengers to ride in; and The second device is used by the user in a different location than the mobile body, wherein... The first device has: The first communication device communicates with the second communication device of the second device; as well as The camera unit includes at least an indoor camera mounted on a designated seat of the mobile body, capable of capturing images of both the interior and exterior of the mobile body. The second device has: The second communication device communicates with the first communication device; A display device that displays images captured by the camera unit; as well as The second control device switches between a first mode and a second mode according to the user's operation. The first mode causes the display device to display an image captured by the camera unit, including a specified interior of the moving body. The second mode causes the display device to display a displacement image, which is an image formed by replacing the area of ​​the specified interior in the image captured by the camera unit with an image of the exterior of the moving body captured by the camera unit.

2. The information processing system according to claim 1, wherein, The second control device recommends either the first mode or the second mode based on the driving state of the mobile body.

3. The information processing system according to claim 2, wherein, The second control device recommends either the first mode or the second mode based on images of the landscape outside the moving body captured by the camera unit.

4. The information processing system according to claim 2, wherein, The second control device recommends either the first mode or the second mode based on whether the moving body is entering the garage.

5. The information processing system according to claim 2, wherein, The second control device recommends either the first mode or the second mode based on whether other moving objects are approaching the moving object.

6. The information processing system according to claim 1, wherein, The camera unit also includes an outdoor camera disposed outside the moving body. The display device displays the displacement image in the second mode. The displacement image is an image formed by replacing the specified interior area in the image captured by the indoor camera with an image of the exterior of the moving body captured by the outdoor camera.

7. The information processing system according to claim 1, wherein, The first device also has: A first speaker outputs sound emitted by the user, obtained via the first communication device; and The first microphone collects the sounds emitted by the occupants. The second device also has: A second speaker outputs the sound emitted by the occupant, obtained via the second communication device; as well as The second microphone collects the sound emitted by the user. The first communication device transmits the sound collected by the first microphone to the second communication device. The second communication device sends the sound collected by the second microphone to the first communication device.

8. The information processing system according to claim 1, wherein, The second device also includes a detection device for detecting the user's pointing direction. The display device displays an image from the camera unit that corresponds to the pointing direction when viewed from the designated seat.

9. The information processing system according to claim 8, wherein, The second communication device sends the pointing direction information to the first communication device. The first device further includes a first control device, which controls the first communication device to selectively transmit images captured by the camera unit that correspond to the pointing direction obtained via the first communication device to the second communication device. The display device of the second device displays an image obtained via the second communication device that corresponds to the pointing direction when viewed from the designated seat.

10. The information processing system according to claim 8, wherein, The first communication device sends the image captured by the camera unit to the second communication device. The second control device causes the display device to selectively display images from the images captured by the camera unit that correspond to the pointing direction.

11. The information processing system according to claim 8, wherein, The display device is a VR glasses, i.e., a virtual reality glasses display device. The detection device includes physical sensors installed on the VR glasses.

12. The information processing system according to claim 1, wherein, The moving body is a vehicle. The specified interior trim is installed on the pillars or roof of the vehicle.

13. The information processing system according to claim 1, wherein, The display device replaces a portion of a designated object within an interior space containing the moving object in an image captured by the camera unit with an image rendered by computer processing for display.

Images