Information processing system, information processing method, and program

Abstract

This invention provides an information processing system, an information processing method, and a program that can provide users with high-quality images with low latency. [Solution] The information processing system comprises a first device mounted on a mobile body and a second device that communicates with the first device. The first device captures an image of the scenery around the mobile body, extracts a first object from the captured image of the scenery, and transmits the first object to the second device. The second device receives the first object transmitted from the first device, reads the second object from a storage unit where the second object is pre-stored, and generates a superimposed image, which is an image in which the first object is superimposed on the read second object.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] When displaying an image of a scenery seen from a vehicle on a display, a technique of superimposing a virtual object (such as a historical scenery, digital signage, aurora, constellation, etc.) on a partial area of the scenery image is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, due to the data capacity of the image, communication speed, etc., the image quality may deteriorate or a delay may occur in communication.

[0005] One of the objects of the present application is to provide an information processing system, an information processing method, and a program that can provide a user with an image of low latency and high quality in order to solve the above problems. And it improves the immersion, satisfaction, sense of presence, etc. of the user who views the image, and by extension, supports a good connection between urban areas and local areas including the urban periphery, and creates new activities.

Means for Solving the Problems

[0006] The information processing system, information processing method, and program according to this invention adopt the following configuration. (1) A first aspect of the present invention is an information processing system comprising: a first device mounted on a mobile body; and a second device communicating with the first device, wherein the first device has a camera for capturing images of the scenery around the mobile body; an extraction unit for extracting a first object, which is at least one of a dynamic object and a static object, from the image of the scenery captured by the camera; and a transmission unit for transmitting the first object to the second device; and the second device has a receiving unit for receiving the first object transmitted from the first device; and a generation unit for reading a second object from a storage unit in which a second object, which is the other object among the dynamic object and the static object that is not the first object, is pre-stored, and generating a superimposed image, which is an image in which the first object is superimposed on the read second object.

[0007] (2) In a second aspect of the present invention, in the first aspect, the image from which the first object is extracted includes an image of the landscape taken in the past.

[0008] (3) A third aspect of the present invention is that, in the first or second aspect, the image from which the first object is extracted includes a stereoscopic map image provided by the stereoscopic map service.

[0009] (4) A fourth aspect of the present invention further comprises a third device that communicates with the second device in the first or second aspect, the third device having an input interface operated by a user, and the extraction unit extracts from the image as the first object one of the dynamic objects and the static objects selected by the user via the input interface.

[0010] (5) A fifth aspect of the present invention is, in the first or second aspect, the generation unit generates an image as the superimposed image in which a third object is superimposed on the read second object, wherein the third object includes a fictitious object, an object captured in a location other than the landscape, or an object extracted from an image of the landscape captured in the past.

[0011] (6) A sixth aspect of the present invention is an information processing method utilizing an information processing system comprising a first device mounted on a mobile body and a second device communicating with the first device, wherein the first device captures an image of the scenery around the mobile body, extracts a first object which is at least one of a dynamic object and a static object from the captured image of the scenery, transmits the first object to the second device, the second device receives the first object transmitted from the first device, reads the second object from a storage unit which has a pre-stored second object which is the other object among the dynamic object and the static object that is not the first object, and generates a superimposed image which is an image in which the first object is superimposed on the read second object.

[0012] (7) A seventh aspect of the present invention is a program to be executed by computers of a first device mounted on a mobile body and a second device communicating with the first device, the program comprising: capturing an image of the scenery around the mobile body; extracting a first object from the captured image of the scenery, which is at least one of a dynamic object and a static object; transmitting the first object to the second device; receiving the first object transmitted from the first device; reading the second object from a storage unit in which the second object, which is the other of the dynamic object and the static object that is not the first object, is stored in advance; and generating a superimposed image, which is an image in which the first object is superimposed on the read second object.

Advantages of the Invention

[0013] According to the above aspect, an image with low latency and high quality can be provided to the user. As a result, the sense of immersion, satisfaction, and presence of the user viewing the image can be improved.

Brief Description of the Drawings

[0014] [Figure 1] It is a diagram showing an example of the configuration of the information processing system 1 according to the embodiment. [Figure 2] It is a diagram showing an example of user data 360. [Figure 3] It is a diagram showing an example of static objects OBs determined in advance for each coordinate. [Figure 4] It is a diagram showing an example of the configuration of the mobile device 100 according to the embodiment. [Figure 5] It is a diagram showing an example of the arrangement of a part of the mobile device 100 in the mobile body M according to the embodiment. [Figure 6] It is a diagram showing an example of the configuration of the user device 200 according to the embodiment. [Figure 7] It is a diagram for explaining an image corresponding to the pointing direction. [Figure 8] It is a sequence diagram showing the flow of a series of processes of the information processing system 1 according to the embodiment. <ooo0078> [Figure 9] It is a diagram showing an example of a landscape image. [[ID=oo00080]] [Figure 10] It is a diagram showing an example of a dynamic object OBd. [Figure 11] It is a diagram showing an example of a static object OBs.

Modes for Carrying Out the Invention

[0015] Hereinafter, embodiments of an information processing system, an information processing method, and a program of the present invention will be described with reference to the drawings.

[0016] <Configuration of the Information Processing System and Configuration of the Information Processing Device> FIG. 1 is a diagram showing an example of the configuration of an information processing system 1 according to an embodiment. The information processing system 1 includes a mobile device 100, a user device 200, and an information processing device 300.

[0017] The mobile device 100 is mounted on a mobile body M on which a passenger P rides. The mobile body M is typically a vehicle, but may be any mobile body (e.g., a ship, a flying object) as long as a passenger P can ride on it. Also, the passenger P is mainly the driver of the mobile body, but may be a passenger other than the driver (e.g., a passenger in the passenger seat). The mobile device 100 is an example of a "first device".

[0018] The user device 200 is used by a user U at a location different from the mobile body M (it is not excluded that the location is accidentally close). The user device 200 is an example of a "third device".

[0019] Between the mobile device 100 and the user device 200, the voice collected by the microphone is transmitted to the other side and reproduced by the speaker. Thereby, a telephone call is made between the passenger P and the user U. Further, a part of the image captured by the camera unit of the mobile device 100 is displayed on the user device 200. Thereby, MR (Mixed Reality) is provided on the user device 200 side, and the user U can obtain a feeling (pseudo-riding experience) as if riding on the mobile body M while being at a location different from the mobile body M. Further, the passenger P can obtain a feeling as if the user U is actually riding on the mobile body M together by having a conversation with the user U who is pseudo-experiencing boarding the mobile body M via the mobile device 100. Hereinafter, the case of pseudo-experiencing as if the user U is actually riding on the mobile body M may be referred to as "pseudo-riding". The mobile device 100 and the user device 200 do not need to have a one-to-one relationship, and the information processing system 1 may operate by matching one of the plurality of mobile devices 100 and the plurality of user devices 200 in a one-to-many relationship. In the latter case, for example, one passenger P can communicate with a plurality of users U simultaneously or in order.

[0020] The mobile device 100, the user device 200, and the information processing device 300 each communicate with each other via a network NW. The network NW includes, for example, at least one of the following: the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), a mobile communication network, a cellular network, etc. The information processing device 300 may be implemented as a server device or storage device incorporated into a cloud computing system. In this case, the functions of the information processing device 300 may be realized by multiple server devices and storage devices in the cloud computing system. Furthermore, the mobile device 100 mounted on a mobile device M may realize its functions by cooperating with mobile devices 100 mounted on other mobile devices.

[0021] The information processing device 300 processes information provided from the mobile device 100 to the user device 200, and information provided from the user device 200 to the mobile device 100. The information processing device 300 is an example of a "second device".

[0022] As shown in Figure 1, the information processing device 300 includes, for example, a third communication device 310, a third control device 320, and a storage unit 350.

[0023] The third control device 320 includes an acquisition unit 321, a matching processing unit 322, a generation unit 323, a charge management unit 324, and a communication control unit 325. These components are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System On Chip), or by the cooperation of software and hardware. The program may be stored in advance in a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device with a non-transient storage medium), or it may be stored in a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is inserted into a drive device. The program may also be updated as needed via a network NW.

[0024] The third communication device 310 is a communication interface for connecting to the network NW. Communication between the third communication device 310 and the mobile device 100, and communication between the third communication device 310 and the user device 200, may be conducted in accordance with, for example, TCP / IP (Transmission Control Protocol / Internet Protocol). The third communication device 310 is an example of a "receiving unit".

[0025] The acquisition unit 321 acquires various information from the mobile device 100, the user device 200, or other external devices via the third communication device 310.

[0026] The matching processing unit 322 is implemented by a processor, such as a CPU, executing a program (set of instructions) stored in a storage medium. For example, when the third communication device 310 receives a matching request from user U via user device 200, or from crew member P via mobile device 100, the matching processing unit 322 refers to user data 360 to perform matching between matching user U and crew member P, and uses the third communication device 310 to send the communication identification information of crew member P's mobile device 100 to the user device 200 of the matched user U, and sends the communication identification information of user U's user device 200 to the mobile device 100 of the matched crew member P. After receiving these, more real-time communication can be performed between the mobile device 100 and the user device 200, for example, compliant with UDP (User Datagram Protocol).

[0027] Based on the various information acquired by the acquisition unit 321, the generation unit 323 generates information to be provided to the mobile device 100 and the user device 200, generates information indicating the processing results by the matching processing unit 322, and generates information indicating charge information (settlement information) managed by the charge management unit 324.

[0028] The fare management unit 324 manages the fares to be charged to user U based on the information provided to user U, and the fares to be charged to crew member P based on the information provided to crew member P of the mobile vehicle M. The fare management unit 324 may also manage, for example, the compensation paid to user U and crew member P in response to the information provided by user U and crew member P. The fare management unit 324 may also handle settlement processing for user U and crew member P.

[0029] The communication control unit 325 transmits various information (such as superimposed images) generated by the generation unit 323 to the mobile device 100 or to the user device 200 via the third communication device 310.

[0030] The memory unit 350 may be implemented using the various storage devices mentioned above, or an SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The memory unit 350 can store, for example, programs executed by the CPU, user data 360, and provided information DB 362.

[0031] Figure 2 is a diagram showing an example of user data 360. User data 360 includes a crew list 360A and a user list 360B. Crew list 360A includes, for example, a crew ID which is the identification information of a crew member P of a mobile vehicle M, their communication identification information (IP address, etc.), a user ID which is the identification information of the user U to be matched, information about the mobile vehicle the crew member is on, and information on whether or not the information can be provided as set by the crew member. Mobile vehicle information includes, for example, information about the equipment installed on the mobile vehicle M (installed equipment information) and vehicle class information which indicates the size and external shape of the mobile vehicle M. Mobile vehicle information may also include information about the current location, destination, and surrounding conditions of the mobile vehicle M (for example, driving on a road along the coast) which are transmitted from the mobile vehicle M at predetermined intervals. User list 360B includes, for example, a user ID which is the identification information of a crew member P which is the identification information of a crew member P to be matched, and user information which are matched together. User information may include information about physical characteristics (e.g., height and sitting height) and information that can predict physical characteristics (e.g., age). Availability information is information that the mobile device M can or cannot provide, and is set, for example, by the occupant P. Availability information may be set for each piece of equipment installed on the mobile device M, or for each user U. Examples of availability information include, but are not limited to, "Provision of images is permitted," "Provision of audio is not permitted," "Provision of interior images is permitted, but provision of exterior images is not permitted," "Provision of occupant images is not permitted," and "Use of navigation devices is not permitted." Availability information may also include a fee for enabling the provision of the information (service provision fee). User data 360 may be generated in any manner, not limited to the manner shown in Figure 2, as long as it includes this information.

[0032] The provided information DB362 stores various types of information provided to user U or crew member P. This information includes, for example, map information, POI (Point of Interest) information, and computer-generated images (e.g., computer graphics images of people, marks, symbols, icons, etc.). POI information, for example, is information about various shops, theme parks, and landmarks at specific locations and may be included in the map information. Audio information may also be included. Furthermore, the provided information DB362 may include advertising information. This advertising information may include, for example, advertisements related to the mobile vehicle M, advertisements related to user U or crew member P, and advertisements related to the products and services of various shops. The inserted advertising information is managed separately from indoor and outdoor information, and when archived and distributed later, it may differ from the real-time inserted advertising information (for example, closed shops may be updated, and the featured menus may be updated). Advertising information may include, for example, videos and audio.

[0033] The provided information DB362 may include predefined static objects OBs for each coordinate as map information and POI information. Static objects OBs are objects that are relatively stationary with respect to the road on which the moving object M travels (in other words, objects fixed to that location), and may include, for example, bridges, buildings, mountains, trees, tunnels, houses, traffic lights, streetlights, utility poles, guardrails, sound barriers, etc.

[0034] In contrast, the dynamic object OBd, which will be described later, is an object that is relatively movable with respect to the road on which the moving object M travels (in other words, an object that is not fixed in that location), and may include, for example, pedestrians, other moving objects, animals such as dogs and cats, clouds floating in the sky, and waves on the surface of the sea.

[0035] Figure 3 shows an example of static objects (OBs) predetermined for each coordinate system. For example, the coordinates of a bridge are associated with the bridge itself, its associated towers, main cables, and hanger ropes as static objects (OBs). Similarly, the coordinates of an office district are associated with office buildings as static objects (OBs).

[0036] [Configuration of the mobile device] Figure 4 is a diagram showing an example of the configuration of a mobile device 100 according to an embodiment. The mobile device 100 includes, for example, a first communication device 110, a first microphone 120, an external sensor 125, a camera unit 130, a first speaker 140, a first display 150, an HMI (Human-machine Interface) 160, a first control device 170, and a GNSS (Global Navigation Satellite System) receiver 180. The first control device 170 is connected to a controlled device 190 mounted on the mobile device M.

[0037] The first communication device 110 is a communication interface for communicating with the third communication device 310 of the information processing device 300 and the second communication device 210 of the user device 200, which will be described later, via a network NW. The first communication device 110 is an example of a "transmitting unit".

[0038] The first microphone 120 collects at least the voice emitted by the occupant P. The first microphone 120 may be installed inside the vehicle M and have the sensitivity to collect sounds from outside the vehicle M, or it may include a microphone installed inside the vehicle M and a microphone installed outside the vehicle M. Hereinafter, the voice information acquired by the microphone installed inside the vehicle may be referred to as "indoor voice information". The voice collected by the first microphone 120 is transmitted, for example, via the first control device 170 to the information processing device 300 or user device 200 by the first communication device 110. If it is not possible to set up a microphone installed outside the vehicle M, the indoor voice information may be processed based on driving information (vehicle speed, acceleration / deceleration, road vibration, etc.) and the surrounding driving environment to generate pseudo-outdoor voice information. It is also possible to record the positional relationship of the speaker relative to the vehicle M (whether the speaker is inside or outside the vehicle), and processing may be performed on the collected voice according to that positional relationship.

[0039] The external sensor 125 detects the position of objects around the moving object M. The external sensor 125 is, for example, a radar device, a LIDAR (Light Detection and Ranging) sensor, or various other proximity sensors. The radar device emits radio waves such as millimeter waves around the moving object M and detects the radio waves reflected by objects (reflected waves) to detect at least the position (distance and direction) of the objects. The radar device may also detect the position and velocity of objects using the FM-CW (Frequency Modulated Continuous Wave) method. The LIDAR sensor irradiates light (or electromagnetic waves with a wavelength close to light) around the moving object M, measures the scattered light, and detects the distance to the target based on the time from emission to reception. The irradiated light is, for example, pulsed laser light. The radar device and LIDAR sensor can be attached to any part of the moving object M. Alternatively, the external sensor 125 may detect surrounding objects using images captured by the outdoor camera 134 of the camera unit 130.

[0040] The camera unit 130 includes, for example, an indoor camera 132 and an outdoor camera 134. The first speaker 140 outputs audio spoken by user U, which is acquired via the first communication device 110. Details of the camera unit 130 and the arrangement of the first speaker 140 will be explained later with reference to Figure 5.

[0041] The first display 150 virtually displays user U as if the user U were present inside the mobile object M. For example, the first display 150 may display a hologram or display user U in a part of the mobile object M corresponding to a mirror or window.

[0042] The HMI160 consists of a touch panel and a voice response device (agent device), etc. The HMI160 receives various instructions from the occupant P to the mobile device 100 and provides various information to the occupant P.

[0043] The first control device 170 controls various parts of the mobile device 100. The first control device 170 includes, for example, an acquisition unit 172, an extraction unit 174, and a communication control unit 176. These functional units are realized, for example, by a processor such as a CPU executing a program (set of instructions). Some or all of these components may be realized by hardware (including circuitry) such as LSIs, ASICs, FPGAs, GPUs, and SOCs, or by the cooperation of software and hardware.

[0044] The acquisition unit 172 acquires, for example, voice data of the occupant P from the first microphone 120, image data from the camera unit 130, and position data of the mobile body M from the GNSS receiver 180. The image data includes images of the inside of the mobile body M captured by the indoor camera 132 and images of the outside of the mobile body M captured by the outdoor camera 134 (in other words, images of the scenery around the mobile body M).

[0045] If the acquired image data is a landscape image of the surroundings of the moving object M, the extraction unit 174 extracts the target object from that landscape image.

[0046] The object to be extracted is one of the two objects predetermined to be extracted, either a dynamic object (OBd) or a static object (OBs). For example, the object to be extracted is a dynamic object (OBd). As mentioned above, a dynamic object (OBd) can be a pedestrian, another moving object, an animal such as a dog or cat, clouds floating in the sky, or waves on the sea surface. The object to be extracted is an example of a "first object".

[0047] Note that the object to be extracted may be a static object OBs instead of a dynamic object OBd. Whether to use a dynamic object OBd or a static object OBs as the object to be extracted may be optimally and automatically determined by the extraction unit 174, or it may be manually determined by the user U.

[0048] For example, if the object to be extracted is a dynamic object OBD, the extraction unit 174 cuts out the dynamic object OBD from the landscape image and extracts a portion of the area cut out from the landscape image as the object to be extracted.

[0049] The communication control unit 176 may transmit various data acquired by the acquisition unit 172 to the user device 200 or the information processing device 300 via the first communication device 110. Furthermore, if the extraction unit 174 extracts an object to be extracted, the communication control unit 176 may transmit the extracted object (a portion of the landscape image) to the user device 200 or the information processing device 300 via the first communication device 110.

[0050] The GNSS receiver 180 determines the position of the mobile object M based on signals received from GNSS satellites. The position of the mobile object M may also be determined or supplemented by an INS (Inertial Navigation System) that utilizes the speed and acceleration of the mobile object M.

[0051] The controlled device 190 is, for example, an in-vehicle device such as a navigation system mounted on the mobile body M that guides the driver to a destination, or a driver assistance system that controls either or both of the steering and / or speed of the mobile body M to assist the driver P. The controlled device 190 includes, for example, a seat drive system that can adjust the seat's position (front, back, left, and right), orientation, and height. When viewing the video using the user device 200, if the camera unit 130 of the mobile body device 100 is attached to the seat, the impact on the video can be suppressed by prohibiting seat movement. Even if seat movement is permitted, processing such as angle of view conversion may be performed to ensure that the video is not affected when the seat is moved. Furthermore, if user U requests to view video outside the current angle of view of the camera unit 130, the seat drive system may be controlled as a request from the user device 200.

[0052] Figure 5 is a diagram showing an example of the arrangement of some parts of the mobile device 100 in the mobile body M according to the embodiment. The interior camera 132 is attached, for example, to the neck pillow of the passenger seat S2 via attachment 132A and is positioned slightly away from the backrest of the passenger seat S2 toward the direction of travel of the mobile body M. The interior camera 132 has a wide-angle lens and is capable of imaging the area represented by region 132B in the figure. The interior camera 132 can photograph not only the interior of the mobile body M but also the exterior through the window.

[0053] The outdoor camera 134 includes, for example, a plurality of sub-outdoor cameras 134-1 to 134-4. By combining the images captured by the plurality of sub-outdoor cameras 134-1 to 134-4, an image such as a panoramic image capturing the outside of the mobile body M is obtained. The outdoor camera 134 may also include (or in addition to) these a wide-angle camera mounted on the roof of the mobile body M. An indoor camera 132 capable of capturing the area behind the passenger seat S2 may be added, and the mobile body image described later may be generated as a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 by the first control device 170, or it may be generated as a 360-degree panoramic image by appropriately combining images captured by indoor cameras 132 and images captured by outdoor cameras 134.

[0054] The first speaker 140 outputs the voice of user U acquired via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, sub-first speaker 140-1 is located in the center of the instrument panel, sub-first speaker 140-2 is located at the left end of the instrument panel, sub-first speaker 140-3 is located at the right end of the instrument panel, sub-first speaker 140-4 is located at the bottom of the left door, and sub-first speaker 140-5 is located at the bottom of the right door. When the first control device 170 outputs the user U's voice to the first speaker 140, for example, it outputs sound at roughly the same volume from sub-first speakers 140-2 and 140-4, and turns off the other sub-first speakers, thereby localizing the sound image so that the occupant P seated in the driver's seat S1 hears the sound coming from the passenger seat S2. Furthermore, the method of sound image localization is not limited to volume adjustment; it may also be done by shifting the phase of the sound output by each sub-first speaker. For example, to localize the sound image so that it sounds like it is coming from the left, the timing of outputting sound from the left sub-first speaker should be slightly earlier than the timing of outputting the same sound from the right sub-first speaker.

[0055] Furthermore, when the first control device 170 outputs the voice of user U to the first speaker 140, it may direct the sound image to the occupant P so that the voice is heard from a position corresponding to the height of user U's head on the passenger seat S2, and then output the voice emitted by user U to the first speaker 140. In this case, the first speaker 140 needs to have multiple child first speakers 140-k (where k is a number of natural numbers) at different heights.

[0056] <User device configuration> Figure 6 is a diagram showing an example of the configuration of a user device 200 according to an embodiment. The user device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a second display 250, an HMI 260, and a second control device 270.

[0057] The second communication device 210 is a communication interface for communicating with the third communication device 310 of the information processing device 300 and the first communication device 110 of the mobile device 100 via the network NW.

[0058] The second microphone 220 collects the voice emitted by user U. The voice collected by the second microphone 220 is transmitted to the first communication device 110 by the second communication device 210 via, for example, the second control device 270.

[0059] The detection device 230 includes, for example, a direction detection device 232, a head position detection device 234, and a motion sensor 236. The detection device 230 is an example of an "input interface".

[0060] The direction detection device 232 is a device for detecting the direction of direction. The direction of direction is the orientation of the user U's face or gaze, or an orientation based on both. Hereinafter, the direction of direction will be assumed to be an angle in the horizontal plane, that is, an angle without a vertical component, but the direction of direction may also be an angle that includes a vertical component. The direction detection device 232 may include a physical sensor (e.g., an accelerometer, a gyroscope, etc.) attached to the VR goggles described later, or it may be an infrared sensor that detects multiple positions of the user U's head, or a camera that images the user U's head. In any case, the second control device 270 calculates the direction of direction based on the information input from the direction detection device 232. Since various technologies for this are publicly known, a detailed explanation will be omitted.

[0061] The head position detection device 234 is a device for detecting the position (height) of the user U's head. For example, one or more infrared sensors or optical sensors placed around the chair where the user U is seated can be used as the head position detection device 234. In this case, the second control device 270 detects the position of the user U's head based on the presence or absence of detection signals from one or more infrared sensors or optical sensors. Alternatively, the head position detection device 234 may be an accelerometer attached to the VR goggles. In this case, the second control device 270 detects the position of the user U's head by integrating the output of the accelerometer minus the acceleration due to gravity. The head position information obtained in this way is provided to the second control device 270 as height information. The user's head position may also be obtained based on the user U's operation of the HMI 260. For example, the user U may input their height numerically into the HMI 260, or they may input their height using a dial switch included in the HMI 260. In these cases, the head position, i.e., height information, is calculated from the height. Furthermore, user U may input discrete values ​​such as body size: large / medium / small, rather than continuous values, into the HMI260. In this case, height information will be obtained based on the information indicating body size. Alternatively, the user's head height may be obtained simply based on the average adult body size (which may be separated by gender) without specifically obtaining the user's head height.

[0062] The motion sensor 236 is a device for recognizing gesture operations performed by user U. For example, a camera that captures images of user U's upper body is used as the motion sensor 236. In this case, the second control device extracts characteristic points of user U's body (fingertips, wrists, elbows, etc.) from the images captured by the camera and recognizes user U's gesture operations based on the movement of these characteristic points.

[0063] The second speaker 240 outputs the voice emitted by occupant P, which is acquired via the second communication device 210. The second speaker 240 has a function, for example, to change the direction from which the voice is heard. The second control device 270 causes the second speaker to output voice so that the voice is heard by user U from the position of occupant P as seen from the passenger seat S2. The second speaker 240 may include multiple child second speakers 240-n (where n is a number of natural numbers), and sound localization may be achieved by the second control device 270 adjusting the volume of each child second speaker 240-n, or if headphones are attached to the VR goggles, sound localization may be achieved by utilizing the function of the headphones.

[0064] The second display 250 displays an image captured by the camera unit 130 (which may be an image that has undergone the aforementioned merging process, and will be referred to as a moving object image hereinafter). The second display 250 may also display an image of a moving object in a specific directional direction.

[0065] Figure 7 is a diagram illustrating the image corresponding to the direction of direction. In this example, the VR goggles 255 include a physical sensor as a direction detection device 232 and a head position detection device 234, and a second display 250. The second control device 270 detects the direction the VR goggles 255 are facing as the direction of direction φ, using, for example, the center of the user U's head or the center of the VR goggles 255 as Ω and a pre-calibrated direction as the reference direction. Since various methods for this function are already publicly known, a detailed explanation is omitted.

[0066] The second display 250 displays image A2 from the moving object image A1, within an angular range of plus or minus α centered on the directional direction φ, toward the user U. The moving object image A1 has an angle of approximately 240 degrees in the figure, but as mentioned above, the field of view may be expanded by the merging process.

[0067] HMI260 is a touch panel, voice response device (agent device), or one of the switches mentioned above. HMI260 receives various instructions from user U to user device 200. HMI260 is another example of an "input interface".

[0068] The second control unit 270 controls various parts of the user device 200. The second control unit 270 includes, for example, an acquisition unit 272, a display control unit 274, and a communication control unit 276. These functional units are realized, for example, by a processor such as a CPU executing a program (set of instructions). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, GPU, or SOC, or by the cooperation of software and hardware. The user device 200 may have all the functions shown in Figure 6 integrated into a VR goggle.

[0069] The acquisition unit 272 may, for example, acquire voice data of user U from the second microphone 220 or acquire detection data indicating the detection result from the detection device 230. The acquisition unit 272 may also acquire various information and data from the mobile device 100 or the information processing device 300 via the second communication device 210.

[0070] The display control unit 274 causes the second display 250 to display the image of the moving object.

[0071] The communication control unit 276 transmits various data acquired by the acquisition unit 272 to the mobile device 100 and the information processing device 300 via the second communication device 210.

[0072] <Sequence of information processing systems> Figure 8 is a sequence diagram showing a series of processing steps in the information processing system 1 according to this embodiment.

[0073] First, the acquisition unit 172 of the mobile device 100 acquires a landscape image of the surroundings of the mobile device M from the outdoor camera 134, and also acquires the position information (latitude and longitude) of the mobile device M at the time the landscape image was captured from the GNSS receiver 180 (step S100).

[0074] Figure 9 shows an example of a landscape image. IMG in the figure represents a landscape image, and this landscape image (IMG in the figure) shows the scenery when a mobile object M is traveling on a road laid on a bridge. Such landscape images may include other mobile objects traveling around the mobile object M, as well as bridge structures (main towers, main cables, hanger ropes, etc.).

[0075] Next, the extraction unit 174 of the mobile device 100 extracts the target object (for example, the dynamic object OBD) from the landscape image (step S102).

[0076] Figure 10 shows an example of a dynamic object OBD. For example, if the object to be extracted is a dynamic object OBD, the extraction unit 174 extracts other moving objects (adjacent vehicles or preceding vehicles) that appear in the landscape image from the landscape image and extracts these extracted other moving objects as the object to be extracted.

[0077] Next, the communication control unit 176 of the mobile device 100 transmits to the information processing device 300 via the first communication device 110 the extracted object extracted from the landscape image by the extraction unit 174 and the position information of the mobile device M at the time the landscape image was captured (step S104).

[0078] Next, the third communication device 310 of the information processing device 300 receives the location information of the object to be extracted and the mobile body M from the mobile device 100 (step S106). Furthermore, the acquisition unit 321 acquires the location information of the object to be extracted and the mobile body M from the third communication device 310.

[0079] Next, the generation unit 323 of the information processing device 300 generates an image in which the object to be extracted is superimposed on the object that is not to be extracted (hereinafter referred to as the superimposed image) (step S108).

[0080] Non-extractable objects are the other objects among the dynamic object OBD and the static object OBD that are not included in the extraction process. For example, if the object included in the extraction process is the dynamic object OBD, then the non-extractable object will be the static object OBD.

[0081] For example, the generation unit 323 reads static object OBs corresponding to the location information of the moving object M from among the multiple static object OBs stored in the storage unit 350 as provided information DB362. As illustrated in Figure 3, the storage unit 350 stores multiple static object OBs associated with coordinates as provided information DB362. For example, if the position of the moving object M when the landscape image was captured was exactly "AAA", the generation unit 323 reads static object OBs associated with "bridge" and associated with coordinate "AAA" from the storage unit 350.

[0082] Figure 11 shows an example of static objects OBs. As shown in the figure, towers, main cables, hanger ropes, etc., are extracted from the landscape image of the "bridge," and these are stored in the storage unit 350 as static objects OBs related to the "bridge."

[0083] The generation unit 323 then superimposes the dynamic object OBd, which was extracted as the object to be extracted, onto the static object OBs, which is the "bridge". This generates a superimposed image. The superimposed image may be a 360-degree panoramic image.

[0084] Next, the communication control unit 325 transmits the superimposed image to the user device 200 via the third communication device 310 (step S110).

[0085] Next, the second communication device 210 of the user device 200 receives the superimposed image from the information processing device 300 (step S112). Furthermore, the acquisition unit 272 acquires the superimposed image from the second communication device 210.

[0086] Next, the display control unit 274 of the user device 200 displays the superimposed image on the second display 250 (step S114). This completes the series of processes.

[0087] <Summary> According to the embodiments described above, the mobile device 100 (an example of the "first device") captures the scenery around the mobile body M and extracts an object to be extracted (an example of the "first object") from the scenery image, which is at least one of the dynamic object OBd and the static object OBs. The mobile device 100 then transmits the object to be extracted to the information processing device 300 (an example of the "second device").

[0088] The information processing device 300 receives the objects to be extracted transmitted from the mobile device 100 and reads the non-extraction target objects from the storage unit 350, which are the other objects among the dynamic object OBd and static object OBs that are not the objects to be extracted. The information processing device 300 generates a superimposed image, which is an image in which the objects to be extracted are superimposed on the read non-extraction target objects. Then, the information processing device 300 transmits the superimposed image to the user device 200 (an example of the "third device"). In this way, when transmitting a landscape image from the mobile device 100 to the information processing device 300, by transmitting only a portion of the landscape image rather than the entire area of ​​the landscape image, it is possible to provide a low-latency, high-quality image to user U. As a result, the immersion, satisfaction, and sense of presence of user U viewing the landscape image can be improved.

[0089] <Other embodiments (variant examples)> Other embodiments will be described below. In the embodiments described above, the objects to be extracted were mainly dynamic objects OBd (e.g., pedestrians and other moving objects), and the objects not to be extracted were static objects OBs (e.g., bridges and buildings), but the embodiments are not limited to this. For example, the objects to be extracted may be static objects OBs, and the objects not to be extracted may be dynamic objects OBd. In other words, the objects (a part of the landscape image) transmitted from the mobile device 100 to the information processing device 300 may be static objects OBs such as bridges and buildings. In this case, the storage unit 350 stores a plurality of dynamic objects OBd associated with coordinates as provided information DB362.

[0090] Furthermore, in the embodiments described above, the landscape image from which the target object is extracted was described as being captured in real time by the camera unit 130 while the moving body M was moving, but this is not limited to this. For example, the landscape image from which the target object is extracted may be an image captured by the camera unit 130 when the moving body M was moving in the past, that is, a past landscape image that is not in real time.

[0091] Furthermore, the landscape image from which the target objects are extracted may also be a 3D map image provided by a 3D map service.

[0092] Furthermore, in the embodiments described above, it was explained that whether the target object for extraction is a dynamic object OBd or a static object OBs is predetermined, but this is not limited to this. For example, whether the target object is a dynamic object OBd or a static object OBs may be dynamically changed according to the user U's preference.

[0093] For example, user U may use HMI260 to select whether to use a dynamic object OBd or a static object OBs as the object to be extracted. Alternatively, user U may wear VR goggles and gaze at the object they want to set as the object to be extracted to select it. Alternatively, user U may wear VR goggles and use gestures to select the object to be extracted.

[0094] In this case, the communication control unit 276 of the user device 200 transmits the object selection result input to the HMI 260 to the information processing device 300 via the second communication device 210, and transmits the operation result of user U detected by the detection device 230 to the information processing device 300 via the second communication device 210. In response, the third control device 320 of the information processing device 300 may determine that one of the dynamic object OBD and static object OBs selected by user U is the object to be extracted.

[0095] Furthermore, in the embodiments described above, it was explained that a superimposed image is generated in which a dynamic object OBd, which is an object to be extracted, is superimposed on a static object OBs, which is an object not to be extracted, or conversely, a superimposed image is generated in which a static object OBs, which is an object to be extracted, is superimposed on a dynamic object OBd, which is an object not to be extracted, but the invention is not limited to this.

[0096] For example, the generation unit 323 may generate a superimposed image by superimposing a dynamic object OBd, which is an object to be extracted, onto a static object OBs, which is an object not to be extracted, or by superimposing a third object instead.

[0097] Similarly, the generation unit 323 may generate a superimposed image by superimposing a static object OBs, which is an object to be extracted, onto a dynamic object OBd, which is an object not to be extracted, or by superimposing a third object instead.

[0098] The third object may include fictional objects (such as virtual advertisements or virtual creatures), objects included in landscapes photographed in locations other than where the landscape was photographed (such as planets or constellations in outer space photographed in space), and objects extracted from historical landscape images (such as ancient buildings).

[0099] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0100] 1. Information Processing System 100 Mobile device 110 First communication device 120 First Microphone 125 External Sensors 130 Camera Unit 132 Indoor Cameras 134 Outdoor Camera 140 First speaker 150 First Display 160 HMI 170 First control device 180 GNSS receiver 190 Controlled Equipment 200 User Devices 210 Second communication device 220 Second microphone 230 detection device 232 Directional detection device 234 Head position detection device 236 Motion Sensors 240 Second speaker 250 Second display 260 HMI 270 Second control device 300 Information Processing Devices 310 Third communication device 320 Third control device 350 Storage section M Mobile object

Claims

1. The first device mounted on the mobile vehicle, The system comprises a second device that communicates with the first device, The first apparatus is A camera that captures the scenery around the moving object. An extraction unit that extracts a first object, which is one of a dynamic object and one of a static object, from the image of the landscape captured by the camera, and It has a transmitting unit that transmits the first object to the second device, The second device is A receiving unit that receives the first object transmitted from the first device, and The system includes a storage unit that has a second object, which is the other object among the dynamic and static objects that is not the first object, stored in advance, and a generation unit that generates a superimposed image, which is an image in which the first object is superimposed on the read second object, The image from which the first object is extracted includes images of the landscape taken in the past. Information processing system.

2. The image from which the first object is extracted includes a 3D map image provided by the 3D map service. The information processing system according to claim 1.

3. The system further comprises a third device that communicates with the second device, The third device has an input interface operated by a user, The extraction unit extracts from the image, as the first object, one of the dynamic objects and the static objects selected by the user via the input interface. The information processing system according to claim 1.

4. The generation unit generates an image in which the third object is superimposed on the second object that was read out, and this image is called the superimposed image. The third object includes a fictitious object, an object photographed in a location other than the landscape, or an object extracted from a previously taken image of the landscape. The information processing system according to claim 1.

5. The first device mounted on the mobile vehicle, The system comprises a second device that communicates with the first device, The first apparatus is A camera that captures the scenery around the moving object. An extraction unit that extracts a first object, which is one of a dynamic object and one of a static object, from the image of the landscape captured by the camera, and It has a transmitting unit that transmits the first object to the second device, The second device is A receiving unit that receives the first object transmitted from the first device, and The system includes a storage unit that has a second object, which is the other object among the dynamic and static objects that is not the first object, stored in advance, and a generation unit that generates a superimposed image, which is an image in which the first object is superimposed on the read second object, The image from which the first object is extracted includes a 3D map image provided by the 3D map service. Information processing system.

6. The first device mounted on the mobile vehicle, The system comprises a second device that communicates with the first device, The first apparatus is A camera that captures the scenery around the moving object. An extraction unit that extracts a first object, which is one of a dynamic object and one of a static object, from the image of the landscape captured by the camera, and It has a transmitting unit that transmits the first object to the second device, The second device is A receiving unit that receives the first object transmitted from the first device, and The system includes a storage unit that has a second object, which is the other object among the dynamic and static objects that is not the first object, stored in advance, and a generation unit that generates a superimposed image, which is an image in which the first object is superimposed on the read second object, The system further comprises a third device that communicates with the second device, The third device has an input interface operated by a user, The extraction unit extracts from the image, as the first object, one of the dynamic objects and the static objects selected by the user via the input interface. Information processing system.

7. The first device mounted on the mobile vehicle, The system comprises a second device that communicates with the first device, The first apparatus is A camera that captures the scenery around the moving object. An extraction unit that extracts a first object, which is one of a dynamic object and one of a static object, from the image of the landscape captured by the camera, and It has a transmitting unit that transmits the first object to the second device, The second device is A receiving unit that receives the first object transmitted from the first device, and The system includes a storage unit that has a second object, which is the other object among the dynamic and static objects that is not the first object, stored in advance, and a generation unit that generates a superimposed image, which is an image in which the first object is superimposed on the read second object, The generation unit generates an image in which the third object is superimposed on the second object that was read out, and this image is called the superimposed image. The third object includes a fictitious object, an object photographed in a location other than the landscape, or an object extracted from a previously taken image of the landscape. Information processing system.

8. The first device mounted on the mobile vehicle, The system comprises a second device that communicates with the first device, The first apparatus is A camera that captures the scenery around the moving object. An extraction unit for extracting static objects from the image of the landscape captured by the camera, and The device includes a transmitting unit that transmits the static object to the second device, The second device is A receiving unit that receives the static object transmitted from the first device, and The system includes a storage unit that stores dynamic objects in advance, a generation unit that reads the dynamic objects from the storage unit that stores the dynamic objects in advance, and a generation unit that generates a superimposed image which is an image in which the static objects are superimposed on the read dynamic objects, Information processing system.

9. The first device mounted on the mobile vehicle, A second device that communicates with the first device, The system comprises a third device that communicates with the second device, The first apparatus is A camera that captures the scenery around the moving object. An extraction unit that extracts a first object, which is one of a dynamic object and one of a static object, from the image of the landscape captured by the camera, and It has a transmitting unit that transmits the first object to the second device, The second device is A receiving unit that receives the first object transmitted from the first device, A generation unit reads a second object, which is the other object among the dynamic and static objects that is not the first object, from a storage unit that has been stored in advance, and generates a superimposed image, which is an image in which the first object is superimposed on the read second object, and The system includes a matching processing unit that, upon receiving a request from a first user using the first device while aboard the mobile vehicle, or from a second user using the third device, performs matching between the first user and the second user, and transmits the identification information of the third device to the first device. Information processing system.

10. The first device mounted on the mobile vehicle, A second device that communicates with the first device, The system comprises a third device that communicates with the second device, The first apparatus is A camera that captures the scenery around the moving object. A microphone that acquires the voice of the first user riding in the mobile vehicle. An extraction unit that extracts a first object, which is one of a dynamic object and one of a static object, from the image of the landscape captured by the camera, and It has a first transmitting unit that transmits the voice of the first object and the first user to the second device, The second device is A first receiving unit that receives the first object and the voice of the first user transmitted from the first device, A generation unit reads a second object, which is the other object among the dynamic and static objects that is not the first object, from a storage unit that has been stored in advance, and generates a superimposed image, which is an image in which the first object is superimposed on the read second object, and It has a second transmitting unit that transmits the voice of the first user to the third device, The third device is A second receiving unit that receives the voice of the first user transmitted from the second device, and Having a speaker that outputs the voice of the first user, Information processing system.

11. An information processing method utilizing an information processing system comprising a first device mounted on a mobile body and a second device that communicates with the first device, The first device is The scenery around the moving object is captured, From the captured image of the landscape, a first object is extracted, which is either a dynamic object or a static object. The first object is transmitted to the second device, The second device is Upon receiving the first object transmitted from the first device, From a storage unit in which a second object, which is the other object among the dynamic object and the static object that is not the first object, is pre-stored, the second object is read out. A superimposed image is generated, which is an image in which the first object is superimposed on the second object that was read out. The image from which the first object is extracted includes images of the landscape taken in the past. Information processing methods.

12. An information processing method utilizing an information processing system comprising a first device mounted on a mobile body and a second device that communicates with the first device, The first device is The scenery around the moving object is captured, From the captured image of the landscape, a first object is extracted, which is either a dynamic object or a static object. The first object is transmitted to the second device, The second device is Upon receiving the first object transmitted from the first device, From a storage unit in which a second object, which is the other object among the dynamic object and the static object that is not the first object, is pre-stored, the second object is read out. A superimposed image is generated, which is an image in which the first object is superimposed on the second object that was read out. The image from which the first object is extracted includes a 3D map image provided by the 3D map service. Information processing methods.

13. An information processing method utilizing an information processing system comprising a first device mounted on a mobile body, a second device communicating with the first device, and a third device communicating with the second device, The first device is The scenery around the moving object is captured, From the captured image of the landscape, a first object is extracted, which is either a dynamic object or a static object. The first object is transmitted to the second device, The second device is Upon receiving the first object transmitted from the first device, From a storage unit in which a second object, which is the other object among the dynamic object and the static object that is not the first object, is pre-stored, the second object is read out. A superimposed image is generated, which is an image in which the first object is superimposed on the second object that was read out. The third device is Of the dynamic and static objects, one object selected by the user via the input interface is extracted from the image as the first object. Information processing methods.

14. An information processing method utilizing an information processing system comprising a first device mounted on a mobile body and a second device that communicates with the first device, The first device is The scenery around the moving object is captured, From the captured image of the landscape, a first object is extracted, which is either a dynamic object or a static object. The first object is transmitted to the second device, The second device is Upon receiving the first object transmitted from the first device, From a storage unit in which a second object, which is the other object among the dynamic object and the static object that is not the first object, is pre-stored, the second object is read out. A superimposed image is generated, which is an image in which the first object is superimposed on the second object that was read out. An image is generated as the superimposed image by superimposing the third object onto the second object that was read out. The third object includes a fictitious object, an object photographed in a location other than the landscape, or an object extracted from a previously taken image of the landscape. Information processing methods.

15. An information processing method utilizing an information processing system comprising a first device mounted on a mobile body and a second device that communicates with the first device, The first device is The scenery around the moving object is captured, From the captured image of the landscape, static objects are extracted. The static object is transmitted to the second device, The second device is Upon receiving the static object transmitted from the first device, The dynamic object is read from the storage unit where the dynamic object is pre-stored, A superimposed image is generated, which is an image in which the static object is superimposed on the read dynamic object. Information processing methods.

16. An information processing method utilizing an information processing system comprising a first device mounted on a mobile body, a second device communicating with the first device, and a third device communicating with the second device, The first device is The scenery around the moving object is captured, From the captured image of the landscape, a first object is extracted, which is either a dynamic object or a static object. The first object is transmitted to the second device, The second device is Upon receiving the first object transmitted from the first device, From a storage unit in which a second object, which is the other object among the dynamic object and the static object that is not the first object, is pre-stored, the second object is read out. A superimposed image is generated, which is an image in which the first object is superimposed on the second object that was read out. In response to a request received from a first user using the first device while aboard the mobile vehicle, or from a second user using the third device, the system matches the first user with the second user and transmits the identification information of the third device to the first device. Information processing methods.

17. An information processing method utilizing an information processing system comprising a first device mounted on a mobile body, a second device communicating with the first device, and a third device communicating with the second device, The first device is The scenery around the moving object is captured, The voice of the first user riding in the aforementioned mobile vehicle is acquired. From the captured image of the landscape, a first object is extracted, which is either a dynamic object or a static object. The first object and the voice of the first user are transmitted to the second device. The second device is The first device receives the first object and the voice of the first user transmitted from the first device, From a storage unit in which a second object, which is the other object among the dynamic object and the static object that is not the first object, is pre-stored, the second object is read out. A superimposed image is generated, which is an image in which the first object is superimposed on the second object that was read out. The voice of the first user is transmitted to the third device. The third device is The second device receives the voice of the first user transmitted from the second device. Outputting the voice of the first user, Information processing methods.

18. A program to be executed by the computers of a first device mounted on a mobile body and a second device that communicates with the first device, To capture images of the scenery around the moving object, Extracting a first object from the captured image of the landscape, which is either a dynamic object or a static object. Transmitting the first object to the second device, Receiving the first object transmitted from the first device, Reading the second object, which is the other object among the dynamic object and the static object that is not the first object, from a storage unit in which the second object is pre-stored, This includes generating a superimposed image, which is an image in which the first object is superimposed on the second object that was read out, The image from which the first object is extracted includes images of the landscape taken in the past. program.

19. A program to be executed by the computers of a first device mounted on a mobile body and a second device that communicates with the first device, To capture images of the scenery around the moving object, Extracting a first object from the captured image of the landscape, which is either a dynamic object or a static object. Transmitting the first object to the second device, Receiving the first object transmitted from the first device, Reading the second object, which is the other object among the dynamic object and the static object that is not the first object, from a storage unit in which the second object is pre-stored, This includes generating a superimposed image, which is an image in which the first object is superimposed on the second object that was read out, The image from which the first object is extracted includes a 3D map image provided by the 3D map service. program.

20. A program to be executed by the computers of a first device mounted on a mobile body, a second device that communicates with the first device, and a third device that communicates with the second device, To capture images of the scenery around the moving object, Extracting a first object from the captured image of the landscape, which is either a dynamic object or a static object. Transmitting the first object to the second device, Receiving the first object transmitted from the first device, Reading the second object, which is the other object among the dynamic object and the static object that is not the first object, from a storage unit in which the second object is pre-stored, A superimposed image is generated, which is an image in which the first object is superimposed on the second object that was read out. From the dynamic object and the static object, one of the objects selected by the user via the input interface is extracted from the image as the first object. A program that includes this.

21. A program to be executed by the computers of a first device mounted on a mobile body and a second device that communicates with the first device, To capture images of the scenery around the moving object, Extracting a first object from the captured image of the landscape, which is either a dynamic object or a static object. Transmitting the first object to the second device, Receiving the first object transmitted from the first device, Reading the second object, which is the other object among the dynamic object and the static object that is not the first object, from a storage unit in which the second object is pre-stored, A superimposed image is generated, which is an image in which the first object is superimposed on the second object that was read out. This includes generating an image as the superimposed image in which the third object is superimposed on the second object that was read out, The third object includes a fictitious object, an object photographed in a location other than the landscape, or an object extracted from a previously taken image of the landscape. program.

22. A program to be executed by the computers of a first device mounted on a mobile body and a second device that communicates with the first device, To capture images of the scenery around the moving object, Extracting static objects from the captured image of the landscape, To transmit the static object to the second device, Receiving the static object transmitted from the first device, Reading a dynamic object from a storage unit in which the dynamic object is pre-stored, A superimposed image is generated, which is an image in which the static object is superimposed on the read dynamic object. A program that includes this.

23. A program to be executed by the computers of a first device mounted on a mobile body, a second device that communicates with the first device, and a third device that communicates with the second device, To capture images of the scenery around the moving object, Extracting a first object from the captured image of the landscape, which is either a dynamic object or a static object. Transmitting the first object to the second device, Receiving the first object transmitted from the first device, Reading the second object, which is the other object among the dynamic object and the static object that is not the first object, from a storage unit in which the second object is pre-stored, A superimposed image is generated, which is an image in which the first object is superimposed on the second object that was read out. In response to a request received from a first user using the first device while aboard the mobile vehicle, or from a second user using the third device, the system matches the first user with the second user and transmits the identification information of the third device to the first device. A program that includes this.

24. A program to be executed by the computers of a first device mounted on a mobile body, a second device that communicates with the first device, and a third device that communicates with the second device, To capture images of the scenery around the moving object, To acquire the voice of the first user riding in the aforementioned mobile vehicle, Extracting a first object from the captured image of the landscape, which is either a dynamic object or a static object. Transmitting the voice of the first object and the first user to the second device, Receiving the first object and the voice of the first user transmitted from the first device, Reading the second object, which is the other object among the dynamic object and the static object that is not the first object, from a storage unit in which the second object is pre-stored, A superimposed image is generated, which is an image in which the first object is superimposed on the second object that was read out. Transmitting the voice of the first user to the third device, Receiving the voice of the first user transmitted from the second device, Outputting the voice of the first user, A program that includes this.

Images