Information processing system, information processing method, and program

Abstract

To obtain an excellent display image by superimposing a virtual space image matching a real space image on the real space image.SOLUTION: The image generation unit generates a virtual space image corresponding to the imaging range of the imaging unit in the real space on the basis of ComputerGraphics (CG) data. For example, a virtual space image is generated by installing a virtual imaging unit in the virtual space in association with an imaging unit in the real space and imaging a CG object in the virtual space. The image superimposing unit superimposes the virtual space image on the real-space image obtained by imaging the object in the real space with the imaging unit to obtain the display image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present technology relates to an image processing device and an image processing method, and more particularly to an image processing device and an image processing method for obtaining a display image by superimposing a virtual space image on a real space image. [Background technology]

[0002] Mixed reality (MR) is a known imaging technology that combines the real world with virtual reality to create a spatial representation that blends reality and virtuality. Patent Document 1 discloses a technology for a wearable device with a one-to-one correspondence between a camera and a display, in which the position and orientation of an object in a virtual space that is superimposed on a captured image are corrected based on the orientation of the device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-162136 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present technology is to obtain a good image for display by superimposing a virtual space image that matches a real space image. [Means for solving the problem]

[0005] The concept of this technology is: an image generation unit that generates a virtual space image corresponding to the imaging range of the imaging unit in real space based on CG (Computer Graphics) data; an image superimposing unit that obtains an image for display by superimposing the virtual space image on a real space image obtained by capturing an image of an object in the real space with the imaging unit; It is in the image processing device.

[0006] In the present technology, an image generation unit generates a virtual space image corresponding to the imaging range of an imaging unit in real space based on CG data. Then, an image superimposition unit superimposes the virtual space image on a real space image obtained by imaging an object in real space with the imaging unit to obtain a display image. For example, the present technology may further include an imaging unit in real space. Also, for example, the present technology may further include a display unit that displays an image based on the display image.

[0007] For example, the image generation unit may be configured to install a virtual imaging unit in the virtual space corresponding to the imaging unit in the real space, and generate a virtual space image by imaging a CG object in the virtual space with the virtual imaging unit, thereby making it possible to appropriately obtain a virtual space image that matches the real space image obtained by imaging the object in the real space with the imaging unit.

[0008] In this case, for example, the image generation unit may set up the virtual imaging unit in the virtual space in accordance with the position, orientation, and angle of view of the imaging unit in the real space, which makes it easy to set up the virtual imaging unit in the virtual space in correspondence with the imaging unit in the real space.

[0009] In this case, for example, the imaging unit in real space may be attached to a vehicle, and the position and orientation of the imaging unit in real space may be determined based on information about the position and orientation of the vehicle in real space and the attachment position and attachment angle of the imaging unit on the vehicle. This makes it possible to correctly determine the position and orientation of the imaging unit in real space when the imaging unit in real space is attached to a vehicle.

[0010] Here, for example, the vehicle may further include a position estimation unit that estimates the position of the vehicle in real space. This makes it possible to obtain information on the position of the vehicle in real space. And, for example, the vehicle may further include an information storage unit that stores position-attitude correspondence information for determining the vehicle attitude based on the estimated vehicle position in real space. This makes it possible to determine the vehicle attitude based on the estimated vehicle position.

[0011] In this way, with this technology, a virtual space image corresponding to the imaging range of the imaging unit in real space is generated based on CG data, and the virtual space image is superimposed on the real space image obtained by imaging an object in real space with the imaging unit to obtain an image for display.By superimposing a virtual space image that matches the real space image, it is possible to obtain a good image for display.

[0012] In the present technology, for example, the image generation unit may change the CG data in response to environmental changes in the real space, thereby changing the virtual space image superimposed on the real space image in response to environmental changes, thereby enabling a more effective display image to be obtained.

[0013] Furthermore, in the present technology, for example, the image generation unit may generate a plurality of virtual space images corresponding to the imaging ranges of a plurality of imaging units in real space based on the same CG data, and the image superimposition unit may obtain a plurality of display images by superimposing the corresponding virtual space images onto a plurality of real space images obtained by imaging an object in real space with the plurality of imaging units.

[0014] This makes it possible to obtain multiple good display images by superimposing a suitable virtual space image on each of multiple real space images. In this case, the multiple virtual space images are generated based on the same CG data, and it becomes possible to superimpose on each of the multiple real space images a virtual space image that captures the same virtual space (CG space) from the same viewpoint as the real space image.

[0015] In this case, for example, the image generation unit and the image superimposition unit may be configured as a single processing unit that holds CG data.Alternatively, in this case, for example, the image generation unit and the image superimposition unit may be configured as a plurality of processing units, and each processing unit may generate a virtual space image corresponding to the imaging range of the imaging unit in real space based on CG data that it holds or CG data that is held in a common server, and obtain an image for display by superimposing the virtual space image on a real space image obtained by imaging an object in real space with the imaging unit.

[0016] Furthermore, in the present technology, for example, the imaging unit in the real space may be attached to the outside of the vehicle to capture a window image, and the display image may be displayed on a display unit for displaying the window image that is arranged inside the vehicle. In this case, it becomes possible to display on the display unit a window image that changes in accordance with changes in the position and attitude of the vehicle (a display image in which a virtual space image that matches the real space image is superimposed).

[0017] Another concept of the present technology is generating a virtual space image corresponding to an imaging range of an imaging unit in real space based on CG (Computer Graphics) data; a step of superimposing the virtual space image on a real space image obtained by capturing an image of an object in the real space with the imaging unit to obtain an image for display; The image processing method. [Brief explanation of the drawings]

[0018] [Figure 1] 1A and 1B are diagrams showing an example of the appearance of a vehicle to which the present technology is applied, and an example of the layout of displays inside the vehicle. [Figure 2] FIG. 1 is a diagram showing an example of the relationship between a camera attached to the front surface of the exterior of a vehicle, a display provided on the front surface of the interior of the vehicle, and a PC serving as a processing unit. [Figure 3] FIG. 1 is a diagram schematically illustrating a real space and a corresponding virtual space. [Figure 4]FIG. 1 is a block diagram illustrating an example of the configuration of an image processing device mounted on a vehicle. [Figure 5] 10 is a flowchart showing an example of a procedure for generating a display image executed by a PC, for example, at a frame cycle. [Figure 6] FIG. 10 is a block diagram showing another example of the configuration of an image processing device mounted on a vehicle. [Figure 7] FIG. 10 is a block diagram showing another example of the configuration of an image processing device mounted on a vehicle. [Figure 8] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer. [Figure 9] 1 shows an example of the configuration of a taxi to which the present technology is applied. [Figure 10] 1 shows an example of a bus configuration to which the present technology is applied. [Figure 11] FIG. 10 is a diagram showing an example of a virtual space image displayed on three displays arranged on the right side of the interior of a bus at time t1 and the subsequent time t2. [Figure 12] FIG. 1 is a diagram illustrating a train to which the present technology is applied. DETAILED DESCRIPTION OF THE INVENTION

[0019] The following describes modes for carrying out the invention (hereinafter referred to as "embodiments") in the following order: 1. Embodiment 2. Variations

[0020] <1. Embodiment> FIG. 1(a) shows an example of the appearance of a vehicle 10 equipped with an image processing device of the present technology. The vehicle 10 is an autonomous vehicle or a remotely controlled vehicle, and is not provided with windows. A display unit (display) 11 is provided on the exterior of the vehicle 10. This display unit 11 is provided to display advertisements and the like.

[0021] Furthermore, cameras 12a, 12b, 12c, and 12d serving as imaging units for capturing images of the front, right, left, and rear directions are attached to the front, right, left, and rear sides, respectively, of the exterior of vehicle 10. Furthermore, displays 13a, 13b, 13c, and 13d serving as display units for displaying window images are arranged on the front, right, left, and rear sides of the interior of vehicle 10, in positions where windows would be in a conventional vehicle, as shown in FIG.

[0022] In this technology, a display image obtained by superimposing a virtual space image on a real space image is displayed as a window image on each of the displays 13a, 13b, 13c, and 13d. In this case, the display image displayed on each display is obtained by superimposing a virtual space image corresponding to the imaging range of a corresponding camera on a real space image obtained by capturing an object in real space with the corresponding camera. In this way, a display image is obtained in which a matching virtual space image is superimposed on the real space image, and a window image that does not look strange to passengers can be provided.

[0023] The image generation process for generating a virtual space image and the image superimposition process for superimposing this virtual space image on a real space image are performed, for example, by a PC (Personal Computer), as will be described later.

[0024] FIG. 2 shows an example of the relationship between a camera 12a attached to the front surface of the exterior of a vehicle 10, a display 13a provided on the front surface of the interior of the vehicle 10, and a PC 14 serving as a processing unit.

[0025] The camera 12a captures an image of an object in real space to obtain a real space image (an image in front of the vehicle) and sends it to the PC 14. The PC 14 generates a virtual space image corresponding to the imaging range of the camera 12a based on CG data. The PC 14 also generates a display image by superimposing the virtual space image on the real space image sent from the camera 12a and sends it to the display 13a. The display 13a displays the display image. In the illustrated example, the display 13a displays virtual space objects (CG objects) such as fish and turtles together with real space objects such as roads.

[0026] The virtual space images included in the display images of each display are generated based on the same CG data. As a result, displays 13a, 13b, 13c, and 13d each display a virtual space image that captures the same virtual space (CG space) from the same viewpoint as the real space image, allowing passengers in vehicle 10 to observe the same virtual space from different viewpoints on each display. For example, this CG data is changed in response to environmental changes, such as location, time, and weather. This allows the virtual space image superimposed on the real space image to be changed in response to environmental changes, making it possible to obtain a more effective display image.

[0027] Furthermore, the virtual space image included in the display image of each display is generated by, for example, placing a virtual camera (virtual imaging unit) in the virtual space in correspondence with a camera (imaging unit) in the real space and capturing an image of a CG object in the virtual space with this virtual camera. In this case, the virtual camera is placed in the virtual space in accordance with the position, orientation, and angle of view of the camera in the real space. This makes it easy to place a virtual camera in the virtual space in correspondence with the camera in the real space.

[0028] Here, since the camera is attached to the vehicle 10, the position and orientation of the camera in real space are determined based on information about the position and orientation of the vehicle 10 in real space and the mounting position and mounting angle of the camera on the vehicle 10. This makes it possible to correctly determine the position and orientation of the camera in real space when the camera in real space is attached to the vehicle 10.

[0029] FIG. 3(a) shows an example of a real space and a corresponding virtual space. Here, the coordinate systems of these spaces are assumed to be X, Y, and Z Cartesian coordinate systems. For example, a vehicle 10 travels along a course 15 from a start position to a goal position in one run. Since the course 15 is a circular course, the start position and the goal position are the same position.

[0030] The position of the vehicle 10 is expressed by X, Y, and Z values. The attitude of the vehicle 10 (direction of travel, tilt in the forward / backward direction, and tilt in the left / right direction) is expressed by rotation about the X, Y, and Z axes. The position of the vehicle 10 is estimated by a self-position estimation unit configured by, for example, the PC 14. This self-position estimation unit estimates the position of the vehicle 10 using a known self-position estimation method, although a detailed description thereof will be omitted.

[0031] Since the vehicle 10 travels along the course 15, if the PC 14 is equipped with an information storage unit that stores information on the correspondence between the time from the start and the position, the position of the vehicle 10 can be estimated based on the time from the start position using the information on the correspondence between the time and the position.

[0032] Furthermore, since the vehicle 10 travels along the course 15, the attitude of the vehicle 10 at each position is fixed. Therefore, for example, the PC 14 is provided with an information storage unit that stores information on the correspondence relationship between the position and the attitude, and the attitude of the vehicle 10 can be obtained based on the estimated position of the vehicle by using the information on the correspondence relationship between the position and the attitude.

[0033] The PC 14 also includes an information storage unit that stores information on the mounting position of the camera on the vehicle 10, the mounting angle (up / down angle, left / right angle), and information on the camera's angle of view. The mounting position and mounting angle of the camera mounted on the vehicle 10 are set in a coordinate system x, y, z with the position X1, Y1, Z1 of the vehicle 10 as the origin, as shown in FIG. 3(b). In the illustrated example, the mounting position of the camera is at position x1, y1, z1.

[0034] 4 shows an example of the configuration of an image processing device 100 mounted on a vehicle 10. This image processing device 100 has cameras 12a, 12b, 12c, and 12d attached to the front, right side, left side, and rear of the exterior of the vehicle 10, display units 13a, 13b, 13c, and 13d arranged on the front, right side, left side, and rear of the interior of the vehicle 10, and a PC 14 serving as a processing unit.

[0035] The PC 14 has a self-position estimation unit 141, virtual cameras 142a, 142b, 142c, and 142d installed in the virtual space corresponding to the cameras 12a, 12b, 12c, and 12d in the real space, and a display control unit 143, and the display control unit 143 includes a CG data storage unit 144 and a CG superimposition unit 145.

[0036] The self-position estimation unit 141 estimates the position of the vehicle 10. The self-position estimation unit 141 estimates the position of the vehicle 10 based on the time from the start of the vehicle 10, using, for example, information on the correspondence between the time from the start of the vehicle 10 and the position, which is stored in an information storage unit (not shown) included in the PC 14.

[0037] Virtual cameras 142a, 142b, 142c, and 142d are installed in a virtual space in accordance with the positions, orientations, and angles of cameras 12a, 12b, 12c, and 12d in real space, which are determined from the position of vehicle 10 estimated by self-position estimation unit 141, etc.

[0038] In this case, the positions and orientations of cameras 12a, 12b, 12c, and 12d in real space are determined based on information about the position and attitude of vehicle 10 in real space and the mounting positions and mounting angles of cameras 12a, 12b, 12c, and 12d on vehicle 10, respectively.

[0039] Here, the attitude of vehicle 10 in real space is determined based on the position of vehicle 10 estimated by self-position estimation unit 141, for example, by using information on the correspondence relationship between position and attitude stored in an information storage unit (not shown) included in PC 14. In addition, the mounting positions and mounting angles of cameras 12a, 12b, 12c, and 12d on vehicle 10, as well as the angles of view of cameras 12a, 12b, 12c, and 12d, are acquired from an information storage unit (not shown) included in PC 14.

[0040] Virtual cameras 142a, 142b, 142c, and 142d generate virtual space images corresponding to the imaging ranges of cameras 12a, 12b, 12c, and 12d, respectively, based on CG data stored in CG data storage unit 144 of display control unit 143. In this case, virtual cameras 142a, 142b, 142c, and 142d each capture an image of a CG object in a virtual space (CG space) to obtain the virtual space image.

[0041] The real-space images obtained by capturing images of objects in real space with cameras 12a, 12b, 12c, and 12d are sent to display control unit 143 of PC 14. CG superimposition unit 145 of display control unit 143 superimposes the virtual-space images generated by virtual cameras 142a, 142b, 142c, and 142d onto the real-space images acquired by cameras 12a, 12b, 12c, and 12d, respectively, to generate display images of the front, right side, left side, and rear.

[0042] The display images of the front, right side, left side, and rear sides generated by the display control unit 143 are sent to and displayed on the displays 13a, 13b, 13c, and 13d, respectively. In this case, for example, the display image of the front side, in which a virtual space image obtained by capturing an image of a virtual space (CG space) by the virtual camera 142a corresponding to the capturing range is superimposed on the real space image of the front direction of the vehicle 10 captured by the camera 12a, is displayed as a window image on the display 13a. Although detailed description will be omitted, the display images of the corresponding sides are similarly displayed as window images on the other displays.

[0043] The flowchart in FIG. 5 shows an example of the procedure for generating a display image that is executed by the PC 14, for example, at a frame cycle.

[0044] First, in step ST1, PC 14 estimates the position of vehicle 10 in real space using self-position estimation unit 141. Next, in step ST2, PC 14 installs a virtual camera corresponding to the camera of vehicle 10 in virtual space (CG space) based on the position estimation result, etc., and captures an image of the virtual space with this virtual camera to generate a virtual space image. Next, in step ST3, PC 14 obtains an image for display by superimposing the virtual space image captured by the virtual camera on the real space image captured by the camera of vehicle 10.

[0045] Fig. 6 shows an example of the configuration of an image processing device 100A mounted on a vehicle 10. In Fig. 6, parts corresponding to those in Fig. 4 are given the same reference numerals, and detailed description thereof will be omitted as appropriate.

[0046] This image processing device 100A has cameras 12a, 12b, 12c, and 12d attached to the front, right side, left side, and rear exterior of the vehicle 10, display units 13a, 13b, 13c, and 13d arranged on the front, right side, left side, and rear interior of the vehicle 10, and PCs 14a, 14b, 14c, and 14d as processing units.

[0047] The PC 14a generates a virtual space image corresponding to the imaging range of the camera 12a based on the CG data, and superimposes this virtual space image on the real space image (image in front of the vehicle) obtained by imaging an object in real space with the camera 12a to obtain an image for displaying the front, and sends it to the display 13a.

[0048] The PC 14a has a self-position estimation unit 141a, a virtual camera 142a installed in a virtual space corresponding to the camera 12a in the real space, and a display control unit 143a, and the display control unit 143a includes a CG data storage unit 144a and a CG superimposition unit 145a.

[0049] The self-position estimation unit 141a estimates the position of the vehicle 10 in the same manner as the self-position estimation unit 141 in the PC 14 of the image processing device 100 in FIG.

[0050] Similar to the virtual camera 142a in the PC 14 of the image processing device 100 in Figure 4, the virtual camera 142a is installed in a virtual space in accordance with the position, orientation, and angle of the camera 12a in the real space, which are determined from the position of the vehicle 10 estimated by the self-position estimation unit 141a.

[0051] Virtual camera 142a generates a virtual space image corresponding to the imaging range of camera 12a based on CG data stored in CG data storage unit 144a of display control unit 143a. In this case, virtual camera 142a captures an image of a CG object in a virtual space (CG space) to obtain the virtual space image.

[0052] A real-space image obtained by capturing an image of an object in real space with camera 12a is sent to display control unit 143a of PC 14a. CG superimposition unit 145a of display control unit 143a superimposes the virtual-space image generated by virtual camera 142a on the real-space image obtained by camera 12a to generate a display image for the front.

[0053] The display image of the front side generated by the display control unit 143a is sent to the display 13a and displayed thereon. In this case, the display 13a displays the display image of the front side as a window image, in which a virtual space image obtained by capturing an image of a virtual space (CG space) by the virtual camera 142a corresponding to the capturing range is superimposed on a real space image of the front side of the vehicle 10 captured by the camera 12a.

[0054] PCs 14b, 14c, and 14d, although not described in detail here, are configured similarly to PC 14a, and generate virtual space images corresponding to the imaging ranges of cameras 12b, 12c, and 12d, respectively, based on CG data, and superimpose these virtual space images on real space images (images of the right side, left side, and rear of the vehicle) obtained by imaging objects in real space with cameras 12b, 12c, and 12d to obtain display images of the right side, left side, and rear, which are then sent to displays 13b, 13c, and 13d. As a result, the display images of the right side, left side, and rear, respectively, are displayed as window images on displays 13b, 13c, and 13d.

[0055] In this case, the CG data used by PCs 14b, 14c, and 14d is the same as the CG data used by PC 14a, and is used with the time axis synchronized, which allows passengers in vehicle 10 to comfortably observe the same virtual space from different viewpoints on displays 13a, 13b, 13c, and 13d, even if the virtual space is a dynamic space.

[0056] The PCs 14a, 14b, 14c, and 14d in the image processing device 100A shown in FIG. 6 each generate an image for display on one surface, and the processing load can be reduced compared to the PC 14 in the image processing device 100 shown in FIG.

[0057] Fig. 7 shows an example of the configuration of an image processing device 100B mounted on a vehicle 10. In Fig. 7, parts corresponding to those in Fig. 4 and Fig. 6 are given the same reference numerals, and detailed description thereof will be omitted as appropriate.

[0058] This image processing device 100B has cameras 12a, 12b, 12c, and 12d attached to the front, right side, left side, and rear exterior of the vehicle 10, display units 13a, 13b, 13c, and 13d arranged on the front, right side, left side, and rear interior of the vehicle 10, and PCs 14a', 14b', 14c', and 14d' as processing units.

[0059] The PC 14a' generates a virtual space image corresponding to the imaging range of the camera 12a based on the CG data, and superimposes this virtual space image on the real space image (image in front of the vehicle) obtained by imaging an object in real space with the camera 12a to obtain an image for displaying the front, and sends it to the display 13a.

[0060] The PC 14a' has a self-position estimation unit 141a, a virtual camera 142a installed in the virtual space corresponding to the camera 12a in the real space, and a display control unit 143', and the display control unit 143' includes a CG superimposition unit 145a.

[0061] Unlike PC 14a in image processing device 100A of Figure 6, this PC 14a' does not store CG data in display control unit 143', and virtual camera 142a generates a virtual space image using CG data stored, for example, in CG data storage unit 161 of server 16 on the cloud.

[0062] A real-space image obtained by capturing an image of an object in real space with camera 12a is sent to display control unit 143a' of PC 14a'. CG superimposition unit 145a of display control unit 143a' superimposes the virtual-space image generated by virtual camera 142a on the real-space image obtained by camera 12a to generate a display image of the front.

[0063] The display image of the front side generated by the display control unit 143a' is sent to the display 13a and displayed thereon. In this case, the display 13a displays the display image of the front side as a window image, in which a virtual space image obtained by capturing an image of a virtual space (CG space) by the virtual camera 142a corresponding to the capturing range is superimposed on a real space image of the front side of the vehicle 10 captured by the camera 12a.

[0064] PCs 14b', 14c', and 14d', although detailed description will be omitted, are configured similarly to PC 14a', and generate virtual space images corresponding to the imaging ranges of cameras 12b, 12c, and 12d, respectively, based on CG data, and superimpose these virtual space images on real space images (images of the right side, left side, and rear of the vehicle) obtained by imaging objects in real space with cameras 12b, 12c, and 12d to obtain display images of the right side, left side, and rear, and send these to displays 13b, 13c, and 13d. As a result, the display images of the right side, left side, and rear, respectively, are displayed as window images on displays 13b, 13c, and 13d.

[0065] The PCs 14a', 14b', 14c', and 14d' in the image processing device 100B shown in FIG. 7 generate virtual space images using CG data stored in the server 16, and compared to the PCs 14a, 14b, 14c, and 14d in the image processing device 100A shown in FIG. 6, memory capacity can be saved by not storing CG data, and there is also no need to synchronize the time axis of the CG data between the respective PCs.

[0066] "PC hardware configuration example" 8 is a block diagram showing an example of the hardware configuration of a computer 400 (PCs 14, 14a to 14d, 14a' to 14d'). The computer 400 has a CPU 401, a ROM 402, a RAM 403, a bus 404, an input / output interface 405, an input unit 406, an output unit 407, a storage unit 408, a drive 409, a connection port 410, and a communication unit 411. Note that the hardware configuration shown here is an example, and some of the components may be omitted. Furthermore, the computer 400 may further include components other than those shown here.

[0067] The CPU 401 functions as, for example, an arithmetic processing device or a control device, and controls all or part of the operations of the components based on various programs recorded in the ROM 402 , RAM 403 , storage unit 408 or removable recording medium 501 .

[0068] The ROM 402 is a means for storing programs to be read into the CPU 401, data to be used for calculations, etc. The RAM 403 temporarily or permanently stores, for example, the programs to be read into the CPU 401 and various parameters that change as appropriate when the programs are executed.

[0069] The CPU 401, ROM 402, and RAM 403 are connected to one another via a bus 404. On the other hand, various components are connected to the bus 404 via an interface 405.

[0070] The input unit 406 may include, for example, a mouse, a keyboard, a touch panel, a button, a switch, a lever, etc. Furthermore, the input unit 406 may also include a remote controller (hereinafter referred to as a remote control) that can transmit control signals using infrared rays or other radio waves.

[0071] The output unit 407 is a device capable of visually or audibly notifying the user of the acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or headphones, a printer, a mobile phone, or a facsimile.

[0072] The storage unit 408 is a device for storing various types of data. For example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device may be used as the storage unit 408.

[0073] The drive 409 is a device that reads information recorded on a removable recording medium 501 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable recording medium 501 .

[0074] The removable recording medium 501 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, various semiconductor storage media, etc. Of course, the removable recording medium 501 may also be, for example, an IC card equipped with a contactless IC chip, an electronic device, etc.

[0075] The connection port 410 is a port for connecting an external device 502, such as a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI), an RS-232C port, or an optical audio terminal. The external device 502 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0076] The communication unit 411 is a communication device for connecting to the network 503, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), or WUSB (Wireless USB), a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0077] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0078] As explained above, in the image processing devices 100, 100A, and 100B shown in Figures 4, 6, and 7, virtual space images corresponding to the imaging ranges of cameras 12a to 12d in real space are generated based on CG data, and images of objects in real space are captured by cameras 12a to 12d, and the virtual space images are superimposed on the real space images to obtain images for display. It is therefore possible to obtain good images for display by superimposing a virtual space image that matches the real space image.

[0079] In addition, in the image processing devices 100, 100A, and 100B shown in Figures 4, 6, and 7, multiple virtual space images corresponding to the imaging ranges of four cameras 12a to 12d in real space are generated based on the same CG data, and the corresponding virtual space images are superimposed on each real space image obtained by imaging an object in real space with the four cameras 12a to 12d to obtain display images as four window images for the front, right side, left side, and rear.

[0080] In this case, the four virtual space images are generated based on the same CG data, and a virtual space image capturing the same virtual space (CG space) from the same viewpoint as the real space image is superimposed on each of the four real space images. Therefore, the four displays 13a to 13d display superimposed virtual space images capturing the same virtual space (CG space) from the same viewpoint as the real space image, allowing passengers in vehicle 10 to observe the same virtual space from different viewpoints on each display.

[0081] <2. Modifications> In the above-described embodiment, an example has been shown in which cameras 12a, 12b, 12c, and 12d are attached to the front, right side, left side, and rear of the exterior of vehicle 10, and displays 13a, 13b, 13c, and 13d as display units for displaying window images are arranged on the front, right side, left side, and rear of the interior of vehicle 10. However, application of the present technology is not limited to such vehicle 10, and can be similarly applied to other vehicles or moving devices.

[0082] 9 shows a configuration example of a taxi 20 to which the present technology is applied. A camera 22 for capturing real space images is attached to the front exterior of the taxi 20, and a display 23 is disposed inside the vehicle 10 between the driver in the front seat and the passengers in the rear seat. A PC 24 constituting a processing unit is also disposed inside the vehicle 10.

[0083] The camera 22 captures an image of an object in real space to obtain a real space image (an image in front of the vehicle) and sends it to the PC 24. The PC 24 generates a virtual space image corresponding to the imaging range of the camera 22 based on CG data. The PC 24 also generates a display image by superimposing the virtual space image on the real space image sent from the camera 22, and sends the display 23. On the display 23, objects in the virtual space (CG objects) are displayed together with real space objects such as roads and are presented to passengers. This allows passengers to enjoy not only real space images but also virtual space images that respond to changes in, for example, location, time, environment, etc.

[0084] 10 shows a configuration example of a bus 30 to which the present technology is applied. Displays 33Ra, 13Rb, and 13Rc are arranged on the right side of the interior of this bus 30, in the positions of multiple windows (three windows in this case) on a conventional bus. In addition, cameras 23Ra, 23Rb, and 23Rc for capturing real space images are attached to the right side of the exterior of this bus 30, corresponding to the displays 33Ra, 13Rb, and 13Rc, respectively.

[0085] Additionally, displays 33La, 13Lb, and 13Lc are arranged on the left side of the interior of bus 30, in the positions of multiple windows (three in this case) on a conventional bus. Additionally, cameras 32La, 32Lb, and 32Lc for capturing real space images are attached to the left side of the exterior of bus 30, corresponding to displays 33La, 33Lb, and 33Lc, respectively.

[0086] Also located inside this bus 30 are PCs 34Ra, 34Rb, 34Rc, 34La, 34Lb, and 34Lc, which form a processing unit. PCRa forms a processing unit corresponding to the camera 32Ra and the display 33Ra, and the camera 23Ra, the display 33Ra, and the PC 34Ra form one image processing device. In the other parts, too, a camera, a display, and a PC form one image processing device, respectively.

[0087] The image processing device, which is composed of the camera 32Ra, display 33Ra, and PC 34Ra, will now be described. The camera 32Ra captures an image of an object in real space to obtain a real space image (an image of the right side of the bus) and sends it to the PC 34Ra. The PC 34Ra generates a virtual space image corresponding to the imaging range of the camera 32Ra based on CG data. The PC 34Ra also generates a display image by superimposing the virtual space image on the real space image sent from the camera 32Ra and sends it to the display 33Ra. The display image is displayed as a window image on the display 33Ra and presented to passengers.

[0088] Although detailed explanations will be omitted, the other parts of the image processing device operate in a similar manner, and a display image is generated on the PC by superimposing a corresponding virtual space image onto the real space image captured by the camera, and this is sent to the display as a window image and shown to passengers.

[0089] In this case, the same CG data is used in the PCs constituting each image processing device, but the data stored in each PC is used, or data stored in a server on the cloud, for example, is used in common. In this case, when using the CG data stored in each PC, it is necessary to use it in a state where the time axis is synchronized. Also, instead of each image processing device having its own PC, it is possible to configure all processing to be performed by a single PC with high processing power.

[0090] Figure 11(a) shows an example of a virtual space image displayed at time t1 on displays 33Ra, 33Rb, and 33Rc located on the right side of the interior of bus 30. Figure 11(b) shows an example of a virtual space image displayed on displays 33Ra, 33Rb, and 33Rc at time t2, which follows time t1. In this case, the appearance of the CG object in the virtual space, in this case a whale, changes on displays 33Ra, 33Rb, and 33Rc in accordance with the change in the positional relationship between bus 30 and the CG object.

[0091] Fig. 12 shows a train 40 running along tracks 17 to which the present technology is applied. A predetermined number of cars constituting this train, three cars 40a, 40b, and 40c in this case, each have a configuration similar to that of bus 30 shown in Fig. 10, although detailed description will be omitted. As a result, in each of cars 40a, 40b, and 40c, passengers can observe, on displays arranged in the windows, a display image in which a virtual space image generated using the same CG image is superimposed on a real space image, as a window image.

[0092] In the above-described embodiment, an example has been shown in which a display image is displayed on a display, in which a virtual space image generated by a PC is superimposed on a real space image captured by a camera. When the display is placed in a window position of a vehicle, for example, it is also possible to use a transparent display to display only the virtual space image.

[0093] Furthermore, while the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0094] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0095] The present technology can also be configured as follows. (1) an image generation unit that generates a virtual space image corresponding to the imaging range of the imaging unit in real space based on CG (Computer Graphics) data; an image superimposing unit that obtains an image for display by superimposing the virtual space image on a real space image obtained by capturing an image of an object in the real space with the imaging unit; Image processing device. (2) The image generation unit installs a virtual imaging unit in the virtual space corresponding to the imaging unit in the real space, and generates the virtual space image by imaging a CG (Computer Graphics) object in the virtual space with the virtual imaging unit. The image processing device according to (1) above. (3) The image generation unit installs the virtual imaging unit in the virtual space in accordance with the position, orientation, and angle of view of the imaging unit in the real space. The image processing device according to (2) above. (4) the imaging unit in the real space is attached to a vehicle; The position and orientation of the imaging unit in the real space are determined based on information about the position and orientation of the vehicle in the real space, and the mounting position and mounting angle of the imaging unit on the vehicle. The image processing device according to (3) above. (5) The vehicle further includes a position estimation unit that estimates the position of the vehicle in the real space. The image processing device according to (4) above. (6) The vehicle control system further includes an information storage unit that stores information on a correspondence relationship between a position and an attitude of the vehicle, in order to obtain the attitude of the vehicle based on the estimated position of the vehicle in the real space. The image processing device according to (5) above. (7) Further comprising an imaging unit in the real space The image processing device according to any one of (1) to (6). (8) A display unit for displaying an image based on the display image is further provided. The image processing device according to any one of (1) to (7). (9) The image generation unit changes the CG (Computer Graphics) data in response to environmental changes in the real space. The image processing device according to any one of claims 1 to 8. (10) The image generating unit generating a plurality of virtual space images corresponding to the imaging ranges of the plurality of imaging units in the real space, respectively, based on the same CG (Computer Graphics) data; The image superimposing unit obtains a plurality of display images by superimposing the corresponding virtual space images onto a plurality of real space images obtained by capturing images of the object in the real space using the plurality of imaging units. The image processing device according to any one of (1) to (9). (11) The image generating unit and the image superimposing unit are configured as a single processing unit that holds the CG (Computer Graphics) data. The image processing device according to (10) above. (12) The image generating unit and the image superimposing unit are composed of a plurality of processing units, Each processing unit generates a virtual space image corresponding to the imaging range of the imaging unit in the real space based on the CG (Computer Graphics) data stored in itself or the CG (Computer Graphics) data stored in a common server, and obtains a display image by superimposing the virtual space image on a real space image obtained by imaging an object in the real space with the imaging unit. The image processing device according to (10) above. (13) The imaging unit in the real space is attached to the exterior of the vehicle and is used to capture a window image, The display image is displayed on a display unit for displaying a window image, which is arranged inside the vehicle. The image processing device according to any one of (1) to (12) above. (14) generating a virtual space image corresponding to the imaging range of the imaging unit in real space based on CG (Computer Graphics) data; a step of superimposing the virtual space image on a real space image obtained by capturing an image of an object in the real space with the imaging unit to obtain an image for display; Image processing methods. [Explanation of symbols]

[0096] 10. Vehicle 11. Display 12a~12d···Camera 13a~13d···Display 14,14a~14d,14a´~14d´···PC 15 Course 16 Servers 17...railway 20. Taxi 22. Camera 23. Display 24···PC 30 Bus 32Ra~32Rd, 32La~32Ld... Camera 33Ra~33Rd, 33La~33Ld...Display 34Ra~34Rd, 34La~34Ld...PC 40. Train 40a~40c...vehicle 100, 100A, 100B...Image processing device 141,141a... Self-position estimation part 142a~142d···Virtual Camera 143, 143a´···Display control unit 144, 144a...CG data storage unit 145, 145a···CG superimposition section 161 CG data storage unit

Claims

1. an image generation unit that generates a virtual space image corresponding to an imaging range of the imaging unit in real space based on CG (Computer Graphics) data; an image superimposing unit that obtains an image for display by superimposing the virtual space image on a real space image obtained by capturing an image of an object in the real space with the imaging unit; Image processing device.

2. The image generation unit installs a virtual imaging unit in the virtual space corresponding to the imaging unit in the real space, and generates the virtual space image by imaging a CG (Computer Graphics) object in the virtual space with the virtual imaging unit. The image processing device according to claim 1 .

3. The image generation unit installs the virtual imaging unit in the virtual space in accordance with the position, orientation, and angle of view of the imaging unit in the real space. The image processing device according to claim 2 .

4. the imaging unit in the real space is attached to a vehicle, The position and orientation of the imaging unit in the real space are determined based on information about the position and orientation of the vehicle in the real space, and the mounting position and mounting angle of the imaging unit on the vehicle. The image processing device according to claim 3 .

5. The vehicle may further include a position estimation unit that estimates the position of the vehicle in the real space. The image processing device according to claim 4 .

6. The vehicle posture estimation system further includes an information storage unit configured to store position-posture correspondence information for determining the posture of the vehicle based on the estimated position of the vehicle in the real space. The image processing device according to claim 5 .

7. The imaging unit in the real space is further provided. The image processing device according to claim 1 .

8. a display unit that displays an image based on the display image; The image processing device according to claim 1 .

9. The image generation unit changes the CG (Computer Graphics) data in response to environmental changes in the real space. The image processing device according to claim 1 .

10. The image generation unit generating a plurality of virtual space images corresponding to the imaging ranges of the plurality of imaging units in the real space based on the same CG (Computer Graphics) data; The image superimposing unit obtains a plurality of display images by superimposing the corresponding virtual space images onto a plurality of real space images obtained by capturing images of the object in the real space using the plurality of imaging units. The image processing device according to claim 1 .

11. The image generating unit and the image superimposing unit are configured as a single processing unit that holds the CG (Computer Graphics) data. The image processing device according to claim 10.

12. the image generating unit and the image superimposing unit are configured with a plurality of processing units, Each processing unit generates a virtual space image corresponding to the imaging range of the imaging unit in the real space based on the CG (Computer Graphics) data stored in the processing unit itself or the CG (Computer Graphics) data stored in a common server, and obtains a display image by superimposing the virtual space image on a real space image obtained by imaging an object in the real space with the imaging unit. The image processing device according to claim 10.

13. the imaging unit in the real space is attached to the exterior of a vehicle and is used to capture an image of a window, The display image is displayed on a display unit for displaying a window image, which is arranged inside the vehicle. The image processing device according to claim 1

14. generating a virtual space image corresponding to an imaging range of an imaging unit in real space based on CG (Computer Graphics) data; a step of superimposing the virtual space image on a real space image obtained by capturing an image of an object in the real space with the imaging unit to obtain an image for display; Image processing methods.

Images