Image processing apparatus, image processing method, program, and image presentation system

Abstract

The present disclosure relates to an image processing device, an image processing method, a program, and an image presentation system which make it possible to present a more preferable image in a vehicle. An image processing unit, on the basis of a vehicle-outside image obtained by photographing a vehicle-outside environment of a first vehicle during travel, generates a presentation image that is presented inside a second vehicle during travel. The present disclosure is applicable to, for example, a projector-type presentation device.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an image processing method, a program, and an image presentation system, and more particularly to an image processing apparatus, an image processing method, a program, and an image presentation system capable of presenting a more suitable image inside a vehicle.

Background Art

[0002] Patent Document 1 discloses a technique for generating a developed image in which a fisheye image obtained by photographing through a fisheye lens is developed on a cylindrical surface by converting the fisheye image. According to this technique, the three-dimensional position of a subject can be estimated from the developed image with little distortion.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, when an image that does not correspond to the situation outside the vehicle is presented inside the vehicle of a running vehicle, there is a risk of giving a sense of discomfort to the user who views the image.

[0005] The present disclosure has been made in view of such a situation, and enables a more suitable image to be presented inside the vehicle.

Means for Solving the Problems

[0006] The image processing apparatus of the present disclosure is an image processing apparatus including an image processing unit that generates a presentation image to be presented inside a second vehicle running based on an external image obtained by photographing the external environment of a first vehicle running.

[0007] The image processing method of the present disclosure is an image processing method in which an image processing apparatus generates a presentation image to be presented inside a second vehicle in motion based on an external image of the external environment of a first vehicle in motion.

[0008] The program of the present disclosure is a program for causing a computer to execute a process of generating a presentation image to be presented inside a second vehicle in motion based on an external image of the external environment of a first vehicle in motion.

[0009] The image presentation system of the present disclosure includes an image processing apparatus having an image processing unit that generates a presentation image to be presented inside a second vehicle in motion based on an external image of the external environment of a first vehicle in motion, and a presentation apparatus having a presentation unit that presents the presentation image inside the second vehicle.

[0010] In the present disclosure, a presentation image to be presented inside a second vehicle in motion is generated based on an external image of the external environment of a first vehicle in motion.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order.

[0013] 1. Outline of the Technology According to the Present Disclosure 2. First Embodiment (Correction of Fisheye Camera Image) 3. Second Embodiment (Correction of Fisheye Camera Image and Object Recognition Processing) 4. Third Embodiment (Correction of Front Camera Image) 5. Fourth Embodiment (Image Reproduction Corresponding to Travel Speed) 6. Fifth Embodiment (Image Correction and Image Reproduction Corresponding to Travel Speed) 7. Modification Example 8. Configuration Example of Computer

[0014] <1. Outline of Technology According to the Present Disclosure> (Configuration of Image Processing Apparatus) FIG. 1 is a block diagram showing the configuration of an image processing apparatus to which the technology (this technology) according to the present disclosure is applied.

[0015] The image processing apparatus 1 in FIG. 1 is configured as a projector-type presentation apparatus that realizes interactive image presentation. The image processing apparatus 1 presents an image generated by predetermined image processing to a user by projecting the image onto a projector, for example, inside a vehicle such as an automobile.

[0016] The image processing apparatus 1 includes an input unit 10, a graphics display processing unit 20, an output unit 30, an image processing unit 40, and a storage unit 50.

[0017] The input unit 10 is composed of a group of sensors such as an image sensor, a depth sensor, a touch sensor, and a speed sensor, a general input device, and a communication device, and receives as inputs an image projected inside the vehicle, a user's operation, the speed of the vehicle, etc. As an image sensor capable of acquiring an image, a visible light camera, an infrared camera, etc. are used. As a depth sensor capable of acquiring three-dimensional information, a stereo camera, a ToF (Time of Flight) sensor, etc. are used. Input information such as various sensor data received by the input unit 10 is supplied to the graphics display processing unit 20.

[0018] The graphics display processing unit 20 executes processing to display graphics to the user based on the input information from the input unit 10. The graphics display processing unit 20 is configured as a control layer of a general OS (Operating System) that performs drawing control of multi-content such as a window for displaying an application and event distribution such as touch operations for each content. Note that the input information supplied to the graphics display processing unit 20 is also supplied to the image processing unit 40.

[0019] The output unit 30 is composed of a projector as one or more presentation units, and presents an image to the user.

[0020] The image processing unit 40 executes predetermined image processing based on the input information supplied via the graphics display processing unit 20. The image obtained by the image processing is presented by the output unit 30 via the graphics display processing unit 20.

[0021] The storage unit 50 stores information necessary for the image processing executed by the image processing unit 40.

[0022] Note that the configuration of FIG. 1 may be configured as an image presentation system 1 including an image processing apparatus having at least the image processing unit 40 and a presentation apparatus having at least the output unit 30.

[0023] (Application Example of Image Processing Apparatus) Here, an example of applying the image processing apparatus 1 of FIG. 1 to a configuration for projecting an image inside a vehicle will be described.

[0024] FIG. 2 and FIG. 3 are a top view and a side view of a vehicle 80 equipped with the image processing apparatus 1.

[0025] In FIGS. 2 and 3, projectors 81, 82, 83 corresponding to the output unit 30 of the image processing apparatus 1 of FIG. 1 are installed inside the vehicle 80.

[0026] The projector 81 is installed near the center in the vertical and horizontal directions at the rear end inside the vehicle. The projector 82 is installed at the upper part of the right side surface at the rear inside the vehicle. The projector 83 is installed at the upper part of the left side surface at the rear inside the vehicle.

[0027] FIG. 4 is a diagram illustrating the position of a presentation site (projection surface) on which an image is projected inside the vehicle 80.

[0028] When the ceiling 92 is used as the presentation site inside the vehicle 91 of the vehicle 80, the projection image 93 from the projector 81 is projected onto the ceiling 92. When the vehicle 80 is equipped with a sunroof, a slide panel that shields the sunroof is used as the presentation site.

[0029] When the left side glass 94 of the rear seat is used as the presentation site, the projection image 95 from the projector 82 is projected onto the left side glass 94. Also, when the right side glass 96 of the rear seat is used as the presentation site, the projection image 97 from the projector 83 is projected onto the right side glass 96.

[0030] In addition to the ceiling 92, the left side glass 94 and the right side glass 96 of the rear seat, the armrest 98 at the center of the rear seat, the headrest 99 of the driver's seat, the headrest 100 of the passenger seat, the side glass 101 on the passenger seat side, and the side glass 102 on the driver's seat side may be used as the presentation site.

[0031] In this case, a projection image from any one of the projectors 81 to 83 is projected onto the armrest 98, the headrest 99 of the driver's seat, and the headrest 100 of the passenger seat. Also, a projection image from any one of the projectors 82 and 83 is projected onto the side glass 101 on the passenger seat side and the side glass 102 on the driver's seat side.

[0032] With such a configuration, any area inside the vehicle 91 can be used as an interactive presentation site, and a screen can be prepared as needed while making use of the texture of the interior of the vehicle 91. In addition, by using a projector, a large area such as the ceiling can be utilized as a presentation site at a lower cost compared to conventional liquid crystal monitors and the like.

[0033] In recent years, due to the increasing safety standards for vehicles and the expectations of purchasers, car manufacturers and the like need to meet the high goal of "a lightweight and durable car". In contrast, a projector that can present a large image while achieving space savings is considered useful.

[0034] (Operation of the image processing unit) Here, with reference to the flowchart of FIG. 5, the operation flow of the image processing unit 40 provided in the image processing apparatus 1 will be described. The processing in FIG. 5 is executed triggered by, for example, an instruction for presenting an image by the user in the vehicle 80 during traveling.

[0035] In step S1, the image processing unit 40 of the image processing apparatus 1 acquires an external image input by the input unit 10. The external image is an image obtained by photographing the external environment of a predetermined vehicle (the first vehicle) during traveling. The predetermined vehicle may be another vehicle different from the vehicle 81, or may be the same vehicle as the vehicle 80 (the vehicle 80 itself). Further, the external image may be an image photographed in the past from a predetermined vehicle during traveling, or may be an image being photographed in real time from a predetermined vehicle currently traveling.

[0036] In step S2, based on the acquired external image, the image processing unit 40 generates an image (hereinafter referred to as a presentation image) to be presented in the vehicle interior 91 of the vehicle 80 (the second vehicle) during traveling. The presentation image is an image corresponding to the situation outside the vehicle 80.

[0037] In step S3, the image processing unit 40 outputs the generated presentation image to the graphics display processing unit 20. The presentation image output to the graphics display processing unit 20 is presented inside the vehicle 91 of the vehicle 80 by the output unit 30.

[0038] According to the above processing, a presentation image corresponding to the external situation is generated as an image presented inside the vehicle 91 of the traveling vehicle 80, so that a more suitable image can be presented inside the vehicle.

[0039] Hereinafter, each embodiment of the image processing unit 40 will be described.

[0040] <2. First Embodiment> FIG. 6 is a block diagram showing a configuration example of the image processing unit 40A according to the first embodiment of the present technology.

[0041] The image processing unit 40A generates a presentation image to be presented on the ceiling 92, which is the projection plane inside the vehicle 91, based on an external image of the vehicle 80 taken while the vehicle is traveling.

[0042] The image processing unit 40A is composed of an external image acquisition unit 111 and a clipping unit 112.

[0043] The external image acquisition unit 111 acquires an external image from a camera having a fisheye lens (hereinafter referred to as a fisheye camera), which is configured as the input unit 10, and supplies it to the clipping unit 112.

[0044] FIG. 7 is a diagram showing the arrangement of the fisheye camera.

[0045] As shown in FIG. 7, the fisheye camera 130 is provided on the roof (outside the ceiling) of the vehicle 80 and photographs in the zenith direction from the traveling vehicle 80. The image taken by the fisheye camera 130 (hereinafter referred to as a fisheye camera image) is a moving image in which the entire periphery of the vehicle 80 is reflected around the zenith direction of the vehicle 80. That is, the external image acquisition unit 111 acquires the fisheye camera image as the external image.

[0046] Here, the reason a fisheye camera 130 is used instead of a camera having a normal lens with a central projection method is to increase the angle of view.

[0047] The clipping unit 112 corrects the outside-vehicle image (fisheye camera image) from the outside-vehicle image acquisition unit 111 to match the ceiling 92 which is the projection plane inside the vehicle 91, and outputs it as a presentation image. By presenting the presentation image in which the zenith direction outside the vehicle is reflected on the ceiling 92, the user can be made to feel as if they are riding in an open car.

[0048] Next, with reference to the flowchart of FIG. 8, the flow of the operation of the image processing unit 40A will be described.

[0049] In step S11, the outside-vehicle image acquisition unit 111 acquires a fisheye camera image from the fisheye camera.

[0050] In step S12, the clipping unit 112 clips the fisheye camera image.

[0051] For example, assume that a fisheye camera image 140 in which buildings around the traveling vehicle 80 as shown in FIG. 9 are reflected is acquired from the fisheye camera 130. If the fisheye camera image 140 is projected onto the ceiling 92 inside the vehicle 91 as it is, the surrounding distortion will become prominent.

[0052] Therefore, the clipping unit 112 generates an enlarged image 150 by enlarging the fisheye camera image 140, and clips a region CL101 corresponding to the projection plane (ceiling 92) of the region inside the vehicle 91 with the center of the fisheye camera image 140 as a reference.

[0053] As a result, a presentation image with less distortion is output. However, since the presentation image obtained by clipping the region CL101 is affected by the characteristics of the fisheye lens, for example, the top edge portion of a building that originally draws a straight line is distorted.

[0054] Therefore, as shown in FIG. 10, the clipping unit 112 enlarges the fisheye camera image 140, generates a distortion-corrected image 160 after distortion correction, and clips a region CL102 corresponding to the projection plane (ceiling 92) of the vehicle interior 91. The distortion correction here is performed on the premise that the curvature of the fisheye lens is known and uniform, and is performed regardless of the individual differences of the fisheye lenses.

[0055] According to the above configuration and processing, it is possible to output a presentation image with less distortion.

[0056] <3. Second Embodiment> FIG. 11 is a block diagram showing a configuration example of the image processing unit 40B according to the second embodiment of the present technology.

[0057] The image processing unit 40B generates a presentation image to be presented on the ceiling 92, which is the projection plane of the vehicle interior 91, based on an external image of the vehicle 80 taken while the vehicle is running.

[0058] The image processing unit 40B includes an external image acquisition unit 211, a clipping unit 212, an object recognition unit 213, and a correction processing unit 214.

[0059] The external image acquisition unit 211 and the clipping unit 212 have the same functions as the external image acquisition unit 111 and the clipping unit 112 in FIG. 6, respectively. Therefore, the fisheye camera image acquired by the external image acquisition unit 211 is enlarged, (distortion-corrected,) clipped by the clipping unit 112, and supplied to the object recognition unit 213.

[0060] The object recognition unit 213 performs object recognition processing on at least a part of the clipped fisheye camera image, and supplies the processing result to the correction processing unit 214 together with the clipped fisheye camera image.

[0061] For example, for the captured image 220 shown in the upper part of FIG. 12, the object recognition unit 213 discriminates the attributes of the subject (object) in pixel units by semantic segmentation, and divides the captured image 220 for each of those attributes. As a result, a processed image 230 as shown in the lower part of FIG. 12 is obtained. In the processed image 230, as the attributes of the subject, a car, a road, a sidewalk, a house, a wall, a tree, the sky, etc. are discriminated.

[0062] Note that the object recognition unit 213 can perform object recognition processing not only by semantic segmentation but also by other methods.

[0063] The correction processing unit 214 corrects and repairs the clipped fisheye camera image based on the processing result of the object recognition processing from the object recognition unit 213 and the object information stored in the object information definition unit 215, and outputs it as a presentation image. The object information definition unit 215 is realized, for example, in the form of a relational database or a lookup table in the storage unit 50.

[0064] Next, with reference to the flowchart of FIG. 13, the operation flow of the image processing unit 40B will be described.

[0065] Note that the processing in steps S21 and S22 in the flowchart of FIG. 13 is the same as the processing in steps S11 and S12 in the flowchart of FIG. 8, so the description thereof is omitted.

[0066] That is, when the fisheye camera image is clipped in step S22, in step S23, the object recognition unit 213 performs object recognition processing on at least a part of the clipped fisheye camera image.

[0067] In step S24, the correction processing unit 214 corrects the clipped fisheye camera image based on the object information stored in the object information definition unit 215.

[0068] For example, as shown in FIG. 14, it is assumed that a presentation image 240 in which the region CL102 is clipped from the distortion-corrected image 160 described with reference to FIG. 10 is obtained.

[0069] The object recognition unit 213 extracts a portion surrounded by the broken-line rectangular frame 240A from the presentation image 240 and obtains an extracted image 250. Then, the object recognition unit 213 divides the extracted image 250 for each object attribute by performing object recognition processing on the extracted image 250. The obtained processed image 260 is divided into regions of a building, a tree, sky, and a street lamp.

[0070] Then, the correction processing unit 214 corrects the presentation image 240 based on the processing result of the object recognition processing by the object recognition unit 213 and the object information stored in the object information definition unit 215, thereby obtaining a corrected image 270 shown in FIG. 15.

[0071] FIG. 16 is a diagram showing an example of object information.

[0072] The object information OJ201 shown in FIG. 16 is information indicating whether each object has sides or surfaces parallel to the road (ground plane) on which the vehicle travels. In the example of FIG. 16, it is shown that while the building has sides or surfaces parallel to the road, the tree, sky, and street lamp do not have sides or surfaces parallel to the road.

[0073] Here, the correction processing unit 214 corrects the presentation image 240 using the information that "the building has sides or surfaces parallel to the road" so that the top edge of the building recognized in the presentation image 240 is parallel (straight) to the road.

[0074] In the corrected image 270 obtained in this way, as shown by the black region in the figure, a part of it is missing. Therefore, the correction processing unit 214 performs image repair by inpainting on the corrected image 270, for example, to obtain a repaired image 280 and outputs it as a presentation image.

[0075] According to the above configuration and processing, it is possible to output a presentation image with less distortion.

[0076] <4. Third Embodiment> FIG. 17 is a block diagram showing a configuration example of the image processing unit 40C according to the third embodiment of the present technology.

[0077] The image processing unit 40C generates a presentation image to be presented on the ceiling 92, which is the projection plane of the vehicle interior 91, based on an external image of the vehicle 80 taken while the vehicle is running.

[0078] The image processing unit 40C includes an external image acquisition unit 311, a vanishing point detection unit 312, a panel superposition unit 313, a clipping unit 314, a synthesis unit 315, and a blurring processing unit 316.

[0079] The external image acquisition unit 311 acquires an external image from a front camera configured as the input unit 10 and supplies it to the vanishing point detection unit 312.

[0080] FIG. 18 is a diagram showing the arrangement of the front camera.

[0081] As shown in FIG. 18, the front camera 320 is provided at the upper end of the windshield or the front end of the ceiling inside the vehicle 80 and captures the traveling direction of the vehicle 80 while it is running. The image captured by the front camera 320 (hereinafter referred to as the front camera image) is a moving image in which the front of the vehicle 80 in the traveling direction is reflected. That is, the external image acquisition unit 311 acquires the front camera image as the external image.

[0082] The vanishing point detection unit 312 detects the vanishing point from the front camera image from the external image acquisition unit 311. For example, the vanishing point detection unit 312 acquires edge information from the front camera image. The vanishing point detection unit 312 performs a Hough transform on the acquired edge information to output a straight line, and obtains the intersection point of the output straight lines. The vanishing point detection unit 312 obtains a range where many of the obtained intersection points gather, averages the coordinates of the intersection points within the range, and sets the averaged coordinates as the vanishing point.

[0083] In the front camera image, the panel overlapping section 313 overlaps a one-point perspective panel at a position based on the vanishing point detected by the vanishing point detection section 312.

[0084] Here, with reference to FIG. 19, the one-point perspective panel will be described.

[0085] In FIG. 19, the vanishing point VP has been detected from the front camera image FV, and the one-point perspective panel 330 is overlapped so as to match the perspective line leading to the vanishing point VP. The one-point perspective panel 330 is an image for specifying, in the front camera image FV, regions corresponding to the windshield of the vehicle 80, the left and right side glasses of the front seats, and the ceiling when the front camera image FV is regarded as a perspective view drawn by the one-point perspective drawing method.

[0086] Specifically, the one-point perspective panel 330 is composed of a front panel 330F corresponding to the windshield of the vehicle 80, a left side panel 330L corresponding to the left side glass of the front seat, a right side panel 330R corresponding to the right side glass of the front seat, and a ceiling panel 330T corresponding to the ceiling.

[0087] Returning to the description of FIG. 17, the clipping section 314 clips, in the front camera image, the region corresponding to the presentation part (ceiling 92) of the vehicle 80 at predetermined intervals such as per second or per frame. The region images clipped at predetermined intervals are sequentially supplied to the synthesizing section 315.

[0088] The synthesizing section 315 generates a synthesized image by sequentially synthesizing (combining) the region images from the clipping section 314 and supplies it to the blurring processing section 316.

[0089] The blurring processing unit 316 performs blurring processing on the composite image from the composite unit 315 and outputs it as a presentation image. Even if there is no camera that captures the zenith direction outside the vehicle, by presenting a presentation image in which the area corresponding to the ceiling 92 is reflected in the front camera image on the ceiling 92, the user can be made to feel as if they are riding in an open car.

[0090] Next, with reference to the flowchart of FIG. 20, the operation flow of the image processing unit 40C will be described.

[0091] In step S31, the out-of-vehicle image acquisition unit 311 acquires a front camera image from the front camera.

[0092] In step S32, the vanishing point detection unit 312 detects the vanishing point from the front camera image.

[0093] In step S33, the panel superimposing unit 313 superimposes a one-point perspective panel at a position based on the vanishing point in the front camera image.

[0094] For example, assume that a front camera image 340 showing a view in the traveling direction of the vehicle 80 as shown in FIG. 21 is acquired from the front camera 320. In the front camera image 340, in addition to buildings such as buildings standing on both sides of the road, there are vehicles traveling in front of the vehicle 80 and parked vehicles, and road signs installed beside the road.

[0095] When the front camera image 340 is acquired, as shown in FIG. 22, the vanishing point VP is detected from the front camera image 340, and as shown in FIG. 23, a one-point perspective panel 330 is superimposed at a position based on the vanishing point VP in the front camera image 340.

[0096] When the one-point perspective panel 330 is superimposed on the front camera image 340, in step S34, as shown in FIG. 24, the clipping unit 314 clips a partial region CL301 of the ceiling panel 330T corresponding to the ceiling 92 of the vehicle 80 in the front camera image 340 at predetermined time intervals to obtain a region image 350. In the example of FIG. 24, the region CL301 is a region corresponding to the front end of the ceiling in the ceiling panel 330T.

[0097] In step S35, the synthesizing unit 315 sequentially synthesizes the region images 350 clipped at predetermined time intervals. Specifically, as shown in FIG. 25, five region images 350(t - 4) to 350(t) acquired at times t - 4 to t are synthesized so as to be arranged in order from the top.

[0098] Thereafter, in step S36, the blurring processing unit 316 performs blurring processing on the synthesized image in which the region images 350(t - 4) to 350(t) are synthesized, for example, using a Gaussian filter, to obtain a processed image 360 and output it as a presentation image.

[0099] FIG. 26 is a diagram showing an example of projection of a presentation image in the vehicle 80.

[0100] FIG. 26 shows the front glass 371 as seen from the rear seat of the vehicle 80, the left and right side glasses 372L, 372R of the front seat, and the ceiling 373.

[0101] The scenery in front of the vehicle 80 at time t can be seen from the front glass 371, and the presentation image (processed image 360) output at time t is projected onto the ceiling 373 serving as a projection surface.

[0102] When the vehicle 80 is running in autonomous driving mode, in addition to the ceiling 373, the windshield 371 can also be used as the projection surface. In this case, on the windshield 371, an image in which the region corresponding to the front panel 330F of the one-point perspective panel 330 is clipped in the front camera image 340 at time t is projected as the presentation image.

[0103] Furthermore, when the left and right side windows 372L and 372R are used as the projection surfaces, images in which the regions corresponding to the left side panel 330L and the right side panel 330R of the one-point perspective panel 330 are clipped in the front camera image 340 at time t are projected on the left and right side windows 372L and 372R as the presentation images.

[0104] According to the above configuration and processing, an image conforming to the concept of the one-point perspective projection method is output from the front camera image as the presentation image, so that the acceleration felt by a human and the mindset felt from the presentation image can be made to match, and it becomes possible to prevent discomfort and motion sickness in advance.

[0105] In the above configuration and processing, since the vehicle 80 (front camera 320) is moving forward, the scenery in front of the vehicle 80 shown in the region image 350 comes to be shown larger every predetermined time. Therefore, in the region images 350(t - 4) to 350(t) shown in FIG. 25, the space between the buildings standing on both sides of the road in front of the vehicle 80 widens over time, and the outline of the building that originally draws a straight line perpendicular to the road is inclined.

[0106] Therefore, a configuration for correcting the outline of the building recognized in the composite image in which the region images are combined so that the outline of the building becomes vertical using the information that "the building has a vertical outline with respect to the road" will be described.

[0107] FIG. 27 is a block diagram showing another configuration example of the image processing unit 40C of the present embodiment.

[0108] The image processing unit 40C in FIG. 27 includes an object recognition unit 381 and a correction processing unit 382 in addition to the same configuration as the image processing unit 40C in FIG. 17.

[0109] The object recognition unit 381 performs object recognition processing on the front camera image 340 from the out-of-vehicle image acquisition unit 311 in the same manner as the object recognition unit 213 in FIG. 11, and supplies the processing result to the correction processing unit 382.

[0110] The correction processing unit 382 corrects and repairs the composite image obtained by sequentially combining the region images generated by the composite unit 315 based on the processing result of the object recognition processing from the object recognition unit 381 and the object information stored in the object information definition unit 383, and supplies it to the blurring processing unit 316. The object information definition unit 383 is configured in the same manner as the object information definition unit 215 in FIG. 11.

[0111] That is, the correction processing unit 382 corrects the composite image so that the contour of the building recognized in the front camera image 340 is vertical using the information that "the building has a vertical contour with respect to the road".

[0112] The blurring processing unit 316 performs blurring processing on the corrected and repaired composite image from the correction processing unit 382 and outputs it as a presentation image. Note that the blurring processing unit 316 may be omitted when the correction and repair by the correction processing unit 382 are performed with high accuracy.

[0113] According to the above configuration, it is possible to output a more natural presentation image without distortion.

[0114] <5. Fourth Embodiment> FIG. 28 is a block diagram showing a configuration example of an image processing unit 40D according to a fourth embodiment of the present technology.

[0115] The image processing unit 40D generates a presentation image corresponding to the traveling speed of the vehicle 80 based on the traveling speed of the vehicle obtained from an out-of-vehicle image that captures the out-of-vehicle environment of a predetermined vehicle during traveling.

[0116] For example, as the projected image 93 projected onto the ceiling 92 of the vehicle 80 described with reference to FIG. 4, when an out-of-vehicle image captured from a vehicle traveling (or that has traveled) in a completely different location is projected, the image processing unit 40D generates a presentation image that is reproduced in accordance with the traveling speed of the vehicle 80.

[0117] Thereby, it is possible to match the physical acceleration change of the user with the acceleration change of the presentation image, and it becomes possible to make the user feel as if they are driving in a different location.

[0118] The image processing unit 40D is composed of an image acquisition unit 411, a speed estimation unit 412, and a presentation control unit 413.

[0119] The image acquisition unit 411 acquires an out-of-vehicle image that has captured (or is capturing) the out-of-vehicle environment of a vehicle traveling (or that has traveled) in a different location, and supplies it to the speed estimation unit 412.

[0120] For example, as shown in FIG. 29, an out-of-vehicle image from a vehicle 420 traveling (or that has traveled) in a different location may be acquired by being transmitted to the vehicle 80 via a cloud server 431 connected to a predetermined network such as the Internet.

[0121] Alternatively, an out-of-vehicle image from a vehicle 420 traveling in a different location may be acquired by being recorded on a predetermined recording medium 432 and read out in the vehicle 80.

[0122] In FIG. 29, when an out-of-vehicle image that has captured the out-of-vehicle environment of a vehicle that has previously traveled in a different location is acquired, the vehicle 80 and the vehicle 420 may be different vehicles or the same vehicle.

[0123] Also, as the out-of-vehicle image that has captured the out-of-vehicle environment of a vehicle that has previously traveled in a different location, a presentation image (a fish-eye lens image or a front camera image after image processing) generated by the image processing units 40A to 40C (40C') of the first to third embodiments described above may be acquired.

[0124] The speed estimation unit 412 executes speed estimation processing. Specifically, the speed estimation unit 412 performs object recognition processing on the out-of-vehicle image from the image acquisition unit 411, and estimates the traveling speed of the vehicle 420 based on the processing result and the object information stored in the object information definition unit 414. The object information definition unit 414 is realized in a form such as a relational database or a lookup table in the storage unit 50, for example.

[0125] The estimated traveling speed is supplied to the presentation control unit 413. Also, when the traveling speed of the vehicle 420 is included in the metadata of the out-of-vehicle image, the traveling speed of the vehicle 420 included in the metadata of the out-of-vehicle image is directly supplied to the presentation control unit 413.

[0126] The presentation control unit 413 executes image presentation processing. Specifically, the presentation control unit 413 generates a presentation image corresponding to the traveling speed of the vehicle 80 based on the traveling speed of the vehicle 420 from the speed estimation unit 412. The generated presentation image is output to the graphics display processing unit 20 and presented inside the vehicle 91 of the vehicle 80 by the output unit 30.

[0127] Next, with reference to the flowchart of FIG. 30, the operation flow of the image processing unit 40D will be described.

[0128] In step S41, the out-of-vehicle image acquisition unit 411 acquires an out-of-vehicle image.

[0129] In step S42, the speed estimation unit 412 executes speed estimation processing.

[0130] In step S43, the presentation control unit 413 executes image presentation processing.

[0131] According to the above configuration and processing, it becomes possible to present a presentation image corresponding to the situation outside the vehicle inside the vehicle 80 while it is traveling.

[0132] (Flow of speed estimation process) First, the flow of the speed estimation process executed by the speed estimation unit 412 in step S42 of the flowchart in Fig. 30 will be described with reference to the flowcharts in Fig. 31 and 32. The speed estimation process is started, for example, when the user selects an image to be presented in the vehicle interior 91 and the image is acquired by the image acquisition unit 411.

[0133] In step S101, the speed estimation unit 412 determines whether or not speed information indicating the traveling speed of the vehicle 420 (the vehicle from which the outside-of-vehicle image was captured) is added to the outside-of-vehicle image from the image acquisition unit 411 as metadata.

[0134] If it is determined in step S101 that the speed information is added to the vehicle outside image, the vehicle outside image to which the speed information is added is provided as is to the presentation control unit 413, and the process ends.

[0135] On the other hand, if it is determined in step S101 that speed information is not added to the vehicle outside image, the process proceeds to step S102, where speed estimation unit 412 performs object recognition processing on the vehicle outside image.

[0136] For example, when the front camera image 340 shown in Fig. 21 is acquired as the outside-of-vehicle image, the speed estimation unit 412 performs object recognition processing on the front camera image 340 to divide the front camera image 340 into object attributes as shown in Fig. 33. The obtained processed image 440 is divided into areas for buildings, vehicles, roads, and road signs.

[0137] In step S103, the speed estimation unit 412 determines whether or not the object information definition unit 414 contains object information of an object of interest (hereinafter, referred to as an object of interest) among objects recognized by the object recognition process on the outside-of-vehicle image.

[0138] FIG. 34 is a diagram showing an example of object information.

[0139] The object information OJ401 shown in FIG. 34 is information indicating whether each object can move and whether it is an object of a specified size. An object of a specified size generally refers to an object with a determined size. In the example of FIG. 34, buildings and roads are objects that cannot move and are not of a specified size (the size is "invalid"), vehicles are objects that can move and are of a specified size (the size is "valid"), and road signs are objects that cannot move and are of a specified size (the size is "valid"). For an object with a "valid" size, its actual size is also defined in the object information. Such object information is used to determine the object to be speed-estimated, which is the object used for speed estimation as described later.

[0140] Note that, as shown in FIG. 34, the object information OJ401 may include information indicating whether each object has sides or surfaces parallel to the road on which the vehicle travels.

[0141] Now, in step S103, if it is determined that the object information corresponding to the target object exists in the object information definition unit 414, the process proceeds to step S104.

[0142] In step S104, the speed estimation unit 412 determines whether the target object cannot move based on the object information corresponding to the target object. Here, an object moving towards the vehicle 80 is not an object to be speed-estimated, and an object that does not move, such as a road sign, can be an object to be speed-estimated.

[0143] In step S104, if it is determined that the target object cannot move, the process proceeds to step S105.

[0144] In step S105, the speed estimation unit 412 determines whether the size of the target object is equal to or greater than a certain size. For example, it is determined whether the vertical and horizontal lengths of the target object in the out-of-vehicle image are each greater than 20 px (pixels). Here, in order to avoid a decrease in the reliability of the estimated traveling speed, an object with a large area in the out-of-vehicle image that occupies the target object can be the object to be speed-estimated.

[0145] In step S105, if it is determined that the size of the target object is equal to or greater than a certain size, the process proceeds to step S106, and the speed estimation unit 412 adds the target object to the speed-estimated object candidate that is a candidate for the object to be speed-estimated.

[0146] On the other hand, in step S103, if it is determined that the object information corresponding to the target object does not exist in the object information definition unit 414, or in step S104, if it is determined that the target object is movable, or in step S105, if it is determined that the size of the target object is not equal to or greater than a certain size, the processing up to step S106 is skipped.

[0147] Then, in step S107, the speed estimation unit 412 determines whether the processing of steps S103 to S106 has been executed for all the objects recognized by the object recognition process for the out-of-vehicle image.

[0148] If the processing of steps S103 to S106 has not been executed for all the objects, the process returns to step S103, and the processing of steps S103 to S106 is repeated.

[0149] On the other hand, if the processing of steps S103 to S106 has been executed for all the objects, the process proceeds to step S108 in FIG. 32.

[0150] In step S108, the speed estimation unit 412 determines the object to be speed-estimated from among the speed-estimated object candidates.

[0151] Here, when objects with "valid" and "invalid" sizes are mixed in the object information of a plurality of object candidates for speed estimation, the object with a "valid" size is preferentially determined as the object for speed estimation. Also, when only an object with either a "valid" or "invalid" size exists in the object information of a plurality of object candidates for speed estimation, an object with a larger size may be preferentially determined as the object for speed estimation.

[0152] Note that when none of the objects are added to the object candidates for speed estimation, the object for speed estimation is not determined.

[0153] In step S109, the speed estimation unit 412 determines whether the object for speed estimation has been determined.

[0154] In step S109, if it is determined that the object for speed estimation has been determined, the process proceeds to step S110, and the speed estimation unit 412 determines whether the size of the object for speed estimation is "valid".

[0155] In step S110, if it is determined that the size of the object for speed estimation is "valid", the process proceeds to step S111, and the speed estimation unit 412 calculates the absolute speed (actual speed) of the vehicle 420 as the traveling speed of the vehicle 420.

[0156] FIG. 35 is a diagram showing an example of the calculation of the absolute speed. In the example of FIG. 35, among the objects shown in the front camera image 340, assume that a road sign defined as "immovable", "valid" in size, and an actual size of 60 cm in diameter as object information is determined as the object for speed estimation.

[0157] In the example of FIG. 35, a part of the front camera image 340 at time t and at time t + 1 one second later is shown, and it shows how the road sign approaches the vehicle 420 (in the foreground) from time t to time t + 1. Specifically, as shown in FIG. 35, in the front camera image 340, a road sign with a size (lengths in both vertical and horizontal directions) of 30 px has moved 300 px in one second.

[0158] In this case, the actual distance at which the road sign approaches vehicle 80 per second is 300 (px) × 0.6 (m) / 30 (px) = 6 m. That is, the moving speed (km / h) of vehicle 420 is 21.6 km / h obtained by converting 6 m / s, which becomes the absolute speed of vehicle 420.

[0159] As described above, the absolute speed of vehicle 420 is calculated based on the amount of movement of an object of a specified size in the out-of-vehicle image. When there are a plurality of objects of the specified size, the absolute speed may be calculated based on the amount of movement of each object, and the average value or median value of these absolute speeds may be used as the final absolute speed of vehicle 420.

[0160] Now, returning to the flowchart of FIG. 32, in step S110, when it is determined that the size of the object to be speed-estimated is not "valid", that is, when the size of the object to be speed-estimated is "invalid", the process proceeds to step S111.

[0161] In step S111, the speed estimation unit 412 calculates a relative speed based on the moving speed of a predetermined object in the out-of-vehicle image as the running speed of vehicle 420. In this case, although the actual size of the object to be speed-estimated is unknown, it is possible to calculate the moving speed (px / km) of the object to be speed-estimated in the out-of-vehicle image. Therefore, for example, a value obtained by multiplying the moving speed of the object to be speed-estimated in the out-of-vehicle image by a predetermined coefficient is set as the relative speed of vehicle 420.

[0162] On the other hand, in step S109, when it is determined that the object to be speed-estimated has not been determined, that is, when none of the objects have been added to the object candidates for speed estimation, the running speed of vehicle 420 is not estimated and the process ends.

[0163] The speed information representing the running speed (absolute speed or relative speed) of vehicle 420 estimated as described above is added to the out-of-vehicle image acquired by the image acquisition unit 411 and supplied to the presentation control unit 413.

[0164] (Flow of Image Presentation Processing) Next, with reference to the flowchart of FIG. 36, the flow of the image presentation processing executed by the presentation control unit 413 in step S43 of the flowchart of FIG. 30 will be described. The image presentation processing is started, for example, when the user gives an instruction to present an image.

[0165] In step S201, the presentation control unit 413 acquires the traveling speed of the vehicle 80 in motion from a speed sensor configured as the input unit 10.

[0166] In step S202, the presentation control unit 413 determines whether speed information representing the traveling speed of the vehicle 420 (the vehicle in which the out-of-vehicle image was taken) is added to the out-of-vehicle image from the speed estimation unit 412.

[0167] In step S202, if it is determined that speed information is added to the out-of-vehicle image, the process proceeds to step S203.

[0168] In step S203, the presentation control unit 413 determines whether there is an out-of-vehicle image that matches the traveling speed of the vehicle 80, specifically, an out-of-vehicle image to which speed information representing a traveling speed that matches the traveling speed of the vehicle 80 is added.

[0169] In step S203, if it is determined that there is an out-of-vehicle image that matches the traveling speed of the vehicle 80, the process proceeds to step S204, and the presentation control unit 413 outputs the out-of-vehicle image that matches the traveling speed of the vehicle 80 as it is as the presentation image.

[0170] On the other hand, in step S203, if it is determined that there is no out-of-vehicle image that matches the traveling speed of the vehicle 80, the process proceeds to step S205.

[0171] In step S205, the presentation control unit 413 determines whether there is an out-of-vehicle image that is slower than the traveling speed of the vehicle 80, specifically, an out-of-vehicle image to which speed information representing a traveling speed that is slower than the traveling speed of the vehicle 80 is added.

[0172] In step S205, if it is determined that there is an external image slower than the traveling speed of the vehicle 80, the process proceeds to step S206, and the presentation control unit 413 outputs a presentation image configured to reproduce the external image at high speed corresponding to the traveling speed of the vehicle 80.

[0173] On the other hand, in step S205, if it is determined that there is no external image slower than the traveling speed of the vehicle 80, that is, if there is an external image faster than the traveling speed of the vehicle 80, the process proceeds to step S207.

[0174] In step S207, the presentation control unit 413 outputs a presentation image configured to reproduce the external image at low speed corresponding to the traveling speed of the vehicle 80.

[0175] Also, in step S202, if it is determined that no speed information is added to the external image, the process proceeds to step S208, and the presentation control unit 413 outputs the external image as it is, that is, regardless of the traveling speed of the vehicle 80, as a presentation image.

[0176] After each of steps S204, S206, S207, and S208, in step S209, the presentation control unit 413 determines whether or not the user has instructed the end of image reproduction in the vehicle interior 91 of the traveling vehicle 80.

[0177] In step S209, if it is determined that the end of reproduction has not been instructed, the process returns to step S201, and the subsequent processing is repeated. On the other hand, in step S209, if it is determined that the end of reproduction has been instructed, the process ends.

[0178] Here, an example of image reproduction corresponding to the traveling speed of the vehicle 80 will be described.

[0179] (Example of image reproduction when the absolute speed is calculated) FIG. 37 is a diagram showing an example of image reproduction corresponding to the traveling speed of vehicle 80 when the absolute speed of vehicle 420 is calculated. This example is also applicable when the traveling speed of vehicle 420 is included in the metadata of the out-of-vehicle image.

[0180] In each of FIGS. 37A, 37B, and 37C, vehicle 80 travels at 10 km / h in a certain time period and then travels at 20 km / h in the next time period.

[0181] FIG. 37A shows an example of image reproduction when there is an out-of-vehicle image that matches the traveling speed of vehicle 80. In the example of FIG. 37A, there are an out-of-vehicle image with speed information indicating that the absolute speed of vehicle 420 is 10 km / h (hereinafter referred to as a 10 km / h out-of-vehicle image, etc.) and a 20 km / h out-of-vehicle image.

[0182] In this case, while vehicle 80 is traveling at 10 km / h, the 10 km / h out-of-vehicle image is reproduced at 1x speed as the presentation image. Also, while vehicle 80 is traveling at 20 km / h, the 20 km / h out-of-vehicle image is reproduced at 1x speed as the presentation image. In the example of FIG. 37A, the 10 km / h out-of-vehicle image and the 20 km / h out-of-vehicle image are cross-faded and reproduced across the switching of the traveling speed of vehicle 80.

[0183] Note that by sandwiching a solid black image between the two out-of-vehicle images and performing cross-fade reproduction, the unnaturalness of the image switching can be further reduced.

[0184] FIG. 37B shows an example of image reproduction when there is an out-of-vehicle image slower than the traveling speed of vehicle 80. In the example of FIG. 37B, there are a 5 km / h out-of-vehicle image and a 20 km / h out-of-vehicle image.

[0185] Particularly in this case, while the vehicle 80 is traveling at 10 km / h, as the presentation image, an out-of-vehicle image at 5 km / h is repeatedly played back at 2 times the speed by frame skipping or the like. In the example of Figure B, since the playback time of the out-of-vehicle image at 5 km / h is less than the time during which the vehicle 80 is traveling at 10 km / h, the out-of-vehicle image at 5 km / h is repeatedly played back. However, when the playback time of the out-of-vehicle image at 5 km / h is sufficiently long, there is no need for repeated playback. Also in the example of Figure B, the out-of-vehicle image at 5 km / h and the out-of-vehicle image at 20 km / h are cross-faded and played back across the change in the traveling speed of the vehicle 80.

[0186] In Figure C, an example of image playback when there is an out-of-vehicle image faster than the traveling speed of the vehicle 80 is shown. In the example of Figure C, only the out-of-vehicle image at 20 km / h exists.

[0187] Particularly in this case, while the vehicle 80 is traveling at 10 km / h, as the presentation image, the out-of-vehicle image at 20 km / h is played back at 0.5 times the speed by frame interpolation or the like. In the example of Figure C, the out-of-vehicle image at 20 km / h is played back with the playback speed changed across the change in the traveling speed of the vehicle 80.

[0188] According to the above operations, it is possible to match the physical acceleration change of the user with the acceleration change of the presentation image, and it becomes possible to make the user feel as if driving in a different location.

[0189] (Example of image playback when the relative speed is calculated) Figure 38 is a diagram showing an example of image playback corresponding to the traveling speed of the vehicle 80 when the relative speed instead of the absolute speed of the vehicle 420 is calculated.

[0190] Similar to Figure 37, in each of Figures A, B, and C of Figure 38, the vehicle 80 is traveling at 10 km / h in a certain time period and traveling at 20 km / h in the next time period.

[0191] In the example of FIG. 38, in the out-of-vehicle image to which speed information representing the relative speed of the vehicle 420 is added, the slowest relative speed is s, and it is assumed that the relative speed s corresponds to 5 km / h of the vehicle 80.

[0192] In FIG. A, an example of image reproduction when there is an out-of-vehicle image that matches the traveling speed of the vehicle 80 is shown. In the example of FIG. A, there are an out-of-vehicle image to which speed information indicating that the relative speed of the vehicle 420 is 2s (hereinafter referred to as an out-of-vehicle image with a relative speed of 2s, etc.) and an out-of-vehicle image with a relative speed of 4s.

[0193] In this case, while the vehicle 80 is traveling at 10 km / h, the out-of-vehicle image with a relative speed of 2s is reproduced at 1x speed as the presentation image. Also, while the vehicle 80 is traveling at 20 km / h, the out-of-vehicle image with a relative speed of 4s is reproduced at 1x speed as the presentation image. In the example of FIG. A, the out-of-vehicle image with a relative speed of 2s and the out-of-vehicle image with a relative speed of 4s are cross-faded and reproduced across the switching of the traveling speed of the vehicle 80.

[0194] In FIG. B, an example of image reproduction when there is an out-of-vehicle image slower than the traveling speed of the vehicle 80 is shown. In the example of FIG. B, there are an out-of-vehicle image with a relative speed of s and an out-of-vehicle image with a relative speed of 4s.

[0195] Particularly in this case, while the vehicle 80 is traveling at 10 km / h, the out-of-vehicle image with a relative speed of s is repeatedly reproduced at 2x speed by frame skipping or the like as the presentation image. In the example of FIG. B, since the reproduction time of the out-of-vehicle image with a relative speed of s is less than the time during which the vehicle 80 is traveling at 10 km / h, the out-of-vehicle image with a relative speed of s is repeatedly reproduced. However, when the reproduction time of the out-of-vehicle image with a relative speed of s is long enough, there is no need for repeated reproduction. Also in the example of FIG. B, the out-of-vehicle image with a relative speed of s and the out-of-vehicle image with a relative speed of 4s are cross-faded and reproduced across the switching of the traveling speed of the vehicle 80.

[0196] In FIG. C, an example of image reproduction when there is an out-of-vehicle image faster than the traveling speed of the vehicle 80 is shown. In the example of FIG. C, only the out-of-vehicle image with a relative speed of 4s exists.

[0197] Particularly in this case, while the vehicle 80 is traveling at 10 km / h, as the presentation image, an external image of the vehicle at a relative speed of 4 s is reproduced at 0.5 times the speed by frame interpolation or the like. In the example of FIG. C, the external image of the vehicle at a relative speed of 4 s is reproduced with the reproduction speed changed across the change in the traveling speed of the vehicle 80.

[0198] According to the above operation, even when the absolute speed of the vehicle 420 is not calculated, it is possible to match the physical acceleration change of the user with the acceleration change of the presentation image and reduce the sense of discomfort.

[0199] In the above, the slowest relative speed s in the external image of the vehicle 420 has been described as corresponding to 5 km / h of the vehicle 80.

[0200] Not limited to this, in the example of FIG. 38, when the vehicle 80 is traveling by autonomous driving, the relative speed of the external image and the traveling speed of the vehicle 80 may be associated based on the legal speed on the traveling route of the autonomous driving. The legal speed on the traveling route of the autonomous driving is obtained from, for example, map information input by the input unit 10.

[0201] For example, when there are external images with relative speeds s, 2 s, 4 s, and 8 s, the fastest relative speed 8 s is made to correspond to the legal maximum speed on the traveling route of the autonomous driving. Also, an intermediate value between the relative speed s and the relative speed 8 s may be made to correspond to the traveling speed at the time of switching from manual driving to autonomous driving. In the latter case, the reproduction of the external image is started at the timing of switching from manual driving to autonomous driving.

[0202] Also, object recognition processing may be performed in real time on an image obtained by photographing the external environment from the traveling vehicle 80, and the relative speed may be calculated based on roadside trees, buildings, signboards, etc. that are not objects of a specified size, so that the relative speed of the external image and the traveling speed of the vehicle 80 are associated.

[0203] <6. Fifth Embodiment> FIG. 39 is a block diagram showing a configuration example of an image processing unit 40E according to a fifth embodiment of the present technology.

[0204] The image processing unit 40E generates a presentation image corresponding to the traveling speed of the vehicle 80 based on the traveling speed of the vehicle obtained from an out-of-vehicle image that captures the out-of-vehicle environment of a predetermined vehicle during traveling. In the image processing unit 40E, as the out-of-vehicle image, similar to the image processing units 40A to 40C (40C') of the first to third embodiments described above, a fisheye lens image or a front camera image before image processing is acquired.

[0205] The image processing unit 40E in FIG. 39 is different from the image processing unit 40D in FIG. 28 in that an out-of-vehicle image processing unit 511 is provided instead of the image acquisition unit 411.

[0206] The out-of-vehicle image processing unit 511 is configured by any one of the image processing units 40A, 40B, and 40C (40C') of the first to third embodiments described above.

[0207] Next, with reference to the flowchart of FIG. 40, the operation flow of the image processing unit 40E will be described.

[0208] Note that the processes in steps S512 and S513 in the flowchart of FIG. 40 are the same as the processes in steps S42 and S43 in the flowchart of FIG. 30, so the description thereof is omitted.

[0209] That is, in step S511, the out-of-vehicle image processing unit 511 executes image processing on the out-of-vehicle image. Specifically, the out-of-vehicle image processing unit 511 acquires a fisheye lens image or a front camera image, and performs image processing by any one of the operations of the image processing units 40A, 40B, and 40C (40C') of the first to third embodiments described above. Thereafter, the above-described speed estimation processing and image presentation processing are executed on the fisheye lens image or the front camera image on which the image processing has been performed.

[0210] According to the above configuration and processing, even when the presentation image generated based on the fisheye lens image or the front camera image is reproduced, it is possible to match the physical acceleration change of the user with the acceleration change of the presentation image.

[0211] <7. Modification Example> The following configuration can also be applied to the above-described embodiment.

[0212] (Another Example of Projector Arrangement) In the vehicle interior 91 of the vehicle 80, when the ceiling 92 is used as the presentation site, as shown in FIG. 41, a projection image 93 from a projector 611 provided on the headrest 99 of the driver's seat may be projected onto the ceiling 92. In the example of FIG. 41, the projector 611 may be provided on the headrest 100 of the passenger seat instead of the headrest 99 of the driver's seat, or may be provided on both headrests 99 and 100.

[0213] (Another Example of Output Unit) As the presentation site in the vehicle interior 91 of the vehicle 80, a flat panel or a flexible image display (such as a liquid crystal display, an organic EL (Electro-Luminescence), etc.) may be provided as the output unit 30, and the presentation image may be displayed on the image display.

[0214] <8. Example of Computer Configuration> The above-described series of processes can be executed by hardware or by software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer in which the program is incorporated in dedicated hardware or a general-purpose personal computer or the like.

[0215] FIG. 42 is a block diagram showing a configuration example of the hardware of a computer that executes the above-described series of processes by a program.

[0216] The above-described image processing apparatus 1 is realized by a computer 900 having the configuration shown in FIG. 42.

[0217] The CPU 901, ROM 902, and RAM 903 are interconnected by a bus 904.

[0218] Further connected to the bus 904 is an input / output interface 905. Connected to the input / output interface 905 are an input unit 906 composed of a keyboard, a mouse, etc., and an output unit 907 composed of a display, a speaker, etc. Also connected to the input / output interface 905 are a storage unit 908 composed of a hard disk, a non-volatile memory, etc., a communication unit 909 composed of a network interface, etc., and a drive 910 for driving a removable medium 911.

[0219] In the computer 900 configured as described above, the CPU 901 loads and executes, for example, a program stored in the storage unit 908 into the RAM 903 via the input / output interface 905 and the bus 904, whereby the above-described series of processes are performed.

[0220] The program executed by the CPU 901 is recorded, for example, on the removable medium 911 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and installed in the storage unit 908.

[0221] Note that the program executed by the computer 900 may be a program in which processing is performed in time series in accordance with the order described in this specification, or may be a program in which processing is performed in parallel or at a necessary timing such as when a call is made.

[0222] In this specification, a system means a collection of a plurality of components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device in which a plurality of modules are housed in one housing are both systems.

[0223] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

[0224] Also, the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0225] Furthermore, the present disclosure can take the following configurations. (1) An image processing apparatus including an image processing unit that generates a presentation image to be presented inside a second vehicle in motion based on an external image of the first vehicle in motion. (2) The image processing unit generates the presentation image corresponding to the traveling speed of the second vehicle based on the traveling speed of the first vehicle acquired from the external image. The image processing apparatus according to (1). (3) When the traveling speed of the first vehicle matches the traveling speed of the second vehicle, the image processing unit outputs the external image as the presentation image. The image processing apparatus according to (2). (4) When the traveling speed of the first vehicle does not match the traveling speed of the second vehicle, the image processing unit outputs the external image at a reproduction speed corresponding to the traveling speed of the second vehicle as the presentation image. The image processing apparatus according to (3). (5) The traveling speed of the first vehicle is included in the metadata of the external image. An image processing device according to any one of (2) to (4). (6) The image processing unit estimates a traveling speed of the first vehicle based on a processing result of an object recognition process on the outside-of-vehicle image. An image processing device according to any one of (2) to (4). (7) When an object of a specified size is detected by the object recognition process, the image processing unit calculates an absolute speed of the first vehicle based on a movement amount of the object of the specified size in the outside-of-vehicle image. An image processing device according to (6). (8) When an object of a specified size is not detected by the object recognition process, the image processing unit calculates a relative speed based on a moving speed of a predetermined object in the outside-of-vehicle image. An image processing device according to (6) or (7). (9) the outside-of-vehicle image is a fisheye camera image taken in a zenith direction from the first vehicle, The image processing unit generates the presentation image by clipping an area based on the center of the fisheye camera image. An image processing device according to any one of (1) to (8). (10) The image processing unit clips the distortion-corrected fisheye camera image. An image processing device according to (9). (11) The image processing unit generates the presentation image by correcting the object in the fish-eye camera image according to a processing result of an object recognition process on at least a portion of the clipped fish-eye camera image. The image processing device according to (10). (12) the outside-of-vehicle image is a front camera image captured in the traveling direction of the first vehicle, The image processing unit generates the presentation image by sequentially synthesizing area images obtained by clipping an area of the front camera image corresponding to the presentation portion of the presentation image of the second vehicle at predetermined time intervals. The image processing apparatus according to any one of (1) to (8). (13) The image processing unit clips the region based on the vanishing point detected from the front camera image. The image processing apparatus according to (12). (14) The image processing unit generates the presentation image by performing a blurring process on a composite image obtained by sequentially compositing the region images. The image processing apparatus according to (13). (15) The presentation part of the presentation image includes the ceiling inside the second vehicle. The image processing apparatus according to any one of (1) to (14). (16) The first vehicle and the second vehicle are different vehicles. The image processing apparatus according to any one of (1) to (15). (17) The first vehicle and the second vehicle are the same vehicle. The image processing apparatus according to any one of (1) to (15). (18) An image processing apparatus, generates a presentation image to be presented inside the second vehicle during travel based on an external image of the external environment of the first vehicle during travel. An image processing method. (19) Causes a computer to generate a presentation image to be presented inside the second vehicle during travel based on an external image of the external environment of the first vehicle during travel. A program for executing the process. (20) An image processing unit that generates a presentation image to be presented inside the second vehicle during travel based on an external image of the external environment of the first vehicle during travel An image processing apparatus having the same, A presentation unit that presents the presentation image inside the second vehicle A presentation apparatus having the same An image presentation system including the same.

Description of Symbols

[0226] 1 Image processing apparatus, 10 Input unit, 20 Graphics display processing unit, 30 Output unit, 40, 40A to 40E Image processing units, 50 Storage unit

Claims

1. An out-of-vehicle image obtained by photographing the out-of-vehicle environment of a first vehicle during travel, the out-of-vehicle image received by an input unit, and based on the obtained out-of-vehicle image, a presentation image presented inside a second vehicle during travel, an image processing unit that generates the presentation image reproduced in accordance with the travel speed of the second vehicle comprising the out-of-vehicle image is a front camera image that photographs the traveling direction of the first vehicle the image processing unit generates the presentation image by sequentially synthesizing region images obtained by clipping, at predetermined time intervals, a region corresponding to a presentation site of the presentation image including the ceiling inside the second vehicle in the front camera image an image processing apparatus

2. the image processing unit generates the presentation image corresponding to the travel speed of the second vehicle based on the travel speed of the first vehicle obtained from the out-of-vehicle image The image processing apparatus according to claim 1

3. when the travel speed of the first vehicle matches the travel speed of the second vehicle, the image processing unit outputs the out-of-vehicle image as the presentation image The image processing apparatus according to claim 2

4. when the travel speed of the first vehicle does not match the travel speed of the second vehicle, the image processing unit outputs, as the presentation image, the out-of-vehicle image at a reproduction speed corresponding to the travel speed of the second vehicle The image processing apparatus according to claim 3

5. the travel speed of the first vehicle is included in the metadata of the out-of-vehicle image The image processing apparatus according to claim 2

6. the image processing unit estimates the travel speed of the first vehicle based on the processing result of object recognition processing on the out-of-vehicle image The image processing apparatus according to claim 2

7. when an object of a specified size is detected by the object recognition processing, the image processing unit calculates the absolute speed of the first vehicle based on the movement amount of the object of the specified size in the out-of-vehicle image The image processing apparatus according to claim 6

8. when an object of a specified size is not detected by the object recognition processing, the image processing unit calculates a relative speed based on the moving speed of a predetermined object in the out-of-vehicle image The image processing apparatus according to claim 6

9. the image processing unit clips the region based on the vanishing point detected from the front camera image The image processing apparatus according to claim 1

10. The image processing unit generates the presentation image by performing a blurring process on a composite image obtained by sequentially combining the region images. The image processing apparatus according to claim 9.

11. The first vehicle and the second vehicle are different vehicles. The image processing apparatus according to claim 1.

12. The first vehicle and the second vehicle are the same vehicle. The image processing apparatus according to claim 1.

13. An image processing apparatus that generates a presentation image to be presented inside a second vehicle during travel, based on a front camera image that captures the traveling direction of a first vehicle and is an out-of-vehicle image of the out-of-vehicle environment of the first vehicle during travel, and the presentation image is reproduced in accordance with the traveling speed of the second vehicle, acquires the front camera image received as input by an input unit, generates the presentation image by sequentially combining region images obtained by clipping, at predetermined time intervals, a region corresponding to a presentation site of the presentation image that includes the ceiling inside the second vehicle, in the acquired front camera image. Image processing method.

14. A computer that generates a presentation image to be presented inside a second vehicle during travel, based on a front camera image that captures the traveling direction of a first vehicle and is an out-of-vehicle image of the out-of-vehicle environment of the first vehicle during travel, and the presentation image is reproduced in accordance with the traveling speed of the second vehicle, acquires the front camera image received as input by an input unit, generates the presentation image by sequentially combining region images obtained by clipping, at predetermined time intervals, a region corresponding to a presentation site of the presentation image that includes the ceiling inside the second vehicle, in the acquired front camera image. A program for executing the process.

15. An image processing unit that obtains an out-of-vehicle image that is an out-of-vehicle image of the out-of-vehicle environment of a first vehicle during travel, the out-of-vehicle image being received as input by an input unit, and generates a presentation image to be presented inside a second vehicle during travel, based on the obtained out-of-vehicle image, and the presentation image is reproduced in accordance with the traveling speed of the second vehicle An image processing apparatus having; A presentation unit that presents the presentation image inside the second vehicle A presentation apparatus having and is provided with The out-of-vehicle image is a front camera image that captures the traveling direction of the first vehicle. The image processing unit generates the presentation image by sequentially synthesizing region images obtained by clipping, at each predetermined time, a region corresponding to a presentation site of the presentation image including the ceiling inside the second vehicle in the front camera image. An image presentation system.

Images