Image processing device, image processing method, program product and image rendering system

Abstract

The present disclosure relates to an image processing apparatus, an image processing method, a program, and an image presentation system that make it possible to present a more preferable image in a vehicle. An image processing unit generates a presentation image that is presented inside a second vehicle during travel, on the basis of a vehicle exterior image obtained by capturing a vehicle exterior environment of a first vehicle during travel. The present disclosure is applicable to, for example, a projector-type presentation apparatus.

Description

Technical Field

[0001] This disclosure relates to image processing apparatus, image processing methods, programs, and image presentation systems, and more particularly to image processing apparatus, image processing methods, programs, and image presentation systems capable of presenting more suitable images to the interior of a vehicle. Background Technology

[0002] PTL 1 discloses a technique for generating a developed image of a fisheye image on a cylindrical surface by transforming a fisheye image captured via a fisheye lens. According to this technique, the three-dimensional position of a subject can be estimated from the developed image with minimal distortion.

[0003] Reference List

[0004] Patent documents

[0005] PTL 1: JP 2012-226645 A Summary of the Invention

[0006] Technical issues

[0007] However, users may feel uncomfortable when an image that does not correspond to the conditions outside a moving vehicle is presented as an image of the vehicle's interior.

[0008] This disclosure was made in view of such circumstances and to enable the presentation of more suitable images into the interior of the vehicle.

[0009] Solution to the problem

[0010] The image processing apparatus disclosed herein is an image processing apparatus that includes an image processing unit configured to generate a presentation image to be presented into the interior of a moving second vehicle based on an image of the exterior of the vehicle obtained by capturing an image of the environment outside the exterior of a moving first vehicle.

[0011] The image processing method disclosed herein is an image processing method comprising: generating, by an image processing device, a rendering image to be rendered into the interior of a moving second vehicle based on an image of the exterior of the vehicle obtained by capturing an image of the environment outside the exterior of a first vehicle in motion.

[0012] The program disclosed herein is a program for causing a computer to perform the following process: generating a rendering image to be rendered into the interior of a moving second vehicle based on an image of the exterior of the vehicle obtained by capturing an image of the environment outside the exterior of a first moving vehicle.

[0013] The image rendering system disclosed herein is an image rendering system comprising: an image processing device including an image processing unit configured to generate a rendering image to be rendered into the interior of a moving second vehicle based on an image of the exterior of the vehicle obtained by capturing an image of the environment outside the exterior of a moving first vehicle; and a rendering device including a rendering unit configured to render the rendering image into the interior of the second vehicle.

[0014] In this disclosure, a rendering image to be rendered into the interior of a moving second vehicle is generated based on an image of the exterior of the vehicle obtained by capturing an image of the environment outside the first vehicle in motion. Attached Figure Description

[0015] Figure 1 This is a block diagram illustrating the configuration of an image processing apparatus to which the technology according to this disclosure is applied.

[0016] Figure 2 This is a diagram illustrating an application example of an image processing device.

[0017] Figure 3 This is a diagram illustrating an application example of an image processing device.

[0018] Figure 4 This is a diagram illustrating an application example of an image processing device.

[0019] Figure 5 This is a flowchart illustrating the operation flow of the image processing unit.

[0020] Figure 6 This is a block diagram illustrating an example configuration of an image processing unit according to the first embodiment.

[0021] Figure 7 This is a diagram showing the arrangement of the fisheye camera.

[0022] Figure 8 This is a flowchart illustrating the operation flow of the image processing unit.

[0023] Figure 9 This is a diagram illustrating an example of correction for a fisheye camera image.

[0024] Figure 10 This is a diagram illustrating an example of correction for a fisheye camera image.

[0025] Figure 11 This is a block diagram illustrating an example configuration of the image processing unit of the second embodiment.

[0026] Figure 12 This is a diagram illustrating semantic segmentation.

[0027] Figure 13 This is a flowchart illustrating the operation flow of the image processing unit.

[0028] Figure 14 This is a diagram illustrating an example of correction for a fisheye camera image.

[0029] Figure 15 This is a diagram illustrating an example of correction for a fisheye camera image.

[0030] Figure 16 This is a diagram showing an example of object information.

[0031] Figure 17 This is a block diagram illustrating an example configuration of an image processing unit according to a third embodiment.

[0032] Figure 18 This is a diagram showing the arrangement of the front cameras.

[0033] Figure 19 This is a diagram showing a one-point perspective panel.

[0034] Figure 20 This is a flowchart illustrating the operation flow of the image processing unit.

[0035] Figure 21 This is an example diagram showing an image from a front-facing camera.

[0036] Figure 22 This is a diagram showing the detection of vanishing points.

[0037] Figure 23 This is a diagram showing the overlay of a single-point perspective panel.

[0038] Figure 24 This is a diagram illustrating clipping.

[0039] Figure 25 This is a diagram illustrating the synthesis and blurring of a region image.

[0040] Figure 26 This is a diagram showing an example of a projection that renders an image.

[0041] Figure 27 This is a block diagram illustrating another configuration example of the image processing unit.

[0042] Figure 28 This is a block diagram illustrating an example configuration of an image processing unit according to a fourth embodiment.

[0043] Figure 29 This is a diagram showing the acquisition of images of the vehicle's exterior.

[0044] Figure 30 This is a flowchart illustrating the operation flow of the image processing unit.

[0045] Figure 31 This is a flowchart illustrating the process of velocity estimation.

[0046] Figure 32 This is a flowchart illustrating the process of velocity estimation.

[0047] Figure 33 This is a diagram illustrating an example of object recognition processing.

[0048] Figure 34 This is a diagram showing an example of object information.

[0049] Figure 35 This is a diagram illustrating an example of calculating absolute velocity.

[0050] Figure 36 This is a flowchart illustrating the image rendering process.

[0051] Figure 37 This is a diagram showing an example of image reproduction corresponding to the vehicle's speed.

[0052] Figure 38 This is a diagram showing an example of image reproduction corresponding to the vehicle's speed.

[0053] Figure 39 This is a block diagram illustrating an example configuration of an image processing unit according to a fifth embodiment.

[0054] Figure 40 This is a flowchart illustrating the operation flow of the image processing unit.

[0055] Figure 41 This is another example of a projector arrangement.

[0056] Figure 42 This is a block diagram illustrating an example of a computer configuration. Detailed Implementation

[0057] Hereinafter, a mode for implementing this disclosure (hereinafter referred to as an embodiment) will be described. The description will proceed in the following order.

[0058] 1. Summary of the technology according to this disclosure

[0059] 2. First Embodiment (Correction of Fisheye Camera Images)

[0060] 3. Second Embodiment (Fisheye Camera Image Correction and Object Recognition Processing)

[0061] 4. Third embodiment (correction of front camera image)

[0062] 5. Fourth Embodiment (Image Reproduction Corresponding to Driving Speed)

[0063] 6. Fifth Embodiment (Image Correction and Image Reconstruction Corresponding to Driving Speed)

[0064] 7. Variation Examples

[0065] 8. Computer Configuration Example

[0066] <1. Summary of the technology according to this disclosure>

[0067] (Configuration of image processing equipment)

[0068] Figure 1 This is a block diagram illustrating the configuration of an image processing apparatus that applies the technology (the present technology) according to the present disclosure.

[0069] Figure 1 The image processing device 1 is configured as a projector-type presentation device that enables interactive image presentation. For example, in a vehicle such as a car, the image processing device 1 presents an image generated through predetermined image processing to a user by projecting an image via a projector.

[0070] The image processing device 1 includes an input unit 10, a graphics display processing unit 20, an output unit 30, an image processing unit 40, and a storage unit 50.

[0071] The input unit 10 includes a sensor group such as an image sensor, a depth sensor, a touch sensor, and a speed sensor, a general-purpose input device, and a communication device, and receives images projected onto the vehicle, user operations, vehicle speed, etc., as input. Visible light cameras, infrared cameras, etc., are used as image sensors capable of acquiring images. Stereo cameras, time-of-flight (ToF) sensors, etc., are used as depth sensors capable of acquiring 3D information. Input information such as various types of sensor data input through the input unit 10 is provided to the graphics display processing unit 20.

[0072] The graphics display processing unit 20 performs processing to display graphics to the user based on input information from the input unit 10. The graphics display processing unit 20 is configured as the control layer of a general-purpose operating system (OS) that controls the drawing of multiple contents (such as windows for displaying applications) and distributes events such as touch operations to each content. The input information provided to the graphics display processing unit 20 is also provided to the image processing unit 40.

[0073] The output unit 30 consists of a projector that serves as one or more presentation units and presents images to the user.

[0074] The image processing unit 40 performs predetermined image processing based on the input information provided by the graphics display processing unit 20. The image obtained through image processing is then displayed by the output unit 30 via the graphics display processing unit 20.

[0075] Storage unit 50 stores information such as that required for image processing performed by image processing unit 40.

[0076] at the same time, Figure 1 The configuration can be configured as an image rendering system, which includes an image processing device having at least an image processing unit 40 and a rendering device having at least an output unit 30.

[0077] (Example of image processing equipment application)

[0078] Here, we will describe the Figure 1 An example of an image processing device 1 being used in a configuration that projects images onto the interior of a vehicle.

[0079] Figure 2 and Figure 3 These are top and side views of the vehicle 80 equipped with image processing equipment 1.

[0080] exist Figure 2 and Figure 3 In the middle, will be with Figure 1 The projectors 81, 82 and 83 corresponding to the output unit 30 of the image processing device 1 are installed inside the vehicle 80.

[0081] Projector 81 is installed near the center of the rear end of the vehicle interior in both the vertical and horizontal directions. Projector 82 is installed on the upper right side of the rear interior of the vehicle interior. Projector 83 is installed on the upper left side of the rear interior of the vehicle interior.

[0082] Figure 4 This is a diagram showing the location of the part (projection surface) on which an image is projected inside the vehicle 80.

[0083] When the roof 92 is used as a presentation region within the interior 91 of the vehicle 80, a projected image 93 from the projector 81 is projected onto the roof 92. When the vehicle 80 is equipped with a sunroof, the sliding panel that covers the sunroof is used as the presentation region.

[0084] When the left-side window 94 of the rear seat is used as the display area, the projected image 95 from the projector 82 is projected onto the left-side window 94. Furthermore, when the right-side window 96 of the rear seat is used as the display area, the projected image 97 from the projector 83 is projected onto the right-side window 96.

[0085] In addition to the roof 92, the left side glass 94 and the right side glass 96 of the rear seats, the center armrest 98 of the rear seats, the headrest 99 of the driver's seat, the headrest 100 of the passenger seat, the side glass 101 of the passenger seat and the side glass 102 of the driver's seat can also be used as display areas.

[0086] In this configuration, the projected images from any of projectors 81 to 83 are projected onto the armrest 98, the headrest 99 of the driver's seat, and the headrest 100 of the passenger seat. Furthermore, the projected images from any of projectors 82 and 83 are projected onto the side window 101 on the passenger seat side and the side window 102 on the driver's seat side.

[0087] With this configuration, all areas of the vehicle interior 91 can be interactively used as presentation areas, and screens can be provided as needed while utilizing the texture of the vehicle interior 91's interior trim. Furthermore, by using a projector, large areas such as the ceiling can be utilized as presentation areas at a lower cost compared to traditional LCD monitors.

[0088] In recent years, due to rising vehicle safety standards and increasing buyer expectations, automakers and others have needed to achieve above-average goals, such as "owning lightweight and durable vehicles." On the other hand, projectors that can display large images while saving space are considered useful.

[0089] (Operation of the image processing unit)

[0090] Here, we will refer to Figure 5 The operation flow of the image processing unit 40 included in the image processing device 1 is described using a flowchart. It is executed upon user instruction, such as displaying an image, while the vehicle 80 is in motion. Figure 5 The processing.

[0091] In step S1, the image processing unit 40 of the image processing device 1 acquires the vehicle exterior image input through the input unit 10. The vehicle exterior image is an image obtained by capturing the environment outside a predetermined vehicle (first vehicle) that is currently in motion. The predetermined vehicle may be another vehicle different from vehicle 80, or it may be the same vehicle as vehicle 80 (vehicle 80 itself). Furthermore, the vehicle exterior image may be an image captured in the past from a predetermined vehicle that is currently in motion, or it may be an image captured in real time from a predetermined vehicle that is currently in motion.

[0092] In step S2, the image processing unit 40 generates an image (hereinafter referred to as the presentation image) to be presented to the interior 91 of the moving vehicle 80 (the second vehicle) based on the acquired exterior image of the vehicle. The presentation image is an image corresponding to the external conditions of the vehicle 80.

[0093] In step S3, the image processing unit 40 outputs the generated rendering image to the graphics display processing unit 20. The rendering image output to the graphics display processing unit 20 is then displayed on the interior 91 of the vehicle 80 by the output unit 30.

[0094] Based on the above processing, since the image corresponding to the external conditions of the vehicle is generated as an image presented to the interior 91 of the moving vehicle 80, a more suitable image can be presented to the interior of the vehicle.

[0095] Hereinafter, each embodiment of the image processing unit 40 will be described.

[0096] <2. First Embodiment>

[0097] Figure 6 This is a block diagram illustrating a configuration example of an image processing unit 40A according to a first embodiment of the present technology.

[0098] The image processing unit 40A generates a presentation image to be projected onto the ceiling 92, which is the projection surface of the vehicle interior 91, based on an image of the vehicle exterior obtained by capturing the environment outside the moving vehicle 80.

[0099] The image processing unit 40A includes a vehicle exterior image acquisition unit 111 and a cropping unit 112.

[0100] The vehicle exterior image acquisition unit 111 acquires vehicle exterior images from a camera with a fisheye lens (hereinafter referred to as a fisheye camera) configured as an input unit 10, and provides the vehicle exterior images to the cropping unit 112.

[0101] Figure 7 This is a diagram showing the arrangement of the fisheye camera.

[0102] like Figure 7 As shown, a fisheye camera 130 is mounted on the roof (outside the roof) of the vehicle 80 and captures images from the moving vehicle 80 in the zenith direction. The image captured by the fisheye camera 130 (hereinafter referred to as a fisheye camera image) is a moving image reflecting the entire environment of the vehicle 80 in the zenith direction. That is, the vehicle exterior image acquisition unit 111 acquires the fisheye camera image as a vehicle exterior image.

[0103] Here, to increase the field of view, a fisheye camera 130 is used instead of a central projection camera with a regular lens.

[0104] The cropping unit 112 corrects the vehicle exterior image (fisheye camera image) from the vehicle exterior image acquisition unit 111 to fit the ceiling 92, which serves as the projection surface of the vehicle interior 91, and outputs the corrected vehicle exterior image as a presentation image. By presenting the presentation image reflecting the vehicle exterior in the zenith direction onto the ceiling 92, the user can feel as if he / she is in a convertible.

[0105] Next, we will refer to Figure 8 The flowchart is used to describe the operation flow of the image processing unit 40A.

[0106] In step S11, the vehicle exterior image acquisition unit 111 acquires a fisheye camera image from the fisheye camera.

[0107] In step S12, the cutting unit 112 cuts the fisheye camera image.

[0108] For example, suppose that images have already been acquired from the fisheye camera 130, such as Figure 9 The image shown is a fisheye camera image 140 reflecting the buildings surrounding the moving vehicle 80. When the fisheye camera image 140 is projected as is onto the ceiling 92 inside the vehicle 91, peripheral distortion becomes apparent.

[0109] Therefore, the clipping unit 112 generates a magnified image 150 by magnifying the fisheye camera image 140, and clips the area CL101 corresponding to the projection surface (ceiling 92) of the vehicle interior 91 with the center of the fisheye camera image 140 as a reference.

[0110] Therefore, the output image has less distortion. However, since the image obtained through the clipping region CL101 is affected by the characteristics of the fisheye lens, for example, the top of a building that was originally a straight line is distorted.

[0111] Therefore, as Figure 10 As shown, the cropping unit 112 magnifies the fisheye camera image 140, generates a distortion-corrected image 160 by performing distortion correction on the fisheye camera image 140, and crops the region CL102 corresponding to the projection surface (ceiling 92) of the vehicle interior 91. Here, the distortion correction is performed on the premise that the curvature of the fisheye lens is known and consistent, without considering individual differences of fisheye lenses.

[0112] Using the above configuration and processing, a rendered image with minimal distortion can be output.

[0113] <3. Second Embodiment>

[0114] Figure 11 This is a block diagram illustrating a configuration example of an image processing unit 40B according to a second embodiment of the present technology.

[0115] The image processing unit 40B generates a presentation image to be projected onto the ceiling 92, which is the projection surface of the vehicle interior 91, based on an image of the vehicle exterior obtained by capturing the environment outside the moving vehicle 80.

[0116] The image processing unit 40B includes a vehicle exterior image acquisition unit 211, a cropping unit 212, an object recognition unit 213, and a correction processing unit 214.

[0117] The vehicle exterior image acquisition unit 211 and the cropping unit 212 have the same characteristics as... Figure 6 The vehicle exterior image acquisition unit 211 and the cropping unit 112 have the same function. Therefore, the fisheye camera image acquired by the vehicle exterior image acquisition unit 211 is enlarged and cropped (distortion corrected) by the cropping unit 112 and provided to the object recognition unit 213.

[0118] The object recognition unit 213 performs object recognition processing on at least a portion of the cropped fisheye camera image and provides the processing result together with the cropped fisheye camera image to the correction processing unit 214.

[0119] For example, regarding such Figure 12 The captured image 220 shown in the upper part is obtained by the object recognition unit 213 determining the attributes of the subject (object) pixel by pixel based on semantic segmentation, and segmenting the captured image 220 according to the attributes. Therefore, as shown in the upper part, the captured image 220 is obtained. Figure 12 The processed image 230 is shown in the lower part. In the processed image 230, cars, roads, sidewalks, houses, trees, sky, etc., are distinguished as attributes of the subject.

[0120] In addition to semantic segmentation, the object recognition unit 213 can also perform object recognition processing through other methods.

[0121] The correction processing unit 214 corrects / repairs the cropped fisheye camera image based on the object recognition processing result from the object recognition unit 213 and the object information accumulated in the object information definition unit 215, and outputs it as a presentation image. For example, the object information definition unit 215 is implemented in the storage unit 50 in the form of a relational database or lookup table.

[0122] Next, we will refer to Figure 13 The flowchart is used to describe the operation flow of the image processing unit 40B.

[0123] because Figure 13 The processing of steps S21 and S22 in the flowchart and Figure 8 The processes for steps S11 and S12 in the flowchart are the same, so their description will be omitted.

[0124] That is, when the fisheye camera image is cropped in step S22, in step S23, the object recognition unit 213 performs object recognition processing on at least a portion of the cropped fisheye camera image.

[0125] In step S24, the correction processing unit 214 corrects the cropped fisheye camera image based on the object information accumulated in the object information definition unit 215.

[0126] For example, such as Figure 14 As shown, assuming from the reference Figure 10 The distortion-corrected image 160 described above yields a rendering image 240 in which region CL102 has been cropped.

[0127] The object recognition unit 213 extracts the portion enclosed by the rectangular box 240A indicated by the dashed line from the presented image 240 to obtain the extracted image 250. Then, the object recognition unit 213 performs object recognition processing on the extracted image 250 to segment the extracted image 250 according to the attributes of the objects. The obtained processed image 260 is divided into regions of buildings, trees, sky, and lights.

[0128] Then, the correction processing unit 214 corrects the presentation image 240 based on the object recognition processing result from the object recognition unit 213 and the object information accumulated in the object information definition unit 215, in order to obtain Figure 15 The corrected image 270 is shown.

[0129] Figure 16 This is a diagram showing an example of object information.

[0130] Figure 16 The object information OJ201 shown indicates whether each object has an edge or face that is parallel to the road (ground plane) on which the vehicle travels. Figure 16 The example shows that buildings have sides and faces parallel to the road, while trees, the sky, and lights do not.

[0131] Here, the correction processing unit 214 uses information indicating that "the building has sides and faces parallel to the road" to correct the presentation image 240, so that the top side of the building identified in the presentation image 240 becomes parallel to the road (a straight line relative to the road).

[0132] In the corrected image 270 obtained in this way, a portion is missing, as shown by the black area in the figure. Therefore, for example, the correction processing unit 214 obtains the repaired image 280 by performing image inpainting based on the corrected image 270, and outputs the repaired image 280 as the presentation image.

[0133] Using the above configuration and processing, a more distortion-free image can be output.

[0134] <4. Third Embodiment>

[0135] Figure 17 This is a block diagram illustrating a configuration example of an image processing unit 40C according to a third embodiment of the present technology.

[0136] The image processing unit 40C generates a presentation image to be projected onto the ceiling 92, which is the projection surface of the vehicle interior 91, based on an image of the vehicle exterior obtained by capturing the environment outside the moving vehicle 80.

[0137] The image processing unit 40C includes a vehicle exterior image acquisition unit 311, a vanishing point detection unit 312, a panel overlay unit 313, a cropping unit 314, a compositing unit 315, and a blur processing unit 316.

[0138] The vehicle exterior image acquisition unit 311 acquires vehicle exterior images from the front camera configured as input unit 10 and provides them to the vanishing point detection unit 312.

[0139] Figure 18 This is a diagram showing the arrangement of the front cameras.

[0140] like Figure 18 As shown, the front camera 320 is positioned at the upper end of the windshield or the front end of the roof inside the vehicle 80, and captures images of the vehicle 80 in its driving direction. The image captured by the front camera 320 (hereinafter referred to as the front camera image) is a moving image reflecting the front view of the vehicle 80 in its driving direction. That is, the vehicle exterior image acquisition unit 311 acquires the front camera image as the vehicle exterior image.

[0141] The vanishing point detection unit 312 detects vanishing points from the front camera image received from the vehicle exterior image acquisition unit 311. For example, the vanishing point detection unit 312 acquires edge information from the front camera image. The vanishing point detection unit 312 outputs straight lines by performing a Hough transform on the acquired edge information and obtains the intersection points of the output straight lines. The vanishing point detection unit 312 obtains a range containing a large number of the acquired intersection points, averages the coordinates of the intersection points within this range, and sets the averaged coordinates as the vanishing point.

[0142] The panel overlay unit 313 overlays a single-point perspective panel onto a position based on the vanishing point detected by the vanishing point detection unit 312 in the front camera image.

[0143] Here, we will refer to Figure 19 Describe the single-point perspective panel.

[0144] exist Figure 19 In the process, the vanishing point VP is detected from the front camera image FV, and a single-point perspective panel 330 is superimposed to align it with the perspective line pointing towards the vanishing point VP. When the front camera image FV is considered as a perspective view drawn using single-point perspective, the single-point perspective panel 330 is an image used to identify areas in the front camera image FV corresponding to the windshield, the left and right side windows of the front seats, and the roof of the vehicle 80.

[0145] Specifically, the single-point perspective panel 330 includes a front panel 330F corresponding to the windshield of the vehicle 80, a left panel 330L corresponding to the left side window of the front seat, a right panel 330R corresponding to the right side window of the front seat, and a roof panel 330T corresponding to the roof.

[0146] Return to Figure 17 As described above, the cropping unit 314 crops the area corresponding to the presentation part (roof 92) of the vehicle 80 from the front camera image at predetermined time intervals (such as seconds or frames). The cropped area images at predetermined time intervals are sequentially provided to the compositing unit 315.

[0147] The compositing unit 315 sequentially synthesizes (combines) the region images from the cutting unit 314 to generate a composite image, and provides the composite image to the blurring unit 316.

[0148] The blurring unit 316 performs blurring processing on the composite image from the compositing unit 315 and outputs it as a presentation image. Even if there is no camera capturing an image in the zenith direction outside the vehicle, the presentation image reflecting the area corresponding to the roof 92 in the front camera image will be presented to the roof 92, thus making the user feel as if he / she is in a convertible.

[0149] Next, we will refer to Figure 20 The flowchart is used to describe the operation flow of the image processing unit 40C.

[0150] In step S31, the vehicle exterior image acquisition unit 311 acquires the front camera image from the front camera.

[0151] In step S32, the vanishing point detection unit 312 detects the vanishing point from the front camera image.

[0152] In step S33, the panel overlay unit 313 overlays the single-point perspective panel onto the position based on the vanishing point in the front camera image.

[0153] For example, suppose that data has already been acquired from the front camera 320, such as... Figure 21The image shown is a front camera image 340 reflecting the view in the direction of travel of vehicle 80. The front camera image 340 reflects buildings (e.g., structures) on both sides of the road, vehicles traveling in front of vehicle 80, parked vehicles, and road signs installed on the side of the road.

[0154] When the image 340 from the front camera is acquired, such as Figure 22 As shown, the vanishing point VP is detected from the front camera image 340, and as... Figure 23 As shown, a single-point perspective panel 330 is superimposed on a position based on the vanishing point VP in the front camera image 340.

[0155] When the single-point perspective panel 330 is superimposed on the front camera image 340, in step S34, as follows: Figure 24 As shown, the cutting unit 314 cuts out the region CL301, which is part of the roof panel 330T corresponding to the roof 92 of the vehicle 80, from the front camera image 340 at predetermined time intervals to obtain the region image 350. Figure 24 In the example, region CL301 corresponds to the front end of the ceiling in ceiling panel 330T.

[0156] In step S35, the compositing unit 315 sequentially composites the region images 350 cropped at predetermined time intervals. Specifically, as shown... Figure 25 As shown, five region images 350(t-4) to 350(t) acquired at time t-4 to time t are synthesized so that they are arranged sequentially from top to bottom.

[0157] Subsequently, in step S36, the blurring unit 316 performs blurring processing, for example using Gaussian filtering, on the synthesized images that have been synthesized into region images 350(t-4) to 350(t) to obtain the processed image 360, and outputs it as the presentation image.

[0158] Figure 26 This is a diagram showing an example of the projection of an image in vehicle 80.

[0159] Figure 26 The windshield 371, the left side window 372L and the right side window 372R of the front seats, and the roof 373 are shown as viewed from the rear seats of vehicle 80.

[0160] The view of the scenery in front of the vehicle 80 at time t is viewed through the windshield 371, and the rendered image (processed image 360) output at time t is projected onto the ceiling 373, which serves as the projection surface.

[0161] When the vehicle 80 is driving autonomously, the windshield 371 can be used as a projection surface in addition to the roof 373. In this case, an image of the area corresponding to the front panel 330F of the single-point perspective panel 330, which has been cropped from the front camera image 340 at time t, is projected onto the windshield 371 as the presentation image.

[0162] Furthermore, when the left glass 372L and the right glass 372R are used as projection surfaces, an image of the area corresponding to the left panel 330L and the right panel 330R of the single-point perspective panel 330, which has been cropped from the front camera image 340 at time t, is projected onto the left glass 372L and the right glass 372R as the presentation image.

[0163] Based on the above configuration and processing, the image output from the front camera is based on the concept of single-point perspective projection as the presentation image. Therefore, the acceleration perceived by the human can be consistent with the mindset felt from the presentation image, and discomfort and motion sickness can be prevented.

[0164] In the above configuration and processing, since the vehicle 80 (front camera 320) is moving forward, the view in front of the vehicle 80 reflected in the area image 350 increases in size at predetermined time intervals. Therefore, in Figure 25 In the area images 350(t-4) to 350(t) shown, the space between the buildings constructed on both sides of the road in front of vehicle 80 expands over time, so the outlines of the buildings, which were originally straight lines perpendicular to the road, become tilted.

[0165] Therefore, the following configuration will be described, in which information indicating that “the building has a profile perpendicular to the road” is used to correct the profile of the building identified in the synthetic image in which the region image has been synthesized, so that it is perpendicular.

[0166] Figure 27 This is a block diagram illustrating another configuration example of the image processing unit 40C in this embodiment.

[0167] Besides Figure 17 Apart from the components of the image processing unit 40C, Figure 27 The image processing unit 40C' also includes an object recognition unit 381 and a correction processing unit 382.

[0168] Object recognition unit 381 with Figure 11 The object recognition unit 213 performs object recognition processing on the front camera image 340 from the vehicle exterior image acquisition unit 311 in the same manner, and provides the processing result to the correction processing unit 382.

[0169] The correction processing unit 382, ​​based on the object recognition processing results from the object recognition unit 381 and the object information accumulated in the object information definition unit 383, corrects and repairs the composite image generated by the synthesis unit 315 sequentially synthesizing region images, and provides the corrected and repaired composite image to the blur processing unit 316. Figure 11 The object information definition unit 383 is configured in the same way as the object information definition unit 215.

[0170] In other words, the correction processing unit 382 uses information indicating that "the building has a profile perpendicular to the road" to correct the synthetic image so that the profile of the building identified in the front camera image 340 is vertical.

[0171] The blurring unit 316 performs blurring processing on the corrected / repaired composite image from the correction processing unit 382 and outputs it as the presentation image. When the correction / repair is performed by the correction processing unit 382 with high precision, the blurring unit 316 can be omitted.

[0172] Using the above configuration, you can output images that are less distorted and more natural.

[0173] <5. Fourth Embodiment>

[0174] Figure 28 This is a block diagram illustrating a configuration example of an image processing unit 40D according to a fourth embodiment of the present technology.

[0175] The image processing unit 40D generates a presentation image corresponding to the driving speed of the predetermined vehicle 80 based on the driving speed of the vehicle being driven, obtained from the vehicle exterior image of the vehicle's external environment.

[0176] For example, when an image of the exterior of a vehicle captured from a vehicle traveling (driving) in a completely different location is projected onto a reference... Figure 4 When the projected image 93 is projected onto the roof 92 of the vehicle 80, the image processing unit 40D generates a presentation image that reproduces the vehicle 80 at the same speed.

[0177] Therefore, the changes in acceleration in the user's body can be made consistent with the changes in acceleration in the presented image, thus allowing the user to feel as if he / she is driving in a different place.

[0178] The image processing unit 40D includes an image acquisition unit 411, a speed estimation unit 412, and a rendering control unit 413.

[0179] Image acquisition unit 411 acquires images of the vehicle's external environment that have been captured (are being captured) and are traveling (are traveling) in different locations, and provides the images to speed estimation unit 412.

[0180] For example, such as Figure 29 As shown, vehicle exterior images from vehicles 420 traveling (driving) in different locations are sent to vehicle 80 via a cloud server 431 connected to a predetermined network such as the Internet, thus enabling the acquisition of vehicle exterior images.

[0181] In addition, vehicle exterior images can be obtained by recording vehicle exterior images from vehicles 420 traveling in different locations in a predetermined recording medium 432 and reading the vehicle exterior images in vehicle 80.

[0182] exist Figure 29 In the process of acquiring images of the vehicle's exterior by capturing images of the vehicle's external environment while it is traveling in different locations, vehicle 80 and vehicle 420 may be different vehicles or the same vehicle.

[0183] Furthermore, as pre-captured images of the vehicle's external environment while driving in different locations, presentation images (image-processed fisheye lens images or front camera images) generated by the image processing units 40A to 40C (40C') of the first to third embodiments described above can be obtained.

[0184] The speed estimation unit 412 performs speed estimation processing. Specifically, the speed estimation unit 412 performs object recognition processing on the vehicle exterior image from the image acquisition unit 411, and estimates the driving speed of the vehicle 420 based on the processing result and the object information accumulated in the object information definition unit 414. For example, the object information definition unit 414 is implemented in the storage unit 50 in the form of a relational database or lookup table.

[0185] The estimated driving speed is provided to the presentation control unit 413. When the metadata of the vehicle exterior image includes the driving speed of the vehicle 420, the driving speed of the vehicle 420 included in the metadata of the vehicle exterior image is provided to the presentation control unit 413 as is.

[0186] The presentation control unit 413 performs image presentation processing. Specifically, the presentation control unit 413 generates a presentation image corresponding to the driving speed of the vehicle 80 based on the driving speed of the vehicle 420 from the speed estimation unit 412. The generated presentation image is output to the graphics display processing unit 20 and presented to the interior 91 of the vehicle 80 by the output unit 30.

[0187] Next, we will refer to Figure 30The flowchart is used to describe the operation process of the image processing unit 40D.

[0188] In step S41, the vehicle exterior image acquisition unit 411 acquires an exterior image of the vehicle.

[0189] In step S42, the speed estimation unit 412 performs speed estimation processing.

[0190] In step S43, the presentation control unit 413 performs image presentation processing.

[0191] Based on the above configuration and processing, an image corresponding to the external conditions of the vehicle 80 can be presented inside the moving vehicle 80.

[0192] (Flowchart of velocity estimation processing)

[0193] Here, firstly, we will refer to Figure 31 and Figure 32 A flowchart to describe in Figure 30 The flowchart describes the speed estimation process performed by the speed estimation unit 412 in step S42. For example, speed estimation processing begins when the user selects an image to be displayed inside the vehicle 91 and that image is acquired by the image acquisition unit 411.

[0194] In step S101, the speed estimation unit 412 determines whether speed information representing the driving speed of vehicle 420 (the vehicle whose exterior image has been captured) has been added as metadata to the vehicle exterior image from the image acquisition unit 411.

[0195] If it is determined in step S101 that speed information has been added to the vehicle exterior image, the vehicle exterior image with added speed information is provided to the presentation control unit 413 as is, and the processing ends.

[0196] On the other hand, if it is determined in step S101 that no speed information has been added to the vehicle exterior image, the process proceeds to step S102, in which the speed estimation unit 412 performs object recognition processing on the vehicle exterior image.

[0197] For example, when Figure 21 When the forward camera image 340 shown has been acquired as an external image of the vehicle, the speed estimation unit 412 performs object recognition processing on the forward camera image 340 to determine the object recognition based on the following: Figure 33 The attributes of the objects shown are used to divide the foreground camera image 340. The resulting processed image 440 is divided into areas for buildings, cars, roads, and signs.

[0198] In step S103, the speed estimation unit 412 determines whether there is object information in the object information definition unit 414 for an object of interest (hereinafter referred to as an object of interest) among the objects identified by object recognition processing performed on the vehicle exterior image.

[0199] Figure 34 This is a diagram showing an example of object information.

[0200] Figure 34 The object information shown in OJ401 indicates whether each object can be moved and whether each object is a specified size. A specified size object refers to an object with a normally defined size. Figure 34 In the example, buildings and roads are "immovable" and not objects of the specified size (having "invalid" size), vehicles are "movable" and are objects of the specified size (having "valid" size), and road signs are "immovable" and are objects of the specified size (having "valid" size). For objects with "valid" size, the actual size of the object is also defined in the object information. As described later, such object information is used to determine the speed estimation target object, which is the object used for speed estimation.

[0201] like Figure 34 As shown, the object information OJ401 may include information indicating whether each object has an edge or face parallel to the road on which the vehicle travels.

[0202] If it is determined in step S103 that there is object information in the object information definition unit 414 corresponding to the object of interest, then the process proceeds to step S104.

[0203] In step S104, the speed estimation unit 412 determines whether the object of interest is "immovable" based on the object information corresponding to the object of interest. Here, an object moving toward the vehicle 80 is not the speed estimation target object, while non-moving objects such as road signs can be the speed estimation target object.

[0204] If it is determined in step S104 that the object of interest is "not" movable, the process proceeds to step S105.

[0205] In step S105, the speed estimation unit 412 determines whether the size of the object of interest is a certain size or larger. For example, it determines whether the vertical and horizontal lengths of the object of interest in the vehicle exterior image are greater than 20 pixels. Here, to avoid reducing the reliability of the estimated driving speed, objects with a large area in the object of interest in the vehicle exterior image can be the target objects for speed estimation.

[0206] If it is determined in step S105 that the size of the object of interest is equal to or greater than a certain size, the process proceeds to step S106. In step S106, the velocity estimation unit 412 adds the object of interest to the velocity estimation target object candidate. This velocity estimation target object candidate is a candidate for the velocity estimation target object.

[0207] On the other hand, if it is determined in step S103 that there is no object information corresponding to the object of interest in the object information definition unit 414, if it is determined in step S104 that the object of interest "can" be moved, or if it is determined in step S105 that the size of the object of interest is not equal to or greater than a certain size, then the process is skipped until step S106.

[0208] Then, in step S107, the speed estimation unit 412 determines whether the processing of steps S103 to S106 has been performed on all objects identified by the object recognition processing performed on the vehicle exterior image.

[0209] If steps S103 to S106 have not yet been processed for all objects, the processing returns to step S103, and the processing of steps S103 to S106 is repeated.

[0210] On the other hand, if steps S103 to S106 have already been performed on all objects, then the process continues to... Figure 32 Step S108.

[0211] In step S108, the velocity estimation unit 412 determines the velocity estimation target object from the velocity estimation target object candidates.

[0212] Here, when objects with both "valid" and "invalid" sizes exist among multiple candidate velocity estimation target objects, the object with the "valid" size is preferentially identified as the velocity estimation target object. Furthermore, when objects with either "valid" or "invalid" sizes exist among multiple candidate velocity estimation target objects, the object with the larger size can be preferentially identified as the velocity estimation target object.

[0213] When no object is added to the velocity estimation target object candidate, the velocity estimation target object is uncertain.

[0214] In step S109, the velocity estimation unit 412 determines whether the velocity estimation target object has been identified.

[0215] If it is determined in step S109 that the target object for velocity estimation has been identified, the process proceeds to step S110, in which the velocity estimation unit 412 determines whether the size of the target object for velocity estimation is "valid".

[0216] If the size of the target object for speed estimation is determined to be "valid" in step S110, the process proceeds to step S111. In step S111, the speed estimation unit 412 calculates the absolute speed (actual speed) of the vehicle 420 as the driving speed of the vehicle 420.

[0217] Figure 35 This is a diagram illustrating an example of calculating absolute velocity. In Figure 35 In the example, it is assumed that among the objects reflected in the foreground camera image 340, a road sign with information that is defined as "immovable", has an "effective" size and an actual size of 60 cm in diameter is identified as the target object for speed estimation.

[0218] exist Figure 35 The example shows a portion of the front camera image 340 at time t and one second later at time t+1, and shows the state of a road sign approaching vehicle 420 (front side) from time t to time t+1. Specifically, as Figure 35 As shown, a road sign with a size of 30px (vertical length and horizontal length) moves 300px per second in the front camera image 340.

[0219] In this scenario, the actual distance the road sign travels in one second towards vehicle 80 is 300 (px) × 0.6 (m) / 30 (px) = 6 (m). That is, vehicle 420's speed (km / h) is 21.6 km / h, converted from 6 m / s; this is vehicle 420's absolute speed.

[0220] As described above, the absolute speed of vehicle 420 is calculated based on the amount of movement of objects of a specified size in the external image of the vehicle. If there are multiple objects of a specified size, the absolute speed can be calculated based on the amount of movement of each object, and the average or median of the absolute speeds can be used as the final absolute speed of vehicle 420.

[0221] Return to Figure 32 The flowchart shows that if it is determined in step S110 that the size of the velocity estimation target object is not "valid", that is, if the size of the velocity estimation target object is "invalid", then the process proceeds to step S111.

[0222] In step S111, the speed estimation unit 412 calculates the relative speed based on the moving speed of a predetermined object in the vehicle's external image, as the driving speed of the vehicle 420. In this case, although the actual size of the speed estimation target object is unknown, the moving speed (px / km) of the speed estimation target object in the vehicle's external image can be calculated. Therefore, for example, the value obtained by multiplying the moving speed of the speed estimation target object in the vehicle's external image by a predetermined coefficient is set as the relative speed of the vehicle 420.

[0223] On the other hand, when it is determined in step S109 that no speed estimation target object has been determined, that is, when no object is added to the speed estimation target object candidate, the process ends without estimating the driving speed of vehicle 420.

[0224] Speed ​​information, representing the estimated driving speed (absolute speed or relative speed) of vehicle 420 as described above, is added to the external image of the vehicle acquired by image acquisition unit 411 and provided to presentation control unit 413.

[0225] (Image rendering and processing workflow)

[0226] Next, we will refer to Figure 36 A flowchart to describe in Figure 30 The flowchart shows the image rendering process executed by the rendering control unit 413 in step S43. For example, the image rendering process is started by user instruction to render an image.

[0227] In step S201, the presentation control unit 413 obtains the driving speed of the moving vehicle 80 from the speed sensor configured as input unit 10.

[0228] In step S202, the presentation control unit 413 determines whether speed information representing the driving speed of vehicle 420 (the vehicle whose exterior image has been captured) has been added to the vehicle exterior image from the speed estimation unit 412.

[0229] If it is determined in step S202 that speed information has been added to the vehicle's external image, the process proceeds to step S203.

[0230] In step S203, the presentation control unit 413 determines whether there is a vehicle exterior image that matches the driving speed of the vehicle 80, specifically whether there is a vehicle exterior image that has been added with speed information indicating a driving speed that matches the driving speed of the vehicle 80.

[0231] If it is determined in step S203 that there is a vehicle exterior image with the same driving speed as vehicle 80, the process proceeds to step S204. In step S204, the presentation control unit 413 outputs the vehicle exterior image with the same driving speed as vehicle 80 as the presentation image.

[0232] On the other hand, if it is determined in step S203 that there is no external image of the vehicle that matches the driving speed of the vehicle 80, the process proceeds to step S205.

[0233] In step S205, the presentation control unit 413 determines whether there is a vehicle exterior image that is slower than the vehicle 80's driving speed, specifically whether there is a vehicle exterior image that has been added with speed information indicating a driving speed slower than the vehicle 80's driving speed.

[0234] If it is determined in step S205 that there is a vehicle exterior image that is slower than the vehicle 80's driving speed, the process proceeds to step S206, in which the presentation control unit 413 outputs a presentation image that reproduces the vehicle exterior image at a high speed corresponding to the vehicle 80's driving speed.

[0235] On the other hand, if it is determined in step S205 that there is no vehicle exterior image that is slower than the vehicle 80, that is, if there is a vehicle exterior image that is faster than the vehicle 80, then the process proceeds to step S207.

[0236] In step S207, the presentation control unit 413 outputs a presentation image that reproduces the external image of the vehicle at a low speed corresponding to the driving speed of the vehicle 80.

[0237] If it is determined in step S202 that no speed information is added to the vehicle exterior image, the process proceeds to step S208. In step S208, the presentation control unit 413 outputs the vehicle exterior image as a presentation image, that is, regardless of the vehicle 80's driving speed.

[0238] After each of steps S204, S206, S207 and S208, in step S209, the presentation control unit 413 determines whether the user has instructed to end image reproduction in the interior 91 of the moving vehicle 80.

[0239] If it is determined in step S209 that no indication to end reproduction has been given, the process returns to step S201 and the subsequent processing is repeated. On the other hand, if it is determined in step S209 that an indication to end reproduction has been given, the process ends.

[0240] Here, an example of image reproduction corresponding to the driving speed of vehicle 80 will be described.

[0241] (Example of image reproduction when absolute velocity is calculated)

[0242] Figure 37 This is a diagram illustrating an example of image reproduction corresponding to the travel speed of vehicle 80 when the absolute speed of vehicle 420 is calculated. This example is also applied when the metadata of the vehicle exterior image includes the travel speed of vehicle 420.

[0243] exist Figure 37 In each of A, B, and C, vehicle 80 travels at a speed of 10 km / h for a certain period of time and at a speed of 20 km / h for a subsequent period of time.

[0244] Figure A illustrates an example of image reproduction when an external image of the vehicle is present, matching the driving speed of vehicle 80. In the example in Figure A, there are external images of the vehicle 420 with added speed information indicating an absolute speed of 10 km / h (hereinafter referred to as the 10 km / h external image, etc.) and an external image of the vehicle at 20 km / h.

[0245] In this scenario, when vehicle 80 is traveling at 10 km / h, the exterior image of the vehicle at 10 km / h is reproduced at 1x speed as the presentation image. Furthermore, when vehicle 80 is traveling at 20 km / h, the exterior image of the vehicle at 20 km / h is reproduced at 1x speed as the presentation image. In the example in Figure A, when the vehicle 80's speed switches between 10 km / h and 20 km / h, the exterior images of the vehicle at 10 km / h and 20 km / h are reproduced with a crossfade.

[0246] Meanwhile, by inserting a single black image between two vehicle exterior images and then cross-fading in and out, the unnaturalness of image transitions can be further reduced.

[0247] Figure B illustrates an example of image reproduction when there are exterior images of the vehicle traveling at a speed slower than 80 km / h. In the example in Figure B, there are exterior images of the vehicle traveling at 5 km / h and 20 km / h.

[0248] In this scenario, specifically when vehicle 80 is traveling at 10 km / h, the exterior image of the vehicle at 5 km / h is reproduced at twice the speed as the presentation image, based on frame thinning out and other factors. In the example of Figure B, although the exterior image of the vehicle at 5 km / h is reproduced repeatedly because its reproduction time is shorter than the time it takes for vehicle 80 to travel at 10 km / h, it is unnecessary to reproduce it repeatedly when the reproduction time is sufficiently long. In the example of Figure B, when vehicle 80's speed switches between 5 km / h and 20 km / h, the exterior images of the vehicle at 5 km / h and 20 km / h are reproduced with crossfade-in / fade-out.

[0249] Figure C illustrates an example of image reproduction when there is an external image of the vehicle traveling at a speed faster than 80 km / h. In the example in Figure C, only an external image of the vehicle at 20 km / h exists.

[0250] In this case, specifically when vehicle 80 is traveling at 10 km / h, the external image of the vehicle at 20 km / h is reproduced as the presentation image at 0.5 times the speed, based on frame interpolation, etc. In the example of Figure C, when the driving speed of vehicle 80 switches between the two, the external image of the vehicle at 20 km / h is reproduced at a different reproduction speed.

[0251] Based on the above operation, the user's physical acceleration changes can be made consistent with the acceleration changes in the presented image, thus allowing the user to feel as if he / she is driving in a different place.

[0252] (Example of image reproduction when relative velocity is calculated)

[0253] Figure 38 This is an example of image reproduction corresponding to the travel speed of vehicle 80 when relative speed is calculated instead of the absolute speed of vehicle 420.

[0254] and Figure 37 Similarly, in Figure 38 In each of A, B, and C, vehicle 80 travels at 10 km / h for a certain period of time and at 20 km / h for a subsequent period of time.

[0255] exist Figure 38 In the example, suppose that in the vehicle exterior image where speed information representing the relative speed of vehicle 420 has been added, the slowest relative speed is s, and this relative speed s corresponds to 5 km / h for vehicle 80.

[0256] Figure A illustrates an example of image reproduction when an external image of the vehicle, which is traveling at the same speed as vehicle 80, is available. In the example in Figure A, there are external images of the vehicle with added speed information indicating a relative speed of 2 seconds for vehicle 420 (hereinafter referred to as external images of the vehicle with a relative speed of 2 seconds, etc.) and external images of the vehicle with a relative speed of 4 seconds.

[0257] In this scenario, when vehicle 80 is traveling at 10 km / h, an image of the vehicle's exterior with a relative speed of 2 seconds is reproduced at 1x speed as the presented image. Furthermore, when vehicle 80 is traveling at 20 km / h, an image of the vehicle's exterior with a relative speed of 4 seconds is reproduced at 1x speed as the presented image. In the example in Figure A, when the vehicle 80's speed switches between a relative speed of 2 seconds and a relative speed of 4 seconds, the images of the vehicle's exterior with a relative speed of 2 seconds and a relative speed of 4 seconds are reproduced with crossfade-in / fade-out.

[0258] Figure B illustrates an example of image reproduction when there is an external image of the vehicle traveling at a speed slower than that of the vehicle at 80. In the example in Figure B, there are external images of the vehicle with a relative speed of s and external images of the vehicle with a relative speed of 4s.

[0259] In this case, specifically, when vehicle 80 is traveling at 10 km / h, the vehicle exterior image with a relative speed of s is repeatedly reproduced as the presentation image at twice the speed, based on frame sparsity, etc. In the example of Figure B, although the vehicle exterior image with a relative speed of s is repeatedly reproduced because its reproduction time is less than the time during which vehicle 80 is traveling at 10 km / h, it is unnecessary to repeat the reproduction of the vehicle exterior image with a relative speed of s if its reproduction time is long enough. In the example of Figure B, when the vehicle 80's speed switches between a relative speed of s and a relative speed of 4s, the vehicle exterior image with a relative speed of s and the vehicle exterior image with a relative speed of 4s are also reproduced with crossfade-in / fade-out.

[0260] Figure C illustrates an example of image reproduction when there is an external image of the vehicle traveling at a speed faster than 80 mph. In the example in Figure C, only an external image of the vehicle with a relative speed of 4 s exists.

[0261] In this case, specifically when vehicle 80 is traveling at 10 km / h, an external image of the vehicle with a relative speed of 4 seconds is reproduced as the presentation image at 0.5 times the speed, based on frame interpolation, etc. In the example of Figure C, when the driving speed of vehicle 80 switches between these speeds, an external image of the vehicle with a relative speed of 4 seconds is reproduced at a different reproduction speed.

[0262] Based on the above operation, even without calculating the absolute speed of vehicle 420, the acceleration changes in the user's body can be made consistent with the acceleration changes in the presented image, thus reducing discomfort.

[0263] In the above description, the slowest relative speed s in the external image of vehicle 420 corresponds to 5 km / h for vehicle 80.

[0264] This disclosure is not limited thereto, and Figure 38 In the example, when vehicle 80 is driving under autonomous driving conditions, the relative speed of the vehicle's external image can be correlated with the vehicle 80's speed based on the legal speed along the autonomous driving route. The legal speed along the autonomous driving route is obtained from map information input, for example, through input unit 10.

[0265] For example, when vehicle exterior images with relative speeds of s, 2s, 4s, and 8s are available, the highest relative speed of 8s is made to correspond to the maximum legal speed on the autonomous driving route. Alternatively, an intermediate value between relative speeds s and 8s can be made to correspond to the driving speed when switching from manual to autonomous driving. In the latter case, the reproduction of the vehicle exterior images begins at the timing of the switch from manual to autonomous driving.

[0266] In addition, the relative speed of the vehicle's external image can be correlated with the vehicle's speed by performing object recognition processing on images of the vehicle's external environment captured from the moving vehicle 80 in real time, and calculating the relative speed based on objects such as roadside trees, buildings, and signs that are not of a specified size.

[0267] <6. Fifth Embodiment>

[0268] Figure 39 This is a block diagram illustrating a configuration example of the image processing unit 40E according to a fifth embodiment of the present technology.

[0269] The image processing unit 40E generates a presentation image corresponding to the driving speed of the predetermined vehicle 80 based on the driving speed of the vehicle being driven, obtained from an external image of the vehicle's external environment. In the image processing unit 40E, as in the image processing units 40A to 40C (40C') of the first to third embodiments described above, a fisheye lens image or a front camera image before image processing is performed is acquired as the external image of the vehicle.

[0270] Figure 39 Image processing unit 40E and Figure 28 The difference with the image processing unit 40D is that an external vehicle image processing unit 511 is provided instead of the image acquisition unit 411.

[0271] The vehicle exterior image processing unit 511 is configured as any one of the image processing units 40A, 40B and 40C (40C') of the first to third embodiments described above.

[0272] Next, we will refer to Figure 40 The flowchart is used to describe the operation flow of the image processing unit 40E.

[0273] because Figure 40 The processing of steps S512 and S513 in the flowchart and Figure 30 The processes in steps S42 and S43 of the flowchart are the same, so their description will be omitted.

[0274] That is, in step S511, the vehicle exterior image processing unit 511 performs image processing on the vehicle exterior image. Specifically, the vehicle exterior image processing unit 511 acquires a fisheye lens image or a front camera image, and performs image processing according to the operation of any one of the image processing units 40A, 40B, and 40C (40C') in the first to third embodiments described above. Thereafter, the speed estimation processing and image rendering processing described above are performed on the image-processed fisheye lens image or front camera image.

[0275] Based on the above configuration and processing, even when reproducing a presentation image generated based on a fisheye lens image or a front camera image, the acceleration changes on the user's body can be made consistent with the acceleration changes in the presentation image.

[0276] <7. Variation Examples>

[0277] The following configurations can also be applied to the above embodiments.

[0278] (Other examples of projector setups)

[0279] When the roof 92 is used as a presentation part in the interior 91 of the vehicle 80, such as Figure 41 As shown, a projected image 93 from a projector 611 mounted on the headrest 99 of the driver's seat can be projected onto the ceiling 92. Figure 41 In the example, the projector 611 may be mounted on the headrest 100 of the passenger seat instead of the headrest 99 of the driver's seat, or it may be mounted on both headrests 99 and 100.

[0280] (Other examples of output units)

[0281] As a presentation part inside the vehicle 80 91, a flat panel or flexible image display (liquid crystal display, organic electroluminescent (EL) or the like) can be provided as an output unit 30, and the presented image can be displayed on the image display.

[0282] <8. Computer Configuration Example>

[0283] The above series of processes can also be performed using hardware or software. When the series of processes are performed using software, the program for the software is installed from the program recording medium onto a computer or general-purpose personal computer embedded in dedicated hardware.

[0284] Figure 42 This is a block diagram illustrating an example of a computer hardware configuration for executing a program to perform the series of processes described above.

[0285] The aforementioned image processing device 1 comprises having Figure 42 The computer 900 with the configuration shown is implemented.

[0286] CPU 901, ROM 902 and RAM 903 are connected via bus 904.

[0287] The input / output interface 905 is further connected to the bus 904. An input unit 906, including a keyboard and mouse, and an output unit 907, including a display and speakers, are connected to the input / output interface 905. Furthermore, a storage unit 908, including a hard disk, non-volatile memory, etc., a communication unit 909, including a network interface, etc., and a driver 910 that drives the removable medium 911 are connected to the input / output interface 905.

[0288] In a computer 900 with the above configuration, for example, a CPU 901 performs the above series of processes by loading a program stored in a storage unit 908 into RAM 903 via an input / output interface 905 and a bus 904 and executing the program.

[0289] The program executed by the CPU 901 is recorded on, for example, a removable medium 911, or provided via a wired or wireless transmission medium (such as a local area network, the Internet, or digital broadcasting) to be installed in the storage unit 908.

[0290] The program executed by the computer 900 may be a program that is processed sequentially in the order described in this specification, or it may be a program that is processed in parallel or at required time intervals (e.g., when called).

[0291] In this specification, a system refers to a collection of constituent elements (devices, modules (components), etc.), and all constituent elements may or may not be located in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, as well as a device in which multiple modules are housed in one housing, are all systems.

[0292] The embodiments of this technology are not limited to the above embodiments, and various changes can be made without departing from the spirit of this technology.

[0293] The beneficial effects described in this specification are merely exemplary and not limiting, and other beneficial effects may be achieved.

[0294] In addition, this disclosure can be configured as follows.

[0295] (1) An image processing apparatus, the image processing apparatus comprising an image processing unit configured to generate a presentation image to be presented into the interior of a moving second vehicle based on an image of the exterior of the vehicle obtained by capturing an image of the environment outside the exterior of a first vehicle in motion.

[0296] (2) The image processing apparatus according to (1), wherein the image processing unit generates a presentation image corresponding to the driving speed of the second vehicle based on the driving speed of the first vehicle obtained from the external image of the vehicle.

[0297] (3) According to the image processing device described in (2), when the driving speed of the first vehicle is the same as the driving speed of the second vehicle, the image processing unit outputs an external image of the vehicle as a presentation image.

[0298] (4) According to the image processing device of (3), when the driving speed of the first vehicle is inconsistent with the driving speed of the second vehicle, the image processing unit outputs a vehicle exterior image with a reproduction speed corresponding to the driving speed of the second vehicle as a presentation image.

[0299] (5) The image processing apparatus according to any one of (2) to (4), wherein the driving speed of the first vehicle is included in the metadata of the vehicle exterior image.

[0300] (6) The image processing apparatus according to any one of (2) to (4), wherein the image processing unit estimates the speed of the first vehicle based on the processing result of object recognition processing performed on an image of the exterior of the vehicle.

[0301] (7) The image processing apparatus according to (6), wherein when an object of a specified size is detected by object recognition processing, the image processing unit calculates the absolute speed of the first vehicle based on the amount of movement of the object of the specified size in the external image of the vehicle.

[0302] (8) The image processing apparatus according to (6) or (7), wherein when no object of a specified size is detected by object recognition processing, the image processing unit calculates a relative speed based on the moving speed of a predetermined object in the external image of the vehicle.

[0303] (9) The image processing apparatus according to any one of (1) to (8), wherein the vehicle exterior image is a fisheye camera image obtained by capturing an image from the first vehicle in the zenith direction, and

[0304] The image processing unit generates the rendered image by cropping a region based on the center of the fisheye camera image.

[0305] (10) The image processing apparatus according to (9), wherein the image processing unit performs cropping on the distortion-corrected fisheye camera image.

[0306] (11) The image processing apparatus according to (10), wherein the image processing unit generates a presentation image by correcting objects in the fisheye camera image based on the processing result of object recognition processing performed on at least a portion of the cropped fisheye camera image.

[0307] (12) The image processing apparatus according to any one of (1) to (8), wherein the vehicle exterior image is an image obtained by capturing an image in the direction of travel of the first vehicle from a forward camera, and

[0308] The image processing unit generates a presentation image by sequentially synthesizing region images, which are obtained by cropping regions corresponding to the presentation parts of the presentation image of the second vehicle from the front camera image at predetermined time intervals.

[0309] (13) The image processing apparatus according to (12), wherein the image processing unit cuts out a region based on a vanishing point detected from a front camera image.

[0310] (14) The image processing apparatus according to (13), wherein the image processing unit generates a presentation image by performing blurring processing on a composite image obtained by sequentially synthesizing the region images.

[0311] (15) The image processing apparatus according to any one of (1) to (14), wherein the display area for displaying the image includes the roof inside the second vehicle.

[0312] (16) The image processing apparatus according to any one of (1) to (15), wherein the first vehicle and the second vehicle are different vehicles.

[0313] (17) The image processing apparatus according to any one of (1) to (15), wherein the first vehicle and the second vehicle are the same vehicle.

[0314] (18) An image processing method, comprising:

[0315] An image processing device generates a rendering image to be presented into the interior of a moving second vehicle based on an image of the exterior of the vehicle obtained by capturing an image of the environment outside the first moving vehicle.

[0316] (19) A program that causes a computer to perform the following processes:

[0317] An image to be rendered into the interior of a moving second vehicle is generated based on an image of the vehicle's exterior obtained by capturing the environment outside the first moving vehicle.

[0318] (20) An image rendering system, comprising:

[0319] An image processing apparatus, comprising an image processing unit configured to generate a rendering image to be rendered into the interior of a moving second vehicle based on an image of the vehicle exterior obtained by capturing an image of the environment outside a moving first vehicle.

[0320] A presentation device, the presentation device including a presentation unit configured to present a presentation image onto the interior of a second vehicle.

[0321] Reference tag list

[0322] 1 Image processing equipment

[0323] 10 Input Units

[0324] 20 Graphics display processing units

[0325] 30 Output Units

[0326] Image processing units 40, 40A to 40E

[0327] 50 storage units

Claims

1. An image processing apparatus including an image processing unit configured to generate a presentation image to be presented to an interior of a second vehicle that is traveling, based on a vehicle exterior image obtained by capturing an environment outside a first vehicle that is traveling, wherein the image processing unit generates the presentation image corresponding to a traveling speed of the second vehicle, based on a traveling speed of the first vehicle acquired from the vehicle exterior image; when the traveling speed of the first vehicle coincides with the traveling speed of the second vehicle, the image processing unit outputs the vehicle exterior image as the presentation image; when the traveling speed of the first vehicle does not coincide with the traveling speed of the second vehicle, the image processing unit outputs the vehicle exterior image at a reproduction speed corresponding to the traveling speed of the second vehicle as the presentation image. the traveling speed of the first vehicle is included in metadata of the vehicle exterior image.

2. The image processing device according to claim 1, wherein the image processing unit estimates the traveling speed of the first vehicle based on a processing result of object recognition processing performed on the vehicle exterior image.

3. The image processing device according to claim 1, wherein when an object of a specified size is detected by the object recognition processing, the image processing unit calculates an absolute speed of the first vehicle based on an amount of movement of the object of the specified size in the vehicle exterior image.

4. The image processing device according to claim 3, wherein when an object of a specified size is not detected by the object recognition processing, the image processing unit calculates a relative speed based on a speed of movement of a predetermined object in the vehicle exterior image.

5. The image processing device according to claim 3, wherein the vehicle exterior image is a fisheye camera image obtained by capturing an image in a zenith direction from the first vehicle, and 6. The image processing device according to claim 1, wherein the image processing unit generates the presentation image by clipping a region with a center of the fisheye camera image as a reference. the image processing unit clips the fisheye camera image that has been subjected to distortion correction.

7. The image processing device according to claim 6, wherein the image processing unit generates the presentation image by correcting an object in the fisheye camera image according to a processing result of object recognition processing performed on at least a part of the clipped fisheye camera image.

8. The image processing device according to claim 7, wherein the vehicle exterior image is a front camera image obtained by capturing an image in a traveling direction of the first vehicle, and 9. The image processing device according to claim 1, wherein the image processing unit generates the presentation image by sequentially synthesizing region images obtained by clipping regions corresponding to a presentation site of the presentation image of the second vehicle at a predetermined time interval in the front camera image. the image processing unit clips a region with a vanishing point detected from the front camera image as a reference.

10. The image processing device according to claim 9, wherein the image processing unit generates the presentation image by performing blur processing on a synthesized image obtained by sequentially synthesizing the region images.

11. The image processing device according to claim 10, wherein the presentation site of the presentation image includes a ceiling inside the second vehicle.

12. The image processing device according to claim 1, wherein the first vehicle and the second vehicle are different vehicles.

13. The image processing device according to claim 1, wherein the first vehicle and the second vehicle are the same vehicle.

14. The image processing device according to claim 1, wherein 15.An image processing method comprising: generating, by an image processing apparatus, a presentation image to be presented to an interior of a second vehicle that is traveling, based on a vehicle exterior image obtained by capturing an environment outside a first vehicle that is traveling, wherein the image processing apparatus generates the presentation image corresponding to a traveling speed of the second vehicle, based on a traveling speed of the first vehicle acquired from the vehicle exterior image; ​ the image processing device outputs the vehicle exterior image as the presentation image when the traveling speed of the first vehicle coincides with the traveling speed of the second vehicle; and the image processing device outputs the vehicle exterior image at the reproduction speed corresponding to the traveling speed of the second vehicle as the presentation image when the traveling speed of the first vehicle does not coincide with the traveling speed of the second vehicle.

16. A program product for causing a computer to execute the following processing: generating, by an image processing device, a presentation image to be presented to an interior of a second vehicle that is traveling, based on a vehicle exterior image obtained by capturing an environment outside a first vehicle that is traveling, wherein the image processing device generates the presentation image corresponding to a traveling speed of the second vehicle based on a traveling speed of the first vehicle acquired from the vehicle exterior image; the image processing device outputs the vehicle exterior image as the presentation image when the traveling speed of the first vehicle coincides with the traveling speed of the second vehicle; and the image processing device outputs the vehicle exterior image at the reproduction speed corresponding to the traveling speed of the second vehicle as the presentation image when the traveling speed of the first vehicle does not coincide with the traveling speed of the second vehicle.

17. An image presentation system comprising: an image processing device including an image processing unit configured to generate a presentation image to be presented to an interior of a second vehicle that is traveling, based on a vehicle exterior image obtained by capturing an environment outside a first vehicle that is traveling, and a presentation device including a presentation unit configured to present the presentation image to the interior of the second vehicle, wherein the image processing unit generates the presentation image corresponding to a traveling speed of the second vehicle based on a traveling speed of the first vehicle acquired from the vehicle exterior image; the image processing unit outputs the vehicle exterior image as the presentation image when the traveling speed of the first vehicle coincides with the traveling speed of the second vehicle; and the image processing unit outputs the vehicle exterior image at the reproduction speed corresponding to the traveling speed of the second vehicle as the presentation image when the traveling speed of the first vehicle does not coincide with the traveling speed of the second vehicle.

Images