Image processing device, and image processing program

Abstract

This image processing device includes: an acquisition unit that acquires first image data indicating color information about a subject in a spatial direction intersecting a depth direction, the subject existing outside a mobile body, and second image data indicating a distance in the depth direction to the subject; and a generation unit that, on the basis of the first image data and the second image data acquired by the acquisition unit, generates third image data in which color information about a light beam from a display unit having a pixel group for displaying a stereoscopic image of the subject via a lens array to a visual recognition position for visually recognizing the display unit is reproduced as color information about reflected light from the subject, and outputs the third image data to the display unit.

Description

Image processing device and image processing program Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2023-208092, filed on December 8, 2023, the contents of which are incorporated herein by reference.

[0002] The present invention relates to an image processing device and an image processing program.

[0003] The vehicular display system of Patent Document 1 below includes a display means, a detection means for detecting a target person present around the vehicle, and a control means for controlling the display means. The display means includes a light field display capable of displaying different images in multiple directions outside the vehicle. The detection means includes a camera and a target person detection unit for detecting the target person from the captured image. The control means includes a setting unit and a display control unit that control the display of an image corresponding to the detected target person in the direction of the target person.

[0004] However, the above-mentioned Patent Document 1 does not disclose how to display the image displayed on the light field display to the driver.

[0005] Patent Publication No. 2021-20523

[0006] The image processing device of the first disclosed technology has an acquisition unit that acquires first image data that indicates color information in a spatial direction that intersects with the depth direction of a subject located outside a moving body, and second image data that indicates the distance to the subject in the depth direction, and a generation unit that generates third image data that reproduces color information of light rays from a display unit having a group of pixels that displays a three-dimensional image of the subject via a lens array to a viewing position where the display unit is viewed as color information of light reflected from the subject, based on the first image data and the second image data acquired by the acquisition unit, and outputs the third image data to the display unit.

[0007] The image processing device of the second disclosed technology has an acquisition unit that acquires first image data indicating color information of a subject and second image data indicating the depth distance to the subject, and a generation unit that generates third image data based on information on the angle of light emitted from pixels of a pixel group of a display unit having a pixel group that displays a three-dimensional image of the subject via a lens array, and the first image data and the second image data acquired by the acquisition unit, and outputs the third image data to the display unit.

[0008] The image processing device of the third disclosed technology has an acquisition unit that acquires first image data indicating color information of a subject and second image data indicating the depth distance to the subject, and a generation unit that identifies color information at pixel positions of a pixel group of a display unit having a pixel group that displays a three-dimensional image of the subject via a lens array based on the first image data and the second image data acquired by the acquisition unit, generates third image data, and outputs the third image data to the display unit.

[0009] The image processing program of the fourth disclosed technology causes a processor to execute an acquisition process to acquire first image data indicating color information in a spatial direction that intersects with the depth direction of a subject located outside a moving body, and second image data indicating the distance to the subject in the depth direction, and a generation process to generate third image data that reproduces color information of light rays from a display unit having a group of pixels that displays a three-dimensional image of the subject via a lens array to a viewing position where the display unit is viewed, as color information of light reflected from the subject, based on the first image data and the second image data acquired by the acquisition process, and output the third image data to the display unit.

[0010] FIG. 1 is an explanatory diagram showing an example of use of a digital mirror. FIG. 2 is a block diagram showing an example of the hardware configuration of an image processing device. FIG. 3 is a perspective view of an LFD. FIG. 4 is a plan view of an LF lens array and a flat display unit. FIG. 5 is a side cross-sectional view of an LFD. FIG. 6 is a block diagram showing an example of the functional configuration of an image processing device. FIG. 7 is an explanatory diagram showing an example of distance information of distance image data. FIG. 8 is an explanatory diagram (part 1) showing an example of a lens position table. FIG. 9 is a side view of an LF lens and a pixel region. FIG. 10 is an explanatory diagram (part 2) showing an example of a lens position table. FIG. 11 is an explanatory diagram showing an example of an output angle table. FIG. 12 is an explanatory diagram showing the relationship between an LFD and a subject. FIG. 13 is an explanatory diagram showing an example of determining the closest point of approach. FIG. 14 is a flowchart showing an example of an image processing procedure executed by an image processing device. FIG. 15 is an explanatory diagram showing an example of a display of an LF image. FIG. 16 is an explanatory diagram showing example 1 of a highlighted display of a stereoscopic image. FIG. 17 is an explanatory diagram showing example 2 of a highlighted display of a stereoscopic image. FIG. 18 is an explanatory diagram showing an example of determining the proximity of a subject.

[0011] <Example of Use of Electronic Mirror> Fig. 1 is an explanatory diagram showing an example of use of an electronic mirror. Fig. 1 shows an example of use of an electronic mirror mounted on a truck. Note that electronic mirrors are not limited to trucks and can also be applied to other moving objects including passenger cars, buses, motorcycles, and the like.

[0012] The truck 100 has a tractor 101 and a trailer 102. In Fig. 1, a plan cross section of the tractor 101 is shown. The tractor 101 has a driver's seat where a driver D sits. The trailer 102 is coupled to the tractor 101.

[0013] The tractor 101 has a right camera 111A and a left camera 111B, and the trailer 102 has a rear camera 111C at its rear end. When there is no need to distinguish between the right camera 111A, the left camera 111B, and the rear camera 111C, they will be simply referred to as cameras 111. The cameras 111 generate two-dimensional image data and range image data by capturing images.

[0014] The right camera 111A is provided on the right side of the tractor 101 facing the direction of the trailer 102. Therefore, the right camera 111A captures an image of a blind spot on the right side of the truck 100 that is not visible to the driver D, and generates two-dimensional image data and distance image data.

[0015] The left camera 111B is provided on the left side of the tractor 101 facing the direction of the trailer 102. Therefore, the left camera 111B captures an image of the blind spot on the left side of the truck 100 that is not visible to the driver D, and generates two-dimensional image data and distance image data.

[0016] The rear camera 111C is provided at the rear end of the trailer 102 facing in the opposite direction to the tractor 101. Therefore, the rear camera 111C captures an image of a blind spot on the rear side of the truck 100 that cannot be seen by the driver D, and generates two-dimensional image data and distance image data.

[0017] The method for generating the distance image data may be a known method. For example, a stereo camera or a sensor using infrared rays, a laser, or ultrasonic waves may be used as the camera 111.

[0018] Further, a right light field display (hereinafter, LFD) 112A, a left LFD 112B, and a rear LFD 112C are provided inside the tractor 101. When there is no need to distinguish between the right LFD 112A, the left LFD 112B, and the rear LFD 112C, they will be simply referred to as LFDs 112.

[0019] The LFD 112 is a display unit that reproduces the light reflected from an object according to the position from which a person views the object, thereby displaying the object in a three-dimensional manner. In other words, by reproducing the mechanism in which the light entering a person's eyes changes depending on the direction from which the person looks, an image as seen from the direction from which the person looks is displayed. The LFD 112 reproduces the light reflected from the object, allowing the object to be viewed in a three-dimensional manner. The object here is the subject captured by the camera 111. Therefore, the LFD 112 displays the subject captured by the camera 111 in a three-dimensional manner.

[0020] The right LFD 112A is disposed in front of the driver D on the right side inside the tractor 101. The right LFD 112A is connected to the right camera 111A. The right LFD 112A displays an image captured by the right camera 111A. The combination of the right camera 111A and the right LFD 112A forms a right electronic mirror.

[0021] The left LFD 112B is disposed on the left front side of the driver D inside the tractor 101. The left LFD 112B is connected to the left camera 111B. The left LFD 112B displays an image captured by the left camera 111B. The combination of the left camera 111B and the left LFD 112B forms a left electronic mirror.

[0022] The rear side LFD 112C is disposed in the center front inside the tractor 101. The rear side LFD 112C is connected to the rear camera 111C. The rear side LFD 112C displays an image captured by the rear camera 111C. The combination of the rear camera 111C and the rear side LFD 112C forms a rear side electronic mirror.

[0023] Here, for convenience, a coordinate system 120 is shown. The X-axis of the coordinate system 120 indicates the pixel arrangement direction along the vehicle width direction of the truck 100 on the display plane of the LFD 112. The Y-axis indicates a direction perpendicular to the display plane of the LFD 112. The Y-axis is also the optical axis direction of the camera 111. The Z-axis indicates the pixel arrangement direction perpendicular to the X-axis and Y-axis on the display plane of the LFD 112. The display plane of the LFD 112 is sometimes referred to as the XZ plane.

[0024] 2 is a block diagram showing an example of the hardware configuration of an image processing device. The image processing device 200 has an image processing unit 201. The image processing unit 201 is connected to an input device 202, a camera 111, and an LFD 112. The image processing device 200 is configured to include at least the image processing unit 201 among the image processing unit 201, the input device 202, an ECU 203, the camera 111, and the LFD 112.

[0025] The image processing unit 201 has a processor 211, a memory 212, and an image processing circuit 213. The processor 211 controls the image processing unit 201. The memory 212 stores an image processing program as well as other necessary programs and data. The memory 212 also serves as a working area for the processor 211. The image processing circuit 213 is a circuit that executes image processing.

[0026] The image processing circuit 213 is an integrated circuit that performs image processing such as color interpolation, white balance adjustment, edge enhancement, gamma correction, and gradation conversion on image data output from the camera 111. Specifically, the image processing circuit 213 may be realized by a programmable logic device (PLD) such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0027] The input device 202 inputs various data to the image processing unit 201 directly or via an Electronic Control Unit (hereinafter referred to as ECU). The input device 202 may be, for example, an input key or a touch panel mounted on a display different from the LFD 112. The input device 202 may also be an operation button for adjusting the angle of the light beam from the LFD 112. The input device 202 may also be an operation button for operating the wipers or windows.

[0028] <LFD 112> Fig. 3 is a perspective view of the LFD 112. Fig. 4 is a plan view of the LF lens array and the flat display unit. Fig. 5 is a side cross-sectional view of the LFD 112.

[0029] The LFD 112 has an LF lens array 301 and a flat display unit 302. The LF lens array 301 is a lenticular lens in which a plurality of LF lenses 310, each having a substantially semi-cylindrical (half-cylindrical) shape, are arranged in the X-axis direction. The longitudinal direction of the LF lenses 310 (the axial direction of the cylinder) is tilted at a predetermined angle θ from the Z-axis direction. The diagonal lines of the LF lens array 301 indicate valleys V that are boundaries between the LF lenses 310. The tilt angle θ is stored in the memory 212 as parameter information.

[0030] 4, the planar display unit 302 has a set of pixels p arranged in a matrix on the XZ plane. The planar display unit 302 may be configured to emit light rays from the pixels p by self-emission, or may be configured to transmit light from a backlight through the pixels p and emit the light. The pixels p emit light rays via the LF lens array 301. The lens pitch LP is the distance between the optical axes of adjacent LF lenses 310, and is the width of the LF lenses 310 in the X-axis direction. The lens pitch LP is stored in the memory 212 as parameter information.

[0031] The side cross section of the LFD 112 in Fig. 5 shows the FF cross section in Fig. 4. The planar display unit 302 has a pixel region 502 corresponding to each LF lens 310. The pixel region 502 is a collection of pixels p arranged in a matrix on the XZ plane. In this example, the pixel region 502 has 13 pixels p arranged in the X-axis direction. The arrangement length of the 13 pixels p in the X-axis direction corresponds to the lens pitch LP.

[0032] <Example of functional configuration of image processing device 200> Fig. 6 is a block diagram showing an example of the functional configuration of the image processing device 200. The image processing unit 201 has an acquisition unit 611 and a generation unit 612. Specifically, the acquisition unit 611 and the generation unit 612 are realized, for example, by causing the processor 211 to execute a program stored in the memory 212 shown in Fig. 2 or by the image processing circuit 213 shown in Fig. 2.

[0033] The acquisition unit 611 acquires two-dimensional image data 601 and distance image data 602 output from the camera 111, which is an imaging unit. The generation unit 612 generates LF image data 603 based on the two-dimensional image data 601 and distance image data 602 acquired by the acquisition unit 611.

[0034] The two-dimensional image data 601 is image data that indicates color information in a spatial direction (for example, the XZ plane) that intersects with the depth direction (for example, the Y-axis direction) of the subject. Each pixel of the two-dimensional image data 601 indicates color information (which may be color or grayscale) of the subject at that position. The distance image data 602 is image data that indicates the distance from the camera 111. The LF image data 603 is image data that, when input to the LFD 112, causes the subject to appear as a three-dimensional image.

[0035] Here, a specific example of the two-dimensional image data 601, distance image data 602, and LF image data 603 will be described when the camera 111 captures images of the ground M, object J, object K, object L, and infinitely distant background N. Note that the objects J, K, and L are closest to the camera 111 in this order.

[0036] The two-dimensional image data 601 includes a ground M1 representing a two-dimensional image of the ground M, an object J1 representing a two-dimensional image of the object J, an object K1 representing a two-dimensional image of the object K, an object L1 representing a two-dimensional image of the object L, and a background N1 representing a two-dimensional image of the background N. The two-dimensional image data 601 is data that defines color information of the ground M, the object J, the object K, the object L, and the background N by pixel p.

[0037] The distance image data 602 is image data that indicates the distance to the subject in the depth direction. Specifically, for example, the distance image data 602 includes ground M2 that indicates a distance image of ground M, object J2 that indicates a distance image of object J, object K2 that indicates a distance image of object K, object L2 that indicates a distance image of object L, and background N2 that indicates a distance image of background N. The distance image data 602 is data that specifies, in terms of pixels p, distance information from the camera 111 to the ground M, object J, object K, object L, and background N. The distance image data 602 is, for example, grayscale image data.

[0038] The LF image data 603 is image data that reproduces color information of reflected light from a subject corresponding to the position (driver's seat) at which the driver D views the LFD 112. Specifically, for example, the LF image data 603 includes a ground M3 that shows an LF image of the ground M, an object J3 that shows an LF image of the object J, an object K3 that shows an LF image of the object K, an object L3 that shows an LF image of the object L, and a background N3 that shows an LF image of the background N.

[0039] The LF image data 603 is data in which the angle of a light ray emitted from pixel p via the LF lens array 301 is specified based on the positional relationship between pixel p and the LF lens 310, and the position of pixel p is determined to display color information of the ground M, object J, object K, object L, and background N in the two-dimensional image data 601 based on the angle of the light ray and distance information of the ground M, object J, object K, object L, and background N in the distance image data 602. In the LF image data 603, the outline of each subject appears as stripes corresponding to the tilt of the LF lens 310.

[0040] 7 is an explanatory diagram showing an example of distance information of the distance image data 602. The distance information of the distance image data 602 is called a depth value. The smaller the depth value (darker the color), the closer the object is to the camera 111 in the Y-axis direction, and the larger the depth value (lighter the color), the farther the object is from the camera 111 in the Y-axis direction. In this example, the depth value reaches its maximum value (white) at 10 m from the camera 111, and the depth value remains at its maximum value (white) from 10 m onwards.

[0041] <Positional Relationship Between Pixel and LF Lens> Next, the positional relationship between the pixel p and the LF lens 310 will be specifically described.

[0042] FIG. 8 is an explanatory diagram (part 1) showing an example of a lens position table. The lens position table 800 is a table in which the lens position of each pixel p is set, and is stored in the memory 212. FIG. 8 shows a state in which the lens position has not yet been set for any pixel p. x indicates the position of pixel p on the planar display unit 302 in the X-axis direction. z indicates the pixel position of pixel p on the planar display unit 302 in the Z-axis direction. Pixel p may be expressed as pixel p(x, z) using pixel positions x and z on the planar display unit 302.

[0043] 9 is a side view of the LF lens 310 and the pixel region 502. In Fig. 9, as an example, the length of the pixel p(x, z) in the X-axis direction is "3" and the length of the pixel p(x, z) in the Z-axis direction is "3". In this example, since 13 pixels p(x, z) are arranged in the X-axis direction in the pixel region 502, the lens pitch LP of the LF lens 310 is, for example, "39".

[0044] Fig. 10 is an explanatory diagram (part 2) showing an example of the lens position table 800. Fig. 10 shows a state in which a lens position is set for each pixel p(x, z). The coordinate value of the lens position of the upper left vertex pixel p(0,0) is set to "0.0", and using the lens position of this pixel p(0,0) as a reference, the coordinate values ​​of the lens positions of other pixels p(x, z) are calculated using the following formulas (1) and (2).

[0045] r(x,z)=fmod(n,LP)...(1) n=3x+3z×tan(πθ / 180)+offset...(2)

[0046] In the above formula (1), r is the lens position of pixel p(x, z). fmod() is a function that calculates the floating-point remainder r(x, z) when the dividend n is divided by the divisor, the lens pitch LP. The above formula (2) returns the calculation result on the right side as n, which is substituted into the above formula (1).

[0047] In the above formula (2), the coefficient "3" of pixel position x is an example of the length of the LF lens 310 in the X-axis direction of the pixel p(x,z) portion, and the coefficient "3" of pixel position z is an example of the length of the LF lens 310 in the Z-axis direction of the pixel p(x,z) portion. In this example, the coefficients of pixel positions x and z are set to "3", but can be set arbitrarily depending on the lengths of the LF lens 310 in the X-axis direction and Z-axis direction of the pixel p(x,z) portion.

[0048] In the above formula (2), θ is the tilt angle of the lens 331. The positional relationship between the LF lens 310 of the LF lens array 301 and the pixel p(x, z) of the flat display unit 302 contains an error depending on the accuracy of bonding the LF lens array 301 and the flat display unit 302. For this reason, the tilt angle θ of the lens 331 differs for each LFD 112. Therefore, θ may be set to different values ​​for multiple LFDs 112.

[0049] In the above formula (2), offset is an adjustment value that takes into account errors due to the accuracy of bonding the LF lens array 301 and the flat display unit 302, and is information that indicates the characteristics of the LF lens array 301. It is stored in the memory 212 as parameter information for each pixel p(x, z).

[0050] The positional relationship between the LF lenses 310 of the LF lens array 301 and the pixels p(x, z) of the flat display unit 302 includes an error due to the accuracy of bonding the LF lens array 301 and the flat display unit 302. For this reason, the position of the valley V of the lens pitch in the X-axis direction of the LF lens 310 differs for each LFD 112. Therefore, the offset may be set to a different value between multiple LFDs 112.

[0051] Furthermore, even for the same pixel p(x, z), the angle of the light ray emitted from pixel p(x, z) differs depending on the installation position of the LFD 112, specifically, between LFDs 112A to 112C. Therefore, the offset is set taking into consideration the position of the driver's seat from which the driver D views the LFD 112 and the installation position of the LFD 112.

[0052] <Emission angle of light ray from pixel p(x, z)> Fig. 11 is an explanatory diagram showing an example of an emission angle table. The emission angle table 1100 is a table in which the emission angle of a light ray emitted from each pixel p(x, z) is set. The data structure of the emission angle table 1100 is the same as that of the lens position table 800, and the set value is the emission angle around the Z axis from pixel p(x, z) rather than the lens position r. The emission angle φ(x, z) from pixel p(x, z) is calculated using the following formula (3):

[0053] φ(x,z)={vw / (LP-1)}×r(x,z)-(vw / 2)...(3)

[0054] In the above formula (3), vw is the viewing angle of the LF lens 310 determined by the characteristics of the LF lens 310, and is set to, for example, 25 degrees. In this manner, the emission angle φ(x, z) is calculated using the above formula (3). With reference to FIG. 9 , it is assumed that the emission angle φ(x, z) of the light ray bc emitted from the central pixel pc in the pixel region 502 is calculated to be 0 degrees. In other words, the light ray bc emitted from the pixel pc is emitted perpendicular to the XZ plane via the LF lens 310.

[0055] Furthermore, the emission angle φ(x, z) of the light ray br emitted from the rightmost pixel pr is an angle tilted by 12.5 degrees from the emission angle φ(x, z) of the light ray bc with respect to the XZ plane via the LF lens 310. The emission angle φ(x, z) of the light ray bl emitted from the leftmost pixel prl is an angle tilted by −12.5 degrees from the emission angle φ(x, z) of the light ray bc with respect to the XZ plane via the LF lens 310.

[0056] <Relationship between LFD 112 and subject> Figure 12 is an explanatory diagram showing the relationship between the LFD 112 and the subject. In Figure 12, for convenience of explanation, the shapes of the objects J, K, and L are drawn partly differently from those in Figures 6 and 7. The vertical axis represents the depth value of the distance image data 602 from the camera 111, which is the value in the Y-axis direction.

[0057] The thick dotted line indicates a plane representing the two-dimensional image data 601 and the depth image data 602. That is, the two-dimensional image data 601 has color information on the thick dotted line, and the depth image data 602 has depth values, which are distance information, on the thick dotted line. Hereinafter, the plane indicated by this thick dotted line will be referred to as the image plane 1200. For ease of explanation, the LFD 112 is placed at the position of the camera 111 to represent the emission of light rays from pixel p(x, z).

[0058] A light ray ba1 from a pixel pa1 in the pixel region 502a is emitted through the LF lens 310a at an emission angle φ and reaches a point hl1 on the object L. A light ray ba2 from a pixel pa2 in the pixel region 502a is emitted through the LF lens 310a at an emission angle φ and reaches a point hl2 on the object L. However, the point hl2 is not visible to the camera 111, i.e., it is not captured.

[0059] A light ray bb1 from a pixel pb1 in the pixel region 502b is emitted through the LF lens 310b at an emission angle φ and reaches a point hj1 on the object J. A light ray bb2 from a pixel pb2 in the pixel region 502b is emitted through the LF lens 310b at an emission angle φ and reaches a point hj2 on the object J.

[0060] A light ray bc1 from pixel pc1 in pixel region 502c is emitted through LF lens 310c at an emission angle φ and reaches point hk1 on object K. A light ray bc2 from pixel pc2 in pixel region 502c is emitted through LF lens 310c at an emission angle φ, but does not hit object K and travels to infinity. For convenience, it is assumed that the light ray bc2 reaches background N at the maximum depth value.

[0061] When there is no need to distinguish between the rays ba1, ba2, bb1, bb2, bc1, and bc2, they are written as rays b. When there is no need to distinguish between the points hl1, hl2, hj1, hj2, hk1, and hk2, they are written as points h.

[0062] The coordinate origin O of the coordinate system 120 is set to the position of the leftmost pixel p0. The emission angle φ of the ray b from each pixel p is set as described in Fig. 11, and the ray b is a line segment connecting the position (x, y) (where y = 0) of the pixel p from which the ray b originates) and the coordinate value (xh, yh) of the point h where the ray b arrives. Therefore, the ray b is defined by a function of the linear ray equation given by the following equation (4).

[0063] y=αx+β...(4)

[0064] y is the Y-axis coordinate value on ray b, and is the depth value of the distance image data 602. x is the X-axis coordinate value of pixel p that emits ray b. α is the slope of ray b. β is the intercept of ray b, for example, the intersection of ray b and the Y-axis. The above formula (4) does not depend on the Z-axis coordinate value of pixel p.

[0065] The color information of the two-dimensional image data 601 at the point h where the ray b reaches becomes the color information of the pixel p from which the ray b emerges. For example, the color information of pixel pa1 becomes the color information of point hl1.

[0066] On the other hand, the arrival point hl2 of the light ray ba2 is not represented in the two-dimensional image data 601. In this case, the image processing unit 201 identifies the point at which the light ray ba2 approaches the image plane 1200 closest.

[0067] FIG. 13 is an explanatory diagram showing an example of determining the point of closest approach. In FIG. 13, the light ray ba2 is used as an example. The image processing unit 201 scans the X-axis value x of the arrival point hl2 in units of pixel p, and generates perpendicular lines q1, q2, q3, q4, q5, q6, ... sequentially from the light ray ba2 at the X-axis value x after scanning (when there is no need to distinguish between the perpendicular lines q1, q2, q3, q4, q5, q6, ..., they will be referred to as perpendicular lines q). The image processing unit 201 determines the point on the image plane 1200 that is the shortest distance from the light ray ba2 among the perpendicular lines q that arrive at the image plane 1200 as the point of closest approach. In this example, the intersection HL2 between the perpendicular line q5 and the object L is the point of closest approach.

[0068] In addition, the image processing unit 201 may calculate the difference between the coordinate value y, which is the depth value of the light ba2 at the scanned coordinate value x, and the depth value on the distance image data 602 at the scanned coordinate value x, and determine the coordinate value (x, y) where this difference is smallest as the closest point.

[0069] <Image Processing Procedure> FIG. 14 is a flowchart showing an example of an image processing procedure executed by the image processing device 200.

[0070] (Step S1401 ) The camera 111 captures an image of a subject, generates two-dimensional image data 601 and depth image data 602 , and outputs them to the image processing unit 201 .

[0071] (Step S1402 ) The image processing unit 201 acquires the two-dimensional image data 601 and the range image data 602 from the camera 111 .

[0072] (Step S1403 ) The image processing unit 201 executes LF image data generation processing and outputs the LF image data 603 to the LFD 112 .

[0073] For example, if the lens position table 800 and the emission angle table 1100 are not stored in the memory 212, the image processing unit 201 substitutes the parameters of the above formulas (1) and (2) into the above formulas (1) and (2) to calculate the lens position r(x, z) for each pixel p(x, z), calculates the emission angle φ(x, z) of the light ray from the pixel p(x, z) for each pixel p(x, z) using the above formula (3), and generates a linear equation (formula (4)) of the light ray emitted at the emission angle φ(x, z) for each pixel p(x, z) using the above formula (4).

[0074] As a result, it is not necessary to store the lens position table 800 and the output angle table 1100 in the memory 212, and therefore it is possible to reduce the size of the memory 212 or increase the free space of the memory 212 accordingly.

[0075] Furthermore, if the lens position table 800 is stored in the memory 212 but the emission angle table 1100 is not stored in the memory 212, the image processing unit 201 reads out the lens position r(x, z) from the lens position table 800 and substitutes it into the above formula (3), calculates the emission angle φ(x, z) of the light ray from pixel p(x, z) for each pixel p(x, z) using the above formula (3), and generates a linear equation (formula (4)) of the light ray emitted at the emission angle φ(x, z) for each pixel p(x, z) using the above formula (4).

[0076] As a result, it is not necessary to store the output angle table 1100 in the memory 212, which can reduce the size of the memory 212 or increase the free space in the memory 212. Furthermore, since the lens position table 800 is stored in the memory 212, it is not necessary to calculate the lens position r(x, z) using the above formulas (1) and (2), which improves the readout speed of the lens position r(x, z).

[0077] Furthermore, if the emission angle table 1100 is stored in the memory 212 but the lens position table 800 is not stored in the memory 212, the image processing unit 201 reads out the emission angle φ(x, z) of the ray from pixel p(x, z) for each pixel p(x, z) from the emission angle table 1100, and generates, for each pixel p(x, z), a linear equation (equation (4)) of the ray emitted at the emission angle φ(x, z) using the above equation (4).

[0078] As a result, it is not necessary to store lens position table 800 in memory 212, which makes it possible to reduce the size of memory 212 or increase the available space in memory 212. Furthermore, since emission angle table 1100 is stored in memory 212, it is not necessary to calculate lens position r(x, z) using equations (1) and (2) above, and to calculate emission angle φ(x, z) from pixel p(x, z) using equation (3) above, which improves the readout speed of emission angle φ(x, z) from pixel p(x, z).

[0079] Furthermore, if the lens position table 800 and the output angle table 1100 are not stored in the memory 212, but the linear equation of the light ray for each pixel p(x, z) of the above equation (4) is stored in the memory 212, the image processing unit 201 reads out the linear equation of the light ray (equation (4)) for each pixel p(x, z).

[0080] As a result, it is not necessary to store lens position table 800 and emission angle table 1100 in memory 212, which makes it possible to reduce the size of memory 212 or increase the available space in memory 212. Furthermore, since the linear equation (equation (4)) of the light ray for each pixel p(x, z) is stored in memory 212, it is not necessary to calculate lens position r(x, z) using equations (1) and (2) above, calculate emission angle φ(x, z) from pixel p(x, z) using equation (3) above, and generate the linear equation (equation (4)) of the light ray for each pixel p(x, z), which makes it possible to speed up the generation of LF image data 603.

[0081] Then, the image processing unit 201 uses the linear equation of the ray emitted at the emission angle φ(x, z) given by the above equation (4) to assign color information of the two-dimensional image data 601 at the arrival point and the closest point on the image plane 1200 to the pixel p(x, z) from which the ray originates, as described with reference to Figures 12 and 13. In this way, LF image data 603 is generated.

[0082] (Step S1404) The LFD 112 displays an LF image based on the LF image data 603 from the image processing unit 201, so that the subject appears as a stereoscopic image.

[0083] 15 is an explanatory diagram showing an example of a display of an LF image. The LFD 112 displays an LF image 1500 based on the LF image data 603. That is, when color information of the LF image data 603 is set in each pixel p(x, z) of the planar display unit 302 of the LFD 112, objects J, K, and L are displayed three-dimensionally, and the driver D can visually recognize them as three-dimensional images J3, K3, and L3.

[0084] 16 is an explanatory diagram showing an example 1 of a three-dimensional image highlighting display. (A) is an example of a highlighting display in which the range of highlighting is 0≦y<d1, (B) is an example of a highlighting display in which the range of highlighting is d1≦y<d2, and (C) is an example of a highlighting display in which the range of highlighting is d2≦y.

[0085] Highlighting is a process in which, for example, the image processing unit 201 sets the brightness and saturation of pixels p(x, z) whose depth value y falls within the range of highlighting to a predetermined amount higher, or sets the brightness and saturation of pixels p(x, z) whose depth value y falls outside the range of highlighting to a predetermined amount lower.

[0086] Fig. 17 is an explanatory diagram showing an example 2 of highlighted display of a stereoscopic image. In Fig. 17, the image processing unit 201 generates LF image data 603 for a portion within the range of LF display and displays an LF image 1701, and outside the range of LF display, the image processing unit 201 does not generate LF image data 603 for that portion but displays a two-dimensional image 1702 using two-dimensional image data 601.

[0087] Specifically, for example, outside the range of LF display, the image processing unit 201 sets the depth value of the pixel in the distance image data 602 to 0, and assigns the color information of that pixel in the two-dimensional image data 601 to a pixel p(x, z) at a corresponding two-dimensional position on the planar display unit 302. For example, when the object L in Fig. 12 is outside the range of LF display, the image processing unit 201 assigns the color information of point hl1 in the two-dimensional image data 601 to pixel pa11 at a corresponding two-dimensional position on the planar display unit 302 that corresponds to the pixel position of point hl1.

[0088] (A) is an example of an LF display in which the range of depth value y is 0≦y<d1, (B) is an example of an LF display in which the range of highlighting is d1≦y<d2, and (C) is an example of an LF display in which the range of depth value y is d2≦y.

[0089] The image processing unit 201 may highlight the LF display ranges (A) to (C) as shown in FIG.

[0090] <Example of determining approach of subject> Figure 18 is an explanatory diagram showing an example of determining approach of a subject. The image processing unit 201 determines whether the subject has approached or moved away from the camera 111 by referring to the depth value y between the distance image data 602 of successive frames. The image processing unit 201 may highlight subjects that have approached by a predetermined depth value y or more, as shown in Figure 16. The image processing unit 201 may also highlight subjects other than subjects that are farther away by the predetermined depth value y or more, as shown in Figure 16.

[0091] 18, it is assumed that it is determined that object J is approaching camera 111 by a predetermined depth value y or more. In this case, the image processing unit 201 highlights the LF image, for example, as shown in (A) of FIG. 16, within the range of depth value y from camera 111 to object J after approach.

[0092] Also, suppose that it is determined that the object K is at a distance of a predetermined depth value y or more from the camera 111. In this case, the image processing unit 201 highlights the LF image, for example, in the range of the depth value y after the object K after the distance, as shown in (C) of FIG.

[0093] <Example of Split Display> Additionally, the image processing unit 201 may display the LF image data 603 of one of the right LFD 112A, the left LFD 112B, and the rear LFD 112C on that LFD 112, and may also display LF image data 603 based on the two-dimensional image data 601 and the distance image data 602 from a camera 111 other than the camera 111 corresponding to that LFD 112. This allows the driver D to simultaneously view the display contents of two or more LFDs 112 by viewing one LFD 112. This reduces the number of times the driver D changes his or her line of sight while driving, thereby supporting safe driving.

[0094] For example, the image processing unit 201 may divide the rear LFD 112C into left and right halves, and control the display so that the LF image displayed on the right LFD 112A is displayed in the right split display area, and the LF image displayed on the left LFD 112B is displayed in the left split display area.

[0095] In addition, the image processing unit 201 may divide the right LFD 112A into upper and lower halves, and control the display so that the LF image displayed on the right LFD 112A is displayed in the upper split display area, and the LF image displayed on the left LFD 112B is displayed in the lower split display area.

[0096] In addition, the image processing unit 201 may divide the right LFD 112A into upper and lower halves, and control the display so that the LF image displayed on the right LFD 112A is displayed in the upper split display area, and the LF image displayed on the rear LFD 112C is displayed in the lower split display area.

[0097] Similarly, the image processing unit 201 may divide the left LFD 112B into upper and lower halves, and control the display so that the LF image displayed on the left LFD 112B is displayed in the upper split display area, and the LF image displayed on the right LFD 112A is displayed in the lower split display area.

[0098] Similarly, the image processing unit 201 may divide the left LFD 112B into upper and lower halves, and control the display so that the LF image displayed on the left LFD 112B is displayed in the upper split display area, and the LF image displayed on the rear LFD 112C is displayed in the lower split display area.

[0099] As described above, the image processing device 200 of this embodiment generates LF image data 603 based on the two-dimensional image data 601 and the distance image data 602, and therefore can display a three-dimensional image of a subject outside the moving body inside the moving body. Furthermore, because the image processing device 200 generates the LF image data 603 based on the two-dimensional image data 601 and the distance image data 602, it can generate the LF image data 603 faster than a stereo camera method that generates distance image data from multiple pieces of two-dimensional image data.

[0100] In addition, LF image data 603 is generated based on the two-dimensional image data 601 and distance image data 602 acquired from the camera 111, so that a three-dimensional image of a subject outside the moving body captured by the camera 111 can be displayed three-dimensionally inside the moving body in real time.

[0101] In this embodiment, the optical axis of the camera 111 is directed in the direction opposite to the traveling direction of the truck 100, but the optical axis of the camera 111 is not limited to this direction. The optical axis of the camera 111 can be set arbitrarily depending on the purpose of the person viewing the LFD 112, such as the driver D, in the moving body.

[0102] In the LFD 112, the LF lens array 301 is configured such that the LF lenses 310 are arranged in the X-axis direction, but may be configured such that the lenses are arranged in the Z-axis direction. In addition, a lenticular lens is used as an example in the LFD 112, but a lens array in which the LF lenses 310 are arranged in a honeycomb pattern on the XZ plane may also be used.

[0103] The present invention is not limited to the above-described contents, and may be implemented by any combination thereof. Furthermore, other embodiments conceivable within the scope of the technical concept of the present invention are also included in the scope of the present invention.

[0104] 100 Truck, 111 Camera, 200 Image processing device, 201 Image processing unit, 211 Processor, 212 Memory, 213 Image processing circuit, 301 LF lens array, 302 Planar display unit, 310 LF lens, 601 Two-dimensional image data, 602 Range image data, 603 LF image data, 611 Acquisition unit, 612 Generation unit, 800 Lens position table, 1100 Emission angle table, b Light ray, LP Lens pitch, r(x,z) Lens position, θ Tilt angle, φ(x,z) Emission angle

Claims

1. An image processing device having: an acquisition unit that acquires first image data indicating color information in a spatial direction intersecting the depth direction of a subject existing outside a moving body, and second image data indicating a distance to the subject in the depth direction; and a generation unit that generates third image data based on the first image data and the second image data acquired by the acquisition unit, reproducing color information of a light ray from a display unit having a group of pixels that displays a three-dimensional image of the subject via a lens array to a viewing position where the display unit is viewed, as color information of reflected light from the subject, and outputs the third image data to the display unit.

2. An image processing device according to claim 1, wherein the acquisition unit acquires the first image data and the second image data from an imaging unit that images the subject.

3. An image processing device as described in claim 2, further comprising a memory unit that stores a function that defines a light ray from each pixel of the pixel group, and the generation unit determines from the second image data a first distance in the depth direction to a two-dimensional position in the spatial direction at which the light ray from the pixel reaches the subject from the imaging unit based on the function, determines color information of the two-dimensional position from the first image data, and assigns the color information of the determined two-dimensional position to the pixel from which the light ray originates.

4. An image processing device as described in claim 3, further comprising a memory unit which stores the emission angle of a light ray emitted from the pixel through the lens array for each pixel, and the generation unit generates the function for each pixel based on the emission angle and the position of the pixel, and based on the generated function, identifies the first distance from the second image data, identifies color information of the two-dimensional position from the first image data, and assigns the identified color information of the two-dimensional position to the pixel from which the light ray is emitted.

5. An image processing device as described in claim 3, comprising a memory unit which stores the lens positions of the pixels corresponding to the lens array, and the generation unit calculates the emission angle of a light ray emitted from each pixel through the lens array based on the lens position, for each pixel, based on the calculated emission angle and the position of the pixel, generates the function for each pixel based on the calculated emission angle and the position of the pixel, identifies the first distance from the second image data based on the generated function, identifies color information of the two-dimensional position from the first image data, and assigns the identified two-dimensional position to the pixel from which the light ray is emitted.

6. An image processing device as described in claim 3, wherein the generation unit calculates a lens position of the pixel corresponding to the lens array based on the position of the pixel and the lens pitch of the lens array, calculates an exit angle of a light ray emitted from the pixel through the lens array for each pixel based on the calculated lens position, generates the function for each pixel based on the calculated exit angle and the position of the pixel, identifies the first distance from the second image data based on the generated function, identifies color information of the two-dimensional position from the first image data, and assigns the identified two-dimensional position to the pixel from which the light ray is emitted.

7. An image processing device as described in claim 3, wherein the generation unit, if the first distance is within a predetermined distance range, identifies the first distance from the second image data based on the function, identifies color information of the two-dimensional position from the first image data, and assigns the identified color information of the two-dimensional position to the pixel from which the light ray is emitted.

8. An image processing device as described in claim 3, wherein, when the two-dimensional position does not exist in the second image data, the generation unit identifies from the second image data a second distance that is closest to the light ray and another two-dimensional position at the second distance, identifies color information of the other two-dimensional position from the first image data, and assigns the color information of the other two-dimensional position to the pixel from which the light ray is emitted.

9. An image processing device according to any one of claims 1 to 8, comprising: an imaging section that images the subject and outputs the first image data and the second image data; and the display section.

10. An image processing device according to claim 9, comprising a plurality of the imaging units, the generation unit generates the third image data for each of the imaging units based on the first image data and the second image data from each of the imaging units, and the display unit divides and displays a stereoscopic image for each of the third image data.

11. An image processing device according to any one of claims 1 to 10, wherein the display unit displays a stereoscopic image of the subject inside the moving body.

12. An image processing device having: an acquisition unit that acquires first image data indicating color information of a subject and second image data indicating the depth distance to the subject; and a generation unit that generates third image data based on information on the angle of light emitted from pixels of a pixel group of a display unit having a pixel group that displays a three-dimensional image of the subject via a lens array, and the first image data and the second image data acquired by the acquisition unit, and outputs the third image data to the display unit.

13. An image processing device having: an acquisition unit that acquires first image data indicating color information of a subject and second image data indicating a depth direction distance to the subject; and a generation unit that generates third image data by identifying color information at pixel positions of a pixel group of a display unit having a pixel group that displays a three-dimensional image of the subject via a lens array based on the first image data and the second image data acquired by the acquisition unit, and outputs the third image data to the display unit.

14. An image processing program that causes a processor to execute: an acquisition process that acquires first image data indicating color information in a spatial direction intersecting the depth direction of a subject outside a moving body, and second image data indicating the distance to the subject in the depth direction; and a generation process that generates third image data based on the first image data and the second image data acquired by the acquisition process, reproducing color information of light rays from a display unit having a group of pixels that displays a three-dimensional image of the subject via a lens array to a viewing position where the display unit is viewed, as color information of light reflected from the subject, and outputs the third image data to the display unit.

Images