Image processing apparatus, mobile device, image processing method, and computer program

Abstract

To provide an image processing apparatus which can generate a virtual viewpoint image with high reproducibility of object shape, a mobile body, an image processing method, and a computer program.SOLUTION: An image processing apparatus 200 includes: a plurality of imaging units each including an imaging optical system and an imaging element for generating a first image signal and a second image signal having predetermined parallax from an optical image made incident through the optical system; and a data generation unit equipped with a development unit which generates multiple pieces of image data based on outputs from the imaging units, and a distance data generation unit which generates distance data for each pixel of the multiple pieces of image data, based on the first image signal and the second image signal generated by each of the imaging units; and a video generation unit which generates a virtual viewpoint image viewed from a predetermined viewpoint, based on the multiple pieces of image data and the distance data for each pixel of the multiple pieces of image data.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a moving body, an image processing method, a computer program, and the like.

Background Art

[0002] There is known a technique for converting an image captured by a camera into an image viewed from an arbitrary viewpoint (hereinafter referred to as a virtual viewpoint) different from the shooting direction of the camera. Using this technique, an image (hereinafter referred to as a virtual viewpoint image) obtained by converting the images captured by a plurality of cameras installed in a vehicle into an image viewed from a virtual viewpoint around the vehicle is widely used for purposes such as vehicle driving and parking support.

[0003] Regarding such a virtual viewpoint image creation technique, Patent Document 1 discloses the following technique. First, as a spatial model representing the surrounding environment of a vehicle, a 3D model combining a planar model and a curved surface model is generated, and spatial data is generated by mapping the captured image onto the spatial model. Next, a virtual viewpoint image is created by referring to the spatial data and generating an image viewed from an arbitrary viewpoint. Also, Patent Document 1 discloses a technique for creating a virtual viewpoint image with reduced distortion of an obstacle by generating a plane perpendicular to the road surface as a spatial model according to the position of a three-dimensional object existing within a predetermined distance from the vehicle.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technique described in Patent Document 1 has a problem that distortion and collapse of a three-dimensional object remain on the virtual viewpoint image due to the difference between the set spatial model and the shape of the actual three-dimensional objects around the vehicle.

[0006] One objective of the present invention is to solve the above problems and provide an image processing device capable of generating virtual viewpoint images with high fidelity to the object shape. [Means for solving the problem]

[0007] One aspect of the image processing apparatus of the present invention is: A plurality of imaging means each having an optical system and an image sensor that generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system, A developing means that generates multiple image data based on the outputs of multiple imaging means, Distance data generation means that generates distance data for each pixel of the plurality of image data based on the first image signal and the second image signal generated by each of the plurality of imaging means, The system includes a video generation means that generates a virtual viewpoint video viewed from a predetermined virtual viewpoint based on a plurality of image data and distance data for each pixel of the plurality of image data. death, The aforementioned video generation means is A 3D model generation means that generates a 3D model based on the aforementioned distance data, A texture mapping means for generating a textured 3D model by mapping the image data onto the aforementioned 3D model, The rendering means includes a rendering means that generates a virtual viewpoint image, which is an image of the textured 3D model viewed from a virtual viewpoint, which is an arbitrary viewpoint position. The 3D model generation means determines the distance data for generating the 3D model based on the distance data in the overlapping imaging regions of the multiple imaging means, and based on the result of determining the occlusion regions of the multiple imaging means. It is characterized by the following: [Effects of the Invention]

[0008] According to the present invention, it is possible to realize an image processing device capable of generating virtual viewpoint images with high fidelity to the object shape. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram illustrates the positional relationship between the imaging unit and the vehicle in an embodiment of the present invention. [Figure 2] This is a functional block diagram showing the configuration of a virtual viewpoint video generation device according to an embodiment. [Figure 3](A) and (B) are schematic diagrams showing the configuration of the imaging device 102 according to the embodiment. [Figure 4] (A) to (D) are schematic diagrams showing the relationship between the subject distance and incident light in the imaging surface phase difference method. [Figure 5] (A) and (B) are flowcharts showing the operation of the data generation unit according to the embodiment. [Figure 6] It is a flowchart showing the operation of the virtual viewpoint video generation device according to the embodiment. [Figure 7] (A) and (B) are diagrams showing the positional relationship between the imaging unit and the three-dimensional objects around the vehicle. [Figure 8] (A) to (C) are diagrams showing an example of the correction process of the distance data by the 3D model generation unit. [Figure 9] It is a diagram showing an example of the 3D model generated by the 3D model generation unit.

Mode for Carrying Out the Invention

[0010] [Embodiment 1] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are denoted by the same reference numerals, and overlapping descriptions are omitted or simplified.

[0011] FIG. 1 is a diagram for explaining the positional relationship between the imaging unit and the vehicle in the embodiment of the present invention. In the present embodiment, as shown in FIG. 1, imaging units 100A to 100D as imaging means are installed at intervals in front of, on the left side, behind, and on the right side of the vehicle 100 as a moving body, respectively, to image the periphery of the vehicle 100. The optical systems of the imaging units 100A to 100D in the present embodiment are assumed to include a fisheye lens or a wide-angle lens capable of widely photographing the surroundings, and the angle of view (photographing range) of each of the imaging units 100A to 100D is shown by a dotted line in FIG. 1. In the present embodiment, four imaging units 100A to 100D are provided as described above, but a plurality of them may be provided. The configuration of the imaging units 100A to 100D will be described later.

[0012] Next, the configuration of the virtual viewpoint video generation device 200 as an image processing device according to this embodiment will be described with reference to FIG. 2. FIG. 2 is a functional block diagram showing the configuration of the virtual viewpoint video generation device according to the embodiment.

[0013] Note that some of the functional blocks shown in FIG. 2 are realized by causing a computer (not shown) included in the virtual viewpoint video generation device 200 to execute a computer program stored in a memory (not shown) as a storage medium. However, some or all of them may be realized by hardware. As the hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like can be used.

[0014] Also, each of the functional blocks shown in FIG. 2 does not have to be built in the same housing, and may be constituted by separate devices connected via signal paths. Note that in this embodiment, the imaging units 100A to 400D have the same configuration, but they do not have to have the same configuration.

[0015] The imaging units 100A to 400D each include an imaging optical system 101 and an imaging element 102. The imaging optical system 101 can form an image (optical image) of a subject on the imaging element 102 and has an exit pupil at a position separated from the imaging element 102 by a predetermined distance.

[0016] The imaging element 102 is, for example, a CMOS image sensor, and includes a pixel region in which pixels having a photoelectric conversion function are two-dimensionally arranged. Each pixel region has two photoelectric conversion units (first photoelectric conversion unit, second photoelectric conversion unit), and photoelectrically converts the subject image formed on the imaging element 102 to generate an image signal based on the subject image.

[0017] The image sensors 102 of the imaging units 100A to 100D output a first image signal based on the signal output from the first photoelectric conversion unit and a second image signal based on the signal output from the second photoelectric conversion unit to the data generation units 210A to 200D, respectively. Here, the imaging units 100A to 100D function as multiple imaging means, each having an optical system and an image sensor that generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system. In this embodiment, the configuration of the data generation units 210A to 200D is the same, but it does not have to be the same.

[0018] As described above, since the image sensor 102 is configured to output a first image signal and a second image signal, the virtual viewpoint image generation device 200 in this embodiment can measure the distance to the surrounding environment of the vehicle 100 using the image plane phase difference method described later.

[0019] Each data generation unit 210A to 210D has a development unit 211 that generates image data for each pixel, including red, green, and blue color signals and a luminance signal, based on a first image signal and a second image signal. Each development unit 211 functions as a development means that generates multiple image data based on the outputs of multiple imaging means.

[0020] Furthermore, each data generation unit 210A to 210D has a distance data generation unit 212 that generates distance data indicating distance information for each pixel. The distance data generation unit 212 functions as a distance data generation means that generates distance data for each pixel of a plurality of image data based on the first image signal and the second image signal generated by the plurality of imaging means.

[0021] Since the image data and distance data are generated from the same image signal, the image data and distance data are obtained at the same time. In other words, they are synchronized. The data generation unit 210 transmits the image data and distance data to the display image generation unit 220 and the reliability calculation unit 260.

[0022] Furthermore, the same number of data generation units 210A to 210D (four in this example) are provided as the number of imaging units 100A to 100D. Each of the data generation units 210A to 210D generates multiple image data and multiple distance data corresponding to the imaging units 100A to 100D based on the image signals from the imaging units 100A to 100D, and outputs them to the display image generation unit 220 and the reliability calculation unit 260. However, for simplicity, only the wiring of the imaging unit 100A and the data generation unit 210A is shown in Figure 2.

[0023] Furthermore, the data generation units 210A to 210D may each be provided within the imaging units 100A to 100D, and the configuration of the virtual viewpoint image generation device 200 is not limited to that described in this embodiment.

[0024] The display image generation unit 220 uses multiple image data and multiple distance data received from data generation units 210A to 210D to generate a virtual viewpoint image from an arbitrary viewpoint position. In this embodiment, the display image generation unit 220 consists of a 3D model generation unit 221, a texture mapping unit 222, a rendering unit 223, and the like.

[0025] The 3D model generation unit 221 functions as a 3D model generation means that generates a 3D model based on distance data. The texture mapping unit 222 functions as a texture mapping means that generates a textured 3D model by mapping image data onto the 3D model.

[0026] The rendering unit 223 functions as a rendering means that generates a virtual viewpoint image, which is an image of a textured 3D model viewed from a virtual viewpoint at an arbitrary viewpoint position, and creates a virtual viewpoint image of the area around the moving object. Details of the operation of these 3D model generation unit 221, texture mapping unit 222, and rendering unit 223 will be described later.

[0027] The virtual viewpoint is determined based on information regarding the virtual viewpoint determined by the virtual viewpoint determination unit 231 (hereinafter referred to as virtual viewpoint information). Details of the virtual viewpoint information will be described later. The display image generation unit 220 functions as an image generation means that generates a virtual viewpoint image as seen from a predetermined virtual viewpoint based on a plurality of image data and distance data for each pixel of the plurality of image data. The virtual viewpoint image generated by the display image generation unit 220 is transmitted to the image transmission unit 240 and the display unit 250.

[0028] The video transmission unit 240 is composed of, for example, a wireless communication unit and has the function of transmitting virtual viewpoint video to an external location outside the vehicle 100. The external location outside the vehicle 100 is, for example, the remote control room of the vehicle 100, and the operator in the remote control room can remotely control the vehicle while viewing this virtual viewpoint video.

[0029] The display unit 250 in this embodiment functions as a display means and is composed of, for example, a liquid crystal display, and displays the virtual viewpoint image received from the display image generation unit 220 for the occupants of the vehicle 100 as a moving object. The communication unit 230 has the function of receiving information for generating virtual viewpoint information from an external source, such as a remote control room.

[0030] The virtual viewpoint determination unit 231 generates virtual viewpoint information based on information received from the communication unit 230. The information received from the communication unit 230 may be external information such as from a remote control room, as mentioned above, or it may be information indicating the viewpoint generated by a vehicle occupant operating an unillustrated control unit. Alternatively, it may be linked to vehicle control information, for example, viewpoint information facing the direction in which the vehicle's turn signal is activated.

[0031] The reliability calculation unit 260 functions as a reliability determination means for calculating the reliability of the distance data generated by the distance data generation unit 212. The method for calculating reliability will be described later. The above describes the configuration of the virtual viewpoint video generation device 200 according to this embodiment.

[0032] In this embodiment, the virtual viewpoint image generation device 200 is mounted on the vehicle 100, but some parts of the virtual viewpoint image generation device 200 may be mounted on an external device located away from the vehicle 100. For example, some or all of the data generation unit 210, display image generation unit 220, virtual viewpoint determination unit 231, display unit 250, reliability calculation unit 260, etc., may be mounted on an external device.

[0033] Next, the ranging principle using the image plane phase difference method (image plane phase difference ranging method or image plane phase difference detection method) with respect to the image sensor 102 will be explained using Figures 3(A), (B) and 4(A) to (D). Figures 3(A) and (B) are schematic diagrams showing the configuration of the image sensor 102 according to the embodiment, and Figure 3(A) is a top view of the image sensor 102 as seen from the direction of light incidence.

[0034] The image sensor 102 is composed of multiple 2x2 pixel groups 310 arranged in a matrix. Each pixel group 310 has green pixels G1 and G2 for detecting green light, red pixels R for detecting red light, and blue pixels B for detecting blue light. In the pixel group 310, green pixels G1 and G2 are arranged diagonally. Each pixel also has a first photoelectric conversion unit 311 and a second photoelectric conversion unit 312.

[0035] Figure 3(B) is a cross-sectional view of the pixel group 310 in the I-I' section of Figure 3(A). Each pixel is composed of a light guide layer 314 and a light receiving layer 315, etc. The light guide layer 314 is a light guide member that efficiently guides the light beam incident on the pixel to the light receiving layer 315 and includes a microlens 313, a color filter that allows light in a wavelength band corresponding to the color of light detected by each pixel to pass through, and wiring for image readout and pixel driving.

[0036] The light-receiving layer 315 is a photoelectric conversion unit that converts light incident through the light guide layer 314 into electrical signals and outputs them as electrical signals. The light-receiving layer 315 has a first photoelectric conversion unit 311 and a second photoelectric conversion unit 312 arranged side by side in the horizontal direction of the image sensor.

[0037] Figures 4(A) to 4(D) are schematic diagrams showing the relationship between subject distance and incident light in the image plane phase-difference imaging method. Figure 4(A) is a schematic diagram showing the exit pupil 401 of the imaging optical system 101, the green pixel G1 of the image sensor 102, and the light incident on the first photoelectric conversion unit 311 and the second photoelectric conversion unit 312 of the green pixel G1. The image sensor 102 has multiple pixels, but for simplicity, we will describe one green pixel G1.

[0038] The microlens 313 of the green pixel G1 is arranged such that it is optically conjugate to the exit pupil 401 and the light-receiving layer 315. As a result, the light beam that passes through the first pupil region 410, which is a partial pupil region within the exit pupil 401, is incident on the first photoelectric conversion unit 311. Similarly, the light beam that passes through the second pupil region 420, which is a partial pupil region, is incident on the second photoelectric conversion unit 312.

[0039] Each pixel's first photoelectric conversion unit 311 converts the received light beam into electricity and outputs a signal. A first image signal is generated from the signals output from the multiple first photoelectric conversion units 311 contained in the image sensor 102. The first image signal shows the intensity distribution of the image formed on the image sensor 102 by the light beam that mainly passed through the first pupil region 410.

[0040] Similarly, the second photoelectric conversion unit 312 of each pixel converts the received light beam into electricity and outputs a signal. A second image signal is generated from the signals output from the multiple second photoelectric conversion units 312 included in the image sensor 102. The second image signal shows the intensity distribution of the image formed on the image sensor 102 by the light beam that mainly passed through the second pupil region 420.

[0041] The relative positional shift between the first and second image signals (hereinafter referred to as the disparity amount) is a quantity corresponding to the defocus amount. The relationship between the disparity amount and the defocus amount will be explained using Figures 4(B), (C), and (D).

[0042] Figures 4(B), (C), and (D) are schematic diagrams illustrating the relative positional relationship between the image sensor 102 and the imaging optical system 101. In the figures, 411 indicates the first light beam passing through the first pupil region 410, and 421 indicates the light beam passing through the second pupil region 420.

[0043] Figure 4(B) shows the state when the image is in focus, with the first light beam 411 and the second light beam 421 converging on the image sensor 102. At this time, the parallax between the first image signal formed by the first light beam 411 and the second image signal formed by the second light beam 421 is 0.

[0044] Figure 4(C) shows a state where the image is defocused in the negative direction of the w-axis. At this time, the disparity between the first image signal formed by the first light beam 411 and the second image signal formed by the second light beam 421 is not zero, but has a negative value.

[0045] Figure 4(D) shows the state where the image is defocused in the positive direction of the w-axis. At this time, the disparity between the first image signal formed by the first light beam 411 and the second image signal formed by the second light beam 421 is not zero, but has a positive value.

[0046] A comparison of Figures 4(C) and (D) shows that the direction in which parallax occurs reverses depending on whether the amount of defocus is positive or negative. Furthermore, the geometric relationship shows that the amount of parallax corresponds to the amount of defocus. Therefore, as will be described later, the amount of parallax between the first image signal and the second image signal can be detected using a region-based matching method, and the amount of parallax can be converted into a defocus amount via a predetermined conversion coefficient. Moreover, by using the imaging formula of the imaging optical system 101, which will be described later, the amount of defocus on the image side can be converted into the distance to the object.

[0047] The above is an explanation of the distance measurement principle using the image plane phase difference method. In this embodiment, the distance data generation unit 212 generates distance data from the first image signal and the second image signal using the image plane phase difference method.

[0048] Next, the detailed operation of the data generation unit 210 will be explained using Figure 5. Figures 5(A) and (B) are flowcharts illustrating the operation of the data generation unit according to the embodiment. Note that the operation of each step in the flowcharts of Figures 5(A) and (B) is performed by a CPU (not shown) acting as a computer within the virtual viewpoint image generation device 200 executing a computer program stored in memory (not shown).

[0049] First, the development process performed by the development unit 211 of the data generation unit 210 to generate image data from an image signal will be explained using Figure 5(A). This development process is executed when the development unit 211 receives an image signal from the image sensor 102.

[0050] In step S501, the CPU instructs the developing unit 211 to perform a process that generates a composite image signal by combining the first image signal and the second image signal input from the image sensor 102. By combining the first image signal and the second image signal, an image signal based on the image formed by the light beam that passed through the entire area of ​​the exit pupil 401 can be obtained.

[0051] When the horizontal pixel coordinate of the image sensor 102 is denoted as u and the vertical pixel coordinate as v, the composite image signal Im(u,v) of the pixel (u,v) can be expressed by equation 1 using the first image signal Im1(u,v) and the second image signal Im2(u,v).

number

[0052] In step S502, the CPU instructs the development unit 211 to perform a correction process for defective pixels in the composite image signal. A defective pixel is a pixel in the image sensor 102 that cannot produce a normal signal output. First, the development unit 211 acquires information indicating the coordinates of the defective pixels of the image sensor 102, which is pre-recorded in a recording unit (not shown). Next, the development unit 211 generates a composite image signal of the defective pixel using a median filter that replaces the composite image signal of the pixels surrounding the defective pixel with the median value of the composite image signals of the pixels surrounding the defective pixel.

[0053] The above is one example of the defective pixel correction process performed by the development unit 211. Alternatively, another method for correcting defective pixels may be used, which involves generating the signal value of a defective pixel by interpolating it using pre-prepared coordinate information of the defective pixel and the signal values ​​of surrounding pixels.

[0054] In step S503, the CPU instructs the development unit 211 to apply a light intensity correction process to the composite image signal to compensate for the reduction in light intensity around the field of view caused by the imaging optical system 101. As a method of light intensity correction, the composite image signal can be corrected by multiplying it by a gain value that makes the relative light intensity ratio between pre-prepared fields of view constant. For example, the development unit 211 performs light intensity correction by multiplying the composite image signal of each pixel by a gain that has the characteristic of increasing from the central pixel to the peripheral pixels of the image sensor 102.

[0055] In step S504, the CPU instructs the development unit 211 to perform noise reduction processing on the composite image signal. As a method for reducing noise, for example, Gaussian filtering may be used.

[0056] In step S505, the CPU causes the development unit 211 to perform demosaicing on the composite image signal, generating image data in which each pixel has a red (R), green (G), and blue (B) color signal and a luminance signal. As an example of demosaicing, a method may be used to generate color information for each pixel using linear interpolation for each color channel.

[0057] In step S506, the CPU causes the development unit 211 to perform gradation correction (gamma correction processing) using a predetermined gamma value. The image data Idc(u,v) of pixel (u,v) after gradation correction is expressed by equation 2 using the image data Id(u,v) of pixel (u,v) before gradation correction and the gamma value γ.

[0058]

number

[0059] The above describes the development process performed by the development unit 211. Note that the development process is not limited to this example. For example, the development unit 211 may perform only a part of the process described in steps S501 to S506. Alternatively, the development unit 211 may perform color conversion processing using a color matrix, and is not limited to any processing necessary for visually inspecting the image information output from the image sensor 102.

[0060] Next, the distance data generation process performed by the distance data generation unit 212 of the data generation unit 210 will be explained using Figure 5(B). Distance data is data that associates information corresponding to the distance from the imaging unit 100A to 100D to the subject with each pixel.

[0061] In step S511, the CPU generates a first luminance image signal using the first image signal and a second luminance image signal using the second image signal, with the distance data generation unit 212. At this time, the distance data generation unit 212 generates the luminance image signal by combining the image signal values ​​of the red, green, and blue pixels of each pixel group 310 using predetermined coefficients. Alternatively, the distance data generation unit 212 may generate the luminance image signal by performing demosaicing using linear interpolation and then combining the red, green, and blue channels by multiplying each by a predetermined coefficient.

[0062] In step S512, the CPU causes the distance data generation unit 212 to correct the light intensity balance between the first luminance image signal and the second luminance image signal. The light intensity balance correction is performed by multiplying at least one of the first luminance image signal and the second luminance image signal by a correction coefficient.

[0063] The correction coefficient is pre-calculated and stored so that the luminance ratio between the first luminance image signal and the second luminance image signal obtained by imaging uniform illumination after the position adjustment of the imaging optical system 101 and the image sensor 102 remains constant. The distance data generation unit 212 multiplies at least one of the first luminance image signal and the second luminance image signal by the correction coefficient to generate a first image signal and a second image signal with light intensity balance correction applied.

[0064] In step S513, the CPU instructs the distance data generation unit 212 to perform noise reduction processing on the first luminance image signal and the second luminance image signal to which light intensity balance correction has been applied. As a specific example of noise reduction processing, the distance data generation unit 212 may apply a low-pass filter to each luminance image signal to suppress high spatial frequency bands.

[0065] Alternatively, the distance data generation unit 212 may apply a bandpass filter that transmits a predetermined spatial frequency band to each luminance image signal. In this case, the effect of reducing the influence of the correction error of the light intensity balance correction performed in step S512 can also be obtained.

[0066] In step S514, the CPU causes the distance data generation unit 212 to calculate the disparity amount, which is the relative positional shift between the first luminance image signal and the second luminance image signal. The distance data generation unit 212 sets a point of interest within the first luminance image corresponding to the first luminance image signal and sets a matching region centered on the point of interest.

[0067] Next, the distance data generation unit 212 sets a reference point within the second luminance image corresponding to the second luminance image signal, and sets a reference region centered on the reference point.

[0068] The distance data generation unit 212 calculates the correlation between a first luminance image contained within the matching area and a second luminance image contained within the reference area while sequentially moving the reference point, and sets the reference point with the highest correlation as the corresponding point. The distance data generation unit 212 sets the amount of the relative positional shift between the point of interest and the corresponding point as the amount of disparity at the point of interest.

[0069] The distance data generation unit 212 can calculate the amount of disparity at multiple pixel positions by calculating the amount of disparity while sequentially moving the point of interest. The distance data generation unit 212 identifies a value indicating the disparity value for each pixel and generates disparity image data, which is data showing the disparity distribution.

[0070] Furthermore, the distance data generation unit 212 can use known methods for calculating the correlation index used to determine the amount of disparity. For example, the distance data generation unit 212 can use a method called NCC (Normalized Cross-Correlation) which evaluates the normalized cross-correlation between luminance images. Alternatively, the distance data generation unit 212 may use a method that evaluates the degree of difference as the correlation index.

[0071] The distance data generation unit 212 can, for example, use SAD (Sum of Absolute Difference), which evaluates the sum of absolute differences between luminance images. Alternatively, it can use SSD (Sum of Squared Difference), which evaluates the sum of squared differences.

[0072] In step S515, the CPU uses the distance data generation unit 212 to convert the disparity amount of each pixel in the disparity image data into a defocus amount, thereby obtaining the defocus amount for each pixel. Based on the disparity amount of each pixel in the disparity image data, the distance data generation unit 212 generates defocus image data that indicates the defocus amount for each pixel.

[0073] The distance data generation unit 212 uses the disparity amount d(u,v) of pixel (u,v) in the disparity image data and the conversion coefficient K(u,v) to calculate the defocus amount ΔL(u,v) of pixel (u,v) from equation 3.

[0074]

number

[0075] Furthermore, if the imaging optical system 101 has a characteristic of field curvature in which the focal position changes between the central and peripheral fields of view, then, if the amount of field curvature is Cf(u,v), the parallax amount d(u,v) can be converted to the defocus amount ΔL(u,v) using equation 4. Note that the amount of field curvature Cf(u,v) is a value that depends on the field of view.

[0076]

number

[0077] In step S516, the CPU uses the distance data generation unit 212 to convert the defocus amount ΔL(u,v) of pixel (u,v) into a distance value D(u,v) to the object at pixel (u,v), thereby generating distance data. The defocus amount ΔL can be converted using the imaging relationship of the imaging optical system 101 to calculate the distance value D to the object.

[0078] That is, when the focal length of the imaging optical system 101 is f and the distance from the principal point on the image side to the image sensor 102 is Ipp, the defocus amount ΔL(u,v) can be converted to a distance value D(u,v) to the object using the imaging formula in equation 5. In this embodiment, this distance value D(u,v) for each pixel (u,v) is called distance data.

[0079]

number

[0080] In the explanation so far, the focal length f and the distance Ipp from the principal point on the image side to the image sensor 102 have been assumed to be constant values ​​regardless of the field of view, but this is not the only option. If the imaging magnification of the imaging optical system 101 changes significantly with each field of view, at least one of the focal length f or the distance Ipp from the principal point on the image side to the image sensor 102 may be set to a value that changes with each field of view. The above is an example of the distance data generation process performed by the distance data generation unit 212.

[0081] Next, the reliability calculation process of the reliability calculation unit 260 will be explained. In the disparity amount calculation process described in step S514 of Figure 5(B), the distance data generation unit 212 searches for corresponding points using the correlation between the first luminance image signal and the second luminance image signal.

[0082] Therefore, if there is a lot of noise in the first luminance image signal (for example, noise caused by optical shot noise), or if the change in the signal value of the luminance image signal within the matching region is small, it may not be possible to correctly evaluate the correlation. In such cases, a disparity amount with a large error may be calculated compared to the correct disparity amount. If the error in the disparity amount is large, the error in the distance data generated in step S516 of Figure 5(B) will also be large.

[0083] The reliability calculation unit 260 performs a reliability calculation process to calculate the reliability of the parallax amount (parallax reliability). Parallax reliability is an index that indicates how much error is contained in the calculated parallax amount. For example, the ratio of the standard deviation to the average value of the signal values ​​included in the matching area (hereinafter referred to as the average value) can be evaluated as the parallax reliability. The standard deviation increases when the change in signal values ​​within the matching area (so-called contrast) is large. The average value increases when the amount of light incident on the pixel is large.

[0084] Furthermore, if the amount of light incident on a pixel is large, the amount of light shot noise increases. In other words, the average value has a positive correlation with the amount of noise. Therefore, the ratio of the average value to the standard deviation (standard deviation / average value) corresponds to the ratio of the contrast magnitude to the amount of noise. That is, if the contrast is sufficiently large relative to the amount of noise, it can be estimated that the error in calculating the amount of disparity will be small.

[0085] In this embodiment, the ratio of the mean value to the standard deviation is defined as the parallax confidence. The higher the parallax confidence, the smaller the error in the calculated parallax amount, and the more accurate the parallax amount is considered to be. Since there is a positive correlation between parallax confidence and distance data, the confidence of the distance data can also be expressed using this parallax confidence.

[0086] Furthermore, the reliability calculation unit 260 may calculate the reliability of the parallax amount based on the characteristics of the imaging optical system 101 of each of the multiple imaging units 100A to 100D. In this embodiment, the characteristic of the imaging optical system 101 is the focal length, and the reliability calculation unit 260 determines that the longer the focal length, the higher the reliability. The above is one example of the reliability calculation process by the reliability calculation unit 260.

[0087] Next, the operation sequence of the virtual viewpoint image generation device 200 will be described using Figure 6. Figure 6 is a flowchart showing the operation of the virtual viewpoint image generation device according to the embodiment. Note that each step in the flowchart of Figure 6 is performed by a CPU (not shown) acting as a computer within the virtual viewpoint image generation device 200 executing a computer program stored in memory (not shown).

[0088] In step S601, the CPU captures images of the area around the vehicle using multiple imaging units 100A to 100D installed on the vehicle 100. The image sensors 102 of the imaging units 100A to 100D output a first image signal based on the signal output from the first photoelectric conversion unit and a second image signal based on the signal output from the second photoelectric conversion unit to the data generation units 210A to 210D, respectively. Here, step S601 functions as an imaging step that acquires imaging outputs from multiple imaging means.

[0089] Alternatively, when outputting the first image signal from the first photoelectric conversion unit and then outputting the image signal from the second photoelectric conversion unit, the sum (combined) output of the first and second image signals may be output. Then, the second image signal may be obtained by subtracting the first image signal from the above combined output.

[0090] In step S602, the CPU uses the development unit 211 of the data generation units 210A to 210D to perform development processing on the first image signal and the second image signal output from the imaging units 100A to 100D, respectively, and generates image data. Here, step S602 functions as a development step that generates multiple image data based on the imaging outputs of multiple imaging means. The development unit 211 outputs each image data to the texture mapping unit 222 and the reliability calculation unit 260.

[0091] In step S603, the CPU performs distance data generation processing using the first image signal and the second image signal output from the imaging units 100A to 100D, respectively, with the distance data generation unit 212 of the data generation units 210A to 210D.

[0092] Here, step S603 functions as a distance data generation step, which generates distance data for each pixel of a plurality of image data based on the first and second image signals generated by the plurality of imaging means. The distance data generation unit 212 outputs this distance data to the 3D model generation unit 221 and the confidence calculation unit 260.

[0093] In step S604, the CPU acquires information regarding the position and orientation of multiple imaging units 100A to 100D (hereinafter referred to as imaging viewpoint information) using the 3D model generation unit 221 and the texture mapping unit 222, respectively.

[0094] In this embodiment, the imaging viewpoint information includes position and orientation information of the imaging units 100A to 100D within a predetermined coordinate system, and for example, includes position information and orientation information indicating the optical axis direction of the imaging device. The imaging viewpoint information may also include field of view information of the imaging device, such as the focal length or principal point position of the imaging device. Based on this imaging viewpoint information, known methods can be used to associate each pixel of the imaging data with the position of the subject present in the imaging data.

[0095] In this embodiment, as shown in Figure 1, the coordinate system is defined as follows: the vertical direction of the vehicle 100 is the x-axis, the horizontal direction is the y-axis, and the height direction is the z-axis. The origin of the coordinate system is the point that is the center of the vehicle and the ground surface when viewed along the z-axis. Furthermore, the coordinate system is a right-handed coordinate system, where the direction of travel of the vehicle 901 is the positive x-axis, the left side of the vehicle 901 is the positive y-axis, and the direction towards the sky is the positive z-axis. Hereafter, the above coordinate system will be referred to as the world coordinate system. However, the coordinate system and the position of the origin are not limited to these, and any arbitrary coordinate system may be set.

[0096] In step S605, the CPU uses the 3D model generation unit 221 to integrate multiple distance data corresponding to the field of view of each imaging unit 100A to 100D, which are output from the distance data generation unit 212, to generate a 3D model of the area around the vehicle 100. Here, each of the multiple distance data represents the distance from each imaging unit 100A to 100D to the subject.

[0097] The 3D model generation unit 221 uses the imaging viewpoint information from the imaging units 100A to 100D acquired in step S604 to convert multiple distance data into distances from the origin of the world coordinate system (hereinafter referred to as world coordinate system distance data), which is then used for 3D model generation.

[0098] Furthermore, the 3D model generation unit 221 can generate a 3D model in the overlapping imaging regions of the multiple imaging units 100A to 100D (in this example, the overlapping imaging regions of two adjacent imaging units), taking into account the occlusion of each of the multiple imaging units 100A to 100D. This process will be explained using Figure 7.

[0099] Figures 7(A) and 7(B) show the positional relationship between the imaging unit and three-dimensional objects around the vehicle. Figure 7(A) shows the positional relationship between the imaging units 100A to 100D installed on the vehicle 100 and three-dimensional objects around the vehicle (spheres 700 and 701 in this example). Figure 7(B) shows the field of view 710A of imaging unit 100A and the field of view 710B of imaging unit 100B in this case.

[0100] Although imaging units 100A and 100B have overlapping imaging regions, the three-dimensional objects (spheres 700 and 701) in the overlapping imaging region are not equally included in the field of view 710A of imaging unit 100A and the field of view 710B of imaging unit 100B. In this example, sphere 701 is included in the field of view 710A of imaging unit 100A, but from the perspective of imaging unit 100B, it is an occlusion region obscured by sphere 700 and is not included in the field of view 710B.

[0101] Therefore, the 3D model generation unit 221 creates a 3D model representing the sphere 701 based on world coordinate system distance data corresponding to the field of view of the imaging unit 100A. Similarly, although the sunspot region 702 of the sphere 700 is included in the field of view 710B of the imaging unit 100B, from the perspective of the imaging unit 100A, it becomes an occlusion region occluded by the sphere 700 and is not included in the field of view 710A.

[0102] Therefore, the 3D model of the occlusion region of the imaging unit 100A, among the 3D models representing the sphere 700, is created by the 3D model generation unit 221 based on the world coordinate system distance data corresponding to the imaging unit 100B. Similarly, the 3D model of the occlusion region of the imaging unit 100B is generated by the 3D model generation unit 221 based on the world coordinate system distance data corresponding to the imaging unit 100A.

[0103] In this way, the 3D model generation unit 221 determines the occlusion regions of the multiple imaging means based on distance data in the overlapping imaging regions of the multiple imaging means, and determines the distance data for generating the 3D model based on the result.

[0104] Furthermore, whether or not occlusion occurs in the field of view of each of the multiple imaging units 100A to 100D can be determined, for example, by checking whether the difference in world coordinate system distance data corresponding to imaging units 100A and 100B in the overlapping imaging area is greater than or equal to a predetermined value. If the distance data is greater than or equal to a certain value, it is determined that occlusion has occurred.

[0105] Alternatively, the determination may be made based on whether the difference in pixel values ​​of the image data from imaging units 100A and 100B is greater than or equal to a certain level. If the difference in pixel values ​​is greater than or equal to a certain level, it can be determined that different subjects are being photographed, and therefore occlusion is detected.

[0106] The above describes the method for generating a 3D model that takes occlusion into consideration in step S605, as performed by the 3D model generation unit 221.

[0107] Furthermore, the 3D model generation unit 221 does not experience the occlusion described above in the field of view of each imaging unit 100A to 100D in the overlapping imaging region. Therefore, in the region where multiple imaging units 100A to 100D can capture images, the 3D model may be created using the confidence level calculated by the confidence level calculation unit 260.

[0108] A specific example of this case will be described as a method for generating a 3D model in the overlapping imaging area of ​​imaging units 100A and 100B. The 3D model generation unit 221 selects the world coordinate system distance data with the higher reliability calculated by the reliability calculation unit 260 from among the two world coordinate system distance data corresponding to imaging units 100A and 100B in the overlapping imaging area and creates a 3D model. In other words, the 3D model generation unit 221 determines the distance data to be used for 3D model generation based on reliability.

[0109] As mentioned above, this reliability may be a value obtained from the characteristics of the imaging optical system 101. In other words, the reliability of the distance data may be determined based on the characteristics of the optical system of the imaging means. For example, the focal lengths f of the imaging optical systems 101 of imaging units 100A and 100B are different. In this case, the reliability calculation unit 260 determines that the data captured by the imaging optical system with the longer focal length f has higher reliability. In other words, the characteristics of the optical system used to determine reliability include the focal length. Therefore, in the overlapping imaging region, the 3D model generation unit 221 generates a 3D model based on the world coordinate system distance data corresponding to the imaging unit with the longer focal length f.

[0110] As another example, let's consider a case where the imaging optical system 101 of both the imaging units 100A and 100B has the characteristic that the imaging magnification changes significantly according to the angle of view, and the focal length f changes with each angle of view. The characteristics of this imaging optical system 101 are, for example, that the magnification is high at the center of the angle of view and decreases towards the periphery of the angle of view, and that the focal length becomes shorter towards the periphery of the angle of view.

[0111] In this case, the closer the object is to the center of the field of view, the higher the confidence calculated by the confidence calculation unit 260 will be. Therefore, the 3D model generation unit 221 may generate a 3D model using the world coordinate system distance data from which the same subject is closer to the center of the field of view among the two world coordinate system distance data.

[0112] Furthermore, the 3D model generation unit 221 may correct distance values ​​in areas with low reliability of distance data based on the results of image data analysis. An example of such correction processing will be explained using Figure 8.

[0113] Figures 8(A) to 8(C) show an example of distance data correction processing by the 3D model generation unit. Figure 8(A) is an image resulting from region segmentation processing such as instance segmentation performed on image data by the 3D model generation unit 221.

[0114] In Figure 8(A), regions with identical patterns indicate areas that have been determined to be the same object region through region segmentation processing. This region segmentation processing allows us to obtain the analysis result that region 800 in the image data corresponds to the same object (in this example, a person). The following section will focus on explaining this region 800.

[0115] Figure 8(B) is a diagram illustrating the reliability of distance data calculated by the reliability calculation unit 260. In this example, the patterned areas in the figure represent areas where the reliability is above a predetermined value, and the areas without patterns represent areas where the reliability is below a predetermined value.

[0116] In other words, Figure 8(B) shows that the confidence level of the distance data for region 800A, which corresponds to the contour of the person's region 800, is above a predetermined value, while the confidence level of the distance data for region 800B, which does not have a contour, is below a predetermined value. This is because regions without contours or regions with low contrast result in larger errors when calculating the disparity amount in Figure 5(B). Thus, in the example in Figure 8(B), the confidence level is determined based on the contrast of the image data corresponding to the distance data.

[0117] In Figure 8(B), the outline of the region 800 of the person shown in Figure 8(A) is shown with a solid line. The 3D model generation unit 221 corrects the distance data of region 800B, which has a confidence level below a predetermined value, based on the distance data of region 800A of the same object, which has a confidence level equal to or greater than a predetermined value. An example of this correction will be explained using Figure 8(C).

[0118] Figure 8(C) is an enlarged view of a portion of the distance data for the person's region 800 in Figure 8(B). D(u1,v1) and D(u4,v1) are distance data belonging to region 800A, and their confidence level is above a predetermined value. D(u2,v1) and D(u3,v1) are distance data belonging to region 800B, and their confidence level is below a predetermined value.

[0119] Here, we will explain the correction process of the 3D model generation unit 221 for distance values ​​D(u2, v1) and D(u3, v1) whose reliability is less than a predetermined value.

[0120] First, the 3D model generation unit 221 refers to the region division processing results and determines which region D(u2, v1) and D(u3, v1) belong to. In this example, it is determined that D(u2, v1) and D(u3, v1) belong to the region 800 of the person.

[0121] Next, the 3D model generation unit 221 identifies distance data within the human region that are located near D(u2, v1) and D(u3, v1) and have a confidence level equal to or greater than a predetermined value. In this example, D(u1, v1) and D(u4, v1) are identified.

[0122] Next, the 3D model generation unit 221 linearly interpolates the distance data of D(u2, v1) and D(u3, v1) using the distance values ​​D(u1, v1) and D(u4, v1) that exist in region 800A of the same person, where the confidence level is above a predetermined value, and replaces them with the result. In other words, the 3D model generation unit 221 corrects the distance data of a predetermined region of an object that has been determined to have a confidence level below a predetermined value, based on the distance data of a region within the same object where the confidence level is above a predetermined value.

[0123] The above is an example of how the 3D model generation unit 221 corrects distance data for a predetermined object region with low reliability based on the analysis results of the image data, using distance data for the same object region with high reliability. The 3D model generation unit 221 then performs 3D model generation processing using this corrected distance data. However, the method of correction processing is not limited to this.

[0124] As another example, the 3D model generation unit 221 determines whether the unreliable distance data for a given object region is within a range that is above the lower limit and below the upper limit of the highly reliable distance data for the surrounding identical object region. If it is outside this range, the 3D model generation unit 221 may replace the unreliable distance data with a value within the range.

[0125] Generally, in distance measurement using parallax images, the parallax almost disappears when the object to be measured is far away, resulting in variations in distance measurement accuracy. Therefore, the 3D model generation unit 221 may create a 3D model for areas that are more than a predetermined distance away from the vehicle 100 by using another 3D model that has been pre-recorded in a recording unit (not shown).

[0126] This process will be explained using Figure 9. Figure 9 shows an example of a 3D model generated by the 3D model generation unit 221. The three-dimensional object model 902 is a 3D model representing a three-dimensional object (in this example, a person) that exists around the vehicle 100. The ground contact surface model 903 is a 3D model representing the ground contact surface of the vehicle 100. The curved surface model 904 is a 3D model that extends from the plane of the ground contact surface model 903 in the positive z-axis direction of the world coordinate system and surrounds the vehicle 100. The 3D model generation unit 221 generates a 3D model consisting of the three-dimensional object model 902 around the vehicle, the ground contact surface model 903 representing the road surface on which the vehicle is located, and the curved surface model 904, as shown in Figure 9.

[0127] The three-dimensional object model 902 and the ground surface model 903, which are located within a predetermined distance 905 from the vehicle 100, are both created by the 3D model generation unit 221 based on world coordinate system distance data. On the other hand, the curved surface model 904 is a model that has been pre-recorded in a recording unit (not shown).

[0128] The distance 905 from the vehicle 100 to the starting point of the curved surface model 904 may be predetermined based on the characteristics of the imaging optical system 101 of the imaging units 100A to 100D. Alternatively, the 3D model generation unit 221 may set the distance 905 to the starting point of the curved surface model 904 if it determines that the average reliability of data representing a distance greater than a certain predetermined value in the world coordinate system distance data is lower than a predetermined value.

[0129] Furthermore, the curved surface model 904 is not limited to the model described above; any 3D model that has length in the z-axis direction of the world coordinate system, that is, a height, is acceptable. The reason for defining a 3D model with height is to suppress distortion and tilting in the virtual viewpoint image for three-dimensional objects such as people and buildings that are farther away from vehicle 100 than a distance of 905. Other examples of 3D models include a bowl-shaped spherical model.

[0130] The above is an example of the 3D model generation process performed by the 3D model generation unit 221 for areas located beyond a certain distance from the vehicle. Processing as in this embodiment can suppress unnatural distortion of the shape of areas located beyond a predetermined distance from the vehicle 100, where the accuracy of the world coordinate system distance data may vary.

[0131] The above is a specific example of the 3D model generation process performed by the 3D model generation unit 221 in this embodiment. The 3D model generation unit 221 outputs the generated 3D model to the texture mapping unit 222.

[0132] Returning to Figure 6, in step S606, the CPU performs texture mapping processing using the texture mapping unit 222 based on the image data and imaging viewpoint information acquired from the development unit 211, and the 3D model acquired from the 3D model generation unit 221. Texture mapping processing is the process of generating a textured 3D model by establishing a positional correspondence between the 3D model and the image data. The texture mapping processing will be explained below.

[0133] The texture mapping unit 222 uses the image data received from the development unit 211 as a texture image for texture mapping. Specifically, the texture mapping unit 222 determines the pixel values ​​of the image data corresponding to the distance data used to generate the 3D model as the texture for the corresponding part of the 3D model.

[0134] Furthermore, in the overlapping imaging regions of imaging units 100A to 100D, if a 3D model is generated by averaging multiple distance data, the pixel values ​​of the corresponding multiple image data may be blended to determine the texture.

[0135] Furthermore, the texture mapping unit 222 predetermines the pixel values ​​of the texture to be applied to the occlusion region that is not included in all of the field angles of the imaging units 101A to 101D. In this embodiment, the texture of the occlusion region is defined as black (r,g,b)=(0,0,0), that is, a texture with R, G, and B signals of zero. Note that the pixels prepared for the occlusion region do not have to be black; in that case, the pixel values ​​to be set can be the pixel values ​​of the desired color.

[0136] By performing the above process for every pixel of the 3D model, a 3D model with a texture mapped to it is generated. However, the texture mapping process described above is just one example, and any method that can map textures from multi-view image data to a 3D model may be used.

[0137] For example, if the aforementioned curved surface model 904 is composed of a collection of triangular polygons, the texture mapping unit 222 can perform texture mapping by using a known method to associate the vertices of each polygon with a texture.

[0138] The texture mapping unit 222 outputs the 3D model with the texture mapped to it to the rendering unit 223.

[0139] In step S607, the CPU determines virtual viewpoint information based on the information received from the communication unit 230 using the virtual viewpoint determination unit 231 and outputs it to the rendering unit 223.

[0140] The virtual viewpoint information, like the imaging viewpoint information, includes positional information of the virtual viewpoint within a predetermined coordinate system, such as positional information of the virtual viewpoint and attitude information indicating the optical axis direction. Furthermore, the virtual viewpoint information includes field-of-view information from the virtual viewpoint, resolution information of the virtual viewpoint image, etc. In addition, the virtual viewpoint information may include distortion parameters and imaging parameters, etc. In this embodiment, the predetermined coordinate system is the same as the world coordinate system described above.

[0141] Furthermore, the information received from the communication unit 230 may be, for example, information indicating a virtual viewpoint, input by the occupant of the vehicle 100 via an operating unit such as a remote control or touch panel display (not shown). In this case, the virtual viewpoint determination unit 231 determines the virtual viewpoint information according to the instructions.

[0142] Another example of information received from the communication unit 230 is information related to changes in the direction of travel of the vehicle 100, such as control information for the vehicle 100's turn signals. In this case, the virtual viewpoint determination unit 231 determines virtual viewpoint information corresponding to the direction of travel.

[0143] In step S608, the CPU, using the rendering unit 223, performs rendering on the textured 3D model obtained from the texture mapping unit 222 based on the virtual viewpoint information obtained from the virtual viewpoint determination unit 231, thereby generating a virtual viewpoint image. This rendering process generates an image of the textured 3D model as seen from the virtual viewpoint position.

[0144] Furthermore, the method for rendering a 3D model with textures as seen from a virtual viewpoint is publicly known, and any method may be used, so the explanation will be omitted. The rendering unit 223 outputs the rendered virtual viewpoint image to the video transmission unit 240 and the display unit 250.

[0145] In step S609, the CPU displays the received virtual viewpoint video on the display unit 250, and in step S610, the CPU transmits the received virtual viewpoint video to the outside of the vehicle 100 on the video transmission unit 240. Here, steps S605 to S609 function as video generation steps that generate a virtual viewpoint video viewed from a predetermined virtual viewpoint based on a plurality of image data and distance data for each pixel of the plurality of image data.

[0146] In this embodiment, in step S605, the 3D model generation unit 221 utilizes the 3D model using all the distance data input to it. In this embodiment, all the distance data refers to the imaging data from the imaging units 100A to 100D obtained from the distance data generation unit 212, which is the distance data for the entire circumference of the vehicle 100.

[0147] In this embodiment, by recording the 3D model of the entire surroundings of the vehicle 100 generated in this way, the virtual viewpoint can be changed at any time to check the area around the vehicle 100. Therefore, even when the vehicle 100 is not being driven, the virtual viewpoint image of the area around the vehicle 100 can be checked, improving the functionality of the dashcam and other devices.

[0148] However, the 3D model generation unit 221 may, when generating a 3D model, use virtual viewpoint information obtained from the virtual viewpoint determination unit 231 and generate a 3D model using only the distance data necessary for generating a virtual viewpoint image. For example, if the virtual viewpoint information indicates the area to the left front of the vehicle 100, the 3D model generation unit 221 may generate a 3D model using only the distance data obtained from the image information of the imaging units 100A and 100B for virtual viewpoint generation.

[0149] In other words, the 3D model generation unit 221 may select distance data to be used for 3D model generation from among multiple distance data based on the position of the virtual viewpoint. This reduces the processing load on the display image generation unit 220 for the virtual viewpoint image, making it suitable for viewing virtual viewpoint images of the vehicle 100 in real time, such as for parking assistance or remote control.

[0150] As described above, the virtual viewpoint image generation device 200 in this embodiment can reduce the positional and temporal discrepancies between the 3D model and the texture. Furthermore, it is possible to realize virtual viewpoint images with high fidelity to the object shape using a smaller device configuration than conventional devices.

[0151] In the above-described embodiment, an example was given in which the virtual viewpoint image generation device 200, as an image processing device, is mounted on a moving body such as a vehicle 100. However, the moving body in this embodiment is not limited to vehicles such as automobiles, but may be any moving body such as a train, ship, airplane, robot, or drone. Furthermore, the image processing device in this embodiment includes those mounted on such moving bodies. In addition, the virtual viewpoint image generation device 200 as an image processing device in this embodiment includes an external device that is positioned away from the moving body and used to remotely control the moving body.

[0152] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible in accordance with the spirit of the present invention, and these modifications are not excluded from the scope of the present invention.

[0153] For example, in the above embodiment, an example was described using an image sensor consisting of one image sensor using an image plane phase difference method (image plane phase difference distance measuring method or image plane phase difference detection method), but the image sensor may also be a stereo camera consisting of two image sensors. That is, the image sensor that generates a first image signal and a second image signal having a predetermined parallax from an optical system and an optical image incident through the optical system may include one or two image sensors.

[0154] This embodiment includes the following configurations, methods, and computer programs.

[0155] (Configuration 1) An image processing apparatus characterized by comprising: a plurality of imaging means each having an optical system and an image sensor that generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system; a developing means that generates a plurality of image data based on the outputs of the plurality of imaging means; a distance data generation means that generates distance data for each pixel of the plurality of image data based on the first image signal and the second image signal generated by the plurality of imaging means; and an image generation means that generates a virtual viewpoint image viewed from a predetermined virtual viewpoint based on the plurality of image data and the distance data for each pixel of the plurality of image data.

[0156] (Configuration 2) The image processing apparatus according to Configuration 1, wherein the image generation means comprises a 3D model generation means for generating a 3D model based on the distance data, a texture mapping means for generating a textured 3D model by mapping the image data onto the 3D model, and a rendering means for generating a virtual viewpoint image which is an image of the textured 3D model viewed from the virtual viewpoint which is an arbitrary viewpoint position.

[0157] (Configuration 3) The image processing apparatus according to Configuration 2, characterized in that a plurality of imaging means are arranged spaced apart from each other to image the area around a moving object, and the rendering means creates a virtual viewpoint image of the area around the moving object.

[0158] (Configuration 4) The image processing apparatus according to Configuration 2 or 3, wherein the 3D model generation means determines the distance data for generating the 3D model based on the distance data in the overlapping imaging regions of the plurality of imaging means, and based on the result of determining the occlusion region of the plurality of imaging means.

[0159] (Configuration 5) An image processing apparatus according to any one of Configurations 2 to 4, further comprising a reliability determination means for determining the reliability of the distance data, wherein the 3D model generation means determines the distance data to be used for 3D model generation based on the reliability.

[0160] (Configuration 6) The image processing apparatus according to Configuration 5, characterized in that the reliability determination means determines the reliability of the distance data based on the characteristics of the optical system of the imaging means.

[0161] (Configuration 7) The image processing apparatus according to Configuration 6, characterized in that the characteristics of the optical system include the focal length.

[0162] (Configuration 8) The image processing apparatus according to Configuration 5, characterized in that the reliability determination means determines the reliability based on the contrast of the image data corresponding to the distance data.

[0163] (Configuration 9) The image processing apparatus according to any one of Configurations 5 to 8, characterized in that the 3D model generation means corrects the distance data of a predetermined region of an object whose reliability has been determined to be less than a predetermined value by the reliability determination means based on the distance data of a region within the same object whose reliability is equal to or greater than the predetermined value.

[0164] (Configuration 10) The image processing apparatus according to any one of Configurations 2 to 9, characterized in that the 3D model generation means selects the distance data to be used for 3D model generation from among a plurality of distance data based on the position of the virtual viewpoint.

[0165] (Configuration 11) A mobile body characterized by comprising: a plurality of imaging means each having an optical system and an image sensor that generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system; a developing means that generates a plurality of image data based on the outputs of the plurality of imaging means; a distance data generation means that generates distance data for each pixel of the plurality of image data based on the first image signal and the second image signal generated by the plurality of imaging means, respectively; an image generation means that generates a virtual viewpoint image viewed from a predetermined virtual viewpoint based on the plurality of image data and the distance data for each pixel of the plurality of image data; and a display means that displays the virtual viewpoint image generated by the image generation means.

[0166] (Configuration 12) The image processing apparatus according to any one of Configurations 1 to 11, characterized in that the image sensor includes one or two image sensors.

[0167] (Method 1) An image processing method characterized by comprising: an imaging step of acquiring imaging output from a plurality of imaging means each equipped with an optical system and an image sensor that generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system; a development step of generating a plurality of image data based on the imaging outputs of the plurality of imaging means; a distance data generation step of generating distance data for each pixel of the plurality of image data based on the first image signal and the second image signal generated by the plurality of imaging means, and an image generation step of generating a virtual viewpoint image viewed from a predetermined virtual viewpoint based on the plurality of image data and the distance data for each pixel of the plurality of image data.

[0168] (Program 1) A computer program for controlling each means of an image processing apparatus described in any one of configurations 1 to 11 or a mobile body described in configuration 12 using a computer.

[0169] Furthermore, the present invention may also be realized by supplying a storage medium containing software program code (control program) that realizes the functions of the embodiments described above to a system or device. It can also be realized by the computer (or CPU or MPU) of the system or device reading and executing the computer-readable program code stored on the storage medium. In that case, the program code read from the storage medium itself will realize the function of the embodiment described above, and the storage medium storing that program code will constitute the present invention. [Explanation of symbols]

[0170] 100: Vehicles 100: Imaging Unit 101: Imaging Optical System 102: Image sensor 210: Data generation unit 211: Developing Department 212: Distance data generation unit 220: Display image generation unit 221: 3D Model Generation Unit 222: Texture Mapping Section 223: Rendering section 230: Communications Department 231: Virtual viewpoint determination unit 240: Video transmission unit 250: Display section 260: Confidence Calculation Unit 200: Virtual viewpoint image generation device

Claims

1. A plurality of imaging means each having an optical system and an image sensor that generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system, A developing means that generates multiple image data based on the outputs of multiple imaging means, Distance data generation means that generates distance data for each pixel of the plurality of image data based on the first image signal and the second image signal generated by each of the plurality of imaging means, The system includes a plurality of image data and a video generation means that generates a virtual viewpoint video viewed from a predetermined virtual viewpoint based on the distance data for each pixel of the plurality of image data, The aforementioned video generation means is A 3D model generation means that generates a 3D model based on the distance data, A texture mapping means for generating a textured 3D model by mapping the image data onto the aforementioned 3D model, The rendering means has a rendering means that generates a virtual viewpoint image, which is an image of the textured 3D model viewed from a virtual viewpoint, which is an arbitrary viewpoint position. The 3D model generation means is characterized in that it determines the distance data for generating the 3D model based on the distance data in the overlapping imaging regions of the plurality of imaging means, and based on the result of determining the occlusion region of the plurality of imaging means.

2. A plurality of imaging means each having an optical system and an image sensor that generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system, A developing means that generates multiple image data based on the outputs of multiple imaging means, Distance data generation means that generates distance data for each pixel of the plurality of image data based on the first image signal and the second image signal generated by each of the plurality of imaging means, The system includes a plurality of image data and a video generation means that generates a virtual viewpoint video viewed from a predetermined virtual viewpoint based on the distance data for each pixel of the plurality of image data, The aforementioned video generation means is A 3D model generation means that generates a 3D model based on the distance data, A texture mapping means for generating a textured 3D model by mapping the image data onto the aforementioned 3D model, The rendering means has a rendering means that generates a virtual viewpoint image, which is an image of the textured 3D model viewed from a virtual viewpoint, which is an arbitrary viewpoint position. The system further includes a reliability determination means for determining the reliability of the distance data, The 3D model generation means is an image processing apparatus characterized by determining the distance data to be used for 3D model generation based on the reliability.

3. The aforementioned reliability determination means is The image processing apparatus according to claim 2, characterized in that the reliability of the distance data is determined based on the characteristics of the optical system of the imaging means.

4. The image processing apparatus according to claim 3, characterized in that the characteristics of the optical system include the focal length.

5. The aforementioned reliability determination means is The image processing apparatus according to claim 2, characterized in that the reliability is determined based on the contrast of the image data corresponding to the distance data.

6. The 3D model generation means is The image processing apparatus according to claim 5, characterized in that the distance data of a predetermined region of an object whose reliability is determined to be less than a predetermined value by the reliability determination means is corrected based on the distance data of a region within the same object whose reliability is equal to or greater than the predetermined value.

7. A plurality of imaging means each having an optical system and an image sensor that generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system, A developing means that generates multiple image data based on the outputs of multiple imaging means, Distance data generation means that generates distance data for each pixel of the plurality of image data based on the first image signal and the second image signal generated by each of the plurality of imaging means, A video generation means that generates a virtual viewpoint video viewed from a predetermined virtual viewpoint based on a plurality of image data and distance data for each pixel of the plurality of image data, The system includes a display means for displaying the virtual viewpoint image generated by the image generation means, The aforementioned video generation means is A 3D model generation means that generates a 3D model based on the distance data, A texture mapping means for generating a textured 3D model by mapping the image data onto the aforementioned 3D model, The rendering means has a rendering means that generates a virtual viewpoint image, which is an image of the textured 3D model viewed from a virtual viewpoint, which is an arbitrary viewpoint position. The 3D model generation means is characterized by determining the distance data for generating the 3D model based on the distance data in the overlapping imaging regions of the plurality of imaging means, and based on the result of determining the occlusion region of the plurality of imaging means.

8. An imaging step in which imaging output is acquired from a plurality of imaging means each equipped with an optical system and an image sensor that generates a first image signal and a second image signal having a predetermined parallax from an optical image incident through the optical system, A development step that generates a plurality of image data based on the imaging output of a plurality of imaging means, A distance data generation step that generates distance data for each pixel of the plurality of image data based on the first image signal and the second image signal generated by each of the plurality of imaging means, The process includes a video generation step of generating a virtual viewpoint video viewed from a predetermined virtual viewpoint based on a plurality of image data and distance data for each pixel of the plurality of image data, The aforementioned video generation step is: A 3D model generation step that generates a 3D model based on the distance data, A texture mapping step generates a textured 3D model by mapping the image data onto the aforementioned 3D model, The rendering step includes generating a virtual viewpoint image, which is an image of the textured 3D model viewed from a virtual viewpoint, which is an arbitrary viewpoint position. The 3D model generation step is an image processing method characterized by determining the distance data for generating the 3D model based on the distance data in the overlapping imaging regions of the plurality of imaging means, and the result of determining the occlusion regions of the plurality of imaging means.

9. A computer program for causing a computer to function as an image processing apparatus according to any one of claims 1 to 6 or as a means for a mobile body according to claim 7.

Images