Image processing device and computer program product

By setting virtual viewpoints and interpolation processing when multiple imaging devices capture images, the blind spots and dual image problems in the overlapping parts are solved, and the generation of wide-field images and image quality improvement is achieved.

CN114556912B9Active Publication Date: 2024-11-08KK TOKAI RIKA DENKI SEISAKUSHO +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080072055.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-10-17
Filing Date
2020-07-10
Publication Date
2024-11-08
Estimated Expiration
2040-07-10

AI Technical Summary

Technical Problem

The prior art When capturing images using multiple imaging devices, there are blind spots and dual image problems in overlapping parts, especially when enlarging the field of view.

Method used

By setting virtual viewpoints and performing interpolated image generation, the interpolated image generated by virtual viewpoints and the actually captured image are stitched, blind spots and dual images in the overlapping parts are reduced, and multiple virtual viewpoints are used to adjust the image difference to ensure the stitched image Continuity.

Benefits of technology

It effectively reduces blind spots and dual images in the overlapping capture part, provides a wide field of view images, while avoiding the step phenomenon of stitching images and improving image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114556912B9_ABST
    Figure CN114556912B9_ABST
Patent Text Reader

Abstract

An image processing device that combines a plurality of images to generate an image with a wide field of view, the plurality of images being captured by a plurality of image capture devices so as to form an overlapping portion where image capture areas overlap with each other. The image processing device includes: an acquisition unit that acquires images captured by the plurality of image capture devices; a virtual viewpoint setting unit that sets at least one virtual viewpoint corresponding to the overlapping portion; an interpolated image generation unit that generates an image from the perspective of the virtual viewpoint set by the virtual viewpoint setting unit by interpolating the images captured by the plurality of image capture devices; and a splicing unit that splices the interpolated image generated by the interpolated image generation unit with portions of the images captured by the plurality of image capture devices other than the overlapping portion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an image processing device and an image processing program for merging vehicle surrounding images captured in separate segments. Background Art

[0002] In a technique in which images captured by two or more imaging devices installed spaced apart from each other are provided with overlapping areas (overlapping portions) and merged together, misalignment between images is basically corrected by referring to a predetermined virtual screen (projection position).

[0003] When this is performed, a blind spot that is not reflected in the display may exist in front of the virtual screen, and an area where the imaged subject appears as a double image may exist behind the virtual screen.

[0004] The wider the spacing between the two cameras used to capture the overlapping images, the larger the blind spot will be and the greater the amount of misalignment of the dual images will be.

[0005] Japanese Patent Application Laid-Open (JP-A) No. 2012-244527 describes a low-cost technology for presenting images observed from multiple viewpoints distributed over a wider range.

[0006] More specifically, the technology described in JP-A No. 2012-244527 includes: an imaging unit for capturing an image of an imaging subject using a plurality of imaging devices; an image supplementing unit for supplementing an image captured from a given position within a plane constituted by a plurality of line segments connecting the positions of all or some of the plurality of imaging devices based on the captured image; and a given viewpoint image generating unit for generating an image of the imaging subject as if captured from a given point within an imaging range common to each of the plurality of imaging devices based on an imaging device surface image constituted by images imaged by the plurality of imaging devices and the supplementing images.

[0007] JP-A No. 2013-118508 describes reducing the amount of misalignment of the peripheral edge of a subject at a boundary line in a composite image.

[0008] More specifically, the image projection screen setting unit sets a plurality of projection screens that are perpendicular to a virtual line of sight connecting a virtual viewpoint to a vanishing point. The plurality of projection screens are at different distances from the virtual viewpoint. The image merging unit forms a plurality of synthetic image candidates for each of the projection screens by projecting a rear image, a left rear image, and a right rear image onto the projection screen around the intersection between the projection screen and a boundary plane extending parallel to the virtual line of sight and spaced a predetermined distance from the virtual line of sight. The image evaluation unit evaluates the misalignment amount of the peripheral edge of the object at the boundary within the synthetic image for each of the synthetic images, and the image selection unit selects the synthetic image with the smallest misalignment amount from the synthetic image. This makes it possible to reduce the misalignment amount at the peripheral edge of the object at the boundary of the synthetic image. Summary of the invention

[0009] [Technical issues]

[0010] The technology of JP-A No. 2012-244527 describes an example of interpolating an image captured from a given position within a plane formed by a plurality of line segments connecting the positions of all or some of a plurality of cameras (note that JP-A No. 2012-244527 uses the term "supplement", but it can be understood here as being synonymous with "interpolation"). However, this technology is used to interpolate images between cameras for three-dimensional animation, in which a plurality of cameras are arranged in an arc arrangement around a single viewpoint, and there is no specific mention of eliminating blind spots or double images. In addition, there is no description or suggestion of interpolating in a two-dimensional image (still image or moving image) for an image using a plurality of cameras to expand the field of view.

[0011] On the other hand, JP-A No. 2013-118508 describes an example of reducing the amount of misalignment at a joint within an image captured using two camera devices. However, there is no specific mention of eliminating blind spots or double images within an overlapping image region.

[0012] An object of the present disclosure is to obtain an image processing device and an image processing program capable of reducing blind spots and double images in overlapping captured portions when images captured using a plurality of camera devices and including overlapping portions are merged to expand the field of view.

[0013] [Solution to the problem]

[0014] The image processing device of the first aspect of the present disclosure is an image processing device that combines multiple images to generate an image with a wide field of view, the multiple images being captured by multiple image capture devices, thereby forming an overlapping portion where the image capture areas overlap with each other. The image processing device includes: an acquisition unit that acquires images captured by multiple image capture devices; a virtual viewpoint setting unit that sets at least one virtual viewpoint corresponding to the overlapping portion; an interpolated image generation unit that generates an image from the perspective of the virtual viewpoint set by the virtual viewpoint setting unit by interpolating the images captured by the multiple image capture devices; and a stitching unit that stitches together the interpolated image generated by the interpolated image generation unit and the portion of the images captured by the multiple image capture devices other than the overlapping portion.

[0015] According to the first aspect of the present disclosure, the acquisition section acquires images captured by a plurality of image capture devices, and the virtual viewpoint setting section sets at least one virtual viewpoint corresponding to the overlapping portion.

[0016] The interpolation image generation section generates an image from the perspective of the virtual viewpoint set by the virtual viewpoint setting section by interpolating the images captured by the plurality of image capture devices.

[0017] The splicing section splices together the interpolation image generated by the interpolation image generating section and portions other than overlapping portions of the images captured by the plurality of image capturing devices.

[0018] Providing a virtual viewpoint enables obtaining an image with a wide field of view without causing undesirable problems such as the occurrence of a double image of an imaged subject or loss of the imaged subject.

[0019] An image processing device according to another aspect of the present disclosure includes: an acquisition unit that acquires an image capable of supplementing the field of view of a driver of a vehicle, the image being captured by a plurality of image capture devices as separate segments, thereby forming an overlapping portion in which image capture areas overlap with each other; a virtual viewpoint setting unit that sets at least one virtual viewpoint corresponding to the overlapping portion; an interpolated image generation unit that generates an image from the perspective of a virtual viewpoint set by the virtual viewpoint setting unit by interpolating the images captured by the plurality of image capture devices; a splicing unit that splices the interpolated image generated by the interpolated image generation unit with portions of the images captured by the plurality of image capture devices other than the overlapping portion; and an output unit that outputs the image spliced ​​by the splicing unit to an output device.

[0020] In this other aspect of the present disclosure, as an example, an image capable of supplementing the field of view of the driver of the vehicle may represent a rearward field of view. When driving a vehicle, there may be a blind spot to the rear. The image processing device using the present disclosure can present an image to assist in checking the rear without double images of the imaging subject or loss of the imaging subject.

[0021] In the present disclosure, in the interpolated image, the virtual viewpoint is closer to one of the image capture devices when moving toward a side spliced ​​with an image captured by the one of the image capture devices, and the virtual viewpoint is set to be closer to the other of the image capture devices when moving toward a side spliced ​​with an image captured by the other of the image capture devices.

[0022] In the present disclosure, the number of virtual viewpoints set is the same as the number of pixels depending on the width size of the interpolation image.

[0023] In the present disclosure, a virtual viewpoint is located at an inner division point between respective viewpoint positions of a plurality of image capture devices that capture the overlapping portion.

[0024] In the present disclosure, the interpolation image generation section detects a difference due to parallax between images, and transforms the image in which the difference is detected to perform a difference adjustment on a virtual viewpoint.

[0025] Transforming the images to adjust for the difference completely eliminates the double image.

[0026] The present disclosure also includes a distance image detection section that detects a distance to an imaging subject in the overlapping portion.

[0027] Employing the distance image detection section can reduce the computational load of difference detection during image processing, and can also generate a higher quality interpolated image.

[0028] An image processing program according to another aspect of the present disclosure causes a computer to operate as any one of the above-described image processing devices.

[0029] Note that the process of generating images from respective virtual viewpoints by the interpolation image generating section can be applied to the viewpoint interpolation technique, for example.

[0030] [Beneficial Effects of the Invention]

[0031] The present disclosure as described above demonstrates the beneficial effect of reducing blind spots and double images in overlapping captured portions when images captured using multiple cameras and including overlapping portions are merged to expand the field of view. In addition to the above beneficial effects, setting a sufficient number of virtual viewpoints enables providing a stitched image in which no steps occur. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] [ Figure 1 ] Figure 1 is a schematic configuration diagram showing the configuration of an image processing apparatus according to the first exemplary embodiment.

[0033] [ Figure 2 ] Figure 2 : is a front view showing an array state of virtual imaging devices viewed in a direction intersecting optical axes of two imaging devices applied to the image processing device according to the first exemplary embodiment.

[0034] [ Figure 3 ] Figure 3 is shown in more detail Figure 1 Schematic configuration diagram of a viewpoint interpolation image generating unit of an image processing device shown in FIG.

[0035] [ Figure 4 ] Figure 4 is a control flow chart showing the flow of processing by the image processing apparatus according to the first exemplary embodiment.

[0036] [ Figure 5A ] Figure 5A is a conceptual diagram illustrating the result of a merging process and a viewpoint generation process of an overlapping portion using a virtual camera array according to the first exemplary embodiment.

[0037] [ Figure 5B ] Figure 5B It is shown in Figure 5A A conceptual diagram of a result of a merging process of images captured by two imaging devices in a comparative example of the illustrated example.

[0038] [ Fig. 6A ] Fig. 6A : is a front view for explaining the principle behind image merging using a virtual camera array of the present disclosure when viewed in a direction intersecting the optical axis of the camera, and shows a misalignment width W0 of a double image when images captured by two real cameras are merged.

[0039] [ Figure 6B ] Figure 6B is corresponding to Fig. 6A , showing a misalignment width W1 of a dual image when a single virtual camera is provided between two real cameras.

[0040] [ Figure 6C ] Figure 6C is corresponding to Fig. 6A , showing a misalignment width W2 of a double image when three virtual cameras are arranged between two real cameras.

[0041] [ Figure 7 ] Figure 7 It is a front view viewed in a direction intersecting the optical axis of the imaging device in order to explain the principle behind the blind spot elimination in the first exemplary embodiment.

[0042] [ Figure 8 ] Figure 8 is a schematic configuration diagram illustrating an image processing apparatus according to a modified example of the first exemplary embodiment.

[0043] [ Fig. 9A ] Fig. 9A is a plan view of a vehicle according to a second exemplary embodiment.

[0044] [ Fig. 9B ] Fig. 9B yes Fig. 9A A side view of the vehicle is shown.

[0045] [ Fig.10 ] Fig.10 is a schematic diagram showing the interior of a vehicle compartment of a vehicle according to a second exemplary embodiment viewed from the center of a rear seat toward a front seat.

[0046] [ Fig.11A ] Fig.11A : is a front view showing an array state of virtual camera devices viewed in a direction intersecting with optical axes of three camera devices applied to a vehicle according to the second exemplary embodiment.

[0047] [ Fig. 11B ] Fig. 11B is a conceptual diagram illustrating the result of a merging process and a viewpoint generation process of an overlapping portion using a virtual camera array according to the second exemplary embodiment. DETAILED DESCRIPTION

[0048] First Exemplary Embodiment

[0049] Figure 1 An image processing apparatus 100 employing a virtual camera array according to a first exemplary embodiment of the present disclosure is shown.

[0050] The image processing apparatus 100 includes an L camera 102 and an R camera 104 serving as real cameras. In the image processing apparatus 100, images captured by the L camera 102 and the R camera 104 are combined and output to a monitor 106.

[0051] The monitor 106 includes a display driver 108 and a display screen 110. A composite image is displayed on the display screen 110 based on the image information received from the image processing apparatus 100.

[0052] like Figure 2 As shown, the L camera 102 and the R camera 104 are arranged along Figure 2Note that although the L camera 102 and the R camera 104 are distinguished as left (L) and right (R) due to their relative positions in the first exemplary embodiment, the respective cameras may be arranged in a straight line. Figure 2 The depth direction (longitudinal direction) of the L camera 102 and the R camera 104 are arranged in a straight line. Figure 2 The respective field of view ranges WL, WR are indicated by the single-point dashed lines in FIG. Within these ranges, images reflected on the virtual screen 114 described later are indicated by oblique lines (similarly also applies to FIG. 5 ). These are used as display images.

[0053] Note that Figure 2 As shown, the optical axes of the L camera 102 and the R camera 104 are preferably not parallel to each other, but diverge outwardly relative to each other, so that the L camera 102 and the R camera 104 have different fields of view from each other. Note that although the respective optical axes of the L camera 102 and the R camera 104 diverge outwardly relative to each other, these optical axes may be parallel.

[0054] The images captured by the L camera 102 and the R camera 104 include an overlapping portion 112 (see Figure 2 ).

[0055] If the image information from the L camera 102 and the image information from the R camera 104 are simply merged at the overlapping portion 112, the disparity caused by the parallax between the two cameras causes a phenomenon in which the imaging subject at the far end of the virtual screen 114 appears twice (double image). Such double images depend on the spacing (distance) between the L camera 102 and the R camera 104 (described in detail later with reference to FIG. 6 ).

[0056] Furthermore, if the image information from the L camera 102 and the image information from the R camera 104 are simply combined, the blind spot area 116 (see Figure 2 ) will appear in front of the virtual screen 114.

[0057] In the first exemplary embodiment, based on the image captured by the L camera 102 and the image captured by the R camera 104 (each including Figure 2The difference-adjusted images from the virtual viewpoints are generated within the field of view range WL, WR in the image capture image (in the image capture image) so as to reduce the incidence of double images and the blind spot area 116. Note that although the viewpoints can be generated at uniform intervals, setting the viewpoints to have narrower intervals can make the joining less noticeable. In addition, the joining at the positions connected together by the image stitching section can be made less noticeable by setting a virtual viewpoint near the position of the L camera 102 on one side of the generated image stitched with the portion other than the overlapping portion of the image captured by the L camera 102, and setting a virtual viewpoint near the position of the R camera 104 on one side of the generated image stitched with the portion other than the overlapping portion of the image captured by the R camera 104.

[0058] like Figure 1 As shown, the L camera 102 and the R camera 104 are each connected to a viewpoint interpolation image generation section 118 of the image processing device 100. The viewpoint interpolation image generation section 118 generates a virtual viewpoint between the L camera 102 and the R camera 104, and generates an image from the perspective of the virtual viewpoint as an interpolation image.

[0059] That is, Figure 2 As shown, the virtual camera 120 can be placed between the L camera 102 and the R camera 104, as if the interpolated image is captured from each viewpoint. The number of virtual cameras 120 is indicated by the viewpoint position indication unit 122. The number of virtual cameras 120 is at least 1, and its maximum actual number depends on the image width of the generated overlapping portion. In this example, a different virtual viewpoint is set for each single pixel width of the generated image. The number of virtual cameras 120 set can be determined according to the imaging subject to be captured by the L camera 102 and the R camera 104 and the accuracy requirements (such as the misalignment width limit of the dual image).

[0060] The viewpoint position indicating section 122 is connected to the viewpoint interpolated image generating section 118, and determines the number and positions of the virtual cameras 120 based on the images captured by the L camera 102 and the R camera 104. Each viewpoint position is represented by a value t of an internal ratio t:1-t of the position of the L camera 102 and the position of the R camera 104. The viewpoint position information is derived to the difference detecting section 128 and the image mixing section 136 as a map t(x, y) representing the corresponding t value of each pixel (x, y) in the overlapping portion.

[0061] Each t is a numerical value (decimal) from 0 to 1. For example, t=0 corresponds to the position of the L camera 102, and t=1 corresponds to the position of the R camera 104. Therefore, a value closer to 0 is adopted on the side of the image spliced ​​by the image splicing unit 124 to the area captured only by the L camera 102, and conversely, a value closer to 1 is adopted on the side of the image spliced ​​by the image splicing unit 124 to the area captured only by the R camera 104, with a continuous change between the two. For example, t=0.5 corresponds to the middle position.

[0062] The viewpoint interpolation image generation unit 118 sets the number of virtual camera devices 120 (see Figure 2 The interpolated image generated corresponding to the virtual camera device optical axis (viewpoint) cluster) is sent to the image stitching unit 124.

[0063] The images captured by the L camera 102 and the R camera 104 are also input to the image stitching section 124. The image stitching section 124 stitches together the image captured by the L camera 102, the image captured by the R camera 104, and a plurality of interpolated images.

[0064] The synthesized image generated by the stitching by the image stitching section 124 is output to the output section 126 , and is displayed on the display screen 110 of the monitor 106 as described above.

[0065] Detailed Description of the Viewpoint Interpolation Image Generator 118

[0066] Figure 3 It is shown Figure 2 2 is a schematic configuration diagram of details of the calculation processing of the viewpoint interpolation image generation section 118 shown in FIG. 1 in order to generate an interpolation image in response to the viewpoint instruction from the viewpoint position instruction section 122 .

[0067] like Figure 3 As shown, the respective images captured by the L camera 102 and the R camera 104 are input to the difference detection section 128. The difference detection section 128 is also input with the viewpoint indication information from the viewpoint position indication section 122.

[0068] The difference detection unit 128 calculates the difference map d of the images captured by the L camera 102 and the R camera 104 based on the following equations (1) and (2). Note that x and y represent coordinates in a two-dimensional image. The difference map d(x, y) with reference to the L camera 102 is derived in the following manner.

[0069] [Mathematical formula 1]

[0070] {d|L(x, y)==R(x+d, y)}...(1)

[0071] This difference map d(x,y) is a map of values ​​that align each pixel L(x,y) in the L image with the R image if it is shifted horizontally by -d(x,y) pixels. In practice, this is not a strict alignment in the mathematical sense, but a calculation using known stereo matching techniques. Stereo matching includes various techniques such as block matching, semi-global block matching; there is no limitation on the stereo matching techniques used in the present disclosure.

[0072] Similarly, the difference map d(x, y) with reference to the R camera 104 is derived as follows.

[0073] [Mathematical formula 2]

[0074] {d|L(xd,y)==R(x,y)}...(2)

[0075] Formula (1) relates to the difference due to parallax with reference to the L camera 102, and formula (2) relates to the difference due to parallax with reference to the R camera 104. Although both calculations give almost the same results, since there are areas captured only by the L camera 102 and there are areas captured only by the R camera 104, it is preferred that formula (1) and formula (2) are used in combination with each other.

[0076] Of the calculation results of the difference detection unit 128 , the calculation result of equation (1) is sent to the difference adjustment unit 130 of the L camera 102 , and the calculation result of equation (2) is sent to the difference adjustment unit 132 of the R camera 104 .

[0077] The viewpoint position information from the viewpoint position instruction unit 122 is input to each of the difference adjustment units 130 and 132 .

[0078] L Difference adjustment of the imaging device 102

[0079] The disparity adjustment unit 130 of the L camera 102 calculates the amount of transformation dL for adjusting the disparity using equation (3) based on the calculation result of equation (1) and the viewpoint position information.

[0080] [Mathematical formula 3]

[0081] dL(x, y)=d(x, y)*t(x, y)...(3)

[0082] The calculation result of the difference adjustment unit 130 is sent to the image conversion unit 134 of the L camera 102. The image conversion unit 134 is input with the image information captured by the L camera 102.

[0083] The image conversion section 134 performs conversion processing using equation (4) based on the calculation result of equation (3) by the difference adjustment section 130 and the image information from the L camera 102 so as to correct the difference due to parallax in the image captured by the L camera 102 .

[0084] [Formula 4]

[0085] L'(x, y)=L(x-dL, y)...(4)

[0086] After the conversion process performed by the image conversion unit 134 , the image information from the L camera 102 is sent to the image mixing unit 136 .

[0087] R Difference adjustment of the camera 104

[0088] The disparity adjustment unit 132 of the R camera 104 calculates the conversion amount dR for adjusting the disparity due to parallax using equation (5) based on the calculation result of equation (1) and the viewpoint position information.

[0089] [Formula 5]

[0090] dR(x, y)=d(x, y)*(1−t(x, y)) (5) The calculation result of the difference adjustment unit 132 is sent to the image conversion unit 138 of the R camera 104. The image conversion unit 138 is input with the image information captured by the R camera 104.

[0091] The image conversion section 138 performs conversion processing using equation (6) based on the calculation result of equation (5) by the difference adjustment section 132 and the image information from the R camera 104 so as to correct the difference due to parallax in the image captured by the R camera 104 .

[0092] [Mathematical formula 6]

[0093] R'(x, y)=R(x+dR, y)...(6)

[0094] After the conversion process performed by the image conversion unit 138 , the image information from the R camera 104 is sent to the image mixing unit 136 .

[0095] The image mixing unit 136 receives the viewpoint instruction information from the viewpoint position instruction unit 122 .

[0096] The image mixing unit 136 superimposes the image from each viewpoint (the image from the viewpoint of each virtual camera) based on the equation (7).

[0097] [Formula 7]

[0098] I(x, y)=(1-t)*L'(x, y)+t*R'(x, y)...(7)

[0099] Such merging is called alpha blending. A gradual change is achieved and steps caused by image quality differences between cameras are eliminated by setting the adoption ratio (1-t):1 to the inverse of the internal ratio of the position of the L camera 102 to the position of the R camera 104. In addition, the area where the images are stitched together uses almost 100% stitched camera images, thus eliminating steps between the stitched images.

[0100] The interpolated image I (x, y) superimposed in the overlapping portion by the image mixing section 136 is temporarily stored in the interpolated image storage section 140, and is sent to the above-mentioned image stitching section 124 at a predetermined timing (in a state in which synchronization has been achieved between the image from the L camera 102 and the image from the R camera 104). Note that inter-camera synchronization must be performed before the processing of the difference detection section 128.

[0101] Refer to the following Figure 4 The flowchart of FIG. 1 illustrates the operation of the first exemplary embodiment.

[0102] Figure 4 is a control flow chart showing the processing flow of the image processing apparatus. Note that the image processing may be controlled based on a predetermined control program executed by a single processor, or may be controlled by Figure 1 and Figure 3 Parallel processing in the illustrated block units implements one or more processes. Alternatively, parallel processing can be implemented by producing a processor board incorporating integrated circuits such as application specific integrated circuits (ASICs), each with a specific purpose.

[0103] like Figure 4 As shown, at step 150, images captured by the L camera 102 and the R camera 104 are imported. The process then transfers to step 152, where viewpoint position indication information is imported, and then the process transfers to step 154.

[0104] At step 154 ​​, a difference detection process with reference to the L camera 102 and a difference detection process with reference to the R camera 104 are executed.

[0105] At the next step 156 , a discrepancy adjustment process based on a discrepancy with reference to the L camera 102 and a discrepancy adjustment process based on a discrepancy with reference to the R camera 104 are performed.

[0106] At the next step 158 , an image conversion process with reference to the L camera 102 and an image conversion process with reference to the R camera 104 are performed.

[0107] Note that the difference detection processing of step 154, the difference adjustment processing of step 156, and the image conversion processing of step 158 are preferably implemented by parallel processing of the reference L camera 102 and the reference R camera 104 in order to reduce the processing time.

[0108] At the next step 160, the images corresponding to the overlapping portions of the reference L camera 102 and the reference R camera 104 which have been subjected to the transformation processing at step 158 are mixed to generate an interpolated image. The process then transfers to step 162, and the interpolated image is temporarily stored in the interpolated image storage section 140 (see Figure 3 )middle.

[0109] At the next step 164, the non-overlapping portion of the image captured by the L camera 102 and the image captured by the R camera 104 is imported, and then at step 166, the image stored in the interpolation image storage section 140 (see Figure 3 ) and the processing is transferred to step 168.

[0110] At step 168 , the image captured by the L camera 102 , the image captured by the R camera 104 , and the interpolated image are stitched together, and the process moves to step 170 .

[0111] The image display process is performed at step 170. In the image display process, the image splicing unit 124 (see Figure 1 ) are output to the monitor 106 via the output unit 126, and the display driver 108 is driven to display the stitched image on the display screen 110.

[0112] At the next step 172, it is determined whether the power of the image processing apparatus 100 has been turned off. In the case where the determination is negative, the process returns to step 150, and the above process is repeated to display an animated image. In the case where the determination at step 172 is positive, the present routine ends.

[0113] FIG. 5 shows a case where a virtual camera is set in the overlapping portion 112 in the merging process of the L camera 102 and the R camera 104 (first exemplary embodiment, see Figure 5A ) is simply combined with two camera devices (ie, the L camera device 102 and the R camera device 104) (for a comparative example, see Figure 5B Note that in the comparative example, for example, the process of merging images by alpha blending can be applied by adopting the image mixing section 136 of the first exemplary embodiment.

[0114] exist Figure 5BIn the comparative example shown, an imaged subject 142 in the overlapping portion 112 behind the virtual screen 114 is captured by both the L camera 102 and the R camera 104 and appears as a dual image when displayed on the virtual screen 114 as a result of the difference caused by parallax.

[0115] On the other hand, Figure 5A In the illustrated first exemplary embodiment, the imaging subject 142 is captured by some of the plurality of virtual cameras 120 that have been provided, thereby eliminating double images.

[0116] Note that although the imaging subject is preferably captured by a single virtual camera 120, in practice it is more often captured by two virtual cameras 120. However, since the difference caused by the parallax between the two virtual cameras 120 is very small, no double image is produced even though some distortion may occur.

[0117] The principle behind eliminating double images by setting up the virtual camera device 120

[0118] FIG. 6 illustrates the principle behind eliminating double images, particularly as a result of placing a virtual camera 120 between real cameras (ie, the L camera 102 and the R camera 104).

[0119] Fig. 6A A situation is shown in which images captured by two real cameras (L camera 102 and R camera 104) are used, for example, in Figure 5B The alpha blending process applied in the comparative example of FIG. 1 is merged, resulting in a double image with a misaligned width W0.

[0120] Figure 6B Shows Fig. 6A However, a single virtual camera 120 is provided between the L camera 102 and the R camera 104. Figure 6B A dual image is also generated in .

[0121] However, since a single virtual camera 120 is provided, the configuration is different from the configuration in which the virtual camera 120 is not provided (see Fig. 6A )compared to, Figure 6B The misalignment width W1 of the dual images in is Fig. 6A The misalignment width W0 of the double image in is half (W1=0.5×W0).

[0122] also, Figure 6C Shows Figure 6BHowever, another virtual camera 120 is provided between the L camera 102, the R camera 104 and the virtual camera 120. For convenience, three virtual cameras 120 are provided between the L camera 102 and the R camera 104. Figure 6C A dual image is also generated in .

[0123] However, since three virtual camera devices 120 are provided, the configuration is different from the configuration in which no virtual camera device 120 is provided (see Fig. 6A )compared to, Figure 6C The misalignment width W2 of the dual images in is Fig. 6A The misalignment width W0 of the double image in is one quarter (W2=0.5×(0.5×W0)=0.25×W0).

[0124] That is, the shorter the spacing (distance) between the L camera 102 and the R camera 104, the smaller the misalignment width of the double image becomes (proportionately). Note that setting the required number of virtual cameras 120 enables the difference to be reduced in proportion to the number of virtual cameras 120 set, thereby enabling the double image to be eliminated.

[0125] Note that even setting a single virtual camera 120 has an effect (i.e., halving the misalignment width of the dual images), and the maximum number of virtual cameras 120 that can be set depends on the width of the overlapping images. The number of virtual cameras 120 that are set can be set according to the accuracy requirements corresponding to the purpose of the captured images, etc.

[0126] Note that in the first exemplary embodiment, when an image is generated from the viewpoint of the virtual camera 120, an image captured by the L camera 102 and an image captured by the R camera 104 are used in combination. By using these images in combination, the blind spot of the L camera 102 and the blind spot of the R camera 104 are each freely Figure 7 The images of the other cameras shown complement each other, making it possible to eliminate the blind spot of the virtual camera 120 .

[0127] Modification example

[0128] Figure 8 1 is a schematic configuration diagram showing an image processing apparatus 100 according to a modified example of the first exemplary embodiment. Figure 3 ) are assigned the same figure marks, and the description of these configurations is omitted.

[0129] In the first exemplary embodiment, the difference due to parallax is detected based on the viewpoint position indication information from the viewpoint position indication section 122 .

[0130] However, in the modified example, the distance image generation section 144 is provided as means for providing information used in difference detection.

[0131] Visible light radar, infrared radar, light detection and ranging (LIDAR), projector-based triangulation, time-of-flight method, etc. may be applied in the range image generation unit 144 .

[0132] The distance image generation section 144 generates a depth map (z(x, y)), and sends the depth map to the difference detection section 128 of the viewpoint interpolation image generation section 118 .

[0133] The difference detection section 128 can detect the difference map d without performing the complicated calculations required by equations (1) and (2).

[0134] Second Exemplary Embodiment

[0135] The following is a description of a second exemplary embodiment of the present disclosure.

[0136] Fig. 9A is a top view showing a vehicle 10 according to the present exemplary embodiment, Fig. 9B 1 is a side view showing the vehicle 10 .

[0137] like Fig. 9A As shown, the right camera adapter 12 is attached to the right side of the vehicle 10 relative to the forward direction. An RH camera head 14 having an optical axis extending toward the right rear of the vehicle 10 is attached to the right camera adapter 12. The RH camera head 14 captures the Fig. 9A The area AR is surrounded by the single-point dashed line R in .

[0138] In addition, if Fig. 9A As shown, the left camera adapter 16 protrudes from the left side of the vehicle 10 relative to the forward direction. An LH camera head 18 having an optical axis extending toward the left rear of the vehicle 10 is attached to the left camera adapter 16. The LH camera head 18 captures the Fig. 9A The area AL surrounded by the single-point dashed line L in .

[0139] Likewise Fig. 9A As shown, the RR camera head 20 having an optical axis extending rearward from the vehicle 10 (the optical axis is substantially parallel to the road surface) is attached to the vehicle width direction center portion of the rear portion of the vehicle 10. The RR camera head 20 captures the Fig. 9A The area AC is surrounded by the single-point dashed line C in .

[0140] The area AR and the area AC partially overlap each other so that the images captured by them partially overlap. The area AL and the area AC also partially overlap each other so that the images captured by them partially overlap.

[0141] like Fig.10 As shown, a dedicated back-up camera 22 is also attached to the rear of the vehicle 10 separately from the RR camera head 20. The dedicated back-up camera 22 is configured to capture an area AB surrounded by a dotted line B, and the captured image is displayed on an information display monitor 24 (see FIG. 24 ) in the vehicle cabin when the vehicle 10 is backing up. Fig.10 ) on. The optical axis is set so that the area AB includes the rear end portion of the vehicle 10 (eg, the rear bumper), and the optical axis is directed downward compared to the optical axis of the RR camera head 20.

[0142] That is, the above-mentioned RH camera head 14, LH camera head 18 and RR camera head 20 are used to assist the driver during normal driving of the vehicle 10 (whether moving forward, reversing or stationary), and therefore have a different purpose from the dedicated reversing camera 22, which is used to specifically assist the driver when the vehicle 10 is reversing.

[0143] Fig.10 It is a schematic diagram of the interior of the vehicle cabin of the vehicle 10 viewed from the center of the rear seats toward the front seats.

[0144] The above-described right camera adapter 12 is located in front of the right side window 26 of the vehicle 10 , and the RH camera head 14 is attached to the right camera adapter 12 .

[0145] The RH monitor 32 is attached to the lower portion of the right A-pillar 30 between the right window 26 and the front window 28. The RH monitor 32 is configured to display the area AR (see FIG. 1 ) behind the right side of the vehicle captured by the RH camera head 14. Fig. 9A ) is displayed in a mirror-inverted state. That is, the RH camera head 14 and the RH monitor 32 have a function equivalent to an optical right rearview mirror.

[0146] The above-mentioned left camera adapter 16 is located in front of the left side window 34 of the vehicle 10 , and the LH camera head 18 is attached to the left camera adapter 16 .

[0147] The LH monitor 38 is attached to the lower portion of the left A-pillar 36 between the left window 34 and the front window 28. The LH monitor 38 is configured to display the area AL (see FIG. 1 ) behind the left side of the vehicle captured by the LH camera head 18. Fig. 9A ) is displayed in a mirror-inverted state. That is, the LH camera head 18 and the LH monitor 38 have a function equivalent to an optical left rearview mirror.

[0148] The information display monitor 24 is attached to the upper portion of the center console 40 in the vehicle compartment. The information display monitor 24 displays an image captured by the dedicated back-up camera device 22 when a shift lever 42 of a transmission (not shown in the drawings) of the vehicle 10 is in reverse gear (R gear).

[0149] The information display monitor 24 is used as a multi-function monitor, and for example, in addition to displaying the vehicle rear image as described above, the information display monitor 24 also coordinates with the navigation system and the audio system not shown to display a navigation screen (a screen with navigation information, etc.) or an audio screen (a screen with information related to music, etc.) when a switching operation is performed. For example, a map image and an image of the driving position of the vehicle on the map image are displayed on the navigation screen based on the position information acquired using the GPS function installed in the vehicle 10. Note that in the case where the information display monitor 24 is configured by a touch panel, the information display monitor 24 can also be used as an input device.

[0150] Note that the inside rearview mirror 44 is attached to an upper center portion of the front window 28 (or ceiling) of the vehicle 10 of the present exemplary embodiment.

[0151] The inside rearview mirror 44 has a mirror function that functions as an optical mirror and a monitor function that displays an image (corresponding to the monitor 106 (see FIG. 1 ) of the first exemplary embodiment. Figure 1 ))Both.

[0152] The mirror function enables the driver, etc. to check the situation behind the vehicle within their field of view (by moving their line of sight) while driving.

[0153] The monitor function enables display of a composite image composed of images captured by the RH camera head 14 , the LH camera head 18 , and the RR camera head 20 .

[0154] As shown in Fig. 9, the RH camera head 14, the LH camera head 18, and the RR camera head 20 mounted to the vehicle 10 capture the overlapping portion (first overlapping portion 50A) between the area AR and the area AC, and the overlapping portion (second overlapping portion 50B) between the area AL and the area AC. In other words, there are two overlapping portions where the virtual camera 120 of the present disclosure can be set.

[0155] If this is according to the L camera 102 and the R camera 104 of the first exemplary embodiment (see Figure 2 ), then for the first overlapping portion 50A, the LH camera head 18 corresponds to the L camera 102, and the RR camera head 20 corresponds to the R camera 104.

[0156] Furthermore, with respect to the second overlapping portion 50B, the RR camera head 20 corresponds to the L camera 102 , and the RH camera head 14 corresponds to the R camera 104 .

[0157] In the second exemplary embodiment, the virtual camera 120 is provided in each of the two overlapping portions (the first overlapping portion 50A and the second overlapping portion 50B), as shown in FIG. Fig.11A shown.

[0158] In the second exemplary embodiment, the image processing apparatus 100 described in the first exemplary embodiment (see Figure 3 ) are applied to the first overlapping portion 50A and the second overlapping portion 50B, respectively, to detect the difference due to parallax, adjust the detected difference, transform the image from the real camera device, superimpose the image at each overlapping portion (the first overlapping portion 50A and the second overlapping portion 50B), and generate an interpolated image. This makes it possible to eliminate (reduce) the generation of a double image of the imaging subject 142 on the virtual screen 114, such as Fig. 11B shown.

[0159] For reference, Table 1 illustrates the correspondence between the real camera devices (L camera 102, R camera 104) used to generate an interpolated image using the virtual camera 120 in the overlapping portion 112 of the first exemplary embodiment and the real camera devices (RH camera head 14, LH camera head 18, RR camera head 20) used to generate an interpolated image using the virtual camera 120 in each overlapping portion (first overlapping portion 50A and second overlapping portion 50B) of the second exemplary embodiment.

[0160] [Table 1]

[0161]

[0162] Note that although the real camera in the second exemplary embodiment is configured by the RH camera head 14, the LH camera head 18, and the RR camera head 20, there is no limitation on the number and installation position of the real camera. For example, the RR camera head 20 may be provided as a pair of RR cameras distributed between the left and right, rather than being provided at the center of the vehicle 10 (to give a total of four real cameras and three overlapping portions).

[0163] Although in the second exemplary embodiment, the rear image from the vehicle 10 is captured and displayed on the inside rearview mirror 44 for driving assistance, the image processing device of the present disclosure may also be applied to a monitoring camera device or the like.

[0164] For example, in the case of performing surveillance by capturing a given area in separate segments using a plurality of surveillance cameras, a plurality of videos from the respective surveillance cameras are displayed on a monitor (or alternatively, the display is updated at regular intervals), requiring the observer to sequentially adjust their sight lines between the respective images. However, providing overlapping portions in the image capture areas of the plurality of surveillance cameras and merging the images using the image processing apparatus of the present disclosure enables generation of a single image with a wide field of view without the occurrence of double images or loss of visibility of the imaging subject, thereby enabling movement of the observer's viewpoint to be kept to a minimum.

[0165] The following supplements related to the above-described exemplary embodiments are also disclosed.

[0166] Supplement 1

[0167] An image processing device characterized by merging a plurality of images to generate an image with a wide field of view, the plurality of images being captured by a plurality of image capturing devices so as to form overlapping portions where image capturing areas overlap with each other, the image processing device comprising:

[0168] an acquisition unit that acquires images captured by the plurality of image capture devices;

[0169] a virtual viewpoint setting unit, which sets at least one virtual viewpoint corresponding to the overlapping portion;

[0170] an interpolation image generating section that generates an image from a perspective of the virtual viewpoint set by the virtual viewpoint setting section by interpolating the images captured by the plurality of image capturing devices; and

[0171] A splicing section splices the interpolation image generated by the interpolation image generating section with a portion of the images captured by the plurality of image capturing devices other than the overlapping portion.

[0172] Supplement 2

[0173] An image processing device, comprising:

[0174] an acquisition section that acquires an image capable of supplementing a field of view of a driver of the vehicle, the image being captured as separate segments by a plurality of image capture devices, thereby forming an overlapping portion where image capture areas overlap each other;

[0175] a virtual viewpoint setting unit, which sets at least one virtual viewpoint corresponding to the overlapping portion;

[0176] an interpolation image generating section that generates an image from a perspective of the virtual viewpoint set by the virtual viewpoint setting section by interpolating the images captured by the plurality of image capturing devices;

[0177] a splicing section that splices together the interpolation image generated by the interpolation image generating section and portions of the images captured by the plurality of image capturing devices other than the overlapping portions; and

[0178] An output unit outputs the image spliced ​​by the splicing unit to an output device.

[0179] Supplement 3

[0180] The image processing device according to Supplement 1 or Supplement 2 is characterized in that, in the interpolated image, the virtual viewpoint is closer to the one of the image capture devices when moving toward the side spliced ​​with the image captured by one of the image capture devices, and the virtual viewpoint is set to be closer to the other of the image capture devices when moving toward the side spliced ​​with the image captured by the other of the image capture devices.

[0181] Supplement 4

[0182] The image processing device according to any one of Supplement 1 to Supplement 3 is characterized in that the number of the virtual viewpoints set is the same as the number of pixels depending on the width size of the interpolated image.

[0183] Supplement 5

[0184] The image processing device according to any one of Supplement 1 to Supplement 4 is characterized in that the virtual viewpoint is located at an internal division point between the respective viewpoint positions of the plurality of image capture devices that capture the overlapping portion.

[0185] Supplement 6

[0186] The image processing device according to any one of Supplement 1 to Supplement 5 is characterized in that the interpolated image generation unit detects a difference caused by parallax between images and transforms the image in which the difference is detected to adjust the difference of the virtual viewpoint.

[0187] Supplement 7

[0188] The image processing device according to any one of Supplement 1 to Supplement 6 is characterized in that it also includes a distance image detection unit, which detects the distance to the imaging subject in the overlapping part.

[0189] Supplement 8

[0190] An image processing program characterized by causing a computer to operate as the image processing device described in any one of Supplement 1 to Supplement 6.

[0191] The disclosure of Japanese Patent Application No. 2019-190453 filed on October 17, 2019 is incorporated herein by reference in its entirety.

[0192] All cited documents, patent applications, and technical standards mentioned in this specification are incorporated by reference into the specification to the same extent as if each individually cited document, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0193] [reference numerals list]

[0194] First Exemplary Embodiment

[0195] 100 Image processing device

[0196] 102 L Camera device (image capturing device)

[0197] 104 R Camera device (image capturing device)

[0198] 106 Monitor

[0199] 108 Display Driver

[0200] 110 Display

[0201] 112 Overlapping

[0202] 114 Virtual Screen

[0203] 116 Blind Spot Area

[0204] 118 Viewpoint interpolation image generation unit

[0205] 120 Virtual Camera Device

[0206] 122 Viewpoint position indication unit

[0207] 124 Image stitching unit (stitching unit)

[0208] 126 Output

[0209] 128 Difference detection unit (acquisition unit, virtual viewpoint setting unit)

[0210] 130, 132 Difference Adjustment Department

[0211] 134, 138 Image conversion unit

[0212] 136 Image mixing unit (interpolation image generation unit)

[0213] 140 interpolation image storage unit

[0214] 142 Imaging Subject

[0215] 144 Distance image generation unit

[0216] Second Exemplary Embodiment

[0217] 10 Vehicles

[0218] 12 Right camera adapter

[0219] 14 RH camera head (image capture device)

[0220] 16 Left camera adapter

[0221] 18 LH Camera Head (Image Capture Device)

[0222] 20 RR camera head (image capture device)

[0223] 22 Dedicated reversing camera

[0224] 24 Information Display Monitor

[0225] 26 Right window

[0226] 28 front window

[0227] 30 Right A-pillar

[0228] 32 RH Monitor

[0229] 34 Left window

[0230] 36 Left A-pillar

[0231] 38 LH Monitor

[0232] 40 Center console

[0233] 42 Gear lever

[0234] 44 Interior rearview mirror

Claims

1. An image processing device, the image processing device merging a plurality of images to generate an acquisition unit that acquires images captured by the plurality of image capture devices; A virtual viewpoint setting unit is configured to set the plurality of image capturing devices corresponding to the overlapping portion. The interpolation image generating unit generates an interpolation image by performing interpolation on the images captured by the plurality of image capturing devices. a splicing unit for combining the interpolated image generated by the interpolated image generating unit with the interpolated image generated by the plurality of images; 2. An image processing device, include: an acquisition unit, which acquires an image capable of supplementing the field of view of a driver of the vehicle, the image being A virtual viewpoint setting unit is configured to set the plurality of image capturing devices corresponding to the overlapping portion. The interpolation image generating unit generates an interpolation image by performing interpolation on the images captured by the plurality of image capturing devices. a splicing unit for combining the interpolated image generated by the interpolated image generating unit with the interpolated image generated by the plurality of images; An output unit outputs the image spliced ​​by the splicing unit to an output device.

3. The image processing device according to claim 1 or 2, It is characterized in that In the 4. The image processing device according to claim 1 or 2, It is characterized in that Set 5. The image processing device according to claim 1 or 2, It is characterized in that The virtual 6. The image processing device according to claim 1 or 2, It is characterized in that The inside 7. The image processing device according to claim 1 or 2, It is characterized in that Also includes 8. An image processing program product, which enables a computer to perform the image processing as claimed in any one of claims 1 to 6