Distance measurement device and method

Abstract

To provide a technology capable of reducing the difference in the number of effective pixels between multiple images and improving the performance and accuracy of distance measurement in stereo shooting and distance measurement.SOLUTION: A distance measuring device includes an imaging device and a processing device. The processing device includes; a property-matched image generation unit that inputs two or more images including a first image of a subject imaged from a first viewpoint and a second image of the subject imaged from a second viewpoint to generate two or more images as property-matched images in which the properties of the first image and the properties of the second image are matched; and a distance measuring unit that calculates the distance to the subject by calculating parallax based on the two or more images of the property-matched images. The property-matched image generation unit generates a property-matched image by projecting 800 a target pixel 701 in at least one of the first image and the second image onto a corresponding pixel position 702 in the other image.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to distance measurement technology. [Background technology]

[0002] In the field of imaging and distance measurement, there are many cases where 3D sensing based on distance information of the entire surroundings is required in addition to image information of the entire surroundings, such as in autonomous driving and people flow analysis.

[0003] One technique for obtaining two pieces of information or images about a subject (or object) at once is to use a stereo camera. A stereo camera captures images of an object from two cameras, one on the left and one on the right, in each direction. Based on the two captured images, the distance from the stereo camera to the object can be calculated.

[0004] For example, Patent Publication No. 4388530 (Patent Document 1) describes a technology for a single-camera omnidirectional binocular image acquisition device that includes a first reflecting unit, a second reflecting unit, a third reflecting unit, and an image capturing unit, and captures omnidirectional images from a first viewpoint and a second viewpoint as binocular images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 4388530 Summary of the Invention [Problem to be solved by the invention]

[0006] Patent Document 1 describes a binocular image acquisition device, in other words, a stereo camera, which uses multiple hyperbolic mirrors as shown in Figures 5C and 7, and is capable of capturing images from two viewpoints with a single image sensor, particularly an omnidirectional image as shown in Figure 8.

[0007] In stereo cameras and distance measurement technology, the distance to the subject is calculated using two or more images of the same subject taken from two or more different viewpoints. In this case, if there is a difference in the effective pixel count between the multiple captured images, the performance of the calculated distance information will be reduced.

[0008] Therefore, when performing stereo photography and distance measurement, it is preferable to make the effective pixel counts of a plurality of images uniform, in other words, to reduce the difference in the effective pixel counts.

[0009] An object of the present invention is to provide a technology that can reduce the difference in the effective pixel count between multiple images in stereo photography and distance measurement, and can improve the performance and accuracy of distance measurement. [Means for solving the problem]

[0010] A representative embodiment of the present disclosure has the following configuration: A distance measurement device according to the embodiment includes an imaging device that captures images of a subject, and a processing device that acquires and processes the images from the imaging device, wherein the processing device has a characteristic-matched image generation unit that receives input of two or more images including a first image captured of the subject from a first viewpoint and a second image captured of the subject from a second viewpoint different from the first viewpoint, and generates two or more images as characteristic-matched images in which characteristics of the first image and characteristics of the second image are matched, and a distance measurement unit that calculates a distance to the subject by calculating a parallax based on the two or more images as the characteristic-matched images, wherein the characteristic-matched image generation unit generates the characteristic-matched image by projecting a pixel of interest in at least one of the first image and the second image onto a corresponding pixel position in the other image. [Effects of the Invention]

[0011] According to a representative embodiment of the present disclosure, in regard to stereo photography and distance measurement, it is possible to reduce the difference in the effective pixel count between multiple images, thereby improving the performance and accuracy of distance measurement. Problems, configurations, effects, etc. other than those described above will be described in the description of the embodiment of the invention. [Brief explanation of the drawings]

[0012] [Figure 1] 1 shows the configuration of a distance measurement device according to a first embodiment. [Figure 2] 1 shows an example of the configuration of an imaging device according to a first embodiment. [Figure 3] 3 shows an example of an image captured by an imaging device in the first embodiment. [Figure 4] 3 shows the pixel count distribution of a captured image in the first embodiment. [Figure 5] 10 shows image processing for generating a panoramic image in a comparative example to the first embodiment. [Figure 6] An interpolation method will be described as a first example of a panoramic image generation method in a comparative example to the first embodiment. [Figure 7] A downsampling method will be described as a second example of a panoramic image generation method in a comparative example to the first embodiment. [Figure 8] FIG. 3 is a schematic explanatory diagram of projection in the first embodiment. [Figure 9] 3 is an explanatory diagram showing the elevation / depression angle and image height characteristics of the imaging device according to the first embodiment. [Figure 10] 4 shows a change in pixel number distribution in the characteristic matching process in the first embodiment. [Figure 11] 10 shows an example of projection and panoramic image generation in the first embodiment. [Figure 12] 3 shows a processing flow of the processing device in the first embodiment. [Figure 13] 1 shows the configuration of a distance measurement device according to a second embodiment. [Figure 14] 10 shows a first configuration example of an imaging device according to a second embodiment. [Figure 15]10 shows a second configuration example of the imaging device in the second embodiment. [Figure 16] 10 shows a first example of a captured image in the second embodiment. [Figure 17] 10 shows a second example of a captured image in the second embodiment. [Figure 18] FIG. 10 is a schematic explanatory diagram of projection in the second embodiment. [Figure 19] 10 shows the configuration of a distance measuring device according to a third embodiment. [Figure 20] 10 shows an example of the configuration of an imaging device according to a third embodiment. [Figure 21] FIG. 11 is a schematic explanatory diagram of projection in the third embodiment. [Figure 22] 10 shows the configuration of a distance measuring device according to a fourth embodiment. [Figure 23] 10 shows an example of a Gaussian filter according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, identical parts are generally designated by the same reference numerals, and repeated explanations will be omitted. In the drawings, the representation of components may not represent their actual positions, sizes, shapes, scopes, etc., in order to facilitate understanding of the invention. Components that are approximate or similar in shape, etc., are also included within the scope of the present invention.

[0014] For the sake of explanation, when describing processing by a program, the program, function, processing unit, etc. may be described as the main body, but the main hardware body for these is a processor, or a controller, device, computer, system, etc. that is composed of the processor, etc. A computer executes processing according to a program read into memory using resources such as memory and communication interfaces as appropriate through the processor. This realizes predetermined functions, processing units, etc. A processor is composed of, for example, semiconductor devices such as a CPU or GPU. A processor is composed of devices or circuits that are capable of performing predetermined calculations. Processing is not limited to software program processing, but can also be implemented using dedicated circuits. Dedicated circuits such as FPGAs, ASICs, and CPLDs can be used.

[0015] The program may be pre-installed as data on the target computer, or may be distributed as data from a program source to the target computer. The program source may be a program distribution server on a communication network, or a non-transitory computer-readable storage medium (e.g., a memory card). The program may be composed of multiple modules. The computer system may be composed of multiple devices. The computer system may be composed of a client-server system, a cloud computing system, an IoT system, etc. Various data and information may be composed of structures such as, for example, tables and lists, but are not limited to these. Expressions such as identification information, identifiers, IDs, names, and numbers are interchangeable.

[0016] <First Embodiment> A distance measurement device and method according to the first embodiment will be described with reference to Figs. 1 to 12. The distance measurement device 1 according to the first embodiment shown in Fig. 1 has a characteristic-matched image generation unit 103 in a processing device 102, and performs processing to match the characteristics of two images captured by an imaging device 101 from two viewpoints using the characteristic-matched image generation unit 103. In other words, the characteristic-matched image generation unit 103 corrects differences in characteristics between the two images through image processing. A distance measurement unit 105 uses the two images after characteristic matching to calculate the distance based on a calculation of the parallax.

[0017] In the first embodiment, the concept of effective pixel count (also represented by the symbol E) is used as one of the characteristics. The effective pixel count E is an index that indicates how many pixels of information a panoramic image, which is a plurality of images used in distance calculation, holds relative to a panoramic image, which is a plurality of original images directly obtained from the image capture device 101. In other words, the effective pixel count E is an amount that indicates to what extent the panoramic image holds the pixel information, amount of information, etc. of the original image. In other words, the effective pixel count E is the number of pixels used in distance calculation in the pixel count distribution of images of the same image.

[0018] [Distance measuring device (1)] 1 shows an example of the configuration of a distance measurement device 1 according to the first embodiment, and in particular shows an example of the functional block configuration of a processing device 102. The distance measurement device 1 includes an imaging device 101 which is an imaging unit, and a processing device 102 which is a signal processing unit. The imaging device 101 includes an optical system and an image sensor 111 shown in FIG. 2, which will be described later.

[0019] [Imaging device (1)] 2 shows an example of the configuration of the optical system, image sensor 111, and the like, as an example of the configuration of the imaging device 101 of the distance measurement device 1 of the first embodiment. The optical system in the first embodiment is composed of components other than the image sensor 111, and has an upper hyperbolic mirror 108, an outer periphery hyperbolic mirror 112, an inner periphery hyperbolic mirror 113, and a lens 110 which is an imaging optical system. The outer periphery hyperbolic mirror 112 and the inner periphery hyperbolic mirror 113 configure a lower hyperbolic mirror 109. The image sensor 111, the lens 110 which is an imaging optical system, the upper hyperbolic mirror 108, the outer periphery hyperbolic mirror 112, and the inner periphery hyperbolic mirror 113 are arranged on the same axis 190 which is an axis of rotational symmetry (an axis arranged in the Z direction in this example).

[0020] As shown in the figure, the optical system in the first embodiment is roughly divided into a first section disposed on the upper side in the Z direction and a second section disposed on the lower side in the Z direction and spaced apart from the first section. Each of the first and second sections is configured using a hyperbolic mirror. The first section has an upper hyperbolic mirror 108 and a lens 110 disposed approximately concentrically. The second section has an outer hyperbolic mirror 112 and an inner hyperbolic mirror 113 disposed approximately concentrically.

[0021] The required number of image sensors 111 are used depending on the configuration of the optical system. In the first embodiment, the imaging device 101 has one image sensor 111. Light 116 arriving from the subject 115 includes light 118 and light 120, and these lights reach the image sensor 111 through the optical system. The image sensor 111 receives the arriving light, converts it into an image signal, and outputs it to the processing device 102. The image sensor 111 and the processing device 102 in FIG. 1 are connected by a signal line. Note that the subject 115 is illustrated in a simplified schematic diagram, for example, as an arrow image. As a specific example, the arrowhead of the arrow image may correspond to a person's head, and the base of the arrow may correspond to a person's feet.

[0022] Here, the imaging device 101 outputs a plurality of image signals obtained by capturing an image of the subject 115 from different viewpoints according to the configuration of the optical system. In the first embodiment, the distance measurement device 1 of FIG. 1 acquires an image signal g1 corresponding to two images (also referred to as a first image and a second image) captured by the imaging device 101 from two different viewpoints (also referred to as a first viewpoint and a second viewpoint) of the subject 115 as stereo photography. One of the two viewpoints, the first viewpoint (also referred to as an upper viewpoint), corresponds to the viewpoint at which the light 118 in FIG. 2 is imaged, and the other, the second viewpoint (also referred to as a lower viewpoint), corresponds to the viewpoint at which the light 120 in FIG. 2 is imaged. That is, the imaging device 101 of the first embodiment can obtain two images, particularly an omnidirectional image, of the subject 115 from two viewpoints as a single image via a single common image sensor 111.

[0023] In other words, the imaging device 101 in the first embodiment has, as an imaging optical system, a first optical system that projects a first image corresponding to a first image onto an image sensor, and a second optical system that projects a second image corresponding to a second image onto the image sensor. The imaging optical system has multiple hyperbolic mirrors as elements that constitute the first optical system and the second optical system.

[0024] [Distance measuring device (2)] Returning to Figure 1, the processing device 102 is a device in which a signal processing unit is implemented, and is composed of, for example, a computer or a semiconductor integrated circuit, and is equipped with a processor, memory, a communication interface, etc. The implementation details of the processing device 102 are not limited, and it may be realized mainly by software program processing, or mainly by a dedicated circuit, or may be realized by a combination of multiple devices.

[0025] The processing device 102 has a characteristic-matched image generating unit 103, an image processing unit 104, a distance measuring unit 105, and an output interface 106 as functional blocks realized by circuits and program processing.

[0026] The processing device 102 processes two images in the image signal g1 input from the image sensor 111 of the imaging device 101 to calculate distance information g4. The processing device 102 outputs the calculated distance information g4 and the like as output data g5 from the output interface 106 to an external control device 107. The external control device 107 is any system or device that uses the distance measurement device 1. As an example, the control device 107 may be an electronic control unit (ECU) of an automobile or a computer such as a user's PC.

[0027] Each element of the processing device 102 will now be described. The characteristic matching image generation unit 103 performs characteristic matching processing on the input image signal g1, and outputs an image signal g2 including a characteristic matching image generated as a result of this processing to the image processing unit 104. The characteristic matching processing here is, for example, processing for matching the effective pixel count E between images from a plurality of different viewpoints. The effective pixel count E will be described later.

[0028] The image processing unit 104 performs image processing on the image signal g2 generated by the characteristic-matched image generation unit 103. The image processing here refers to, for example, a general geometric transformation, such as transforming the coordinate system of the two images of the image signal g2, such as a one-point perspective coordinate system, into another coordinate system, such as a cylindrical coordinate system. In the first embodiment, the image before the geometric transformation is an image captured in all directions of 360 degrees around the axis 190 in FIG. 2, and is a panoramic image of concentric rings as shown in FIG. 3, which will be described later. The one-point perspective coordinate system before the transformation is a coordinate system having parameters in the radial direction R and the circumferential direction C in the panoramic image as shown in FIG. 3, etc. The image after the transformation is a rectangular panoramic image as shown in FIG. 4, FIG. 11, etc., which will be described later. The cylindrical coordinate system after the transformation is a coordinate system having parameters in the vertical and horizontal directions in the panoramic image as shown in FIG. 4, etc.

[0029] The distance measurement unit 105 performs distance calculations, for example, using commonly known stereo processing, on the image signal g3 after the geometric transformation, i.e., on two panoramic images with matched characteristics, and obtains distance information as a result of the calculation. Stereo processing is a process for calculating distance information from parallax information between multiple images acquired from multiple different viewpoints. The distance measurement unit 105 outputs an image signal g4 to the output interface 106, which includes the distance information calculated by the stereo processing and, if necessary, the image signal g3 (i.e., the panoramic image) processed by the image processing unit 104. Note that the distance measurement unit 105 or another functional block may generate a distance image in an image format as the distance information. A distance image is an image in which the distance to the subject is expressed for each pixel using color, brightness, etc.

[0030] The output interface 106 outputs output data g5 including image signals g4 such as distance information output from the distance measurement unit 105 to an external control device 107. The output interface 106 is a part in which a communication interface with the control device 107 and the like are implemented.

[0031] The external control device 107 includes a processor, a memory, a communication interface, etc. The control device 107 receives and acquires the output data g5 from the distance measurement device 1 and uses it as desired. The control device 107 (for example, an ECU) may control the processing device 102 and the imaging device 101 of the distance measurement device 1. The control device 107 controls a control target such as an automobile using, for example, the distance information of the output data g5. Examples of control include automatic driving control and alert output based on the distance to other vehicles or people around the host vehicle.

[0032] [Imaging device (2)] Next, the configuration of the imaging device 101 in Fig. 2 will be described in detail. The imaging device 101 is configured to have an upper hyperbolic mirror 108, a lower hyperbolic mirror 109, an imaging optical system, specifically a lens 110, and an image sensor 111. Furthermore, the lower hyperbolic mirror 109 is configured to have an outer peripheral hyperbolic mirror 112 and an inner peripheral hyperbolic mirror 113. These hyperbolic mirrors are arranged concentrically on the same axis 190.

[0033] Upper hyperbolic mirror 108 has an opening 114 near the top of the hyperbolic surface near axis 190. A lens 110, which is an imaging optical system, is disposed in opening 114. The imaging optical system is composed of, for example, a single lens or a lens combination consisting of multiple lenses. An image sensor 111 is disposed above lens 114, which is an imaging optical system, in the space defined by upper hyperbolic mirror 108.

[0034] In lower hyperbolic mirror 109, inner circumference hyperbolic mirror 113 is arranged near axis 190, and outer circumference hyperbolic mirror 112 is arranged concentrically around inner circumference hyperbolic mirror 113. In other words, inner circumference hyperbolic mirror 113 is arranged on the top of outer circumference hyperbolic mirror 112.

[0035] In other words, the upper hyperbolic mirror 108 is a first reflecting device that reflects a first image from a first viewpoint. The outer hyperbolic mirror 112 is a second reflecting device that reflects a second image from a second viewpoint. The inner hyperbolic mirror 113 is a third reflecting device that further reflects the first image reflected from the first reflecting device. The lens 110, which is an imaging optical system, receives both the first image reflected from the third reflecting device via the first reflecting device and the second image reflected from the second reflecting device and forms an image. The image sensor 110 then captures the image formed by the lens 110 as a single image (see FIG. 3, described below) that includes two images from two viewpoints.

[0036] [Hyperbolic Mirror] Here, the properties of each of the hyperbolic mirrors, the upper hyperbolic mirror 108, the outer hyperbolic mirror 112, and the inner hyperbolic mirror 113, will be described. The hyperbolic surfaces of the upper hyperbolic mirror 108, the outer hyperbolic mirror 112, and the inner hyperbolic mirror 113 correspond to quadratic surfaces expressed by the following equation 1, where the conic constant κ is less than -1. Here, in equation 1, c is the curvature on the optical axis (in other words, the axial curvature), r is the radial distance from the optical axis, and z(r) is the amount of sag of the surface in the optical axis direction, with the vertex on the optical axis as the origin. The optical axis is axis 190 in FIG. 2.

[0037] TIFF0007756008000001.tif17170

[0038] A hyperboloid generally has two foci. The coordinates of these foci are expressed by the following equation 2, with the vertex of the surface as the reference. f represents the focus. In the ± (plus / minus sign) in equation 2, when the sign is +, it represents the coordinates of the focus inside the hyperboloid, and when the sign is -, it represents the coordinates of the focus outside the hyperboloid. Hereinafter, the focus inside the hyperboloid will be referred to as the first focus (or inner focus), and the focus outside the hyperboloid will be referred to as the second focus (or outer focus).

[0039] TIFF0007756008000002.tif15170

[0040] A hyperbolic mirror has the property of reflecting light rays heading toward the first focus into light rays that converge at the second focus, and conversely, it has the property of reflecting light rays that emerge from the second focus as if they had emerged from the first focus.

[0041] 2, subject 115 is assumed to be arranged in a space around the entire periphery of image capture device 101, or within a range of a portion of the entire periphery. Note that in FIG. 2, the coordinate system of the space in which image capture device 101 is arranged is also shown as (R, C, Z) or (X, Y, Z). R is the radial direction, C is the circumferential direction, and Z is the vertical direction. It is assumed that a certain subject 115 is arranged away from image capture device 101 in the radial direction R or X direction, within a certain angular range in the circumferential direction C.

[0042] Of the light 116 traveling from the subject 115 toward the imaging device 101, there is light 118 (also referred to as first light) traveling toward a first focal point 117 (in other words, an upper viewpoint or a first viewpoint), which is a focal point inside the upper hyperbolic mirror 108, and light 120 (also referred to as second light) traveling toward a first focal point 119 (in other words, a lower viewpoint or a second viewpoint), which is inside the lower hyperbolic mirror 109, particularly the outer peripheral hyperbolic mirror 112.

[0043] Light 118 traveling toward the first focal point 117 of the upper hyperbolic mirror 108 is reflected as light (e.g., light ray a1, in other words, reflected light a1) traveling toward the second focal point, which is located outside the two focal points of the upper hyperbolic mirror 108, due to the nature of the hyperbolic mirror.

[0044] 2, the first focus, which is the inner of the two focuses of inner hyperbolic mirror 113, is positioned near the second focus, which is the outer focus of upper hyperbolic mirror 108. The two focuses positioned so as to approximately coincide within a nearby range in this manner correspond to focus 119. In this configuration, light directed toward focus 119, which corresponds to the second focus of upper hyperbolic mirror 108, is emitted by upper hyperbolic mirror 108 as light directed toward focus 119, which is the first focus on the inside of inner hyperbolic mirror 113.

[0045] The light heading toward the inner circumference side hyperbolic mirror 113 is reflected by the inner circumference side hyperbolic mirror 113 and becomes light (for example, light ray a3, in other words, reflected light a3) heading toward the lens 110, which is an imaging optical system.

[0046] On the other hand, light 120 (second light) of light 116 from subject 115 traveling toward first focal point 119 of outer-circumferential hyperbolic mirror 112 is reflected as light (for example, light ray a2, in other words, reflected light a2) traveling toward a second focal point outside outer-circumferential hyperbolic mirror 112 due to the properties of the hyperbolic mirror. The reflected light becomes light traveling toward lens 110, which is an imaging optical system.

[0047] 2 has these reflected lights, that is, reflected light (e.g., light ray a3) from inner circumference-side hyperbolic mirror 113 via reflection on upper hyperbolic mirror 108, which corresponds to first light 118 from subject 115, and reflected light (e.g., light ray a2) from outer circumference-side hyperbolic mirror 112, which corresponds to second light 120 from subject 115. This optical system has a configuration in which lens 110, which is an imaging optical system, is arranged so as to form an image of these reflected lights on image sensor 111.

[0048] With this configuration, the imaging device 101 can obtain two images of the subject 115 viewed from two viewpoints, a focal point 117 corresponding to an upper viewpoint and a focal point 119 corresponding to a lower viewpoint, using a single image sensor 111. Similarly, when the subject 115 is disposed all around the imaging device 101, the imaging device 101 can obtain two omnidirectional images of the subject 115 viewed from two viewpoints using a single image sensor 111.

[0049] [Stereo Image] 3 shows an example of an image 201 obtained when an object 115 is photographed by the imaging device 101 of FIG. 2, in other words, a stereo image. The image 201 includes an upper viewpoint region 202 in which the surroundings are photographed from a focal point 117 corresponding to the upper viewpoint in FIG. 2, and a lower viewpoint region 203 in which the surroundings are photographed from a focal point 119 corresponding to the lower viewpoint. The entire image 201 is illustrated as a rectangle, and two ring-shaped regions, the upper viewpoint region 202 and the lower viewpoint region 203, are included within the rectangular region of the image 201. The portion of the image 201 other than the two ring-shaped regions may be a simple background pixel region, or may be configured so that there are no pixels in this portion, i.e., a circular image is obtained.

[0050] The upper viewpoint region 202 and the lower viewpoint region 203 are concentric ring-shaped image regions. In other words, the upper viewpoint region 202 is the first image, or inner ring image region, and the lower viewpoint region 203 is the second image, or outer ring image region. A circle near the center 200 of the upper viewpoint region 202 is missing as an area that cannot be captured. The example of FIG. 3 shows a case where the upper viewpoint region 202 and the lower viewpoint region 203 are entirely included in the image 201. However, this is not limiting. Because the image capture area varies depending on the size and position of the imaging surface of the image sensor 111 and the characteristics of the imaging optical system 110, the upper viewpoint region 202 and the lower viewpoint region 203 may not be ring-shaped around the entire 360-degree circumference, but may be partially missing. In other words, the imaging device 101 may be configured to capture only an area within a predetermined angle or position in the circumferential direction C.

[0051] In a stereo camera, the distance to the subject is calculated using two or more images of the same subject captured from two or more different viewpoints. In this case, as mentioned above, if there is a difference in the effective pixel count E between the multiple images, the performance of the distance information may be degraded. Therefore, it is preferable to make the effective pixel count E the same between the multiple images that are the subject of distance measurement. The above-mentioned issues will be explained in more detail below.

[0052] The light imaged in the upper viewpoint region 202 in FIG. 3 is light that has been reflected twice, once by the upper hyperbolic mirror 108 and once by the inner hyperbolic mirror 113 in FIG. 2. In contrast, the light imaged in the lower viewpoint region 203 is light that has been reflected once by the outer hyperbolic mirror 112. Therefore, the images formed in the upper viewpoint region 202 and the lower viewpoint region 203 are inverted in the radial direction R depending on whether the number of reflections is even or odd. An image in the upper viewpoint region 202, for example, image (or region) 204, depicts subject 115 in the radial direction R from the inner periphery to the outer periphery. For example, a person's head, indicated by an arrowhead, is located on the outer periphery. In contrast, an image in the lower viewpoint region 203, for example, image (or region) 205, is inverted, depicting subject 115 in the radial direction R from the outer periphery to the inner periphery. For example, the person's head indicated by the arrowhead is located on the inner circumference.

[0053] Furthermore, since the upper viewpoint area 202 and the lower viewpoint area 203 are ring-shaped areas, the number of pixels in the circumferential direction C increases as the distance from the center 200 of the ring in the radial direction R increases.

[0054] As a result, in an image 204 in the upper viewpoint region 202 and an image 205 in the lower viewpoint region 203, which both show the same subject 115, the number of pixels in the circumferential direction C at corresponding locations in the images 204 and 205, such as location 206 and location 207, which show the same part of the subject 115, is different.

[0055] For example, (A) and (B) in FIG. 3 show enlarged views of the base of an arrow image, which is the same corresponding portion of subject 115, in image 204 in upper viewpoint region 202 and image 205 in lower viewpoint region 203. (A) shows an enlarged view of portion 206 in image 204, and (B) shows an enlarged view of portion 207 in image 205. In these images, each square represents a pixel. The shaded gray areas are pixels that show the base of the arrow image. Thus, the number of pixels differs between the corresponding portions of subject 115 in upper viewpoint region 202 and lower viewpoint region 203. In the image of portion 206 in (A), the width of the base is one pixel, while in the image of portion 207 in (B), the width of the base is five pixels.

[0056] [Number of pixels] 4 is a diagram showing an expanded view of the ring-shaped upper viewpoint region 202 and lower viewpoint region 203 in FIG. 3, and is a schematic diagram showing the change in the number of pixels in the circumferential direction (circumferential direction C) depending on the distance from the center 200 of the ring in the radial direction R. The number of pixels in the circumferential direction C is also represented by N. In FIG. 4, the distance from the center 200 in the radial direction R is also represented in the vertical direction (Y direction), and the pixels in the circumferential direction C are also represented in the horizontal direction (X direction).

[0057] The expanded upper viewpoint region 301 and lower viewpoint region 302 are obtained by cutting open the upper viewpoint region 202 and lower viewpoint region 203 in FIG. 3 along line 208 in FIG. 3 (one radius from the center 200 to the outermost periphery) and stretching the circumferential direction C horizontally (the horizontal direction, X direction in FIG. 4). In the expanded regions, the circumferential direction C is a straight horizontal line. In FIG. 4, the upper viewpoint region 301 and lower viewpoint region 302 are each trapezoidal in shape.

[0058] 4 represents the number of pixels N in the circumferential direction. As the distance from the center 200 in the radial direction R increases, the number of pixels N in the circumferential direction increases, for example, from NL (representing Low) to NM (representing Middle) to NH (representing High), and the horizontal length also increases accordingly. The number of pixels NL is the number of pixels in the circumferential direction C at a distance r1 from the bottom edge of the upper viewpoint region 301 (the inner circumference of the ring in the upper viewpoint region 202). The number of pixels NM is the number of pixels in the circumferential direction C at a distance r2 from the top edge of the upper viewpoint region 301 (the outer circumference of the ring in the upper viewpoint region 202), in other words, the number of pixels in the circumferential direction C at a distance r2 from the bottom edge of the lower viewpoint region 302 (the inner circumference of the ring in the lower viewpoint region 203). The number of pixels NH is the number of pixels in the circumferential direction C at a distance r3 from the upper side of the lower viewpoint region 302 (the outer periphery of the ring in the lower viewpoint region 203). The numbers of pixels NL, NM, and NH are larger in this order, and NL <NM<NHである。

[0059] As a result, the difference in the number of pixels N at corresponding locations in upper viewpoint region 301 and lower viewpoint region 302 in Figure 4 can be grasped from the difference in length in the horizontal direction (X direction). This difference in the number of pixels N at corresponding locations reduces the performance of distance measurement in distance measurement unit 105 in the subsequent stage in Figure 1. This issue will be described in detail below.

[0060] [Comparative examples and issues] FIG. 5 shows how, in a distance measurement device of the comparative example, image processing is performed by image processing unit 104 of FIG. 1 on upper viewpoint region 301 and lower viewpoint region 302, which have different numbers of pixels N in the circumferential direction in FIG. 4, to generate images with the same number of pixels N in the circumferential direction. This image processing is a geometric transformation that generates a panoramic image with the same number of pixels N in the circumferential direction. Pre-processing image 400 of FIG. 5(A) is assumed to be the same as post-development image 400 of FIG. 4. Image processing unit 104 and the like here are components of the comparative example. In order for distance measurement unit 105 to calculate distance information, images of the upper viewpoint region and lower viewpoint region with the same number of pixels in the circumferential direction are required.

[0061] 5B, in this comparative example, an upper panoramic image 401 and a lower panoramic image 402 are generated as images in which the upper viewpoint region and the lower viewpoint region are converted into rectangles and the number of pixels N in the circumferential direction is the same, by image processing in the image processing unit 104. In this case, the closer the effective pixel numbers E of corresponding portions of the upper panoramic image 401 and the lower panoramic image 402 are, the higher the performance of distance measurement in the distance measurement unit 105 becomes.

[0062] Here, the effective pixel count E is an index representing how many pixels of information from the original image are held in the panoramic image. For example, the effective pixel count E in the circumferential direction C at a certain position in the radial direction R of the upper panoramic image 401 represents how many pixels of information are held out of the number of pixels in the circumferential direction C at the corresponding position in the radial direction R of the upper viewpoint region 301.

[0063] Note that inversion of the image in the radial direction R in the unfolded image (image before processing) 400, for example, inversion between images 411 and 412 relating to the same image, is corrected when generating a panoramic image through image processing. This correction involves, for example, converting the lower viewpoint region 302 into a rectangle and inverting it in the radial direction R to produce the lower panoramic image 402. Therefore, in FIG. 5B, the upper panoramic image 401 and the lower panoramic image 402 after image processing have the same orientation of the images shown in, for example, images 413 and 414.

[0064] There are several methods for generating a panoramic image with the same number of pixels N in the circumferential direction as in FIG. 5(B).

[0065] [Comparison example: Interpolation] For example, Fig. 6 shows a method using interpolation as one method for generating a panoramic image in a distance measurement device of a comparative example. This method, as shown in Fig. 6(A), is a method of performing interpolation processing so that the pixel count distribution of the trapezoids in upper viewpoint region 301 and lower viewpoint region 302 becomes the pixel count distribution of rectangles 501 and 502. This method performs interpolation processing so that the pixel count distribution matches the largest pixel count in the radial direction R (the pixel count N at distance r3 in this example) in lower viewpoint region 302 (in other words, trapezoidal image) which has a larger pixel count N in the circumferential direction. In other words, this method interpolates the pixel count N in the circumferential direction C at each position in the radial direction R so that the two trapezoidal images are aligned with the region with the largest pixel count N in the circumferential direction R.

[0066] The image of rectangle 501 is an image after image processing of interpolation from the upper viewpoint region 301. The image of rectangle 502 is an image after image processing of interpolation from the lower viewpoint region 302. Before being flipped upside down, the image of rectangle 501 corresponding to the upper viewpoint region 301 becomes an interpolated panoramic image 503 corresponding to the upper viewpoint region 301 as shown in FIG. 6B, and the image of rectangle 502 corresponding to the lower viewpoint region 302 becomes an interpolated panoramic image 504 corresponding to the lower viewpoint region 302. Furthermore, these images are flipped upside down to align the upside-down orientation of the images, resulting in interpolated panoramic image 503 and panoramic image 504 as shown in FIG. 6C.

[0067] Interpolation is a process such as the following. For example, in the trapezoid of the upper viewpoint area 301 in (A) of FIG. 6, the number of pixels N in the circumferential direction at a position with a distance r1 in the radial direction R corresponding to the short side of the trapezoid is NL. In the trapezoid of the larger lower viewpoint area 302, the number of pixels N in the circumferential direction at a position with a distance r3 in the radial direction R corresponding to the long side of the trapezoid is NH. Interpolation processing is performed so that the number of pixels NL of the short side of the trapezoid in the upper viewpoint area 301 matches the number of pixels NH of the long side of the trapezoid in the lower viewpoint area 302. In this interpolation processing, new pixels are evenly inserted between the original pixels in a pixel group of one line of the original number of pixels NL. A new pixel value calculated from the pixel values of the surrounding original pixels is stored in each new pixel.

[0068] When generating a panoramic image by using such an interior, the effective number of pixels E of the upper panoramic image 503 in (C) of FIG. 6 and the effective number of pixels E of the lower panoramic image 504 have different regions. For example, in the two images of the upper panoramic image 503 and the lower panoramic image 504, in the region with the number of pixels N as the pixel number NM in the upper part of the image in the radial direction R (in other words, the upper part of the images 511 and 512 shown), the effective number of pixels E is aligned as the effective number of pixels EM. On the other hand, in the two images, in the regions with the number of pixels N as the pixel numbers NL and NM in the lower part of the image in the radial direction R (shown by the broken line frames), the effective number of pixels E becomes the effective number of pixels EL and the effective number of pixels EH respectively, and they are different. Since NL < NM as the number of pixels, EL < EH as the effective number of pixels. In the two images of the upper panoramic image 503 and the lower panoramic image 504, the relationship of the effective number of pixels E is EL < EM < EH.

[0069] In the comparative example, in the distance measurement unit 105 in the latter stage of FIG. 1, after detecting the corresponding points (for example, the corresponding locations of the images 513 and 514) between the upper panoramic image 503 and the lower panoramic image 504, the parallax is calculated at those corresponding points. However, at that time, the detection accuracy of the corresponding points decreases between the regions with different effective numbers of pixels E, and as a result, the performance of calculating the parallax and thus the distance measurement decreases.

[0070] [Comparison example: Downsampling] Figure 7 shows a method using downsampling as another method for generating a panoramic image with a uniform number of pixels. As shown in Figure 7(A), this method downsamples the pixel count distribution of the trapezoids of upper viewpoint region 301 and lower viewpoint region 302 so that it becomes the pixel count distribution of rectangles 601 and 602. In other words, this method downsamples the pixel count N in the circumferential direction C at each position in the radial direction R so that the pixel count N in the circumferential direction R in the two trapezoidal images becomes the smallest.

[0071] The image of rectangle 601 is an image after image processing of downsampling from upper viewpoint region 301. The image of rectangle 602 is an image after image processing of downsampling from lower viewpoint region 302. As shown in FIG. 7B, the image of rectangle 601 corresponding to upper viewpoint region 301 becomes downsampled panoramic images 603 and 604 after inversion. Panoramic images 603 and 604 shown in (B) are shown stretched relative to the side length of the pixel count NL in (A).

[0072] Downsampling is, for example, the following process. For example, in the trapezoid of upper viewpoint region 301 in FIG. 7A, the number of pixels N in the circumferential direction at a position at a distance r1 in the radial direction R corresponding to the short side of the trapezoid is NL. In the larger trapezoid of lower viewpoint region 302, the number of pixels N in the circumferential direction at a position at a distance r3 in the radial direction R corresponding to the long side of the trapezoid is NH. Downsampling is performed so that the number of pixels NH on the long side of the trapezoid in lower viewpoint region 302 matches the number of pixels NL on the short side of the trapezoid in upper viewpoint region 301. In this process, pixels to be thinned out are evenly spaced between the original pixels in one line of pixels with the original number of pixels NH. Each new pixel remaining after thinning stores a new pixel value calculated from the pixel value of the new pixel and the pixel values ​​of the surrounding pixels to be thinned out.

[0073] Another example of downsampling processing is to thin out pixels in a group of pixels on one line of the original pixel count NH, either evenly or in a certain area in the circumferential direction C, and the pixel values ​​of the thinned out pixels are deleted without being reflected in the new pixels.

[0074] 7B, when a panoramic image is generated using such a method using downsampling, the effective pixel count E of the upper panoramic image 603 and the effective pixel count E of the lower panoramic image 604 are all the effective pixel count EL corresponding to the pixel count NL at each position in the radial direction R. That is, the effective pixel count E is the same for the entire upper panoramic image 603 and the lower panoramic image 604.

[0075] Furthermore, (C) of FIG. 7 shows an image obtained by stretching the two images of (B) so that the number of pixels NL in the circumferential direction C matches the number of pixels NH on the original longest side.

[0076] Therefore, with this method, there is no degradation in distance measurement performance due to differences in the effective pixel count E. However, because this method involves downsampling, the number of pixels from which distance can be measured decreases across the two images. For example, after downsampling, the number of pixels NH on the long side of the trapezoid in the lower viewpoint area 302 from which distance can be measured decreases, as in the effective pixel count EL. For this reason, it is desirable to maintain the effective pixel count E in the panoramic image as close as possible to the pixel count N of the original image.

[0077] When generating a rectangular panoramic image such as that shown in Fig. 5 by image processing for distance calculation from an original image such as the ring-shaped image shown in Fig. 3 or the trapezoidal image shown in Fig. 4, the above-mentioned problem can be summarized into two viewpoints. The first viewpoint is to make the effective pixel count E at each position in the radial direction R as uniform as possible for the two images of upper viewpoint region 301 and lower viewpoint region 302 after image processing, in other words, after geometric transformation, in order to reduce the difference in effective pixel count E as much as possible. The second viewpoint is to maintain the number of pixels that allows distance measurement as much as possible for these two images after image processing, in other words, after geometric transformation, in comparison with the original image.

[0078] In order to solve the above problems, the distance measurement device 1 of the first embodiment has the configuration described below. In the distance measurement device 1, the characteristic-matched image generation unit 103 of FIG. 1 performs a characteristic-matched image generation process when generating a panoramic image from an original image so as to make it possible to align the effective pixel count E at corresponding locations of the image while maintaining the effective pixel count E (in other words, the number of pixels from which distance calculation is possible) to the maximum extent possible. That is, in the first embodiment, a characteristic-matched image is generated so as to satisfy the above two viewpoints in a balanced manner. The process performed by the characteristic-matched image generation unit 103 and its effects will be described below.

[0079] In the first embodiment, the characteristic-matched image generation unit 103 generates a characteristic-matched image in which the effective pixel number E is matched by projecting the pixels of the lower viewpoint region 203 in the image 201 of FIG. 3 based on the input image signal g1 onto the corresponding pixel positions as an image of the upper viewpoint region 202 (FIG. 8).

[0080] [Feature-Consistent Image Generation and Projection] FIG. 8 is an explanatory diagram schematically illustrating how the characteristic-matching image generation unit 103 projects pixels in the lower viewpoint region 203 of the image 201 onto corresponding pixel positions in the upper viewpoint region 202 in the first embodiment. The characteristic-matching image generation unit 103 performs a projection 800 (also indicated by an arrow or "h(r)") of a pixel of interest 701 in the lower viewpoint region 203 onto a corresponding pixel position 702 in the upper viewpoint region 202 that corresponds to the pixel of interest 701. The pixel of interest 701 is a pixel located at a distance r in the radial direction R. The corresponding pixel position 702 is located at a distance r' in the radial direction R. The characteristic-matching image generation unit 103 performs an operation similar to this projection 800 for all pixels in the lower viewpoint region 203. This completes the projection of the lower viewpoint region 203. Note that the projection in FIG. 8 is basically a concept that follows mathematical projection.

[0081] Next, a method for determining corresponding pixel position 702 for pixel of interest 701 in embodiment 1 will be described. As shown in Fig. 8, the distance from center 703 of the ring of upper viewpoint region 202 and lower viewpoint region 203 to pixel of interest 701 in radial direction R is set to r, and the distance from center 703 to corresponding pixel position 702 is set to r'. In this case, a function h(r) that satisfies the following equation 3 is referred to as a projection function here.

[0082] TIFF0007756008000003.tif7170

[0083] Determining the corresponding pixel position 702 for the pixel of interest 701 corresponds to determining this projection function h(r). This projection function can be determined using, for example, the image height characteristics of the optical system.

[0084] [Image height characteristics of optical system] 9 is a cross-sectional view of the image capturing device 101 in FIG. 2 taken along a plane including the upper focal point 117 and the lower focal point 119, and is an explanatory diagram showing the elevation and depression angles when the image capturing device 101 is used as a reference. The elevation and depression angles 802 (magnitude: θ) represent signed angles in the up and down directions from a horizontal line 801 (corresponding to the X direction, for example) when the image capturing device 101 is used as a reference. The image height characteristic is a characteristic that indicates the position in the image 201 in FIG. 3 at which the subject 115, located in the direction of the elevation and depression angle θ from the image capturing device 101, is captured.

[0085] Assume that the image height characteristics of the upper viewpoint region 202 are expressed as r' = f(θ), and the image height characteristics of the lower viewpoint region 203 are expressed as r = g(θ). In this case, an image of a subject 115 that is far from the image capture device 101 and at θ = θ0 is formed at a position with an image height of r' = f(θ0) and r = g(θ0). Therefore, based on the distance r from the center 703 to the pixel of interest 701, the image height characteristics f(θ) of the upper viewpoint region 202, and the image height characteristics g(θ) of the lower viewpoint region 203, the distance r' from the center 703 to the corresponding pixel position 702 can be calculated using the following equation 4.

[0086] TIFF0007756008000004.tif9170

[0087] Therefore, the projection function h(r) is given by the following equation 5.

[0088] TIFF0007756008000005.tif8170

[0089] Referring to FIG. 8, a specific example of a projection function using image height characteristics for imaging device 101 in Embodiment 1 is shown. Let ra be the radius at the position closest to center 703 in upper viewpoint region 202, and rb be the radius at the position farthest from center 703 in lower viewpoint region 203. Let rc be the radius at the boundary between upper viewpoint region 202 and lower viewpoint region 203. Here, the image height characteristics of upper viewpoint region 202 and lower viewpoint region 203 are assumed to be the same shape but inverted. This configuration for image height characteristics corresponds to the optical system ( FIG. 2 ) in Embodiment 1 where upper hyperbolic mirror 108 and lower hyperbolic mirror 109 are configured using the same hyperbolic mirror. In this case, the image height characteristics of upper viewpoint region 202 are r=f(θ), and the image height characteristics of lower viewpoint region 203 are expressed by the following Equation 6.

[0090] TIFF0007756008000006.tif7170

[0091] Therefore, the projection function h(r) can be calculated as the following equation 7 using equation 5.

[0092] TIFF0007756008000007.tif7170

[0093] [Property Matching and Projection] Next, the effect of the above-mentioned characteristic matching process will be described with reference to FIGS. 3, 8, 10, and 11. FIG. 10 is an explanatory diagram showing a change in the number of pixels due to the characteristic matching process (particularly projection) by the characteristic matching processing unit 103 in the distance measurement device 1 of the first embodiment. The target image is assumed to be the same as the image 201 in FIG. 3 and the two images (upper viewpoint region 301 and lower viewpoint region 302) in FIG. 4. In FIG. 10, the characteristic matching processing unit 103 performs characteristic matching process to project pixels in the lower viewpoint region 302 onto corresponding pixel positions in the upper viewpoint region 301. This characteristic matching process corresponds to downsampling the pixel count distribution of the lower viewpoint region 302 to match the pixel count distribution of the upper viewpoint region 302, as in the form of a trapezoid 901.

[0094] As a comparative example to the first embodiment, there is a method in which the trapezoidal upper viewpoint region 301 and lower viewpoint region 302 of FIG. 3 are actually created by cutting them apart along dashed line 208, and then the trapezoidal lower viewpoint region 302 is downsampled to become the smaller trapezoid 901 of FIG. 10. This method also enables processing similar to the projection processing shown in FIG. 8. However, this comparative example method requires two processes: expanding a ring region such as the image of FIG. 3 into a trapezoid, and downsampling the trapezoidal region. The processing in this comparative example requires one more process than the projection processing shown in FIG. 8. Therefore, the projection processing shown in FIG. 8 has the advantage of reducing the accumulated error caused by each process.

[0095] [Projective and panoramic image generation] Fig. 11 is a schematic diagram showing how a panoramic image is generated from images after characteristic matching processing by projection. Fig. 11(A) shows trapezoidal image 301 in upper viewpoint region 301 and trapezoidal image 901 in lower viewpoint region 302 as characteristic-matched images after characteristic matching processing, and rectangular image 1001 and rectangular image 1002 as images after interpolation processing from these two images. Fig. 11(B) shows panoramic image 1003 in upper viewpoint region 301 and panoramic image 1004 in lower viewpoint region 302 as rectangular panoramic images after panoramic image generation by interpolation processing from the rectangular images in Fig. 11(A) and after inversion.

[0096] The distance measurement device 1 performs interpolation conversion using the image 301 and image 901 after the above-described characteristic matching process by projection in the image processing unit 104 of Fig. 1 to obtain rectangles 1001 and 1002 shown in Fig. 11(A). As a result, an upper panoramic image 1003 and a lower panoramic image 1004 shown in Fig. 11(B) are generated.

[0097] In the first embodiment, the image in upper viewpoint region 202 in Fig. 3 is used as is for the geometric transformation in image processing unit 104. In contrast, the projection in Fig. 8 is applied to the image in lower viewpoint region 203, and a characteristic-matched image is generated as the image after projection, such as the change from trapezoidal image 301 to image 901 in Fig. 10. The characteristic-matched image after projection is used for the geometric transformation in image processing unit 104, and a panoramic image such as that in Fig. 11 is generated.

[0098] The method of FIG. 11 differs from the comparative example interpolation method described above with reference to FIG. 6 in that the upper panoramic image 1003 and the lower panoramic image 1004 generated by this method have an effective pixel count E of EM corresponding to pixel count NM in the upper portion of the image, and an effective pixel count E of EL corresponding to pixel count NL in the lower portion of the image. That is, after conversion, the two images 1113 and 1114 representing the same image in the upper viewpoint region 301 and the lower viewpoint region 302 have the same distribution of effective pixel count E at each position in the radial direction R. The upper panoramic image 1003 and the lower panoramic image 1004 each have a distribution that changes from the effective pixel count EL to the effective pixel count EM in the radial direction R, for example, from the bottom to the top of the subject images 1113 and 1114. Image 1113 corresponds to image 1111 before conversion, and image 1114 corresponds to image 1112 before conversion. In other words, the method of the first embodiment has a smaller difference in the effective pixel count E between the two upper and lower images than the method of the comparative example.

[0099] 7, the effective pixel count E of the entire panoramic image is EL. In contrast, the effective pixel count E of the upper panoramic image 1003 and the lower panoramic image 1004 generated by the method in the first embodiment is EM, and the effective pixel count E is maintained relative to the original image. In other words, the method in the first embodiment maintains the effective pixel count E (the number of pixels from which distance can be measured) throughout the entire image to a greater extent than the method in the comparative example.

[0100] As described above, the characteristic matching process in the first embodiment maintains the effective pixel count E to the maximum extent possible when the original image is converted into a panoramic image, and also achieves a match in the effective pixel count E of corresponding portions of the subject image. That is, in the first embodiment, the panoramic image is generated so as to satisfy the above-mentioned two viewpoints in a balanced manner. Then, using this panoramic image, distance calculation can be performed with high accuracy.

[0101] [Distance measurement flow] Next, with reference to the flow of Fig. 12, an example of processing performed by the characteristic-matched image generating unit 103, image processing unit 104, and distance measuring unit 105 by the processor of the processing device 102 of Fig. 1 according to the first embodiment will be described. First, in step S101, the processing device 102 refers to a projection function that is stored in advance and corresponds to the image height characteristics of the imaging device 101. This projection function is calculated in advance based on the above-mentioned Figs. 8 to 9, and is stored in memory as data information.

[0102] Next, in step S102, the characteristic-matching image generation unit 103 uses the projection function referenced in step S101 to project the lower viewpoint region 203 in Figure 8 onto the corresponding location in the upper viewpoint region 202, thereby generating images 301 and 901 as characteristic-matching images such as those in Figures 10 and 11.

[0103] Next, in step S103, the image processing unit 104 generates a panoramic image using the characteristic-matched image (image signal g2 in FIG. 1) created in step S102.

[0104] Next, in step S104, the distance measurement unit 105 performs distance measurement using the panoramic image (image signal g3 in FIG. 1) created in step S103.

[0105] Next, in step S105, if the image for which distance measurement was performed in step S104 is one frame in the video, the processing device 102 determines whether or not to perform distance measurement on the image of the next frame. If the processing device 102 will also perform distance measurement on the next frame (S105-YES), it returns to step S102 and performs the same processing. If the processing device 102 will not perform distance measurement on the next frame or if there is no next frame (S105-NO), it ends the processing of this flow. The output interface 106 outputs the data after the above processing (output data g5 in FIG. 1).

[0106] [Distance measurement method] The distance measurement method of the first embodiment is a method having steps executed by the processor of the distance measurement device 1, particularly the processing device 102, in accordance with the functional blocks in Fig. 1 and the flow in Fig. 12. The distance measurement method of the first embodiment has the steps of capturing an image of the subject with the imaging device 101, generating a characteristic-matched image in which the characteristics (particularly the effective pixel count E) between the two images are matched based on the captured images from two viewpoints, generating two rectangular panoramic images by image processing such as geometric transformation based on the characteristic-matched image, and measuring the distance to the subject based on the two rectangular panoramic images.

[0107] [Effects (1)] As described above, according to the distance measurement device 1 and distance measurement method of the first embodiment, by projecting pixels in the lower viewpoint region 203 of Fig. 8 onto corresponding pixel positions in the upper viewpoint region 202, a characteristic-matched image (Fig. 11) is generated in which the effective pixel numbers E are matched between the images of the two upper and lower viewpoints and the number of pixels that allows distance measurement from the original image is maintained. The distance measurement device 1 of the first embodiment can improve the performance of distance measurement by performing distance measurement using this characteristic-matched image.

[0108] According to the first embodiment, a relatively simple process using projection can be used to correct the difference in the effective pixel count E between multiple captured images (or multiple image regions) used for distance measurement and calculation so as to reduce the difference. In other words, correction can be performed to match the characteristics between multiple images. For example, it is possible to maintain the effective pixel count E of an image used for distance calculation as close as possible to the effective pixel count E of the original image, in other words, to at least reduce the difference in the effective pixel count E between those images. As a result, distance calculation can be performed using the corrected image, in other words, the image after the matching of characteristics, thereby improving the performance and accuracy of distance measurement.

[0109] In the first embodiment, the main processes preceding the distance calculation by distance measurement unit 105 in Fig. 1 are only two: characteristic matching processing by characteristic matching image generation unit 103, i.e., projection processing, and geometric transformation, i.e., processing for conversion to a panoramic image, by image processing unit 104. These relatively simple processes align the characteristics of the two input images (images of upper viewpoint region 202 and lower viewpoint region 203 in Fig. 3), enabling highly accurate distance calculation without requiring complex transformation, such as complex filtering, according to the two images.

[0110] The following modification of the first embodiment is also possible. In this modification, a process of generating a trapezoidal characteristic-matched image as shown in FIG. 10 by cutting and unfolding the image of FIG. 3, a projection as shown in FIG. 8, and conversion to a panoramic image as shown in FIG. 11 are performed. However, in this modification, unlike the first embodiment, it is necessary to convert the concentric ring-shaped image region of FIG. 3 into a trapezoidal image region as shown in FIG. 10 and downsample the trapezoidal image region. Therefore, this modification requires more processing steps than the first embodiment. Compared to the modification, the first embodiment does not require such processing and only requires projection processing, so the number of processing steps is smaller and it is possible to reduce the accumulated error overall.

[0111] The projection and panoramic image generation method of the first embodiment shown in FIG. 11 can align the distribution of the effective pixel count E in the radial direction R and the circumferential direction C of the two captured images (upper viewpoint region 202 and lower viewpoint region 203) compared to the interpolation method of FIG. 6; in other words, it can reduce the difference in the effective pixel count E. Furthermore, the method of the first embodiment can maintain the number of pixels (in other words, the amount of information) from the original image to a greater extent than the downsampling method of FIG. 7. That is, the method of the first embodiment can achieve a good balance between aligning the effective pixel count E and maintaining the number of pixels from the original image.

[0112] <Embodiment 2> The distance measurement device of embodiment 2 will be described using Figure 13 onwards. The basic configuration of embodiment 2 etc. is the same as embodiment 1, and the following mainly describes the components of embodiment 2 etc. that are different from embodiment 1. The configuration of embodiment 2 differs from embodiment 1 in that, as shown in Figure 13, two fisheye cameras (camera 1201 and camera 1202) are used as imaging device 101B, and processing device 101B generates characteristic-matched images for each of the two captured images. The fisheye camera is a camera equipped with a fisheye lens, and in embodiment 2, the fisheye lens has a wide angle of view that is greater than 180 degrees.

[0113] In the first embodiment, the characteristic-matched image unit 103 in FIG. 1 generates a characteristic-matched image for two input images from two viewpoints, one above the other, by matching one image to the characteristics of the other. Specifically, image 901, a characteristic-matched image as shown in FIG. 11, is generated by the projection in FIG. 8 so that the characteristics of the image in the lower viewpoint region 302 in FIG. 4 match the characteristics of the image in the upper viewpoint region 302. In contrast, in the distance measurement device 1B in the second embodiment in FIG. 13, the characteristic-matched image generation unit 103B generates a first characteristic-matched image for two images from two viewpoints, one above the other, by the first projection so that the characteristics of one image match the characteristics of the other image, and generates a second characteristic-matched image for the other image by the second projection so that the characteristics of the other image match the characteristics of the one image. The image processing unit 104B generates two panoramic images from the two characteristic-matched images. The distance measurement unit 105B calculates the distance based on the two panoramic images.

[0114] In the distance measurement device 1B of the second embodiment, as shown in Fig. 16 (to be described later) and other figures, since a single projection from one image onto the other image is not sufficient to match the characteristics of images from two viewpoints, one above the other, two mutual projections (the first and second projections) are performed by adding a second projection from the other image onto the one image. These two projections have a corresponding relationship, and the basic concept is the same as that of Fig. 8.

[0115] In the second embodiment, the two fisheye cameras, camera 1201 and camera 1202, are, for example, cameras of the same type and characteristics. Therefore, the same projection function can be applied to the two projections. As a modified example, if the two cameras are of different types and characteristics, the two projections may be performed as two different projections that take into account the characteristics of each camera.

[0116] [Distance measuring device] 13 shows a schematic configuration of distance measurement device 1B according to the second embodiment. Distance measurement device 1B includes imaging device 101B, which is an imaging unit, and processing device 102B. Imaging device 101B has two fisheye cameras (in other words, imaging devices), camera 1201 and camera 1202. Camera 1201 is a first camera or first fisheye camera, and camera 1202 is a second camera or second fisheye camera. Camera 1201 and camera 1202 are arranged so that their optical axes are roughly coaxial.

[0117] 13, on the one hand, the first block B1 of the characteristic-matched image generating unit 103B performs first characteristic matching processing by first projection on a first image (image g11) captured by the first camera 1201, and obtains a first characteristic-matched image (image g13) as a result. On the other hand, the distance measurement device 1B also performs second characteristic matching processing by second projection on a second image (image g12) captured by the second camera 1202, and obtains a second characteristic-matched image (image g14) as a result.

[0118] [Imaging device] An example configuration of the imaging device 101B according to the second embodiment will be described with reference to FIGS. 14 and 15. FIG. 14 is a schematic diagram illustrating an example configuration of the imaging device 101B. The imaging device 101B in FIG. 13 illustrates a case where the configuration shown in FIG. 14 is applied. In FIG. 14, the imaging device 101B is disposed in a space of a coordinate system indicated by (X, Y, Z) in the figure. The Z direction is the vertical direction. The imaging device 101B is configured using two fisheye cameras, camera 1201 and camera 1202, each with a field of view of 180 degrees or more. A field of view 1204 indicated by an arc-shaped arrow indicates the field of view of the first fisheye camera 1201, and a field of view 1205 indicates the field of view of the second fisheye camera 1202. In FIG. 14, as an example, the fields of view 1204 and 1205 of the fisheye cameras 1201 and 1202 each have a field angle of 200 degrees relative to the Z axis, which is the optical axis 1203.

[0119] The imaging device 101B uses two fisheye cameras and therefore has two image sensors: camera 1201 has a lens and image sensor 1201S, and camera 1202 has a lens and image sensor 1202S.

[0120] The first fisheye camera 1201 and the second fisheye camera 1202 have approximately the same optical axis 1203 and are arranged back-to-back on the optical axis 1203. In this example, the optical axis 1203 is arranged along the Z direction (in other words, the Z axis), which is the vertical direction. On the optical axis 1203, with respect to the center Q, the viewpoint (in other words, the entrance pupil) of the camera 1201 is arranged at a position at a predetermined distance in the positive direction of the Z axis (upward in FIG. 14) and facing in the positive direction, and the viewpoint of the camera 1202 is arranged at a position at the same distance in the negative direction of the Z axis (downward in FIG. 14) and facing in the negative direction. Note that the cameras 1201 and 1202 may be arranged in contact with each other and integrally.

[0121] Because the first fisheye camera 1201 and the second fisheye camera 1202 have a field of view of 180 degrees or more, a common field of view 1206 exists as a field of view where the field of view 1204 of the first fisheye camera 1201 and the field of view 1205 of the second fisheye camera 1202 overlap. Note that although the field of view on the YZ plane is illustrated in Fig. 14, the field of view has an angle of view all around the Z axis, which is the optical axis 1203, including the X direction. Therefore, the common field of view 1206 also exists not only on the left and right sides in the Y direction as illustrated, but also all around the 360-degree perimeter around the optical axis 1203, in other words, roughly in the shape of a ring on the circumference when viewed on the XY plane.

[0122] An object 115 within this common field of view 1206, for example, an object 115 located in a generally horizontal direction (the Y direction in this example), can be photographed from two different viewpoints, a first viewpoint and a second viewpoint, by the first fisheye camera 1201 and the second fisheye camera 1202, thereby obtaining two images, a first image and a second image. Therefore, distance measurement is possible in the common field of view 1206 through stereo processing using these two fisheye cameras. Because the fields of view of the first fisheye camera 1201 and the second fisheye camera 1202 are 200 degrees, when the optical axes 1203 of these cameras are aligned, the common field of view 1206 has an angle of view of 20 degrees relative to the Y direction (i.e., the Y axis) shown in the figure. Therefore, to ensure this common field of view 1206 around the entire circumference of the fisheye cameras (i.e., around the entire circumference of the optical axis 1203), a deviation of up to ±20 degrees from the optical axis 1203 is acceptable.

[0123] FIG. 15, like FIG. 14, is a schematic diagram illustrating another example configuration of image capture device 101B in Embodiment 2. The configuration shown in FIG. 15 may be applied to image capture device 101B. In FIG. 15, first fisheye camera 1201 and second fisheye camera 1202 have approximately the same optical axis 1203 and are arranged facing each other on optical axis 1203. On optical axis 1203 (Z axis in this example), with respect to center Q, the viewpoint of camera 1201 is arranged at a predetermined distance in the positive direction of the Z axis and facing in the negative direction, and the viewpoint of camera 1202 is arranged at the same distance in the negative direction of the Z axis and facing in the positive direction. As in FIG. 14, field of view 1204 of camera 1201 and field of view 1205 of camera 1202 are 180 degrees or more, particularly 200 degrees.

[0124] 14, in the facing-to-face arrangement configuration of Fig. 15, a common field of view 1206 exists as a field of view where the field of view 1204 of the first fisheye camera 1201 and the field of view 1205 of the second fisheye camera 1202 overlap. Using these two cameras, distance measurement becomes possible in this common field of view 1206 through stereo processing. As described above, when the fields of view of cameras 1201 and 1202 are 200 degrees, the common field of view 1206 is 20 degrees. Furthermore, in order to ensure the common field of view 1206 around the entire circumference of the fisheye cameras, a deviation of up to ±20 degrees in the optical axis 1203 is permissible.

[0125] 13 to 17, the images captured by the imaging device 101B will be described below. In FIG. 14, light C21 of light C20 from the subject 115 passes through the common field of view 1206 and enters the first viewpoint of the first fisheye camera 1201 to form a first image, and light C22 passes through the common field of view 1206 and enters the first viewpoint of the second fisheye camera 1202 to form a second image. The image sensor 1201S of the first fisheye camera 1201 outputs a first image obtained by capturing the first image, and the image sensor 1202S of the second fisheye camera 1202 outputs a second image obtained by capturing the second image. Although not shown, the same operation occurs in FIG. 14.

[0126] 14 and 15, the two fisheye cameras, camera 1201 and camera 1202, are arranged symmetrically with respect to the YZ plane (for example, the horizontal plane) including the center Q.

[0127] [Captured image] FIG. 16 shows an example of an image obtained by capturing an object 115 within a common field of view 1206 using two fisheye cameras arranged back-to-back as shown in FIG. 14. (A) shows a first image (i.e., an upper image) captured from a first viewpoint by the first fisheye camera 1201 (i.e., the upper camera). (B) shows a second image (i.e., a lower image) captured from a second viewpoint by the second fisheye camera 1202 (i.e., the lower camera). The first image in (A) has a ring-shaped image area 1403 within a rectangular image 1401. The second image in (B) has a ring-shaped image area 1404 within a rectangular image 1402. These ring-shaped image areas are similar to the omnidirectional images shown in FIGS. 3 and 8.

[0128] The first fisheye camera 1201 and the second fisheye camera 1202 in FIG. 14 have opposite shooting directions on the optical axis 1203. Therefore, an image 1401 captured by the first fisheye camera 1201 and an image 1402 captured by the second fisheye camera 1202 are inverted in the radial direction R. Within the image region 1403, an image (or region) 1405 corresponds to a first image of the object 115, and within the image region 1404, an image (or region) 1406 corresponds to a second image of the object 115. In the first image of the image 1405, an arrow image is captured in a direction from the inner periphery to the outer periphery in the radial direction R. In contrast, in the second image of the image 1406, which is in an inverted state, an arrow image is captured in a direction from the outer periphery to the inner periphery in the radial direction R. A portion 1407 and a portion 1408 are the same corresponding portion (e.g., the base) of the arrow image.

[0129] Therefore, in the second embodiment, when an image region 1403 of image 1401 and an image region 1404 of image 1402 are developed into a panoramic image, the effective pixel count E differs between corresponding locations of the same image (for example, location 1407 and location 1408), as in the first embodiment. That is, the number of pixels in the circumferential direction C of each image varies depending on the position in the radial direction R. This reduces the performance of distance measurement.

[0130] In order to prevent this degradation in distance measurement performance, distance measurement device 1B of embodiment 2 in Fig. 13 generates a characteristic-matched image in characteristic-matched image generation unit 103B, which is an image in which the characteristics of these two images are matched. In embodiment 2, the characteristic to be matched is also the effective pixel count E when a panoramic image is created. If the same or similar performance cameras are used for first fisheye camera 1201 and second fisheye camera 1202, image region 1403 in image 1401 and image region 1404 in image 1402 will have approximately the same number of pixels.

[0131] In this case, enlarged views of the base portions (points 1407 and 1408) of the same arrow image in image 1405 in image area 1403 and image 1406 in image area 1404 of object 115 are shown in (C) and (D) of FIG. 16. The enlarged view in (C) shows the image of point 1407, and the enlarged view in (D) shows the image of point 1408. In (C), the width of the base portion is one pixel in the horizontal direction, and in (D), the width of the base portion is five pixels in the horizontal direction. As such, it can be seen that for the same portion of object 115, image 1406 in the second image taken by lower camera 1202 has a larger number of pixels than image 1405 in the first image taken by upper camera 1201.

[0132] 17 similarly shows enlarged views of the tip portions of the first and second images of the arrow image of subject 115. The enlarged view in (C) shows an image of a portion 1501 in image (or region) 1405 in (A), and the enlarged view in (D) shows an image of a portion 1502 in image (or region) 1406 in (B). In these enlarged views, it can be seen that, for the arrowhead portion at the tip of the arrow image, image 1405 of the first image taken by upper camera 1201 has a larger number of pixels than image 1406 of the second image taken by lower camera 1202, which is the opposite of the base portion in FIG.

[0133] 16 and 17, in the images taken by the two fisheye cameras above and below imaging device 101B, the magnitude relationship between the number of pixels in circumferential direction C at corresponding locations on the image of subject 115 in image region 1403 and image region 1404 changes depending on the distance from the center of the region in radial direction R. In such a case, characteristic-matched image generating unit 103 in FIG. 1 according to embodiment 1 cannot generate an image with a consistent effective pixel number E simply by projecting pixels in one region (e.g., image region 1403) onto corresponding pixel positions in the other region (e.g., image region 1404).

[0134] The above applies not only to the back-to-back arrangement in FIG. 14 but also to the facing arrangement in FIG.

[0135] Therefore, in the second embodiment, to solve the above problem, characteristic-matched image generation unit 103B in FIG. 13 generates a characteristic-matched image by projecting pixels in image region 1403 of the first image captured by upper camera 1201 and pixels in image region 1404 of the second image captured by lower camera 1202 onto corresponding pixel positions. The characteristic to be matched is the effective pixel count E in circumferential direction C at each position in radial direction R. In FIG. 16, this mutual projection, in other words, the concept of two projections, is illustrated by arrow 1410.

[0136] FIG. 13 illustrates this mutual projection configuration as an example of the functional block configuration of the characteristic-matched image generation unit 103B. Regarding the first and second images of the image signal g1, the characteristic-matched image generation unit 103B inputs the first image g11 to the first block B1 and the second image g12 to the second block B2. The characteristic-matched image generation unit 103B performs a first characteristic matching process using a first projection on the first image g11 using the first block B1, thereby obtaining an image g13 whose characteristics (effective pixel number E) are matched with those of the second image g12. Similarly, the characteristic-matched image generation unit 103B performs a second characteristic matching process using a second projection on the second image g12 using the second block B2, thereby obtaining an image g14 whose characteristics are matched with those of the first image g11. The first block B1 and the second block B2 have similar processing functions. In FIG. 13, the first block B1 and the second block B2 are configured to process in parallel, but this is not limited to this, and the configuration may also be such that the processing is performed sequentially in the order of the first block B1, the second block B2, and so on.

[0137] [Generation of characteristic-matched images] A method for generating a characteristic-matched image in the second embodiment will be described with reference to Fig. 18. Fig. 18 is a schematic diagram showing how a pixel of interest is projected onto a corresponding pixel position in a first image and a second image such as Fig. 16, which are images captured using the fisheye camera configuration of Fig. 14 or Fig. 15 in the second embodiment. (A) shows one first projection, and (B) shows the other second projection. For example, the first projection is a projection from the first image to the second image, and the second projection is a projection from the second image to the first image.

[0138] First, in (A), characteristic-matching image generation unit 103B (particularly first block B1) projects a pixel of interest 1601 in image region 1403 of the first image captured by upper camera 1201 onto corresponding pixel position 1602. Pixel of interest 1601 is a pixel located at a distance r from center q in radial direction R. Corresponding pixel position 1602 is the position of a pixel at a corresponding location in image region 1404 of the second image, located at a distance r' from center q in radial direction R. Note that image region 1403, for example, ranges from distance rc to distance rd in radial direction R. Distances r and r' are within the range from distance rc to distance rd.

[0139] A first characteristic-matched image (image g13 in FIG. 13) is created by such a first projection. At this time, the corresponding pixel position 1602 is determined by the projection function h(r), similar to FIG. 8 in the first embodiment. The projection function h(r) may be determined using, for example, the image height characteristic, as shown in Equation 5 above.

[0140] Similarly, in (B), a second characteristic-matched image (image g14 in FIG. 13) is created by a second projection using the same process as in (A) for image region 1404 of the second image captured by lower camera 1202. The characteristic-matched image generation unit 103B (particularly the second block B2) projects pixel of interest 1611 in image region 1404 onto corresponding pixel position 1612. Corresponding pixel position 1612 is the position of a pixel at a corresponding location in image region 1403 of the first image.

[0141] In the distance measurement device 1B (particularly the processing device 102B) of the second embodiment, the characteristic-matching image generation unit 103B generates a characteristic-matching image by projecting the first image and the second image onto corresponding locations using the above-described method. The processing device 102B converts the generated characteristic-matching images into rectangular panoramic images in the image processing unit 104 using the image signals g2 from images g13 and g14. The distance measurement unit 105 then calculates the distance to the subject based on these two panoramic images.

[0142] [Effects etc. (2)] As described above, according to the second embodiment, for two images from two viewpoints obtained by the imaging device 101B of Fig. 14 etc., it is possible to generate characteristic-matched images in which the effective pixel counts E are matched, even if the images have different pixel counts depending on the position in the radial direction R within the images. Furthermore, according to the second embodiment, distance measurement is performed using these images, which makes it possible to improve the performance of distance measurement.

[0143] <Third Embodiment> The distance measurement device of the third embodiment will be described with reference to Figure 19 and subsequent figures. The distance measurement device 1C of the third embodiment in Figure 19 differs from the first embodiment in that two cameras are arranged in parallel in the same direction as stereo cameras in an imaging device 101C, and the captured image is not a panoramic image.

[0144] Furthermore, in the third embodiment, similarly to the first embodiment, a characteristic-matched image is generated by projection from one image, the first image, obtained from the imaging unit to the other image, the second image. This projection is basically the same as the projection in the first embodiment.

[0145] [Distance measuring device] 19 shows the configuration of a distance measurement device 1C according to the third embodiment. This distance measurement device 1C includes an imaging device 101C and a processing device 102C. The configuration of this distance measurement device 1C is the same as that of the first embodiment (FIG. 1) or the second embodiment (FIG. 13), except for the imaging device 101C. In addition, in the third embodiment, the original images obtained from the two cameras are rectangular images, so that panoramic transformation (in other words, panoramic image generation) in the image processing unit 104 is not required.

[0146] [Imaging device] FIG. 20 is a schematic explanatory diagram illustrating an example configuration of an imaging device 101C according to the third embodiment. The imaging device 101C, which is an imaging unit, is configured using a camera 1701 and a camera 1702 as two cameras that form a stereo camera. In FIG. 20, the imaging device 101C is placed in a space of the illustrated (X, Y, Z) coordinate system. The Z direction is the vertical direction. The cameras 1701 and 1702 are placed in parallel at predetermined positions spaced a predetermined distance apart on a straight line 1700 in the X direction. The camera 1701 is a first camera having a first viewpoint, and the camera 1702 is a second camera having a second viewpoint.

[0147] Image capturing device 101C has two image sensors because it uses two cameras (camera 1701 and camera 1702). Camera 1701 has a lens and image sensor 1701S, and camera 1702 has a lens and image sensor 1702S.

[0148] The optical axis 1703 of the first camera 1701 and the optical axis 1704 of the second camera 1702 are approximately parallel to each other. The optical axis 1703 of the camera 1701 and the optical axis 1704 of the camera 1702 are arranged in the same direction, which in this example is the horizontal Y direction.

[0149] The field of view of the first camera 1701 and the field of view of the second camera 1702 exist as field of view corresponding to the angle of view of each camera. The field of view of the first camera 1701 and the field of view of the second camera 1702 may be different. In this example, the field of view of the second camera 1702 is assumed to be smaller than that of the first camera 1701. In other words, in this example, the first camera 1701 has a wide field of view due to a wide angle, and the second camera 1702 has a narrower field of view due to a narrower angle than that of the first camera 1701. However, the number of pixels of the image captured by the first camera 1701 and the image captured by the second camera 1702 is assumed to be approximately the same. The first image of the first camera 1701 and the second image of the second camera 1702 are assumed to have approximately the same image size but different resolutions (in other words, pixel densities).

[0150] 20, there is a common field of view where the fields of view of the first camera 1701 and the second camera 1702 overlap. Using these two cameras, distance measurement is possible through stereo processing in this common field of view. In this example, the field of view of the second camera 1702 is smaller than that of the first camera 1701. Therefore, the number of pixels in the portion of the object 115 captured in the first image captured by the first camera 1701 and the second image captured by the second camera 1702 is smaller in the first image captured by the first camera 1701 than in the second image.

[0151] Therefore, as a comparative example to the third embodiment, a difference occurs in the effective pixel number E in an image obtained by processing a first image captured by a first camera 1701 and a second image captured by a second camera 1702 as they are in an image processing unit, i.e., a panoramic image for distance measurement, which results in a decrease in the performance of distance measurement.

[0152] Therefore, in order to solve the above problem, the distance measurement device 1C of embodiment 3 generates a characteristic-matched image by projecting pixels in the area of ​​the first image of the first camera 1701 onto corresponding pixel positions in the area of ​​the second image of the second camera 1702 in the characteristic-matched image generation unit 103C of the processing device 102C of Figure 19.

[0153] [Generation and projection of characteristic-matched images] A method for generating a characteristic-matched image in the third embodiment will be described below with reference to Fig. 20 and Fig. 21. In Fig. 20, with optical axis 1703 of first camera 1701 as the reference, the magnitude of signed angle 1705 in the horizontal direction (X direction in Fig. 20) is taken as θx, and the magnitude of signed angle in the vertical direction (Z direction in Fig. 20) is taken as θy. The symbols x and y correspond to the coordinate system (x, y) in the camera image in Fig. 21.

[0154] FIG. 21 is a schematic explanatory diagram showing a method for specifying pixel positions and pixel projection in the second image acquired by the second camera 1702 in FIG. 20. FIG. 21 shows a rectangular image 1801, which is the second image acquired by the second camera 1702. In image 1801, the upper left pixel 1802 is set as the origin. From the origin pixel 1802, the x-axis is set horizontally to the right, and the y-axis is set vertically downward. These (x, y) values ​​are used to specify a pixel in image 1801 (in other words, position coordinates). The pixel specification method is similar for the first image acquired by the first camera 1701 in FIG. 21.

[0155] 21 does not show the first image captured by the first camera 1701, but shows the corresponding pixel position 1804 in the first image superimposed on the second image, image 1801. The first image is a rectangular image of the same size as the second image, but has a different resolution from the second image due to the difference in field of view as shown in FIG.

[0156] Here, a method for determining a corresponding pixel position using image height characteristics when projecting a pixel in image 1801, which is the second image captured by second camera 1702 (in other words, a pixel of interest), onto a corresponding pixel position in the first image captured by first camera 1701 will be described. In Fig. 21, an arrow 1805 from pixel of interest 1803 in the first image to corresponding pixel position 1804 in the second image corresponds to the concept of projection. This concept of projection is the same as in embodiment 1 or 2, but in more detail, there are projections in two directions in the image: the x direction (in other words, the x axis, the horizontal direction in the image) and the y direction (in other words, the y axis, the vertical direction in the image).

[0157] As a modified example, the mechanism of embodiment 3 can also be applied in the case where the first image captured by the first camera 1701 and the second image captured by the second camera 1702 have different image sizes but the same resolution.

[0158] 21, the x-coordinate and y-coordinate of a pixel of interest 1803 are set to x2 and y2, respectively, and the x-coordinate and y-coordinate of a corresponding pixel position 1804 are set to x1 and y1, respectively. Furthermore, the image height characteristics of the first camera 1701 in the horizontal direction (x direction) are set to x1 = f1(θx), and the image height characteristics in the vertical direction (y direction) are set to y1 = g1(θy). Similarly, the image height characteristics of the second camera 1702 in the horizontal direction (x direction) are set to x2 = f2(θx), and the image height characteristics in the vertical direction (y direction) are set to y2 = g2(θy).

[0159] Using the image height characteristics of each camera, it is possible to determine the x-coordinate x1 and y-coordinate y1 of the corresponding pixel position 1804. First, the x-coordinate x1 of the corresponding pixel position 1804 can be calculated as shown in the following equation 8 using x1=f1(θx) and x2=f2(θx).

[0160] TIFF0007756008000008.tif8170

[0161] Similarly, the y coordinate y1 of the corresponding pixel position 1804 can be calculated as shown in the following equation 9 using y1=g1(θy) and y2=g2(θy).

[0162] TIFF0007756008000009.tif8170

[0163] As a result, it is possible to determine the corresponding pixel position using the image height characteristics of the first camera 1701 and the second camera 1702. By projection using this image height characteristic, it is possible to match the effective pixel number E as a characteristic of the second image captured by the second camera 1702 with the effective pixel number E as a characteristic of the first image captured by the first camera 1701. Note that in the above example, the pixel of interest 1803 and the corresponding pixel position 1804 correspond to the pixel at the distance r and the corresponding pixel position at the distance r' in FIG. 8 in terms of the correspondence relationship with the first embodiment.

[0164] As described above, in the third embodiment, characteristic matching processing is performed by characteristic matching processing unit 103C in FIG. 19 so as to project pixels of the second image captured by second camera 1702 onto corresponding pixel positions of the first image captured by first camera 1701. This allows for the generation of images with a consistent effective pixel count E. Characteristic matching processing unit 103C in FIG. 19 outputs the first image and the second image after characteristic matching. Image processing unit 104 performs image processing for distance measurement on these two images. Distance measurement unit 105 then calculates the distance based on the two rectangular images after these image processing.

[0165] [Effects etc. (3)] As described above, according to the third embodiment, in the case of a stereo camera configuration such as that shown in FIG. 20, the characteristics of the two images can be matched mainly by simple processing using projection, thereby improving the accuracy of distance measurement.

[0166] <Fourth Embodiment> The distance measurement device of the fourth embodiment will be described with reference to Fig. 22. The distance measurement device 1D of the fourth embodiment in Fig. 22 differs from the first embodiment in that processing by a filtering unit 108 is added between the characteristic-matched image generation unit 103 and the image processing unit 104 in a processing device 102D.

[0167] [Distance measuring device] 22 shows the configuration of a distance measurement device 1D according to the fourth embodiment. This distance measurement device 1D includes an imaging device 101 and a processing device 102D. For example, this distance measurement device 1D includes the same imaging device 101 as the imaging device 101 according to the first embodiment. In addition to the components according to the first embodiment, the processing device 102D includes a filtering unit 108 between the characteristic-matched image generation unit 103 and the image processing unit 104.

[0168] The filtering unit 108 is a block that performs uniform filtering on the characteristic-matched image generated by the characteristic-matched image generation unit 103. This makes it possible to achieve non-uniform filtering on the original image. The original image is two images of the image signal g1 from the imaging device 101.

[0169] Here, uniform filtering refers to filtering using an LPF (low-pass filter) such as a general Gaussian filter, where the cutoff frequency of the LPF is always constant regardless of the position in the image. In contrast, non-uniform filtering refers to filtering using a filter with a different cutoff frequency depending on the position in the image. This non-uniform filtering is effective when filtering with a cutoff frequency according to the distance from a certain point in the image, such as image 201 described above in FIG. 2, where the number of pixels (particularly the number of pixels in the circumferential direction C) varies depending on the distance in the radial direction R from the certain point in the image. This type of filtering will be described in detail below.

[0170] In the example of the fourth embodiment, the image to which the filter is applied (i.e., the image generated by the characteristic-matched image generating unit 103) is assumed to be the characteristic-matched image by projection as shown in Fig. 8 created in the first embodiment. Also, in the example of the fourth embodiment, an LPF such as a general Gaussian filter is used as the uniform filter, and the reciprocal of the cutoff frequency of the LPF is set to n pixels.

[0171] Note that the "n" in "n pixels" refers to the number of pixels in the vertical and horizontal dimensions of the range over which the filter effect extends. FIG. 23 shows an example of a Gaussian filter. In this example of the Gaussian filter kernel, "n" refers to a 5x5 area over which the filter effect extends. In the example of the Gaussian filter kernel in FIG. 23, the range over which the filter effect extends is the area where the weighting value is appropriately set, excluding invalid areas where the value is set to 0 or 1, for example.

[0172] In the lower viewpoint region 203 in Fig. 8, a pixel at a position a distance r in the radial direction R from the center 703 of the region is projected to a corresponding pixel position where the distance r' from the center 703 is r' = h(r) by the projection function of the above-mentioned equation 5. This projection produces an effect equivalent to that of a filter whose reciprocal of the cutoff frequency is r / h(r) acting on the pixel at the distance r from the center 703 in the image 201. This projection is similarly performed by the characteristic-matching image generation unit 103 in Fig. 22.

[0173] Furthermore, filtering unit 108 applies a uniform filter with n pixels as the reciprocal of the cutoff frequency to the projected region, i.e., the characteristic-matching image. This effectively applies a filter with n pixels as the reciprocal of the cutoff frequency, expressed by the following equation 10, to lower viewpoint region 203 of image 201.

[0174] TIFF0007756008000010.tif13170

[0175] Equation 10 means that the cutoff frequency changes depending on the distance r from the center 703. This realizes a non-uniform filter for pixels in the lower viewpoint region 203 of the image 201, in which the cutoff frequency changes depending on the distance from the center where the pixel is located.

[0176] With the above-described filtering, if the amount of blur varies depending on the distance from the center of the image due to, for example, distortion of the camera lens of the imaging device 101, it is possible to correct the blur so that it is uniform across the entire image by appropriately selecting the cutoff frequency.

[0177] [Effects etc. (4)] As described above, according to the fourth embodiment, a characteristic-matched image is generated and uniform filtering is performed on the characteristic-matched image, thereby realizing non-uniform filtering of the original image. According to the fourth embodiment, a correction can be performed to reduce the difference in the effective pixel number E between multiple captured images used for distance measurement, by relatively simple processing using projection and filtering.

[0178] Although the embodiments of the present disclosure have been specifically described above, they are not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present disclosure. Except for essential components, components can be added, deleted, or replaced in each embodiment. Unless otherwise specified, each component can be singular or plural. A combination of each embodiment is also possible. [Explanation of symbols]

[0179] 1...distance measurement device, 101...imaging device, 102...processing device, 103...characteristic matching image generation unit, 104...image processing unit, 105...distance measurement unit, 106...output interface, 107...control unit, 201...image, 202...upper viewpoint area, 203...lower viewpoint area, 701...pixel of interest, 702...corresponding pixel position, 703...center, 800...projection.

Claims

1. an imaging device that captures an image of a subject; a processing device that acquires and processes the image from the imaging device; Equipped with The processing device includes: a characteristic-matched image generation unit that receives input of two or more images including a first image obtained by photographing the subject from a first viewpoint and a second image obtained by photographing the subject from a second viewpoint different from the first viewpoint, and generates two or more images as characteristic-matched images in which characteristics of the first image and characteristics of the second image are matched; a distance measurement unit that calculates a distance to the subject by calculating a parallax based on two or more images as the characteristic-matched images; and the characteristic is an effective pixel number as the number of pixels that can be used for distance measurement related to the pixel number distribution of the same image in the image, an optical system of the imaging device causes a difference between the number of effective pixels of the first image and the number of effective pixels of the second image; the characteristic-matched image generating unit generates the characteristic-matched image by projecting a pixel of interest in at least one of the first image and the second image onto a corresponding pixel position in the other image. Distance measuring device.

2. 2. The distance measurement device according to claim 1, the characteristic-matched image generating unit generates the characteristic-matched image by projecting a pixel of interest in the first image onto a corresponding pixel position in the second image. Distance measuring device.

3. An imaging device that captures an image of a subject; a processing device that acquires and processes the image from the imaging device; Equipped with The processing device includes: a characteristic-matched image generation unit that receives input of two or more images including a first image obtained by photographing the subject from a first viewpoint and a second image obtained by photographing the subject from a second viewpoint different from the first viewpoint, and generates two or more images as characteristic-matched images in which characteristics of the first image and characteristics of the second image are matched; a distance measurement unit that calculates a distance to the subject by calculating a parallax based on two or more images as the characteristic-matched images; and the characteristic-matched image generating unit generates the characteristic-matched image by projecting a pixel of interest in at least one of the first image and the second image onto a corresponding pixel position in the other image; The characteristic matching image generating unit generating a first characteristic-matching image by projecting a pixel of interest in the first image onto a corresponding pixel location in the second image; generating a second characteristic-matching image by projecting a pixel of interest in the second image onto a corresponding pixel location in the first image; the first characteristic-matched image and the second characteristic-matched image are defined as the characteristic-matched image; Distance measuring device.

4. An imaging device that captures an image of a subject; a processing device that acquires and processes the image from the imaging device; Equipped with The processing device includes: a characteristic-matched image generation unit that receives input of two or more images including a first image obtained by photographing the subject from a first viewpoint and a second image obtained by photographing the subject from a second viewpoint different from the first viewpoint, and generates two or more images as characteristic-matched images in which characteristics of the first image and characteristics of the second image are matched; a distance measurement unit that calculates a distance to the subject by calculating a parallax based on two or more images as the characteristic-matched images; and the characteristic-matched image generating unit generates the characteristic-matched image by projecting a pixel of interest in at least one of the first image and the second image onto a corresponding pixel position in the other image; the characteristic-matched image generation unit determines the corresponding pixel position of the pixel of interest in the projection based on image height characteristics of an optical system that captures the first image of the imaging device and image height characteristics of an optical system that captures the second image. Distance measuring device.

5. 5. The distance measurement device according to claim 1, 3 or 4, the processing device has an image processing unit that performs image processing to convert a coordinate system of the characteristic-matched image into a coordinate system for distance calculation in the distance measurement unit. Distance measuring device.

6. 5. The distance measurement device according to claim 1, 3 or 4, the processing device has a filtering unit that applies a filter with a fixed cutoff frequency to the characteristic matching image; Distance measuring device.

7. 5. The distance measurement device according to claim 1, 3 or 4, the imaging device includes a single image sensor and an imaging optical system that forms an image of light from the subject on the image sensor; the imaging optical system includes a first optical system that projects a first image corresponding to the first image onto the image sensor, and a second optical system that projects a second image corresponding to the second image onto the image sensor; Distance measuring device.

8. 8. The distance measurement device according to claim 7, the imaging optical system has a plurality of hyperbolic mirrors as elements constituting the first optical system and the second optical system; Distance measuring device.

9. 8. The distance measurement device according to claim 7, The first image and the second image in the image are arranged in a concentric ring shape. Distance measuring device.

10. 5. The distance measurement device according to claim 1, 3 or 4, The imaging device is a first optical system for capturing the first image, the first optical system including a first lens and a first image sensor; a second optical system for capturing the second image, the second optical system including a second lens and a second image sensor; The first optical system and the second optical system are arranged back to back or facing each other on the same axis. Distance measuring device.

11. 5. The distance measurement device according to claim 1, 3 or 4, The imaging device is a first optical system for capturing the first image, the first optical system including a first lens and a first image sensor; a second optical system for capturing the second image, the second optical system including a second lens and a second image sensor; The first optical system and the second optical system are arranged with their optical axes in the same direction and parallel to each other. Distance measuring device.

12. 10. The distance measurement device according to claim 9, the processing device has an image processing unit that performs image processing to convert a coordinate system of the characteristic-matched image into a coordinate system for distance calculation in the distance measurement unit, the image processing unit converts the concentric ring-shaped first image and the second image in the characteristic-matched image into a rectangular panoramic image, and inverts one of the obtained first panoramic image and second panoramic image so as to match the orientation of the image of the subject, thereby obtaining two images for distance calculation by the distance measurement unit. Distance measuring device.

13. A distance measurement method for a distance measurement device including an imaging device that captures an image of a subject and a processing device that acquires and processes the image from the imaging device, a characteristic-matched image generation step in which the processing device inputs two or more images from the images, including a first image obtained by photographing the subject from a first viewpoint and a second image obtained by photographing the subject from a second viewpoint different from the first viewpoint, and generates two or more images as characteristic-matched images in which characteristics of the first image and characteristics of the second image are matched; a distance measurement step of calculating a distance to the subject by calculating a parallax based on two or more images as the characteristic-matched images; and the characteristic is an effective pixel number as the number of pixels that can be used for distance measurement related to the pixel number distribution of the same image in the image, an optical system of the imaging device causes a difference between the number of effective pixels of the first image and the number of effective pixels of the second image; the characteristic-matched image generating step generates the characteristic-matched image by projecting a pixel of interest in at least one of the first image and the second image onto a corresponding pixel position in the other image. Distance measurement method.

14. A distance measurement method for a distance measurement device including an imaging device that captures an image of a subject, and a processing device that acquires and processes the image from the imaging device, comprising: a characteristic-matched image generation step in which the processing device inputs two or more images from the images, including a first image obtained by photographing the subject from a first viewpoint and a second image obtained by photographing the subject from a second viewpoint different from the first viewpoint, and generates two or more images as characteristic-matched images in which characteristics of the first image and characteristics of the second image are matched; a distance measurement step of calculating a distance to the subject by calculating a parallax based on two or more images as the characteristic-matched images; and the characteristic-matched image generating step generates the characteristic-matched image by projecting a pixel of interest in at least one of the first image and the second image onto a corresponding pixel position in the other image; The characteristic-matching image generating step includes: generating a first characteristic-matching image by projecting a pixel of interest in the first image onto a corresponding pixel location in the second image; generating a second characteristic-matching image by projecting a pixel of interest in the second image onto a corresponding pixel location in the first image; the first characteristic-matched image and the second characteristic-matched image are defined as the characteristic-matched image; Distance measurement method.

15. A distance measurement method for a distance measurement device including an imaging device that captures an image of a subject, and a processing device that acquires and processes the image from the imaging device, comprising: a characteristic-matched image generation step in which the processing device inputs two or more images from the images, including a first image obtained by photographing the subject from a first viewpoint and a second image obtained by photographing the subject from a second viewpoint different from the first viewpoint, and generates two or more images as characteristic-matched images in which characteristics of the first image and characteristics of the second image are matched; a distance measurement step of calculating a distance to the subject by calculating a parallax based on two or more images as the characteristic-matched images; and the characteristic-matched image generating step generates the characteristic-matched image by projecting a pixel of interest in at least one of the first image and the second image onto a corresponding pixel position in the other image; the characteristic-matched image generating step determines the corresponding pixel position of the pixel of interest in the projection based on image height characteristics of an optical system that captures the first image of the imaging device and image height characteristics of an optical system that captures the second image. Distance measurement method.

Images