Image processing system, mobile body, image processing method, and computer program

Abstract

To provide an image processing system capable of generating an entire-celestial-sphere composite image and distance measurement information using a small number of imaging unit.SOLUTION: An image processing system includes a first imaging unit comprising a first optical system having a maximum half view angle θa and a second imaging unit comprising a second optical system having a maximum half view angle θb, where configuration is performed so as to satisfy θa+θb>180° and an optical axis of the first optical system and an optical axis of the second optical system are set oppositely to each other. The image processing system also includes: an image composition unit for generating entire-celestial-sphere composite image information on the basis of outputs of the first imaging unit and the second imaging unit; and a distance measurement processing unit for generating distance information on the basis of an output of a view angle at which the first imaging unit and the second imaging unit overlap.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing system, a moving object, an image processing method, a computer program, and the like. [Background technology]

[0002] Conventionally, there are known mobile objects that move autonomously and perform work in various places such as homes, office buildings, houses, logistics centers, etc. Such mobile objects can perform autonomous driving by estimating their own position and generating driving routes using output information from sensors mounted on the mobile objects.

[0003] One known technology for acquiring information about the surrounding environment of an autonomous vehicle is a distance measurement technology that captures disparity images of a subject viewed from two different viewpoints and calculates the depth value of a specific area of ​​the subject based on the disparity images.

[0004] There is also known a technology that uses a camera equipped with an optical lens with a wide viewing angle (angle of view), for example, a fisheye lens, to capture an image of a subject from two viewpoints. There is also known a technology that performs remote monitoring and image recognition using video information acquired by a sensor mounted on a moving object.

[0005] For example, Patent Document 1 discloses a technique for performing omnidirectional imaging and omnidirectional distance measurement by using a pair of omnidirectional imaging units mounted on a moving object. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-102072 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, the conventional technology disclosed in Patent Document 1 has a problem in that it is difficult to capture images of the lower part of the imaging unit or blind spots between imaging units, etc. Therefore, when attempting to capture images of the entire celestial sphere (360° horizontally, 180° vertically) and measure distances in all directions (360° horizontally), the number of imaging units required increases.

[0008] Therefore, an object of the present invention is to provide an image processing system capable of generating a spherical composite image and distance measurement information using a small number of imaging units. [Means for solving the problem]

[0009] In order to achieve the above object, an image processing system according to one aspect of the present invention comprises: a first imaging unit including a first optical system having a maximum half angle of view of θa; a second imaging unit including a second optical system having a maximum half angle of view of θb; θa+θb>180° The above is configured to satisfy the above. The optical axis of the first optical system and the optical axis of the second optical system are arranged in opposite directions to each other, an image synthesis unit that generates synthetic image information of the entire celestial sphere based on outputs of the first imaging unit and the second imaging unit; The imaging device further comprises a distance measurement processing unit that generates distance information based on an output of a viewing angle at which the first imaging unit and the second imaging unit overlap. Effect of the Invention

[0010] According to the present invention, it is possible to provide an image processing system capable of generating a composite image of the celestial sphere and distance measurement information using a small number of imaging units. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic perspective view of an imaging device according to a first embodiment. [Diagram 2] 1 is a functional block diagram showing an example of the configuration of an image processing system 200 according to a first embodiment. [Diagram 3]5A and 5B are diagrams showing examples of images captured by the imaging system according to the first embodiment. [Figure 4] 2 is a functional block diagram showing an example of the configuration of a distance measurement processing unit 204. FIG. [Diagram 5] 13A and 13B are diagrams showing examples of results of panorama processing performed by image conversion units 402 and 403 according to the first embodiment. [Figure 6] 13A and 13B are diagrams showing examples of results of trimming performed by the image processing system according to the first embodiment. [Figure 7] FIG. 1 is a diagram illustrating the principle of stereo ranging. [Figure 8] 4A to 4C are diagrams showing an example of the results of photographing and distance measurement performed by the image processing system according to the first embodiment. [Figure 9] 13(A) to 13(C) are diagrams illustrating the characteristics of the optical system in the second embodiment. [Figure 10] FIG. 11 is an external perspective view of a moving body according to a third embodiment. [Figure 11] FIG. 11 is a functional block diagram showing a configuration example of an autonomous driving system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In the drawings, the same members or elements are given the same reference numerals, and duplicated descriptions thereof will be omitted or simplified.

[0013] (Embodiment 1) A first embodiment of the present invention will be described below.

[0014] Fig. 1 is a perspective schematic diagram of an imaging device according to embodiment 1, and the imaging device 100 shown in Fig. 1 includes a first imaging unit 1a and a second imaging unit 1b. The first imaging unit 1a and the second imaging unit 1b are provided in a housing 4 with their respective optical axes 5a and 5b coaxially and in opposite directions to each other.

[0015] The first imaging unit 1a includes an imaging element 3a and a first optical system 2a, and the second imaging unit 1b includes an imaging element 3b and a second optical system 2b. The imaging elements 3a and 3b convert the light incident through the first optical system 2a and the second optical system 2b into electrical signals, respectively.

[0016] The imaging elements 3a and 3b may be a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

[0017] For the first optical system 2a and the second optical system 2b, optical systems that satisfy the following formula 1 are used, where the maximum half angle of view of the first optical system 2a is θa and the maximum half angle of view of the second optical system 2b is θb. θa+θb>180° (Equation 1)

[0018] In an optical system that satisfies formula 1, an overlapping angle of view occurs between the images captured by the first imaging unit 1a and the second imaging unit 1b. For example, if θa is 95° and θb is 95°, the angle of view of θa from 85° to 95° and the angle of view of θb from 85° to 95° are overlapping angles of view.

[0019] The first optical system 2a and the second optical system 2b do not have to have the same angle of view. At least one of the first optical system 2a and the second optical system 2b is not limited to an optical system with a fixed angle of view, and may be an optical system with a variable angle of view that satisfies the above formula 1. When an optical system with a variable angle of view is used, the overlapping area of ​​the images acquired by the first imaging unit 1a and the second imaging unit 1b becomes variable, and the vertical position of the distance information of 360° in the horizontal direction acquired in the distance measurement process described later can be changed.

[0020] That is, it becomes possible to arbitrarily change the vertical position of the distance measurement information, and it is possible to efficiently obtain distance information in the range required by the imaging device 100. Note that the first imaging unit 1a and the second imaging unit 1b need not necessarily be the same type of camera, but may be a combination of different types of cameras, such as an infrared camera and a visible light camera.

[0021] Fig. 2 is a functional block diagram showing an example of the configuration of an image processing system 200 according to embodiment 1. Note that some of the functional blocks shown in Fig. 2 are realized by causing a CPU or the like serving as a computer (not shown) included in the image processing system 200 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0022] However, a part or all of these may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0023] In addition, the functional blocks shown in Fig. 2 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. That is, a part of the image processing system 200 may be provided in, for example, an external server. The above explanation regarding Fig. 2 also applies to Figs. 4 and 11.

[0024] The image processing system 200 includes a first imaging unit 1a, a second imaging unit 1b, image processing units 202 and 203, a distance measurement processing unit 204, an image synthesis unit 205, an object detection unit 208, and a display unit 209. The image processing units 202 and 203 perform various image processing such as white balance adjustment, demosaic processing, gain / offset adjustment, gamma correction, and color correction on the imaging signals output from the first imaging unit 1a and the second imaging unit 1b, respectively.

[0025] Distance measurement processing unit 204 performs distance measurement using a method described later, and creates omnidirectional (360° horizontal) distance information 206. Image synthesis unit 205 performs panoramic expansion on the images captured by first imaging unit 1a and second imaging unit 1b, and then synthesizes them into one image to create spherical (360° horizontal, 180° vertical) synthetic image information 207. That is, image synthesis unit 205 generates spherical synthetic image information based on the outputs of first imaging unit 1a and second imaging unit 1b.

[0026] The panoramic development method may be based on equirectangular projection or Mercator projection, and the present invention is not limited to this method. The object detection unit 208 detects objects existing around the image capture device 100 by image recognition based on the distance information 206 and the composite image information 207. That is, the object detection unit 208 detects objects existing around the image capture device 100 based on the captured images and distance information of the first and second image capture units.

[0027] The object to be detected is, for example, the position of an object such as a vehicle, a building, or a person. Note that only distance information may be used for object detection, or only synthetic image information may be used, and the present invention is not limited to the method of object detection.

[0028] In addition, various information based on the detection results, such as information indicating various objects detected in the panoramic image and their coordinates on the image, and information such as a superimposed image of the object detection result on the panoramic image using a bounding box, etc. may be generated and output. The display unit 209 displays the distance information 206, the composite image information 207, and object information detected by the object detection unit 208.

[0029] 3A and 3B are diagrams showing examples of images acquired by the imaging device according to the first embodiment. In the imaging device as shown in FIG. 1, it is assumed that the first imaging unit 1a and the second imaging unit 1b each have a wide viewing angle optical system with a half angle of view of 95°, the imaging device 100 is installed so that the optical axis 5a is perpendicular to the ground, and an outdoor image is captured. In this case, the first image 300a and the second image 300b shown in FIG. 3A and 3B are acquired by the upper and lower first imaging unit 1a and second imaging unit 1b, respectively.

[0030] The first image 300a shows the sky 307 in the center, and the ground around it in the outer periphery, while the second image 300b shows the ground 309 and road markings 308, etc. in the center, and the ground around them in the outer periphery.

[0031] In the first image 300a and the second image 300b, for example, pedestrians 301a and 301b facing sideways, buildings 302a and 302b, pedestrians 303a and 303b facing forward, and vehicles 304a and 304b are shown. In the first image 300a and the second image 300b, trees 305a and 305b and buildings 306a and 306b are also shown.

[0032] Depending on the installation position of the imaging device 100 and the surrounding environment, the above-mentioned object may not be captured in the first image 300a and the second image 300b. Also, the installation position of the imaging device 100 is not limited to this, and the first image 300a and the second image 300b are acquired according to the installation position.

[0033] Fig. 4 is a functional block diagram showing an example of the configuration of the distance measurement processing unit 204. Image conversion units 402 and 403 perform panoramic development processing on the celestial sphere images such as the first image 300a and the second image 300b shown in Fig. 3 output from the image processing units 202 and 203, using calibration information 401 such as internal and external camera parameters.

[0034] As described above, the panoramic development method may be based on the equirectangular projection or the Mercator projection.

[0035] 5(A) and (B) are diagrams showing examples of the results of panoramic processing performed by the image conversion units 402 and 403 in embodiment 1, which are panoramic expansions of the first image 300a and the second image 300b as shown in FIG.

[0036] The first panoramic image 500a is a panoramic expansion of the first image 300a and represents composite image information of 360° horizontally and 180° vertically, and the second panoramic image 500b is a panoramic expansion of the second image 300b in the same manner as the first image.

[0037] Cropping processing units 404 and 405 in FIG. 4 determine the superimposition angle of view for the panoramic images output from the image conversion units 402 and 403 using calibration information 401 such as camera internal parameters previously acquired, and cut out the superimposition area from each panoramic image.

[0038] An example of the overlapping region cut-out process will be described with reference to Fig. 6. Figs. 6(A) and (B) are diagrams showing examples of the results of trimming (cut-out) performed by the image processing system according to the first embodiment. That is, Figs. 6(A) and (B) show the results of the trimming process for cutting out the overlapping regions (85° to 95°) of the first panoramic image 500a and the second panoramic image 500b as shown in Fig. 5.

[0039] The first trimmed image 600a is obtained by trimming the vertical 85° to 95° portion of the first panoramic image 500a, which is the overlapping region of the first panoramic image 500a and the second panoramic image 500b, from the first panoramic image 500a. The second trimmed image 600b is obtained by trimming the vertical 85° to 95° portion of the second panoramic image 500b, which is the overlapping region of the first panoramic image 500a and the second panoramic image 500b, from the second panoramic image 500b.

[0040] As described above, the trimming processing units 404 and 405 cut out the image area of ​​the viewing angle where the first imaging unit and the second imaging unit overlap. Note that the determination of the overlapping area is not limited to this, and a method of determining the overlapping area based on the texture of the image may be used, and the present invention is not limited to this determination method.

[0041] The distance calculation unit 406 in Fig. 4 performs stereo ranging based on the cropped images output from the cropping processing units 404 and 405, and generates distance information. That is, the distance measurement processing unit 204 generates distance information based on the output of the viewing angle where the first and second imaging units overlap. Fig. 7 is a diagram showing the principle of stereo ranging. Since the measurement object 701 looks different in the upper and lower images, it is imaged at position 702 in image 700a and at position 703 in image 700b.

[0042] Here, images 700a and 700b correspond to the first trimmed image 600a and the second trimmed image 600b output from the trimming processing units 404 and 405. Images 702 and 703 formed by the objects to be measured in the respective images are, for example, pedestrians 301a and 301b facing sideways in FIG.

[0043] Alternatively, the objects may be buildings 302a and 302b, pedestrians 303a and 303b facing forward, vehicles 304a and 304b, trees 305a and 305b, buildings 306a and 306b, as well as animals and structures on the road.

[0044] At this time, a disparity 704 occurring between the images 700a and 700b is calculated by performing stereo matching between the images 700a and 700b in the vertical direction of FIG. 7 and searching for pixel positions with high similarity in local feature amounts.

[0045] The above-mentioned method of measuring the parallax is only an example, and is not limited to this. Since the focal length 705 and the base length 706 are known values ​​determined by the installation positions of the optical system and the imaging unit on the housing, the distance to the measurement target 701 can be obtained from the similarity relationship of the triangle by using the value of the parallax 704 measured on the image.

[0046] By performing the above processing on the trimmed images output from the trimming processing units 404 and 405, for example the first trimmed image 600a and the second trimmed image 600b in Fig. 6, distance information corresponding to the trimmed images is calculated. Furthermore, based on the calculated distance information, information such as a distance image showing the distance to various objects present within the field of view may be generated and output.

[0047] FIG. 8 is a diagram showing an example of the results of imaging and distance measurement performed by the image processing system according to the first embodiment, in which the entire celestial sphere of the imaging device 100 is displayed as a panoramic image, and object detection results and distance information are displayed together with bounding boxes.

[0048] The distance displayed in the bounding box is calculated based on a distance image based on distance information 206 and the coordinates on the image of the object in the panoramic image detected by object detection unit 208. In other words, the distance to the object is calculated by statistical processing such as the average value of the distance image within the object. Note that the display method of the composite image information, distance information, and object detection result is not limited to the example in FIG.

[0049] As described above, according to the first embodiment, the entire celestial sphere (360° horizontally and 180° vertically) of the imaging device 100 can be captured using a small number of imaging units (two imaging units, the first imaging unit 1a and the second imaging unit 1b in this example), and composite image information of the celestial sphere of the imaging device can be acquired.

[0050] In addition, distances can be measured in all directions (360° horizontally) to obtain the distance to a specific object existing around the image capture device 100. This makes it possible to realize the image processing system 200 that acquires composite image information of the entire celestial sphere and distance measurement information in all directions from the image capture device 100 while suppressing an increase in the number of parts and reducing costs and power consumption required for manufacturing, installation, and operation.

[0051] (Embodiment 2) A second embodiment of the present invention will be described below. In the second embodiment, the optical characteristics of the optical system of the first embodiment are limited to improve the distance measurement accuracy of the image processing system 200.

[0052] That is, in embodiment 2, the first optical system 2a and the second optical system 2b of the first imaging unit 1a and the second imaging unit 1b are optical systems that generate an image circle having a characteristic that the resolution at the peripheral angle of view is higher than the resolution near the center of the optical axis.

[0053] This makes it possible to increase the number of pixels in the trimmed images output from the trimming processing units 404 and 405 as shown in Fig. 4. Also, distance measurement errors and variations can be reduced, and disparity can be calculated and distance measurement can be performed with high accuracy.

[0054] Figures 9(A) to (C) are diagrams for explaining the characteristics of the optical system in embodiment 2. In embodiment 2, the first optical systems 2a and 2b have approximately the same characteristics. In the following, the characteristics of the first optical system 2a will be described as an example.

[0055] 9(A) is a side view of the angles of view of a first imaging unit 1a and a second imaging unit 1b installed in an imaging device 100. The first imaging unit 1a has a first optical system 2a and an imaging element 3a as shown in FIG. 1. The first optical system 2a has different imaging magnifications at a first angle of view (first field of view) 901 and a second angle of view (second field of view) 902 that is closer to the periphery than the first angle of view 901.

[0056] The imaging surface (light receiving surface) of the image sensor 3a includes a first region for capturing an image of an object included in a first angle of view 901, and a second region for capturing an image of an object included in a second angle of view 902. The number of pixels per unit angle of view in the second region is greater than the number of pixels per unit angle of view in the first region.

[0057] In other words, the resolution of the first imaging unit 1a at the second angle of view (second region) is higher than the resolution at the first angle of view (first region). Similarly, the resolution of the second imaging unit 1b at the second angle of view (second region) is higher than the resolution at the first angle of view (first region).

[0058] Fig. 9(B) is a diagram showing the image height y at each half angle of view on the imaging surface (light receiving surface) of the image sensor 3a in the form of contour lines. Fig. 9(C) is a diagram showing the relationship between the half angle of view θ and the image height y (projection characteristics of the first optical system 2a) in the first quadrant of Fig. 9(B).

[0059] 9C, the first optical system 2a is configured such that its projection characteristic y(θ) differs between angles of view less than a predetermined half angle of view θc and angles of view equal to or greater than the half angle of view θc. Therefore, when the increase in image height y per unit of half angle of view θ is defined as the resolution, the resolution of the first optical system 2a differs depending on the angle of view (the area on the light receiving surface of the image sensor).

[0060] This local resolution can be expressed as the differential value dy(θ) / dθ of the projection characteristic y(θ) at the half angle of view θ. For example, the greater the slope of the projection characteristic y(θ) in Fig. 9(C), the higher the resolution. Also, Fig. 9(B) shows that the greater the interval between the contour lines of the image height y at each half angle of view, the higher the resolution.

[0061] The first optical system 2a of the second embodiment has a projection characteristic in which the rate of increase of image height y (the slope of the projection characteristic y(θ) in FIG. 9(C)) is small in the central region near the optical axis, and the rate of increase of image height y becomes larger as the angle of view becomes larger in the peripheral region outside the central region.

[0062] In Fig. 9B, a first region 903 including the center corresponds to an angle of view less than the half angle of view θc, and a second region 904 outside the first region corresponds to an angle of view equal to or greater than the half angle of view θc. The angle of view less than the half angle of view θc corresponds to the first angle of view 901 in Fig. 9A, and the angle of view equal to or greater than the half angle of view θc corresponds to the second angle of view 902 in Fig. 9A.

[0063] Similarly, the angle of view of the second optical system 2b of the second imaging unit 1b that is less than the half angle of view θc corresponds to the first angle of view 901 in Fig. 9A, and the angle of view that is equal to or greater than the half angle of view θc corresponds to the second angle of view 902 in Fig. 9A. Note that at least a portion of the overlapping area (e.g., a portion between 85° and 95° vertically) of the first panoramic image 500a and the second panoramic image 500b is set to be equal to or greater than the half angle of view θc. Note that it is desirable that the entire overlapping area be equal to or greater than the half angle of view θc.

[0064] As described above, the first region 903 is a relatively low-resolution region, and the second region 904 is a relatively high-resolution region. Also, the first region 903 is a high-distortion region with relatively more distortion, and the second region 904 is a low-distortion region with relatively less distortion. Therefore, in the second embodiment, the first region 903 may be called a low-resolution region or a high-distortion region, and the second region 904 may be called a high-resolution region or a low-distortion region.

[0065] Note that the characteristics shown in Fig. 9 are only an example and are not limited to this. For example, the low-resolution area and the high-resolution area of ​​the optical system do not have to be configured as concentric circles, and each area may have a distorted shape. Also, the center of gravity of the low-resolution area and the center of gravity of the high-resolution area do not have to coincide with each other.

[0066] In addition, the centers of gravity of the low-resolution region and the high-resolution region may be shifted from the center of the light receiving surface of the image sensor. In the optical system of the second embodiment, it is sufficient that the low-resolution region is formed near the optical axis, and the high-resolution region is formed on the peripheral side away from the optical axis (i.e., outside the low-resolution region).

[0067] The first optical system 2a and the second optical system are configured to satisfy the following formula 2, where f is the focal length of each of them, θ is the half angle of view, y is the image height on the image plane, y(θ) is a projection characteristic that indicates the relationship between the image height y and the half angle of view θ, and θmax is the maximum half angle of view possessed by the optical system. That is, the first optical system 2a is configured so that the projection characteristic y(θ) is different from 2ftan(θ / 2) (stereoscopic projection method).

[0068] 0.2<2×f×tan(θmax / 2) / y(θmax)<0.92 (Formula 2) In an optical system having such optical characteristics, the magnification in the radial direction with respect to the optical axis can be adjusted by adjusting the projection characteristic y(θ).

[0069] This makes it possible to control the aspect ratio in the radial and circumferential directions relative to the optical axis, making it possible to obtain a high-resolution image with little distortion in the peripheral areas despite a wide angle of view, unlike conventional fisheye lenses.

[0070] Moreover, by satisfying Expression 2, the resolution in the second region 904 can be made higher than that of an optical system using the stereoscopic projection method. Note that, if the upper limit of Expression 2 is exceeded, the resolution in the second region 904 becomes low, and the difference in resolution with respect to the first region 903 becomes small, which is not preferable.

[0071] Moreover, if the lower limit of the formula 2 is not satisfied, it becomes difficult to satisfactorily correct various aberrations such as curvature of field. Note that the above formula 2 is only one example, and the optical system in the second embodiment is not limited to this.

[0072] By configuring the optical system as described above, high resolution can be obtained in the peripheral second angle of view 902 region, while the increase in image height y per unit half angle of view θ can be reduced in the low resolution region, and imaging can be performed at a wide angle of view. In other words, high resolution can be obtained in the high resolution region while the imaging range is a wide angle of view equivalent to that of a fisheye lens.

[0073] In the second embodiment, in the high-resolution region, the projection characteristics are approximated to the central projection method (y=f×tanθ) and the equidistant projection method (y=f×θ), which are the projection characteristics of a general imaging optical system. Therefore, in the high-resolution region, the optical distortion is small, and it is possible to generate a fine image.

[0074] By arranging the first imaging unit 1a and the second imaging unit 1b having the optical characteristics described above as described above, it is possible to capture high-resolution images of the peripheral angle of view of the imaging device 100. In addition, since the high-resolution areas of the first imaging unit 1a and the second imaging unit 1b overlap with the overlapping areas of the two imaging units, it is possible to obtain distance measurement information with a higher resolution than that of a conventional fisheye lens.

[0075] (Embodiment 3) In the third embodiment, a form will be described in which the imaging device 100 is mounted on, for example, an autonomous robot, which is a type of moving object.

[0076] 10 is a perspective view of the exterior of a moving body 1001 according to embodiment 3. The imaging device 100 is provided on the upper part of the moving body 1001 via a support 1000.

[0077] 11 is a functional block diagram showing a configuration example of an autonomous driving system according to embodiment 3. The autonomous driving system includes an image processing system 200 and an autonomous driving processing system 1100. A moving object 1001 includes the autonomous driving system and is capable of autonomous driving.

[0078] In the third embodiment, the moving object is equipped with at least a first imaging unit and a second imaging unit. The other parts of the image processing system may be provided in, for example, an external server. The configuration of the image processing system 200 is the same as that of the first embodiment, and therefore the description thereof will be omitted.

[0079] Next, a description will be given of the autonomous driving processing system 1100. A map creation unit 1101 creates a map 1102 required for estimating the self-position and orientation and generating a route, based on the distance information 206 created by the image processing system 200.

[0080] The map 1102 is three-dimensional spatial data such as three-dimensional point cloud data and coordinate data of characteristic points that represent the characteristics of structures such as objects existing in space, floors, walls, and ceilings of buildings, etc. The map 1102 can be updated at any time while the mobile object 1001 is traveling.

[0081] The self-position and orientation estimation unit 1103 estimates the position and orientation of the moving object 1001 based on the distance information 206 and the map 1102 while the moving object 1001 is moving.

[0082] A driving route generation unit 1104 generates a driving route based on the distance information 206, the output of the object detection unit 208, the self-position and orientation estimation result by the self-position and orientation estimation unit 1103, and the detection result of the map 1102. A mobile object control unit 1105 controls the moving direction and speed of the mobile object 1001 based on the driving route generated by the driving route generation unit 1104. A drive unit 1106 drives the wheels of the mobile object 1001, etc.

[0083] As described above, the moving body 1001 equipped with the autonomous driving system including the imaging device 100 in the third embodiment performs autonomous driving based on spherical composite image information and omnidirectional distance information of the moving body 1001 acquired by the imaging device 100. Therefore, it is possible to realize the autonomous driving of the moving body 1001 while suppressing an increase in the number of parts provided in the moving body 1001 and suppressing the costs and energy required for manufacturing, installation, and operation.

[0084] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to the above-mentioned embodiments, and various modifications and combinations of the above-mentioned embodiments are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The above-mentioned embodiments include the following combinations.

[0085] (Configuration 1) An image processing system comprising: a first imaging unit having a first optical system with a maximum half angle of view of θa; and a second imaging unit having a second optical system with a maximum half angle of view of θb, configured so as to satisfy θa+θb>180°, wherein an optical axis of the first optical system and an optical axis of the second optical system are installed in opposite directions to each other; an image synthesis unit that generates synthetic image information of a celestial sphere based on outputs from the first imaging unit and the second imaging unit; and a ranging processing unit that generates distance information based on outputs from a viewing angle at which the first imaging unit and the second imaging unit overlap.

[0086] (Configuration 2) The image processing system according to configuration 1, wherein at least one of the first optical system and the second optical system is an optical system with a variable angle of view.

[0087] (Configuration 3) The image processing system according to configuration 1, characterized in that when the focal length of each of the first optical system and the second optical system is f, the image height is y, and the half angle of view is θ, the projection characteristic y(θ) satisfies the following condition: 0.2<2×f×tan(θmax / 2) / y(θmax)<0.92.

[0088] (Configuration 4) The image processing system described in any one of configurations 1 to 3, characterized in that the distance measurement processing unit has a cropping processing unit that cuts out an image area of ​​a field of view where the first imaging unit and the second imaging unit overlap, and a distance calculation unit that generates the distance information based on the output of the cropping processing unit.

[0089] (Configuration 5) The image processing system according to any one of configurations 1 to 4, further comprising a detection unit that detects objects present in the surroundings based on the images captured by the first imaging unit and the second imaging unit and the distance information.

[0090] (Configuration 6) A moving body comprising the first imaging unit and the second imaging unit of the image processing system according to any one of configurations 1 to 5.

[0091] (Method) An image processing method using an imaging device having a first imaging unit with a first optical system having a maximum half angle of view of θa, and a second imaging unit with a second optical system having a maximum half angle of view of θb, where θa+θb>180° is satisfied and the optical axis of the first optical system and the optical axis of the second optical system are installed in opposite directions to each other, the image processing method generating composite image information of the celestial sphere based on outputs of the first imaging unit and the second imaging unit, and generating distance information based on outputs of a viewing angle at which the first imaging unit and the second imaging unit overlap.

[0092] (Program) A computer program for controlling each part of the image processing system according to any one of configurations 1 to 5 or the moving object according to configuration 6 by a computer.

[0093] In order to realize a part or all of the control in the above-described embodiment, a computer program for realizing the functions of the above-described embodiment may be supplied to an image processing system or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the image processing system or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0094] 100: Imaging device 200: Image processing system 1a: First imaging unit 1b: Second imaging unit 2a: First optical system 2b: Second optical system 3a: imaging element of first imaging unit 1a 3b: imaging element of second imaging unit 1b 5a: Optical axis of the first imaging unit 1a 5b: Optical axis of the second imaging unit 1b

Claims

1. a first imaging unit including a first optical system having a maximum half angle of view of θa; a second imaging unit including a second optical system having a maximum half angle of view of θb; θa+θb>180° The above is configured to satisfy the above. The optical axis of the first optical system and the optical axis of the second optical system are arranged in opposite directions to each other, an image synthesis unit that generates synthetic image information of a celestial sphere based on outputs of the first imaging unit and the second imaging unit; a distance measurement processing unit that generates distance information based on an output of a viewing angle at which the first imaging unit and the second imaging unit overlap.

2. 2. The image processing system according to claim 1, wherein at least one of the first optical system and the second optical system is an optical system with a variable angle of view.

3. When the focal length of each of the first optical system and the second optical system is f, the image height is y, and the half angle of view is θ, the projection characteristic y(θ) satisfies the following condition: 0.2<2×f×tan(θmax / 2) / y(θmax)<0.92 2. The image processing system according to claim 1, wherein the following is satisfied:

4. The distance measurement processing unit includes a trimming processing unit that cuts out an image area of ​​a viewing angle where the first imaging unit and the second imaging unit overlap; 2. The image processing system according to claim 1, further comprising: a distance calculation unit that generates the distance information based on an output of the trimming processing unit.

5. 2. The image processing system according to claim 1, further comprising a detection unit that detects an object existing in the surroundings based on the images captured by the first imaging unit and the second imaging unit and the distance information.

6. 6. A moving body comprising the first imaging unit and the second imaging unit of the image processing system according to claim 1.

7. a first imaging unit including a first optical system having a maximum half angle of view of θa; a second imaging unit including a second optical system having a maximum half angle of view of θb; θa+θb>180° Fulfilling An image processing method using an imaging device in which an optical axis of the first optical system and an optical axis of the second optical system are arranged in opposite directions to each other, generating composite image information of a celestial sphere based on outputs of the first imaging unit and the second imaging unit; An image processing method comprising: generating distance information based on outputs of a viewing angle where the first imaging unit and the second imaging unit overlap.

8. A computer program for controlling each part of the image processing system according to any one of claims 1 to 5 or the moving body according to claim 6 by a computer.

Images