Image processing system, mobile object, image processing method, and computer program

Abstract

To provide an image processing system by which displacement is less likely to occur on a subject at a boundary part of a combined image.SOLUTION: An image processing system includes: a first imaging device with an optical system whose angle of field is 180° or more; a second imaging device with an optical system whose angle of field is 180° or more; a recognition part that recognizes an object from images taken by the first imaging device and the second imaging device; a calculation part that calculates a distance up to the object recognized by the recognition part; and an image combination part that combines the images after subjected to planar expansion at a boundary part determined on the basis of the distance calculated by the calculation part.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing system, a moving object, an image processing method, a computer program, and the like. [Background technology]

[0002] Conventionally, there are known autonomous mobile bodies that move in various places such as homes, office buildings, houses, logistics centers, etc. Among such autonomous mobile bodies, there are some that provide an operator with an image captured by an imaging means provided in the mobile body.

[0003] For example, Patent Document 1 discloses a technique for capturing spherical images of the surroundings of a moving object using two fisheye lenses and synthesizing the captured images in order to obtain the surrounding conditions. Patent Document 2 discloses a technique for matching feature points of the captured images to naturally connect them when synthesizing the captured images. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-27390 [Patent Document 2] JP 2020-204874 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology disclosed in the above-mentioned patent document, when a texture that is easy to match is present in the image, this may cause deviations in important subject areas such as people.

[0006] Therefore, one object of the present invention is to provide an image processing system that is less likely to cause deviation of a subject at the boundary of a synthesized image. [Means for solving the problem]

[0007] An image processing system according to one aspect of the present invention comprises: a first imaging device having an optical system with a viewing angle of 180° or more; a second imaging device having an optical system with a viewing angle of 180° or more; a recognition unit that recognizes an object from images captured by the first imaging device and the second imaging device; a calculation unit that calculates a distance to the object recognized by the recognition unit; The image processing device further comprises an image synthesis unit that synthesizes the images developed on a plane at a boundary portion determined based on the distance calculated by the calculation unit. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing system in which deviation of a subject is unlikely to occur at the boundary of a synthesized image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram of a moving body 1 having an autonomous driving system 100 according to a first embodiment of the present invention, and imaging devices 21 and 22 arranged thereon. [Diagram 2] 1A and 1B are diagrams illustrating optical characteristics of the optical system of an image pickup device 21 according to a first embodiment of the present invention. [Diagram 3] 1 is a diagram illustrating an example of the configuration of an autonomous driving system 100 according to a first embodiment of the present invention. [Figure 4] FIG. 2 is an enlarged view of the vicinity of imaging devices 21 and 22 according to the first embodiment of the present invention. [Diagram 5] 1A to 1D are diagrams illustrating an example of a panoramic image according to the first embodiment of the present invention. [Figure 6] 4 is a flowchart showing a process flow of the image synthesis unit 25 according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a diagram illustrating a method for determining an angle of view used for image synthesis according to the first embodiment of the present invention. [Figure 8]1A to 1C are diagrams illustrating a boundary portion of a panoramic image according to the first embodiment of the present invention. [Figure 9] 13(A) to 13(C) are diagrams illustrating a boundary portion of a panoramic image according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In the drawings, the same members or elements are given the same reference numerals, and duplicated descriptions thereof will be omitted or simplified.

[0011] (Embodiment 1) 1 is a schematic diagram of a moving body 1 having an autonomous driving system 100 according to a first embodiment of the present invention, and image capturing devices 21 and 22 arranged thereon. In the first embodiment, the image capturing device 21 is installed on the moving body 1 by a support part 105 so that the optical axis faces upward. That is, the first image capturing device is arranged facing vertically upward.

[0012] Moreover, the imaging device 22 is installed so that its optical axis faces the ground direction, which is the opposite direction to the optical axis of the imaging device 21. That is, the second imaging device is installed facing vertically downward. The imaging devices 21 and 22 are part of the elements that make up the image processing system 2 described later, and the imaging device 21 functions as the first imaging device, and the imaging device 22 functions as the second imaging device.

[0013] The optical axes of the imaging devices 21 and 22 are arranged to overlap on the same straight line. This arrangement can reduce positional deviation between the images when the image synthesis unit 25 (described later) synthesizes the images captured by the imaging devices 21 and 22.

[0014] In this embodiment, the optical systems of the imaging devices 21 and 22 have a characteristic of generating an image circle having a higher resolution at the peripheral angle of view compared to the vicinity of the center of the optical axis, and form an image on the light receiving surface of an imaging element described later. In this embodiment, the optical systems of the imaging devices 21 and 22 have approximately the same characteristics, and the characteristics of the optical system of the imaging device 21 will be described below as an example.

[0015] The imaging device 21 includes an optical system 21a and an imaging element 21b. The optical system 21a has a characteristic that the imaging magnification differs between a first angle of view (first field of view) 10a and a second angle of view (second field of view) 10b that is closer to the periphery than the first angle of view 10a.

[0016] The imaging surface (light receiving surface) of the imaging element 21b includes a first region for imaging an object included in the first angle of view 10a and a second region for imaging an object included in the second angle of view 10b. At this time, the number of pixels per unit angle of view in the second region is greater than the number of pixels per unit angle of view in the first region. In other words, the resolution of the imaging device 21 at the second angle of view (second region) is higher than the resolution at the first angle of view (first region).

[0017] The combined angle of view of the first angle of view 10a and the second angle of view 10b corresponds to the imaging angle of view of the imaging device 21, which is 180° or more. The imaging angle of view of the imaging device 22 is also 180° or more, and the imaging angles of view of the imaging devices 21 and 22 overlap in the peripheral portion.

[0018] That is, the imaging device 21 as the first imaging device has an optical system with a viewing angle of 180° or more, and the imaging device 22 as the second imaging device also has an optical system with a viewing angle of 180° or more. Therefore, the surroundings of the image processing system 2 can be photographed in a wide range (360° in this embodiment).

[0019] 2A and 2B are diagrams illustrating the optical characteristics of the optical system of the imaging device 21 according to the first embodiment of the present invention. Fig. 2A is a diagram showing the image height y at each half angle of view on the imaging surface (light receiving surface) of the imaging element 21b in the form of contour lines. Fig. 2B is a diagram showing the relationship between the half angle of view θ and the image height y (projection characteristics of the optical system 21a) in the first quadrant of Fig. 2A.

[0020] 2B, the optical system 21a is configured such that its projection characteristic y(θ) differs between angles of view less than a predetermined half angle of view θa and angles of view equal to or greater than the half angle of view θa. That is, the optical system 21a is configured such that, when the increase in image height y per unit half angle of view θ is defined as the resolution, the resolution differs depending on the angle of view (the area on the light receiving surface of the image sensor).

[0021] This local resolution can be expressed as the differential value dy(θ) / dθ of the projection characteristic y(θ) at the half angle of view θ. For example, the greater the slope of the projection characteristic y(θ) in Fig. 2(B), the higher the resolution. Also, Fig. 2(A) shows that the greater the interval between the contour lines of the image height y at each half angle of view, the higher the resolution.

[0022] The optical system 21a of this embodiment has a projection characteristic in which the rate of increase of the image height y (the slope of the projection characteristic y(θ) in FIG. 2(B)) is small in the central region near the optical axis, and the rate of increase of the image height y becomes larger as the angle of view becomes larger in the peripheral region outside the central region.

[0023] 2A, a first region 21c including the center corresponds to an angle of view less than the half angle of view θa, and a second region 21d outside the first region corresponds to an angle of view equal to or greater than the half angle of view θa. The angle of view less than the half angle of view θa corresponds to the first angle of view 10a in FIG. 1, and the angle of view equal to or greater than the half angle of view θa corresponds to the second angle of view 10b in FIG. 1.

[0024] 1. Similarly, the angle of view less than the half angle of view θa of the optical system 22a of the imaging device 22 corresponds to the angle of view 11a in FIG. 1, and the angle of view equal to or greater than the half angle of view θa corresponds to the angle of view 11b in FIG.

[0025] As described above, the first region 21c is a relatively low-resolution region, and the second region 21d is a relatively high-resolution region. Also, the first region 21c is a high-distortion region with relatively more distortion, and the second region 21d is a low-distortion region with relatively less distortion. Therefore, in this embodiment, the first region 21c may be called a low-resolution region or a high-distortion region, and the second region 21d may be called a high-resolution region or a low-distortion region.

[0026] The characteristics shown in Fig. 2 are merely an example, and are not limited to these. For example, the low-resolution area and the high-resolution area of ​​the optical system do not have to be configured as concentric circles, and each area may have a distorted shape. Also, the center of gravity of the low-resolution area and the center of gravity of the high-resolution area do not have to coincide with each other.

[0027] In addition, the centers of gravity of the low-resolution region and the high-resolution region may be offset from the center of the light receiving surface of the image sensor. In the optical system of this embodiment, it is sufficient that the low-resolution region is formed near the optical axis, and the high-resolution region is formed on the peripheral side (outside the low-resolution region) away from the optical axis.

[0028] The optical system 21a is configured to satisfy the following formula 1, where f is the focal length, θ is the half angle of view, y is the image height on the image plane, y(θ) is a projection characteristic that indicates the relationship between the image height y and the half angle of view θ, and θmax is the maximum half angle of view of the optical system. That is, the optical system 21a is configured so that the projection characteristic y(θ) is different from 2ftan(θ / 2) (stereoscopic projection method). 0.2<2×f×tan(θmax / 2) / y(θmax)<0.92 (Formula 1)

[0029] In this way, when the focal length of each of the optical systems of the first imaging device and the second imaging device is f, the image height is y, and the half angle of view is θ, the projection characteristic y(θ) is set to satisfy the above formula 1. Furthermore, in an optical system having such optical characteristics, the magnification in the radial direction with respect to the optical axis can be adjusted by adjusting the projection characteristic y(θ).

[0030] This makes it possible to control the aspect ratio in the radial and circumferential directions relative to the optical axis, making it possible to obtain a high-resolution image with little distortion in the peripheral areas despite a wide angle of view, unlike conventional fisheye lenses.

[0031] Moreover, by satisfying formula 1, the resolution in the second region 21d can be made higher than that of a stereoscopic projection optical system. However, if the upper limit of formula 1 is exceeded, the resolution in the second region 21d becomes low, and the difference with the resolution in the first region 21c becomes small, which is not preferable.

[0032] Moreover, if the lower limit of the formula 1 is not satisfied, it becomes difficult to satisfactorily correct various aberrations such as curvature of field. Note that the above formula 1 is only one example, and the optical system in this embodiment is not limited to this.

[0033] By configuring the optical system as described above, high resolution can be obtained in the high resolution region, while the increase in image height y per unit half angle of view θ can be reduced in the low resolution region, making it possible to capture a wider angle of view. Therefore, while providing an imaging range with a wide angle of view equivalent to that of a fisheye lens, high resolution can be obtained in the high resolution region.

[0034] In addition, in this embodiment, in the high resolution region (low distortion region), the projection characteristics are approximate to the central projection method (y=f×tanθ) and the equidistant projection method (y=f×θ), which are the projection characteristics of a general imaging optical system.

[0035] Therefore, in the high-resolution region (low-distortion region), optical distortion is small, and it is possible to generate a fine image. By arranging the imaging devices 21 and 22 having the optical characteristics described above as described above, it is possible to capture the surroundings of the moving object 1 with high resolution.

[0036] Fig. 3 is a diagram showing a configuration example of the autonomous driving system 100 according to the first embodiment of the present invention. Some of the functional blocks shown in Fig. 3 are realized by causing a CPU or the like as a computer (not shown) included in the autonomous driving system 100 to execute a computer program stored in a memory (not shown) as a storage medium.

[0037] However, some or all of these may be realized by hardware. For the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used. Furthermore, the functional blocks shown in Fig. 3 do not have to be built into the same housing, and may be configured by separate devices connected to each other via signal paths.

[0038] The autonomous driving system 100 according to this embodiment includes an image processing system 2, a map data generating device 3, an autonomous driving control device 4, a display device 5, etc. In this embodiment, the autonomous driving system 100 is installed in a moving body 1, thereby enabling the moving body 1 to perform autonomous driving. Although the moving body in this embodiment is equipped with at least a first imaging device and a second imaging device, a part of the autonomous driving system 100 may be provided in, for example, an external server.

[0039] The image processing system 2 has imaging devices 21 and 22, image processing units 23 and 24, and an image synthesis unit 25. The imaging device 21 has an optical system 21a and an imaging element 21b, and the imaging device 22 has an optical system 22a and an imaging element 22b.

[0040] The optical systems 21a and 22a of the imaging devices 21 and 22 are each composed of one or more optical lenses and have the optical characteristics shown in Figures 2(A) and 2(B). The optical systems 21a and 22a form (focus) their respective optical images on the light receiving surfaces of the imaging elements 21b and 22b, respectively.

[0041] The imaging elements 21b and 22b function as imaging means, and perform photoelectric conversion of the formed optical image to output an imaging signal. Note that, for example, R, G, and B color filters are arranged for each pixel in a Bayer array or the like on the light receiving surface of the imaging elements 21b and 22b, and R, G, and B pixel signals are sequentially output from the imaging devices 21 and 22, respectively, as imaging signals.

[0042] The image processing units 23 and 24 perform various image processing such as white balance adjustment, demosaic processing, gain / offset adjustment, gamma correction, and color correction on the imaging signals output from the imaging devices 21 and 22, respectively.

[0043] The image synthesis unit 25 performs panoramic expansion on the images captured by the imaging devices 21 and 22, respectively, and then synthesizes them into one image to create a synthetic image. That is, the images captured by the imaging device 21 as the first imaging device and the imaging device 22 as the second imaging device are expanded on a plane, and then a synthetic image is created.

[0044] The panoramic development method may be based on the equirectangular projection or the Mercator projection, and the present invention is not limited to this method. The details of the composite image creation process performed by the image composition unit 25 will be described later with reference to Figs. 4 to 9.

[0045] The map data generating device 3 has a distance measuring sensor 31 using LiDAR (Light Detection And Ranging) or the like, and a map creating unit 32. The map creating unit 32 creates a map 33 required for estimating the self-position and orientation and generating a route, based on the measurement values ​​of the distance measuring sensor 31.

[0046] The map 33 is three-dimensional spatial data such as three-dimensional point cloud data and coordinate data of characteristic points that represent the characteristics of structures such as objects existing in space, floors, walls, and ceilings of buildings, etc. The map data generating device 3 can update the map 33 at any time while the mobile object 1 is traveling.

[0047] The autonomous driving control device 4 is a device mounted on the moving body 1 and drives the moving body 1, and includes a detection unit 41, a self-position and attitude estimation unit 42, a driving path generation unit 43, a moving body control unit 44, and a drive unit 45.

[0048] The detection unit 41 detects gestures as predetermined actions by image recognition. For example, a person raising their hand is detected as an action to call the moving object 1. It is also possible to detect gestures such as holding up a finger or winking, in addition to raising a hand. Alternatively, it is also possible to detect a person performing a suspicious action with a predetermined pattern.

[0049] The self-position and attitude estimation unit 42 estimates the position and attitude of the moving body 1 on which the autonomous driving control device 4 is mounted, based on the measurement values ​​of the distance measuring sensor 31 and the map 33 while the moving body 1 is moving.

[0050] The driving route generation unit 43 generates a driving route based on the measurement values ​​of the distance measurement sensor 31, the map 33, the self-position and orientation estimation result by the self-position and orientation estimation unit 42, and the recognition result by the detection unit 41. For example, when the detection unit 41 detects a person making an action of calling the moving object 1, a driving route is generated with the position of the detected person as a destination point.

[0051] The moving body control unit 44 controls the moving direction, speed, etc. of the moving body 1 equipped with the autonomous driving control device 4 based on the output result of the driving route generation unit. The drive unit 45 drives the wheels, etc. of the moving body 1 equipped with the autonomous driving control device 4.

[0052] Furthermore, the map 33 can be accessed from both the map data generation device 3 and the autonomous driving control device 4, and may be stored in either the map data generation device 3 or the autonomous driving control device 4, or may be stored in both.

[0053] The display device 5 has an image display unit 51 such as a liquid crystal monitor that displays an image captured by the image processing system 2, and displays an image on the image display unit 51, in which images captured by the two imaging devices are synthesized into a single image by the image synthesis unit 25. In this embodiment, the display device 5 is installed on the mobile object 1, but it may be installed in a location separate from the mobile object 1 and configured to receive and display an image transferred from the mobile object 1 to be displayed via wireless communication.

[0054] An example of the composite image creation process by the image composition unit 25 will be described with reference to Fig. 4 to Fig. 9. Fig. 4 is an enlarged view of the vicinity of the imaging devices 21 and 22 according to the first embodiment of the present invention. In this embodiment, the imaging devices 21 and 22 are installed at positions separated from each other in the vertical direction with respect to the ground, and the inter-device distance 400 between the imaging devices 21 and 22 is, for example, about 150 mm.

[0055] In addition, the imaging devices 21 and 22 are each capable of capturing an image with an angle of view of 180° or more, and Θ1 and Θ2 are each, for example, 184°. Since the imaging devices 21 and 22 each have an angle of view of 180° or more, there is an overlapping area (401, 402) captured by both the imaging devices 21 and 22.

[0056] 5(A) to (D) are diagrams for explaining examples of panoramic images according to the first embodiment of the present invention, and Fig. 5(A) is a diagram showing an example of an image 500 in the sky direction of the image processing system 2 captured by the imaging device 21. Furthermore, 501 represents the outer periphery of the image circle of the image 500, and 502 represents the center point of the image circle of the image 500. The image circle of the image 500 is represented by a circle of radius r centered on the center point 502.

[0057] 5(B) is a diagram showing an example of an image 510 of the image processing system 2 in the ground direction captured by the imaging device 22, in which 511 indicates the outer periphery of the image circle of the image 510 and 512 indicates the center point of the image circle of the image 510. The images 500 and 510 are images formed on the imaging elements 21b and 22b by the optical systems 21a and 22a of the imaging devices 21 and 22 described above.

[0058] For example, the ceiling C1 and the moving object 1 itself in the images 500 and 510 are imaged at a position corresponding to the first area 21c described in Fig. 2(A). On the other hand, the person P1 present in the horizontal peripheral position of the moving object 1 is imaged at a position corresponding to the second area 21d.

[0059] That is, in the images 500 and 510, the ceiling C1 and the moving object 1 itself that exist in the sky or on the ground above the image processing system 2 are captured with lower resolution than an image captured by an imaging device using a fisheye lens with equidistant projection characteristics. On the other hand, objects that exist in the horizontal peripheral direction of the image processing system 2, such as person P1, are captured with higher resolution and less distortion than an image captured by an imaging device using a fisheye lens with equidistant projection characteristics.

[0060] 5(C) is a diagram showing an example of image 500P obtained by panorama expansion of image 500. In image 500P, only the face of person P1 present at the horizontal periphery is recorded, and the part below the face is not recorded.

[0061] 5(D) is a diagram showing an example of image 510P obtained by panorama expansion of image 510. In image 510P, the lower part of the face of person P1 present at the horizontal periphery is recorded, and the upper part of the face is not recorded. In addition, since the angles of view of images 500 and 510 are each 180° or more, the area from around the eyebrows to around the neck of person (P1) is recorded in both images 500 and 510.

[0062] Fig. 6 is a flowchart showing a processing flow of the image synthesis unit 25 according to the first embodiment of the present invention. Note that the operation of each step in the flowchart of Fig. 6 is performed sequentially by a CPU serving as a computer in the image processing system executing a computer program stored in a memory.

[0063] First, in step S601, the CPU of the image processing system performs the panoramic development described with reference to Fig. 5 on the images captured by the image capture devices 21 and 22. That is, the images captured by the first image capture device and the second image capture device are planarly developed. Note that the method of this panoramic development is not limited to the above and may be based on the equirectangular projection or the Mercator projection, as described above.

[0064] Next, the process proceeds to step S602 (recognition step: recognition section), where the CPU of the image processing system detects a human face from the panoramic images such as 500P and 510P in FIG. 5. That is, the object is recognized from the planar image. The coordinate position and size of the detected face are also saved. Next, the process proceeds to step S603, where the CPU of the image processing system determines whether or not a face is present at the boundary (seam) where the images captured by the imaging devices 21 and 22 are combined.

[0065] The method of determination is to refer to the coordinate position and size of the face saved in step S602 and determine whether or not a face is present near the outer periphery 501 in Fig. 5(C) and the outer periphery 511 in Fig. 5(D). For example, if any part of a face (hair, eyebrows, eyes, nose, mouth, etc.) is present within a predetermined number of pixels from the outer periphery 501 in Fig. 5(C) and the outer periphery 511 in Fig. 5(D), it is determined that a face is present.

[0066] If it is determined in step S603 that a face is present, the CPU of the image processing system calculates and sets the distance D to the face in step S604 (calculation step: calculation unit). That is, the distance to the recognized object is calculated.

[0067] The distance D is calculated by, for example, a method of calculating disparity by stereo matching. Specifically, the disparity is calculated by stereo matching of feature points of pixels of the face of a person (P1) recorded near both the outer periphery 501 in Fig. 5(C) and the outer periphery 511 in Fig. 5(D). That is, the distance to the recognized object is calculated by stereo matching between the object in the image captured by the first imaging device and the object in the image captured by the second imaging device.

[0068] Since stereo matching is widely known, a description thereof will be omitted here. It is also possible to obtain the distance D using a sensor such as LiDAR. That is, the distance to a recognized object may be measured based on information on the reflected light by irradiating the object with a laser beam.

[0069] On the other hand, if it is determined in step S603 that there is no face, then in step S608, the CPU of the image processing system sets the distance D to a specific value (predetermined value). This predetermined value may be any value, and may be, for example, 5 m.

[0070] After calculating the distance D to the face in step S604, the CPU of the image processing system determines in step S605 whether the distance D to the face has been calculated. If the distance D to the face has been calculated in step S605, the process proceeds to step S606, and if the distance D has not been calculated, the process proceeds to step S608.

[0071] An example of when the distance D cannot be calculated is when the brightness difference in the captured image is very small when the distance is calculated by stereo matching. Since stereo matching extracts feature points of an image to obtain parallax information, when the brightness difference in the image is small, the feature points of the image cannot be extracted.

[0072] If the distance D to the face can be calculated in step S605, in step S606, the CPU of the image processing system determines the boundary when synthesizing images based on the distance D calculated in step S604. In this step S606, in order to determine the boundary (seam), the angle of view to be used for synthesis among the images captured by the image capture devices 21 and 22 is determined. The method of determination will be described with reference to FIG.

[0073] 7 is a diagram for explaining a method for determining an angle of view used for compositing images according to the first embodiment of the present invention. Since the distance D to the face of person P1 is known in step S604, and the value of the difference 400 in the distance between the image capture devices 21 and 22 is also known, if the value of 400 is B, the value of Θ in FIG. 7 can be calculated by Equation 2.

[0074]

number

[0075] When Θ is found using Equation 2, the angle of view to be used for synthesis is determined from among the images captured by the image capture devices 21 and 22. Once the angle of view is determined, the pixels corresponding to Θ in the panoramic images 500P and 510P described in FIG. 5 are determined based on the projection characteristics described in FIG.

[0076] The pixel corresponding to this Θ is determined as the boundary part. Next, in step S607 (combining step), the CPU of the image processing system combines the images at the boundary part determined in step S606. That is, the two images developed on a plane are combined at the boundary part determined based on the calculated distance. A detailed example of image combination will be described with reference to Figs. 8(A) to (C).

[0077] 8(A) to 8(C) are diagrams for explaining the boundary portion of a panoramic image according to the first embodiment of the present invention, and for explaining image synthesis of images 500P and 510P. Fig. 8(A) is a diagram showing the boundary portion of an image captured by the imaging device 21, determined in step S606. 801 is the boundary portion determined in step S606. 500PA is an image for synthesis, and indicates the image to be used for synthesis.

[0078] 8B is a diagram showing the boundary portion of the image captured by the imaging device 22, which was determined in step S606. Reference numeral 811 denotes the boundary portion determined in step S606. Reference numeral 510PA denotes an image for synthesis, which indicates an image to be used for synthesis.

[0079] Next, the process proceeds to step S607, where the CPU of the image processing system composites 500PA and 510PA, which are images within the range to be used for composite determined in step S606. The composite result is shown in FIG. 8C, and the face of person P1 can be composited and displayed without being misaligned. The dotted line 821 indicates the boundary. Note that the actual composite image is not displayed as a dotted line.

[0080] As described above, according to the first embodiment, by determining the boundary based on the distance to the important subject, it is possible to realize an image processing system that does not cause a shift in the important subject of the composite image.

[0081] (Embodiment 2) 9(A) to 9(C) are diagrams illustrating the boundary portion of a panoramic image according to the second embodiment of the present invention. Fig. 9(A) is an image captured by the imaging device 21, and shows the entire face of person P1. Fig. 9(B) is an image captured by the imaging device 22, and shows a part of the face of person P1 cut off.

[0082] Therefore, all pixels of image 500P captured by imaging device 21 are used, and pixels (500PB) below the face of the image captured by imaging device 22 are used to synthesize the images. In other words, the pixels between 901 and 512 are used for image synthesis. As a result, the face of person P1 can be displayed without being misaligned, as shown in FIG. 9(C).

[0083] The dotted line 902 indicates the boundary and is not displayed as a dotted line in the actual composite image. That is, in the second embodiment, the entire image captured by the first imaging device, including the recognized predetermined object, is used for composition, and the image captured by the second imaging device is used for composition except for the portion excluding the predetermined object. As a result, while the distance to the face of person P1 is calculated in the first embodiment, this is not necessary in the second embodiment, which has the advantage of reducing the processing load.

[0084] (Embodiment 3) In the first embodiment, an example in which one face is detected has been described. However, when a plurality of faces are detected, the synthesis process may be performed by adjusting the distance to the face that is closest.

[0085] Alternatively, the composition process may be performed according to the distance to the face that is displayed largest. In other words, when multiple faces are detected from a captured image, the distance to the closest face or the face that is displayed largest may be calculated, and the composition process may be performed according to that distance.

[0086] Also, a gesture may be detected by the detection unit 41, and the synthesis process may be performed by adjusting the distance to the person who performed the gesture. That is, a predetermined action may be detected by the detection unit, and the boundary may be determined based on the distance to the object that performed the predetermined action.

[0087] (Embodiment 4) In the first embodiment, a human face is detected and the facial portion is smoothly synthesized. That is, the object recognized from the captured image is a human face. However, it is not limited to a human face, and it may be configured to detect other objects. For example, it may be an airplane, a vehicle, a bicycle, a train, a model, or an animal.

[0088] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to the above-mentioned embodiments, and various modifications and combinations of the above-mentioned embodiments are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The above-mentioned embodiments include the following combinations.

[0089] (Configuration 1) An image processing system comprising: a first imaging device equipped with an optical system having a viewing angle of 180° or more; a second imaging device equipped with an optical system having a viewing angle of 180° or more; a recognition unit that recognizes an object from images captured by the first imaging device and the second imaging device; a calculation unit that calculates the distance to the object recognized by the recognition unit; and an image synthesis unit that synthesizes the images developed on a plane at a boundary determined based on the distance calculated by the calculation unit.

[0090] (Configuration 2) The image processing system according to configuration 1, wherein the recognition unit develops the images captured by the first imaging device and the second imaging device into a plane and recognizes an object.

[0091] (Configuration 3) The image processing system according to configuration 1 or 2, wherein the object recognized from the image is a human face.

[0092] (Configuration 4) The image processing system according to any one of configurations 1 to 3, characterized in that when multiple faces are detected from the image, the system calculates the distance to the face that is closest or the face that is displayed largest.

[0093] (Configuration 5) The image processing system described in any one of configurations 1 to 4, characterized in that the image synthesis unit uses the entire image captured by the first imaging device, which includes a recognized specified object, for synthesis, and uses the image captured by the second imaging device for synthesis, excluding the portion of the specified object.

[0094] (Configuration 6) An image processing system according to any one of configurations 1 to 5, further comprising a detection unit for detecting a predetermined motion, and the image synthesis unit determines the boundary area based on a distance to an object that has performed the predetermined motion.

[0095] (Configuration 7) An image processing system according to any one of configurations 1 to 6, characterized in that, when the focal length of each of the optical systems of the first imaging device and the second imaging device is f, the image height is y, and the half angle of view is θ, the projection characteristic y(θ) satisfies the following formula: 0.2<2×f×tan(θmax / 2) / y(θmax)<0.92.

[0096] (Configuration 8) The image processing system according to any one of configurations 1 to 7, wherein the first imaging device is disposed facing vertically upward, and the second imaging device is disposed facing vertically downward.

[0097] (Configuration 9) An image processing system according to any one of configurations 1 to 8, characterized in that the distance to the recognized object is calculated by stereo matching between the object in the image captured by the first imaging device and the object in the image captured by the second imaging device.

[0098] (Configuration 10) The image processing system according to any one of configurations 1 to 9, wherein the distance to the recognized object is measured based on information on reflected light by irradiating the object with laser light.

[0099] (Configuration 11) A moving body comprising the first imaging device and the second imaging device of the image processing system according to any one of configurations 1 to 10.

[0100] (Method) An image processing method using an image processing system having a first imaging device equipped with an optical system with a viewing angle of 180° or more, and a second imaging device equipped with an optical system with a viewing angle of 180° or more, the image processing method comprising: recognizing an object from images captured by the first imaging device and the second imaging device; calculating the distance to the recognized object; and synthesizing the images expanded on a plane at a boundary determined based on the calculated distance.

[0101] (Program) A computer program for controlling each part of the image processing system according to any one of configurations 1 to 10 or the moving object according to configuration 11 by a computer.

[0102] In order to realize a part or all of the control in the above-described embodiment, a computer program for realizing the functions of the above-described embodiment may be supplied to an image processing system or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the image processing system or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0103] 1: Moving object 2: Image processing system 3: Map data generator 4: Autonomous driving control device 5:Display device 10a: First angle of view of the imaging device 21 10b: Second angle of view of the imaging device 21 11a: First angle of view of the imaging device 22 11b: second angle of view of the imaging device 22 21: Imaging device 21a:Optical system 21b: Image sensor 22: Imaging device 22a:Optical system 22b: Image sensor 25: Image synthesis unit 31: Distance sensor 32: Map Creation Department 33: Map 41: Detection section 42:Self-position and orientation estimation unit 43: Route generation unit 44: Mobile control unit 45: Drive unit 51: Image display unit

Claims

1. A first imaging device having an optical system with a viewing angle of 180° or more; a second imaging device having an optical system with a viewing angle of 180° or more; a recognition unit that recognizes an object from images captured by the first imaging device and the second imaging device; a calculation unit that calculates a distance to the object recognized by the recognition unit; An image processing system comprising: an image synthesis unit that synthesizes the images developed on a plane at a boundary portion determined based on the distance calculated by the calculation unit.

2. 2. The image processing system according to claim 1, wherein the recognition unit performs planar development of the images captured by the first imaging device and the second imaging device to recognize an object.

3. 2. The image processing system according to claim 1, wherein the object recognized from the image is a human face.

4. 2. The image processing system according to claim 1, wherein when a plurality of faces are detected from the image, the distance to the face that is closest or the face that is displayed largest is calculated.

5. The image processing system according to claim 1, characterized in that the image synthesis unit uses the entire image captured by the first imaging device, which includes a recognized specified object, for synthesis, and uses the image captured by the second imaging device for synthesis, excluding the portion of the specified object.

6. A detection unit for detecting a predetermined operation is provided, 2. The image processing system according to claim 1, wherein the image synthesis unit determines the boundary portion based on a distance to an object that has performed the predetermined action.

7. When the focal length of each of the optical systems of the first imaging device and the second imaging device is f, the image height is y, and the half angle of view is θ, the projection characteristic y(θ) is expressed by the following formula: 0.2<2×f×tan(θmax / 2) / y(θmax)<0.92 2. The image processing system according to claim 1, wherein the following is satisfied:

8. 2. The image processing system according to claim 1, wherein the first imaging device is disposed facing vertically upward, and the second imaging device is disposed facing vertically downward.

9. 2. The image processing system according to claim 1, wherein the distance to the recognized object is calculated by stereo matching between the object in the image captured by the first imaging device and the object in the image captured by the second imaging device.

10. 2. The image processing system according to claim 1, wherein the distance to the recognized object is measured based on information on reflected light by irradiating the object with a laser beam.

11. 11. A moving body comprising the first imaging device and the second imaging device of the image processing system according to claim 1.

12. A first imaging device having an optical system with a viewing angle of 180° or more; and a second imaging device having an optical system with a viewing angle of 180° or more, An image processing method comprising: recognizing an object from images captured by the first imaging device and the second imaging device; calculating a distance to the recognized object; and synthesizing the planarly developed images at a boundary determined based on the calculated distance.

13. A computer program for controlling each part of the image processing system according to any one of claims 1 to 10 or the moving body according to claim 11 by a computer.

Images