Image processing apparatus, image processing method and program

Abstract

To enable displaying an image that is easily and visually recognized while suppressing increase of a circuit scale and cost, and delay.SOLUTION: An image processing apparatus acquires a first image in which an optical image based on an optical system including a high distortion area having large distortion and a low distortion area having small distortion is imaged, and acquires a second image having a part of an image area cut out from the first image. The image processing apparatus, then, expands / contracts the second image at an expansion / contraction magnification corresponding to distortion of the optical system, and outputs, to a display device, the expanded / contracted second image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing technology for generating an image to be displayed on a display device from a captured image.

Background Art

[0002] There is an image display system that displays a moving image captured by a camera attached to the rear of a vehicle such as an automobile on an in-vehicle display and an electronic rearview mirror. Hereinafter, a camera attached to the rear of a vehicle is referred to as a rear camera, and a moving image captured by the rear camera is referred to as a rear image. Here, the rear camera often has an optical system that can obtain a wide viewing angle. Since such a wide-angle optical system is likely to have a large distortion, the rear image obtained by the rear camera is subjected to distortion correction. In addition, there is also an image display system for the electronic rearview mirror that cuts out and displays an image of an area such as directly behind the vehicle from the rear image.

[0003] Patent Document 1 discloses an imaging device having an optical system with optical characteristics that cause the magnification ratios of light to be imaged to differ between a first region and a second region on the imaging surface. In Patent Document 1, the first region is a local region within the second region, the image of the second region is subjected to distortion correction and displayed on an in-vehicle display, and the image of the first region is cut out from the distorted-corrected image and displayed on an electronic rearview mirror.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In addition, there is also an image display system that can switch the image displayed on the in-vehicle display to an image of an area of interest, such as an image behind the vehicle or an image below the vehicle. In the case of such a system, the method of distortion correction for the rear image is also appropriately switched according to the switching of the area of interest to be displayed. However, when the method of distortion correction is switched, the image of the area cut out from the rear image after distortion correction for display on the electronic rearview mirror also changes, resulting in a loss of usability for the user. In this case, it is also conceivable to provide a configuration for performing dedicated distortion correction processing on the image of the area cut out for display on the electronic rearview mirror, but doing so will increase the circuit scale and cost. In particular, it is desirable that the image displayed on the electronic rearview mirror be a high-frame-rate image. However, since the distortion correction process takes a certain amount of processing time, the delay becomes large, and it may not be possible to obtain the desired frame rate. On the other hand, if distortion correction is not performed on the image of the area cut out for display on the electronic rearview mirror in order to reduce the delay, a deformed image with large distortion remaining will be displayed, resulting in an image that is very difficult to view.

[0006] Therefore, an object of the present invention is to be able to display an easily visible image while suppressing an increase in circuit scale, cost, and delay.

Means for Solving the Problems

[0007] The image processing apparatus of the present invention includes an acquisition means for acquiring a first image obtained by imaging an optical image formed by an optical system including a high-distortion area with large distortion and a low-distortion area with small distortion, a cutting-out means for acquiring a second image obtained by cutting out a part of the image area from the first image, and a stretching and shrinking processing means for stretching and shrinking the second image by a magnification of stretching and shrinking corresponding to the distortion of the optical system, and is characterized in that the second image stretched and shrunk by the stretching and shrinking processing means is output to a display device.

Effects of the Invention

[0008] According to the present invention, it is possible to display an image that is easy to view while suppressing an increase in circuit scale, cost, and delay.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments described below do not limit the present invention, and not all combinations of the features described in this embodiment are essential for the solution means of the present invention. The configuration of the embodiment can be appropriately modified or changed according to the specifications of the device to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). Also, a configuration may be formed by appropriately combining a part of each of the embodiments described later. In the following embodiments, the same configuration will be described with the same reference numerals.

[0011] <The First Embodiment> FIG. 1(A) is a block diagram showing a functional configuration example of the main part of an information processing system 100 including an image processing apparatus according to the first embodiment. Further, FIG. 1(B) is a diagram showing a hardware configuration example of the information processing system 100 according to the present embodiment. The information processing system 100 illustrated in the present embodiment is assumed to be a system mounted on a vehicle such as an automobile.

[0012] As an example, the information processing system 100 of the present embodiment is a system capable of displaying a moving image (rear image) captured by a camera (referred to as a rear camera 121) attached to the rear of a vehicle such as an automobile on an in-vehicle display 151 and an electronic rearview mirror 152. The rear camera 121 has an optical system 101 and an imaging device 102 in FIG. 1(A). Details of the optical system 101 and the imaging device 102 will be described later. The rear camera 121 is connected to an information processing apparatus 130 mounted on the vehicle via a communication i / f (interface) 131. In FIG. 1(A), the first display unit 105 corresponds to the in-vehicle display 151, and the second display unit 109 corresponds to the electronic rearview mirror 152.

[0013] The in-vehicle display 151 is attached to the dashboard or the like of the vehicle and is connected to the information processing apparatus 130 via a communication i / f 136. Maps, route guidance information, TV screens, channel selection information, various setting information, etc. are displayed on the in-vehicle display 151, and it is also possible to display the rear image. Maps, route guidance information, TV screens, channel selection information, and various setting information are appropriately switched and displayed when the vehicle is moving forward (while driving) or stopped. The display of the rear image on the in-vehicle display 151 is performed when the vehicle is reversing. Thereby, the driver of the vehicle can check the situation behind the vehicle when reversing the vehicle.

[0014] The electronic rearview mirror 152 is attached, for example, to the upper part of the front window of a vehicle, and is a display device that can display an image of the area directly behind the vehicle, that is, an image similar to the range visible from a general rearview mirror, among the rear images captured by the rear camera 121. The electronic rearview mirror 152 is connected to the information processing device 130 via the communication i / f 137. An image of the area directly behind the vehicle is displayed on the electronic rearview mirror 152 when the vehicle is moving forward (while driving), moving backward, or stopped. This enables the driver of the vehicle to check the situation behind the vehicle when moving forward (while driving), moving backward, or stopped.

[0015] The ECU (Electronic Control Unit) 135 is an in-vehicle microcomputer that performs overall control of the vehicle, such as operation control of the vehicle engine or drive motor, and control of various in-vehicle electronic devices. The ECU 135 uses the RAM 133 as a work memory and executes the OS (Operating System) and various programs stored in the ROM 134. In addition, the ECU 135 of the present embodiment also has a function as an image processing device that generates an image to be displayed on the in-vehicle display 151 and the electronic rearview mirror 152 based on the rear image captured by the rear camera 121. That is, in the case of the present embodiment, the processing in each functional unit such as the signal processing unit 103, distortion correction unit 104, cutout unit 106, stretching processing unit 107, and region determination unit 110 in FIG. 1(A) is executed by the ECU 135. For example, the ECU 135 uses the RAM 133 as a work memory and realizes each function such as the signal processing unit 103, distortion correction unit 104, cutout unit 106, stretching processing unit 107, and region determination unit 110 by executing the program of the present embodiment stored in the ROM 134.

[0016] The input device 153 is a user input device that includes various switches and buttons provided in the vehicle, and touch panels incorporated in the in-vehicle display 151 and the electronic rearview mirror 152. Note that a GUI (Graphical User Interface) may be displayed on the in-vehicle display 151 and the electronic rearview mirror 152, and a user (such as a driver or a passenger) can input various instructions via the GUI. Further, the input device 153 may include a function of acquiring instructions by the user's voice. The input device 153 is connected to the information processing device 130 via the communication i / f 138.

[0017] The external storage device 122 is a recording device capable of recording various data and information on a storage medium such as a removable memory card, and is connected to the information processing device 130 via the communication i / f 132. The external storage device 122 records a rear image captured by the rear camera 121, an image generated for display on the in-vehicle display 151 from the rear image, an image generated for display on the electronic rearview mirror 152 from the rear image, and the like. Further, the image recorded in the external storage device 122 can be read out and displayed on the in-vehicle display 151 or the like. The external storage device 122 can also be used when rewriting a program in the information processing device 130.

[0018] The wireless communication device 139 is a communication device for performing wireless communication with the outside by any one or a plurality of various wireless communication methods. The wireless communication device 139 can wirelessly communicate programs, image data, and various other information. The program wirelessly communicated may include the program according to the present embodiment. The image data wirelessly communicated may include a rear image captured by the rear camera 121, an image generated for display on the in-vehicle display 151 from the rear image, an image generated for display on the electronic rearview mirror 152 from the rear image, and the like.

[0019] The communication I / Fs 131, 132, 136 to 138, RAM 133, ROM 134, ECU 135, and wireless communication device 139 of the information processing apparatus 130 described above are mutually connected via a system bus.

[0020] FIG. 2 is a diagram used to explain the optical characteristics of the lens system included in the optical system 101. FIG. 2(A) is a diagram showing the range (image circle) of the optical image incident on the imaging surface of the imaging device 102 via the optical system 101, and is a diagram representing the relationship between the angle θ formed by the optical axis of the optical system 101 and the incident light ray and the image height y on the imaging surface in the form of contour lines per unit angle. FIG. 2(B) is a diagram showing the projection characteristics indicating the relationship between the angle θ formed by the optical axis of the optical system 101 and the incident light ray and the image height y [mm] on the imaging surface of the imaging device 102. In FIG. 2(B), the angle θ formed by the optical axis of the optical system 101 and the incident light ray is taken as the horizontal axis, and the image height y (imaging height) on the imaging surface of the imaging device 102 is shown as the vertical axis. Note that the angle θmax corresponds to the semi-angle of view.

[0021] As shown in Fig. 2(B), the lens system of the optical system 101 is configured such that in the half field angle, the projection characteristic y(θ) is different between a region where the angle θ formed by the optical axis of the optical system 101 and the incident light ray is less than a predetermined angle θa and a region where the angle θ is greater than or equal to the predetermined angle θa. In other words, the optical system 101 of the present embodiment has a characteristic such that the increase amount of the image height y on the imaging surface of the imaging device 102 is different between a region where the angle is less than the predetermined angle θa and a region where the angle is greater than or equal to the angle θa. Here, when the increase amount of the image height y per unit angle of the angle θ formed by the optical axis of the optical system 101 and the incident light ray within the half field angle increases, the number of pixels per unit angle increases, which corresponds to an increase in the resolution of the image on the imaging surface of the imaging device 102. Conversely, when the increase amount of the image height y per unit angle decreases, the number of pixels per unit angle decreases, which corresponds to a decrease in the resolution of the image on the imaging surface of the imaging device 102. That is, according to the optical system 101 of the present embodiment, the resolution of the image on the imaging surface of the imaging device 102 is different between a region where the angle is less than the predetermined angle θa and a region where the angle is greater than or equal to the angle θa. In the case of the optical system 101 of the present embodiment, the region closer to the center, which is the region where the angle is less than the predetermined angle θa, is a high-resolution region because the increase amount of the image height y per unit angle is large. On the other hand, the outer region, which is the region where the angle is greater than or equal to the predetermined angle θa, is a low-resolution region because the increase amount of the image height y per unit angle is small. In the optical system 101 of the present embodiment, the predetermined angle θa is defined in advance as the first threshold value. In the present embodiment, the region less than the first threshold value (less than the predetermined angle θa) is referred to as the high-resolution region 2a, and the region greater than or equal to the first threshold value (greater than or equal to the predetermined angle θa) is referred to as the low-resolution region 2b.

[0022] Also, as shown in FIG. 2(B), in the region less than a predetermined angle θa (less than the first threshold value), the increase amount of the image height y on the imaging surface of the imaging device 102 is substantially constant, while in the region of a predetermined angle θa or more (the first threshold value or more), the increase amount of the image height y is not constant. This indicates that the image distortion is small in the region less than the predetermined angle θa and large in the region of a predetermined angle θa or more. That is, the high-resolution region 2a less than the predetermined angle θa is a low-distortion region with relatively small distortion compared to the low-resolution region 2b, and the low-resolution region 2b of a predetermined angle θa or more is a high-distortion region with relatively large distortion compared to the high-resolution region 2a. Therefore, in this embodiment, the high-resolution region 2a that is less than the predetermined angle θa (less than the first threshold value) is also referred to as the low-distortion region 2a, while the low-resolution region 2b that is a predetermined angle θa or more (the first threshold value or more) is also referred to as the high-distortion region 2b. That is, it can also be said that the low-distortion region 2a is a region where the optical distortion by the optical system 101 is less than the first threshold value, while the high-distortion region 2b is a region where the optical distortion is the first threshold value or more. Also, from these facts, it can be rephrased that the first threshold value is a threshold value for the optical distortion in the optical system 101.

[0023] In the above description, an example of dividing into the high-resolution region 2a (low-distortion region 2a) or the low-resolution region 2b (high-distortion region 2b) based on whether it is less than or equal to the first threshold value (θa) is given, but it is not limited to this example. For example, a region including the region with the highest number of pixels per unit angle may be set as the high-resolution region 2a (low-distortion region 2a). In this case, a region with a number of pixels per unit angle less than or equal to a second threshold value, which is smaller than the number of pixels per unit angle in the high-resolution region 2a (low-distortion region 2a), is set as the low-resolution region 2b (high-distortion region 2b). Note that the projection characteristics of the optical system 101 are not limited to the projection characteristics shown in FIG. 2(B).

[0024] The imaging device 102 has, for example, a CMOS image sensor or a CCD image sensor, and acquires image data for each frame of the rear image by imaging the optical image formed on the imaging surface by the optical system 101. On the imaging surface of the imaging device 102, color filters of, for example, R (red), G (green), and B (blue) of the three primary colors are arranged for each pixel. It is assumed that the arrangement of the RGB color filters is, for example, a Bayer arrangement.

[0025] The signal processing unit 103 performs de-Bayer processing on the image data obtained by the imaging device 102 and converts it into RGB raster format image data. Further, the signal processing unit 103 performs, if necessary, white balance adjustment, gain and offset adjustment, gamma processing, color matrix processing, reversible compression processing, etc. on the image data. Hereinafter, unless otherwise explicitly stated, the image data is simply referred to as an image.

[0026] The distortion correction unit 104 performs processing for correcting the distortion caused by the projection characteristics of the optical system 101 on the rear image after the processing by the signal processing unit 103. That is, the distortion correction unit 104 performs distortion correction processing corresponding to the distortion based on the projection characteristics of the optical system 101. Note that the distortion correction unit 104 may have a plurality of distortion correction modes that can respectively correspond to, for example, a central projection method (y = f·tanθ), an equidistant projection method (y = f·θ), a combination of a plurality of these projection methods, or even another projection method. The distortion correction unit 104 can also select a distortion correction mode corresponding to the projection characteristics of the optical system 101 from among the plurality of distortion correction modes and perform distortion correction processing. As an example of the distortion correction processing, for each distortion correction mode, there is a LUT (look-up table) of correction coefficients corresponding to the projection characteristics, and a process of correcting the position of each pixel or interpolating pixels using the correction coefficients of the LUT selected from them can be mentioned. Note that f is the focal length of the optical system 101, and θ is the angle formed by the optical axis of the optical system 101 and the incident light ray. In addition to the example using the LUT, distortion correction processing may be performed in which the correction coefficients corresponding to the projection characteristics are obtained by calculation each time, and the position of each pixel is corrected using the correction coefficients.

[0027] As described above, the first display unit 105 is the in-vehicle display 151. The rear image that has been distortion-corrected by the distortion correction unit 104 is displayed on the in-vehicle display 151. In this embodiment, the rear image that has been distortion-corrected by the distortion correction unit 104 is displayed on the first display unit 105, but a rear image that has not been distortion-corrected may be displayed.

[0028] As described above, the second display unit 109 is the electronic rearview mirror 152. Although details will be described later, an image that has been cut out from the rear image captured by the rear camera in a manner to be described later and further subjected to a scaling process with a scaling magnification corresponding to the distortion of the optical system 101 is displayed on the electronic rearview mirror 152. Thus, an image of a region cut out from the rear image is displayed on the electronic rearview mirror 152, and the cutting out from the rear image is performed by the cutting-out unit 106.

[0029] The region determination unit 110 determines the region to be cut out by the cutting-out unit 106 from the rear image after the processing by the signal processing unit 103. In the case of this embodiment, the region determination unit 110 determines the cut-out region that can be displayed on the electronic rearview mirror 152 of the second display unit 109. Then, the cutting-out unit 106 cuts out a partial image region from the rear image after the processing by the signal processing unit 103 based on the information on the cut-out region determined by the region determination unit 110.

[0030] Here, in the case of this embodiment, the area cut out by the cut-out part 106 is an image area that does not include the high-distortion area 2b (low-resolution area 2b) where the distortion in the vertical direction is equal to or greater than the first threshold value among the rear images after the processing in the signal processing part 103. That is, the cut-out part 106 cuts out the image area of the low-distortion area 2a (high-resolution area 2a) where the distortion in the vertical direction is less than the first threshold value among the rear images after the processing in the signal processing part 103. Note that the area cut out by the cut-out part 106 may be an image area that includes the area where the number of pixels per unit angle in the vertical direction is the highest among the rear images after the processing of the signal processing part 103. That is, the cut-out part 106 cuts out so as not to include an image area that is equal to or less than a second threshold value, which is a value smaller than the number of pixels per unit angle in the high-resolution area 2a (low-distortion area 2a), among the rear images after the processing of the signal processing part 103.

[0031] The stretching processing part 107 is a stretching processing part that stretches the image of the area cut out by the cut-out part 106 by the magnification of expansion and contraction corresponding to the distortion of the optical system 101. For example, when the magnification of expansion and contraction is 1 times, the stretching processing is non-stretching. When the magnification of expansion and contraction is a positive magnification greater than 1, the stretching processing is stretching processing. When the magnification of expansion and contraction is greater than 0 and less than 1, the stretching processing is compression processing. In the case of the example of this embodiment, the magnification of expansion and contraction is the magnification of stretching for stretching the image in the horizontal direction. That is, in the case of this embodiment, the stretching processing part 107 performs stretching processing for stretching the image area cut out by the cut-out part 106 in the horizontal direction. Note that in this embodiment, the horizontal direction is defined as the data reading direction of the imaging element 102.

[0032] In the example of this embodiment, as described above, the cut-out portion 106 cuts out an image region with less distortion in the vertical direction from the rear image after the processing by the signal processing unit 103. Therefore, in the horizontal direction of the cut-out image region, not only the low-distortion region 2a (high-resolution region 2a) but also the high-distortion region 2b (low-resolution region 2b) is included. Thus, the stretching processing unit 107 selects and applies stretching processing with a stretching magnification suitable for each region to the low-distortion region 2a and the high-distortion region 2b in the horizontal direction in the image region cut out by the cut-out portion 106. In this embodiment, the stretching magnification can also be referred to as the stretching amount or the stretching coefficient. The stretching processing unit 107 of this embodiment is capable of executing a plurality (at least two) of stretching processes with different stretching magnifications, and selects and applies stretching processing with a stretching magnification suitable for the low-distortion region 2a and the high-distortion region 2b in the horizontal direction, respectively. It is assumed that the stretching magnifications in these plurality of stretching processes are predetermined values according to the distortion based on the projection characteristics of the optical system 101. In the case of this embodiment, for the high-distortion region 2b, stretching processing with a magnification higher than the stretching magnification applied to the low-distortion region 2a is applied.

[0033] In this way, the stretching processing unit 107 of this embodiment selects and applies stretching processing with a stretching magnification corresponding to the magnitude of the distortion in the horizontal direction in the image region cut out by the cut-out portion 106. That is, in the stretching processing unit 107, stretching processing corresponding to the magnitude of the distortion in the horizontal direction is performed on the image of the region cut out by the cut-out portion 106 and including the low-distortion region 2a and the high-distortion region 2b in the horizontal direction. In the case of this embodiment, the stretching processing unit 107 uses the resolution of the image in the low-distortion region (image height (number of pixels) per unit angle) as a reference, divides the high-distortion region into unit angles, and performs stretching processing with a stretching magnification corresponding to the resolution of the reference image in the low-distortion region for each of these unit angles. Thereby, the image after the stretching processing by the stretching processing unit 107 becomes an image with reduced distortion in the horizontal direction.

[0034] In the above example, it was assumed that the magnification factors in the plurality of stretching processes were each a predetermined magnification factor. However, the stretching processing unit 107 may change the magnification factor of the stretching process according to the change in distortion in the horizontal direction of the cut-out image area. Further, for example, the stretching processing unit 107 may change the magnification factor according to the resolution of the image in the horizontal direction of the cut-out image area.

[0035] The storage unit 108 stores information on the optical characteristics (projection characteristics) in the optical system 101 as illustrated in FIG. 2(B). In the case of the present embodiment, it is assumed that the storage unit 108 stores a look-up table in which, as information on the optical characteristics of the optical system 101, for example, the coordinates on the imaging surface of the imaging device 102 are associated with the image height data (number of pixels) per unit angle. The stretching processing unit 107 acquires the image height per unit angle in the horizontal direction from the look-up table of the storage unit 108 based on each coordinate on the imaging surface of the imaging device 102 corresponding to the image area cut out by the cut-out unit 106. Here, since the image height per unit angle corresponds to the magnitude of distortion (in this case, the distortion in the horizontal direction) as described above, the stretching processing unit 107 can determine the magnification factor corresponding to the magnitude of distortion in the horizontal direction. Thereby, the stretching processing unit 107 can perform stretching processing with a magnification factor corresponding to the magnitude of distortion in the horizontal direction of the image area cut out by the cut-out unit 106.

[0036] Then, the image cut out by the cut-out unit 106 and stretched by the stretching processing unit 107 is output to the second display unit 109. The second display unit 109 is the electronic rearview mirror 152 as described above. Therefore, an image in which the highly distorted area in the vertical direction is removed and the distortion is reduced by stretching processing in the horizontal direction, that is, the image cut out by the cut-out unit 106 and stretched by the stretching processing unit 107, is displayed on the electronic rearview mirror 152.

[0037] FIG. 3 is a flowchart showing the processing flow in each functional unit after the cut-out unit 106 included in the image processing apparatus according to the first embodiment. First, in step S31, the area determination unit 110 determines the extraction area when extracting the image to be displayed on the electronic rearview mirror 152 from the image captured by the rear camera 121. In the present embodiment, the extraction area may be set in advance as an initial value, or may be the extraction area set last at the previous startup. As the value for setting the extraction area, the coordinate values of the image can be used. In addition, the area determination unit 110 can also set the extraction area based on the value instructed by the user via the input device 153. The instruction by the user can be assumed to be an instruction to move the extraction area parallel in an arbitrary direction by an arbitrary amount on the rear image displayed on the in-vehicle display 151. In addition, the instruction by the user can also be assumed to be an instruction for the user to touch a desired extraction area via a touch panel or the like within the rear image displayed on the in-vehicle display 151. Any method may be used for determining the extraction area as long as the area to be displayed on the electronic rearview mirror 152 can be determined from the rear image after the processing by the signal processing unit 103.

[0038] FIG. 4(A) shows an example of the image 400 captured by the imaging device 102 and processed by the signal processing unit 103, and the extraction area 401 determined as the area to be displayed on the electronic rearview mirror 152 for the image 400. The image 400 is a rear image formed on the imaging surface of the imaging device 102 by the optical system 101 having the optical characteristics described in FIG. 2 above and captured. Note that the lines of each circle drawn in the image 400 are lines indicating virtual circles representing the image height for each unit angle shown in FIG. 2(A) in a contour line shape, and are not lines actually existing in the image. In the image 400 of FIG. 4(A), the inside of the outermost virtual circle is the area of the optical image formed on the imaging surface by the optical system 101, and the outside of the outermost virtual circle is the area of a black image where no optical image exists (there is no incident light).

[0039] Next, in step S32, the extraction unit 106 extracts the image of the extraction area 401 determined as the area to be displayed on the electronic rearview mirror 152 in step S31 from the image 400 of FIG. 4(A).

[0040] Next, in step S33, the stretching processing unit 107 determines whether the image area cut out in step S32 includes a highly distorted area in the horizontal direction. In the case of this embodiment, whether a highly distorted area is included in the horizontal direction is known in advance from the projection characteristics of the optical system 101. If it is determined in step S33 that no highly distorted area is included, the processing of the information processing system 100 transitions to step S35. On the other hand, if it is determined that a highly distorted area is included, the stretching processing unit 107 proceeds to step S34 and performs horizontal stretching processing on the cut-out image area. In the case of this embodiment, since the cut-out image area includes a highly distorted area in the horizontal direction as described above, it will proceed to step S34.

[0041] When proceeding to step S34, the stretching processing unit 107 performs stretching processing on the cut-out image area as described above. That is, the stretching processing unit 107 divides the highly distorted area into unit angles based on the resolution of the image (image height (number of pixels) per unit angle) in the low-distorted area, and for each of these unit angles, performs processing to stretch it to be equivalent to the resolution of the reference image in the low-distorted area. The image 402 shown in FIG. 4(B) is an image after performing horizontal stretching processing on the image area cut out based on the cut-out area 401 in FIG. 4(A).

[0042] Note that in the above example, an example of performing stretching processing by changing the stretching magnification according to the resolution of the image within the highly distorted area has been described, but stretching processing may be uniformly performed on the highly distorted area at a constant magnification. Also, for example, stretching processing may be performed on the low-distorted area so that the resolution of the entire image becomes uniform.

[0043] After that, in step S35, the stretching processing unit 107 outputs the image 402 shown in FIG. 4(B) to the electronic room mirror 152 of the second display unit 109 for display. At this time, the stretching processing unit 107 may perform stretching processing at a magnification that is uniform in the horizontal direction and the vertical direction according to the display resolution and aspect ratio of the electronic room mirror 152.

[0044] FIG. 5(A) is a diagram showing an example of setting an expansion coefficient as a value representing the horizontal expansion magnification for the example of the image height and the angle θ shown in FIG. 2(B) described above. The stretching processing unit 107 changes the stretching coefficient based on the Q point in the figure. The Q point preferably corresponds to a predetermined angle θa at which the above-described projection characteristics change, but it does not have to correspond. In the example of FIG. 5(A), the stretching processing in the horizontal direction is performed with a constant stretching coefficient up to the reference image height (the Q point in FIG. 5(A)), and thereafter, the stretching processing is performed with a stretching coefficient that increases according to the change amount of the image height.

[0045] In the present embodiment, at the time of cutting by the cutting unit 106, it is in accordance with the display of the electronic room mirror 152, and since the cutting is performed so that the high distortion region is not included in the vertical direction, the stretching processing in the vertical direction is not performed. However, the present invention is not limited to this example, and when cutting is performed such that a high distortion region is included in the vertical direction, stretching processing may also be performed in the vertical direction.

[0046] FIG. 5(B) is a diagram showing an example of setting the vertical expansion coefficient for the example of the image height and the angle θ shown in FIG. 2(B) described above. Note that the stretching processing in the vertical direction does not have to be the same as the processing performed in the horizontal direction by the stretching processing unit 107 described above. For example, as shown in FIG. 5(B), the stretching processing in the vertical direction may use the first stretching coefficient up to the reference image height (the Q point in FIG. 5(B)), and thereafter, use stretching coefficients that are switched such as the second stretching coefficient and the third stretching coefficient as the image height increases.

[0047] The stretching process in the stretching processing unit 107 of this embodiment can be realized, for example, by adjusting the read timing of a line memory (not shown). That is, as the resolution of the image decreases, the read timing (period) is lengthened, so that the low resolution of the image can be simply corrected. For example, the stretching process can be simply realized by periodically reading the line memory a plurality of times. In the case of this example, since the stretching process can be realized only by adjusting the read timing, it is possible to realize it without preparing a dedicated circuit for performing stretching / compression with high image quality.

[0048] As described above, in the first embodiment, the image to be displayed on the in-vehicle display 151 is the image after performing distortion correction processing on the rear image including the high distortion region and the low distortion region captured via the optical system 101. On the other hand, the image to be displayed on the electronic rearview mirror 152 is the image obtained by performing stretching processing (stretching process in the example of this embodiment) on the image region cut out from the rear image before the distortion correction processing at the magnification corresponding to the distortion of the optical system 101. That is, in the first embodiment, on the electronic rearview mirror 152, an image is displayed in which the distortion of the image in the high distortion region is made less noticeable by stretching the image horizontally at the stretching magnification corresponding to the distortion of the image. Also, in the case of this embodiment, the processing for the image displayed on the electronic rearview mirror 152 is the stretching process based on the stretching magnification, so the processing delay is less compared to the case of performing distortion correction processing, and thus it is possible to display at a desired high frame rate. Also, the stretching process for the image for the electronic rearview mirror is the stretching process in only one horizontal direction, so the stretching processing unit 107 can have a smaller circuit scale compared to a distortion correction circuit such as the distortion correction unit 104.

[0049] <Second Embodiment> Hereinafter, the information processing system of the second embodiment will be described. Here, mainly the differences from the first embodiment will be described. FIG. 6 is a block diagram showing the main functional configuration of an information processing system 600 including an image processing apparatus according to the second embodiment. In the information processing system 600 of FIG. 6, the optical system 101, the imaging device 102, the signal processing unit 103, the distortion correction unit 104, the storage unit 108, the region determination unit 110, the first display unit 105, and the second display unit 109 are the same as those in the example of FIG. 1 described above, and thus their descriptions are omitted. Also, since the hardware configuration of the information processing system 600 of the second embodiment is the same as that in FIG. 1(B), its illustration and description are omitted.

[0050] The cutout unit 106 of the first embodiment described above performs a process of cutting out an image according to the cutout region determined by the region determination unit 110. In contrast, the cutout unit 601 of the second embodiment cuts out a region with low distortion in the vertical direction in the horizontal direction from the rear image after the process in the signal processing unit 103. In other words, the cutout unit 601 cuts out an image region excluding the regions with high and low distortion in the vertical direction from the rear image after the process in the signal processing unit 103.

[0051] Also, in the case of the second embodiment, the stretching processing unit 602 performs stretching processing on the region with low distortion in the vertical direction cut out by the cutout unit 601, and the image after the stretching processing is sent to the display image cutout unit 603. The display image cutout unit 603 cuts out the image of the cutout region determined by the region determination unit 110 from the image after the stretching processing by the stretching processing unit 602. That is, in the case of the second embodiment, the image displayed on the electronic mirror 152 of the second display unit 109 is the image cut out by the display image cutout unit 603.

[0052] Furthermore, in the case of the second embodiment, object recognition units 604a and 604b are provided. The object recognition unit 604a performs object recognition processing on the rear image after the distortion correction processing is performed by the distortion correction unit 104, that is, the image that will be displayed on the in-vehicle display 151 of the first display unit 105, and superimposes the object recognition result on the image. On the other hand, the object recognition unit 604b performs object recognition processing on the image cut out by the display image cut-out unit 603, that is, the image that will be displayed on the electronic rearview mirror 152 of the second display unit 109, and superimposes the object recognition result on the image. As an example, the object recognition processing can use object recognition processing using a pre-trained model or processing that discriminates between the background and moving objects from the difference between frames such as the so-called background difference method and recognizes the moving objects. The object recognition processing is performed to detect an object approaching the vehicle on which the information processing system 600 of the present embodiment is mounted, and the object recognition result is displayed on the in-vehicle display 151 or the electronic rearview mirror 152. Thereby, a user who drives the vehicle of the present embodiment can recognize the approach of an object when there is an object approaching the vehicle.

[0053] FIG. 7 is a flowchart showing the flow of processing in each functional unit after the cut-out unit 601 included in the image processing apparatus of the second embodiment. In step S74, the region determination unit 110 determines a cut-out region to be displayed on the electronic rearview mirror 152 in the same manner as step S31 of FIG. 3 described above. In the case of the second embodiment, the information indicating the cut-out region determined by the region determination unit 110 is sent to the display image cut-out unit 603. After this step S74, the processing in the information processing system 600 proceeds to step S73.

[0054] Also, in the case of the second embodiment, in step S71, the cut-out unit 601 cuts out, in the horizontal direction, a region with low distortion in the vertical direction from the rear image after the processing by the signal processing unit 103, that is, cuts out a region excluding regions with high and low distortion in the vertical direction.

[0055] FIG. 8(A) and FIG. 8(B) are diagrams used to explain the cutting process by the cutout section 601 of the second embodiment. The image 800 in FIG. 8(A) is a rear image captured by the imaging device 102 and processed by the signal processing unit 103. Similar to the image 400 in FIG. 4(A) described above, the inside of the outermost virtual circle in the image 800 of FIG. 8(A) is the region of the optical image formed by the optical system 101, and the outside of the outermost virtual circle is a black image region where no optical image exists.

[0056] In the case of the second embodiment, the cutout section 601 cuts out, in the horizontal direction, a region with low distortion in the vertical direction from the image 800 in FIG. 8(A). FIG. 8(B) is a diagram showing the image 800 in FIG. 8(A) with regions other than the region 801 with low distortion in the vertical direction masked. Here, also in the second embodiment, similar to the first embodiment described above, the region with low distortion is a region less than the first threshold value, and the region with high distortion is a region equal to or greater than the first threshold value. Alternatively, the region with low distortion is a region including the region with the highest number of pixels per unit angle, and the region with high distortion is a region less than or equal to a second threshold value smaller than the number of pixels per unit angle in the region with low distortion.

[0057] In the example of the image 800 in FIG. 8(A), since the region closer to the center is a region with low distortion as also described in FIG. 2 above, the region 801 excluding the mask portion 810 in FIG. 8(B) is a region with low distortion in the vertical direction. Note that the mask portion 810 in FIG. 8(B) is drawn to make it easier to understand the region with low distortion in the vertical direction on the drawing, and the image 800 is not actually masked. Of course, the image 800 can be actually masked. In the case of the second embodiment, the cutout section 601 performs a cutout process of cutting out, from the image 800 in FIG. 8(A), the region with low distortion in the vertical direction, that is, the region 801 excluding the mask portion 810 shown in FIG. 8(B). Then, after step S71, the processing of the information processing system 600 proceeds to step S72.

[0058] When proceeding to step S72, the stretching processing unit 602 performs a stretching process in the horizontal direction on the vertically low-distortion region 801 cut out in step S71. The stretching process in the stretching processing unit 602 is the same as that described in the first embodiment above. That is, the stretching processing unit 602 divides the high-distortion region for each unit angle based on the resolution of the image of the low-distortion region (image height (number of pixels) per unit angle), and performs a stretching process so as to be equivalent to the resolution of the reference image. After this step S72, the processing of the information processing system 600 proceeds to step S73.

[0059] When proceeding to step S73, the display image cutting unit 603 cuts out the image of the cutout region determined by the region determination unit 110, that is, the region image adjusted to the electronic room mirror 152, from the image after the stretching process by the stretching processing unit 602. FIG. 8(C) is a diagram showing the image 802 cut out by the display image cutting unit 603 from the image after the stretching process by the stretching processing unit 602. That is, in the second embodiment, the image 802 is an image obtained by stretching the image of the region 801 cut out as a vertically low-distortion region from the image 800 in FIG. 8(A), and then cutting it out according to the cutout region determined by the region determination unit 110. At this time, the display image cutting unit 603 may perform a scaling process at a magnification that is uniform in the horizontal and vertical directions according to the display resolution and aspect ratio of the electronic room mirror 152.

[0060] Thereafter, in step S75, the image 802 cut out by the display image cutting unit 603 is displayed on the electronic room mirror 152 of the second display unit 109 through the object recognition process in the object recognition unit 604b.

[0061] In the second embodiment, a low-distortion region in the vertical direction is cut out from the rear image without distortion correction, and further, an image cut out from an image obtained by performing horizontal stretching processing on the image of the low-distortion region in the vertical direction is displayed on the electronic rearview mirror 152. Thus, similarly to the first embodiment, an image in which the deformation of the image of the highly distorted region is suppressed can be displayed on the electronic rearview mirror 152.

[0062] Also in the second embodiment, the object recognition unit 604a performs object recognition from the image after distortion correction by the distortion correction unit 104, and the object recognition unit 604b performs object recognition from an image obtained by horizontally stretching the image of the low-distortion region in the vertical direction. That is, for example, when performing recognition using an image of an object deformed by distortion, there is a risk that the recognition accuracy will decrease. In contrast, in this embodiment, object recognition is performed using an image in which distortion is suppressed, that is, an image in a state where the deformation of the object in the image is suppressed. According to this embodiment, high-precision object recognition with a decrease in object recognition accuracy suppressed is possible.

[0063] <Other Embodiments> The object recognition described in the second embodiment is also applicable to the first embodiment. When applying object recognition to the first embodiment, an object recognition unit can be provided after the distortion correction unit 104 and after the stretching processing unit 107.

[0064] Also in the above-described embodiment, whether the rear image is a highly distorted region or a low-distorted region is determined based on the resolution of the image per unit angle. However, for example, it may be determined based on the optical distortion defined by the following formula (1).

[0065] ((Y - Y') / Y') * 100 (%) Formula (1) In formula (1), Y is the actual image height, Y' is the ideal image height, Y' = f * tanθ, f is the focal length of the optical system 101, and θ is the angle formed by the optical axis of the optical system 101 and the incident light ray.

[0066] In the information processing system of each of the above-mentioned embodiments, for example, the image displayed on the in-vehicle display 151 may be switchable to an image of a region of interest such as not only the area directly behind the vehicle but also the area below the vehicle. When displayed on the in-vehicle display 151, appropriate distortion correction may be performed on the image of the area directly behind the vehicle and the image of the area of ​​interest such as the area below the vehicle. Here, in each of the above-mentioned embodiments, the image displayed on the electronic rear-view mirror 152 is an image of an area cut out from an image on which distortion correction processing has not been performed. Therefore, even if the method of distortion correction for the image displayed on the in-vehicle display 151 is appropriately switched, the switching does not affect the display of the electronic rear-view mirror 152, and usability is not impaired.

[0067] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0068] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an acquisition means for acquiring a first image obtained by capturing an optical image by an optical system including a high distortion region and a low distortion region; A cutout means for acquiring a second image by cutting out a part of an image area from the first image; a scaling processing means for scaling the second image by a scaling factor corresponding to the distortion of the optical system, An image processing apparatus, characterized in that the second image that has been expanded and contracted by the expansion and contraction processing means is output to a display device. (Configuration 2) An acquisition means for acquiring a first image obtained by imaging an optical image by an optical system including a high-distortion region with a large distortion and a low-distortion region with a small distortion; A cutting-out means for cutting out a part of the image area from the first image to obtain a second image; An expansion and contraction processing means for expanding and contracting the second image by a magnification of expansion and contraction corresponding to the distortion of the optical system; A display image cutting-out means for cutting out an image to be output to a display device from the second image that has been expanded and contracted by the expansion and contraction processing means; An image processing apparatus, characterized by comprising the above. (Configuration 3) The first image has correction means for performing correction processing according to the projection characteristics of the optical system, The image processing apparatus according to Configuration 1 or 2, characterized in that the first image subjected to the correction processing is output to a display device different from the display device. (Configuration 4) The image processing apparatus according to Configuration 3, characterized in that the correction means performs correction processing for correcting the distortion of the image based on the projection characteristics of the optical system. (Configuration 5) Performing recognition processing for recognizing an object from the image output to the display device, and superimposing the recognition result by the recognition processing on the image output to the display device, the image processing apparatus according to any one of Configurations 1 to 4. (Configuration 6) The low-distortion region of the optical system is a region where the distortion of the optical system is less than a first threshold value, The high-distortion region of the optical system is a region where the distortion of the optical system is equal to or greater than the first threshold value, the image processing apparatus according to any one of Configurations 1 to 5. (Configuration 7) The image processing apparatus according to Configuration 6, characterized in that the cutting-out means performs cutting-out from the first image so as not to include an image region where the distortion in the vertical direction is equal to or greater than the first threshold value. (Configuration 8) The stretching and shrinking processing means stretches and shrinks the second image at a magnification such that the resolution per unit angle in the high distortion region corresponds to the reference resolution, based on the number of pixels per unit angle within the angle of view in the low distortion region, in the image processing apparatus according to Configuration 7. (Configuration 9) The low distortion region of the optical system is a region including a region where the number of pixels per unit angle within the angle of view is the highest, The high distortion region of the optical system is a region where the number of pixels per unit angle within the angle of view is equal to or less than a second threshold value, which is smaller than the number of pixels per unit angle in the low distortion region, in the image processing apparatus according to any one of Configurations 1 to 5. (Configuration 10) The cutting-out means cuts out an image region from the first image that does not include a region where the number of pixels per unit angle in the vertical direction within the angle of view is equal to or less than the second threshold value, which is smaller than the number of pixels per unit angle in the low distortion region, in the image processing apparatus according to Configuration 9. (Configuration 11) The stretching and shrinking processing means stretches and shrinks the second image at a magnification such that the resolution per unit angle in the high distortion region corresponds to the reference resolution, based on the resolution per unit angle in the low distortion region, in the image processing apparatus according to Configuration 9 or 10. (Configuration 12) The stretching and shrinking processing means includes at least two stretching and shrinking processes with different magnification factors, and selects which stretching and shrinking process to use according to the distortion of the optical system, in the image processing apparatus according to any one of Configurations 1 to 11. (Configuration 13) The stretching and shrinking processing means changes the magnification factor according to the change in the distortion within the second image, in the image processing apparatus according to any one of Configurations 1 to 12. (Configuration 14) The image processing apparatus according to any one of Configurations 1 to 13, wherein the scaling processing means scales the horizontal direction of the second image according to the scaling magnification. (Configuration 15) The image processing apparatus according to any one of Configurations 1 to 14, wherein the scaling processing means scales the second image only when the image area cut out by the cutting-out means includes the highly distorted area. (Configuration 16) The image processing apparatus according to any one of Configurations 1 to 15, wherein the scaling processing means has a line memory into which the second image is input, and scales the second image by adjusting the read timing of the line memory according to the distortion. (Method 1) An acquisition step of acquiring a first image obtained by imaging an optical image by an optical system including a highly distorted area with a large distortion and a low distorted area with a small distortion; A cutting-out step of obtaining a second image by cutting out a partial image area from the first image; A scaling processing step of scaling the second image by a scaling magnification according to the distortion of the optical system, and An image processing method characterized by outputting the second image scaled in the scaling processing step to a display device. (Method 2) An acquisition step of acquiring a first image obtained by imaging an optical image by an optical system including a highly distorted area with a large distortion and a low distorted area with a small distortion; A cutting-out step of cutting out a partial image area from the first image to obtain a second image; A scaling processing step of scaling the second image by a scaling magnification according to the distortion of the optical system; A display image cutting-out step of cutting out an image to be output to a display device from the second image scaled in the scaling processing step; An image processing method characterized by including the steps. (Program 1) A program for causing a computer to function as the image processing apparatus according to any one of Configurations 1 to 16.

Description of Signs

[0069] 100: Information processing system, 101: Optical system, 103: Signal processing unit, 104: Distortion correction unit, 106: Extraction unit, 105: First display unit, 106: Extraction unit, 107: Expansion processing unit, 108: Memory unit, 109: Second display unit, 110: Region determination unit

Claims

1. an acquisition means for acquiring a first image obtained by capturing an optical image by an optical system including a high distortion region and a low distortion region; a cutout means for acquiring a second image by cutting out a part of an image area from the first image; a scaling processing means for scaling the second image by a scaling factor corresponding to the distortion of the optical system, The image processing device further comprises: outputting the second image, which has been subjected to the expansion / contraction processing by the expansion / contraction processing means, to a display device.

2. an acquisition means for acquiring a first image obtained by capturing an optical image by an optical system including a high distortion region and a low distortion region; a cutout means for cutting out a part of an image area from the first image to obtain a second image; a scaling processing means for scaling the second image by a scaling factor corresponding to the distortion of the optical system; a display image cutout means for cutting out an image to be output to a display device from the second image that has been expanded or contracted by the expansion / contraction processing means; 13. An image processing device comprising:

3. a correction unit that performs a correction process on the first image in accordance with the projection characteristic of the optical system; 3. The image processing apparatus according to claim 1, wherein the first image on which the correction process has been performed is output to a display device other than the display device.

4. 4. The image processing apparatus according to claim 3, wherein the correction means performs a correction process for correcting image distortion caused by the projection characteristics of the optical system.

5. 5. The image processing device according to claim 4, further comprising: a recognition process for recognizing an object from the image output to the display device; and a result of the recognition process being superimposed on the image output to the display device.

6. the low distortion region of the optical system is a region in which the distortion of the optical system is less than a first threshold; 3. The image processing device according to claim 1, wherein the high distortion area of ​​the optical system is an area in which the distortion of the optical system is equal to or greater than the first threshold value.

7. 7. The image processing apparatus according to claim 6, wherein the cropping means crops the first image so as not to include an image area in which distortion in the vertical direction is equal to or greater than the first threshold value.

8. 8. The image processing device according to claim 7, wherein the scaling processing means scales the second image at a magnification ratio such that a resolution per unit angle in the high distortion region corresponds to a reference resolution based on a number of pixels per unit angle within an angle of view in the low distortion region.

9. the low-distortion region of the optical system is a region including a region in which the number of pixels per unit angle within an angle of view is the highest, 3. The image processing device according to claim 1, wherein the high distortion region of the optical system is a region in which the number of pixels per unit angle within the angle of view is equal to or less than a second threshold value, the second threshold value being a value smaller than the number of pixels per unit angle in the low distortion region.

10. 10. The image processing device according to claim 9, wherein the extraction means extracts from the first image an image area that does not include an area where the number of pixels per unit angle in the vertical direction within the angle of view is equal to or less than the second threshold, the number of pixels per unit angle being a value smaller than the number of pixels per unit angle in the low-distortion area.

11. 11. The image processing device according to claim 10, wherein the scaling processing means scales the second image at a magnification ratio such that a resolution per unit angle in the low distortion region is set as a standard and a resolution per unit angle in the high distortion region becomes equivalent to the standard resolution.

12. 3. The image processing device according to claim 1, wherein the enlargement / reduction processing means includes at least two enlargement / reduction processes having different enlargement / reduction magnifications, and selects which enlargement / reduction process to use depending on the distortion of the optical system.

13. 3. The image processing apparatus according to claim 1, wherein the enlargement / reduction processing means changes a magnification of the enlargement / reduction in accordance with a change in the distortion in the second image.

14. 3. The image processing apparatus according to claim 1, wherein the scaling processing means scales the second image in the horizontal direction in accordance with the scaling factor.

15. 3. The image processing apparatus according to claim 1, wherein the expansion / contraction processing means expands or contracts the second image only when the image area cut out by the cutting means includes the highly distorted area.

16. 3. The image processing device according to claim 1, wherein the expansion / contraction processing means has a line memory to which the second image is input, and expands or contracts the second image by adjusting a read timing of the line memory in accordance with the distortion.

17. an acquisition step of acquiring a first image obtained by capturing an optical image by an optical system including a high distortion region and a low distortion region; a cutting step of acquiring a second image by cutting out a part of an image area from the first image; a scaling processing step of scaling the second image by a scaling factor corresponding to the distortion of the optical system, an image processing method comprising: outputting the second image, which has been subjected to the expansion / contraction processing step, to a display device;

18. an acquisition step of acquiring a first image obtained by capturing an optical image by an optical system including a high distortion region and a low distortion region; a cutting step of cutting out a part of an image area from the first image to obtain a second image; a scaling process step of scaling the second image by a scaling factor corresponding to the distortion of the optical system; a display image extraction step of extracting an image to be output to a display device from the second image that has been expanded or contracted in the expansion / contraction processing step; 13. An image processing method comprising:

19. Computer, an acquisition means for acquiring a first image obtained by capturing an optical image by an optical system including a high distortion region and a low distortion region; a cutout means for acquiring a second image by cutting out a part of an image area from the first image; a scaling processing means for scaling the second image by a scaling factor corresponding to the distortion of the optical system, a program that causes the image processing device to function as an image processing device that outputs the second image that has been scaled by the scale processing means to a display device.

20. Computer, an acquisition means for acquiring a first image obtained by capturing an optical image by an optical system including a high distortion region and a low distortion region; a cutout means for cutting out a part of an image area from the first image to obtain a second image; a scaling processing means for scaling the second image by a scaling factor corresponding to the distortion of the optical system; a display image cutout means for cutting out an image to be output to a display device from the second image that has been expanded or contracted by the expansion / contraction processing means; A program that causes the image processing device to function as an image processing device having the above-mentioned configuration.

Images