Image processing device, mobile object, control method, and computer program

Abstract

To achieve an image processing apparatus that can create, by using an image from one imaging means, a first image with high resolution and a second image with low resolution capable of displaying and checking a wide range of a lower part.SOLUTION: An image processing apparatus has: acquisition means that acquires an image from imaging means that picks up an image of the rear of a movable body; image creation means that cuts out, from the image, a first area formed of an area including the center of a sensor surface of the imaging means to create a first image, and cuts out a second area from an image area different from the first area to create a second image; and display control means that displays the first image and displays the second image when the movable body moves backward. The image creation means creates the first image and the second image so that a lower area of the first area and a part of an upper area of the second area overlap each other.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device capable of acquiring images with different resolutions, a moving object, a control method, and a computer program. [Background technology]

[0002] In recent years, there has been a demand to replace the room mirrors (rear-view mirrors) installed in vehicles with electronic room mirrors. Patent Document 1 discloses an electronic room mirror system that is composed of an imaging means that captures the area behind the vehicle outside and a display means inside the vehicle, and that displays the image captured by the imaging means on a display inside the vehicle, allowing the driver to check the situation behind the vehicle outside.

[0003] On the other hand, there are rearview confirmation systems that allow the driver to check the blind spot behind the vehicle when backing up, etc. Patent Document 2 discloses a rearview confirmation system that allows the driver to check the blind spot behind the vehicle when backing up, etc. by installing a camera to capture an image behind the vehicle and displaying the captured image inside the vehicle cabin. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-95202 [Patent Document 2] Japanese Patent Application Laid-Open No. 2004-345554 Summary of the Invention [Problem to be solved by the invention]

[0005] The camera used as the imaging means for capturing images for the electronic rearview mirror is required to have high resolution so that the driver can check the situation behind them in greater detail, while the camera for the rearview system is required to capture a wider range of images to check safety in a wider area, including blind spots behind the vehicle and areas to the rear and sides, in order to avoid collisions when reversing, etc.

[0006] Therefore, when installing an electronic rearview mirror system and a rearview confirmation system simultaneously in a vehicle, there is a problem that the in-vehicle camera system becomes complicated if a camera for the electronic rearview mirror system and a camera for the rearview confirmation system are installed separately.

[0007] Therefore, in consideration of the above problems, the present invention aims to provide an image processing device that can generate a high-resolution first image and a low-resolution second image that can display and confirm a wide area below from an image taken by a single imaging means. [Means for solving the problem]

[0008] In order to achieve the above object, an image processing device according to one aspect of the present invention comprises: an acquisition means for acquiring an image from an imaging means for capturing an image of the rear of the moving object; an image generating means for generating a first image by cutting out a first region formed by a region including the center of a sensor surface of the imaging means from the image, and for generating a second image by cutting out a second region from an image region different from the first region; a display control means for displaying the first image and displaying the second image when the moving body is reversing, The image generating means generates the first image and the second image so that a lower area of ​​the first area and a partial upper area of ​​the second area overlap with each other. Along with The lower end of the second region is entirely below the lower end of the first region, and The first image and the second image are generated so that the entire upper end of the second region is below the upper end of the first region. It is characterized by: [Effects of the Invention]

[0009] According to the present invention, it is possible to realize an image processing device that can generate a high-resolution first image and a low-resolution second image that can display and confirm a wide area below from an image taken by a single imaging means. [Brief explanation of the drawings]

[0010] [Figure 1] 2 is a diagram illustrating the positional relationship between a vehicle 1 and an imaging unit 2. FIG. [Figure 2] 10A and 10B are diagrams illustrating the optical characteristics of the imaging unit 2. FIG. [Figure 3] 1 is a block diagram showing a configuration of a camera system according to a first embodiment. [Figure 4] 10(A) to 10(D) are diagrams for explaining examples of clipping regions in the first embodiment. [Figure 5] 10 is a flowchart showing a processing flow of a camera system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, preferred embodiments of the present invention will be described by way of example with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified. [Example]

[0012] In this embodiment, an improved method for realizing both a high-definition display for an electronic rearview mirror and a wide-area display for rearview confirmation using a single camera will be described. FIG. 1 is a diagram illustrating the positional relationship between a vehicle 1 and an imaging unit 2. As shown in FIG. In this embodiment, as shown in Fig. 1, an imaging unit 2 is installed behind a vehicle 1, such as an automobile, which is a moving body. The imaging unit 2 is installed so that the area behind the vehicle, which is a moving body, is its imaging range. The imaging unit 2 has an optical system that forms an optical image, and a sensor that captures the optical image formed by the optical system on the light receiving section of the imaging unit 2. It is desirable to install the imaging unit 2 so that the optical axis of the optical system is approximately horizontal.

[0013] The imaging unit 2 used in this embodiment has an optical system with a sufficient angle of view and imaging magnification in the central angle of view region in order to obtain a high-definition image of a distant object with a wide angle of view. The optical system of the imaging unit 2 in this embodiment will be described with reference to FIG.

[0014] 2A and 2B are diagrams for explaining the optical characteristics of the image capture unit 2. Fig. 2A is a diagram showing, in contour lines, the image height y at each half angle of view on the sensor surface of the image capture unit 2 of the optical system included in the image capture unit 2 in this embodiment. Fig. 2B is a diagram showing the projection characteristics, which represent the relationship between the image height y and the half angle of view θ of the optical system included in the image capture unit 2 in this embodiment. In Fig. 2B, the half angle of view (the angle between the optical axis and the incident light) θ is plotted on the horizontal axis, and the image height (image height) y on the sensor surface (image plane) of the image capture unit 2 is plotted on the vertical axis.

[0015] As shown in FIG. 2B, the optical system of the imaging unit 2 in this embodiment is configured so that its projection characteristic y(θ) differs between regions less than a predetermined half angle of view θa and regions greater than or equal to the half angle of view θa. Therefore, when the resolution is defined as the increase in image height y per unit half angle of view θ, the resolution differs depending on the region. This local resolution can also be expressed as the differential value dy(θ) / dθ of the projection characteristic y(θ) at the half angle of view θ. In other words, the greater the slope of the projection characteristic y(θ) in FIG. 2B, the higher the resolution. It can also be said that the greater the interval between the image heights y at each half angle of view on the contour lines in FIG. 2A, the higher the resolution.

[0016] In this embodiment, the area near the center formed on the sensor surface when the half angle of view θ is less than a predetermined half angle of view θa is called a high-resolution area 41 (FIG. 2(A)), and the area near the outside where the half angle of view θ is equal to or greater than the predetermined half angle of view θa is called a low-resolution area 42 (FIG. 2(A)).

[0017] The optical system of the imaging unit 2 in this embodiment is configured so that its projection characteristic y(θ) is greater than f×θ (f is the focal length of the optical system of the imaging unit 2) in the high-resolution region 41. In addition, the projection characteristic y(θ) in the high-resolution region is set to be different from the projection characteristic in the low-resolution region. Furthermore, when θmax is the maximum half angle of view of the optical system of the imaging unit 2, the ratio θa / θmax of θa to θmax is preferably equal to or greater than a predetermined lower limit, for example, 0.15 to 0.16.

[0018] Furthermore, the ratio θa / θmax of θa to θmax is preferably equal to or less than a predetermined upper limit, for example, 0.25 to 0.35. For example, if θmax is 90°, the predetermined lower limit is 0.15, and the predetermined upper limit is 0.35, then θa should preferably be determined in the range of 13.5 to 31.5°. Furthermore, the optical system of the imaging unit 2 is configured so that its projection characteristic y(θ) also satisfies the following equation 1.

[0019]

number

[0020] By configuring the optical system as described above, resolution can be obtained in the high-resolution region 41, while the increase in image height y per unit half angle of view θ can be reduced in the low-resolution region 42, making it possible to capture an image with a wider angle of view. Therefore, high resolution can be obtained in the high-resolution region 41 while maintaining an imaging range with a wide angle of view equivalent to that of a fisheye lens.

[0021] Furthermore, in this embodiment, in the high-resolution range, the projection characteristics are close to those of the central projection method (y=f×tanθ) or the equidistant projection method (y=f×θ), which are the projection characteristics of an optical system for normal imaging, so that optical distortion is small and a high-resolution display is possible. Therefore, a natural sense of perspective can be obtained when viewing surrounding vehicles such as preceding and following vehicles, and good visibility can be achieved by suppressing degradation of image quality. It should be noted that the projection characteristic is not limited to the one shown in FIG. 2, since the same effect can be obtained as long as the projection characteristic y(θ) satisfies the condition of the above-mentioned equation 1.

[0022] Next, the configuration of the camera system (image processing device) in this embodiment will be described with reference to FIG. FIG. 3 is a block diagram showing the configuration of the camera system according to the first embodiment. The camera system in this embodiment includes an imaging unit 2, an image processing unit 3, a first display unit 4, a second display unit 5, and the like.

[0023] The imaging unit 2 is composed of an optical system, a sensor, an image output interface, etc. The optical system is composed of one or more optical lenses and is configured to satisfy the condition of equation 1, and is configured to have the characteristics shown in Figure 2, for example. Light that passes through the optical system forms an image on the light-receiving surface of the sensor. The sensor converts the received light into an electrical signal and outputs it as image data. The sensor can be a CMOS image sensor or a CCD image sensor. The image data output from the sensor is output to the image processing unit 3 via an image output interface.

[0024] The image processing unit 3 is composed of an image correction unit 300, a first image generation unit 311, a second image generation unit 312, a display control unit 320, a CPU 330, a RAM 340, a ROM 350, a communication unit 360, etc. The image correction unit 300, the first image generation unit 311, the second image generation unit 312, the display control unit 320, the CPU 330, the RAM 340, the ROM 350, and the communication unit 360 are connected to each other via a bus 399.

[0025] The CPU 330 as a computer controls each operation block of the image processing unit 3, and the ROM 350 as a storage medium stores a control computer program that describes the processing procedures of the CPU 330. The RAM 340 serves as a work memory and temporarily stores the control computer program and data.

[0026] 3 may be realized by hardware, or by causing the CPU 330 to execute a computer program stored in the ROM 350. As the hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), etc. may be used.

[0027] The communication unit 360 communicates with other devices mounted on the vehicle 1, such as a navigation system and a vehicle control device that controls the driving of the vehicle, using protocols such as CAN, FlexRay, and Ethernet.

[0028] The image correction unit 300 performs image processing such as development processing and color adjustment on the input image data from the imaging unit 2, and outputs the result to the first image generation unit 311 and the second image generation unit 312. More specifically, the image data input in a Bayer array from the imaging unit 2 is de-Bayer processed to convert it into image data in an RGB raster format. Furthermore, various correction processes such as white balance adjustment, gain / offset adjustment, gamma processing, and color matrix processing are also performed.

[0029] Note that a part or all of the processing performed by the image correction unit 300 in this embodiment may be performed by the imaging unit 2 by providing an image processing LSI or the like in the imaging unit 2. Alternatively, a part of the image processing may be performed by a logic circuit stacked in the sensor of the imaging unit 2. The first image generation unit 311 and the second image generation unit 312 each cut out a predetermined area from the image data input from the image correction unit 300 to generate and output a first image and a second image. The cut-out areas are configured to be individually settable for the first image generation unit 311 and the second image generation unit 312. Details of the areas cut out by the first image generation unit 311 and the second image generation unit 312 will be described later.

[0030] The display control unit 320 outputs for display at least one of the first image and the second image, which are the cropped image data input from the first image generation unit 311 and the second image generation unit 312, respectively. The display control unit 320 also controls whether the image data is to be displayed on the first display unit 4 or the second display unit 5. If the display resolution and display frequency of the first display unit 4 and the second display unit 5 differ from the cropped image data, the display resolution and display frequency are converted to match those of the respective display units before output.

[0031] Here, the display control unit 320 functions as an output unit that outputs at least one of the first image and the second image for display. Also, the first display unit 4 and the second display unit 5 function as display units that display at least one of the first image and the second image output from the output unit. Note that, although the first display unit 4 and the second display unit 5 are configured separately as display units in the embodiment of Fig. 3, they may share a common display screen.

[0032] Furthermore, it is also possible to perform predetermined geometric transformation processing (for example, distortion correction) on each piece of image data after the cutout. Furthermore, instead of outputting each piece of image data after the cutout, it is also possible to output map images of a car navigation system, CG images for user interfaces for air conditioning, sound, and vehicle control, etc., acquired via the communication unit 360.

[0033] The first display unit 4 is a display that displays images based on input image data, and is installed in the vehicle 1, for example, near the center of the vehicle width direction above the driver's seat, with the display screen facing the rear of the vehicle. Note that a half mirror or the like may be used so that it can be used as a mirror when not used as a display. Alternatively, it may be equipped with a touch panel or operation buttons and be capable of receiving instructions from the user and outputting them to the image processing unit 3.

[0034] The second display unit 5 is a display that displays based on input image data, and is installed, for example, around an operation panel near the center of the vehicle width in front of the driver's seat. Note that the vehicle as a moving body is equipped with a navigation system (not shown), an audio system, a vehicle control device (engine and motor) that controls the driving of the vehicle, and the like.

[0035] For example, the second display unit can display UI images for controlling a navigation system, an audio system, a vehicle control device, etc. Also, it is equipped with a touch panel and operation buttons, and is configured to be able to receive instructions from the user. The display panels of the first display unit 4 and the second display unit 5 may be liquid crystal displays, organic EL displays, or the like.

[0036] Next, the regions cut out by the first image generating section 311 and the second image generating section 312 of the image processing section 3 will be described with reference to FIG. 4A to 4D are diagrams for explaining examples of cut-out areas in Example 1, in which 41 indicates a high-resolution area that is an area less than a predetermined half angle of view θa, and 42 indicates a low-resolution area that is an area equal to or greater than the predetermined half angle of view θa.

[0037] In this embodiment, the first image generation unit 311 generates a first image by cutting out a first region from an image region corresponding to the high-resolution region 41 within an image read from the imaging element. That is, the first image generation unit 311 is controlled by the CPU 330 to cut out an image region 43 for electronic rearview mirror as the first region, which is an image region surrounded by a solid line in FIG. 4 , to generate and output the first image. Furthermore, the display control unit 320 is controlled by the CPU 330 to display the image data of the first image generated by the first image generation unit 311 on the first display unit 4.

[0038] The second image generating unit 312 generates the second image by cutting out a second area from an image area different from the first area in the image read from the imaging element. That is, the second image generating unit 312 is controlled by the CPU 330 to cut out, as the second area, a backup monitor image area 44, which is the image area surrounded by a dashed line in Fig. 4(A), for example, to generate and output the second image.

[0039] At this time, the CPU 330 determines the upper end (upper edge) 44a of the image area 44 for the rearview monitor and the upper end (upper edge) 43a of the image area 43 for the electronic rearview mirror so that the upper end (upper edge) 44a of the image area 44 for the rearview monitor is lower than the upper end (upper edge) 43a of the image area 43 for the electronic rearview mirror.

[0040] Furthermore, the CPU 330 determines the lower end (lower side) 44b of the image area 44 for the rearview monitor and the lower end (lower side) 43b of the image area 43 for the electronic rearview mirror so that the lower end (lower side) 44b of the image area 44 for the rearview monitor is lower than the lower end (lower side) 43b of the image area 43 for the electronic rearview mirror. That is, in the example of Figure 4(A), the image generation unit generates the first image and the second image so that the entire bottom end of the second region is lower than the bottom end of the first region, and so that the entire top end of the second region is lower than the top end of the first region.

[0041] Then, the CPU 330 controls the display control unit 320 to display the image data generated by the first image generation unit 311 on the first display unit 4, and the image data generated by the second image generation unit 312 on the second display unit 5. As a result, a first image obtained by cutting out the image region 43 for the electronic rearview mirror from the image data captured by the imaging unit 2 is displayed on the first display unit 4, and a second image obtained by cutting out the image region 44 for the back monitor is displayed on the second display unit 5.

[0042] In this embodiment, the high-resolution area 41 is configured to have projection characteristics that are similar to the central projection method (y=f×tanθ) or equidistant projection method (y=f×θ) of a normal imaging optical system, as described above. Therefore, the image for the electronic rearview mirror displayed on the first display unit 4 has a higher resolution than that of the low-resolution area 42, and can display the area behind the vehicle in greater detail.

[0043] Furthermore, because the high-resolution area 41 has little optical distortion, the image for the electronic rearview mirror can also be displayed with little distortion, allowing the driver to view the area behind the vehicle with a more natural perspective. Furthermore, the image recognition accuracy can be improved when recognizing the license plate of a vehicle behind or people in the image.

[0044] On the other hand, since the bottom end 44b of the image area 44 for the back monitor is set lower than the bottom end 43b of the image area 43 for the electronic rearview mirror, the rear of the vehicle 1 and the area near the ground, which is in the driver's blind spot, can be displayed and confirmed over a wider area when, for example, backing up the vehicle 1. This is due to the fact that the increase in image height y per unit half angle of view θ is reduced in the low-resolution area 42, and the parameters of the optical system of the imaging unit 2 are set so that a wider angle of view can be captured.

[0045] As described above, the upper end 44a of the rearview monitor image area 44 is set lower than the upper end 43a of the electronic rearview mirror image area 43. Therefore, compared to when the upper end 44a of the rearview monitor image area 44 is set higher than the upper end 43a of the electronic rearview mirror image area 43 or when the captured image is displayed as is as an image for the rearview monitor, the area near the ground can be displayed larger and more precisely, allowing for safer driving.

[0046] Furthermore, if a fisheye lens or the like having the same maximum angle of view as the optical system of the imaging unit 2 of this embodiment is used to extract an image with an angle of view equivalent to the electronic rearview mirror image area 43 of this embodiment, the optical distortion will be large and geometric transformation processing to correct the optical distortion will be necessary. The geometric transformation processing here refers to processing to correct image distortion by enlarging or reducing the image differently for each area when displaying it.

[0047] On the other hand, since the high-resolution area 41 of the imaging unit 2 in this embodiment is configured to reduce optical distortion, geometric transformation processing for correcting optical distortion is not necessary, resulting in the effect of reducing the processing load. In other words, there is no need to perform geometric transformation processing on the first image. On the other hand, the second image may be subjected to geometric transformation processing and output from the output unit.

[0048] As described above, according to this embodiment, by operating the imaging unit 2, the image processing unit 3, the first display unit 4, and the second display unit 5, a high-definition, low-distortion display for an electronic rearview mirror and a wide-area display for rearview confirmation can be obtained with a single camera.

[0049] 4(A) has been described taking as an example a case where both the electronic rearview mirror image area 43 and the back monitor image area 44 are cut out in a rectangular shape. However, the cut-out shape is not limited to a rectangle, as long as the upper end 44a and the lower end 44b of the back monitor image area 44 are determined to be lower than the upper end 43a and the lower end 43b of the electronic rearview mirror image area 43, respectively. For example, as shown in FIG. 4(B), part of the cut-out shape of the back monitor image area 45 may be a curve.

[0050] In this case, the boundary of the area is determined so that the lower end (lower sides 450 to 452) of the backup monitor image area 45 is lower than the lower end (lower side) 43b of the electronic rearview mirror image area 43 and its extension. That is, the vertical direction is the y-coordinate axis, and the y-coordinate value increases upward. In this case, the boundary of the area is determined so that the y-coordinate of the lower end (lower side) 43b of the electronic rearview mirror image area 43 is smaller than the y-coordinate of the lower end (lower side) 43b of the electronic rearview mirror image area 43.

[0051] By making the lower end (lower sides 450 to 452) of the rear monitor image area 45 curved in this way, it is possible to display an area closer to the ground, which is closer to the vehicle, than in the case of Figure 4(A), and to display a wider area for rear confirmation.

[0052] 4C, the electronic rearview mirror image area 43 and the rearview monitor image area 46 may be set separately. In this case, the bottom edge (both ends 460 and 462) of the rearview monitor image area 46, which is located outside the extension lines (two-dot chain lines) of the left edge 43c and the right edge 43d of the electronic rearview mirror image area 43, is set above the bottom edge (bottom side) 43b of the electronic rearview mirror image area 43.

[0053] This allows a single camera to display both a high-definition electronic rearview mirror and a wide area near the ground for rearward confirmation, while also providing a wider view to the left and right. Since optical distortion increases toward the outside of the low-resolution area 42, it is desirable to correct the optical distortion by performing geometric transformation processing in the display control unit 320 before displaying the image.

[0054] In the example of FIG. 4(D), the upper end 47a of the rear monitor image area 47 is set to be above the horizontal line 48 (or the horizon) on the captured image. This setting is more desirable because it allows the driver to more easily grasp the sense of distance behind the vehicle.

[0055] 4(A) to 4(D), the upper ends 44a, 45a, 46a, and 47a of the rearview monitor image areas 44, 45, 46, and 47 are set so that they are below the upper end 43a of the electronic rearview mirror image area 43 and above the lower end 43b of the electronic rearview mirror image area 43. That is, the first image and the second image are generated so that the upper end of the second area is between the upper and lower ends of the first area. Therefore, the electronic rearview mirror image area 43 and the rearview monitor image areas 44 to 47 partially overlap, and the same subject behind the vehicle is displayed on the first display unit 4 and the second display unit 5.

[0056] This provides a further advantage that the images displayed on the first display unit 4 and the second display unit 5 allow the driver to more easily grasp the sense of distance behind the vehicle. Furthermore, in the examples of Figures 4(A) to (D), the right ends of the rearview monitor image areas 44 to 47 are set to the right of the right end of the electronic rearview mirror image area 43, and the left ends of the rearview monitor image areas 44 to 47 are set to the left of the left end of the electronic rearview mirror image area 43.

[0057] That is, the image generation unit generates the first image and the second image so that the left end of the image area 43 for the electronic rearview mirror as the first area is on the left side of the left end of the image area 44-47 for the back monitor as the second area, and the right end of the second area is on the right side of the right end of the first area. By doing this, a single camera can be used to display both a high-definition electronic rearview mirror and a wide area near the ground for rearward confirmation, and furthermore, the rear monitor display can display a wider area in the left and right directions.

[0058] As described above, in this embodiment, the image generator generates the first image and the second image so that at least a portion of the lower edge of the second region is lower than the lower edge of the first region, and so that at least a portion of the upper edge of the second region is lower than the upper edge of the first region. Therefore, the area near the ground can be displayed larger and more precisely than in the prior art.

[0059] In this embodiment, the electronic rearview mirror image area 43 and the rearview monitor image areas 44 to 47 are individually displayed on the first display unit 4 and the second display unit 5, respectively. However, the display method is not limited to this as long as the electronic rearview mirror image area 43 and the rearview monitor image area 44 that satisfy the above-mentioned cut-out conditions can be presented to the driver. For example, the display control unit 320 may combine the electronic rearview mirror image area 43 and the images cut out from the rearview monitor image areas 44 to 47 so that they are arranged side by side, and display them on only one of the first display unit 4 and the second display unit 5.

[0060] Alternatively, only one of the first display section 4 and the second display section 5 may be configured to selectively switch and display images cut out from the electronic rearview mirror image area 43 and the back monitor image areas 44 to 47. Alternatively, the CPU 330 may be configured to acquire information as to whether the drive gear of the vehicle 1 is in reverse gear or not via the communication unit 360. Then, only when it is determined that the vehicle is in reverse gear, an image cut out from the backup monitor image areas 44 to 47 may be displayed on either the first display unit 4 or the second display unit 5. [Example]

[0061] Next, a description will be given of Example 2. This example differs from Example 1 in that the image processing unit 3 changes the size of the cropped area of ​​the image to be output in accordance with the control state of the vehicle. The imaging unit 2, image processing unit 3, first display unit 4, second display unit 5 and their internal configurations in this embodiment are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0062] Fig. 5 is a flowchart showing the processing flow of the camera system in Example 2. Note that the operation of each step in Fig. 5 is performed by the CPU 330 as a computer executing a computer program stored in the ROM 350 as a memory.

[0063] 5, the CPU 330 acquires vehicle control information from a vehicle control device (not shown) that is mounted on the vehicle 1 and performs drive control, etc., via the communication unit 360. In this embodiment, for example, traveling speed information in the reverse direction of the vehicle is acquired.

[0064] Next, in step S510, the CPU 330 determines the image area 43 for the electronic rearview mirror and sets the image area 43 for the electronic rearview mirror by outputting area information to the first image generation unit 311. The image area 43 for the electronic rearview mirror is set, for example, within the range of the high-resolution area 41 in FIG. 4(A) as in the first embodiment.

[0065] Next, in step S520, CPU 330 determines whether the vehicle control information acquired in step S500 satisfies a predetermined condition. If it is determined that the condition is satisfied, the process proceeds to step S530, and if it is determined that the condition is not satisfied, the process proceeds to step S535.

[0066] Specifically, in this embodiment, it is determined whether the vehicle's traveling speed in the backward direction is less than a predetermined value based on the traveling speed information acquired in step S500. If CPU 330 determines that the traveling speed is less than the predetermined value, it advances the process to step S530. On the other hand, if CPU 330 determines that the traveling speed of the vehicle in the backward direction is equal to or greater than the predetermined value, it advances the process to step S535.

[0067] Next, in step S530, the CPU 330 selects, as the backup monitor image area, for example, a backup monitor image area 44 with a narrow angle as shown in Fig. 4(A), and sets it by outputting area information to the second image generation unit 312. However, at least a part of the upper end 44a and at least a part of the lower end 44b of the backup monitor image area 44 are set to be lower than the upper end 43a and the lower end 43b of the electronic rearview mirror image area 43, respectively.

[0068] In step S535, the CPU 330 determines the back monitor image area 44 to be wider-angle and sets it by outputting area information to the second image generation unit 312. That is, the area is set to be wider-angle with a relatively larger cropping range than the back monitor image area 44 determined by the CPU 330 in step S530. The first image and second image generated by the image generation step consisting of steps S510, S530, S535, etc. are then output from the display control unit 320 to the display unit for display. (Output Step)

[0069] As described above, the image generating unit is characterized in that it changes the second area in accordance with the vehicle state information. That is, for example, the rear monitor image area 47 shown in Fig. 4(D) is used. The upper end 47a of the rear monitor image area 47 in Fig. 4(D) extends above the upper end 44a of the rear monitor image area 44 in Fig. 4(A) set in step S330.

[0070] In this embodiment, the left edge, right edge and bottom edge of the rear monitor image area 47 are the same as those of the rear monitor image area 44, for example. When the process of step S530 or step S535 is completed, the CPU 330 returns to step S500 and repeats the processes from step S500 to S535 in the same manner.

[0071] As described above, by executing the processes from step S500 to S535 by CPU 330, the display mode can be appropriately changed so that a more important range can be confirmed according to the vehicle control state. In other words, when the vehicle is reversing at a speed faster than a predetermined value, the vehicle will travel a long distance in a short period of time, so the driver needs to check for safety over a wider range or a longer distance.

[0072] Therefore, if it is determined in step S520 that the vehicle speed is faster than a predetermined value, a wider-angle rearview monitor image area 47 is displayed in step S535. Therefore, the upper end 47a of the rearview monitor image area 47 has a wider viewing angle than the upper end 44a of the rearview monitor image area 44 when the driving speed is relatively slow, so the driver can see obstacles in a wider range or at a greater distance, allowing for safer driving.

[0073] On the other hand, when the vehicle is traveling at a relatively slow speed, the driver needs to check bollards, white lines in parking areas, etc. on the displayed image. Therefore, if it is determined in step S520 that the vehicle speed is less than a predetermined value, in step S530, the rear monitor image area 44 is changed so that its upper end 44a is lower than the upper end 47a of the rear monitor image area 47 for when the traveling speed is fast. This causes the area near the ground to be enlarged and displayed, making it easier for the driver to check bollards, white lines in parking areas, etc. near the ground.

[0074] As described above, in this embodiment, a high-resolution display for the electronic rearview mirror and a wide-area display near the ground for rearview confirmation are simultaneously acquired by a single camera, and the cropped area of ​​the image output is changed depending on the vehicle control status. Furthermore, based on the vehicle's traveling speed information included in the vehicle status information, it is determined whether the traveling speed is faster than a predetermined speed, and the upper end of the second area when it is determined that the traveling speed is faster than the predetermined speed is set higher than the upper end of the second area when it is not determined that the traveling speed is faster than the predetermined speed. Therefore, the driver can easily check the more important area depending on the vehicle control status.

[0075] Furthermore, if it is determined that the vehicle is reversing based on the vehicle's traveling direction information included in the vehicle status information, the second image may be output from the output unit, and if it is not determined that the vehicle is reversing, the second image may be controlled not to be output from the output unit. In this embodiment, the case where speed information is used as the vehicle control state has been described as an example, but for example, the CPU 330 may acquire obstacle information around the vehicle from the imaging unit 2 or another sensor, and set the range of the cut-out area based on the obstacle information.

[0076] Specifically, the imaging unit 2 or another sensor detects the presence of an obstacle behind the vehicle 1, and if it is determined that an obstacle is present, the back monitor image area 44 is widened so that the obstacle is included in the back monitor image area 44. If it is determined that no obstacle is present, the back monitor image area 44 is narrowed to display the area near the ground in detail. In other words, if it is determined that an obstacle is present based on obstacle information around the vehicle that is included in the vehicle status information, it is desirable to make the second area larger than when it is determined that no obstacle is present so that the obstacle is included in the second area.

[0077] Note that other sensors for detecting obstacles, such as distance sensors such as LiDAR, sonar, radar, and TOF, may be mounted on the vehicle 1, and when an object is detected closer than a predetermined distance, it may be determined that an obstacle is present. Also, object detection may be performed based on the results of image recognition of image data from the imaging unit 2, and when a predetermined type of object, such as a person, is detected, it may be determined that an obstacle is present.

[0078] Furthermore, if an obstacle is moving, its speed and direction of movement can be detected from the image, and the faster the obstacle is moving, the wider the second area can be in the direction of its speed of movement. This configuration allows for safer driving. It should be noted that the mobile body in the above-described embodiment is not limited to an automobile, but may be any mobile device such as a ship, an airplane, a robot, or a drone, and includes these. This embodiment can also be applied to the case where a moving object is remotely controlled.

[0079] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention.

[0080] Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to a camera system (image processing device) or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the camera system (image processing device) or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0081] 1: Vehicle 2: Imaging unit 3: Image processing section 4: 1st display section 5:Second display section 5 300: Image correction unit 311: First image generation unit 312: Second image generation unit 320: Display control unit 330:CPU 340:RAM 350:ROM 360: Communications Department

Claims

1. an acquisition means for acquiring an image from an imaging means for capturing an image of the rear of the moving object; an image generating means for generating a first image by cutting out a first region formed by a region including the center of a sensor surface of the imaging means from the image, and for generating a second image by cutting out a second region from an image region different from the first region; a display control means for displaying the first image and displaying the second image when the moving object is moving backward; the image generating means generates the first image and the second image so that a lower area of ​​the first area and a partial upper area of ​​the second area overlap each other; The lower end of the second region is entirely below the lower end of the first region, and An image processing device comprising: generating the first image and the second image such that the entire upper end of the second region is below the upper end of the first region.

2. the image is captured using an optical system that forms an optical image on a sensor surface of the imaging means, the sensor surface having a high-resolution area closer to the center and less than a predetermined half angle of view from the center, and a low-resolution area closer to the outer side of the sensor surface and not less than the predetermined half angle of view, The image processing device described in claim 1, characterized in that the image generation means generates the first image by cutting out the first area from an image area corresponding to the high-resolution area, and generates the second image by cutting out the second area from an image area different from the first image and including the low-resolution area.

3. The image generating means The left end of the second region is to the left of the left end of the first region, and 3. The image processing device according to claim 1, wherein the first image and the second image are generated so that the right edge of the second area is located to the right of the right edge of the first area.

4. The image generating means 4. The image processing device according to claim 1, wherein the first image and the second image are generated so that the upper end of the second area is between the upper end and the lower end of the first area.

5. The image generating means 5. The image processing device according to claim 1, wherein the first image and the second image are generated so that the upper end of the second region is above the horizon or horizontal line on the captured image.

6. When the focal length of the imaging means is f, the half angle of view is θ, the image height on the image plane is y, and the projection characteristic representing the relationship between the image height y and the half angle of view θ is y(θ), 6. The image processing device according to claim 1, wherein y(θ) in the first region is greater than f×θ and is different from the projection characteristic in the second region.

7. 7. The image processing device according to claim 6, wherein the first area is configured to have projection characteristics that are approximate to a central projection method (y=f*tan θ) or an equidistant projection method (y=f*θ).

8. When θmax is the maximum half angle of view of the optical system of the imaging means, and A is a predetermined constant, [Equation 1] 8. The image processing apparatus according to claim 6, wherein the image processing apparatus is configured to satisfy the following:

9. 9. The image processing device according to claim 1, wherein the first image is not subjected to geometric transformation processing, and the second image is subjected to predetermined geometric transformation processing and output from the display control means.

10. A communication means for acquiring vehicle status information is provided.

10. The image processing device according to claim 1, wherein the image generating means changes the second area in accordance with the vehicle state information.

11. 11. The image processing device according to claim 10, wherein the image generating means determines whether the vehicle is traveling at a speed faster than a predetermined speed based on the vehicle's traveling speed information included in the vehicle status information, and sets the upper end of the second area when the vehicle is determined to be traveling at a speed faster than the predetermined speed higher than the upper end of the second area when the vehicle is not determined to be traveling at a speed faster than the predetermined speed.

12. 12. The image processing device according to claim 10, wherein when the image generating means determines that an obstacle is present based on obstacle information around the vehicle included in the vehicle status information, the image generating means makes the second area larger than when it determines that no obstacle is present, and includes the obstacle in the second area.

13. When the image generating means determines that the vehicle is moving backward based on the traveling direction information of the vehicle included in the vehicle state information, the image generating means outputs the second image from the display control means; 13. The image processing device according to claim 10, wherein when it is not determined that the vehicle is moving backward, the second image is not output from the display control means.

14. The image processing device according to any one of claims 1 to 13 is installed, a display unit that displays at least one of the first image and the second image output from the display control unit.

15. an acquisition step of acquiring an image from an imaging means for imaging the rear of the moving object; an image generating step of generating a first image by cutting out a first region formed by a region including the center of a sensor surface of the imaging means from the image, and generating a second image by cutting out a second region from an image region different from the first region; a display control means for displaying the first image and displaying the second image when the moving body is reversing; The image generating step generates the first image and the second image so that a lower area of ​​the first area and a partial upper area of ​​the second area overlap each other, and The lower end of the second region is entirely below the lower end of the first region, and A control method comprising generating the first image and the second image such that the entire upper end of the second area is below the upper end of the first area.

16. A computer program for controlling the image processing device according to any one of claims 1 to 13 or the means of the moving body according to claim 14 by a computer.

Images