Image processing apparatus, movable body, control method, and computer program

Abstract

To achieve an image processing apparatus that can create, by using an image from one imaging means, a first image with high resolution and a second image with low resolution capable of displaying and checking a wide range of a lower part.SOLUTION: An image processing apparatus has: acquisition means that acquires an image from imaging means that picks up an image of the rear of a movable body; image creation means that cuts out, from the image, a first area formed of an area including the center of a sensor surface of the imaging means to create a first image, and cuts out a second area from an image area different from the first area to create a second image; and display control means that displays the first image and displays the second image when the movable body moves backward. The image creation means creates the first image and the second image so that a lower area of the first area and a part of an upper area of the second area overlap each other.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device capable of acquiring images with different resolutions, a moving object, a control method, and a computer program. [Background technology]

[0002] In recent years, there has been a demand to replace the room mirror (rear-view mirror) installed in a vehicle with an electronic room mirror. Patent Document 1 discloses an electronic room mirror system that is composed of an imaging means having an imaging range of the area behind the vehicle outside and a display means inside the vehicle, and that displays an image captured by the imaging means on a display inside the vehicle, allowing the driver to check the situation behind the vehicle outside.

[0003] On the other hand, there is a rearview confirmation system that allows the driver to check the blind spot behind the vehicle when reversing, etc. Patent Document 2 discloses a rearview confirmation system that allows the driver to check the blind spot behind the vehicle when reversing, etc. by installing a camera to capture an image behind the vehicle and displaying the captured image inside the vehicle cabin. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-95202 A [Patent Document 2] JP 2004-345554 A Summary of the Invention [Problem to be solved by the invention]

[0005] The camera serving as the imaging means for capturing images for the electronic rear-view mirror described above is required to have a high resolution so that the driver can check the relatively distant rearward scene in greater detail, whereas the camera for the rearview system is required to capture a wider range of images to check the safety of a wider range, including blind spots behind the vehicle and the rear sides, in order to avoid collisions when reversing, etc.

[0006] Therefore, when installing an electronic rearview mirror system and a rearview confirmation system simultaneously in a vehicle, there is a problem that the in-vehicle camera system becomes complicated if a camera for the electronic rearview mirror system and a camera for the rearview confirmation system are installed separately.

[0007] Therefore, in consideration of the above problems, the present invention aims to provide an image processing device capable of generating a high-resolution first image and a low-resolution second image that can display and confirm a wide area below from an image captured by a single imaging means. [Means for solving the problem]

[0008] In order to achieve the above object, an image processing device according to one aspect of the present invention comprises: An acquisition means for acquiring an image from an imaging means for imaging the rear of the moving object; an image generating means for generating a first image by cutting out a first region formed in a region including the center of a sensor surface of the imaging means from the image, and for generating a second image by cutting out a second region from an image region different from the first region; a display control means for displaying the first image and displaying the second image when the moving object is moving backward; The image generating means generates the first image and the second image such that a lower area of ​​the first area and a partial upper area of ​​the second area overlap with each other. Effect of the Invention

[0009] According to the present invention, it is possible to realize an image processing device capable of generating a high-resolution first image and a low-resolution second image that can display and confirm a wide area below from an image captured by a single imaging means. [Brief description of the drawings]

[0010] [Figure 1] 2 is a diagram illustrating the positional relationship between a vehicle 1 and an imaging unit 2. FIG. [Diagram 2] 1A and 1B are diagrams for explaining optical characteristics of the imaging unit 2. FIG. [Diagram 3] 1 is a block diagram showing a configuration of a camera system according to a first embodiment. [Figure 4] 1A to 1D are diagrams for explaining examples of cut-out regions in the first embodiment. [Diagram 5] 11 is a flowchart showing a processing flow of the camera system in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions will be omitted or simplified. EXAMPLES

[0012] In this embodiment, an improved method for achieving both a high-definition display for an electronic rear-view mirror and a wide-area display for rearview confirmation using a single camera will be described. FIG. 1 is a diagram illustrating the positional relationship between a vehicle 1 and an imaging unit 2. As shown in FIG. In this embodiment, as shown in Fig. 1, an imaging unit 2 is installed behind a vehicle 1, such as an automobile, which is a moving body. The imaging unit 2 is installed so that the area behind the vehicle, which is a moving body, is its imaging range. The imaging unit 2 has an optical system that forms an optical image, and a sensor that captures the optical image formed by the optical system on the light receiving section of the imaging unit 2. It is preferable to install the imaging unit 2 so that the optical axis of the optical system is approximately horizontal.

[0013] In addition, the imaging unit 2 used in this embodiment has an optical system with a sufficient angle of view and imaging magnification in the central angle of view region in order to obtain a high-definition image of a distant object with a wide angle of view. The optical system of the imaging unit 2 in this embodiment will be described with reference to FIG.

[0014] 2A and 2B are diagrams for explaining the optical characteristics of the imaging unit 2, and FIG. 2A is a diagram showing the image height y of the optical system of the imaging unit 2 in this embodiment at each half angle of view on the sensor surface of the imaging unit 2 in the form of contour lines. FIG. 2B is a diagram showing the projection characteristics showing the relationship between the image height y and the half angle of view θ of the optical system of the imaging unit 2 in this embodiment. In FIG. 2B, the half angle of view (angle between the optical axis and the incident light) θ is shown on the horizontal axis, and the imaging height (image height) y on the sensor surface (image surface) of the imaging unit 2 is shown on the vertical axis.

[0015] As shown in FIG. 2B, the optical system of the imaging unit 2 in this embodiment is configured such that its projection characteristic y(θ) differs between an area less than a predetermined half angle of view θa and an area equal to or greater than the half angle of view θa. Therefore, when the increase in image height y per unit half angle of view θ is called resolution, the resolution differs depending on the area. This local resolution can also be expressed as the differential value dy(θ) / dθ of the projection characteristic y(θ) at the half angle of view θ. That is, the greater the inclination of the projection characteristic y(θ) in FIG. 2B, the higher the resolution. It can also be said that the greater the interval between the image heights y at each half angle of view on the contour lines in FIG. 2A, the higher the resolution.

[0016] In this embodiment, the area near the center formed on the sensor surface when the half angle of view θ is less than a predetermined half angle of view θa is called the high-resolution area 41 (FIG. 2(A)), and the area near the outside where the half angle of view θ is equal to or greater than the predetermined half angle of view θa is called the low-resolution area 42 (FIG. 2(A)).

[0017] The optical system of the imaging unit 2 in this embodiment is configured so that its projection characteristic y(θ) is greater than f×θ in the high resolution region 41 (f is the focal length of the optical system of the imaging unit 2). In addition, the projection characteristic y(θ) in the high resolution region is set to be different from the projection characteristic in the low resolution region. Furthermore, when θmax is the maximum half angle of view of the optical system of the imaging unit 2, the ratio θa / θmax of θa to θmax is preferably equal to or greater than a predetermined lower limit, for example, 0.15 to 0.16.

[0018] Also, the ratio θa / θmax of θa to θmax is preferably equal to or less than a predetermined upper limit, for example, 0.25 to 0.35. For example, if θmax is 90°, the predetermined lower limit is 0.15, and the predetermined upper limit is 0.35, then θa is preferably determined within the range of 13.5 to 31.5°. Furthermore, the optical system of the imaging unit 2 is configured so that its projection characteristic y(θ) also satisfies the following formula 1.

[0019]

number

[0020] By configuring the optical system as described above, resolution can be obtained in the high resolution region 41, while the increase in image height y per unit half angle of view θ can be reduced in the low resolution region 42, making it possible to capture a wider angle of view. Therefore, high resolution can be obtained in the high resolution region 41 while providing an imaging range with a wide angle of view equivalent to that of a fisheye lens.

[0021] Furthermore, in this embodiment, in the high-resolution region, the projection characteristics are close to those of the central projection method (y=f×tanθ) and the equidistant projection method (y=f×θ) of the normal imaging optical system, so that the optical distortion is small and it is possible to display in detail. Therefore, a natural perspective can be obtained when viewing surrounding vehicles such as a preceding vehicle and a following vehicle, and good visibility can be obtained by suppressing deterioration of image quality. It should be noted that the projection characteristic y(θ) is not limited to that shown in FIG. 2, since the same effect can be obtained as long as the projection characteristic y(θ) satisfies the condition of the above-mentioned formula 1.

[0022] Next, the configuration of a camera system (image processing device) in this embodiment will be described with reference to FIG. FIG. 3 is a block diagram showing the configuration of the camera system according to the first embodiment. The camera system in this embodiment includes an imaging unit 2, an image processing unit 3, a first display unit 4, a second display unit 5, and the like.

[0023] The imaging unit 2 is composed of an optical system, a sensor, an image output interface, etc. The optical system is composed of one or more optical lenses and is configured to satisfy the condition of mathematical expression 1, and is configured to have the characteristics shown in FIG. The light that passes through the optical system forms an image on the light receiving surface of the sensor. The sensor converts the received light into an electrical signal and outputs it as image data. The sensor may be a CMOS image sensor or a CCD image sensor. The image data output from the sensor is output to the image processing unit 3 via an image output interface.

[0024] The image processing unit 3 is composed of an image correction unit 300, a first image generation unit 311, a second image generation unit 312, a display control unit 320, a CPU 330, a RAM 340, a ROM 350, a communication unit 360, etc. The image correction unit 300, the first image generation unit 311, the second image generation unit 312, the display control unit 320, the CPU 330, the RAM 340, the ROM 350, and the communication unit 360 are each connected to each other via a bus 399.

[0025] The CPU 330 as a computer controls each operation block of the image processing unit 3, and the ROM 350 as a storage medium stores a control computer program describing the processing procedures of the CPU 330. The RAM 340 serves as a work memory in which the control computer program and data are temporarily stored.

[0026] 3 may be realized by hardware, or by causing the CPU 330 to execute a computer program stored in the ROM 350. As the hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), etc. may be used.

[0027] The communication unit 360 communicates with other devices mounted on the vehicle 1, such as a navigation system and a vehicle control device that controls the driving of the vehicle, using a protocol such as CAN, FlexRay, or Ethernet.

[0028] The image correction unit 300 performs image processing such as development processing and color adjustment on the input image data from the imaging unit 2, and outputs the result to the first image generation unit 311 and the second image generation unit 312. More specifically, the image data input in a Bayer array from the imaging unit 2 is de-Bayer processed, and converted into image data in an RGB raster format. In addition, various correction processes such as white balance adjustment, gain / offset adjustment, gamma processing, and color matrix processing are also performed.

[0029] Incidentally, a part or all of the processing performed by the image correction unit 300 in this embodiment may be configured to be performed by the imaging unit 2 by providing an image processing LSI or the like in the imaging unit 2. Alternatively, a part of the image processing may be performed by a logic circuit stacked in the sensor of the imaging unit 2. The first image generating unit 311 and the second image generating unit 312 each cut out a predetermined area from the image data input from the image correction unit 300 to generate and output a first image and a second image. The cut-out areas are configured to be individually settable for the first image generating unit 311 and the second image generating unit 312. Details of the areas cut out by the first image generating unit 311 and the second image generating unit 312 will be described later.

[0030] The display control unit 320 outputs for display at least one of the first image and the second image, which are the cut-out image data input from the first image generation unit 311 and the second image generation unit 312, respectively. It also controls whether the image data is to be displayed on the first display unit 4 or the second display unit 5. If the display resolution and display frequency of the first display unit 4 and the second display unit 5 are different from the cut-out image data, it outputs the image data after converting the display resolution and display frequency to match those of the respective display units.

[0031] Here, the display control unit 320 functions as an output unit that outputs at least one of the first image and the second image for display. Also, the first display unit 4 and the second display unit 5 function as display units that display at least one of the first image and the second image output from the output unit. Note that, although the first display unit 4 and the second display unit 5 as display units are separate in the embodiment of Fig. 3, a common display screen may be used.

[0032] In addition, it is also possible to perform a predetermined geometric transformation process (for example, distortion correction, etc.) on each image data after the cutout. In addition, instead of outputting each image data after the cutout, it is also possible to output a map image of a car navigation system, a CG image for a user interface for air conditioning, audio, or vehicle control, etc., acquired via the communication unit 360.

[0033] The first display unit 4 is a display that performs display based on input image data, and is installed in the vehicle 1, for example, near the center in the vehicle width direction at the front upper part of the driver's seat, with the display screen facing the rear of the vehicle. Note that a half mirror or the like may be used so that it can be used as a mirror when not used as a display. Also, it may be equipped with a touch panel and operation buttons, and be configured to receive instructions from the user and output them to the image processing unit 3.

[0034] The second display unit 5 is a display that displays based on input image data, and is installed, for example, around an operation panel near the center of the vehicle width direction in front of the driver's seat. Note that the vehicle as a moving body is equipped with a navigation system (not shown), an audio system, a vehicle control device (engine and motor) that controls the driving of the vehicle, and the like.

[0035] For example, the second display unit can display UI images for controlling a navigation system, an audio system, a vehicle control device, etc. Also, the second display unit is equipped with a touch panel and operation buttons, and is configured to be able to receive instructions from the user. As the display panels of the first display unit 4 and the second display unit 5, a liquid crystal display, an organic EL display, or the like can be used.

[0036] Next, the regions cut out by the first image generating section 311 and the second image generating section 312 of the image processing section 3 will be described with reference to FIG. 4(A) to (D) are diagrams for explaining examples of cut-out areas in the first embodiment, in which 41 indicates a high-resolution area that is an area less than a predetermined half angle of view θa, and 42 indicates a low-resolution area that is an area equal to or greater than the predetermined half angle of view θa.

[0037] The first image generating unit 311 in this embodiment generates a first image by cutting out a first region from an image region corresponding to the high resolution region 41 in an image read out from the imaging element. That is, the first image generating unit 311 is controlled by the CPU 330 to cut out an image region for electronic rear-view mirror 43 as a first region, which is an image region surrounded by a solid line in Fig. 4, to generate and output the first image. Furthermore, the display control unit 320 is controlled by the CPU 330 to display the image data of the first image generated by the first image generating unit 311 on the first display unit 4.

[0038] The second image generating unit 312 generates the second image by cutting out a second region from an image region different from the first region in the image read from the imaging element. That is, the second image generating unit 312 is controlled by the CPU 330 to cut out the image region 44 for rear monitor as the second region, which is the image region surrounded by the dashed line in Fig. 4(A), to generate and output the second image.

[0039] At this time, the CPU 330 determines the upper end (upper edge) 44a of the image area 44 for the rear monitor and the upper end (upper edge) 43a of the image area 43 for the electronic rearview mirror so that the upper end (upper edge) 44a of the image area 44 for the rearview monitor is lower than the upper end (upper edge) 43a of the image area 43 for the electronic rearview mirror.

[0040] Furthermore, the CPU 330 determines the lower end (lower side) 44b of the image area 44 for the rear monitor and the lower end (lower side) 43b of the image area 43 for the electronic rearview mirror so that the lower end (lower side) 44b of the image area 44 for the rearview monitor is lower than the lower end (lower side) 43b of the image area 43 for the electronic rearview mirror. That is, in the example of Figure 4(A), the image generation unit generates the first image and the second image so that the entire bottom end of the second region is lower than the bottom end of the first region, and so that the entire top end of the second region is lower than the top end of the first region.

[0041] Then, the CPU 330 controls the display control unit 320 to display the image data generated by the first image generation unit 311 on the first display unit 4, and the image data generated by the second image generation unit 312 on the second display unit 5. As a result, a first image obtained by cutting out the image region 43 for the electronic rearview mirror from the image data captured by the imaging unit 2 is displayed on the first display unit 4, and a second image obtained by cutting out the image region 44 for the back monitor is displayed on the second display unit 5.

[0042] In this embodiment, the high-resolution area 41 is configured to have projection characteristics similar to the central projection method (y=f×tanθ) or equidistant projection method (y=f×θ) of a normal imaging optical system as described above. Therefore, the image for the electronic rear-view mirror displayed on the first display unit 4 has a higher resolution than the low-resolution area 42, and can display the rear of the vehicle in greater detail.

[0043] In addition, since the high-resolution area 41 has small optical distortion, the image for the electronic rearview mirror can be displayed with small distortion, allowing the driver to view the rear of the vehicle with a more natural perspective. In addition, the accuracy of image recognition can be improved when recognizing the license plate of the rear vehicle or people.

[0044] On the other hand, since the bottom end 44b of the image area 44 for the rear monitor is set lower than the bottom end 43b of the image area 43 for the electronic rearview mirror, the rear of the vehicle 1 and the vicinity of the ground, which is in the driver's blind spot, can be displayed and confirmed over a wider range when, for example, backing up the vehicle 1. This is due to the fact that the increase in image height y per unit half angle of view θ is reduced in the low-resolution area 42, and the parameters of the optical system of the imaging unit 2 are set so that a wider angle of view can be captured.

[0045] As described above, the upper end 44a of the rearview monitor image area 44 is set lower than the upper end 43a of the electronic rearview mirror image area 43. Therefore, compared to a case where the upper end 44a of the rearview monitor image area 44 is set higher than the upper end 43a of the electronic rearview mirror image area 43 or a case where the captured image is displayed as it is as an image for the rearview monitor, the vicinity of the ground can be displayed larger and more precisely, allowing for safer driving.

[0046] Furthermore, if a fisheye lens having the same maximum angle of view as the optical system of the imaging unit 2 in this embodiment is used to extract an image with an angle of view equivalent to the electronic rearview mirror image area 43 in this embodiment, the optical distortion will be large and a geometric transformation process will be required to correct the optical distortion. The geometric transformation process here refers to a process of correcting image distortion by enlarging or reducing the image differently for each area when it is displayed.

[0047] On the other hand, the high resolution area 41 of the imaging unit 2 in this embodiment is configured to reduce optical distortion, so that the geometric transformation process for correcting the optical distortion is unnecessary, and the processing load can be reduced. That is, the geometric transformation process does not need to be performed on the first image. On the other hand, the geometric transformation process may be performed on the second image and output from the output unit.

[0048] As described above, according to this embodiment, by operating the imaging unit 2, each part of the image processing unit 3, the first display unit 4, and the second display unit 5, it is possible to obtain a high-definition, low-distortion display for an electronic rearview mirror and a wide-area display for rear confirmation with a single camera.

[0049] In the example of Fig. 4(A), both the electronic rearview mirror image area 43 and the rearview monitor image area 44 are cut out in a rectangular shape. However, the cut-out shape is not limited to a rectangle as long as the upper end 44a and the lower end 44b of the rearview monitor image area 44 are determined to be lower than the upper end 43a and the lower end 43b of the electronic rearview mirror image area 43, respectively. For example, as shown in Fig. 4(B), part of the cut-out shape of the rearview monitor image area 45 may be a curve.

[0050] In this case, the boundary of the area is determined so that the bottom end (bottom sides 450-452) of the rear monitor image area 45 is lower than the bottom end (bottom side) 43b of the electronic rearview mirror image area 43 and its extension line. That is, the vertical direction is the y coordinate axis, and the y coordinate value increases in the upward direction. In this case, the boundary of the area is determined so that the y coordinate of the bottom end (bottom side) 43b of the electronic rearview mirror image area 43 is smaller than the y coordinate of the bottom end (bottom side) 43b of the electronic rearview mirror image area 43.

[0051] By making the lower end (lower sides 450 to 452) of the rear monitor image area 45 curved in this manner, it is possible to display an area near the ground that is closer to the vehicle than in the case of FIG. 4(A), and to perform a wider range of rearward confirmation display.

[0052] In addition, when it is desired to display a wider range in the left and right directions in the rearview monitor display, it is advisable to set the electronic rearview mirror image area 43 and the rearview monitor image area 46, for example, as shown in Fig. 4(C). In this case, the lower end (both ends 460 and 462) of the rearview monitor image area 46, which is located outside the extension lines (two-dot chain lines) of the left end 43c and right end 43d of the electronic rearview mirror image area 43, is set to be above the lower end (lower side) 43b of the electronic rearview mirror image area 43.

[0053] In this way, a single camera can be used to both display a high-definition electronic rearview mirror image and a wide area near the ground for rearward confirmation, while also providing a wider area of ​​view to the left and right. Incidentally, since optical distortion increases toward the outside of the low-resolution area 42, it is preferable that the display control unit 320 corrects the optical distortion by performing a geometric transformation process before displaying the area.

[0054] In the example of FIG. 4(D), the upper end 47a of the rear monitor image area 47 is set so as to be above the horizontal line 48 (or the horizon) on the captured image. This is preferable because it allows the driver to more easily grasp the sense of distance behind the vehicle.

[0055] 4(A)-(D), the upper ends 44a, 45a, 46a, and 47a of the image areas 44, 45, 46, and 47 for the rear monitor are set so as to be lower than the upper end 43a of the image area 43 for the electronic rear-view mirror and higher than the lower end 43b of the image area 43 for the electronic rear-view mirror. That is, the first image and the second image are generated so that the upper end of the second area is between the upper and lower ends of the first area. Therefore, the image area 43 for the electronic rear-view mirror and the image areas 44-47 for the rear monitor are partially overlapped, and the same subject is displayed behind the vehicle on the first display unit 4 and the second display unit 5.

[0056] This provides a further effect that the driver can more easily grasp the sense of distance behind the vehicle in the images displayed on the first display unit 4 and the second display unit 5. Furthermore, in the example of Figures 4(A) to (D), the right ends of the rear monitor image areas 44 to 47 are set to the right of the right end of the electronic rearview mirror image area 43, and the left ends of the rear monitor image areas 44 to 47 are set to the left of the left end of the electronic rearview mirror image area 43.

[0057] In other words, the image generation unit generates the first image and the second image so that the left end of the image area 43 for the electronic rearview mirror as the first area is on the left side of the left end of the image area 44-47 for the rear monitor as the second area, and the right end of the second area is on the right side of the right end of the first area. This makes it possible to use a single camera to both display a high-definition electronic rearview mirror and a wide area near the ground for rearward confirmation, while also enabling a wider range of left and right views to be displayed on the rear monitor.

[0058] As described above, in this embodiment, the image generating unit generates the first image and the second image such that at least a part of the lower end of the second region is lower than the lower end of the first region, and at least a part of the upper end of the second region is lower than the upper end of the first region. Therefore, the vicinity of the ground can be displayed larger and more precisely than in the conventional technology.

[0059] In this embodiment, the electronic rearview mirror image area 43 and the rearview monitor image areas 44 to 47 are individually displayed on the first display unit 4 and the second display unit 5, respectively. However, the display method is not limited to this as long as the electronic rearview mirror image area 43 and the rearview monitor image area 44 that satisfy the above-mentioned cut-out conditions can be presented to the driver. For example, the display control unit 320 may combine the images cut out from the electronic rearview mirror image area 43 and the rearview monitor image areas 44 to 47 so as to be arranged side by side, and display them on only one of the first display unit 4 and the second display unit 5.

[0060] Also, the image cut out from the electronic rearview mirror image area 43 and the rear monitor image areas 44 to 47 may be selectively switched and displayed on only one of the first display unit 4 and the second display unit 5. Also, the CPU 330 may be configured to obtain information as to whether or not the driving gear of the vehicle 1 is in reverse gear via the communication unit 360. Then, only when it is determined that the vehicle 1 is in reverse gear, an image cut out from the rear monitor image areas 44-47 may be displayed on either the first display unit 4 or the second display unit 5. EXAMPLES

[0061] Next, a description will be given of Example 2. This example differs from Example 1 in that the image processor 3 changes the size of the cropped area of ​​the image output in accordance with the control state of the vehicle. Incidentally, the imaging unit 2, image processing unit 3, first display unit 4, second display unit 5 and their internal configurations in this embodiment are similar to those in the first embodiment, and therefore description thereof will be omitted.

[0062] Fig. 5 is a flowchart showing a process flow of the camera system in Example 2. Note that the operation of each step in Fig. 5 is performed by the CPU 330 as a computer executing a computer program stored in the ROM 350 as a memory.

[0063] 5, the CPU 330 acquires vehicle control information from a vehicle control device (not shown) that is mounted on the vehicle 1 and performs drive control and the like, via the communication unit 360. In this embodiment, for example, traveling speed information in the reverse direction of the vehicle is acquired.

[0064] Next, in step S510, the CPU 330 determines the image area 43 for the electronic rear-view mirror, and sets the image area 43 for the electronic rear-view mirror by outputting area information to the first image generating unit 311. The image area 43 for the electronic rear-view mirror is set within the range of the high resolution area 41 in FIG. 4(A), for example, as in the first embodiment.

[0065] Next, in step S520, CPU 330 determines whether or not the vehicle control information acquired in step S500 satisfies a predetermined condition. If it is determined that the condition is satisfied, the process proceeds to step S530, and if it is determined that the condition is not satisfied, the process proceeds to step S535.

[0066] Specifically, in this embodiment, it is determined whether the vehicle's traveling speed in the backward direction is less than a predetermined value based on the traveling speed information acquired in step S500. When CPU 330 determines that the traveling speed is less than the predetermined value, it advances the process to step S530. On the other hand, when CPU 330 determines that the traveling speed of the vehicle in the backward direction is equal to or greater than the predetermined value, it advances the process to step S535.

[0067] Next, in step S530, the CPU 330 selects, as the rear monitor image area, for example, a rear monitor image area 44 having a narrow angle as shown in Fig. 4(A) and sets it by outputting area information to the second image generating unit 312. However, at least a part of the upper end 44a and at least a part of the lower end 44b of the rear monitor image area 44 are set to be lower than the upper end 43a and the lower end 43b of the electronic rearview mirror image area 43, respectively.

[0068] In step S535, the CPU 330 determines the rear monitor image area 44 to be wider-angled, and sets it by outputting area information to the second image generating unit 312. That is, the rear monitor image area 44 is set to a wide-angle area with a relatively enlarged cut-out range than the rear monitor image area 44 determined by the CPU 330 in step S530. The first image and the second image generated by the image generating step consisting of steps S510, S530, S535, etc. are then output from the display control unit 320 to the display unit for display. (Output Step)

[0069] As described above, the image generating unit is characterized in that it changes the second area in accordance with the vehicle state information. That is, for example, the rear monitor image area 47 shown in Fig. 4(D) is used. The upper end 47a of the rear monitor image area 47 in Fig. 4(D) extends above the upper end 44a of the rear monitor image area 44 in Fig. 4(A) set in step S330.

[0070] In this embodiment, the left end, right end and bottom end of the rear monitor image area 47 are the same as those of the rear monitor image area 44, for example. Upon completion of the process of step S530 or step S535, CPU 330 returns to step S500, and similarly repeats the processes from step S500 to S535.

[0071] As described above, by executing the processes from step S500 to S535 by CPU 330, the display mode can be appropriately changed so that a more important range can be confirmed according to the vehicle control state. That is, when the traveling speed of the vehicle during backing up is faster than a predetermined value, the vehicle travels a long distance in a short time, so the driver needs to check the safety of a wider range or a longer distance.

[0072] Therefore, when it is determined in step S520 that the vehicle speed is faster than a predetermined value, a wider angle rear-view monitor image area 47 is displayed in step S535. Therefore, the upper end 47a of the rear-view monitor image area 47 has a wider viewing angle above than the upper end 44a of the rear-view monitor image area 44 when the driving speed is relatively slow, so that the driver can check obstacles in a wider range or at a greater distance, and can drive more safely.

[0073] On the other hand, when the vehicle is traveling at a relatively slow speed, the driver needs to check the stop signs, white lines of parking areas, etc. on the displayed image. Therefore, when it is determined in step S520 that the vehicle speed is less than a predetermined value, in step S530, the rear-view monitor image area 44 is changed so that the upper end 44a is lower than the upper end 47a of the rear-view monitor image area 47 when the traveling speed is fast. This causes the area near the ground to be enlarged and displayed, making it easier for the driver to check the stop signs, white lines of parking areas, etc. near the ground.

[0074] As described above, in this embodiment, a high-definition display for an electronic rearview mirror and a wide-area display near the ground for rear confirmation are simultaneously obtained by one camera, and the cropped area of ​​the image to be output is changed according to the vehicle control state. Also, based on the vehicle's running speed information included in the vehicle state information, it is determined whether the running speed is faster than a predetermined running speed, and the upper end of the second area when it is determined that the running speed is faster than the predetermined running speed is set higher than the upper end of the second area when it is not determined that the running speed is faster than the predetermined running speed. Therefore, the driver can easily check the more important area according to the vehicle control state.

[0075] In addition, if it is determined that the vehicle is reversing based on the vehicle's traveling direction information included in the vehicle status information, the second image is output from the output unit, and if it is not determined that the vehicle is reversing, the second image is not output from the output unit. In this embodiment, the case where speed information is used as the vehicle control state has been described as an example, but for example, the CPU 330 may obtain obstacle information around the vehicle from the imaging unit 2 or another sensor, and set the range of the cut-out area based on the obstacle information.

[0076] Specifically, the imaging unit 2 or another sensor detects the presence of an obstacle behind the vehicle 1, and if it is determined that an obstacle is present, the rear monitor image area 44 is widened so that the obstacle is included in the rear monitor image area 44. If it is determined that there is no obstacle, the rear monitor image area 44 is set to a narrow angle condition and the area near the ground is displayed in detail. In other words, if it is determined that there is an obstacle based on obstacle information around the vehicle that is included in the vehicle status information, it is desirable to make the second area larger so that the obstacle is included in the second area, compared to when it is determined that there is no obstacle.

[0077] As another sensor for detecting an obstacle, for example, a distance sensor such as LiDAR, sonar, radar, or TOF may be mounted on the vehicle 1, and when an object is detected at a position closer than a predetermined distance, it may be determined that an obstacle is present. Also, a configuration may be adopted in which object detection is performed based on the result of image recognition of image data from the imaging unit 2, and when a predetermined type of object such as a person is detected, it may be determined that an obstacle is present.

[0078] Furthermore, if an obstacle is moving, its moving speed and direction may be detected from the image, and the faster the obstacle is moving, the wider the second area may be in the direction of its moving speed. With such a configuration, the vehicle can travel safely. In addition, the moving object in the above-described embodiment is not limited to an automobile, but may be any moving device such as a ship, an airplane, a robot, or a drone, and includes them. This embodiment can also be applied to the case where a moving object is remotely controlled.

[0079] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention.

[0080] A computer program for implementing all or part of the control in this embodiment may be supplied to a camera system (image processing device) or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the camera system (image processing device) or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0081] 1: Vehicle 2: Imaging unit 3: Image processing section 4: 1st display section 5:Second display section 5 300: Image correction unit 311: First image generating unit 312: Second image generating unit 320: Display control unit 330: CPU 340:RAM 350:ROM 360: Communications Department

Claims

1. An acquisition means for acquiring an image from an imaging means for imaging the rear of the moving object; an image generating means for generating a first image by cutting out a first region formed by a region including a center of a sensor surface of the imaging means from the image, and for generating a second image by cutting out a second region from an image region different from the first region; a display control means for displaying the first image and displaying the second image when the moving object is moving backward; The image processing device according to claim 1, wherein the image generating means generates the first image and the second image such that a lower area of ​​the first area and a part of an upper area of ​​the second area overlap with each other.

2. the image is captured by using an optical system that forms an optical image on a sensor surface having a high-resolution area closer to the center and less than a predetermined half angle of view from the center of the sensor surface of the imaging means and a low-resolution area on the sensor surface closer to the outer side and not less than the predetermined half angle of view, The image processing device according to claim 1, characterized in that the image generation means generates the first image by cutting out the first area from an image area corresponding to the high resolution area, and generates the second image by cutting out the second area from an image area different from the first image and including the low resolution area.

3. The image generating means includes: The lower end of the second region is entirely below the lower end of the first region, and 3 . The image processing apparatus according to claim 1 , wherein the first image and the second image are generated such that the entire upper end of the second area is below the upper end of the first area.

4. The image generating means includes: The left end of the second region is to the left of the left end of the first region, and 4. The image processing device according to claim 1, wherein the first image and the second image are generated such that a right edge of the second area is located to the right of a right edge of the first area.

5. The image generating means includes:

5. The image processing device according to claim 1, wherein the first image and the second image are generated so that the upper end of the second area is between the upper end and the lower end of the first area.

6. The image generating means includes:

6. The image processing device according to claim 1, wherein the first image and the second image are generated so that the upper end of the second region is above the horizon or horizontal line on the captured image.

7. Let f be the focal length of the imaging means, θ be the half angle of view, y be the image height on the image plane, and y(θ) be the projection characteristic expressing the relationship between the image height y and the half angle of view θ:

7. The image processing device according to claim 1, wherein y(θ) in the first region is greater than f×θ and is different from the projection characteristic in the second region.

8. 8. The image processing device according to claim 7, wherein the first area is configured to have projection characteristics approximating a central projection method (y=f×tan θ) or an equidistant projection method (y=f×θ).

9. Let θmax be the maximum half angle of view of the optical system of the imaging means, and A be a predetermined constant. [0010] 9. The image processing apparatus according to claim 7, wherein the above-mentioned condition is satisfied.

10. 10. The image processing device according to claim 1, wherein the first image is not subjected to a geometric transformation process, and the second image is subjected to a predetermined geometric transformation process and output from the display control means.

11. A communication means for acquiring vehicle status information, 11. The image processing device according to claim 1, wherein the image generating means changes the second area in accordance with the vehicle state information.

12. The image processing device according to claim 11, characterized in that the image generating means determines whether the vehicle is traveling faster than a predetermined traveling speed based on the vehicle's traveling speed information included in the vehicle status information, and sets the upper end of the second area when the vehicle is determined to be traveling faster than the predetermined traveling speed higher than the upper end of the second area when the vehicle is not determined to be traveling faster than the predetermined traveling speed.

13. The image processing device according to claim 11 or 12, characterized in that, when the image generating means determines that there is an obstacle based on obstacle information around the vehicle included in the vehicle status information, the image generating means makes the second area larger than when it determines that there is no obstacle, and includes the obstacle in the second area.

14. When the image generating means determines that the vehicle is moving backward based on the traveling direction information of the vehicle included in the vehicle state information, the image generating means outputs the second image from the display control means; 14. The image processing device according to claim 11, wherein when it is not determined that the vehicle is moving backward, the second image is not output from the display control means.

15. The image processing device according to any one of claims 1 to 14 is installed, a display unit that displays at least one of the first image and the second image output from the display control unit.

16. An acquisition step of acquiring an image from an imaging means for imaging the rear of the moving object; an image generating step of generating a first image by cutting out a first region formed by a region including a center of a sensor surface of the imaging means from the image, and generating a second image by cutting out a second region from an image region different from the first region; a display control means for displaying the first image and displaying the second image when the moving body is moving backward; A control method characterized in that the image generating step generates the first image and the second image so that a lower area of ​​the first area and a portion of an upper area of ​​the second area overlap.

17. A computer program for controlling the image processing device according to any one of claims 1 to 14 or each means of the moving body according to claim 15 by a computer.

Images