Image processing apparatus, moving device, image processing method, and program

Abstract

To allow visual recognition of a situation behind a moving device.SOLUTION: An image processing apparatus has: imaging means that picks up an image of a space behind a first moving device to generate image data; display control means that displays, on a display, first image data obtained by cutting out a center area of an optical axis from the image data; and detection means that detects a second moving device satisfying a predetermined condition from the image data. When the second moving device is detected by the detection means, the display control means displays, on the display, second image data of an area larger than the first image data from the image data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device capable of capturing an image behind a mobile device, a mobile device, an image processing method, and a program. [Background technology]

[0002] Patent document 1 describes a method in which the center of an image is displayed on the display unit during normal driving, and when the distance to the following vehicle is below a threshold, a wide-angle image is displayed on the display unit to reduce the sense of oppression. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-74558 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Technology 1, the decision to switch to a wide-angle image is based on the distance to the following vehicle displayed on the display unit, which has the problem that it is difficult to notice a danger when a dangerous vehicle approaches from an area outside the display unit.

[0005] Therefore, an object of the present invention is to enable the rear situation to be visually confirmed. [Means for solving the problem]

[0006] In order to solve the above problems, the image processing device of the present invention has an imaging means for imaging the rear of a first moving device and generating image data, a display control means for displaying first image data, in which the central area of ​​the optical axis is cut out from the image data, on a display unit, and a detection means for detecting a second moving device that satisfies a predetermined condition from the image data, and is characterized in that when the second moving device is detected by the detection means, the display control means displays second image data of a wider area than the first image data from the image data on the display unit. Effect of the Invention

[0007] According to the present invention, the situation behind can be visually confirmed. [Brief description of the drawings]

[0008] [Figure 1] Fig. 1(A) is a side view of a moving device and an imaging device system in embodiment 1. Fig. 1(B) is a diagram showing, in contour form, image height y at each half angle of view on the light receiving surface of an imaging element by optical system 201 of embodiment 1, and Fig. 1(C) is a diagram showing projection characteristics showing the relationship between image height y and half angle of view θ of optical system 201 of embodiment 1. [Diagram 2] 1 is a block diagram for explaining an example of the configuration of an image processing device 200 according to a first embodiment. [Diagram 3] 13 is a flowchart showing an example of a view angle switching processing program executed by the processing unit 220. [Figure 4] FIG. 11 is a block diagram illustrating an example of the configuration of an image processing device 400 according to a second embodiment. [Diagram 5] 10 is a flowchart showing an example of control for changing the display angle of view based on the operation of a blinker of a vehicle behind in the second embodiment. [Figure 6] FIG. 11 is a block diagram illustrating an example of the configuration of an image processing device 600 according to a third embodiment. [Figure 7] 13 is a flowchart showing an example of control for detecting meandering by a rear vehicle and changing the display angle of view in the fourth embodiment. [Figure 8] FIG. 13 is a block diagram illustrating an example of the configuration of an image processing device 800 according to a fourth embodiment. [Figure 9] 13 is a flowchart showing an example of control for changing the display angle of view upon detection of a passing signal from a vehicle behind in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0010] [Embodiment 1] Fig. 1(A) is a side view of a mobile device and an imaging device system in embodiment 1. As shown in Fig. 1(A), a mobile device 10 (first mobile device) is equipped with an imaging device 20 as an imaging means. The mobile device 10 described here is a vehicle that runs on public roads, such as an automobile, in which a driver 40 rides and which can move to any location.

[0011] The mobile device 10 is provided with a drive source such as an engine or a motor for moving the mobile device 10, a drive control unit for controlling the drive source, and an image processing device 200 including a display unit for displaying image data captured by the imaging device 20. These components together constitute the mobile device.

[0012] The optical system 201 of the imaging device 20 combines multiple lenses to form images with two different angles of view on the light receiving surface of the imaging unit 210. The first angle of view is a narrow angle of view with high resolution and low distortion, with the normal field of view range 30 as the imaging range. This angle of view is used when observing the rear of the mobile device 10 with high resolution when the mobile device 10 is normally traveling. The other angle of view is a wide angle of view with a wide-angle field of view range 31 as the imaging range, capturing a wider range including the normal field of view range 30. This angle of view is used to capture a wide-angle image of the rear of the mobile device 10, for example, when the mobile device 10 is traveling in reverse.

[0013] The optical characteristics of the optical system 201 will be described with reference to Fig. 1(B) and Fig. 1(C). Fig. 1(B) is a diagram showing the image height y at each half angle of view on the light receiving surface of the image sensor included in the imaging unit 210 in the form of contour lines. Fig. 1(C) is a diagram showing the projection characteristics showing the relationship between the image height y and the half angle of view θ of the optical system 201 of the first embodiment. In Fig. 1(C), the half angle of view (angle between the optical axis and the incident light) θ is shown on the horizontal axis, and the image height (image height) y on the light receiving surface (image plane) of the image sensor included in the imaging unit 210 is shown on the vertical axis.

[0014] As shown in FIG. 1C, the optical system 201 is configured such that its projection characteristic y(θ) differs between an area less than a predetermined half angle of view θa and an area equal to or greater than the half angle of view θa. Therefore, when the increase in image height y per unit half angle of view θ is called resolution, the resolution differs depending on the area. This local resolution can also be expressed as the differential value dy(θ) / dθ of the projection characteristic y(θ) at the half angle of view θ. For example, the greater the inclination of the projection characteristic y(θ) in FIG. 1C, the higher the resolution. It can also be said that the greater the interval between the image heights y at each half angle of view on the contour lines in FIG. 1B, the higher the resolution.

[0015] In the first embodiment, when the half angle of view θ is less than a predetermined half angle of view θa, a region closer to the center generated on the light receiving surface of the image sensor is called a high resolution region 201a, and a region closer to the outside where the half angle of view θ is equal to or greater than the predetermined half angle of view θa is called a low resolution region 201b. The angle of view of the high resolution region 201a corresponds to the normal field of view range 30 described above, and the combined angle of view of the high resolution region 201a and the low resolution region 201b corresponds to the wide angle field of view range 31. In the first embodiment, the high resolution region 201a is a low distortion region with relatively little distortion, and the low resolution region 201b is a high distortion region with relatively much distortion. Therefore, in the first embodiment, the high resolution region and the low resolution region may be called a low distortion region and a high distortion region, respectively.

[0016] The optical system 201 is configured so that the projection characteristic y(θ) in the high resolution region (low distortion region) 201a is greater than f×θ (f is the focal length of the optical system 201). In addition, the projection characteristic y(θ) in the high resolution region (low distortion region) is set to be different from the projection characteristic in the low resolution region (high distortion region).

[0017] When θmax is the maximum half angle of view of the optical system 201, it is desirable that the ratio θa / θmax of θa to θmax is equal to or greater than a predetermined lower limit, for example, 0.15 to 0.16 as the predetermined lower limit. It is also desirable that the ratio θa / θmax of θa to θmax is equal to or less than a predetermined upper limit, for example, 0.25 to 0.35. For example, when θa is 90°, the predetermined lower limit is 0.15, and the predetermined upper limit is 0.35, it is desirable to determine θa in the range of 13.5 to 31.5°.

[0018] Furthermore, the optical system 201 is configured such that its projection characteristic y(θ) also satisfies the following equation (1).

[0019]

number

[0020] Here, f is the focal length of the optical system 201 as described above, and A is a predetermined constant. By setting the lower limit to 1, it is possible to increase the central resolution compared to a fisheye lens of the orthogonal projection type (y=f×sinθ) having the same maximum imaging height, and by setting the upper limit to A, it is possible to obtain an angle of view equivalent to that of a fisheye lens while maintaining good optical performance. The predetermined constant A may be determined in consideration of the balance between the resolutions of the high-resolution region and the low-resolution region, and is preferably set to 1.4 to 1.9.

[0021] By configuring the optical system 201 as described above, high resolution can be obtained in the high resolution region 201a, while the increase in image height y per unit half angle of view θ can be reduced in the low resolution region 201b, making it possible to capture a wider angle of view. Therefore, while the imaging range has a wide angle of view equivalent to that of a fisheye lens, high resolution can be obtained in the high resolution region 201a.

[0022] In the first embodiment, in the high-resolution region (low-distortion region), the projection characteristics are approximate to the central projection method (y=f×tanθ) and equidistant projection method (y=f×θ), which are the projection characteristics of a normal imaging optical system. Therefore, optical distortion is small, and it is possible to display in detail. Therefore, a natural sense of perspective can be obtained when viewing a vehicle behind, and good visibility can be obtained by suppressing deterioration of image quality.

[0023] Note that the same effect can be obtained as long as the projection characteristic y(θ) satisfies the condition of the above-mentioned mathematical expression 1, so the first embodiment is not limited to the projection characteristic shown in Fig. 2. Note that in the first embodiment, the optical system 201 having the projection characteristic y(θ) that satisfies the condition of the above-mentioned mathematical expression 1 may be referred to as a different angle of view lens.

[0024] Fig. 2 is a block diagram for explaining an example of the configuration of the image processing device 200 in the first embodiment. Note that one or more components shown in Fig. 2 may be realized by hardware such as an ASIC or a programmable logic array (PLA). The same applies to the block diagrams shown in Figs. 4, 6, and 8 described below.

[0025] The image processing device 200 is a system that displays an image captured by the imaging device 20 installed at the rear of the vehicle on a display device inside the vehicle. The image processing device 200 has the imaging device 20, a processing unit 220, and a display unit 230. The imaging device 20 is composed of an optical system 201 and an imaging unit 210 having the configurations explained in Fig. 1(B) and Fig. 1(C). Here, the imaging device 20 has the optical system 201 that forms an optical image having a low distortion region and a high distortion region on a light receiving surface, and functions as an imaging unit that executes an imaging step of capturing an image behind the first mobile device.

[0026] The processing unit 220 is composed of an image processing unit 221, a display angle of view determination unit 224, a user setting change unit 226, a rear vehicle distance detection unit 223, a reverse gear detection unit 225, and a display angle of view change unit 222. Note that a computer (CPU (Central Processing Unit), microcomputer, etc.) included in the processing unit 220 functions as a control unit that controls the operation of each component of the image processing device 200 based on a computer program stored in a storage medium.

[0027] The imaging unit 210 includes an imaging element (such as a CCD image sensor or a CMOS image sensor), converts an optical subject image formed by the optical system 201 into imaging data, and supplies the data to the image processing unit 221.

[0028] The image processing unit 221 functions as an image processing unit, and performs processes such as WDR (Wide Dynamic Range) correction, gamma correction, LUT (Look Up Table) processing, and distortion correction on the imaging data acquired from the imaging unit 210 to generate image data.

[0029] The distortion correction is performed on at least the imaging data in the high distortion region (low resolution region 201b). This makes it easier to view when displayed on the display unit 230, and improves the recognition rate in the rear vehicle distance detection unit 223. The distortion correction does not need to be performed on the imaging data in the low distortion region (high resolution region 201a). The image processing unit 221 executes the image processing steps as described above, and the image data generated thereby is supplied to the display angle of view changing unit 222 and the rear vehicle distance detection unit 223.

[0030] The display angle of view determination unit 224 determines whether the image to be displayed on the display unit 230 should be in the wide-angle field of view 31 or the normal field of view 30 based on distance information acquired by a rear vehicle distance detection unit 223 described later and the detection result of a reverse gear detection unit 225 described later. Then, the display angle of view determination unit 224 notifies the display angle of view change unit 222 according to the determination result. For example, if the distance information acquired by the rear vehicle distance detection unit 223 is equal to or less than a certain threshold (e.g., 3 m), the display angle of view is determined to be the wide-angle field of view 31, and if it is greater than the threshold, the display angle of view is determined to be the normal field of view 30.

[0031] Moreover, when the reverse gear detection unit 225 notifies that the transmission is in reverse gear, the display angle of view is set to the wide-angle viewing range 31, and when the transmission is not in reverse gear, the display angle of view is set to the normal range. Furthermore, when the reverse gear is detected, the display angle of view is set to the wide-angle viewing range 31 regardless of the result of the rear vehicle distance detection unit 223. When the transmission is not in reverse gear, the display angle of view is determined according to the result of the rear vehicle distance detection unit 223. Moreover, by receiving vehicle type information from the rear vehicle distance detection unit 223 described later, the criterion for determining whether to change the angle of view may be changed according to the vehicle type. For example, since the braking distance of a large vehicle such as a truck is longer than that of a standard vehicle, the threshold value is set to be longer, for example, 10 m, compared to that of a standard vehicle.

[0032] The user setting change unit 226 allows the user to change the determination criterion for whether or not to change to a wide angle in the display angle of view determination unit 224. The changed content is input to the display angle of view determination unit 224.

[0033] The rear vehicle distance detection unit 223 functions as a detection means, and obtains the distance of a rear vehicle present in image data in an area of ​​the wide-angle field of view 31 that does not include the normal field of view 30, using an image processed by the image processing unit 221. For example, the rear vehicle distance detection unit 223 detects a vehicle based on image data other than the first image data described below among image data in an area including a highly distorted area, and calculates the distance between the vehicle and the detected vehicle from the position and size change of the detected vehicle. The distance information calculated by the rear vehicle distance detection unit 223 is notified to the display angle of view determination unit 224.

[0034] Furthermore, the rear vehicle distance detection unit 223 may determine the vehicle type based on data on image features such as shape and color for each vehicle type extracted as a result of machine learning such as deep learning from images of a large number of vehicles collected in advance. The vehicle type information may also be notified to the display angle of view determination unit 224.

[0035] The reverse gear detection unit 225 detects whether the vehicle is in reverse gear or not and notifies the display angle of view determination unit 224 of the detection result.

[0036] The display angle of view change unit 222 generates an image to be displayed on the display unit 230. When the display angle of view determination unit 224 notifies the user that the image is to be displayed in the normal field of view range 30, the display angle change unit 222 cuts out a rectangular narrow-angle image (first image data) from the normal field of view range 30 (low distortion region) in the image data processed by the image processing unit 221, and supplies the cutout to the display unit 230. This causes the display unit 230 to display the first image data.

[0037] Furthermore, when a second moving device that satisfies a predetermined condition is detected from image data of an area including a high distortion area, second image data including the second moving device of the high distortion area is displayed. Note that the second image data may include an image of a low distortion area. Here, the display angle change unit 222 has, as a display control unit, a first display control step of displaying first image data cut out from a low distortion area in the image data, and a second display control step of displaying the above-mentioned second image data under a predetermined condition.

[0038] The image is cut out by storing the image data processed by the image processing unit 221 in a memory such as a RAM and reading out the image to be cut out. The position where the first image data is cut out is a rectangular area in the normal field of view range 30 corresponding to the high resolution area 201a in the image formed by the optical system 201. When the second moving device is detected and the display angle of view determination unit 224 is notified that the image is to be displayed in the wide angle field of view range 31, a rectangular wide angle image (second image data) including the second moving device in the wide angle field of view range 31 is supplied to the display unit 230. Therefore, the second image data including the second moving device in the high distortion area is displayed.

[0039] The image supplied to the display unit 230 by the display angle change unit 222 is subjected to enlargement or reduction processing so as to obtain an optimal size for the size of the display unit 230 before being supplied to the display unit 230. When the display area is switched from the first image data to the second image data, the transition is such that the area gradually widens over transition time t1. Also, when switching back from the second image data to the first image data, the transition is such that the area gradually narrows over transition time t2. For example, the display control unit allows a predetermined transition time when switching between the display of the first image data and the second image data. Here, the transition time t1 and the transition time t2 are different, and may be, for example, 0. <t1<t2とする。

[0040] The transition time t1 may be changed depending on the urgency. For example, when a second moving device approaching the moving device 10 from behind is detected, the transition time t1 is set to a shorter time. If the condition for switching to the second image data is repeatedly satisfied / not satisfied in a short period of time, it becomes difficult to view the image on the display unit 230, so a predetermined waiting time t3 is provided when switching from the second image data to the first image data. The setting range of the waiting time t3 is set to a time longer than 0 sec. It is preferable to make the condition (threshold) when switching the display from the second image data to the first image data different from the condition (threshold) when switching the display from the first image data to the second image data.

[0041] The display unit 230 is a display unit such as a liquid crystal display or an organic EL display, and functions as a display unit that displays image data supplied from the display angle of view changing unit 222 as a display control unit. The display unit 230 is disposed, for example, above the windshield of the mobile device 10, and is used as an electronic rearview mirror.

[0042] Fig. 3 is a flowchart showing an example of a view angle switching processing program executed by the processing unit 220. The flow in Fig. 3 starts when the power source of the moving device 10 is started, and is executed continuously while the power source is driving. Moreover, each step shown in Fig. 3 is controlled by a computer included in the processing unit 220 executing a computer program stored in a storage medium.

[0043] In step S301, the reverse gear detection unit 225 determines whether the vehicle is in reverse gear, and notifies the determination result to the display angle of view determination unit 224. If it is determined that the vehicle is in reverse gear, the process proceeds to step S302. If it is determined that the vehicle is not in reverse gear, the process proceeds to step S303.

[0044] In step S302, display angle of view change unit 222 receives a notification of switching to a wide-angle image from display angle of view determination unit 224, and supplies a rectangular wide-angle image (third image data) including wide-angle viewing range 31 processed by image processing unit 221 to display unit 230. For example, in step S302, display angle of view change unit 222 displays the third image data including a high distortion area behind the first mobile device when the first mobile device is moving backward. Note that the wide-angle image of the third image data may have a different range from the wide-angle image of the second image data. Display unit 230 displays the wide-angle image (third image data) supplied from display angle of view change unit 222, and proceeds to step S304.

[0045] In step S304, the reverse gear detection unit 225 determines whether the vehicle is still in reverse gear, and notifies the display angle of view determination unit 224 of the determination result. While the vehicle is still in reverse gear, the process of step S304 is repeated. If reverse gear is no longer detected, the process proceeds to step S305.

[0046] In step S305, display angle of view changing unit 222 receives a notification of switching to a normal angle of view image from display angle of view determination unit 224, and supplies a rectangular narrow-angle image (first image data) cut out from the normal angle of view range from the image processed by image processing unit 221 to display unit 230. Display unit 230 displays the narrow-angle image (first image data) supplied from display angle of view changing unit 222, and ends the series of processes.

[0047] In step S303, display angle of view changing unit 222 receives a notification of switching to a normal angle of view image from display angle of view determination unit 224, and supplies a narrow-angle image (first image data) obtained by cutting out the normal angle of view range from the image processed by image processing unit 221 to display unit 230. When display unit 230 switches to the narrow-angle image (first image data) supplied from display angle of view changing unit 222, the process proceeds to step S306.

[0048] In step S306, rear vehicle distance detection unit 223 measures the distance between the host vehicle and the rear vehicle based on image data other than the first image data among image data of the area including the highly distorted area, and determines whether the distance is less than a threshold value (e.g., 3 m). If there is a rear vehicle whose distance from the host vehicle is less than the threshold value, the process proceeds to step S307. In this case, step S306 functions as a detection step for detecting a second mobile device that satisfies a predetermined condition from image data of the area including the highly distorted area.

[0049] In the first embodiment, the second mobile device that satisfies the above-mentioned predetermined condition refers to a vehicle (diagonally) behind the vehicle whose distance from the vehicle is less than a predetermined distance (for example, less than 3 m) in image data other than the first image data among the image data of the area including the high distortion area. If there is no rear vehicle whose distance from the vehicle is equal to or less than a threshold in image data other than the first image data among the image data of the area including the high distortion area, it is determined that there is no vehicle diagonally behind. Then, the display of the narrow angle image on the display unit 230 is not changed, and the series of processes is terminated as it is. The above-mentioned distance threshold can be changed by the user in the user setting change unit 226.

[0050] In step S307, display angle of view changing unit 222 receives a notification of switching to a wide-angle image from display angle of view determining unit 224, and supplies a wide-angle image (second image data) including the second moving device in the high distortion region to display unit 230. When display unit 230 switches to the wide-angle image (second image data) supplied from display angle of view changing unit 222, the process proceeds to step S308.

[0051] In step S308, the distance between the vehicle and the rear vehicle is measured in image data other than the first image data among image data of an area including a highly distorted area. Then, the display angle of view determination unit 224 determines whether the distance is equal to or greater than a threshold (e.g., 3 m). For example, it determines whether the distance to the diagonally rear vehicle is equal to or greater than a predetermined threshold. If there is a vehicle or the like diagonally rear whose distance to the vehicle is less than the threshold, the process of step S308 is repeated.

[0052] If the distance between the vehicle and the vehicle diagonally behind is equal to or greater than the threshold, the process proceeds to step S309. The distance threshold in step S308 can also be changed by the user using user setting change unit 226. Furthermore, the threshold set in step S306 and the threshold set in step S308 can be different values.

[0053] In step S309, the display state of the wide-angle image (second image data) is maintained for a predetermined waiting time t3 after the display angle of view changing unit 222 receives a notification of switching to a normal angle of view image from the display angle of view determining unit 224. After the waiting time t3 has elapsed, a narrow-angle image (first image data) is supplied to the display unit 230, and after the display unit 230 switches to the narrow-angle image (first image data) supplied from the display angle of view changing unit 222, the series of processes ends.

[0054] As described above, the type of the second moving device may be detectable by the rear vehicle distance detection unit 223. Then, the conditions for displaying the second image data (for example, a threshold value for the distance to the rear vehicle when detecting the second moving device) may be varied depending on the type.

[0055] [Embodiment 2] 4 is a block diagram for explaining an example of the configuration of an image processing device 400 in embodiment 2. The image processing device 400 is different from the image processing device 200 in embodiment 1 in that the processing unit 220 has a rear vehicle blinker detection unit 424 instead of the rear vehicle distance detection unit 223. In addition, an image processed by the image processing unit 221 is supplied to the display angle of view change unit 222 and the rear vehicle blinker detection unit 424.

[0056] The display angle of view determination unit 224 determines whether the image to be displayed on the display unit 230 should be in the wide-angle field of view 31 or the normal field of view 30, based on the detection information from the rear vehicle blinker detection unit 424 and the detection result from the reverse gear detection unit 225. Then, the display angle of view determination unit 224 notifies the display angle of view change unit 222. If the rear vehicle blinker detection unit 424 notifies that a vehicle emitting a blinker has been detected from the rear, the display angle of view is set to the wide-angle field of view 31 (second image data), and if not detected, the display angle of view is set to the normal field of view 30 (first image data).

[0057] Also, when there is a notification from the reverse gear detection unit 225 that the vehicle is in reverse gear, the display angle of view is set to the wide-angle viewing range 31 (third image data), and when the vehicle is not in reverse gear, the display angle of view is set to the normal viewing range 30 (first image data). Furthermore, when the vehicle is in reverse gear, the display angle of view is set to the wide-angle viewing range 31 (third image data) regardless of the result of the rear vehicle blinker detection unit 424. When the vehicle is not in reverse gear, the first image data or the second image data is displayed depending on the result of the rear vehicle blinker detection unit 424.

[0058] The rear vehicle blinker detection unit 424 detects the blinker operation of the rear vehicle from the image processed by the image processing unit 221, and notifies the detected information to the display angle of view determination unit 224. Here, the rear vehicle blinker detection unit 424 detects a moving device blinking a blinker under a predetermined condition (for example, three or more times in succession) as the second moving device.

[0059] Fig. 5 is a flowchart showing an example of control for changing the display angle of view based on the operation of a blinker of a vehicle behind in embodiment 2. The flow in Fig. 5 is started when the power source of the moving device 10 is started, and is executed continuously while the power source is operating. Moreover, each step shown in Fig. 5 is controlled by a computer included in the processing unit 220 executing a computer program stored in a storage medium.

[0060] In step S501, the reverse gear detection unit 225 determines whether the vehicle is in reverse gear and notifies the display view angle determination unit 224. If it is determined that the vehicle is in reverse gear, the process proceeds to step S502. If it is determined that the vehicle is not in reverse gear, the process proceeds to step S503.

[0061] In step S502, display angle of view changing unit 222 receives a notification of switching to a wide-angle image from display angle of view determining unit 224, and supplies a rectangular wide-angle image (third image data) to display unit 230. When display unit 230 switches to displaying the wide-angle image (third image data) supplied from display angle of view changing unit 222, the process proceeds to step S504.

[0062] In step S504, the reverse gear detection unit 225 determines whether the vehicle is still in reverse gear and notifies the display angle of view determination unit 224. While the vehicle is still in reverse gear, the process of step S504 is repeated. If reverse gear is no longer detected, the process proceeds to step S505.

[0063] In step S505, display angle of view changing unit 222 receives a notification of switching to a normal angle of view image from display angle of view determination unit 224, and supplies a rectangular narrow-angle image (first image data) cut out from the normal angle of view range from the image processed by image processing unit 221 to display unit 230. Display unit 230 displays the narrow-angle image (first image data) supplied from display angle of view changing unit 222, and ends the series of processes.

[0064] In step S503, the display angle of view change unit 222 receives a notification of switching to a normal angle of view image from the display angle of view determination unit 224, and supplies a narrow-angle image (first image data) to the display unit 230. When the display unit 230 displays the narrow-angle image (first image data) supplied from the display angle of view change unit 222, the process proceeds to step S506.

[0065] In step S506, the rear vehicle blinker detection unit 424 detects a vehicle activating a blinker in image data other than the first image data among image data of an area including a highly distorted area, and notifies the detected information to the display angle of view determination unit 224. If a vehicle activating a blinker is detected behind the own vehicle, the process proceeds to step S507. If a vehicle activating a blinker is not detected behind the own vehicle, the display of the narrow-angle image (first image data) on the display unit 230 is not changed, and the series of processes ends as is.

[0066] In step S507, display angle of view changing unit 222 receives a notification of switching to a wide-angle image from display angle of view determining unit 224, and supplies a rectangular wide-angle image (second image data) including the second moving device in the high distortion region to display unit 230. When display unit 230 switches to the wide-angle image (second image data) supplied from display angle of view changing unit 222, the process proceeds to step S508.

[0067] In step S508, the rear vehicle blinker detection unit 424 detects a vehicle activating a blinker in image data other than the first image data among image data of an area including a highly distorted area. If a vehicle activating a blinker is detected behind the own vehicle, the process of step S508 is repeated. If a vehicle activating a blinker is no longer detected behind the own vehicle, the process proceeds to step S509.

[0068] In step S509, the display angle of view changing unit 222 continues to display the second image data for standby time t3 after receiving a notification of switching to a normal angle of view image from the display angle of view determining unit 224. After the standby time t3 has elapsed, the display unit 230 supplies a narrow-angle image (first image data), and the display unit 230 switches to the narrow-angle image (first image data) supplied from the display angle of view changing unit 222, and then the series of processes ends.

[0069] In the second embodiment, in steps S506 and S508, the rear vehicle blinker detection unit 424 detects a vehicle that is activating a blinker in image data other than the first image data among image data of an area including a high distortion area. However, in steps S506 and S508, the rear vehicle blinker detection unit 424 may detect a vehicle that is activating a blinker in the wide-angle field of view range 31.

[0070] [Embodiment 3] 6 is a block diagram for explaining a configuration example of an image processing device 600 in embodiment 3. The image processing device 600 is different from the image processing device 200 in embodiment 1 in that the processing unit 220 has a meandering driving detection unit 624 instead of the rear vehicle distance detection unit 223, and an image processed by the image processing unit 221 is supplied to the display angle of view change unit 222 and the meandering driving detection unit 624.

[0071] The display angle of view determination unit 224 determines whether the image to be displayed on the display unit 230 should be in the wide-angle viewing range 31 or the normal viewing range 30 based on the detection information from the meandering detection unit 624 and the detection result from the reverse gear detection unit 225, and notifies the display angle of view change unit 222 of the determination result. If the number of times that the rear vehicle detected by the meandering detection unit 624 has repeatedly moved alternately left and right around the white line is equal to or greater than a threshold value, the second image data is displayed, and if the number of times is less than the threshold value, the first image data is displayed. Furthermore, if there is a notification from the reverse gear detection unit 225 that the vehicle is in reverse gear, the third image data is displayed, and if the vehicle is not in reverse gear, the first image data is displayed.

[0072] Furthermore, when reverse gear is detected, the third image data is displayed regardless of the result of the meandering detection unit 624. When reverse gear is not in, the first image data or the second image data is displayed according to the result of the meandering detection unit 624. Also, by receiving vehicle type information from the meandering detection unit 624 described later, the criteria for determining whether to change the angle of view may be changed according to the vehicle type.

[0073] For example, in the case of large vehicles such as trucks, the risk of collision is high, so the threshold for the number of round trips is set low so that the vehicle can be recognized earlier than the meandering of a normal vehicle. For example, the meandering detection unit 624 may be able to detect the type of the second mobile device, and the conditions for displaying the second image data (for example, the threshold for the number of times the vehicle crosses the boundary between traffic sections when detecting a vehicle as a meandering vehicle) may be varied depending on the type.

[0074] The meandering detection unit 624 stores image recognition patterns for detecting various objects in advance, and detects patterns in images using a predetermined image recognition algorithm based on the image recognition patterns. The image patterns may be defined by machine learning such as deep learning, or may be image recognition patterns specialized for objects that may be present on roads and in the surrounding areas, such as four-wheeled vehicles, two-wheeled vehicles, pedestrians, road installations, lanes, and white lines, which are objects that may be photographed by an imaging device.

[0075] The meandering detection unit 624 detects a vehicle from the image processed by the image processing unit 221 and detects the number of times that the detected position repeatedly moves alternately left and right with the white line as the boundary. When detecting the repeated movement of the vehicle alternately left and right with the white line as the boundary, the number of times the vehicle moves back and forth is notified to the display angle of view determination unit 224.

[0076] The meandering detection unit 624 holds data on image features such as shape and color for each vehicle type extracted as a result of machine learning such as deep learning from images of many vehicles collected in advance, determines the vehicle type, and notifies the vehicle type information to the display angle of view determination unit 224. Here, the meandering detection unit 624 functions as a detection unit that detects a mobile device that repeatedly crosses the boundary between traffic lanes as a second mobile device.

[0077] Fig. 7 is a flowchart showing an example of control for detecting meandering by a rear vehicle and changing the display angle of view in embodiment 4. The flow in Fig. 7 is started when the power source of the moving device 10 is started, and is executed continuously while the power source is operating. Moreover, each step shown in Fig. 7 is controlled by a computer included in the processing unit 220 executing a computer program stored in a storage medium.

[0078] In step S701, the reverse gear detection unit 225 determines whether the vehicle is in reverse gear and notifies the display view angle determination unit 224. If it is determined that the vehicle is in reverse gear, the process proceeds to step S702. If it is determined that the vehicle is not in reverse gear, the process proceeds to step S703.

[0079] In step S702, display angle of view changing unit 222 receives a notification of switching to a wide-angle image from display angle of view determining unit 224, and supplies the wide-angle image (third image data) to display unit 230. When display unit 230 switches to displaying the wide-angle image (third image data) supplied from display angle of view changing unit 222, the process proceeds to step S704.

[0080] In step S704, the reverse gear detection unit 225 determines whether the vehicle is still in reverse gear and notifies the display angle of view determination unit 224. While the vehicle is still in reverse gear, the process of step S704 is repeated. If reverse gear is no longer detected, the process proceeds to step S705.

[0081] In step S705, display angle of view changing unit 222 receives a notification of switching to a normal angle of view image from display angle of view determination unit 224, and supplies a rectangular narrow-angle image (first image data) cut out from the normal angle of view range from the image processed by image processing unit 221 to display unit 230. Display unit 230 displays the narrow-angle image (first image data) supplied from display angle of view changing unit 222, and ends the series of processes.

[0082] In step S703, the display angle of view changing unit 222 receives a notification of switching to a normal angle of view image from the display angle of view determining unit 224, and supplies the narrow-angle image (first image data) to the display unit 230. When the display unit 230 switches to displaying the narrow-angle image (first image data) supplied from the display angle of view changing unit 222, the process proceeds to step S706.

[0083] In step S706, the meandering detection unit 624 detects the number of times the detected vehicle has repeatedly moved left and right around a lane boundary, such as a white line (or a yellow line, etc.), in the image data of the area including the high distortion area, other than the first image data. Then, the detected information is notified to the display angle determination unit 224. If the number of times the detected vehicle has repeatedly moved left and right around the white line exceeds a threshold (for example, the number of times is two), the vehicle is determined to be a meandering vehicle, and the process proceeds to step S707. If the number of times the detected vehicle has repeatedly moved left and right around the white line does not exceed the threshold, the display of the narrow angle image (first image data) on the display unit 230 is not changed, and the series of processes is terminated as it is.

[0084] In step S707, display angle of view changing unit 222 receives a notification of switching to a wide-angle image from display angle of view determination unit 224, and supplies the wide-angle image (second image data) to display unit 230. For example, the second image data, which is a wide-angle image including a second moving device in a high distortion area, is supplied to display unit 230. When display unit 230 switches to the wide-angle image (second image data) supplied from display angle of view changing unit 222, the process proceeds to step S708.

[0085] In step S708, the meandering detection unit 624 detects the number of times the detected vehicle has repeatedly moved alternately left and right, for example, around a white line, in image data other than the first image data among image data of an area including a highly distorted area. Then, the detected information is notified to the display angle of view determination unit 224. If the number of times the detected vehicle has repeatedly moved alternately left and right, for example, around a white line, exceeds a threshold value (for example, the number of times is two), the vehicle is determined to be meandering, and the process of step S708 is repeated. If the number of times the detected vehicle has repeatedly moved alternately left and right, for example, around a white line, does not exceed the threshold value, the process proceeds to step S709.

[0086] In step S709, the display angle of view changing unit 222 continues to display the second image data for standby time t3 after receiving a notification of switching to a normal angle of view image from the display angle of view determining unit 224. After the standby time t3 has elapsed, a narrow-angle image (first image data) is supplied to the display unit 230, and when the display unit 230 switches to the narrow-angle image (first image data) supplied from the display angle of view changing unit 222, the series of processes ends.

[0087] In steps S706 and S708 of the third embodiment, the number of times the detected vehicle has repeatedly moved back and forth across the lane boundary line is detected in image data other than the first image data among image data of an area including a highly distorted area, and a meandering vehicle is detected. However, in steps S506 and S508, the number of times the detected vehicle has repeatedly moved back and forth across the lane boundary line in the wide-angle field of view 31 may be detected, and a meandering vehicle may be detected based on this.

[0088] [Embodiment 4] 8 is a block diagram for explaining an example of the configuration of an image processing device 800 in the fourth embodiment. The image processing device 800 is different from the image processing device 200 in the first embodiment in that the processing unit 220 has a rear vehicle passing detection unit 824 instead of the rear vehicle distance detection unit 223. In addition, an image processed by the image processing unit 221 is supplied to the display angle of view change unit 222 and the rear vehicle passing detection unit 824.

[0089] The display angle of view determination unit 224 determines whether the image to be displayed on the display unit 230 should be in the wide-angle field of view 31 or the normal field of view 30, based on the detection information from the rear vehicle passing detection unit 824 and the detection result from the reverse gear detection unit 225. Then, the display angle of view determination unit 224 notifies the display angle of view change unit 222 of the determination result. If the rear vehicle passing detection unit 824 detects a vehicle passing to the rear, the second image data is displayed, and if not, the first image data is displayed.

[0090] Also, when there is a notification from the reverse gear detection unit 225 that the vehicle is in reverse gear, the third image data is displayed, and when the vehicle is not in reverse gear, the first image data is displayed. Furthermore, when the vehicle is in reverse gear, the third image data is displayed regardless of the result of the rear vehicle passing detection unit 824. When the vehicle is not in reverse gear, the first image data or the second image data is displayed depending on the result of the rear vehicle passing detection unit 824.

[0091] The rear vehicle passing detection unit 824 detects the passing action of the rear vehicle from the image processed by the image processing unit 221, and notifies the detected information to the display angle of view determination unit 224. Here, the rear vehicle passing detection unit 824 functions as a detection unit that detects a moving device performing a passing action as a second moving device.

[0092] Fig. 9 is a flowchart showing an example of control for detecting a passing operation with a rear vehicle and changing the display angle of view in embodiment 4. The flow in Fig. 9 is started when the power source of the moving device 10 is started, and is executed continuously while the power source is operating. Moreover, each step shown in Fig. 9 is controlled by a computer included in the processing unit 220 executing a computer program stored in a storage medium.

[0093] In step S901, the reverse gear detection unit 225 determines whether the vehicle is in reverse gear and notifies the display view angle determination unit 224. If it is determined that the vehicle is in reverse gear, the process proceeds to step S902. If it is determined that the vehicle is not in reverse gear, the process proceeds to step S903.

[0094] In step S902, display angle of view changing unit 222 receives a notification of switching to a wide-angle image from display angle of view determining unit 224, and supplies the wide-angle image (third image data) to display unit 230. When display unit 230 switches to displaying the wide-angle image (third image data) supplied from display angle of view changing unit 222, the process proceeds to step S904.

[0095] In step S904, the reverse gear detection unit 225 determines whether the vehicle is still in reverse gear and notifies the display angle of view determination unit 224. While the vehicle is still in reverse gear, the process of step S904 is repeated. If reverse gear is no longer detected, the process proceeds to step S905.

[0096] In step S905, the display angle of view changing unit 222 receives a notification of switching to a normal angle of view image from the display angle of view determining unit 224, and supplies a rectangular narrow-angle image (first image data) obtained by cutting out the normal angle of view range from the image processed by the image processing unit 221 to the display unit 230. The display unit 230 displays the narrow-angle image (first image data) supplied from the display angle of view changing unit 222, and ends the series of processes.

[0097] In step S903, the display angle of view change unit 222 receives a notification of switching to a normal angle of view image from the display angle of view determination unit 224, and supplies the narrow-angle image (first image data) to the display unit 230. When the display unit 230 switches to displaying the narrow-angle image (first image data) supplied from the display angle of view change unit 222, the process proceeds to step S906.

[0098] In step S906, the rear vehicle passing detection unit 824 detects a vehicle flashing its lights in image data other than the first image data among image data of an area including a highly distorted area, and notifies the detected information to the display angle of view determination unit 224. If a vehicle flashing its lights is detected behind the own vehicle, the process proceeds to step S907. If a vehicle flashing its lights is not detected behind the own vehicle, the display of the narrow-angle image (first image data) on the display unit 230 is not changed, and the series of processes ends as is.

[0099] In step S907, display angle of view changing unit 222 receives a notification of switching to a wide-angle image from display angle of view determination unit 224, and supplies the wide-angle image (second image data) to display unit 230. For example, the second image data, which is a wide-angle image including the second moving device in the high distortion area, is supplied to display unit 230. When display unit 230 switches to the wide-angle image (second image data) supplied from display angle of view changing unit 222, the process proceeds to step S908.

[0100] In step S908, rear vehicle passing detection unit 824 detects a vehicle flashing its lights in image data other than the first image data among image data of an area including a highly distorted area, and notifies the detected information to display angle of view determination unit 224. If a vehicle flashing its lights behind the host vehicle is detected, the process of step S908 is repeated. If a vehicle flashing its lights behind the host vehicle is no longer detected, the process proceeds to step S909.

[0101] In step S909, the display angle of view changing unit 222 continues to display the second image data for standby time t3 after receiving a notification of switching to a normal angle of view image from the display angle of view determining unit 224. After the standby time t3 has elapsed, a narrow-angle image (first image data) is supplied to the display unit 230. When the display unit 230 switches to the narrow-angle image (first image data) supplied from the display angle of view changing unit 222, the series of processes ends.

[0102] In step S906 and step S908 of the fourth embodiment, a vehicle flashing its lights is detected in image data other than the first image data among image data of an area including a highly distorted area. However, in step S506 and step S508, a vehicle flashing its lights may be detected in the wide-angle field of view 31, thereby detecting a vehicle flashing its lights.

[0103] [Embodiment 5] At least one of the various functions, processes, and methods described in the above-mentioned embodiments can be realized using a program. Hereinafter, in the fifth embodiment, a program for realizing at least one of the various functions, processes, and methods described in the above-mentioned embodiments is called "program X." Furthermore, in the fifth embodiment, a computer for executing program X is called "computer Y." A personal computer, a microcomputer, a CPU (Central Processing Unit), and the like are examples of computer Y. A computer such as an image processing device in the above-mentioned embodiments is also an example of computer Y.

[0104] At least one of the various functions, processes, and methods described in the above-mentioned embodiments can be realized by computer Y executing program X. In this case, program X is supplied to computer Y via a computer-readable storage medium. The computer-readable storage medium in embodiment 5 includes at least one of a hard disk device, a magnetic storage device, an optical storage device, a magneto-optical storage device, a memory card, a ROM, a RAM, and the like. Furthermore, the computer-readable storage medium in embodiment 5 is a non-transitory storage medium.

[0105] [Embodiment 6] The moving device 10 in the above-described embodiment is not limited to an automobile, but may be any device capable of moving, such as a motorcycle, a bicycle, a wheelchair, a ship, an airplane, a robot, or a drone. [Explanation of symbols]

[0106] 10: Mobile device 20: Imaging device 30: Normal field of view 31: Wide-angle viewing range 40: Driver 200: Image processing device 201:Optical system 201a: High resolution area 201b: Low resolution area 210: Imaging unit 220: Processing section 221: Image processing unit 222: Display angle change unit 223: Rear vehicle distance detection unit 224: Display angle determination unit 225: Reverse gear detector 226: User setting change section 230: Display section 400: Image processing device 424: Rear vehicle turn signal detector 600: Image processing device 624: Wandering detection unit 800: Image processing device 824: Rear vehicle passing detection unit

Claims

1. an imaging means for imaging the rear of the first mobile device and generating image data; a display control means for displaying, on a display unit, first image data obtained by cutting out a central area of ​​the optical axis from the image data; a detection means for detecting a second mobile device that satisfies a predetermined condition from the image data; having the display control means, when the second mobile device is detected by the detection means, displays, on the display unit, second image data of a larger area than the first image data from the image data.

13. An image processing device comprising:

2. When the focal length of the optical system is f, the half angle of view is θ, the image height on the image plane is y, and the projection characteristic expressing the relationship between the image height y and the half angle of view θ is y(θ), 2. The image processing apparatus according to claim 1, wherein y(.theta.) in the low distortion region is greater than f.times..theta., and the projection characteristic is different from that in the high distortion region.

3. If θmax is the maximum half angle of view of the optical system, 1<f×sin(θmax) / y(θmax)≦1.9 3. The image processing apparatus according to claim 2, wherein the above-mentioned condition is satisfied.

4. The image processing device according to any one of claims 1 to 3, characterized in that the display control means displays third image data including the high distortion area behind the first mobile device when the first mobile device is moving backward.

5. The image processing device according to any one of claims 1 to 4, characterized in that the detection means detects, in the image data of an area including the high distortion area, a moving device that is present within a predetermined distance from the first moving device as the second moving device in the image data other than the first image data.

6. 6. The image processing device according to claim 1, wherein the detection means detects a mobile device that is flashing a turn signal under a predetermined condition as the second mobile device.

7. 7. The image processing device according to claim 1, wherein the detection means detects a moving device that repeatedly crosses a boundary between traffic sections as the second moving device.

8. 8. The image processing apparatus according to claim 1, wherein the detection means detects a mobile device performing a passing motion as the second mobile device.

9. 9. The image processing device according to claim 1, wherein the display control means provides a predetermined transition time when switching between the display of the first image data and the display of the second image data.

10. 10. The image processing device according to claim 9, wherein the transition time when switching the display from the first image data to the second image data is different from the transition time when switching the display from the second image data to the first image data.

11. The image processing device according to any one of claims 1 to 10, characterized in that the detection means is capable of detecting the type of the second mobile device, and the display control means varies the conditions for displaying the second image data depending on the type.

12. The image processing device according to claim 1 , further comprising an image processing unit that performs distortion correction on the image data captured by the imaging unit to generate image data.

13. A mobile device An imaging means for generating image data; a display control means for displaying first image data obtained by cutting out a central area of ​​the optical axis from the image data; a detection means for detecting a second mobile device that satisfies a predetermined condition from the image data; having the imaging means images a rear area of ​​the moving device, the display control means, when the second mobile device is detected by the detection means, displays second image data from the image data having a larger area than the first image data; A mobile device comprising:

14. an imaging step of capturing an image of a rear area of ​​the first mobile device using an imaging means for generating image data; a first display control step of displaying first image data obtained by cutting out a central area of ​​the optical axis from the image data; a detection step of detecting a second mobile device that satisfies a predetermined condition from the image data; and a second display control step of displaying, when the second mobile device is detected by the detection step, second image data having a larger area than the first image data from the image data.

13. An image processing method comprising:

15. The computer of the image processing device an imaging step of imaging a rear area of ​​the first mobile device using an imaging means; a first display control step of displaying first image data obtained by cutting out a central area of ​​the optical axis from the image data; a detection step of detecting a second mobile device that satisfies a predetermined condition from the image data; a second display control step of displaying second image data having a larger area than the first image data from the image data when the second mobile device is detected by the detection step; A program for executing.

Images