Image processing device, mobile device, image processing method, and program

Abstract

To allow visual recognition of a situation behind a moving device.SOLUTION: An image processing apparatus has: imaging means that picks up an image of a space behind a first moving device to generate image data; display control means that displays, on a display, first image data obtained by cutting out a center area of an optical axis from the image data; and detection means that detects a second moving device satisfying a predetermined condition from the image data. When the second moving device is detected by the detection means, the display control means displays, on the display, second image data of an area larger than the first image data from the image data.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an image processing apparatus capable of photographing the rear of a mobile device, a mobile device, an image processing method, and a program.

Background Art

[0002] Patent Document 1 describes a method of displaying the central part of an image on a display unit during normal driving and displaying a wide-angle image on the display unit to reduce a sense of pressure when the distance from a following vehicle is less than or equal to a threshold value.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Technique 1, since the switching to a wide-angle image is determined based on the distance from a following vehicle displayed in the display unit, there is a problem that it is difficult to notice danger when a dangerous vehicle approaches from an area outside the display unit.

[0005] Therefore, an object of the present invention is to enable visual recognition of the rear situation.

Means for Solving the Problems

[0006] In order to solve the above problems, an image processing apparatus according to the present invention captures an image behind a first mobile device to generate image data single imaging means, The single imaging means generatedThe system includes a display control means for displaying a first image data obtained by extracting the central region of the optical axis from the aforementioned image data on a display unit, and a detection means for detecting a second moving device that satisfies predetermined conditions from the aforementioned image data, wherein the display control means displays a second image data with a wider area than the first image data on the display unit when the second moving device is detected by the detection means. [Effects of the Invention]

[0007] According to the present invention, the situation behind the vehicle can be visually confirmed. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1(A) is a side view of the moving device and imaging device system in Embodiment 1. Figure 1(B) is a diagram showing the contour lines of the image height y at each half-angle of view on the light-receiving surface of the image sensor by the optical system 201 of Embodiment 1, and Figure 1(C) is a diagram showing the projection characteristics representing the relationship between the image height y and the half-angle of view θ of the optical system 201 of Embodiment 1. [Figure 2] This is a block diagram illustrating an example configuration of the image processing device 200 in Embodiment 1. [Figure 3] This flowchart shows an example of a field of view switching processing program executed by the processing unit 220. [Figure 4] This is a block diagram illustrating an example configuration of the image processing device 400 in Embodiment 2. [Figure 5] This flowchart shows an example of controlling the display field of view based on the turn signal operation of a vehicle behind in Embodiment 2. [Figure 6] This is a block diagram illustrating an example configuration of the image processing device 600 in Embodiment 3. [Figure 7] This flowchart shows an example of a control system in Embodiment 4 that detects swerving driving of a vehicle behind and changes the display field of view. [Figure 8]This is a block diagram illustrating an example configuration of the image processing device 800 in Embodiment 4. [Figure 9] This flowchart shows an example of an embodiment 4 in which control is performed to change the display field of view in response to the detection of a passing action with a vehicle behind. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0010] [Embodiment 1] Figure 1(A) is a side view of the mobile device and imaging device system in Embodiment 1. As shown in Figure 1(A), the mobile device 10 (first mobile device) is equipped with an imaging device 20 as an imaging means. The mobile device 10 described here is, for example, a vehicle that travels on public roads, such as an automobile, in which a driver 40 is riding and which can move to any location.

[0011] The mobile device 10 is equipped with a drive source such as an engine or motor for moving the mobile device 10, a drive control unit for controlling the drive source, and an image processing device 200 including a display unit for displaying image data captured by the imaging device 20. The mobile device is composed of these components together.

[0012] The optical system 201 of the imaging device 20 combines multiple lenses to form images of two different angles of view on the light-receiving surface of the imaging unit 210. The first angle of view is a narrow angle of view with high resolution and low distortion, with the normal field of view 30 as the imaging range. This angle of view is used when observing the area behind the mobile device 10 with high resolution during normal driving. The other angle of view is a wide angle of view that captures a wider area including the normal field of view 30, with the wide-angle field of view 31 as the imaging range. This angle of view is used to capture a wide-angle image of the area behind the mobile device 10, for example, when it is driving in reverse.

[0013] Referring to FIGS. 1(B) and 1(C), the optical characteristics of the optical system 201 will be described. FIG. 1(B) is a diagram showing the image height y at each half angle on the light receiving surface of the imaging element included in the imaging unit 210 in a contour line shape. FIG. 1(C) is a diagram showing the projection characteristics representing the relationship between the image height y and the half angle θ of the optical system 201 of Embodiment 1. In FIG. 1(C), the half angle θ (the angle between the optical axis and the incident ray) is taken as the horizontal axis, and the imaging height (image height) y on the light receiving surface (image surface) of the imaging element included in the imaging unit 210 is shown as the vertical axis.

[0014] As shown in FIG. 1(C), the optical system 201 is configured such that its projection characteristics y(θ) are different in the region where the half angle θ is less than a predetermined half angle θa and in the region where the half angle θ is greater than or equal to the half angle θa. Therefore, when the increase amount of the image height y with respect to the half angle θ per unit is defined as the resolution, the resolution varies depending on the region. This local resolution can also be expressed as the differential value dy(θ) / dθ of the projection characteristics y(θ) at the half angle θ. For example, it can be said that the higher the slope of the projection characteristics y(θ) in FIG. 1(C), the higher the resolution. Also, it can be said that the higher the interval of the image height y at each half angle in the contour line shape of FIG. 1(B), the higher the resolution.

[0015] In Embodiment 1, the region closer to the center generated on the light receiving surface of the imaging element when the half angle θ is less than a predetermined half angle θa is called the high-resolution region 201a, and the outer region where the half angle θ is greater than or equal to the predetermined half angle θa is called the low-resolution region 201b. Note that the angular coverage of the high-resolution region 201a corresponds to the above-mentioned normal viewing range 30, and the combined angular coverage of the high-resolution region 201a and the low-resolution region 201b corresponds to the wide-angle viewing range 31. Also, in Embodiment 1, the high-resolution region 201a is a low-distortion region with relatively less distortion, and the low-resolution region 201b is a high-distortion region with relatively more distortion. Therefore, in Embodiment 1, the high-resolution region and the low-resolution region may be referred to as the low-distortion region and the high-distortion region, respectively.

[0016] The optical system 201 is configured such that its projection characteristic y(θ) is greater than f×θ in the high-resolution region (low-distortion region) 201a (where f is the focal length of the optical system 201). Also, the projection characteristic y(θ) in the high-resolution region (low-distortion region) is set to be different from the projection characteristic in the low-resolution region (high-distortion region).

[0017] When θmax is the maximum half-angle that the optical system 201 has, it is desirable that the ratio θa / θmax of θa to θmax is not less than a predetermined lower limit value. For example, it is desirable that the predetermined lower limit value is 0.15 - 0.16. Also, it is desirable that the ratio θa / θmax of θa to θmax is not more than a predetermined upper limit value. For example, it is desirable to set it to 0.25 - 0.35. For example, when θa is 90°, and the predetermined lower limit value is 0.15 and the predetermined upper limit value is 0.35, it is desirable to determine θa within the range of 13.5 - 31.5°.

[0018] Furthermore, the optical system 201 is configured such that its projection characteristic y(θ) also satisfies the following Equation 1.

[0019]

Equation

[0020] Here, f is the focal length of the optical system 201 as described above, and A is a predetermined constant. By setting the lower limit value to 1, the central resolution can be made higher than that of an orthographic projection method (y = f×sinθ) fisheye lens having the same maximum imaging height. By setting the upper limit value to A, a fisheye lens-equivalent angle of view can be obtained while maintaining good optical performance. The predetermined constant A may be determined in consideration of the resolution balance between the high-resolution region and the low-resolution region, and it is desirable to set it to be 1.4 - 1.9.

[0021] By configuring the optical system 201 as described above, high resolution can be obtained in the high-resolution region 201a, while in the low-resolution region 201b, the increase in image height y with respect to the half-angle of view θ per unit can be reduced, making it possible to image a wider angle of view. Therefore, while using a wide angle of view equivalent to that of a fisheye lens as the imaging range, high resolution can be obtained in the high-resolution region 201a.

[0022] In Embodiment 1, in the high-resolution region (low-distortion region), the projection characteristics approximate those of a typical imaging optical system, such as the central projection method (y=f×tanθ) or the equidistant projection method (y=f×θ). Therefore, optical distortion is small, and detailed display is possible. Consequently, a natural sense of depth can be obtained when visually checking vehicles behind, and good visibility can be obtained while suppressing image quality degradation.

[0023] Furthermore, since the same effect can be obtained with any projection characteristic y(θ) that satisfies the conditions of Equation 1 above, Embodiment 1 is not limited to the projection characteristics shown in Figure 2. In Embodiment 1, the optical system 201 having a projection characteristic y(θ) that satisfies the conditions of Equation 1 above may be referred to as a lens with a different field of view.

[0024] Figure 2 is a block diagram illustrating an example configuration of the image processing device 200 in Embodiment 1. Note that one or more components shown in Figure 2 may be implemented by hardware such as an ASIC or a programmable logic array (PLA). The same applies to the block diagrams shown in Figures 4, 6, and 8, which will be described later.

[0025] The image processing device 200 is a system that displays images captured by the imaging device 20 installed at the rear of the vehicle on a display device inside the vehicle. The image processing device 200 includes an imaging device 20, a processing unit 220, and a display unit 230. The imaging device 20 is composed of an optical system 201 and an imaging unit 210, as described in Figures 1(B) and 1(C). Here, the imaging device 20 has an optical system 201 that forms an optical image having low distortion and high distortion regions on the light-receiving surface, and functions as an imaging unit that performs an imaging step to photograph the rear of the first moving device.

[0026] The processing unit 220 consists of an image processing unit 221, a display field of view determination unit 224, a user setting change unit 226, a rear vehicle distance detection unit 223, a reverse gear detection unit 225, and a display field of view change unit 222. The computer (CPU (Central Processing Unit), microcomputer, etc.) included in the processing unit 220 functions as a control unit that controls the operation of each component of the image processing device 200 based on a computer program stored on a storage medium.

[0027] The imaging unit 210 includes an image sensor (such as a CCD image sensor or CMOS image sensor), converts the optical subject image formed by the optical system 201 into imaging data, and supplies it to the image processing unit 221.

[0028] The image processing unit 221 functions as an image processing means and performs processes such as WDR (Wide Dynamic Range) correction, gamma correction, LUT (Look Up Table) processing, and distortion correction on the imaging data acquired from the imaging unit 210 to generate image data.

[0029] Distortion correction is performed on at least the high-distortion region (low-resolution region 201b) of the image data. This makes the image easier to see when displayed on the display unit 230 and improves the recognition rate of the rear vehicle distance detection unit 223. Distortion correction is not required for the image data in the low-distortion region (high-resolution region 201a). The image processing unit 221 performs the image processing steps described above, and the resulting image data is supplied to the display field of view changing unit 222 and the rear vehicle distance detection unit 223.

[0030] The display field of view determination unit 224 determines whether to display the image on the display unit 230 in the wide-angle field of view 31 or the normal field of view 30, based on the distance information acquired by the rear vehicle distance detection unit 223 (described later) and the detection result of the reverse gear detection unit 225 (described later). It then notifies the display field of view change unit 222 according to the determination result. For example, if the distance information acquired by the rear vehicle distance detection unit 223 falls below a certain threshold (for example, 3m), the display field of view is determined to be the wide-angle field of view 31, and if it exceeds the threshold, it is determined to be the normal field of view 30.

[0031] Furthermore, if the reverse gear detection unit 225 notifies that the transmission is in reverse gear, the display field of view is set to the wide-angle field of view range 31, and if the transmission is not in reverse gear, the display field of view is set to the normal range. In addition, when reverse gear is detected, the display field of view is set to the wide-angle field of view range 31 regardless of the result of the rear vehicle distance detection unit 223. If the transmission is not in reverse gear, the display field of view is determined according to the result of the rear vehicle distance detection unit 223. Furthermore, by receiving vehicle type information from the rear vehicle distance detection unit 223, which will be described later, the criteria for determining the field of view change may be changed according to the vehicle type. For example, in the case of large vehicles such as trucks, the braking distance is longer than that of ordinary cars, so the threshold may be made longer, for example, to 10m, compared to ordinary cars.

[0032] The user setting change unit 226 allows the user to change the criteria used by the display field of view determination unit 224 to determine whether or not to change to a wide-angle field of view. The changed settings are input to the display field of view determination unit 224.

[0033] The rear vehicle distance detection unit 223 functions as a detection means and uses the image processed by the image processing unit 221 to obtain the distance of the rear vehicle present in the image data within the wide-angle field of view 31, excluding the normal field of view 30. For example, the rear vehicle distance detection unit 223 detects a vehicle based on image data other than the first image data described later, from the image data in the area including the high-distortion region, and calculates the distance between the vehicle and the detected vehicle from the position and changes in size of the detected vehicle. The distance information calculated by the rear vehicle distance detection unit 223 is notified to the display field of view determination unit 224.

[0034] Furthermore, the rear vehicle distance detection unit 223 may determine the vehicle type based on data regarding image characteristics such as shape and color for each vehicle type, which are extracted as a result of machine learning such as deep learning from a large number of vehicle images collected in advance. The vehicle type information may also be notified to the display field of view determination unit 224.

[0035] The reverse gear detection unit 225 detects whether the vehicle is in reverse gear. The detection result is notified to the display field of view determination unit 224.

[0036] The display field of view changing unit 222 generates the image to be displayed on the display unit 230. If the display field of view determination unit 224 is notified to display within the normal field of view range 30, the image processing unit 221 extracts a rectangular narrow-angle image (first image data) from the normal field of view range 30 (low distortion region) within the processed image data and supplies it to the display unit 230. This causes the display unit 230 to display the first image data.

[0037] Furthermore, if a second moving device that satisfies predetermined conditions is detected from the image data in the region including the high-distortion region, the second image data including the second moving device in the high-distortion region is displayed. The second image data may also include images in the low-distortion region. Here, the display field of view changing unit 222, as a display control unit, has a first display control step of displaying the first image data extracted from the low-distortion region within the image data, and a second display control step of displaying the second image data under predetermined conditions.

[0038] Image extraction is performed by storing the image data processed by the image processing unit 221 in memory such as RAM, and then reading the image to be extracted. The position for extracting the first image data is a rectangular area within the normal field of view 30 corresponding to the high-resolution area 201a of the image formed by the optical system 201. Furthermore, if the second moving device is detected and the display field of view determination unit 224 is notified to display in the wide-angle field of view 31, a rectangular wide-angle image (second image data) including the second moving device within the wide-angle field of view 31 is supplied to the display unit 230. Therefore, the second image data including the second moving device in the high-distortion area is displayed.

[0039] The image supplied to the display unit 230 by the display angle changing unit 222 is enlarged or reduced to the optimal size for the display unit 230 before being supplied to the display unit 230. When switching the display area from the first image data to the second image data, the transition is performed so that it gradually expands over a transition time t1. Similarly, when switching back from the second image data to the first image data, the transition is performed so that it gradually narrows over a transition time t2. For example, the display control unit provides a predetermined transition time when switching the display of the first image data and the second image data. Here, the transition times t1 and t2 are different, for example, 0 <t1<t2とする。

[0040] Furthermore, the transition time t1 may be changed according to the urgency. For example, when a second mobile device is detected approaching the mobile device 10 rapidly from behind, the transition time t1 is set to a shorter time. Note that if the conditions for switching to the second image data are repeatedly met / not met in a short period of time, it will become difficult to see the image on the display unit 230, so a predetermined waiting time t3 is provided when switching from the second image data to the first image data. The setting range for the waiting time t3 is set to a time longer than 0 seconds. It is desirable that the conditions (thresholds) for switching the display from the second image data to the first image data and the conditions (thresholds) for switching the display from the first image data to the second image data be different.

[0041] The display unit 230 is a display unit such as a liquid crystal display or an organic EL display, and functions as a display means for displaying image data supplied from the display angle changing unit 222, which is a display control means. The display unit 230 is also positioned, for example, above the windshield of the mobile device 10 and used as an electronic rearview mirror.

[0042] Figure 3 is a flowchart showing an example of a field-of-view switching processing program executed by the processing unit 220. The flow in Figure 3 starts when the power source of the mobile device 10 is started and continues to run while the power source is operating. Furthermore, each step shown in Figure 3 is controlled by the computer included in the processing unit 220 executing a computer program stored in a storage medium.

[0043] In step S301, the reverse gear detection unit 225 determines whether the vehicle is in reverse gear and notifies the display field of view determination unit 224 of the determination result. If it is determined that the vehicle is in reverse gear, the process proceeds to step S302. If it is determined that the vehicle is not in reverse gear, the process proceeds to step S303.

[0044] In step S302, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to a wide-angle image, and supplies a rectangular wide-angle image (third image data) including the wide-angle field of view 31 processed by the image processing unit 221 to the display unit 230. For example, in step S302, when the first moving device is retracted, the display field of view changing unit 222 displays the third image data including the high-distortion region behind the first moving device. Note that the wide-angle image of the third image data may have a different range from the wide-angle image of the second image data. The display unit 230 displays the wide-angle image (third image data) supplied from the display field of view changing unit 222 and proceeds to step S304.

[0045] In step S304, the reverse gear detection unit 225 determines whether the vehicle is continuously in reverse gear and notifies the display field of view determination unit 224 of the determination result. As long as the vehicle remains in reverse gear, the process in step S304 is repeated. If reverse gear is no longer detected, the process proceeds to step S305.

[0046] In step S305, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to the normal field of view image, and the image processing unit 221 supplies the display unit 230 with a rectangular narrow-angle image (first image data) cropped from the normal field of view range from the processed image. The display unit 230 displays the narrow-angle image (first image data) supplied by the display field of view changing unit 222 and ends the series of processes.

[0047] In step S303, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to the normal field of view image, and the image processing unit 221 cuts out the normal field of view range from the processed image and supplies the narrow-angle image (first image data) to the display unit 230. Once the display unit 230 switches to the narrow-angle image (first image data) supplied by the display field of view changing unit 222, the process proceeds to step S306.

[0048] In step S306, the rear vehicle distance detection unit 223 measures the distance between the vehicle and the rear vehicle based on image data other than the first image data from the image data in the region including the high distortion region, and determines whether the distance is less than a threshold (e.g., 3m). If there is a rear vehicle whose distance from the vehicle is less than the threshold, the process proceeds to step S307. In this case, step S306 functions as a detection step that detects a second moving device that satisfies predetermined conditions from the image data in the region including the high distortion region.

[0049] In Embodiment 1, the second moving device that satisfies the above predetermined conditions refers to a vehicle (diagonally) behind the vehicle that is located less than a predetermined distance (for example, less than 3m) from the vehicle, in the image data other than the first image data in the image data of the region including the high distortion region. If there is no vehicle behind the vehicle that is located less than or equal to a threshold distance from the vehicle in the image data other than the first image data in the image data of the region including the high distortion region, it is determined that there is no vehicle diagonally behind the vehicle. Then, the series of processes is terminated without changing the display of the narrow-angle image on the display unit 230. The above distance threshold can be changed by the user using the user setting change unit 226.

[0050] In step S307, the display angle changing unit 222 receives a notification from the display angle determination unit 224 to switch to a wide-angle image and supplies the wide-angle image (second image data) including the second moving device in the high-distortion region to the display unit 230. Once the display unit 230 switches to the wide-angle image (second image data) supplied from the display angle changing unit 22, the process proceeds to step S308.

[0051] In step S308, the distance between the vehicle and the vehicle behind is measured using image data other than the first image data, from the image data in the region containing the high distortion region. The display field of view determination unit 224 then determines whether the distance is above a threshold (for example, 3m). For example, it determines whether the distance to the vehicle diagonally behind is above a predetermined threshold. If there is a vehicle diagonally behind that is less than the threshold distance from the vehicle, the process in step S308 is repeated.

[0052] If the distance between your vehicle and the vehicle diagonally behind you exceeds a threshold, the system proceeds to step S309. The distance threshold in step S308 can also be changed by the user using the user setting change unit 226. Furthermore, the threshold set in step S306 and the threshold set in step S308 can be set to different values.

[0053] In step S309, after the display angle change unit 222 receives a notification from the display angle determination unit 224 to switch to the normal angle image, it remains in the display state of the wide-angle image (second image data) for a predetermined waiting time t3. After the waiting time t3 has elapsed, it supplies the narrow-angle image (first image data) to the display unit 230, and after the display unit 230 switches to the narrow-angle image (first image data) supplied from the display angle change unit 222, the series of processes ends.

[0054] As mentioned above, the rear vehicle distance detection unit 223 may be capable of detecting the type of the second moving device. Depending on the type, the conditions for displaying the second image data (for example, the threshold distance to the rear vehicle when detecting it as a second moving device) may be made different.

[0055] [Embodiment 2] Figure 4 is a block diagram illustrating an example configuration of the image processing device 400 in Embodiment 2. The difference between the image processing device 400 and the image processing device 200 in Embodiment 1 is that the processing unit 220 has a rear vehicle turn signal detection unit 424 instead of a rear vehicle distance detection unit 223. Also, the image processed by the image processing unit 221 is supplied to the display field of view changing unit 222 and the rear vehicle turn signal detection unit 424.

[0056] The display field of view determination unit 224 determines whether to display the image on the display unit 230 in the wide-angle field of view 31 or the normal field of view 30, based on the detection information from the rear vehicle turn signal detection unit 424 and the detection result from the reverse gear detection unit 225. It then notifies the display field of view change unit 222 of the determination result. If the rear vehicle turn signal detection unit 424 notifies that a vehicle with a turn signal on from the rear has been detected, the display field of view is set to the wide-angle field of view 31 (second image data); otherwise, it is set to the normal field of view 30 (first image data).

[0057] Furthermore, if the reverse gear detection unit 225 notifies that the vehicle is in reverse gear, the display field of view is set to the wide-angle field of view range 31 (third image data), and if the vehicle is not in reverse gear, the display field of view is set to the normal field of view range 30 (first image data). In addition, when reverse gear is detected, the display field of view is set to the wide-angle field of view range 31 (third image data) regardless of the result of the rear vehicle turn signal detection unit 424. If the vehicle is not in reverse gear, either the first or second image data is displayed according to the result of the rear vehicle turn signal detection unit 424.

[0058] The rear vehicle turn signal detection unit 424 detects the operation of the turn signal of the rear vehicle in the image processed by the image processing unit 221 and notifies the display field of view determination unit 224 of the detected information. Here, the rear vehicle turn signal detection unit 424 detects a mobile device that is flashing its turn signal under predetermined conditions (for example, three or more consecutive times) as a second mobile device.

[0059] Figure 5 is a flowchart illustrating an example of control in Embodiment 2 in which the display field of view is changed based on the turn signal operation of a vehicle behind. The flow in Figure 5 starts when the power source of the mobile device 10 is started and is executed continuously while the power source is running. Furthermore, each step shown in Figure 5 is controlled by the computer included in the processing unit 220 executing a computer program stored in the storage medium.

[0060] In step S501, the reverse gear detection unit 225 determines whether the vehicle is in reverse gear and notifies the display field of view determination unit 224. If it is determined that the vehicle is in reverse gear, the process proceeds to step S502. If it is determined that the vehicle is not in reverse gear, the process proceeds to step S503.

[0061] In step S502, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to a wide-angle image and supplies a rectangular wide-angle image (third image data) to the display unit 230. Once the display unit 230 switches to displaying the wide-angle image (third image data) supplied from the display field of view changing unit 22, the process proceeds to step S504.

[0062] In step S504, the reverse gear detection unit 225 determines whether the vehicle is continuously in reverse gear and notifies the display field of view determination unit 224. As long as the vehicle remains in reverse gear, the process in step S504 is repeated. If reverse gear is no longer detected, the process proceeds to step S505.

[0063] In step S505, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to the normal field of view image, and the image processing unit 221 supplies the display unit 230 with a rectangular narrow-angle image (first image data) cut out from the normal field of view range from the processed image. The display unit 230 displays the narrow-angle image (first image data) supplied by the display field of view changing unit 222 and ends the series of processes.

[0064] In step S503, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to the normal field of view image and supplies the narrow-angle image (first image data) to the display unit 230. Once the display unit 230 has displayed the narrow-angle image (first image data) supplied by the display field of view changing unit 222, the process proceeds to step S506.

[0065] In step S506, the rear vehicle turn signal detection unit 424 detects vehicles with turn signals in the image data other than the first image data, among the image data in the region including the high distortion region, and notifies the display field of view determination unit 224 of the detected information. If a vehicle with a turn signal is detected behind the vehicle, the process proceeds to step S507. If no vehicle with a turn signal is detected behind the vehicle, the display unit 230 does not change the display of the narrow-angle image (first image data), and the series of processes ends as is.

[0066] In step S507, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to a wide-angle image and supplies a rectangular wide-angle image (second image data) including the second moving device in the high-distortion region to the display unit 230. Once the display unit 230 switches to the wide-angle image (second image data) supplied from the display field of view changing unit 22, the process proceeds to step S508.

[0067] In step S508, the rear vehicle turn signal detection unit 424 detects vehicles with their turn signals on in the image data other than the first image data, within the image data that includes the high-distortion region. If a vehicle with a turn signal on is detected behind the vehicle, the process in step S508 is repeated. If no vehicles with turn signals on are detected behind the vehicle, the process proceeds to step S509.

[0068] In step S509, after the display angle change unit 222 receives a notification from the display angle determination unit 224 to switch to the normal angle image, it continues to display the second image data for a waiting time t3. After the waiting time t3 has elapsed, it supplies the narrow-angle image (first image data) to the display unit 230, and after the display unit 230 switches to the narrow-angle image (first image data) supplied from the display angle change unit 222, it terminates the series of processes.

[0069] In Embodiment 2, in steps S506 and S508, the rear vehicle turn signal detection unit 424 detects vehicles that are emitting turn signals in image data other than the first image data, among the image data in the region including the high-distortion region. However, in steps S506 and S508, the rear vehicle turn signal detection unit 424 may also detect vehicles that are emitting turn signals in the wide-angle field of view 31.

[0070] [Embodiment 3] Figure 6 is a block diagram illustrating an example configuration of the image processing device 600 in Embodiment 3. The difference between the image processing device 600 and the image processing device 200 in Embodiment 1 is that the processing unit 220 has a meandering driving detection unit 624 instead of a rear vehicle distance detection unit 223, and the image processed by the image processing unit 221 is supplied to the display field of view changing unit 222 and the meandering driving detection unit 624.

[0071] The display field of view determination unit 224 determines whether to display the image on the display unit 230 in the wide-angle field of view 31 or the normal field of view 30 based on the detection information from the meandering driving detection unit 624 and the detection result from the reverse gear detection unit 225, and notifies the display field of view change unit 222 of the determination result. If the number of times the rear vehicle detected by the meandering driving detection unit 624 moves back and forth alternately to the left and right with respect to the white line is greater than or equal to a threshold, the second image data is displayed; otherwise, the first image data is displayed. In addition, if the reverse gear detection unit 225 notifies that the vehicle is in reverse gear, the third image data is displayed; otherwise, the first image data is displayed.

[0072] Furthermore, when reverse gear is detected, the third image data is displayed regardless of the result of the meandering detection unit 624. If reverse gear is not engaged, the first or second image data is displayed according to the result of the meandering detection unit 624. In addition, by receiving vehicle type information from the meandering detection unit 624, which will be described later, the criteria for determining the angle of view change may be changed according to the vehicle type.

[0073] For example, in the case of large vehicles such as trucks, the risk of collision is greater, so the threshold for the number of back-and-forth movements is reduced so that it can be detected earlier than when a regular car is weaving. For example, the weaving detection unit 624 may be capable of detecting the type of second moving device, and the conditions for displaying the second image data (for example, the threshold for the number of times the vehicle crosses the boundary of a traffic lane when detected as a weaving vehicle) may be made different depending on the type.

[0074] The meandering detection unit 624 stores image recognition patterns for detecting various objects in advance and uses a predetermined image recognition algorithm to detect patterns in the image based on these image recognition patterns. The image patterns may be defined by machine learning such as deep learning, or they may be image recognition patterns specialized for objects that can be captured by the imaging device, such as four-wheeled vehicles, two-wheeled vehicles, pedestrians, road fixtures, lanes, white lines, and other objects that may exist on the road and in the surrounding area.

[0075] The meandering driving detection unit 624 detects a vehicle in the image processed by the image processing unit 221 and detects the number of times the detected position moves alternately left and right across the white line. When it detects movement alternately left and right across the white line, it notifies the display field of view determination unit 224 of the number of round trips.

[0076] The meandering detection unit 624 stores data on image features such as shape and color for each vehicle type, extracted as a result of machine learning such as deep learning from images of a large number of vehicles collected in advance. It then determines the vehicle type and notifies the display field of view determination unit 224 of the vehicle type information. Here, the meandering detection unit 624 functions as a detection unit that detects a moving device that repeatedly crosses the boundary of a traffic lane as a second moving device.

[0077] Figure 7 is a flowchart illustrating an example of control in Embodiment 4 that detects the serpentine driving of a vehicle behind and changes the display field of view. The flow in Figure 7 starts when the power source of the mobile device 10 is started and is executed continuously while the power source is running. Furthermore, each step shown in Figure 7 is controlled by the computer included in the processing unit 220 executing a computer program stored in the storage medium.

[0078] In step S701, the reverse gear detection unit 225 determines whether the vehicle is in reverse gear and notifies the display field of view determination unit 224. If it is determined that the vehicle is in reverse gear, the process proceeds to step S702. If it is determined that the vehicle is not in reverse gear, the process proceeds to step S703.

[0079] In step S702, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to a wide-angle image and supplies the wide-angle image (third image data) to the display unit 230. Once the display unit 230 switches to displaying the wide-angle image (third image data) supplied from the display field of view changing unit 22, the process proceeds to step S704.

[0080] In step S704, the reverse gear detection unit 225 determines whether the vehicle is continuously in reverse gear and notifies the display field of view determination unit 224. As long as the vehicle remains in reverse gear, the process in step S704 is repeated. If reverse gear is no longer detected, the process proceeds to step S705.

[0081] In step S705, the display angle change unit 222 receives a notification from the display angle determination unit 224 to switch to the normal angle image, and the image processing unit 221 supplies the display unit 230 with a rectangular narrow-angle image (first image data) cropped from the normal angle range of the processed image. The display unit 230 displays the narrow-angle image (first image data) supplied by the display angle change unit 222 and completes the series of processes.

[0082] In step S703, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to the normal field of view image and supplies the narrow-angle image (first image data) to the display unit 230. Once the display unit 230 switches to displaying the narrow-angle image (first image data) supplied by the display field of view changing unit 222, the process proceeds to step S706.

[0083] In step S706, the meandering driving detection unit 624 detects the number of times the detected vehicle repeatedly moved back and forth across a lane boundary, such as a white line (or yellow line, etc.), in the image data other than the first image data, among the image data in the region containing the high-distortion area. The detected information is then notified to the display field of view determination unit 224. If the number of times the detected vehicle repeatedly moved back and forth across the white line exceeds a threshold (for example, 2 times), it is determined to be a meandering vehicle, and the process proceeds to step S707. If the number of times the detected vehicle repeatedly moved back and forth across the white line does not exceed the threshold, the display of the narrow-angle image (first image data) on the display unit 230 is not changed, and the series of processes ends as is.

[0084] In step S707, the display angle changing unit 222 receives a notification from the display angle determination unit 224 to switch to a wide-angle image and supplies the wide-angle image (second image data) to the display unit 230. For example, the display unit 230 is supplied with second image data, which is a wide-angle image including a second moving device in a high-distortion region. Once the display unit 230 has switched to the wide-angle image (second image data) supplied from the display angle changing unit 22, the process proceeds to step S708.

[0085] In step S708, the meandering detection unit 624 detects the number of times the detected vehicle repeatedly moved back and forth, for example, to the left and right, with respect to a white line, in the image data other than the first image data, among the image data in the region containing the high-distortion area. The detected information is then notified to the display field of view determination unit 224. If the number of times the detected vehicle repeatedly moved back and forth, for example, to the left and right, with respect to a white line, exceeds a threshold (for example, 2 times), it is determined to be a meandering vehicle, and the process in step S708 is repeated. If the number of times the detected vehicle repeatedly moved back and forth, for example, to the left and right, with respect to a white line, does not exceed the threshold, the process proceeds to step S709.

[0086] In step S709, after the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to the normal field of view image, it continues to display the second image data for a waiting time t3. After the waiting time t3 has elapsed, the narrow-angle image (first image data) is supplied to the display unit 230, and once the display unit 230 switches to the narrow-angle image (first image data) supplied from the display field of view changing unit 222, the series of processes is terminated.

[0087] In step S706 and step S708 of Embodiment 3, among the image data in the region including the high-distortion region, the number of times the detected vehicle repeatedly moved back and forth alternately around the lane boundary is detected in the image data other than the first image data, thereby detecting a vehicle driving erratically. However, in step S506 and step S508, the number of times the detected vehicle repeatedly moved back and forth alternately left and right around the lane boundary is detected in the wide-angle field of view 31, and a vehicle driving erratically may be detected by doing so.

[0088] [Embodiment 4] Figure 8 is a block diagram illustrating an example configuration of the image processing device 800 in Embodiment 4. The difference between the image processing device 800 and the image processing device 200 in Embodiment 1 is that the processing unit 220 has a rear vehicle passing detection unit 824 instead of a rear vehicle distance detection unit 223. Also, the image processed by the image processing unit 221 is supplied to the display field of view changing unit 222 and the rear vehicle passing detection unit 824.

[0089] The display field of view determination unit 224 determines whether to display the image on the display unit 230 in the wide-angle field of view 31 or the normal field of view 30 based on the detection information from the rear vehicle flashing detection unit 824 and the detection result from the reverse gear detection unit 225. It then notifies the display field of view change unit 222 of the determination result. If the rear vehicle flashing detection unit 824 detects a vehicle flashing its headlights from behind, the second image data is displayed; otherwise, the first image data is displayed.

[0090] Furthermore, if the reverse gear detection unit 225 notifies that the vehicle is in reverse gear, the third image data is displayed; otherwise, the first image data is displayed. In addition, when reverse gear is detected, the third image data is displayed regardless of the result of the rear vehicle flashing detection unit 824. If the vehicle is not in reverse gear, either the first or second image data is displayed depending on the result of the rear vehicle flashing detection unit 824.

[0091] The rear vehicle flashing detection unit 824 detects the flashing action of a rear vehicle in the image processed by the image processing unit 221 and notifies the display field of view determination unit 224 of the detected information. Here, the rear vehicle flashing detection unit 824 functions as a detection unit that detects a moving device performing a flashing action as a second moving device.

[0092] Figure 9 is a flowchart illustrating an example in Embodiment 4 of controlling the display angle of view in response to the detection of a passing maneuver with a vehicle behind. The flow in Figure 9 begins when the power source of the mobile device 10 starts up and continues to run while the power source is operating. Furthermore, each step shown in Figure 9 is controlled by the computer included in the processing unit 220 executing a computer program stored in the storage medium.

[0093] In step S901, the reverse gear detection unit 225 determines whether the vehicle is in reverse gear and notifies the display field of view determination unit 224. If it is determined that the vehicle is in reverse gear, the process proceeds to step S902. If it is determined that the vehicle is not in reverse gear, the process proceeds to step S903.

[0094] In step S902, the display angle changing unit 222 receives a notification from the display angle determination unit 224 to switch to a wide-angle image and supplies the wide-angle image (third image data) to the display unit 230. Once the display unit 230 switches to displaying the wide-angle image (third image data) supplied by the display angle changing unit 222, the process proceeds to step S904.

[0095] In step S904, the reverse gear detection unit 225 determines whether the vehicle is continuously in reverse gear and notifies the display field of view determination unit 224. As long as the vehicle remains in reverse gear, the process in step S904 is repeated. If reverse gear is no longer detected, the process proceeds to step S905.

[0096] In step S905, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to the normal field of view image, and the image processing unit 221 cuts out the normal field of view range from the processed image and supplies the rectangular narrow-angle image (first image data) to the display unit 230. The display unit 230 displays the narrow-angle image (first image data) supplied from the display field of view changing unit 222 and ends the series of processes.

[0097] In step S903, the display field of view changing unit 222 receives a notification from the display field of view determination unit 224 to switch to the normal field of view image and supplies the narrow-angle image (first image data) to the display unit 230. Once the display unit 230 switches to displaying the narrow-angle image (first image data) supplied by the display field of view changing unit 222, the process proceeds to step S906.

[0098] In step S906, the rear vehicle flashing detection unit 824 detects a vehicle flashing its headlights in the image data other than the first image data, among the image data in the region including the high distortion region, and notifies the display field of view determination unit 224 of the detected information. If a vehicle flashing its headlights is detected behind the vehicle, the process proceeds to step S907. If no vehicle flashing its headlights is detected behind the vehicle, the display unit 230 does not change the display of the narrow-angle image (first image data), and the series of processes ends as is.

[0099] In step S907, the display angle changing unit 222 receives a notification from the display angle determination unit 224 to switch to a wide-angle image and supplies the wide-angle image (second image data) to the display unit 230. For example, the second image data, which is a wide-angle image including a second moving device in a high-distortion region, is supplied to the display unit 230. Once the display unit 230 has switched to the wide-angle image (second image data) supplied from the display angle changing unit 22, the process proceeds to step S908.

[0100] In step S908, the rear vehicle flashing detection unit 824 detects a vehicle flashing its headlights in the image data other than the first image data, among the image data in the region including the high distortion region, and notifies the display field of view determination unit 224 of the detected information. If a vehicle flashing its headlights is detected behind the vehicle, the process in step S908 is repeated. If no vehicle flashing its headlights is detected behind the vehicle, the process proceeds to step S909.

[0101] In step S909, after the display angle change unit 222 receives a notification from the display angle determination unit 224 to switch to the normal angle image, it continues to display the second image data for a waiting time t3. After the waiting time t3 has elapsed, it supplies the narrow-angle image (first image data) to the display unit 230. Once the display unit 230 switches to the narrow-angle image (first image data) supplied by the display angle change unit 222, the series of processes ends.

[0102] In step S906 and step S908 of Embodiment 4, a vehicle flashing its headlights is detected in the image data other than the first image data among the image data in the region including the high-distortion region. However, in step S506 and step S508, a vehicle flashing its headlights may be detected in the wide-angle field of view 31, thereby detecting the vehicle flashing its headlights.

[0103] [Embodiment 5] At least one of the various functions, processes, and methods described in the embodiments described above can be implemented using a program. Hereinafter, in Embodiment 5, the program for implementing at least one of the various functions, processes, and methods described in the embodiments described above will be referred to as "Program X". Furthermore, in Embodiment 5, the computer for executing Program X will be referred to as "Computer Y". Personal computers, microcomputers, CPUs (Central Processing Units), etc., are examples of Computer Y. The computer such as the image processing device in the embodiments described above is also an example of Computer Y.

[0104] At least one of the various functions, processes, and methods described in the embodiments described above can be realized by computer Y executing program X. In this case, program X is supplied to computer Y via a computer-readable storage medium. The computer-readable storage medium in Embodiment 5 includes at least one of a hard disk drive, magnetic storage device, optical storage device, magneto-optical storage device, memory card, ROM, RAM, etc. Furthermore, the computer-readable storage medium in Embodiment 5 is a non-transitory storage medium.

[0105] [Embodiment 6] The mobility device 10 in the above-described embodiment is not limited to an automobile, but may be any mobile device such as a motorcycle, bicycle, wheelchair, ship, airplane, robot, or drone. [Explanation of Symbols]

[0106] 10: Mobile device 20: Imaging device 30: Normal field of view 31: Wide-angle field of view 40: Driver 200: Image processing device 201:Optical system 201a: High resolution area 201b: Low resolution area 210: Imaging Unit 220: Processing Unit 221: Image Processing Unit 222: Display angle change section 223: Rear vehicle distance detection unit 224: Display angle determination unit 225: Reverse gear detection unit 226: User Settings Change Section 230: Display section 400: Image processing device 424: Rear vehicle turn signal detection unit 600: Image processing device 624: Swerving detection unit 800: Image processing device 824: Rear vehicle flashing detection unit

Claims

1. A single imaging means that captures images of the rear of the first mobile device and generates image data, A display control means for displaying first image data, which is obtained by extracting the central region of the optical axis from the image data generated by the single imaging means, on a display unit, A detection means for detecting a second mobile device that satisfies predetermined conditions from the aforementioned image data, It has, When the second moving device is detected by the detection means, the display control means displays a second image data with a wider area than the first image data from the image data on the display unit. An image processing apparatus characterized by the following:

2. The image processing apparatus according to claim 1, characterized in that the detection means detects a mobile device that is flashing its turn signal under predetermined conditions as the second mobile device.

3. The image processing apparatus according to any one of claims 1 to 2, characterized in that the detection means detects a moving device that repeatedly crosses the boundary of a traffic lane as the second moving device.

4. The image processing apparatus according to any one of claims 1 to 3, characterized in that the detection means detects a moving device performing a passing operation as the second moving device.

5. The image processing apparatus according to any one of claims 1 to 4, characterized in that the display control means provides a predetermined transition time when switching the display of the first image data and the second image data.

6. The image processing apparatus according to claim 5, characterized in that the transition time when switching the display from the first image data to the second image data is different from the transition time when switching the display from the second image data to the first image data.

7. The image processing apparatus according to any one of claims 1 to 6, wherein the detection means is capable of detecting the type of the second moving device, and the display control means causes the conditions for displaying the second image data to differ according to the type.

8. The image processing apparatus according to claim 1, further comprising an image processing means for correcting distortion of image data captured by the imaging means to generate image data.

9. A mobile device A single imaging means for generating image data, A display control means for displaying a first image data obtained by extracting the central region of the optical axis from the image data generated by the single imaging means, A detection means for detecting a second mobile device that satisfies predetermined conditions from the image data, It has, The imaging means captures images of the rear of the moving device, The display control means, when the detection means detects the second moving device, displays a second image data from the image data that covers a wider area than the first image data. A mobile device characterized by the following features.

10. An imaging step of photographing the rear of the first mobile device using a single imaging means that generates image data, A first display control step of displaying first image data obtained by cutting out the central region of the optical axis from the image data generated by the single imaging means, A detection step of detecting a second mobile device that satisfies predetermined conditions from the image data, The system includes a second display control step, in which, when the second moving device is detected by the detection step, a second image data set with a wider area than the first image data set is displayed from the image data set. An image processing method characterized by the following:

11. In the computer of the image processing device, An imaging step of generating image data by imaging the area behind the first moving device using a single imaging means, A first display control step of displaying first image data obtained by cutting out the central region of the optical axis from the image data generated by the single imaging means, A detection step of detecting a second mobile device that satisfies predetermined conditions from the image data, If the second moving device is detected by the detection step, a second display control step is performed to display a second image data from the image data that covers a wider area than the first image data. A program to execute.

Images