Perimeter monitoring system, image processing method, and program

Abstract

To improve display of a cut-out region.SOLUTION: A perimeter monitoring system has a camera that picks up an image of a perimeter region of a movable body, an information processing apparatus that processes a picked-up image from the camera, and a monitor that displays the image processed by the information processing apparatus. The camera has an optical unit that is capable of imaging of a distorted region in which the picked-up image has distortion, and a low-distorted region in which the picked-up image has a smaller distortion than the distorted region. The information processing apparatus has: cut-out angle-of-view setting means that sets the angle of view of the picked-up image to be displayed on the monitor; low delay mode determination means that determines whether to perform video display on the monitor with a low delay; and cut-out angle-of-view restriction means that, when the low delay mode determination means determines to perform the video display with a low delay, performs restriction so that the distorted region is not included in a cut-out angle of view set by the cut-out angle-of-view setting means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a peripheral monitoring system, an image processing method, and a program.

Background Art

[0002] In recent years, systems for performing peripheral monitoring with a wide-angle camera installed on a moving body have been spreading rapidly. For example, there is a system in which a wide-angle camera is installed on the upper part of a small mobility with its optical axis facing upward, and the captured image is cut out at an arbitrary angle of view and displayed as a video on a monitor. Further, Patent Document 1 discloses a system in which a wide-angle camera is installed at the rear or side of an automobile, and the captured image is cut out at a plurality of angles of view and displayed as a video on each monitor. In these conventional systems, it is possible to display a video on a plurality of monitors with a single wide-angle camera, which contributes to cost reduction, weight reduction, and improvement of the degree of freedom in installation.

[0003] In addition, a method has been proposed in which an optical system capable of adjusting the projection characteristics (aspect ratio) of the peripheral portion of a lens is used for the above-described wide-angle camera, and the captured image of the peripheral portion of the lens is cut out with low distortion. With such an optical system, it is possible to display the video as it is without performing distortion correction processing, and it is possible to perform peripheral monitoring with low latency.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in a wide-angle camera capable of imaging the peripheral part of a lens with low distortion, depending on the situation of performing peripheral monitoring, there may be a case where it is desired to cut out a wide angle of view including the central part of the lens and display the video. In that case, since the cut-out image may be distorted, it is necessary to perform distortion correction processing. As a result, a time delay corresponding to the distortion correction processing and an increase in arithmetic resources are cited as problems. For this reason, conventionally, there has been room for improvement in the display of the cut-out area.

[0006] The present invention was devised in view of the above problems, and an object thereof is to improve the display of the cut-out area.

Means for Solving the Problems

[0007] A peripheral monitoring system according to an embodiment of the present invention includes a camera that images a peripheral area of a moving body, an information processing device that processes an imaging image of the camera, and a monitor that displays an image processed by the information processing device. The camera has an optical unit capable of imaging a distorted area where the imaging image has distortion and a low-distortion area where the imaging image has less distortion than the distorted area. The information processing device includes a cut-out angle-of-view setting means for setting an angle of view to be displayed on the monitor among the imaging images, a low-latency mode determination means for determining whether to perform video display on the monitor with low latency, and a cut-out angle-of-view restriction means for restricting the cut-out angle of view set by the cut-out angle-of-view setting means so that the distorted area is not included when the low-latency mode determination means determines to perform video display with low latency.

Effects of the Invention

[0008] According to the present invention, the display of the cut-out area can be improved.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

MODE FOR CARRYING OUT THE INVENTION

[0010] Hereinafter, the peripheral monitoring system according to the present invention will be described in detail with reference to the drawings. In each figure, the same member or element is given the same reference numeral, and redundant descriptions thereof are omitted or simplified. In addition, the peripheral monitoring system of the present invention can be embodied in various forms and is not limited to only the embodiments described in the specification.

[0011] <First Embodiment> FIG. 1 is a diagram showing an example of an installation mode of a moving body, a wide-angle camera disposed on the upper part of the moving body, and a display monitor according to the first embodiment of the present invention. FIG. 1 (A) is a side view of the moving body 10, and FIG. 1 (B) is a front view of the moving body 10.

[0012] The mobile body 10 is, for example, a vehicle such as a commercial vehicle shown in FIG. 1, or a small mobility performing autonomous driving. The wide-angle camera 110 is a camera capable of imaging a wide range, for example, with an elevation angle of about 100 degrees and an azimuth angle of about 360 degrees, and is installed on the upper part of the mobile body 10 with its optical axis facing upward so as to be able to image the entire surroundings. The imaging range of the wide-angle camera 110 is composed of a low-distortion region 501 with less distortion in the captured image and a distortion region 502 with distortion peculiar to the wide-angle camera 110 in the captured image. The low-distortion region 501 is a region with less distortion in the captured image than the distortion region 502. The wide-angle camera 110 has projection characteristics such that the low-distortion region 501 occupies a region with a large angle formed with the optical axis. Details of such projection characteristics will be described later with reference to FIG. 3.

[0013] The display monitor 120 is a device that cuts out a partial viewing angle of the entire-surroundings image captured by the wide-angle camera 110 and presents it to the user 50 as video information. In FIG. 1, the display monitor 120 is arranged inside the mobile body 10, but the display monitor 120 may be arranged outside the mobile body 10 for remote peripheral monitoring.

[0014] FIG. 2 is a diagram showing a configuration example of a peripheral monitoring system according to the first embodiment of the present invention. The peripheral monitoring system includes a wide-angle camera 110, a processing unit 200, and a display monitor 120.

[0015] The wide-angle camera 110 has an optical unit 111 and an imaging unit 112. The optical unit 111 has at least one lens and serves to form an image of the light incident from the outside on the light-receiving surface of the imaging unit 112. The imaging unit 112 is an image sensor having a pixel structure arranged two-dimensionally, converts the optical subject image formed by the optical unit 111 into an electrical signal for each pixel, and transmits it to the development / image processing unit 211. The imaging unit 112 is, for example, a CMOS image sensor or a CCD image sensor. CMOS is an abbreviation for Complementary Metal-Oxide-Semiconductor. CCD is an abbreviation for Charge Coupled Device. An RGB color filter is arranged for each pixel on the light-receiving surface of the imaging unit 112 according to a method such as a Bayer array. In order to image the entire 360 degrees around the moving body 10, it is desirable to align the position where the center of the light-receiving surface of the imaging unit 112 intersects the optical axis. However, when the monitoring area is limited to, for example, 180 degrees in front of the moving body 10, sensor shift may be performed so that the front can be imaged with emphasis.

[0016] The processing unit 200 has an image processing unit 210, a video control unit 220, a cropped field-of-view processing unit 230, and a low-latency mode determination unit 240. The processing unit 200 includes a CPU that performs operations and controls, and a ROM and a RAM that are main storage devices. CPU is an abbreviation for Central Processing Unit. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory. Basic setting data and a program for realizing a series of processes according to the present embodiment are stored in the ROM. The CPU calls a program corresponding to the processing content from the ROM and expands it into the RAM, and executes the operations of each block. The processing unit 200 is an example of an information processing device.

[0017] The image processing unit 210 includes a development / image processing unit 211 and a distortion correction processing unit 212. The development / image processing unit 211 respectively performs white balance adjustment and demosaicing on the image data input from the wide-angle camera 110 according to a Bayer array or the like, and converts it into RGB raster format image data. The development / image processing unit 211 further performs various image processes such as demosaicing, WDR correction, gamma correction, and LUT. WDR is an abbreviation for Wide Dynamic Range. LUT is an abbreviation for Look Up Table.

[0018] When the distortion correction processing unit 212 determines that the distortion of the image for which the cutout display is to be performed is large by a distortion correction determination unit 232 described later, the distortion correction processing unit 212 performs coordinate conversion and pixel interpolation processing on the image data to correct the imaging distortion peculiar to the wide-angle lens. The above-described development / image processing unit 211 and distortion correction processing unit 212 may be realized hardware-wise by an FPGA or the like, or may be realized software-wise by executing a program on a CPU or the like. FPGA is an abbreviation for Field Programmable Gate Array. When the processing is realized hardware-wise, the processing time can be increased, and the delay of video display can be reduced.

[0019] The video control unit 220 includes an image cutout unit 221 and a video display unit 222. The image cutout unit 221 performs cutout processing of the image data based on the cutout angle information set by a cutout angle setting unit 231 described later. The video display unit 222 performs video display processing on the image data subjected to the cutout processing by the image cutout unit 221 to the display monitor 120. The video display processing refers to output conversion into a digital signal conforming to a communication standard such as HDMI (registered trademark) or DisplayPort, which is an interface for monitor output. HDMI is an abbreviation for High-Definition Multimedia Iterface.

[0020] The cropped picture angle processing unit 230 includes a cropped picture angle setting unit 231, a distortion correction determination unit 232, and a cropped picture angle limitation unit 233. The cropped picture angle setting unit 231 sets the coordinate values (picture angle information) of the picture angle for cropping and displaying the image captured by the wide-angle camera 110. In the first embodiment, the method of setting the picture angle information is a method in which the user 50 arbitrarily sets the display picture angle through the UI of the display monitor 120. UI is the abbreviation of User Interface. As an example of UI operation, there are methods of obtaining the coordinate values of the four corners of the picture angle in conjunction with operations such as drag-wheel operations using a mouse or slide-pinch operations on a touch panel, or direct input of each coordinate value. As another example of the method of setting picture angle information, when the picture angle of the monitoring area is fixed, the coordinate information of the picture angle to be cropped is stored in the ROM in advance and read out from the ROM each time for setting.

[0021] The distortion correction determination unit 232 refers to the coordinate values of the picture angle set by the cropped picture angle setting unit 231 and determines whether the picture angle for cropped display is distorted. The determination result by the distortion correction determination unit 232 is output to the distortion correction processing unit 212, and if the image to be cropped is distorted, distortion correction processing is performed. On the other hand, if the cropped image is not distorted, video display is performed as it is without performing distortion correction processing. The means for determining whether the image to be cropped is distorted will be described later with reference to FIG. 4.

[0022] The cropped picture angle limitation unit 233 performs a limitation process on the setting of the picture angle coordinate values by the cropped picture angle setting unit 231 based on the determination result of the low-latency mode determination unit 240 described later. The limitation process of picture angle setting will be described later with reference to FIG. 4. By performing the processing by the above-described distortion correction determination unit 232 and cropped picture angle limitation unit 233, the displayable picture angle is limited, but it is possible to perform video display with low latency without performing distortion correction processing.

[0023] The low-latency mode determination unit 240 determines whether it is necessary to perform video display with low latency based on the running status of the moving body 10 or the like. When video display with low latency is required, the cut-out angle-of-view restriction unit 233 shifts to the low-latency mode and performs a restriction process on the setting of the cut-out angle-of-view. On the other hand, when video display with low latency is not required, the cut-out angle-of-view restriction unit 233 shifts to the normal mode and does not perform a restriction process on the setting of the cut-out angle-of-view. As a situation where video display with low latency is required, there is a case where video display at a high frame rate is required due to the high running speed of the moving body 10 or a large relative speed with respect to the target of interest 11 (see FIG. 4) displayed on the display monitor 120. Or, as a situation where video display with low latency is required, there is a case where it is for real-time peripheral monitoring and not for monitoring with recorded video. Also, the user 50 may arbitrarily set the switching between the low-latency mode and the normal mode.

[0024] Next, the optical system of the optical unit 111 included in the wide-angle camera 110 will be described in detail with reference to FIGS. 3(A) and 3(B), as well as Equations 1 and 2. FIGS. 3(A) and 3(B) are diagrams showing an example of the optical characteristics of the optical unit according to the first embodiment of the present invention. FIG. 3(A) is a diagram showing the image height y at each half angle-of-view on the light-receiving surface of the optical unit 111 in the first embodiment in a contour shape. FIG. 3(B) shows the projection characteristic y(θ) with the half angle-of-view (the angle formed by the optical axis and the incident light ray) θ on the horizontal axis and the imaging height (image height) y on the light-receiving surface (image surface) of the optical unit 111 on the vertical axis.

[0025] The projection characteristic y(θ) shown in FIG. 3(B) satisfies the following condition of Equation 1. Here, the focal length of the optical system is f, the half angle-of-view is θ, the image height on the image surface is y, the projection characteristic representing the relationship between the image height y and the half angle-of-view θ is y(θ), and θmax is the maximum half angle-of-view of the optical system. 0.1 < 2×f×tan(θmax / 2) / y(θmax) < 1.2 ··· Equation 1

[0026] More preferably, it satisfies the following Equation 2. 0.2 < 2×f×tan(θmax / 2) / y(θmax) < 0.92 ··· Equation 2

[0027] As shown in the projection characteristics of FIG. 3(B) and Equations 1 and 2, the optical system of the optical unit 111 in the first embodiment is configured such that its projection characteristic y(θ) is different in a region with a small half angle θ (near the optical axis) and a region with a large half angle θ (away from the optical axis). That is, when defining the increase amount of the image height y with respect to the half angle θ per unit (number of pixels per unit angle) as the resolution, it has the characteristic that the resolution increases as the half angle θ increases. This local resolution can be said to be higher as the contour interval of the image height y at the half angle θ in FIG. 3(A) is wider, and as the differential value (slope) dy(θ) / dθ of the projection characteristic y(θ) in FIG. 3(B) is larger. Here, dy(θ) / dθ is taken as the resolution.

[0028] At an angular field of view with a half angle θa or more where the resolution dy(θ) / dθ is higher than the average resolution, it is possible to adjust the projection characteristics (aspect ratio) in the radial direction and circumferential direction with respect to the optical axis, and it is possible to obtain an image with less distortion in the peripheral region of the lens. In the first embodiment, an angular field of view with a half angle θa or more is set as a low distortion region 501 where the projection characteristic (aspect ratio) can be adjusted. Also, an angular field of view smaller than the half angle θa is set as a distortion region 502 with distortion peculiar to a wide-angle lens. Therefore, according to the peripheral monitoring system in the first embodiment, it is possible to cut out the peripheral portion of the lens, which is the monitoring region, as it is without performing distortion correction processing depending on the display angular field of view, and an effect of enabling peripheral monitoring with low latency is achieved.

[0029] Next, the determination process of the presence or absence of distortion of the cut-out image by the distortion correction determination unit 232 and the restriction process of the angular field of view setting by the cut-out angular field restriction unit 233 will be described in detail with reference to FIG. 4. FIG. 4 is a diagram showing the entire surrounding image captured by the wide-angle camera 110 and the region cut out and displayed on the display monitor 120 surrounded by a broken line.

[0030] In the first embodiment, the wide-angle camera 110 has projection characteristics that occupy the low-distortion region 501 in the peripheral part of the lens with a large half field angle θ as shown in FIG. 4, and is characterized in that there is little distortion in the image cut out within the low-distortion region 501.

[0031] For example, when an image is cut out in the cutout region 503a of FIG. 4, the cutout field angle is within the low-distortion region 501, and the distortion of the cutout image is small. Therefore, video display can be performed without performing distortion correction processing. In the example of FIG. 4, the target of interest 11 is within the cutout region 503a. The target of interest 11 is an oncoming vehicle, a following vehicle, etc., and is a target that requires attention such as enhancing visibility with an image having little distortion.

[0032] On the other hand, when the display field angle is changed to the cutout region 503b or the cutout region 503c by the UI operation of the display monitor 120, the cutout field angle protrudes from the low-distortion region 501. Therefore, when video is displayed on the display monitor 120 as it is, the image may be greatly distorted and the visibility may decrease. As a method for determining the presence or absence of distortion in the cutout image, a method of referring to the coordinate values of each of the four corners of the cutout field angle and determining whether the number of pixels in the radial direction and the circumferential direction with respect to the optical axis, or the absolute coordinate values exceed a predetermined value can be mentioned. Such determination processing is executed each time by an interrupt process that is executed each time the field angle coordinate value is changed by a UI operation or the like by the user 50.

[0033] When performing cutout display with a field angle that protrudes from the low-distortion region 501, perform distortion correction processing, or perform cutout field angle restriction processing when low-latency video display is required. The cutout field angle restriction unit 233 restricts the field angle setting so that the cutout region 503b and the cutout region 503c are within the low-distortion region 501 in the case of the low-latency mode in order to eliminate the need for distortion correction processing. As the restriction processing of the field angle setting, when the coordinate values of the four corners of the cutout field angle are set outside the low-distortion region 501, a process of resetting the setting so as to be included within the low-distortion region 501 may be performed, or an error notification may be given to the user 50 to prompt the user to reset the cutout field angle.

[0034] The CPU inside the processing unit 200 executes the program stored in the ROM to perform peripheral monitoring processing by the image processing unit 210, the video control unit 220, the cut-out picture angle processing unit 230, and the low-latency mode determination unit 240. A series of procedures for the peripheral monitoring processing will be described in detail according to the flowchart of FIG. 5. FIG. 5 is a flowchart showing an example of camera image display processing executed by the CPU inside the processing unit 200 according to the first embodiment. For example, a series of peripheral monitoring processing from step S500 is executed by turning on the power of the wide-angle camera 110, the display monitor 120, and the processing unit 200. When the power source (for example, a battery) of the moving body 10 is in the starting state, the wide-angle camera 110, the display monitor 120, and the processing unit 200 may be powered on, or may be powered on by the operation of the user 50. The processing unit 200 may supply power to the wide-angle camera 110 and the display monitor 120.

[0035] In step S501, the wide-angle camera 110 starts the imaging operation. The developing / image processing unit 211 acquires the captured image by the wide-angle camera 110. In step S502, the developing / image processing unit 211 performs developing processing and various image processing on the captured image by the wide-angle camera 110.

[0036] In step S503, the low-latency mode determination unit 240 determines whether it is necessary to perform video display with low latency based on the running status of the moving body 10 and the like. The determination result is stored in the ROM or the like and is read by the cut-out picture angle restriction unit 233 in the processing of step S505 described later.

[0037] In step S504, the cut-out picture angle setting unit 231 sets the picture angle information to be cut out and displayed on the display monitor 120 from the captured image by the wide-angle camera 110. As a method for setting the picture angle information, the user 50 may be requested for the coordinate information of the picture angle using the UI on the display monitor 120, or the coordinate information of the picture angle to be cut out may be stored in the ROM in advance.

[0038] In step S505, the cutout angle-of-view limiting unit 233 determines whether or not the determination result by the low-latency mode determination unit 240 is the low-latency mode. If the cutout angle-of-view limiting unit 233 determines that it is the low-latency mode, the process of step S506 is executed. If the cutout angle-of-view limiting unit 233 determines that it is not the low-latency mode, the process of step S508 is executed. That is, when low-latency video display is required, such as during high-speed driving, the process proceeds to step S506. On the other hand, when low-latency video display is not required, the process proceeds to step S508.

[0039] In step S506, the distortion correction determination unit 232 determines whether or not there is distortion in the image (the image to be cut out) at the cutout angle-of-view set by the cutout angle-of-view setting unit 231. In the first embodiment, the distortion correction determination unit 232 refers to the coordinate values of each of the four corners of the cutout angle-of-view, and determines that there is distortion in the image to be cut out when the number of pixels in the radial direction and the circumferential direction with respect to the optical axis, or the absolute coordinate values exceed a predetermined value. If the distortion correction determination unit 232 determines that there is distortion in the image to be cut out, the process of step S507 is executed. If the distortion correction determination unit 232 determines that there is no distortion in the image to be cut out, the process of step S509 is executed.

[0040] In step S507, since low-latency video display is required and there is distortion in the image to be cut out, the cutout angle-of-view limiting unit 233 performs a limiting process on the angle-of-view setting of the image to be cut out. Examples of the limiting process for the angle-of-view setting include a process of setting the coordinates of the four corners of the cutout angle-of-view to be included within the low-distortion region 501, and a process of notifying the user 50 of an error and prompting the user to re-set the cutout angle-of-view. After the cutout angle-of-view limiting unit 233 performs the limiting process on the angle-of-view setting of the image to be cut out, the process of step S509 is executed.

[0041] In step S508, since low-latency video display is not required, the distortion correction processing unit 212 performs distortion correction processing on the captured image of the wide-angle camera 110. After the distortion correction processing unit 212 performs the distortion correction processing, the process of step S509 is executed.

[0042] In step S509, the image extraction unit 221 extracts the captured image of the wide-angle camera 110 to the angle of view set by the angle-of-view setting unit 231. Also, in step S509, the video display unit 222 performs video display processing on the display monitor 120 for the image data extracted by the image extraction unit 221.

[0043] In step S510, the image processing unit 210 determines whether or not the imaging operation by the wide-angle camera 110 has ended. When the image processing unit 210 determines that the imaging operation has ended, the process of step S511 is executed. On the other hand, when the image processing unit 210 determines that the imaging operation has not ended, the process of step S503 is executed.

[0044] In step S511, the video display unit 222 ends the video display on the display monitor 120. After step S511, the execution of a series of peripheral monitoring processes ends, and when the next imaging operation of the wide-angle camera 110 starts, the process is executed again from step S500.

[0045] By the above-described processing, in the peripheral monitoring system that monitors the entire periphery of the moving body 10, when low-latency video display is required, a restriction is imposed to perform the extraction process only within the low-distortion region. As a result, it becomes possible to extract the captured image as it is without performing distortion correction processing, and thus an effect of enabling low-latency video display is achieved.

[0046] <Second Embodiment> In the first embodiment, a system in which the wide-angle camera 110 is installed on the upper part of the moving body 10 with the optical axis facing upward, and the captured image is extracted at an arbitrary angle of view and video display is performed on the display monitor 120 has been described in detail. In the second embodiment of the present invention, a system in which the wide-angle camera 110 is installed on the side of the moving body 10, and the captured image is extracted at a plurality of angles of view and video display is performed on each display monitor 120 will be described in detail. Note that in the second embodiment, processes other than the processes according to the determination result of the low-latency mode determination unit 240, the optical characteristics of the wide-angle camera 110, and the flowchart of the peripheral monitoring process are the same as those in the first embodiment.

[0047] FIG. 6 is a diagram showing an example of the installation mode of the moving body 10 according to the second embodiment, the wide-angle camera 110 arranged on the side of the moving body 10, and the display monitor 120. FIG. 6(A) is a side view of the moving body 10, and FIG. 6(B) is a plan view of the moving body 10 as viewed from above. The wide-angle camera 110 is arranged on the side so as to be able to image the rear of the moving body 10 with low distortion, and a side view display is performed by a display monitor 120 such as an electronic mirror installed on the side of the moving body 10 as well. FIGS. 6(A) and 6(B) illustrate only the wide-angle camera 110 and the display monitor 120 on one side of the moving body 10, but a similar peripheral monitoring system can be applied to both sides of the moving body 10.

[0048] The configuration in the second embodiment is the same as that shown in FIG. 2 in the first embodiment except for the processing according to the determination result of the low-latency mode determination unit 240. In the second embodiment, as the determination to shift to the low-latency mode by the low-latency mode determination unit 240, it is assumed that the traveling speed of the moving body 10 is high, or the relative speed with the target of interest 11 (see FIG. 7) displayed on the display monitor 120 is large. Also, even if the above does not apply, the user 50 may arbitrarily set the low-latency mode.

[0049] The optical characteristics of the wide-angle camera 110 in the second embodiment are the same as the optical characteristics of the first embodiment shown in FIG. 3.

[0050] Next, the method of restricting the angle-of-view setting by the cut-out angle-of-view restricting unit 233 in the second embodiment will be described in detail with reference to FIG. 7. FIG. 7 is a diagram showing a side image of the moving body 10 captured by the wide-angle camera 110 according to the second embodiment and the area to be cut out and displayed on the display monitor 120 surrounded by a broken line.

[0051] In the second embodiment, the wide-angle camera 110 has projection characteristics that occupy the low-distortion region 501 in the lens peripheral portion with a large half field angle θ as shown in FIG. 7, and is characterized in that there is little distortion in the image cut out within the low-distortion region 501. For example, when the image is cut out in the cutout region 503d in FIG. 7, the cutout field angle is within the low-distortion region 501, and the distortion of the cutout image is small. Therefore, the image in the cutout region 503d can be displayed on the video without performing distortion correction processing. In the example of FIG. 7, the target of interest 11 is within the cutout region 503d.

[0052] On the other hand, when the user 50 changes to the cutout region 503e by operating the UI of the display monitor 120 to check a wide range on the side, the cutout field angle protrudes from the low-distortion region 501. Therefore, if the image in the cutout region 503e is directly displayed on the display monitor 120, the image may be greatly distorted and the visibility may be reduced. When performing a cutout display in the cutout region 503e, distortion correction processing is performed, or when low-latency video display is required, cutout field angle limiting processing is performed. The cutout field angle limiting processing is the same as that in the first embodiment.

[0053] By the above processing, also in the peripheral monitoring system that monitors the side of the moving body 10, when low-latency video display is required, the cutout process is restricted to be performed only within the low-distortion region. As a result, it becomes possible to cut out the captured image as it is without performing distortion correction processing, and thus, the same effect of enabling low-latency video display as in the first embodiment is achieved.

[0054] As described above, the present invention has been described in detail based on its preferred embodiments. However, the present invention is not limited to these specific embodiments, and can be modified in various forms without departing from the gist of the present invention.

[0055] In the first and second embodiments, the projection characteristics of the optical unit 111 in the wide-angle camera 110 are illustrated by the characteristics shown in FIG. 3(B) and Equations (1) and (2). The present invention is not limited to this, and the projection characteristics of the optical unit 111 may be changed according to the installation position of the wide-angle camera 110 in each embodiment and the range to be monitored. For example, when the installation height of the wide-angle camera 110 is low in the first embodiment, the lower distortion region 501 may be made wider. That is, a method of using an optical unit 111 having projection characteristics with a gentle slope of the half angle θ - image height y such that θa shown in FIG. 3 shifts in the direction of a smaller angle of view can be mentioned.

[0056] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

[0057] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0058] The disclosure of this embodiment includes the following configurations, methods, and programs. (Configuration 1) A camera that images a peripheral area of a moving body, an information processing device that processes an image captured by the camera, and a monitor that displays an image processed by the information processing device, The camera has an optical unit capable of imaging a distorted region in which the captured image has distortion and a lower distortion region in which the captured image has less distortion than the distorted region. The information processing device Cropping angle setting means for setting a cropping angle for an angle of view to be displayed on the monitor among the captured images, Low-latency mode determination means for determining whether to perform video display on the monitor with low latency, When the low-latency mode determination means determines that video display is to be performed with low latency, there is a cutout angle limit means for restricting so that the distortion area is not included in the cutout angle set by the cutout angle setting means. A peripheral monitoring system characterized by this. (Configuration 2) When the low-latency mode determination means determines that video display is not to be performed with low latency, the information processing device has a distortion correction processing means for correcting the distortion of the captured image at the angle of view set by the cutout angle setting means. The peripheral monitoring system according to Configuration 1, characterized by this. (Configuration 3) When the cutout angle set by the cutout angle setting means protrudes from the low-distortion area, the distortion correction processing means performs distortion correction processing, and when it is within the low-distortion area, the distortion correction processing means does not perform distortion correction processing. The peripheral monitoring system according to Configuration 2, characterized by this. (Configuration 4) When the traveling speed of the moving body is high, or the relative speed with the target of interest is large, or it is real-time peripheral monitoring, the low-latency mode determination means shifts to the low-latency mode in which video is displayed on the monitor with low latency. The peripheral monitoring system according to any one of Configurations 1 to 3, characterized by this. (Configuration 5) When the cutout angle set by the cutout angle setting means protrudes from the low-distortion area, and when the low-latency mode determination means determines that the monitor shifts to the low-latency mode in which video is displayed with low latency, it is restricted to an angle of view that fits within the low-distortion area. The peripheral monitoring system according to any one of Configurations 1 to 4, characterized by this. (Configuration 6) The camera is arranged so that the monitoring area is within the low-distortion area. The peripheral monitoring system according to any one of Configurations 1 to 5, characterized by this. (Configuration 7) The cutout angle setting means sets the coordinate information of the cutout angle through a UI operation such as a touch operation or a mouse operation on the monitor. The peripheral monitoring system according to any one of Configurations 1 to 6, characterized in that. (Configuration 8) The optical unit is optically designed such that the low-distortion region is a region corresponding to the installation position of the camera and the monitoring region. The peripheral monitoring system according to any one of Configurations 1 to 7, characterized in that. (Method 1) An imaging step of imaging the peripheral area of the moving body, where the captured image has a distorted area with distortion, and a cutout angle setting step of setting the angle of view to be displayed on the monitor among the captured images captured by a camera having an optical unit capable of imaging a low-distortion region where the captured image has less distortion than the distorted region, A low-latency mode determination step of determining whether to perform video display on the monitor with low latency, When the low-latency mode determination step determines that video display is to be performed with low latency, a cutout angle restriction step of restricting the cutout angle set in the cutout angle setting step so that the distorted region is not included. An image processing method characterized by that. (Program 1) A computer, A cutout angle setting means for setting the angle of view to be displayed on the monitor among the captured images captured by a camera having an optical unit capable of imaging the peripheral area of the moving body, where the captured image has a distorted area with distortion and a low-distortion region where the captured image has less distortion than the distorted region, A low-latency mode determination means for determining whether to perform video display on the monitor with low latency, and A cutout angle restriction means for restricting the cutout angle set by the cutout angle setting means so that the distorted region is not included when the low-latency mode determination means determines that video display is to be performed with low latency. A program characterized by causing it to function as such.

Explanation of Signs

[0059] 10 Mobile body 11 Target of interest 50 User 110 Wide-angle camera 111 Optical unit 112 Imaging unit 120 Display monitor 200 Processing unit 210 Image processing unit 211 Development / image processing unit 212 Distortion correction processing unit 220 Video control unit 221 Image extraction unit 222 Video display unit 230 Extracted image angle processing unit 231 Extracted image angle setting unit 232 Distortion correction determination unit 233 Extracted image angle limitation unit 240 Low-latency mode determination unit 501 Low-distortion region 502 Distortion region 503a, 503b, 503c, 503d, 503e Extraction regions

Claims

1. A camera that images a peripheral area of a moving body, an information processing device that processes the captured image of the camera, and a monitor that displays the image processed by the information processing device, the camera has an optical unit capable of imaging a distorted area where the captured image has distortion and a low-distortion area where the captured image has less distortion than the distorted area, the information processing device includes a cutout angle setting means for setting a cutout angle of an angle of view to be displayed on the monitor among the captured images, a low-latency mode determination means for determining whether to perform video display on the monitor with low latency, and a cutout angle restriction means for restricting so that the distorted area is not included in the cutout angle set by the cutout angle setting means when the low-latency mode determination means determines that video display is performed with low latency. A peripheral monitoring system characterized by the above.

2. The information processing device has a distortion correction processing means for correcting the distortion of the captured image at the angle of view set by the cutout angle setting means when the low-latency mode determination means determines that video display is not performed with low latency. The peripheral monitoring system according to claim 1, characterized by the above.

3. The distortion correction processing means performs distortion correction processing when the cutout angle set by the cutout angle setting means extends beyond the low-distortion area, and does not perform distortion correction processing when it is within the low-distortion area. The peripheral monitoring system according to claim 2, characterized by the above.

4. The low-latency mode determination means determines to shift to a low-latency mode in which video is displayed on the monitor with low latency when the traveling speed of the moving body is high, or the relative speed with respect to the target of interest is large, or it is real-time peripheral monitoring. The peripheral monitoring system according to claim 1, characterized by the above.

5. The cutout angle restriction means restricts to an angle of view that fits within the low-distortion area when the cutout angle set by the cutout angle setting means extends beyond the low-distortion area and the low-latency mode determination means determines to shift to a low-latency mode in which video is displayed on the monitor with low latency. The peripheral monitoring system according to claim 1, characterized by the above.

6. The camera is arranged so that the monitoring area is within the low-distortion area. The peripheral monitoring system according to claim 1, characterized by the above.

7. The cutout angle setting means sets the coordinate information of the cutout angle by a UI operation such as a touch operation or a mouse operation on the monitor. The peripheral monitoring system according to claim 1, characterized in that.

8. The optical unit is optically designed so that the low-distortion region is a region corresponding to the installation position of the camera and the monitoring region. The peripheral monitoring system according to claim 1, characterized in that.

9. A cutout angle setting step of setting an angle of view to be displayed on a monitor from among the captured images captured by a camera having an optical unit capable of capturing a peripheral region of a moving body and capable of capturing a distorted region in which the captured image has distortion and a low-distortion region in which the captured image has less distortion than the distorted region; A low-latency mode determination step of determining whether to perform video display on the monitor with low latency; When the low-latency mode determination step determines to perform video display with low latency, a cutout angle restriction step of restricting the cutout angle set in the cutout angle setting step so that the distorted region is not included. An image processing method characterized by that.

10. A computer, Cutout angle setting means for setting an angle of view to be displayed on a monitor from among the captured images captured by a camera having an optical unit capable of capturing a peripheral region of a moving body and capable of capturing a distorted region in which the captured image has distortion and a low-distortion region in which the captured image has less distortion than the distorted region; Low-latency mode determination means for determining whether to perform video display on the monitor with low latency, and Cutout angle restriction means for restricting the cutout angle set by the cutout angle setting means so that the distorted region is not included when the low-latency mode determination means determines to perform video display with low latency. A program characterized by causing the computer to function as such.

Images