Mobile body and control method

Abstract

To generate an image with a high visibility specifically in an operation range of an operation machine or the like and their circumference.SOLUTION: A mobile body includes: a movable part; imaging means having an optical system for forming a first field angle in which an optical image with a high resolution is formed and a second field angle in which the optical image with a low resolution is formed by a field angel of the first field angle onto a light reception surface of an imaging element; and generation means of generating a bird's-eye view image that can recognize a movable range of the movable part and the outside of the movable range of the movable part from an image photographed by the imaging means. The imaging means is installed so that the movable range of the movable part is imaged at the second field angle of the optical system.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention particularly relates to a system installed in a moving body such as a working machine.

Background Art

[0002] In a working machine, such as an excavator, there are many blind spots from the driver. Therefore, when the driver of the working machine visually recognizes a monitoring target such as a worker or an obstacle around the excavator, the driver confirms the monitoring target with a camera installed on the working machine. As a result, during the operation of the working machine, especially in the working range of the working machine, the driver can take actions to prevent contact accidents and the like.

[0003] Patent Document 1 and Patent Document 2 disclose a technique of imaging the periphery of a working machine or the like with a wide-angle camera and displaying an aerial image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the prior art disclosed in the above-mentioned patent documents, there is a problem that the distortion is large in the area far from the working machine, and it is difficult for the driver to visually recognize the monitoring target from the image.

[0006] Therefore, in the present disclosure, an image with high visibility is generated particularly in the working range of a working machine or the like and its periphery.

Means for Solving the Problems

[0007] In order to achieve the above object, the moving body includes a movable part, an imaging unit having an optical system that forms a first angular field of view where a high-resolution optical image is formed and a second angular field of view where an optical image with a lower resolution than the angular field of view of the first angular field of view is formed on the light-receiving surface of the imaging device, and a generation unit that generates an aerial image capable of distinguishing between the movable range of the movable part and outside the movable range of the movable part from the image captured by the imaging unit. The imaging unit is installed so as to image the movable range of the movable part at the second angular field of view of the optical system.

Advantages of the Invention

[0008] Particularly in the working range of a working machine or the like and its surroundings, it becomes possible to generate an image with high visibility.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Mode for Carrying Out the Invention

[0010] (First Embodiment) In the first embodiment, an imaging system that generates an image (so-called bird's-eye view image) of a moving object looking down from a virtual viewpoint existing directly above the moving object will be described. In this embodiment, on the moving object, three imaging devices (hereinafter referred to as cameras) having an optical system capable of acquiring a peripheral angle-of-view region away from the optical axis with high resolution are installed in order to photograph the three directions around the moving object, respectively. In addition, in this embodiment, by allocating a region (high-resolution region) that can be acquired with high resolution to a region (region to be monitored) that may enter a region (dangerous region) where there is a risk of contacting the moving object, the region to be monitored in the image from the virtual viewpoint is made to have low distortion. This is, for example, an installation method that addresses the following issues. From the perspective of preventive safety, it is desirable to visually recognize monitoring targets such as people and obstacles in the region to be monitored and pay attention not to enter the dangerous region. However, when imaging with a wide-angle camera having an optical system such as a fish-eye lens and generating a bird's-eye view image, there is a problem that the distortion of the region to be monitored, which is farther from the camera, becomes large, making it difficult to visually recognize the monitoring target and difficult to grasp the sense of distance. Therefore, in this embodiment, when generating a bird's-eye view image from the captured image of the camera having the optical system 1, the camera is installed so as to allocate a high-resolution region to the region to be monitored. As a result, the distortion of the bird's-eye view image in the region to be monitored is reduced, and the effects of making it easy to visually recognize the monitoring target and easy to grasp the sense of distance can be obtained. Hereinafter, this embodiment will be described in detail.

[0011] FIG. 1 is a diagram for explaining the imaging range of a vehicle (for example, an excavator) and cameras in the first embodiment. This vehicle is an example of a moving object.

[0012] As shown in FIG. 1, cameras 11, 12, and 13 (imaging means) are installed at the rear, right side, and left side positions of the vehicle 10 (moving object), respectively.

[0013] Cameras 11 to 13 are imaging units each having an optical system 1 and an imaging device. The imaging directions of cameras 11 to 13 are set so as to image the rear, the right side, and the left side of the vehicle 10 respectively, and each has an imaging range of, for example, an angle of view of about 180 degrees. Further, the optical axes of the optical systems of cameras 11 to 13 are each installed so as to be obliquely downward with respect to the vehicle 10 when the vehicle 10 is placed on a horizontal road surface.

[0014] Imaging ranges 11a to 13a schematically show the angles of view of cameras 11 to 13, and 11b to 13b are high-resolution regions, which schematically show regions where images can be acquired with high resolution due to the characteristics of the optical systems in the respective cameras, and are shown with thin shading. In each of the cameras 11 to 13, the peripheral angle-of-view regions away from the optical axis can be acquired with high resolution. Note that the imaging ranges and the high-resolution regions of cameras 11 to 14 are actually three-dimensional ranges, but are schematically represented in a planar manner in FIG. 1. Further, the imaging ranges of the respective cameras overlap with the imaging ranges of the other adjacent cameras in the peripheral portions.

[0015] Next, FIG. 2 is a functional block diagram for explaining the configuration of the image processing system 100 in the first embodiment, and the image processing system 100 will be described with reference to FIG. 1. Note that some of the functional blocks shown in FIG. 2 are realized by causing a computer (not shown) included in the image processing system 100 to execute a computer program stored in a storage unit 22 as a storage medium.

[0016] Further, each of the functional blocks shown in FIG. 2 does not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

[0017] In FIG. 2, the image processing system 100 is mounted on a vehicle 10 such as an excavator.

[0018] Cameras 11 to 13 each have an imaging device 11d to 13d that images an optical image, and an optical system 11c to 13c that forms an optical image on the light-receiving surface of the imaging device (13c and 13d are not shown). Thereby, the surrounding situation is acquired as image data.

[0019] The optical systems 1 (first optical systems) of the cameras 11 to 13 have optical characteristics of forming a high-resolution optical image in the peripheral angle-of-view region away from the optical axis and forming a low-resolution optical image in the narrow angle-of-view region around the optical axis. Details of the optical systems 11c to 13c will be described later.

[0020] The imaging elements 11d to 13d are, for example, CMOS image sensors or CCD image sensors, which photoelectrically convert an optical image and output imaging data. The imaging elements 11d to 13d have, for example, RGB color filters arranged in a Bayer array for each pixel, and a color image can be obtained by performing demosaicing processing.

[0021] The image processing device 20 (image processing means) includes an information processing unit 21, a storage unit 22, and various interfaces (not shown) for data and power input / output, and includes various hardware. Further, the image processing device 20 is connected to the cameras 11 to 13, and outputs image data obtained by synthesizing a plurality of image data acquired from each camera to the display unit 30 (display means).

[0022] The information processing unit 21 has an image deformation unit 21a (image deformation means) and an image synthesis unit 21b (image synthesis means). Further, for example, it has a SOC (System On Chip), an FPGA (Field Programable Gate Array), a CPU, an ASIC, a DSP, a GPU (Graphics Processing Unit), and a memory.

[0023] The CPU performs various controls of the entire image processing system 100 including the camera and the display unit by executing a computer program stored in the memory.

[0024] In the first embodiment, the image processing apparatus and the camera are housed in separate enclosures. Further, in the information processing unit 21, the image data input from each camera according to the Bayer array is subjected to demosaicing processing and converted into RGB raster format image data. Furthermore, various image processing and image adjustments such as white balance adjustment, gain / offset adjustment, gamma processing, color matrix processing, reversible compression processing, and lens distortion correction processing are performed.

[0025] Also, after the image deformation unit 21a performs image deformation processing for viewpoint conversion, the image synthesis unit 21b synthesizes a plurality of images so as to connect them.

[0026] The storage unit 22 is an information storage device such as a ROM, and stores information necessary for controlling the entire image processing system 100. Incidentally, the storage unit 22 may be a removable recording medium such as a hard disk or an SD card.

[0027] Also, the storage unit 22 stores, for example, camera information of each of the cameras 11 to 13, a coordinate conversion table for performing image deformation / synthesis processing, and parameters for controlling the image processing system 100. Furthermore, the image data generated by the information processing unit 21 may be recorded.

[0028] The camera information includes the optical characteristics of the optical system 1, the number of pixels of the imaging elements 11d to 13d, photoelectric conversion characteristics, gamma characteristics, sensitivity characteristics, frame rate, image format information, mounting position coordinates in the vehicle coordinate system of the camera, and the like. Also, the camera information may include not only the design values of the camera but also adjustment values that are unique values for each camera individual.

[0029] The display unit 30 has a liquid crystal display or an organic EL display as a display panel, and displays an image output from the image processing apparatus 20. Thereby, the driver can grasp the situation around the vehicle. Note that the number of display units is not limited to one. Two or more display units may display patterns of different viewpoints of the synthesized image, a plurality of images acquired from the cameras, and other information.

[0030] Next, the optical characteristics of the optical system 1 of cameras 11 to 13 will be described in detail.

[0031] First, with reference to FIGS. 3 and 4, the optical characteristics of the optical system 1 will be described.

[0032] FIG. 3(A) is a diagram showing the image height y1 at each half field angle on the light-receiving surface of the imaging device of the optical system 1 in the first embodiment in the form of contour lines. Further, FIG. 3(B) is a diagram showing the projection characteristics representing the relationship between the image height y1 and the half field angle θ1 of the optical system 1 in the first embodiment. In FIG. 3(B), the half field angle (the angle formed by the optical axis and the incident light ray) θ1 is taken as the horizontal axis, and the imaging height (image height) y1 on the light-receiving surface (image surface) of the camera is taken as the vertical axis.

[0033] FIGS. 4(A) and (B) are diagrams showing the image height at each half field angle on the light-receiving surface of each optical system in the form of contour lines. FIG. 4(A) shows the optical system 1, and FIG. 4(B) shows the optical system of the equidistant projection method. That is, FIG. 3(A) and FIG. 4(A) are the same. Further, in FIGS. 3 and 4, 40a is a high-resolution region and is shown by being thinly shaded. Also, 40b is a low-resolution region.

[0034] As shown in FIG. 4(B), a lens of the equidistant projection method (y = fθ), which is common as a fish-eye lens, has a constant resolution at each image height position and has projection characteristics of a proportional relationship.

[0035] On the other hand, as shown in the projection characteristics of FIG. 3(B), the optical system 1 in the first embodiment is configured such that its projection characteristics y1(θ1) change in a region with a small half field angle θ1 (near the optical axis) and a region with a large half field angle (far from the optical axis). That is, when the increase amount of the image height y1 with respect to the half field angle θ1 per unit (that is, the number of pixels per unit angle) is defined as the resolution, the resolution varies depending on the region.

[0036] This local resolution can also be expressed as the derivative value dy1(θ1) / dθ1 at the half angle θ1 of the projection characteristic y1(θ1). That is, it can be said that the higher the slope of the projection characteristic y1(θ1) in Fig. 3(B), the higher the resolution. Also, it can be said that the higher the resolution, the greater the interval between the image heights y1 at each half angle of the contour line in Fig. 3(A).

[0037] In the first embodiment, the region from the center formed on the light-receiving surface of the sensor when the half angle θ1 is less than a predetermined half angle θ1a is called the low-resolution region 40b, and the outer region where the half angle θ1 is greater than or equal to the predetermined half angle θ1a is called the high-resolution region 40a. That is, the optical system 1 (the first optical system) forms a first optical image having a low-resolution region 40b corresponding to an angle of view less than the first angle of view (half angle θ1a) and a high-resolution region corresponding to an angle of view greater than or equal to the first angle of view (half angle θ1a).

[0038] The camera also captures the first optical image formed by the first optical system to generate image data.

[0039] Note that the value of the half angle θ1a is an example for explaining the optical system 1 and is not an absolute value.

[0040] Also, the high-resolution region 40a corresponds to the high-resolution regions 11b to 13b in Fig. 1.

[0041] Looking at the projection characteristic in Fig. 3(B), it can be seen that in the low-resolution region 40b where the angle of view is small near the optical axis, the increase rate (slope) of the image height y1 is small, and as the angle of view gradually increases, the increase rate (slope) becomes larger. This is a projection characteristic with a more significant change in slope than the generally known stereographic projection (y = 2f×tan(θ / 2)).

[0042] To realize these characteristics, it is preferable to satisfy the conditions shown in the following mathematical formula 1.

[0043]

Equation

[0044] y1(θ1): The projection characteristic representing the relationship between the half field angle θ1 of the first optical system and the image height y1 on the image plane. θ1max: The maximum half field angle of the first optical system (the angle formed by the outermost chief ray and the optical axis). f1: The focal length of the first optical system. A is a predetermined constant, which may be determined considering the balance of the resolution between the high-resolution region and the low-resolution region. It is preferably about 0.92, and more preferably about 0.8.

[0045] If it exceeds the lower limit of Equation (1), image curvature, distortion aberration, etc. will deteriorate and good image quality cannot be obtained. If it exceeds the upper limit, the difference in resolution between the central region and the peripheral region will decrease and the required projection characteristics cannot be realized.

[0046] Figure 5 is a graph showing an example of the equidistant projection and resolution characteristics of the optical system 1 in the first embodiment. The horizontal axis is the half field angle θ, and the vertical axis is the resolution, which is the number of pixels per unit angle of the image. In equidistant projection, the resolution is constant at any half field angle position, while the optical system 1 has the characteristic that the resolution increases at positions with a larger half field angle.

[0047] By using the optical system 1 with the above characteristics, for example, while imaging a wide field angle equivalent to that of a fish-eye lens of 180 degrees or the like, in the high-resolution region, a high-resolution image can be obtained. That is, in the optical system 1, the peripheral field angle region far from the optical axis becomes the high-resolution region, and it is possible to obtain a high-resolution image with less distortion on the far side of the vehicle.

[0048] In addition, as long as the optical system 1 has the projection characteristic y1(θ1) that satisfies the conditions of the above Equation (1), the same effect can be obtained. Therefore, the optical system 1 of the first embodiment is not limited to the projection characteristics shown in FIGS. 3 to 5.

[0049] FIG. 6 is a flowchart for explaining the flow of the image processing method executed by the information processing unit 21 of the first embodiment. Using the processing flow of FIG. 6, the processing contents executed by the image deformation unit 21a and the image composition unit 21b will also be described. The processing flow of FIG. 6 is controlled in frame units, for example, by a CPU inside the information processing unit 21 executing a computer program in the memory.

[0050] When the image processing system 100 is powered on, or triggered by an operation by the driver, a change in the driving state, etc., the processing flow of FIG. 6 starts.

[0051] In step S11, the information processing unit 21 acquires the image data in three directions of FIG. 1 of the vehicle 10 captured by the cameras 11 to 13. In this embodiment, the imaging by the cameras 11 to 13 is performed simultaneously (synchronously).

[0052] In step S12, the information processing unit 21 performs an image deformation process of converting the acquired image data into an image from a virtual viewpoint.

[0053] At this time, the image deformation unit 21a deforms the images acquired from the cameras 11 to 13 based on the calibration data stored in the storage unit 22. Alternatively, it may be deformed based on various parameters such as a coordinate conversion table based on the calibration data. The content of the calibration data includes internal parameters of the camera such as the lens distortion amount of each camera and the deviation from the sensor position, and external parameters representing the relative positional relationship between each camera and between the vehicle.

[0054] The viewpoint conversion will be described with reference to FIG. 7. FIG. 7 is a diagram for explaining the virtual viewpoint and image deformation of the first embodiment, where the vehicle 10 is present on the road surface 60. Note that the side cameras 12 and 13 are not shown. The vehicle 10 is imaging the rear with the camera 11, and the imaging range of the camera 11 includes the road surface 60 around the vehicle 10. The information processing unit 21 projects the image acquired by the camera 11 onto the road surface 60 as a projection plane, and assuming that there is a virtual camera at the virtual viewpoint 50 directly above the vehicle, the information processing unit 21 performs coordinate conversion (deformation) on the image as if the virtual camera were photographing the projection plane. That is, the information processing unit 21 generates a virtual viewpoint image (so-called bird's-eye view image) from the virtual camera (virtual viewpoint) by performing coordinate conversion on the image photographed by the camera.

[0055] Note that the information processing unit 21 can project the image onto the projection plane and obtain the image from a different viewpoint by performing coordinate conversion, using various parameters included in the calibration data. It is assumed that the calibration data has been calculated by performing camera calibration in advance. Also, considering the virtual camera as an orthographic projection camera, the generated image can be an image with no distortion and easy to perceive the sense of distance.

[0056] Also, the side cameras 12 and 13 (not shown) can perform similar processing to deform the image. Also, the projection plane does not have to be a plane imitating the road surface, and for example, it can have a bowl-shaped three-dimensional shape. Also, the position of the virtual viewpoint does not have to be directly above the vehicle, and for example, it can be a viewpoint looking obliquely forward or backward from the vehicle, or a viewpoint looking at the surroundings from inside the vehicle.

[0057] Returning to the description of the flow of FIG. 6, in step S13, the information processing unit 21 synthesizes a plurality of images that have been converted and deformed in step S12.

[0058] In step S14, the information processing unit 21 outputs the image synthesized in step S13 and displays it on the display unit 30. As a result, the user can view the image from the virtual viewpoint with high resolution.

[0059] Hereinafter, the information processing unit 21 can display an aerial image as a video by repeatedly executing the flow of FIG. 6 in frame units. By viewing the video of the aerial image, the user can continuously grasp the position of relative obstacles with high resolution.

[0060] When the vehicle has movable parts such as an arm or a bucket like an excavator, the range where it may come into contact with the movable part (hereinafter referred to as the danger area) is wide, and it is difficult for the user to know up to where the danger area is when checking safety in the bird's-eye view image. The range where it may come into contact with the movable part is the movable range of the movable part. When the vehicle body turns, it becomes even more difficult to understand. Therefore, it is desirable to represent the boundary between the danger area and the area where there is a possibility of entering the surrounding danger area (hereinafter referred to as the area to be monitored) with a line or the like and superimpose it on the bird's-eye view image. In this embodiment, the area to be monitored is outside the movable range of the movable part. As a result, the user can distinguish between the danger area and the area to be monitored. As an example, the boundary line between the danger area and the area to be monitored may be set on the outer periphery of the maximum range where the vehicle and the monitoring target may come into contact. In this case, for example, if the vehicle is an excavator, the outermost outer periphery of the range through which the excavator passes when the boom, arm, and bucket are extended to the maximum on the far side from the excavator and turned is the boundary line. The setting of the boundary line is not limited to this, and it may be set on the outer periphery of a range with more margin than the above example.

[0061] Next, an example of the installation method of the camera having the optical system 1 when allocating a high-resolution area to the area to be monitored will be described with reference to FIG. 8. FIG. 8 illustrates an example of the camera 11 installed behind the vehicle. Note that the camera 11 faces directly behind the vehicle, and FIG. 8 is a view from the side with respect to the traveling direction of the vehicle. Let the installation height of the camera 11 be h1, the horizontal length from the focal point of the camera 11 to the boundary line be r1, the half-angle at which the projection characteristic y(θ) of the optical system changes be θ1a, and the angle formed by the direction directly below and the optical axis center direction from the focal point of the camera 11 be Φ1. When the above θ1a, h1, and r1 are known, Φ1 can be expressed by the following formula 2.

[0062] [Number]

[0063] Therefore, if the camera is installed so as to tilt the optical axis center by Φ1 around the focal point from directly below, a high-resolution area can be assigned to the area to be monitored. In FIG. 8, the angular field of view that can be imaged with high resolution is shown by being thinly shaded. Here, h1 needs to satisfy the condition that the range of the desired bird's-eye view image is higher than the height at which imaging is possible. Here, an example of the installation method that can assign a high-resolution area to the area to be monitored is shown, but it is not limited thereto, as long as most of the high-resolution area is assigned to the area to be monitored.

[0064] As described above, for the imaging system that generates an image (bird's-eye view image) of looking down on a vehicle from a virtual viewpoint existing directly above the vehicle by installing four cameras each having an optical system capable of acquiring a high-resolution peripheral angular field area away from the optical axis for photographing the four directions around the moving object. In the present embodiment, when generating a bird's-eye view image from the captured image of the camera having the optical system 1, the camera is installed so as to assign a high-resolution area to the area to be monitored. As a result, the distortion of the bird's-eye view image in the area to be monitored is reduced, and it is possible to obtain the effect that the monitoring target is easy to visually recognize and the sense of distance is easy to grasp.

[0065] (Second Embodiment) In the second embodiment, the three directions around the moving body are further imaged respectively. For this purpose, three cameras having an optical system capable of acquiring a narrow-angle region around the optical axis with high resolution on the moving body are installed, and an imaging system for generating an image (bird's-eye view image) of looking down on the vehicle from a virtual viewpoint existing directly above the vehicle will be described. In this embodiment, by allocating a high-resolution region, which is a region that can be acquired with high resolution, to a region (region to be monitored) that may enter a region (hazardous region) where there is a risk of contacting the moving body, the region to be monitored in the image from the virtual viewpoint is made to have low distortion. From the viewpoint of preventive safety, it is desirable to visually recognize monitoring targets such as people and obstacles in the region to be monitored and pay attention not to enter the hazardous region. However, when imaging with a wide-angle camera having an optical system such as a fish-eye lens and generating a bird's-eye view image, there is a problem that the distortion of the region to be monitored, which is farther from the camera, becomes large, making it difficult to visually recognize the monitoring target and also difficult to grasp the sense of distance. In this embodiment, when generating a bird's-eye view image from the captured images of the cameras having the optical system 2, the cameras are installed so as to allocate a high-resolution region to the region to be monitored. As a result, it is possible to obtain an effect that the distortion of the bird's-eye view image in the region to be monitored is reduced, making it easier to visually recognize the monitoring target and easier to grasp the sense of distance.

[0066] FIG. 9 is a diagram for explaining the imaging range of a vehicle (for example, an excavator) and cameras in the second embodiment. As shown in FIG. 9, cameras 14, 15, and 16 (imaging means) are installed at the rear, right side, and left side positions of the vehicle 10 (moving body), respectively.

[0067] The cameras 14 to 16 are imaging units having an optical system and an image sensor. The imaging directions of the cameras 14 to 16 are set so as to image the rear, right side, and left side of the vehicle 10 respectively as the imaging ranges, and each has an imaging range of, for example, about 180 degrees of the angle of view. Also, the optical axes of the optical systems of the cameras 14 to 16 are installed so as to be obliquely downward with respect to the vehicle 10 when the vehicle 10 is placed on a horizontal road surface.

[0068] The imaging ranges 14a to 16a schematically show the field angles of the cameras 14 to 16. 14b to 16b are high-resolution regions, which schematically show the regions where images can be acquired with high resolution due to the characteristics of the optical systems in the respective cameras, and are shown in thinly shaded notation. Each of the cameras 14 to 16 can acquire a high-resolution image in a narrow field angle region around the optical axis. Note that the imaging ranges and high-resolution regions of the cameras 14 to 16 are actually three-dimensional ranges, but are schematically represented in a planar manner in FIG. 9. Also, the imaging ranges of the respective cameras overlap at the peripheral portions with the imaging ranges of the other adjacent cameras.

[0069] Next, FIG. 10 is a functional block diagram for explaining the configuration of the image processing system 200 in the second embodiment, and the image processing system 200 will be described with reference to FIG. 10. Note that some of the functional blocks shown in FIG. 10 are realized by causing a computer (not shown) included in the image processing system 200 to execute a computer program stored in a storage unit 22 as a storage medium.

[0070] Also, each of the functional blocks shown in FIG. 10 does not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

[0071] In FIG. 10, the image processing system 200 is mounted on a vehicle 10 such as a shovel car.

[0072] Each of the cameras 14 to 16 includes an image sensor 14d to 16d that captures an optical image, and an optical system 14c to 16c that forms an optical image on the light-receiving surface of the image sensor (16c and 16d are not shown).

[0073] Thereby, the surrounding situation is acquired as image data.

[0074] The optical system 2 (second optical system) included in the cameras 14 to 16 has an optical characteristic of forming a high-resolution optical image in a narrow field angle region around the optical axis and forming a low-resolution optical image in a peripheral field angle region away from the optical axis.

[0075] Next, the optical characteristics of the optical system 2 included in the cameras 14 to 16 will be described in detail.

[0076] First, with reference to FIGS. 11 and 12, the optical characteristics of the optical system 2 will be described.

[0077] FIG. 11(A) is a diagram showing, in a contour line manner, the image height y2 at each half field angle on the light receiving surface of the imaging element of the optical system 2 in the second embodiment. Further, FIG. 11(B) is a diagram showing the projection characteristics representing the relationship between the image height y2 and the half field angle θ2 of the optical system 2 in the second embodiment. In FIG. 11(B), the half field angle (the angle formed by the optical axis and the incident light ray) θ2 is taken as the horizontal axis, and the imaging height (image height) y2 on the light receiving surface (image surface) of the cameras 14 to 16 is taken as the vertical axis.

[0078] FIGS. 12(A) and (B) are diagrams showing, in a contour line manner, the image height at each half field angle on the light receiving surface of the imaging element of each optical system. FIG. 12(A) shows the optical system 2, and FIG. 12(B) shows the equidistant projection method. That is, FIG. 11(B) and FIG. 12(B) are the same. Further, in FIGS. 11 and 12, 41a is a high resolution region and is shown by being thinly shaded. Also, 41b is a low resolution region.

[0079] As shown in FIG. 12(B), a lens of the equidistant projection method (y = fθ), which is common as a fish-eye lens, has a constant resolution at each image height position and has projection characteristics of a proportional relationship.

[0080] On the other hand, as shown in the projection characteristics of FIG. 11(A), the optical system 2 included in the cameras 14 to 16 is configured such that its projection characteristics y2(θ2) change in a region where the half field angle θ2 is small (near the optical axis) and a region where the half field angle θ2 is large (far from the optical axis). That is, when the increase amount of the image height y2 with respect to the half field angle θ2 per unit (that is, the number of pixels per unit angle) is defined as the resolution, the resolution varies depending on the region.

[0081] This local resolution can also be expressed as the derivative value dy2(θ2) / dθ2 at the half-angle θ2 of the projection characteristic y2(θ2). That is, it can be said that the higher the slope of the projection characteristic y2(θ2) in Fig. 11(B), the higher the resolution. Also, it can be said that the higher the resolution, the greater the interval between the image heights y2 at each half-angle of the contour lines in Fig. 11(A).

[0082] In the optical system 2 in the second embodiment, the region near the center generated on the sensor surface when the half-angle θ2 is less than a predetermined half-angle θ2b is called the high-resolution region 41a, and the outer region where the half-angle θ2 is greater than or equal to the predetermined half-angle θ2b is called the low-resolution region 41b. That is, the optical system 2 (the second optical system) forms a second optical image having a high-resolution region 41a corresponding to an angle of view less than the second angle of view (half-angle θ2b) and a low-resolution region 41b corresponding to an angle of view greater than or equal to the second angle of view.

[0083] Here, in Fig. 12(A), the value of θ2 corresponding to the image height position at the boundary between 41a and 41b is θ2b, and the angle of view of the high-resolution region 41a corresponds to the high-resolution regions 14b to 16b in Fig. 9.

[0084] The optical system 2 (the second optical system) is configured such that in the high-resolution region 41a, the projection characteristic y2(θ2) representing the relationship between the half-angle θ2 of the second optical system and the image height y2 on the image plane is greater than f2×θ2. However, f2 is the focal length of the second optical system of the cameras 14 to 16. Also, the projection characteristic y2(θ2) in the high-resolution region is set to be different from the projection characteristic in the low-resolution region.

[0085] When θ2max is the maximum half-angle of the optical system 2, it is desirable that the ratio θ2b / θ2max of θ2b to θ2max is equal to or greater than a predetermined lower limit value. For example, it is desirable that the predetermined lower limit value is 0.15 to 0.16. Also, it is desirable that the ratio θ2b / θ2max of θ2b to θ2max is equal to or less than a predetermined upper limit value. For example, it is desirable to set it to 0.25 to 0.35. For example, when θ2max is 90°, the predetermined lower limit value is 0.15, and the predetermined upper limit value is 0.35, it is desirable to determine θ2b in the range of 13.5 to 31.5°.

[0086] Furthermore, the optical system 2 (second optical system) is configured to satisfy the following formula 3.

[0087]

Equation

[0088] y2(θ2): Projection characteristic representing the relationship between the half angle θ2 of the second optical system and the image height y2 on the image plane θ2max: Maximum half angle of the second optical system (angle formed by the outermost chief ray from the optical axis) f2: Focal length of the second optical system Here, B is a predetermined constant. By setting the lower limit value to 1, the central resolution can be made higher than that of a fisheye lens with an orthographic projection method (y = f×sinθ) having the same maximum imaging height. By setting the upper limit value to B, a good optical performance can be maintained while obtaining a field of view equivalent to that of a fisheye lens. The predetermined constant B may be determined in consideration of the balance between the resolution in the high-resolution region and the low-resolution region, and it is desirable to set it to be in the range of 1.9 to 1.4.

[0089] FIG. 13 is a graph showing an example of the equidistant projection and resolution characteristics of the optical system 2 in the second embodiment. The horizontal axis represents the half angle θ, and the vertical axis represents the resolution which is the number of pixels per unit angle of view. In equidistant projection, the resolution is constant at any half angle position, whereas the optical system 2 has the characteristic that the resolution becomes higher at positions with a smaller half angle.

[0090] By using the optical system 2 having the above characteristics, for example, while imaging a wide field of view equivalent to that of a fisheye lens such as 180 degrees, a high-resolution image can be obtained in the high-resolution region.

[0091] That is, in the optical system 2, the vicinity of the optical axis becomes a high-resolution region, and since it has characteristics approximated to the central projection method (y = f × tan θ) or the equidistant projection method (y = f × θ), which are the projection characteristics of a normal imaging optical system, the optical distortion is small and fine display is possible. Therefore, a natural sense of perspective can be obtained when visually observing a monitoring target in the area to be monitored, and deterioration of the image quality can be suppressed to obtain good visibility.

[0092] In addition, as long as the optical system 2 has projection characteristics y2(θ2) that satisfy the conditions of the above formula (2), the same effect can be obtained. Therefore, the optical system 2 of the second embodiment is not limited to the projection characteristics shown in FIGS. 11 to 13.

[0093] The description of the image sensor, the image processing apparatus 20 (image processing means), the information processing unit 21, the storage unit 22, and the display unit 30 and the processing flow are the same as those in the first embodiment, and thus are omitted.

[0094] When the vehicle has movable parts such as an arm or a bucket like an excavator, the range where it may come into contact with the vehicle body (hereinafter referred to as the danger area) is variable, and it is difficult to know where the danger area is when performing a safety check using an aerial image. When the vehicle body turns, it becomes even more difficult to understand. Therefore, it is desirable to superimpose and display on the aerial image a line as the boundary between the danger area and the area (hereinafter referred to as the area to be monitored) that may intrude into the danger area around it. The boundary line between the danger area and the area to be monitored may be set on the outer periphery of the maximum range where the vehicle and the monitoring target may come into contact. For example, when the vehicle is an excavator, the outer periphery of the range through which the excavator passes when the boom, arm, and bucket are extended to the maximum on the far side from the excavator and turned is the boundary line. The setting of the boundary line is not limited to this, and it may be set on the outer periphery of a range with more margin than the above example.

[0095] Next, an example of the installation method of the camera having the optical system 2 when allocating the high-resolution area to the area to be monitored will be described with reference to FIG. 14. FIG. 14 illustrates an example of the camera 14 installed behind the vehicle. Note that the camera 14 faces directly behind the vehicle, and FIG. 14 is a view from the side with respect to the traveling direction of the vehicle. Let the installation height of the camera 14 be h2, the horizontal length from the focal point of the camera 14 to the boundary line 1 be r2, the half angle of view at which the projection characteristic y(θ) of the optical system changes be θ2b, and the angle formed by the direction directly below and the optical axis center direction from the focal point of the camera 14 be Φ2. When the above θ2b, h2, and r2 are known, Φ2 can be expressed by the following formula.

[0096]

Equation

[0097] Therefore, the camera can allocate the high-resolution area to the area to be monitored by installing it so that the optical axis center is tilted by Φ2 around the focal point from the directly below direction. In FIG. 14, the angle of view that can be imaged with high resolution is shown by thinly shading. Here, h2 needs to satisfy the condition that the height is equal to or greater than the height at which the desired bird's-eye view image range can be imaged. Here, an example of the installation method that can allocate the high-resolution area to the area to be monitored has been shown, but it is not limited thereto, and it is sufficient that most of the high-resolution area is allocated to the area to be monitored.

[0098] As described above, for the imaging system that generates an image (bird's-eye view image) of looking down on the vehicle from a virtual viewpoint existing directly above the vehicle by installing four cameras having an optical system capable of acquiring a narrow angle-of-view area around the optical axis with high resolution for photographing the four directions around the vehicle as the moving object, in this embodiment, when generating a bird's-eye view image from the captured image of the camera having the optical system 2, the camera is installed so as to allocate the high-resolution area to the area to be monitored. As a result, the distortion of the bird's-eye view image in the area to be monitored is reduced, and the effects of making it easier to visually recognize the monitoring target and making it easier to grasp the sense of distance can be obtained.

[0099] In the above-described first and second embodiments, an example where the optical system is rotationally symmetric has been described. However, the present invention is not limited to this, and an optical characteristic that is non-rotationally symmetric may also be used.

[0100] In the above-described first and second embodiments, an example where three cameras are installed rearward and on the left and right has been shown. However, the present invention is not limited to this.

[0101] In the above-described first and second embodiments, the case where the monitoring target is visually confirmed using the bird's-eye view image displayed on the display unit 30 has been described. Not only visually, but also using a learned model for monitoring target recognition obtained by learning learning data including the monitoring target using a GPU (Graphics Processing Unit) or the like, it may be detected that there is a monitoring target in the bird's-eye view image. By allocating a high-resolution area to the area to be monitored, it is expected that the monitoring target will be easily photographed in detail and the detection accuracy of the monitoring target will be improved. Furthermore, in the above-described embodiment, it has been explained that the distortion is reduced by allocating a high-resolution area to the area to be monitored. Since the distortion is small, it is also advantageous for recognition by the learning model, and an improvement in the recognition rate can be expected. When detected by the recognition means, a mark may be superimposed on the monitoring target shown in the bird's-eye view image to make it easier to understand, or the driver may be notified by sound or light.

[0102] In the above-described first and second embodiments, the image sensor is assumed to be a general image sensor. However, a sensor capable of acquiring depth such as a dual-pixel CMOS sensor may be used as additional information for safety confirmation. For example, by adding depth information to the bird's-eye view image displayed on the display unit 30, it becomes easier to visually recognize a monitoring target that is difficult to visually recognize, such as the same color as the road surface. Also, in recognition by the learning model, by using depth information together with the bird's-eye view image, an improvement in the recognition rate of the monitoring target or a decrease in the false recognition rate can be expected.

[0103] In the above-described first and second embodiments, it is assumed that the display unit 30 is installed in the driver's seat and the driver checks the display unit 30. However, the display unit 30 may be installed outside the driver's seat and used, for example, for safety confirmation during remote operation.

[0104] As described above, the preferred embodiments of the present invention have been explained. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0105] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in the computer of the system or device to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0106] Note that the present invention is not limited to the above-described embodiments as they are, and at the implementation stage, the components can be modified and embodied without departing from the gist thereof. Further, various inventions can be formed by appropriately combining a plurality of components disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0107] Note that in the above-described embodiments, "at least one of A and B" may mean only A, only B, or both A and B.

[0108] The disclosure of the present embodiment includes the following configurations and methods.

[0109] (Configuration 1) A movable part, Imaging means having an optical system that forms a first angle of view where a high-resolution optical image is formed and a second angle of view where a lower-resolution optical image than the angle of view of the first angle of view is formed on the light-receiving surface of an image sensor, Generating means for generating an aerial image capable of distinguishing the movable range of the movable part and outside the movable range of the movable part from the image captured by the imaging means, The imaging means is installed so as to image the movable range of the movable part at the second angle of view of the optical system A moving body characterized by the following.

[0110] (Configuration 2) The moving body according to Configuration 1, further comprising display means for displaying the bird's-eye view image generated by the generation means.

[0111] (Configuration 3) The moving body according to Configuration 1 or 2, wherein a boundary between the movable range of the movable part and outside the movable range of the movable part is set on the outermost outer periphery of a range where the movable part may come into contact.

[0112] (Configuration 4) The moving body according to any one of Configurations 1 to 3, wherein the optical system forms a first viewing angle in a peripheral viewing angle region away from the optical axis and a second viewing angle in a viewing angle region around the optical axis.

[0113] (Configuration 5) The imaging means is installed such that, when a switching angle between a first viewing angle and a second viewing angle of the imaging means is a half viewing angle θ1a, a region imaged at the half viewing angle θ1a coincides with a boundary line between a danger region and a region to be monitored. The moving body according to Configuration 4.

[0114] (Configuration 6) The moving body according to any one of Configurations 1 to 3, wherein the optical system forms a first viewing angle in a viewing angle region around the optical axis and a second viewing angle in a peripheral viewing angle region away from the optical axis.

[0115] (Configuration 7) The moving body according to Configuration 6, wherein the imaging means is installed such that, when a switching angle between a first viewing angle and a second viewing angle of the imaging means is a half viewing angle θ2b, a region imaged at the half viewing angle θ2b coincides with a boundary line between a danger region and a region to be monitored.

Claims

1. A movable part, imaging means having an optical system that forms a first angular field of view in which a high-resolution optical image is formed and a second angular field of view in which an optical image with a lower resolution than the angular field of view of the first angular field of view is formed on a light-receiving surface of an image sensor, generating means for generating an aerial image capable of distinguishing the movable range of the movable part and outside the movable range of the movable part from the image captured by the imaging means, and the imaging means is installed so as to image the movable range of the movable part at the second angular field of view of the optical system A moving body characterized by this.

2. The moving body according to claim 1, further comprising display means for displaying the aerial image generated by the generating means.

3. The moving body according to claim 1, wherein a boundary between the movable range of the movable part and outside the movable range of the movable part is set on the outermost outer periphery of a range where there is a possibility of contacting the movable part.

4. The moving body according to claim 1, wherein the optical system forms the first angular field of view in an angular field region away from the optical axis and the second angular field of view in an angular field region around the optical axis.

5. The imaging means is installed such that when the angular field at which the first angular field of view and the second angular field of view of the imaging means are switched is a half angular field θ1a, the boundary line between the danger area and the area to be monitored coincides with the area imaged at the half angular field θ1a. The moving body according to claim 4.

6. The moving body according to claim 1, wherein the optical system forms the first angular field of view in an angular field region around the optical axis and the second angular field of view in an angular field region away from the optical axis.

7. The imaging means is installed such that when the angular field at which the first angular field of view and the second angular field of view of the imaging means are switched is a half angular field θ2b, the boundary line between the danger area and the area to be monitored coincides with the area imaged at the half angular field θ2b. The moving body according to claim 6.

8. A movable part, imaging means having an optical system that forms a first angular field of view in which a high-resolution optical image is formed and a second angular field of view in which an optical image with a lower resolution than the angular field of view of the first angular field of view is formed on a light-receiving surface of an image sensor, and the imaging means is installed so as to image the movable range of the movable part at the second angular field of view of the optical system, and a control method for a moving body, A control method characterized by having a generation step of generating an aerial image capable of distinguishing the movable range of the movable part and outside the movable range of the movable part from the image captured by the imaging means.

Images