Imaging system and vehicle

Abstract

To provide an imaging system capable of monitoring the rear of a vehicle and a cargo area using a single imaging device.SOLUTION: An imaging system includes an imaging device disposed in a vehicle having a cargo bed, and a display image generating unit that outputs image data generated by the imaging device to an image display unit, and the imaging device is disposed so as to be able to simultaneously capture images of a first area behind the vehicle and a second area including the cargo bed.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an imaging system, a vehicle, and the like. [Background technology]

[0002] A moving platform transport vehicle (hereinafter referred to as a dump truck) that transports soil and sand used in civil engineering works has a problem that there are many blind spots when the operator sitting in the driver's seat in the driver's cab sees the rear. For example, it is difficult to see directly behind the vehicle using the rearview mirror due to the soil and sand piled in the platform.

[0003] In particular, when discharging soil, the operator had to check the rear of the vehicle through a small gap between the loading platform and the vehicle body. For this reason, in Patent Document 1, the transport vehicle is equipped with two surveillance camera devices, a surveillance camera inside the loading platform and a surveillance camera behind the vehicle, and the images captured by these surveillance cameras are displayed on a monitor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-105124 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the configuration of Patent Document 1, there was an issue that the camera installed on the lower running body could not monitor the loading platform, and the camera installed on the top of the loading platform could not monitor the rear when the loading platform was moving, making it difficult to achieve both rear monitoring and loading platform monitoring with a single camera.

[0006] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide an imaging system capable of monitoring the rear of a vehicle and the cargo bed with a single imaging device. [Means for solving the problem]

[0007] In order to achieve the above object, an imaging system according to one aspect of the present invention comprises: An imaging device disposed in a vehicle having a loading platform; a display image generating unit that outputs the image data generated by the imaging device to an image display unit, The imaging device is characterized in that it is disposed so as to be able to simultaneously capture images of a first area behind the vehicle and a second area including the cargo bed. Effect of the Invention

[0008] According to the present invention, it is possible to provide an imaging system capable of monitoring the rear of a vehicle and the cargo bed with a single imaging device. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an external view that typically illustrates the external appearance of a dump truck that is an example of a transport vehicle according to a first embodiment of the present invention. [Diagram 2] 5A and 5B are diagrams illustrating the optical characteristics of the optical system according to the present embodiment. [Diagram 3] 1 is a functional block diagram showing an example of the configuration of an imaging system 107 including an imaging device 105 and a control unit 12 according to the present embodiment. [Figure 4] 2 is a diagram illustrating an example of a dump truck, which is an example of a transport vehicle, and an example of an imaging range of an imaging device 105 in the first embodiment. FIG. [Diagram 5] 4A to 4C are diagrams illustrating an example of a platform movable area and a camera imaging range in the first embodiment. [Figure 6] 3 is a diagram showing an example of an image captured by the imaging device 105 of the first embodiment. FIG. [Figure 7] FIG. 2 is a diagram showing an example of a cut-out image displayed by the image display unit 106 of the first embodiment. [Figure 8] 1 is a diagram showing an example of an image captured when a loading platform 102 is in a raised state in the first embodiment. FIG. [Figure 9]FIG. 8 is a diagram showing an example of an image cut out from an image 801 in which the loading platform 102 is in a raised state. [Figure 10] FIG. 11 is a diagram showing an example of a recognition result image displayed by an image display unit 106 in the third embodiment. [Figure 11] 5 is a flowchart showing an example of processing performed by the imaging device of the first embodiment. [Figure 12] 12 is a continuation of the flowchart in FIG. 11. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0011] <Embodiment 1> 1 is an external view that typically shows the external appearance of a dump truck that is an example of a transport vehicle according to embodiment 1. Note that, although this embodiment is described taking a dump truck as an example, the transport vehicle may be any vehicle equipped with a loading platform, and may be, for example, a crawler carrier, a container loading vehicle, an articulated dump truck, or any other transport vehicle equipped with a movable loading platform.

[0012] 1, a transport vehicle 100 in the first embodiment is composed of a running body 101 having wheels and a loading platform 102. The loading platform 102 can be tilted in the vertical direction, that is, raised and lowered, with the lower part of the rear end of the loading platform 102 as a fulcrum, by a hydraulic pump 103 provided on the running body 101. The loading platform 102 has a width approximately equal to that of the running body 101 in the width direction.

[0013] The loading platform 102 is made up of a loading platform and a front wall, left and right side walls, and a rear door that surround the loading platform. These walls and the rear door, together with the loading platform, form a substantially rectangular recess in which soil and sand are loaded.

[0014] In this embodiment, the imaging device 105 is installed on the upper part of the front wall of the loading platform 102, and is arranged so as to be able to image the rear of the transport vehicle 100. As a result, the imaging device 105 can image cargo such as sand and soil loaded on the loading platform 102, and can also image vehicles behind. The captured image can be displayed on the image display unit 106. The imaging device 105 has an optical system 10 and an imaging unit 11.

[0015] Then, when the loading platform 102 is raised and tilted, and the rear door is opened by its own weight, the soil and sand loaded on the loading platform 102 slides off the loading platform 102 and is discharged onto the ground. The hydraulic pump 103 of the running body 101 raises or lowers the loading platform 102 to change the lifted or lowered state of the loading platform 102.

[0016] Also provided is an angle detection sensor 104 that detects the rising or lowering state of the loading platform 102, and this angle detection sensor 104 detects a change in the inclination angle 20a between the running body 101 and the loading platform 102, for example, by measuring the extended length of the hydraulic pump 103. Reference numeral 108 denotes a warning unit, which includes warning lights such as a rotating light and a Patlite (registered trademark), and which turns on a red warning light, for example, when it detects the approach of an object (person, object, etc.).

[0017] Next, the optical characteristics of the optical system 10 of the imaging device 105 will be described in detail. Figures 2(A) and (B) are diagrams for explaining the optical characteristics of the optical system according to this embodiment. Figure 2(A) is a diagram showing the image height y at each angle of view on the light receiving surface of the imaging element included in the imaging unit 11 in the form of contour lines.

[0018] Fig. 2(B) is a diagram showing projection characteristics indicating the relationship between image height y, which indicates the height of an image formed by the optical system 10 of the imaging device 105, and half angle of view θ. In Fig. 2(B), the horizontal axis indicates half angle of view θ (the angle between the optical axis and the incident light ray), and the vertical axis indicates image height y on the light receiving surface (image plane) of the imaging element.

[0019] 2B, the optical system 10 of the image pickup device 105 is configured so that its projection characteristic y(θ) differs between an area less than a predetermined half angle of view θa and an area equal to or greater than the half angle of view θa. Therefore, the optical system 10 is configured so that, when the increase in image height y per unit half angle of view θ is defined as the resolution, the resolution differs depending on the area on the light receiving surface of the image pickup element.

[0020] This local resolution can be expressed as the differential value dy(θ) / dθ of the projection characteristic y(θ) at a half angle of view θ. For example, the greater the gradient of the projection characteristic y(θ) in Fig. 2(B), the higher the resolution. Also, the greater the interval between the image heights y at each angle of view on the contour lines in Fig. 2(A), the higher the resolution.

[0021] The optical system of this embodiment has projection characteristics in which the rate of increase of image height y (the slope of the projection characteristic y(θ) in FIG. 2(B)) is large in the central region near the optical axis, and the rate of increase of image height y becomes smaller in the peripheral region outside the central region as the angle of view becomes larger.

[0022] In the first embodiment, when the half angle of view θ is less than a predetermined half angle of view θa, a region closer to the center generated on the light receiving surface of the image sensor is called a high resolution region 10a, and a region closer to the center where the half angle of view θ is equal to or greater than the predetermined half angle of view θa is called a low resolution region 10b. That is, the optical system forms an optical image with a first resolution in a central region including the center of the light receiving surface of the image sensor, and forms an optical image with a second resolution lower than the first resolution in a peripheral region around the central region.

[0023] In the first embodiment, the high resolution region 10a is a low distortion region with relatively little distortion, and the low resolution region 10b is a high distortion region with relatively much distortion. Therefore, in the first embodiment, the high resolution region and the low resolution region are sometimes called the low distortion region and the high distortion region, respectively.

[0024] Note that the characteristics shown in Fig. 2(A) are just an example and are not limited to this. For example, the low-resolution area and the high-resolution area of ​​the optical system do not have to be configured as concentric circles, and each area may have a distorted shape.

[0025] The center of gravity of the low resolution region and the center of gravity of the high resolution region do not have to coincide with each other. The center of gravity of the low resolution region and the center of gravity of the high resolution region may be offset from the center of the light receiving surface of the image sensor. In the optical system of this embodiment, it is sufficient that the high resolution region is formed near the optical axis, and the low resolution region is formed on the peripheral side away from the optical axis, that is, outside the high resolution region.

[0026] The optical system 10 is configured so that the projection characteristic y(θ) in the high resolution region (low distortion region) 10a is greater than f×θ (f is the focal length of the optical system 10). Moreover, the projection characteristic y(θ) in the high resolution region (low distortion region) is set to be different from the projection characteristic in the low resolution region (high distortion region).

[0027] When θmax is the maximum half angle of view of the optical system 10, the ratio θa / θmax of θa to θmax is preferably equal to or greater than a predetermined lower limit, for example, 0.15 to 0.16, and the ratio θa / θmax of θa to θmax is preferably equal to or less than a predetermined upper limit, for example, 0.25 to 0.35.

[0028] For example, if θa is 90°, the predetermined lower limit is 0.15, and the predetermined upper limit is 0.35, then θa is desirably determined within the range of 13.5 to 31.5°. However, the above is just one example, and the present invention is not limited to this.

[0029] Furthermore, the optical system 10 is configured so that its projection characteristic y(θ) also satisfies the following formula 1.

number

[0030] Here, f is the focal length of the optical system 10 as described above, and A is a predetermined constant. In formula 1, by setting the lower limit to 1, it is possible to increase the central resolution compared to a fisheye lens of the orthogonal projection type (y=f×sinθ) having the same maximum imaging height. Also, by setting the upper limit to A in formula 1, it is possible to obtain an angle of view equivalent to that of a fisheye lens while maintaining good optical performance.

[0031] The predetermined constant A may be determined in consideration of the balance between the resolution of the high resolution region and the resolution of the low resolution region, and is preferably set to 1.4 to 1.9. However, the above is only one example, and the present invention is not limited to this.

[0032] By configuring the optical system 10 as described above, high resolution can be obtained in the high resolution region 10a, while the increase in image height y per unit half angle of view θ can be reduced in the low resolution region 10b, making it possible to capture a wider angle of view. Therefore, while the imaging range has a wide angle of view equivalent to that of a fisheye lens, high resolution can be obtained in the high resolution region 10a.

[0033] In the first embodiment, in the high-resolution region (low-distortion region), the projection characteristics are approximate to the central projection method (y=f×tanθ) and the equidistant projection method (y=f×θ), which are the projection characteristics of a normal imaging optical system. Therefore, in the high-resolution region (low-distortion region), the optical distortion is small, and it is possible to display a fine image.

[0034] Note that the first embodiment is not limited to the projection characteristic shown in FIG. 2, since the same effect can be obtained as long as the projection characteristic y(θ) satisfies the condition of the above-mentioned mathematical expression 1.

[0035] The advantage of obtaining the high-resolution region 10a with little distortion is that it enables display with less delay. If the displayed image has a large distortion, it becomes difficult to grasp the positional relationship of the objects shown on the screen, so it becomes necessary to perform distortion correction processing on the captured image.

[0036] When performing distortion correction, there are methods that can be considered, such as processing by hardware using an FPGA or processing by software using a CPU, but either method will result in delays.

[0037] In contrast, in this embodiment, distortion of the imaging area displayed on the image display unit 106 can be suppressed by the optical characteristics of the optical system 10, so distortion correction is not necessary, and display with lower delay is possible.

[0038] Fig. 3 is a functional block diagram showing a configuration example of an imaging system 107 including an imaging device 105 of this embodiment and a control unit 12. Note that some of the functional blocks shown in Fig. 3 are realized by causing a CPU or the like as a computer (not shown) included in the imaging system 107 to execute a computer program stored in a memory (not shown) as a storage medium.

[0039] However, a part or all of these may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0040] 3 may not be built in the same housing, and may be configured by separate devices connected to each other via signal paths. For example, at least a part of the control unit 12, some or all of the image display unit 106, warning unit 108, etc. may be provided in an external terminal or the like separate from the mobile object.

[0041] 3 is a system for displaying an image captured by an imaging device 105 installed on the upper part (torii) of a loading platform 102 of a transport vehicle 100 on an image display unit 106. The imaging system 107 also includes the imaging device 105, a control unit 12, an image display unit 106, a warning unit 108, a hydraulic pump 103, an angle detection sensor 104, and the like.

[0042] As described above, the imaging device 105 is installed on the vehicle bed to monitor the rear of the vehicle and the bed. Also, as described above, it has the optical system 10 and the imaging unit 11. The imaging unit 11 has an imaging element, and the optical system 10 has at least one lens, and guides light incident from the outside to the light receiving surface of the imaging element of the imaging unit 11 to form an optical image. The optical characteristics of the optical system 10 have been described in detail above.

[0043] That is, it has optical properties that form a high-resolution optical image in the angle of view region around the optical axis (corresponding to high-resolution region 10a in Figure 2), and form a low-resolution optical image in the angle of view region around the optical axis (corresponding to low-resolution region 10b in Figure 2).

[0044] The imaging unit 11 includes an imaging element (image sensor) for converting an optical subject image formed by the optical system 10 into an electrical signal. The imaging element is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0045] The electrical signal converted by the imaging element is converted into a predetermined image signal within the imaging unit 11 and is output to the control unit 12 at the downstream side.

[0046] The control unit 12 includes, for example, a SOC (System On Chip) / FPGA (Field Programmable Gate Array), a CPU as a computer, and a memory as a storage medium. The CPU executes computer programs stored in the memory to perform various controls of the entire imaging system.

[0047] The control unit 12 has various interfaces for image input and output, and outputs images to the image display unit 106. Note that the imaging device 105 may perform some or all of the functions performed by the control unit 12.

[0048] The control unit 12 is connected to an imaging device 105 , an image display unit 106 , an angle detection sensor 104 , and an alarm unit 108 , and controls the image display unit 106 , the alarm unit 108 , and the hydraulic pump 103 according to the results of processing within the control unit 12 .

[0049] The image display unit 106 is for displaying at least one image of the side rear of the vehicle or the top of the loading platform, and displays an image from the control unit 12. As shown in Fig. 1, the image display unit 106 is provided in the driver's seat of the transport vehicle 100, but a monitor of, for example, a mobile terminal outside the transport vehicle 100 may be used, in which case the image signal from the control unit 12 is displayed using a communication method such as wireless.

[0050] The control unit 12 acquires an image signal from the imaging device 105, and also acquires the tilt angle of the loading platform 102 from the angle detection sensor 104. The development processing unit 13 develops the image signal transmitted from the imaging unit 11 into an image, and performs processes such as WDR (Wide Dynamic Range) correction, gamma correction, LUT (Look Up Table) processing, and video signal extraction. It also performs various types of image processing such as distortion correction and object detection.

[0051] The distortion correction unit 15 performs distortion correction processing using a known technique to correct distortion of all or part of the input image. The distortion correction unit 15 also performs coordinate conversion to display the image input from the imaging unit on the rectangular screen of the display unit. If the image input from the imaging unit 11 has little distortion, the signal can be sent directly to the display image generation unit 17 without performing distortion correction.

[0052] The image signal processed by distortion correction unit 15 is input to object detection processing unit 16 and display image generation unit 17. By performing distortion correction in distortion correction unit 15, not only is the image easier to view when displayed on image display unit 106, but the detection rate of object detection processing performed inside object detection processing unit 16, which will be described later, is improved.

[0053] The object detection processing unit 16 performs object detection processing using the image signal output from the distortion correction unit 15, and determines whether or not an obstacle such as a vehicle or a person exists in the image. Deep learning is used for object detection. For example, it is preferable to use YOLO (You Only Look Once) as deep learning, which is easy to learn and has fast detection.

[0054] Other deep learning methods that may be used include SSD (Single Shot Multi Box Detector), Faster R-CNN (Regional Convolution Neural Network), Fast R-CNN, R-CNN, etc.

[0055] The object detection result includes four-point coordinate information, which is the coordinates of the four vertices of a rectangle (bounding box) indicating the position where the object is detected, and object name information indicating the classification of the detected object. Note that the object detection process by the object detection processing unit 16 may apply different learning results and detect different objects depending on the angle of view of the input image.

[0056] That is, the learning results of obstacles such as vehicles and people may be applied to the upper part of the angle of view, and the learning results of foreign objects such as cargo may be applied to the lower part of the angle of view. In this way, for example, a detection process for a cargo 603 on the loading platform 102 is executed, and a detection process for a falling cargo and dumping of the cargo is executed. In this way, by limiting the detection target according to the angle of view, it is possible to realize an accurate and high-speed detection process.

[0057] The display image generating unit 17 generates an image to be displayed on the image display unit 106. The display image generating unit 17 generates a display image by performing, for example, image cropping processing in accordance with the display resolution of the image display unit 106 based on the image signal received from the distortion correction unit 15. That is, the display image generating unit 17 outputs image data generated by the imaging device to the image display unit 106.

[0058] Also, multiple regions may be cut out from the image signal received from the distortion correction unit 15 and then synthesized to generate a single display image. For example, an image of the rear of the transport vehicle 100 and an image of the ground surface direction where soil is unloaded from the loading platform 102 may be cut out and then synthesized to generate a single display image.

[0059] Furthermore, the display image generation unit 17 receives the object detection result from the object detection processing unit 16 via the judgment processing unit 18, and synthesizes a warning image by superimposing the result on the above-mentioned display image. That is, a rectangle (bounding box) indicating the position where the object was detected is synthesized into the display image.

[0060] The determination processing unit 18 receives the object detection result from the object detection processing unit 16 and determines whether or not to stop the ascent and descent (tilt operation) of the loading platform 102. For example, the discharge status of soil and sand from the loading platform 102 may be detected, and as soon as the discharge is completed, the determination may be made to stop the ascent and descent (tilt operation) of the loading platform 102. Alternatively, when the detected discharge status approaches a predetermined level, the determination may be made to stop the ascent and descent (tilt operation) of the loading platform 102.

[0061] Also, it may be determined whether to stop or not depending on the classification of the detected object. For example, it may be determined to stop for safety when an obstacle such as a vehicle or a person is detected. When it is determined to stop, the determination processing unit 18 immediately sends a stop signal to the hydraulic pump 103, and the hydraulic pump 103 stops rising and falling.

[0062] The determination processing unit 18 also plays a role in sending a warning signal to the warning unit 108. This makes it possible to control the warning unit 108 to issue a warning when the object detection processing unit 16 detects an object.

[0063] The warning unit 108 includes a warning light such as a rotating light or a Patlite (registered trademark). The warning light may be configured with an LED, and the color emitted may be controlled depending on the type of warning. The warning unit 108 may also include a warning buzzer that issues a warning sound, and the sound emitted may be controlled depending on the type of warning.

[0064] For example, the warning unit 108 may turn on a red warning light when the object detection processing unit 16 detects a person or an obstacle in the high resolution region 10a, and may turn on a yellow warning light when the object detection processing unit 16 detects a person or an obstacle in the low resolution region 10b. In addition, if the warning unit 108 is a warning buzzer, the buzzer sound may be changed depending on the classification of the detected resolution region.

[0065] 4 is a diagram for explaining an example of a dump truck, which is an example of a transport vehicle in the first embodiment, and an example of an imaging range of the imaging device 105. Note that the imaging device 105 will be explained as an example installed on the upper part of the loading platform of the vehicle.

[0066] As shown in Fig. 4, in this embodiment, an imaging device 105 is installed on the upper part of the vehicle's loading platform facing the rear of the transport vehicle 100, thereby making it possible to capture an image of an area required for rear monitoring. In Fig. 4, 10a corresponds to the high-resolution area 10a in Fig. 2 and may be called the first area. Also, 10b corresponds to the low-resolution area 10b in Fig. 2 and may be called the second area.

[0067] 4, the imaging device 105 is disposed so as to be able to simultaneously capture images of a first area behind the vehicle and a second area including the loading platform. The imaging device 105 is disposed on the loading platform 102 so that an image of a blind spot area behind the vehicle that cannot be seen from the driver's seat of the transport vehicle 100 is formed in the first area.

[0068] In addition, the optical system 10 forms an optical image of the first region in a central region of the light receiving surface of the image sensor, and forms an optical image of the second region in a peripheral region of the light receiving surface of the image sensor. Therefore, the image sensor 105 can capture the first region with a higher resolution than the second region to generate image data.

[0069] Fig. 5 is a diagram illustrating an example of the platform movable area and the imaging range of the camera in embodiment 1. As shown in Fig. 5, the imaging device 105 is disposed so that the tilt angle 20a detected based on the extension amount of the hydraulic pump 103 is included in the high resolution area 10a.

[0070] In other words, the imaging device 105 is positioned so that the rear of the vehicle is included in the high resolution area 10a (first area) even if the loading platform 102 is raised or lowered (tilted) and the optical axis direction of the imaging device changes as a result.

[0071] Specifically, when the inclination angle 20a is, for example, a maximum inclination angle of 50° and the high resolution area 10a is ±60° from the optical axis, it is desirable to arrange the transport vehicle 100 so that the horizontal direction is always included within the high resolution area 10a. In other words, it is desirable for the optical axis of the camera to be about ±60°.

[0072] In this embodiment, the imaging device 105 is installed so that the viewing angle region of the high resolution region 10a described in Fig. 4 captures an image of the rear region of the transport vehicle 100. In addition, the imaging device 105 is installed so that the lower viewing angle region of the low resolution region 10b captures an image of the inside of the loading platform 102. Therefore, a single imaging device 105 can obtain images for monitoring the rear of the vehicle and images for monitoring the loading platform.

[0073] Therefore, a high-resolution and less-distorted image of the rear of the vehicle can be obtained by the image display unit 106. Note that, in order to adjust the field of view of the image displayed on the image display unit 106, the installation direction (optical axis direction) of the imaging device 105 may be appropriately adjusted.

[0074] In addition, in the imaging system 107 of this embodiment, a predetermined cropping process is performed on the image within the high-resolution area 10a described in FIG. 2 according to the inclination angle 20a of the loading platform 102, and the cropped image is output to the image display unit 106.

[0075] That is, the display image generating unit 17 generates a display image by cutting out a partial image area of ​​the image captured by the imaging device 105, and changes the image area to be cut out in accordance with the tilt angle of the loading platform .

[0076] Fig. 6 is a diagram showing an example of an image captured by the imaging device 105 of the embodiment 1. An image 601 shown in Fig. 6 is an image before the cropping process when the loading platform 102 is in the lowest position, i.e., when the inclination angle 20a is 0°, and a vehicle 602 behind the transporter vehicle 100 and a load 603 on the loading platform 102 are displayed in the image 601.

[0077] In this way, the imaging device 105 of this embodiment can include the rear of the transport vehicle 100 and the loading platform 102 in one angle of view, as in the image 601. In addition, in Fig. 6, 604 indicates a virtual boundary between the angle of view of the high resolution region 10a and the angle of view of the low resolution region 10b.

[0078] The virtual boundary line 604 is not displayed on the screen of the image display unit 106, but may be selectively displayed. The inside of the boundary line 604 is the field of view of the high resolution region 10a, and the outside of the boundary line 604 is the field of view of the low resolution region 10b.

[0079] Fig. 7 is a diagram showing an example of a cut-out image displayed by the image display unit 106 of the first embodiment. The cut-out image shown in Fig. 7 is a horizontal rectangular image of the rear of the transport vehicle 100 from the image 601, and is a rectangular image of the rear of the vehicle output to the image display unit 106 for the rear-view mirror when viewed from the driver. In the example of Fig. 7, a vehicle 602 following the transport vehicle 100 is displayed.

[0080] Although the vehicle 602 is distant from the transport vehicle 100, it can be clearly imaged with high resolution because it is present in the high resolution area 10a of the imaging device 105. In this embodiment, the load 603 present on the platform 102 is present in the lower part of the angle of view of the low resolution area 10b of the imaging device 105, but since it is close, the load 603 can also be clearly imaged.

[0081] In this manner, in this embodiment, the imaging device 105 is placed on the upper part of the front wall of the loading platform 102 of the transport vehicle 100, facing the rear of the transport vehicle 100, thereby enabling a single unit to both monitor the rear and the loading platform.

[0082] 8 is a diagram showing an example of an image captured when the loading platform 102 is in an elevated state in embodiment 1. An image 801 is an image captured when the loading platform 102 is in an elevated state (inclined state), and is an image before cropping processing when the inclination angle 20a is, for example, 50°.

[0083] 5, the imaging device 105 is disposed so that the horizontal direction of the transport vehicle 100 is always included within the high resolution area 10a. Therefore, even in an image 801 in which the platform 102 is raised, the vehicle 602 and the cargo 603 can be included within one angle of view without adjusting the installation direction (optical axis direction) of the imaging device 105. However, the rear vehicle 602 is shifted upward within the high resolution area 10a.

[0084] 9 is a diagram showing an example of an image cut out from an image 801 in which the loading platform 102 is in a raised state, and processing is performed to cut out the upper part of the high resolution area 10a in accordance with the inclination angle 20a. Therefore, the rear vehicle 602 and the like can be clearly photographed with high resolution, and the cut-out image can be displayed on the image display unit 106 for the rear-view mirror.

[0085] In this example, there is a load 603 on the loading platform 102, and although it is located at the bottom of the angle of view of the low-resolution area 10b of the imaging device 105, the load 603 can also be clearly imaged because it is close.

[0086] In this manner, in the first embodiment, the imaging device 105, which has a high resolution at the center of the angle of view, is disposed on the upper portion of the front wall of the loading platform 102 of the transport vehicle 100, facing the rear of the transport vehicle 100, and the rear image is cut out according to the angle of the loading platform, thereby making it possible to monitor the rear with an appropriate angle of view. Also, the loading platform can be monitored.

[0087] (Embodiment 2) In addition, as the second embodiment, a general wide-angle camera or a fish-eye camera may be used as the imaging device 105. In that case, an image with high resolution at the center of the angle of view as described in Fig. 4 cannot be obtained, and the resolution at the center of the angle of view is lower than that of the imaging device 105 described in this embodiment, but it is possible to recognize a subject to a certain degree.

[0088] (Embodiment 3) Next, an example of displaying an image and an image recognition result on the image display unit 106 in the third embodiment will be described. Fig. 10 is a diagram showing an example of a recognition result image displayed by the image display unit 106 in the third embodiment. The display image shown in Fig. 10 shows the rear of the transport vehicle 100, and an image of the load 603 on the loading platform 102 is included in the lower part of the screen.

[0089] These images are images captured by the imaging device 105, which have been subjected to various development processes by the development processing unit 13, and further subjected to image distortion correction by the distortion correction unit 15. Furthermore, the object detection results by the object detection processing unit 16 are also superimposed on the images, and the images are displayed as a composite image.

[0090] Specifically, when the object detection processing unit 16 detects that a part of the load has fallen, a falling object detection frame 603a is displayed as a warning on the right side of the screen, as shown in Fig. 10. The warning includes displaying predetermined characters or a frame on the image display unit 106. By displaying such a warning, the operator in the cab can easily notice the falling of the load.

[0091] In addition, even if the transport vehicle 100 is an automatic or remotely operated vehicle, the operator can notice the falling of the load by displaying an image such as that shown in Fig. 10 on the terminal of the operator at a remote location. Therefore, the safety of the transport vehicle 100 can be prevented from being deteriorated due to the falling of the load.

[0092] Fig. 11 is a flowchart showing an example of processing by the imaging device of the first embodiment, and Fig. 12 is a flowchart continuing from Fig. 11. A CPU (not shown) in the control unit 12 calls up a program corresponding to the processing content from a ROM, loads it into a RAM, and executes this flowchart to perform processing of each step in Figs. 11 and 12.

[0093] While the power supply of the transport vehicle 100 is on or while the transport vehicle 100 is traveling, the processing of the flowcharts shown in FIGS. 11 and 12 is repeatedly executed, for example, periodically.

[0094] First, in step S101, the CPU of the control unit 12 acquires imaging data by controlling the aperture (not shown) of the optical system 10 of the imaging device 105, which is a rear camera of the transport vehicle 100, the accumulation time of the imaging unit 11, etc. The imaging data includes the rear of the transport vehicle 100 and the loading platform 102 as shown in FIG.

[0095] Next, in step S102, the CPU of the control unit 12 controls the development processing unit 13, the distortion correction unit 15, and the display image generation unit 17 to generate a display image from the above-mentioned imaging data and display it on the image display unit 106. As described above, this is display data combining the rear image of the transport vehicle 100 and the image of the loading platform 102. Note that the display image is displayed as, for example, a moving image.

[0096] In step S103, the CPU of the control unit 12 determines whether the loading platform 102 has risen (inclined) using the tilt detection processing unit 14 based on the output of the angle detection sensor 104. If NO is determined, the process proceeds to step S104, and if YES is determined, the process proceeds to step S110.

[0097] Next, steps S104 to S109 and steps S110 to S122 are executed exclusively by the CPU of the control unit 12, as shown in Fig. 11. First, steps S104 to S109 will be described.

[0098] First, in step S104, the CPU of the control unit 12 controls the development processing unit 13, the distortion correction unit 15, and the display image generation unit 17 to perform cropping processing on the image region below the horizontal angle of view of the image data captured by the imaging device 105. The image cropped here is a rectangular image with an angle of view equivalent to that of a rearview mirror as shown in FIG. 7, for example, and is an image in a high-resolution region.

[0099] Next, in step S105, the CPU of the control unit 12 outputs the cut-out image of the high-resolution area to the image display unit 106 for display. Next, in step S106, the CPU of the control unit 12 controls the object detection processing unit 16 to execute object detection processing (image recognition) for the entire image area below the horizontal of the angle of view of the image data captured by the imaging device 105. Note that the image area subjected to the object detection processing at this time also includes the image of the angle of view of the loading platform reflected in the low-resolution area.

[0100] Incidentally, the learning data applied when performing the object detection process in step S106 may be limited to a predetermined load, so that only the load on the loading platform 102 is detected.

[0101] Next, in step S107, the CPU of the control unit 12 determines whether or not a drop of an object (such as a load) was detected in step S106. If a drop of an object (such as a load) was not detected, the flow charts of Figures 11 and 12 end, and if a drop of an object (such as a load) was detected, the flow proceeds to step S108.

[0102] Here, step S107 functions as a determination step (determination section) that issues a warning when a fall of an object loaded on the loading platform is detected.

[0103] Next, in step S108, the CPU of the control unit 12 superimposes information about the detected load on the display image displayed in step S105. Specifically, the CPU calculates the detected position of the load on the image based on the detection result by the object detection processing unit 16, and adds a load detection frame to the image. As a result, a falling object detection frame 603a is displayed as a warning on the right side of the screen, as shown in FIG. 10. As described above, the warning includes the display of a predetermined character or frame by the image display unit.

[0104] Next, in step S109, the CPU of the control unit 12 controls the judgment processing unit 18 to issue a warning using the warning unit 108. For example, when a drop of a loaded object or the like is detected, a red warning light is turned on. Alternatively, a warning sound may be issued. That is, the warning includes the issuance of a predetermined color or a predetermined sound.

[0105] This makes it possible to notify not only the operator of the transport vehicle 100 but also people around the transport vehicle 100 of the detection of a dropped load or the like. After the process of step S109, the flow of Fig. 11 and Fig. 12 is terminated. However, as described above, the flow of Fig. 11 and Fig. 12 is repeatedly executed, for example, periodically.

[0106] Next, steps S110 to S122 will be described. If the determination in step S103 is YES, in step S110, the CPU of the control unit 12 controls the development processing unit 13, the distortion correction unit 15, and the display image generation unit 17 to perform processing to cut out high-resolution imaging data according to the inclination angle of the loading platform 102.

[0107] That is, in step S110, the CPU of the control unit 12 cuts out a part of the high resolution area 10a from the image data captured by the imaging device 105 based on the tilt angle of the platform 102 detected by the tilt detection processing unit 14 in step S103.

[0108] Specifically, as the inclination angle of the loading platform 102 increases, the image of the rear vehicle 602 and the like shifts toward the top of the screen as shown in FIG. 8, so the image of the upper part of the high resolution area 10a is cut out.

[0109] Next, in step S111, the CPU of the control unit 12 outputs the cut-out image to the image display unit. Therefore, even if the loading platform 102 tilts, the image cut out from the image area shifted upward (the image area corresponding to the rear vehicle or the like) can be displayed on the image display unit 106 for the rearview mirror.

[0110] Next, in step S112, the CPU of the control unit 12 controls the object detection processing unit 16 to execute object detection processing on image data including a low-resolution region captured by the imaging device 105. At that time, the learning data applied to the object detection processing is limited to predetermined objects such as people and predetermined objects, thereby executing detection of only approaching objects (people, etc.).

[0111] Next, in step S113, the CPU of the control unit 12 determines that the approach of an object (person, etc.) was detected in step S112. If the approach of an object (person, etc.) is not detected, the process proceeds to step S114, and if the approach of an object (person, etc.) is detected, the process proceeds to step S120. Here, step S113 functions as a determination step (determination unit) that issues a warning when a predetermined object approaches the loading platform while the loading platform is tilted.

[0112] In step S114, the CPU of the control unit 12 controls the object detection processing unit 16 to execute object detection processing on the high-resolution area of ​​the image data captured by the imaging device 105. At that time, the learning data applied to the object detection processing is limited to objects such as people, objects, etc., so that only approaching objects (people, etc.) are detected.

[0113] Next, in step S115, the CPU of the control unit 12 determines whether or not the approach of an object (person, etc.) was detected in step S114. If the approach of an object (person, etc.) was not detected, the process proceeds to step S116, and if the approach of an object (person, etc.) was detected, the process proceeds to step S120. Here, step S115 functions as a determination step (determination unit) that issues a warning when a predetermined object approaches the loading platform while the loading platform is tilted.

[0114] Next, in step S116, the CPU of the control unit 12 controls the object detection processing unit 16 to execute object detection processing on the low-resolution area of ​​the image data captured by the imaging device 105. At that time, by limiting the learning data applied to the object detection processing to predetermined objects such as soil and sand, detection of only the load (soil and sand) unloaded from the loading platform 102 is executed.

[0115] Next, in step S117, the CPU of the control unit 12 determines whether or not completion of unloading of the load (such as earth and sand) has been detected in step S116. If completion of unloading has been detected, that is, if no load (such as earth and sand) remaining on the loading platform 102 has been detected, the process proceeds to step S118.

[0116] In step S117, if the completion of unloading is not detected, that is, if the load (such as earth and sand) to be unloaded remaining on the loading platform 102 is detected, the process returns to step S112, and the processes from step S112 to step S117 are repeated. In other words, the process is repeated until the completion of unloading is detected in step S117, that is, until the load (such as earth and sand) remaining on the loading platform 102 is no longer detected.

[0117] In step S118, the CPU of the control unit 12 superimposes information about the load (earth and sand) to be discharged on the image of the image display unit 106. Specifically, the CPU superimposes text, marks, etc. on the image displayed on the image display unit 106 to indicate that discharge has been completed, i.e., that the load (earth and sand) to be discharged has disappeared from the loading platform 102.

[0118] Next, in step S119, the CPU of the control unit 12 controls the judgment processing unit 18 and issues (notifies) a warning regarding the progress of the work using the warning unit 108. For example, the CPU notifies that the dumping is completed, that is, that there is no more load (such as earth and sand) to be dumped, by flashing a red warning light. This makes it possible to notify not only the operator of the transport vehicle 100 but also those around the transport vehicle 100 that there is no more load (such as earth and sand) to be dumped.

[0119] In step S120, the CPU of the control unit 12 superimposes information about the object (person, etc.) on the display image. Specifically, based on the detection result by the object detection processing unit 16, the CPU calculates the detection position of the object (person, etc.) on the image, and adds an object (person, etc.) detection frame to the image. As a result, when an object (person, etc.) approaches the image displayed on the image display unit 106, for example, a blinking red detection frame is superimposed.

[0120] Next, in step S121, the CPU of the control unit 12 controls the judgment processing unit 18 to issue a safety warning using the warning unit 108. For example, when the approach of an object (person, etc.) is detected, a red warning light is turned on. This makes it possible to notify not only the operator of the transport vehicle 100 but also those around the transport vehicle 100 of the approach of an object (person, etc.).

[0121] Next, in step S122, the CPU of the control unit 12 controls the determination processing unit 18 to send a stop signal to the hydraulic pump 103 to stop the lifting of the platform 102. Incidentally, the lowering of the platform 102 may be started in conjunction with this.

[0122] Thus, according to this embodiment, even if the operator of the transport vehicle 100 is performing work such as unloading soil and does not notice an object (person, etc.) approaching the loading platform 102, a warning of the approach of the object (person, etc.) can be displayed on the screen or a warning light can be used to warn the operator.

[0123] Furthermore, according to the above embodiment, the operator of the transport vehicle 100 can monitor both the rear and the loading platform based on the contents displayed on the image display unit 106 and the detection results of the object detection processing unit 16, by simply using one imaging device.

[0124] In the above-described embodiment, the optical characteristics of the optical system 10 of the imaging device 105 cause differences in resolution, such as the low-resolution region 10b and the high-resolution region 10a, in the image output from the imaging device, but this is not limited to this.

[0125] For example, the resolution may be different depending on the pixel density of the imaging unit 11 of the imaging device 105. Specifically, the pixel density of the sensor may be made sparser toward the periphery of the angle of view, so that the resolution of the periphery of the angle of view is relatively lower than that of the center of the angle of view.

[0126] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to the above-mentioned embodiments, and various modifications and combinations of the above-mentioned embodiments are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The above-mentioned embodiments include the following combinations.

[0127] (Configuration 1) An imaging system comprising: an imaging device disposed in a vehicle having a cargo bed; and a display image generation unit that outputs image data generated by the imaging device to an image display unit, wherein the imaging device is disposed so as to be able to simultaneously image a first area behind the vehicle and a second area including the cargo bed.

[0128] (Configuration 2) The imaging system according to configuration 1, wherein the imaging device images the first region at a higher resolution than the second region to generate the image data.

[0129] (Configuration 3) The imaging system described in Configuration 2, wherein the imaging device has an imaging element and an optical system that forms an optical image on the light receiving surface of the imaging element, the optical system forms an optical image of a first resolution in a central region including the center of the light receiving surface of the imaging element, and forms an optical image of a second resolution lower than the first resolution in a peripheral region around the central region, and the optical system forms an optical image of the first region in the central region and forms an optical image of the second region in the peripheral region.

[0130] (Configuration 4) The imaging system described in any one of configurations 1 to 3, characterized in that the imaging device is positioned so that the first area includes the rear of the vehicle even when the loading platform moves.

[0131] (Configuration 5) The imaging system described in any one of configurations 1 to 4, characterized in that the imaging device is arranged on the loading platform so that an image of a blind spot area behind the vehicle that cannot be seen from the driver's seat of the vehicle is formed in the first area.

[0132] (Configuration 6) An imaging system described in any one of configurations 1 to 5, characterized in having a judgment unit that issues a warning when a fall of an object loaded on the loading platform is detected or when a specified object approaches the loading platform while the loading platform is tilted.

[0133] (Configuration 7) The imaging system according to configuration 6, wherein the warning includes emission of a predetermined color or a predetermined sound.

[0134] (Configuration 8) The imaging system according to configuration 6 or 7, wherein the warning includes display of a predetermined character or frame by the image display unit.

[0135] (Configuration 9) The imaging system described in any one of configurations 1 to 8, characterized in that the imaging device is installed on the loading platform and is installed so as to be able to image the area behind the vehicle, and the first area is arranged to include the area behind the vehicle even if the loading platform tilts and the optical axis direction of the imaging device changes.

[0136] (Configuration 10) An imaging system described in any one of configurations 1 to 9, characterized in that the display image generation unit generates a display image by cutting out a partial image area of ​​an image captured by the imaging device, and the display image generation unit changes the image area in accordance with the inclination angle of the loading platform.

[0137] (Configuration 11) A vehicle, characterized in that the imaging device of the imaging system according to any one of configurations 1 to 9 is disposed on the loading platform, and the imaging device is capable of imaging the rear.

[0138] In order to realize a part or all of the control in the above-mentioned embodiment, a computer program for realizing the functions of the above-mentioned embodiment may be supplied to an imaging system or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the imaging system or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0139] 100: Transport vehicle 101: Running body 102: Cargo bed 103: Hydraulic pump 104: Angle detection sensor 105: Imaging device 106: Image display unit 107: Imaging system 108: Warning section 10:Optical system 11: Imaging section 12: Control unit 13: Development processing section 14: Tilt detection processing unit 15: Distortion correction section 16: Object detection processing unit 17: Display image generation unit 18: Judgment processing unit

Claims

1. An imaging device disposed in a vehicle having a loading platform; a display image generating unit that outputs the image data generated by the imaging device to an image display unit, 13. An imaging system, comprising: the imaging device disposed so as to be capable of simultaneously imaging a first area behind the vehicle and a second area including the cargo bed.

2. 2. The imaging system according to claim 1, wherein the imaging device captures the first region at a higher resolution than the second region to generate the image data.

3. the imaging device includes an imaging element and an optical system that forms an optical image on a light receiving surface of the imaging element; the optical system forms an optical image having a first resolution in a central region including a center of the light receiving surface of the image sensor, and forms an optical image having a second resolution lower than the first resolution in a peripheral region around the central region; the optical system forms an optical image of the first area in the central area and an optical image of the second area in the peripheral area; 3. The imaging system according to claim 2.

4. 2. The imaging system according to claim 1, wherein the imaging device is disposed so that the first area includes a rear area of ​​the vehicle even when a movement of the loading platform occurs.

5. 2. The imaging system according to claim 1, wherein the imaging device is disposed on the loading platform so that an image of a blind spot area behind the vehicle that cannot be seen from a driver's seat of the vehicle is formed in the first area.

6. 2. The imaging system according to claim 1, further comprising a judgment unit that issues a warning when a fall of an object loaded on the loading platform is detected or when a specified object approaches the loading platform while the loading platform is tilted.

7. 7. The imaging system according to claim 6, wherein the warning includes emission of a predetermined color or a predetermined sound.

8. 7. The imaging system according to claim 6, wherein the warning includes displaying a predetermined character or a frame by the image display unit.

9. the imaging device is installed on the loading platform and is installed so as to be able to image a rear area of ​​the vehicle, The imaging system according to claim 1, characterized in that the first area is arranged to include the rear of the vehicle even when the loading platform is tilted and the optical axis direction of the imaging device is changed.

10. The display image generating unit generates a display image by cutting out a partial image area of ​​an image captured by the imaging device, The imaging system according to claim 1 , wherein the display image generating unit changes the image area in accordance with an inclination angle of the loading platform.

11. 10. A vehicle, comprising: an imaging device of an imaging system according to claim 1, disposed on a loading platform; and capable of imaging a rear area by the imaging device.

Images