Imaging system

Abstract

To provide an imaging system that can simultaneously capture images of an aircraft in both the direction of travel and the direction of landing.SOLUTION: An imaging system (201) includes an imaging unit (102) arranged in an aircraft (101) and an output unit (202) that outputs image data acquired by the imaging unit to a display unit (215), and the imaging unit is arranged so as to be able to simultaneously capture images of a first imaging area in the direction of the aircraft's travel and a second imaging area in the direction of the aircraft's landing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging system that ensures safe flight by attaching a wide-angle camera to an aircraft and using the wide-angle camera to simultaneously capture images of the aircraft's flight direction and landing direction. [Background technology]

[0002] In recent years, demonstration experiments have been conducted for the commercial use of next-generation aircraft, such as flying cars (official name: electric vertical take-off and landing aircraft). To ensure safety during flight, it is expected that cameras will be installed outside the aircraft to capture images of the surrounding area.

[0003] Patent Document 1 discloses a control system for an unmanned aerial vehicle that achieves a safe landing by attaching a camera to the bottom of the drone, extracting the position of the landing point on the image from the captured image, and controlling so that the central angle of view of the camera coincides with the center of the landing point in the captured image. Patent Document 2 discloses a landing control method that provides a display device on a platform where an unmanned aerial vehicle takes off and lands to display landing environment information, captures the displayed information with an aircraft camera, and controls a safe landing based on the recognition result. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-138681 A [Patent Document 2] JP 2020-149640 A Summary of the Invention [Problem to be solved by the invention]

[0005] Both Patent Document 1 and Patent Document 2 aim to ensure safety during landing by capturing images of the landing site using a camera. However, to ensure safe flight of an aircraft, it is necessary to capture images not only during landing (in the landing direction) but also in the forward direction of flight.

[0006] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide an imaging system capable of simultaneously capturing images in the direction of travel and the direction of landing of an aircraft. [Means for solving the problem]

[0007] An imaging system as one aspect of the present invention has an imaging unit disposed on an aircraft and an output unit that outputs image data acquired by the imaging unit to a display unit, the imaging unit being disposed so as to be able to simultaneously image a first imaging area in the direction of travel of the aircraft and a second imaging area in the direction of landing of the aircraft.

[0008] Other objects and features of the present invention will be described in the following embodiments. Effect of the Invention

[0009] According to the present invention, it is possible to provide an imaging system capable of simultaneously capturing images of the flight direction and landing direction of an aircraft. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is an explanatory diagram of an aircraft in which an imaging unit is disposed in the first embodiment. [Diagram 2] FIG. 2 is a block diagram of an imaging system in each embodiment. [Diagram 3] 5 is a flowchart illustrating a control method of the imaging system in each embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a flight direction and a coordinate region in the first embodiment. [Diagram 5] FIG. 2 is an explanatory diagram of a cut-out image in the first embodiment. [Figure 6] FIG. 11 is an explanatory diagram of the characteristics of the optical portion in the second embodiment. [Figure 7] 13 is an explanatory diagram of resolution conversion using an imaging unit that captures an image at a central angle of view with low resolution and at a peripheral angle of view with high resolution in the second embodiment. FIG. [Figure 8]13 is an explanatory diagram of the characteristics of an optical portion in the third embodiment. FIG. [Figure 9] 13 is an explanatory diagram of resolution conversion using an imaging unit that captures images at a central angle of view with high resolution and at a peripheral angle of view with low resolution in a third embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In each drawing, the same reference numerals are used to refer to the same components, and duplicated explanations will be omitted.

[0012] <First embodiment> First, an imaging system according to a first embodiment will be described. The imaging system according to this embodiment has an imaging unit equipped with one wide-angle camera (described later), and the imaging unit is attached to the aircraft at a position that allows imaging of the flight direction and landing direction of the flying object. The imaging system also performs partial resolution conversion processing (described later) according to the flight angle of the flying object, thereby performing high-resolution imaging of the flight direction and landing direction with one camera, thereby achieving a safe flight.

[0013] FIG. 1 is an explanatory diagram of an air vehicle in which an imaging unit 102 is disposed in this embodiment, and shows an example of an installation position for attaching the imaging unit 102 to the air vehicle 101 and an imaging area 103 of the imaging unit 102. In this embodiment, the air vehicle 101 is exemplified as a flying car (aircraft). However, a flying car is an example of the air vehicle 101, and the air vehicle 101 encompasses all aircraft that fly in the sky and take off and land on the ground, such as airplanes, helicopters, and drones. In addition, the air vehicle 101 can be operated in any manner, such as manned or unmanned.

[0014] 1, in this embodiment, the imaging unit 102 is installed on the outside of the aircraft, at the front lower part, of the aircraft 101. The imaging unit 102 is preferably installed forward of the front wheel of the aircraft 101. In addition, the imaging unit 102 is preferably installed below a shock absorber (such as a bumper in a vehicle) that is installed in front of the aircraft.

[0015] The imaging unit 102 is a camera (wide-angle camera) capable of capturing an image of an arbitrary area with a wide angle of view. The imaging range (imaging area 103) of the imaging unit 102 is preferably 100 degrees to 180 degrees. By installing the imaging unit 102 at the position shown in FIG. 1, an imaging area 103 including the traveling direction to the landing direction of the aircraft 101 is captured. Here, the traveling direction is the forward direction as seen from the pilot's seat of the aircraft 101. Also, the landing direction is the direction in which the landing port, such as a vertiport, exists when the aircraft 101 is about to take off or land on the landing port. Therefore, when the aircraft 101 is about to land on the landing port, the imaging area 103 includes the traveling direction at that time and the landing port. However, as long as the imaging unit 102 can capture an image of the range of the imaging area 103, the mounting position of the imaging unit 102 may be a position other than the lower front outside the aircraft 101.

[0016] 2 is a block diagram of an image capturing system 201 according to this embodiment. The image capturing system 201 includes an image capturing unit 102, a development recognition unit 202, and an integration processing unit 203.

[0017] First, the outline of the process flow will be described. As described above, the imaging unit 102 captures an image of an arbitrary wide-angle area and acquires image data. The imaging unit 102 is connected to the development recognition unit 202. The imaging unit 102 transmits the acquired image data to the development recognition unit 202.

[0018] The development recognition unit 202 performs a predetermined development process (image processing) and image recognition process on the image data transmitted from the imaging unit 102. Specifically, the development recognition unit 202 performs development processing, resolution conversion processing, recognition processing, and the like. The development recognition unit 202 is connected to the integration processing unit 203. The development recognition unit 202 transmits the recognition result by the predetermined image recognition processing to the integration processing unit 203. The development recognition unit 202 is also connected to the display unit 215. The development recognition unit 202 performs a predetermined image processing on the image data acquired by the imaging unit 102, and then outputs the image data to the display unit 215 arranged at a position different from the imaging unit 102 (for example, a position visible to the passenger of the aircraft 101). In other words, the development recognition unit 202 functions as an output unit that outputs the image data acquired by the imaging unit 102 to the display unit 215.

[0019] The integrated processing unit 203 determines the dangers during the flight and landing of the flying object 101 based on the recognition results transmitted from the development recognition unit 202. The integrated processing unit 203 also detects information related to the flight mode and flight angle of the flying object 101 and sends the detected information to the development recognition unit 202.

[0020] Next, the configuration of each block of the imaging system 201 shown in Fig. 2 will be described in detail. The imaging unit 102 has an optical unit 204 and an image sensor (imaging element) 205. The optical unit 204 has a lens (optical system) and a lens barrel, and collects light incident from the outside with a wide angle of view. The optical unit 204 then forms an optical subject image and outputs the subject image to the image sensor 205.

[0021] The image sensor 205 has a microlens, a color filter, and a photodiode that is a light receiving element in each pixel (not shown). The image sensor 205 converts the subject image (optical image) formed by the optical unit 204 into an electrical signal (photoelectric conversion), converts it into predetermined image data, and transmits it to the development recognition unit 202 at the downstream.

[0022] Next, the development recognition unit 202 will be described. The development recognition unit 202 has an image processing unit 206, a high-resolution area calculation unit 207, a resolution conversion processing unit 208, a recognition area determination unit 209, and a recognition unit 210. The development recognition unit 202 also has a CPU (Central Processing Unit) that performs calculations and control, and a ROM (Read Only Memory) and a RAM (Random Access Memory) that are main storage devices. Basic setting data is stored in the ROM, and the CPU calls up a program corresponding to the processing content from the ROM and expands it in the RAM to execute the operation of each block. However, in this embodiment, the development recognition unit 202 is configured with a CPU, but is not limited to this. As a more practical configuration, each block may be implemented with a semiconductor integrated circuit such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The development recognition unit 202 performs development processing by the image processing unit 206, partial resolution conversion processing by the resolution conversion processing unit 208, and object recognition by the recognition unit 210 on image data transmitted from the imaging unit 102.

[0023] The image processing unit 206 performs image processing such as demosaic, WDR (Wide Dynamic Range) correction, gamma correction, and distortion correction on the image data transmitted from the image sensor 205. However, the processing performed by the image processing unit 206 is not limited to these and can be changed as necessary. The image data processed by the image processing unit 206 is input to a resolution conversion processing unit 208.

[0024] The high resolution area calculation unit 207 calculates an area in the image data where the resolution conversion process is performed based on the flight angle information of the flying object 101 input from the flight angle detection unit 213. Specifically, the high resolution area calculation unit 207 acquires information about the flight angle of the flying object 101 detected by the flight angle detection unit 213, and generates coordinate information indicating the coordinates in the image data where the resolution conversion process is performed based on the information about the flight angle. That is, the high resolution area calculation unit 207 determines an area to be converted into high resolution among the imaging area of ​​the imaging unit 102 according to the flight angle. The coordinate information calculated by the high resolution area calculation unit 207 is transmitted to the resolution conversion processing unit 208. The high resolution area calculation unit 207 may also determine an area to be converted into high resolution based on flight mode information of the flying object 101 transmitted from the flight mode detection unit 212. Specifically, when the flight mode is a normal flight state (normal mode), the high resolution area calculation unit 207 determines the traveling direction of the flying object 101 as the area to be converted into high resolution. On the other hand, when the flight mode is the landing state (landing mode), the high resolution area calculation unit 207 determines the landing direction of the flying object 101 as the area to be subjected to high resolution conversion.

[0025] The resolution conversion processing unit 208 performs high-resolution processing on the coordinate area in the image data based on the coordinate information input from the high-resolution area calculation unit 207. That is, the resolution conversion processing unit 208 converts the area determined by the high-resolution area calculation unit 207 to high resolution, and converts the area excluding the area to low resolution to generate image data. The high-resolution processing is performed using a known technology such as super-resolution technology. However, this embodiment is not limited to this, and the high-resolution processing may be performed using other methods. The image data partially subjected to the high-resolution processing by the resolution conversion processing unit 208 is input to the recognition unit 210. Here, high resolution is a resolution at which the recognition unit 210 can recognize small objects or distant objects included in the image data. In this case, the resolution is preferably at a level at which the recognition unit 210 can recognize objects that may affect the flight of the flying object 101.

[0026] The recognition area determination unit 209 calculates an area for recognition based on flight mode information of the flying object 101 transmitted from the flight mode detection unit 212, and sends coordinate information to the recognition unit 210. That is, the recognition area determination unit 209 determines a recognition area in an image, which is an area for performing image recognition processing, based on information indicating the in-flight state of the flying object 101 transmitted from the flight mode detection unit 212, and sends information about the recognition area to the recognition unit 210. This makes it possible to reduce the amount of data and processing time required for the recognition processing by the recognition unit 210.

[0027] The recognition unit 210 performs a recognition process on the image data input from the resolution conversion processing unit 208 to detect the presence or absence of an object in the area determined by the recognition area determination unit 209. The recognition unit 210 recognizes the target object in the recognition area determined according to the flight state. For the recognition process, YOLO (You Only Look Once), which is a deep learning method that is easy to learn and has fast detection, is used, but other deep learning methods may also be used. The results of recognition by the recognition unit 210 (type of recognition target, coordinate information) are transmitted to the integration processing unit 203. In addition, the recognition process may be performed using other methods other than deep learning.

[0028] Next, the integrated processing unit 203 will be described. The integrated processing unit 203 has a danger determination unit 211, a flight mode detection unit 212, and a flight angle detection unit 213. The integrated processing unit 203 is, for example, an electronic control device such as an ECU (Electronic Control Unit) or a FADEC (Full Authority Digital Engine Control), and controls the flying object 101 in an integrated manner.

[0029] The danger determination unit 211 determines whether the flight or landing of the flying object 101 is dangerous based on the recognition information (information indicating the recognition result of the recognition unit 210) acquired from the recognition unit 210. The result determined by the danger determination unit 211 is transmitted to the warning unit 214. The flight mode detection unit 212 detects the flying state of the flying object 101 (whether it is in a normal flight state or a landing preparation state) and transmits information regarding the flight mode to the recognition area determination unit 209. The flight angle detection unit 213 detects the real-time flight angle of the flying object 101 and transmits the detection result to the high-resolution area calculation unit 207. The flight angle detection unit 213 acquires information regarding the flight angle of the flying object 101, for example, from an acceleration sensor or the like arranged at a predetermined position of the flying object 101.

[0030] The warning unit 214 issues a warning sound (alert) to the pilot and passengers of the flying object 101 when it is determined to be dangerous based on the determination result (recognition result) input from the danger determination unit 211. However, the warning method is not limited to the method of issuing a warning sound as described above, and may be a warning display on the display unit 215 such as a monitor screen in the pilot's seat. In addition, the flight control unit 216 may automatically control the flight of the flying object 101 so as to avoid danger. When the flying object 101 is an unmanned flight, such automatic control is effective. The flight control unit 216 controls the flying object 101 so as to avoid collision with the object based on the result recognized by the recognition unit 210.

[0031] Next, a process flow in this embodiment (a control method for the imaging system 201) will be described with reference to Fig. 3. Fig. 3 is a flowchart showing a control method for the imaging system 201. The process in Fig. 3 is executed by a control unit (not shown), such as a CPU of the development recognition unit 202, calling up a program corresponding to the processing content from a ROM and loading the program in a RAM. The process in Fig. 3 is also always executed in the imaging system 201.

[0032] First, in step S301, the imaging unit 102 captures an image of an arbitrary region with an appropriate exposure time. The appropriate exposure time is, for example, a value calculated as an optimal value as a result of the imaging unit 102 determining the brightness of the subject using an auto exposure function. In this embodiment, as described above, the imaging unit 102, which is a wide-angle camera, is attached to the flying object 101 at the position shown in FIG. 1, so that images of the forward traveling direction and the landing direction directly below are captured simultaneously as viewed from the flying object 101 (see the imaging area 103). Note that in this embodiment, it is assumed that the flying object 101 lands in a vertical direction. However, this embodiment is not limited to this, and can also be applied to flying objects that land in an oblique direction. In this case, the landing direction is an oblique direction.

[0033] Next, in step S302, the flight angle detection unit 213 detects the flight angle of the aircraft 101. Fig. 4(a) is an explanatory diagram showing the flight of the aircraft 101, and the flight angle of the aircraft 101 is 401 in Fig. 4(a). As shown in Fig. 4(a), the flight angle 401 is an angle θ in the pitch direction formed by the aircraft direction 403 of the aircraft 101 with respect to a horizontal reference 402. The order of steps S301 and S302 may be reversed, and these processes may be performed simultaneously.

[0034] Next, in step S303, the resolution conversion processing unit 208 increases the resolution of an arbitrary coordinate region of the image data. FIG. 4(b) is an explanatory diagram of the coordinate region. As shown in FIG. 4(b), the coordinate region is a region in the direction of travel (first imaging region) 404 and a region in the direction of landing (second imaging region) 405. A method of calculating the coordinate region to be increased in resolution will be described below. Note that FIG. 4(c) is a diagram showing the rooftop 406(a) of a building on which the flying object 101 lands and a landing port 406(b) arranged on the rooftop 406(a) of the building when the building is viewed from above.

[0035] First, the area 404 in the direction of travel is a predetermined angle of view area, for example, 180 degrees horizontally and 60 degrees vertically, in front of the pilot's seat of the flying object 101 regardless of the flight angle 401. However, this angle of view area is an example and can be changed as appropriate. Next, the area 405 in the landing direction is calculated by the high-resolution area calculation unit 207 based on the flight angle 401. Here, the area 405 in the landing direction is an area including a landing port 406(b) arranged on the roof of a building 406(a) or the like in the imaging area when the flying object 101 is in the landing posture. The area 405 in the landing direction is an angle of view area of ​​180 degrees horizontally and 60 degrees vertically, starting from a high-resolution area boundary 408 in the landing direction, which is obtained by shifting the angle of view in the pitch upward by the angle of the flight angle 401 with respect to an imaging limit 407 in the downward direction of the angle of view in the imaging area 103. However, this angle of view area is an example and can be changed as appropriate.

[0036] In this embodiment, the control in the pitch direction is described, but the flight angle detection unit 213 may acquire the amount of change in angle in the roll direction or yaw direction, and the high resolution area calculation unit 207 may adjust and control the landing direction area 405. In addition, when the flying object 101 lands in an oblique direction, a rotation unit 218 that rotates the imaging unit 102 in a hardware (physically) manner and a rotation control unit 217 that controls the rotation angle of the rotation unit 218 may be provided to change the landing direction area 405. The rotation unit 218 can rotate the imaging unit 102 to change the imaging area of ​​the imaging unit 102 according to the flight angle detected by the flight angle detection unit 213. In addition, the high resolution area calculation unit 207 can adjust and control the landing direction area 405 based on the relative position between the landing port 406(b) acquired from image recognition processing of image data, etc., and the flying object 101. In addition, the rotation unit 218 may rotate the imaging unit 102 based on flight mode information of the flying object 101 transmitted from the flight mode detection unit 212.

[0037] Next, the resolution conversion processing unit 208 performs a high-resolution process on the image data so that the areas (area 404 in the direction of travel and area 405 in the direction of landing) calculated by the high-resolution area calculation unit 207 have high resolution. This enables the recognition unit 210 to recognize small objects and distant objects that exist in the direction of travel and the direction of landing of the flying object 101 regardless of the flight angle 401.

[0038] Fig. 5 is an explanatory diagram of a cut-out image 501 obtained by cutting out a part of image data that has been subjected to high-resolution processing. However, it is assumed that the cut-out image 501 has been subjected to distortion correction processing by the image processing unit 206. Fig. 5 shows in a mesh form how the area including the landing port 406(b) has been made high-resolution by making the area 405 in the landing direction high-resolution.

[0039] Returning to FIG. 3, the flight mode detection unit 212 subsequently detects the flight mode of the flying object 101 in step S304. That is, in step S304, the flight mode detection unit 212 determines whether the flying object 101 is in a normal flight state or a landing state (landing preparation state). Then, the recognition area determination unit 209 determines an area in the imaging area 103 where recognition processing of the object is performed. If it is determined that the flight mode is in a normal flight state, the recognition area determination unit 209 sends coordinate information of the area 404 in the traveling direction to the recognition unit 210, and proceeds to step S305. On the other hand, if it is determined that the flight mode is in a landing state, the recognition area determination unit 209 sends coordinate information of the area 405 in the landing direction to the recognition unit 210, and proceeds to step S306.

[0040] In step S305, since it is determined in step S304 that the flight state is normal, the recognition unit 210 performs object recognition processing for the area 404 in the traveling direction in the captured image. In step S306, since it is determined in step S304 that the aircraft is in the landing state, the recognition unit 210 performs object recognition processing on the area 405 in the landing direction in the captured image.

[0041] Next, in step S307, the risk determination unit 211 determines whether or not there is a risk of collision depending on the recognition result of the object in step S305 or step S306. That is, when an object (body) is recognized by the image recognition process in step S305 or step S306, the risk determination unit 211 determines whether or not there is a risk of collision between the recognized object and the aircraft 101 by the aircraft 101 continuing to fly. The risk determination unit 211 may determine whether or not there is an object that may hinder flight or landing, such as another aircraft, a bird, or an obstacle. If it is determined that there is a risk in step S307, the process proceeds to step S308. On the other hand, if it is determined that there is no risk, the process ends.

[0042] In step S308, the warning unit 214 issues a warning to the passengers (pilot, etc.) of the flying object 101, and urges them to take action to avoid a collision with the object. As described above, the warning method performed by the warning unit 214 includes a voice warning and a warning display on the display unit 215 such as a monitor screen in the pilot's seat. In addition, in a case where the flying object 101 is performing unmanned operation, in step S308, instead of the above-mentioned warning process, the flight control unit 216 may automatically control the flying object 101 to perform an operation to avoid a collision with the object. Specifically, the flight control unit 216 controls the flying object 101 so as to avoid a collision with the object based on the result of recognition by the recognition unit 210.

[0043] As described above, in this embodiment, the imaging unit 102, which is a wide-angle camera, is attached at a position where it can simultaneously capture images of the traveling direction and landing direction of the flying object 101. The imaging system 201 increases the resolution of the traveling direction and landing direction according to the flight angle 401, thereby performing high-resolution imaging of the traveling direction and landing direction using the imaging unit 102, which is a single camera, and thus achieving a safe flight.

[0044] <Second embodiment> Next, an imaging system according to a second embodiment will be described. In this embodiment, an imaging unit 102 with a peripheral angle of view of high resolution and a central angle of view of low resolution is attached at a position where high-resolution imaging can be performed in the direction of flight and the direction of landing of the flying object 101. The imaging system of this embodiment variably controls the area in which high-resolution imaging is performed according to the flight angle 401, thereby performing high-resolution imaging in the direction of flight and the direction of landing using the imaging unit 102, which is a single camera, and achieving safe flight.

[0045] The basic configuration of the imaging system of this embodiment is similar to that of the imaging system 201 described with reference to Fig. 2. However, the optical section 204 of this embodiment has a lens (optical system) capable of forming an optical image on the light receiving surface of the image sensor 205, in which the central angle of view (first angle of view) of the imaging region has a low resolution (first resolution) and the peripheral angle of view (second angle of view) of the imaging region has a high resolution (second resolution).

[0046] That is, the optical unit 204 has an optical system having different imaging magnifications at a first angle of view (first field of view) and a second angle of view (second field of view) that is closer to the periphery than the first angle of view. The imaging surface (light receiving surface) of the image sensor 205 includes a first region that images an object included in the first angle of view, and a second region that images an object included in the second angle of view. At this time, the number of pixels per unit angle of view in the second region is greater than the number of pixels per unit angle of view in the first region. In other words, the resolution (second resolution) at the second angle of view (second region) of the imaging unit 102 is higher than the resolution (first resolution) at the first angle of view (first region).

[0047] The optical characteristics of the optical system in the optical unit 204 in this embodiment will be described with reference to Fig. 6(a) and (b). Fig. 6(a) is a diagram showing the image height y at each half angle of view on the imaging surface (light receiving surface) of the image sensor 205 in the form of contour lines. Fig. 6(b) is a diagram showing the relationship between the half angle of view θ and the image height y (projection characteristics of the optical unit 204) in the first quadrant of Fig. 6(a).

[0048] As shown in FIG. 6B, the optical unit 204 is configured such that its projection characteristic y(θ) differs between angles of view less than a predetermined half angle of view θa and angles of view equal to or greater than the half angle of view θa. Therefore, the optical unit 204 is configured such that, when the increase in image height y per unit of half angle of view θ is taken as the resolution, the resolution differs depending on the angle of view (area on the light receiving surface of the image sensor). This local resolution can be expressed as a differential value dy(θ) / dθ of the projection characteristic y(θ) at the half angle of view θ. For example, the greater the inclination of the projection characteristic y(θ) in FIG. 6B, the higher the resolution. Also, in FIG. 6A, the greater the interval between the contour lines of the image height y at each half angle of view, the higher the resolution.

[0049] The optical section 204 of this embodiment has a projection characteristic in which the rate of increase of image height y (the slope of the projection characteristic y(θ) in FIG. 6(b)) is small in the central region near the optical axis, and the rate of increase of image height y becomes larger as the angle of view becomes larger in the peripheral region outside the central region.

[0050] In Fig. 6(a), a first region 601a including the center corresponds to an angle of view less than a half angle of view θa, and a second region 601b outside the first region 601a corresponds to an angle of view equal to or greater than the half angle of view θa. The angle of view less than the half angle of view θa corresponds to a region excluding regions 404 and 405 in Fig. 7(a) described below (the central angle of view of the imaging region), and the angle of view equal to or greater than the half angle of view θa corresponds to regions 404 and 405 (the peripheral angle of view of the imaging region (the first imaging region and the second imaging region)).

[0051] As described above, the first region 601a is a relatively low-resolution region, and the second region 601b is a relatively high-resolution region. Also, the first region 601a is a high-distortion region with relatively more distortion, and the second region 601b is a low-distortion region with relatively less distortion. Therefore, in this embodiment, the first region 601a may be called a low-resolution region or a high-distortion region, and the second region 601b may be called a high-resolution region or a low-distortion region.

[0052] The characteristics shown in Figs. 6(a) and (b) are merely examples, and are not limited thereto. For example, the low-resolution region and the high-resolution region of the optical system may not be configured in a concentric shape, and each region may have a distorted shape. The center of gravity of the low-resolution region and the center of gravity of the high-resolution region may not coincide. The center of gravity of the low-resolution region and the center of gravity of the high-resolution region may be shifted from the center of the light receiving surface of the image sensor. In the optical system of this embodiment, it is sufficient that the low-resolution region is formed near the optical axis, and the high-resolution region is formed on the peripheral side near the optical axis.

[0053] The optical unit (optical system) 204 is configured to satisfy the following formula (1), where f is the focal length, θ is the half angle of view, y is the image height on the image plane, y(θ) is a projection characteristic that indicates the relationship between the image height y and the half angle of view θ, and θmax is the maximum half angle of view that the optical system has. That is, the optical unit 204 is configured so that the projection characteristic y(θ) is different from 2ftan(θ / 2) (stereoscopic projection method).

[0054] 0.2<2×f×tan(θmax / 2) / y(θmax) <0.92 ···(1) In an optical system with such optical characteristics, the magnification in the radial direction with respect to the optical axis can be adjusted by adjusting the projection characteristic y(θ). This allows the aspect ratio in the radial direction and circumferential direction with respect to the optical axis to be controlled, so that, unlike conventional fisheye lenses, it is possible to obtain a high-resolution image with little distortion in the peripheral area even with a wide angle of view.

[0055] Moreover, by satisfying formula (1), the resolution in the second region 601b can be made higher than that of a stereoscopic projection optical system. If the upper limit of formula (1) is exceeded, the resolution in the second region 601b decreases, and the difference with the resolution in the first region 601a becomes small, which is not preferable. If the lower limit of formula (1) is exceeded, it becomes difficult to satisfactorily correct various aberrations such as field curvature, which is also not preferable.

[0056] It should be noted that the above number (1) is just an example, and the optical system in this embodiment is not limited to this.

[0057] By configuring the optical system as described above, high resolution can be obtained in the high resolution region, while the increase in image height y per unit half angle of view θ can be reduced in the low resolution region, making it possible to capture a wider angle of view. Therefore, while providing an imaging range with a wide angle of view equivalent to that of a fisheye lens, high resolution can be obtained in the high resolution region.

[0058] In this embodiment, the projection characteristics in the high-resolution region (low-distortion region) are approximate to the central projection method (y=f×tanθ) and equidistant projection method (y=f×θ), which are the projection characteristics of a general imaging optical system. Therefore, in the high-resolution region (low-distortion region), optical distortion is small, making it possible to generate a fine image.

[0059] Next, the flow of processing in this embodiment will be described with reference to Fig. 3 and Fig. 7(a) and (b). Fig. 7(a) and (b) are explanatory diagrams of resolution conversion using an imaging unit that captures a central angle of view with low resolution and a peripheral angle of view with high resolution. Note that in Fig. 3, explanations common to the first embodiment will be omitted.

[0060] First, in step S301 in Fig. 3, the imaging unit 102 captures an image of an arbitrary imaging region 103, as in the first embodiment. As shown in Fig. 7(a), the imaging unit 102 is attached to the front lower part of the flying object 101 at a position where it is possible to simultaneously capture an image of an area in the traveling direction (second area, second angle of view) 404 and an area in the landing direction (second area, second angle of view) 405 with high resolution (second resolution).

[0061] On the other hand, the imaging unit 102 of this embodiment captures an image at a low resolution (first resolution) for an area (first angle of view) excluding the area 404 in the direction of travel and the area 405 in the direction of landing from the imaging area 103. Note that in this embodiment, the imaging unit 102 may be attached to a position other than the front lower part of the flying object 101 as long as the range satisfies the above-mentioned imaging conditions (the area 404 in the direction of travel and the area 405 in the direction of landing can be simultaneously captured at high resolution).

[0062] In step S303, the resolution conversion processing unit 208 performs a process of increasing the resolution of the image data so that the area 405 directly below in the landing direction has a high resolution in accordance with the flight angle 401 shown in Fig. 7(b). This enables the imaging unit 102 to capture images of two areas, the area 404 in the direction of travel and the area 405 in the direction of landing, with high resolution regardless of the flight angle 401.

[0063] In this embodiment, in addition to the high-resolution processing, the resolution conversion processing unit 208 performs a low-resolution processing on the image data so that the area 601 previously captured at high resolution by the imaging unit 102 becomes low resolution. This makes it possible to suppress unnecessary bandwidth. Here, the low-resolution processing performed by the resolution conversion processing unit 208 may, for example, be a synthesis compression processing that averages a plurality of pixels including peripheral pixels for a pixel of interest and regards them as one pixel data. However, this embodiment is not limited to this, and other processing that partially reduces the resolution of image data may be adopted.

[0064] In this embodiment, the optical system (optical unit 204) forms an optical image with a first resolution in a central region including the center of the light receiving surface of the image sensor 205, and forms an optical image with a second resolution higher than the first resolution in a peripheral region around the central region. The optical system also forms optical images of the first imaging region (region 404 in the direction of travel) and the second imaging region (region 405 in the direction of landing) in the peripheral region.

[0065] As described above, in this embodiment, the imaging unit 102, which has a high resolution peripheral angle of view and a low resolution central angle of view, is attached at a position where high resolution imaging is possible in the traveling direction and landing direction of the flying object 101. The imaging system of this embodiment variably controls the area in which high resolution imaging is performed according to the flight angle 401, thereby performing high resolution imaging in the traveling direction and landing direction using the imaging unit 102, which is a single camera, and achieving safe flight.

[0066] <Third embodiment> Next, an imaging system according to a third embodiment will be described. In this embodiment, the imaging unit 102, which has a low resolution peripheral angle of view and a high resolution central angle of view, is attached to a position where high resolution imaging can be performed in the direction of flight or landing of the flying object 101. The imaging system of this embodiment variably controls the area in which high resolution imaging is performed according to the flight angle 401, thereby performing high resolution imaging in the direction of flight and landing using the imaging unit 102, which is a single camera, and achieving safe flight.

[0067] The basic configuration of the imaging system of this embodiment is similar to that of the imaging system 201 described with reference to Fig. 2. However, the optical section 204 of this embodiment has a lens (optical system) capable of forming an optical image on the light receiving surface of the image sensor 205, in which the central angle of view (first angle of view) of the imaging area is high resolution (first resolution) and the peripheral angle of view (second angle of view) of the imaging area is low resolution (second resolution).

[0068] That is, the optical unit 204 has an optical system having different imaging magnifications at a first angle of view (first field of view) and a second angle of view (second field of view) that is closer to the periphery than the first angle of view. The imaging surface (light receiving surface) of the image sensor 205 includes a first region that images an object included in the first angle of view, and a second region that images an object included in the second angle of view. At this time, the number of pixels per unit angle of view in the first region is greater than the number of pixels per unit angle of view in the second region. In other words, the resolution (first resolution) at the first angle of view (first region) of the imaging unit 102 is higher than the resolution (second resolution) at the second angle of view (second region).

[0069] The optical characteristics of the optical unit (optical system) 204 in this embodiment will be described with reference to Fig. 8(a) and (b). Fig. 8(a) is a diagram showing the image height y at each half angle of view on the imaging surface (light receiving surface) of the image sensor 205 in the form of contour lines. Fig. 8(b) is a diagram showing the relationship between the half angle of view θ and the image height y (projection characteristics of the optical unit 204) in the first quadrant of Fig. 8(a).

[0070] As shown in FIG. 8B, the optical unit 204 is configured such that its projection characteristic y(θ) differs between angles of view less than a predetermined half angle of view θa and angles of view equal to or greater than the half angle of view θa. Therefore, the optical unit 204 is configured such that, when the increase in image height y per unit of half angle of view θ is defined as the resolution, the resolution differs depending on the angle of view (area on the light receiving surface of the image sensor). This local resolution can be expressed as a differential value dy(θ) / dθ of the projection characteristic y(θ) at the half angle of view θ. For example, the greater the inclination of the projection characteristic y(θ) in FIG. 8B, the higher the resolution. Also, FIG. 8A shows that the greater the interval between the contour lines of the image height y at each half angle of view, the higher the resolution.

[0071] The optical section 204 of this embodiment has a projection characteristic in which the rate of increase of image height y (the slope of the projection characteristic y(θ) in FIG. 8(b)) is large in the central region near the optical axis, and the rate of increase of image height y becomes smaller in the peripheral region outside the central region as the angle of view becomes larger.

[0072] In Fig. 8(a), a first region 801a including the center corresponds to an angle of view less than a half angle of view θa, and a second region 801b outside the first region 801a corresponds to an angle of view equal to or greater than the half angle of view θa. The angle of view less than the half angle of view θa corresponds to the first angle of view, that is, a region (first imaging region) 404 in Fig. 9(a) and a region (second imaging region) 405 in Fig. 9(b) described later. The angle of view obtained by combining the angle of view less than the half angle of view θa and the angle of view equal to or greater than the half angle of view θa corresponds to the second angle of view, that is, the region of the imaging region 103 in Fig. 9(a) excluding the region 404, or the region of the imaging region 103 in Fig. 9(b) excluding the region 405.

[0073] As described above, the first region 801a is a relatively high-resolution region, and the second region 801b is a relatively low-resolution region. Also, the first region 801a is a low-distortion region with relatively little distortion, and the second region 801b is a high-distortion region with relatively much distortion. Therefore, in this embodiment, the first region 801a may be called a high-resolution region or a low-distortion region, and the second region 801b may be called a low-resolution region or a high-distortion region.

[0074] The characteristics shown in Figures 8(a) and (b) are merely examples, and are not limited thereto. For example, the low-resolution region and the high-resolution region of the optical system may not be configured in a concentric shape, and each region may have a distorted shape. In addition, the center of gravity of the low-resolution region and the center of gravity of the high-resolution region may not coincide. In addition, the center of gravity of the low-resolution region and the center of gravity of the high-resolution region may be shifted from the center of the light receiving surface of the image sensor.

[0075] In the optical system of this embodiment, it is sufficient that the high resolution region is formed in the vicinity of the optical axis, and the low resolution region is formed on the peripheral side in the vicinity of the optical axis.

[0076] The optical unit 204 is configured so that the projection characteristic y(θ) in the first region 801a (high resolution region) is greater than f×θ (f is the focal length of the optical unit 204). In addition, the projection characteristic y(θ) in the first region 801a (high resolution region) is set to be different from the projection characteristic in the second region 801b (low resolution region).

[0077] When θmax is the maximum half angle of view of the optical unit 204, it is desirable that the ratio θa / θmax of θa to θmax is equal to or greater than a predetermined lower limit, and for example, the predetermined lower limit is 0.15 to 0.16. It is also desirable that the ratio θa / θmax of θa to θmax is equal to or less than a predetermined upper limit, and for example, 0.25 to 0.35. For example, when θa is 90°, the predetermined lower limit is 0.15, and the predetermined upper limit is 0.35, it is desirable that θa is determined in the range of 13.5 to 31.5°. However, the above is an example, and is not limited thereto.

[0078] Furthermore, the optical unit 204 is configured such that its projection characteristic y(θ) satisfies the following formula (2).

[0079]

number

[0080] Here, f is the focal length of the optical unit 204 as described above, and A is a predetermined constant. By setting the lower limit to 1, it is possible to increase the central resolution compared to a fisheye lens of the orthogonal projection type (y=f×sinθ) having the same maximum imaging height, and by setting the upper limit to A, it is possible to obtain an angle of view equivalent to that of a fisheye lens while maintaining good optical performance. The predetermined constant A may be determined in consideration of the balance between the resolutions of the high-resolution region and the low-resolution region, and is preferably set to 1.4 to 1.9. However, the above is just an example and is not limited to this.

[0081] By configuring the optical unit 204 as described above, a high resolution can be obtained in the first region 801a (high resolution region), while the increase in image height y per unit half angle of view θ can be reduced in the second region 801b (low resolution region), making it possible to capture a wider angle of view. Therefore, while the imaging range has a wide angle of view equivalent to that of a fisheye lens, high resolution can be obtained in the high resolution region.

[0082] Next, the flow of processing in this embodiment will be described with reference to Fig. 3 and Fig. 9(a) to (c). Fig. 9(a) to (c) are explanatory diagrams of resolution conversion using an imaging unit that captures a central angle of view with low resolution and a peripheral angle of view with high resolution. Note that in Fig. 3, explanations common to the first embodiment will be omitted.

[0083] First, in step S301 in Fig. 3, the imaging unit 102 captures an image of an arbitrary imaging area 103, similarly to the first embodiment. As shown in Fig. 9(a), the imaging unit 102 in this embodiment is attached at a position where it is possible to capture a high-resolution image of an area 404 in the direction of travel. However, this embodiment is not limited to this, and as shown in Fig. 9(b), the imaging unit 102 may be attached at a position where it is possible to capture a high-resolution image of an area 405 in the direction of landing.

[0084] In step S303, the resolution conversion processing unit 208 performs a process of increasing the resolution of the image data so that the area 404 in the direction of travel and the area 405 in the direction of landing become high resolution in accordance with the flight angle 401 shown in Fig. 9(c). To achieve this process, the imaging system of this embodiment may be provided with a rotation unit 218 that rotates the imaging unit 102 in a hardware (physically) manner and a rotation control unit 217 that controls the rotation angle of the rotation unit 218, and simultaneously control the angle of view captured by the imaging unit 102. The rotation unit 218 can rotate the imaging unit 102 to change the imaging area of ​​the imaging unit 102 according to the flight angle detected by the flight angle detection unit 213.

[0085] This makes it possible for the imaging system of this embodiment to capture images of two areas, an area 404 in the direction of travel and an area 405 in the direction of landing, at high resolution, regardless of the flight angle 401. In this embodiment, in addition to the high-resolution processing, the resolution conversion processing unit 208 performs a low-resolution processing on the image data so that an area 601 captured in advance at high resolution by the imaging unit 102, among areas excluding the area determined by the high-resolution area calculation unit 207, becomes low resolution.

[0086] In this embodiment, the optical system (optical unit 204) forms an optical image with a first resolution in a central region including the center of the light receiving surface of the image sensor 205, and forms an optical image with a second resolution lower than the first resolution in a peripheral region around the central region. The optical system also forms optical images of the first imaging region (region 404 in the direction of travel) and the second imaging region (region 405 in the direction of landing) in the central region.

[0087] As described above, in this embodiment, the imaging unit 102, which has a low resolution peripheral angle of view and a high resolution central angle of view, is attached at a position where high resolution imaging is possible in the flight direction or landing direction of the flying object 101. The imaging system of this embodiment variably controls the area in which high resolution imaging is performed according to the flight angle 401, thereby performing high resolution imaging in the flight direction and landing direction using the imaging unit 102, which is a single camera, and achieving safe flight.

[0088] In addition, in a normal flight state, if the flying object can fly while maintaining a small flight angle 401, the processing of steps S302 and S303 in FIG. 3 (high-resolution processing of images) may not be performed. For example, this is the case when the flying object 101 is a small drone. In addition, particularly when using the imaging unit 102 described in the second embodiment in which the peripheral angle of view is high resolution and the central angle of view is low resolution, the traveling direction and landing direction can be imaged with high resolution without performing the high-resolution processing. For this reason, the processing of steps S302 and S303 in FIG. 3 (high-resolution processing of images) may not be performed.

[0089] According to each embodiment, it is possible to provide an imaging system capable of simultaneously capturing images of the flight direction and landing direction of an aircraft.

[0090] The disclosure of each embodiment includes the following configurations. (Configuration 1) An imaging unit disposed in the aircraft; an output unit that outputs image data acquired by the imaging unit to a display unit, An imaging system characterized in that the imaging unit is positioned so as to be able to simultaneously image a first imaging area in the direction of travel of the aircraft and a second imaging area in the direction of landing of the aircraft. (Configuration 2) the imaging unit has an image sensor and an optical unit that forms an optical image on a light receiving surface of the image sensor; the optical unit forms an optical image having a first resolution in a central region including a center of the light receiving surface of the image sensor, and forms an optical image having a second resolution higher than the first resolution in a peripheral region around the central region; 2. The imaging system according to configuration 1, wherein the optical unit forms optical images of the first imaging region and the second imaging region in the peripheral region. (Configuration 3) the imaging unit has an image sensor and an optical unit that forms an optical image on a light receiving surface of the image sensor; the optical unit forms an optical image having a first resolution in a central region including a center of the light receiving surface of the image sensor, and forms an optical image having a second resolution lower than the first resolution in a peripheral region around the central region; 2. The imaging system according to configuration 1, wherein the optical unit forms an optical image of the first imaging region or the second imaging region in the central region. (Configuration 4) A flight angle detection unit detects a flight angle of the flying object, The output unit is a high-resolution area calculation unit that determines an area to be converted into a high-resolution image from among the imaging area of ​​the imaging unit in accordance with the flight angle; and a resolution conversion processing unit that converts the area determined by the high resolution area calculation unit into a high resolution and converts an area excluding the high resolution area into a low resolution to generate image data. (Configuration 5) A flight angle detection unit that detects a flight angle of the flying object; a rotation unit that rotates the imaging unit to change an imaging area of ​​the imaging unit according to the flight angle; 4. The imaging system according to any one of configurations 1 to 3, further comprising: a rotation control unit that controls a rotation angle of the rotation unit. (Configuration 6) A flight mode detection unit that detects a flight state of the aircraft; The imaging system according to any one of configurations 1 to 3, further comprising: a recognition unit that recognizes an object within a recognition area determined based on the state detected by the flight mode detection unit. (Configuration 7) 7. The imaging system according to configuration 6, further comprising: a flight control unit that controls the flying object so as to avoid collision with the object recognized by the recognition unit. (Configuration 8) 7. The imaging system according to configuration 6, further comprising a warning unit that performs a warning process for a passenger of the aircraft based on a recognition result of the recognition unit.

[0091] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. Parts of the above-described embodiments may be combined as appropriate. [Explanation of symbols]

[0092] 101 Aircraft (Flying Objects) 102 Imaging unit 201 Imaging System 202 Development recognition unit (output unit) 215 Display section

Claims

1. An imaging unit disposed in the aircraft; an output unit that outputs image data acquired by the imaging unit to a display unit, An imaging system characterized in that the imaging unit is positioned so as to be able to simultaneously image a first imaging area in the direction of travel of the aircraft and a second imaging area in the direction of landing of the aircraft.

2. the imaging unit has an image sensor and an optical unit that forms an optical image on a light receiving surface of the image sensor; the optical unit forms an optical image having a first resolution in a central region including a center of the light receiving surface of the image sensor, and forms an optical image having a second resolution higher than the first resolution in a peripheral region around the central region; The imaging system according to claim 1 , wherein the optical section forms optical images of the first imaging region and the second imaging region in the peripheral region.

3. the imaging unit has an image sensor and an optical unit that forms an optical image on a light receiving surface of the image sensor; the optical unit forms an optical image having a first resolution in a central region including a center of the light receiving surface of the image sensor, and forms an optical image having a second resolution lower than the first resolution in a peripheral region around the central region; The imaging system according to claim 1 , wherein the optical section forms an optical image of the first imaging region or the second imaging region in the central region.

4. A flight angle detection unit detects a flight angle of the flying object, The output unit is a high-resolution area calculation unit that determines an area to be converted into a high-resolution image from among the imaging area of ​​the imaging unit in accordance with the flight angle; The imaging system according to any one of claims 1 to 3, further comprising a resolution conversion processing unit that converts the area determined by the high resolution area calculation unit to a high resolution and converts an area excluding the high resolution area calculation unit to a low resolution to generate image data.

5. A flight angle detection unit that detects a flight angle of the flying object; a rotation unit that rotates the imaging unit to change an imaging area of ​​the imaging unit according to the flight angle; 4. The imaging system according to claim 1, further comprising a rotation control unit that controls a rotation angle of the rotation unit.

6. A flight mode detection unit that detects a flight state of the aircraft; 4. The imaging system according to claim 1, further comprising: a recognition unit that recognizes an object within a recognition area determined based on the state detected by the flight mode detection unit.

7. 7. The imaging system according to claim 6, further comprising a flight control unit that controls the flying object so as to avoid collision with the object recognized by the recognition unit.

8. 7. The imaging system according to claim 6, further comprising a warning unit that performs a warning process for a passenger of the aircraft based on a recognition result of the recognition unit.

Images