Camera system, control method, and program

Abstract

To provide a camera system that can photograph both images of a movable body in a travel direction and bird's-eye images, while reducing the number of cameras.SOLUTION: A camera system has: imaging means that is installed on a movable body, and picks up an image of a first area corresponding to an angle of view equal to or more than a predetermined angle of view and a second area corresponding to an angle of view less than the predetermined angle of view; and processing means that creates first images and second images on the basis of the images picked up by the imaging means. The processing means creates the first images by cutting out part of the images picked up in the first area. The processing means creates the second images by performing distortion correction processing of correcting distortion of the images picked up in an area including the second area.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a camera system, a control method, and a program. [Background technology]

[0002] In recent years, drones and other unmanned aerial vehicles (UAVs) have been used in various industrial fields. When controlling a drone from a remote location, it is not rare to control the drone while visually checking the image of the direction of travel captured by the camera installed on the drone. On the other hand, drones are sometimes flown for the purpose of acquiring overhead images. In this case, the camera installed on the drone needs to capture both the image of the direction of travel for control and the image directly below for acquiring the overhead image.

[0003] Patent Document 1 discloses a mechanism in which one drone holds multiple cameras. This mechanism makes it possible to simultaneously capture images in different directions using multiple cameras. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2020-508615 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technique of installing multiple cameras on a moving object such as a drone increases the size and weight of the moving object, making it difficult to operate.

[0006] Therefore, an object of the present invention is to provide a camera system capable of capturing both an image in the direction of travel of a moving object and an overhead image while minimizing the number of cameras. [Means for solving the problem]

[0007] A camera system according to one embodiment of the present invention comprises an imaging means installed on a moving body for imaging a first region corresponding to an angle of view equal to or greater than a predetermined angle of view and a second region corresponding to an angle of view less than the predetermined angle of view, and a processing means for generating a first image and a second image based on images captured by the imaging means, wherein the processing means generates the first image by cutting out a portion of the image captured in the first region, and the processing means generates the second image by performing a distortion correction process that corrects distortion in the image captured in an area including the second region. Effect of the Invention

[0008] According to the present invention, it is possible to provide a camera system capable of capturing both an image in the traveling direction of a moving object and an overhead image while reducing the number of cameras. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a camera system 100 according to a first embodiment of the present invention. [Diagram 2] 2A and 2B are diagrams illustrating optical characteristics of the optical unit 11 according to the first embodiment of the present invention. Fig. 2A is a diagram showing, in contour form, the image height y at each half angle of view on the light receiving surface of the image sensor of the optical unit 11. Fig. 2B is a diagram showing projection characteristics that indicate the relationship between the image height y and the half angle of view θ of the optical unit 11. [Diagram 3] 1 is a diagram illustrating an imaging range of a drone 200 and a camera 10 according to a first embodiment of the present invention. FIG. [Figure 4] 4A and 4B are diagrams showing images captured by the imaging unit 12. Fig. 4A shows, as an example, an intersection photographed from the sky by the drone 200. Fig. 4B shows another example of an image captured by the imaging unit 12. [Diagram 5] 2 is a diagram showing images displayed on the control image display unit 14 and the overhead image display unit 15. FIG. [Figure 6] 2 is a block diagram showing processing blocks included in a processing unit 13. FIG. [Figure 7] 4 is a flowchart showing a process according to the first embodiment of the present invention. [Figure 8] FIG. 11 is a block diagram showing a configuration according to a second embodiment of the present invention. [Figure 9] 10 is a flowchart showing a process according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In the drawings, the same members or elements are given the same reference numerals, and duplicated descriptions thereof will be omitted or simplified.

[0011] <First embodiment> Fig. 1 is a block diagram showing a camera system 100 according to a first embodiment of the present invention. The camera system 100 is a system that displays an image captured by a camera 10 installed in a drone 200 (see Fig. 3) on a display unit (control image display unit 14 and overhead image display unit 15) that can be seen by a pilot. The camera system 100 has the camera 10, the control image display unit 14, and the overhead image display unit 15. The drone 200 is an example of a moving body.

[0012] The camera 10 is an imaging device installed on the drone 200 to monitor the surroundings of the drone 200, and has an optical unit 11, an imaging unit 12, and a processing unit 13. The optical unit 11 uses at least one lens to form an image of light incident from the outside on the imaging unit 12. The optical characteristics of the optical unit 11 will be described in detail later. The imaging unit 12 has an image sensor, converts the optical subject image formed by the optical unit 11 into an electrical signal, and transmits it to the processing unit 13. The image sensor generates a video signal by a photoelectric conversion function.

[0013] The processing unit 13 includes, for example, a SOC (System On Chip) / FPGA (Field Programmable Gate Array), a CPU as a computer, and a memory as a storage medium. The CPU executes computer programs stored in the memory to perform various controls of the entire system. CPU is an abbreviation for Central Processing Unit.

[0014] Moreover, the processing unit 13 develops the video signal obtained from the imaging unit 12, and performs various image processing such as WDR (Wide Dynamic Range) correction, gamma correction, LUT processing, distortion correction, and clipping. The detailed processing in the processing unit 13 will be described later.

[0015] Processing unit 13 has various interfaces for video input and output, and outputs video to control image display unit 14 and overhead image display unit 15. Typically, control image display unit 14 and overhead image display unit 15 are located away from drone 200. For this reason, video is transmitted from processing unit 13 to control image display unit 14 and overhead image display unit 15 using wireless communication means. Note that some or all of the functions of processing unit 13 may be performed by the display unit (control image display unit 14 and overhead image display unit 15).

[0016] The control image display unit 14 is a monitor that displays an image in the traveling direction of the drone 200, and displays the image from the processing unit 13. The overhead image display unit 15 is a monitor that displays an overhead image directly below the drone 200, and displays the image from the processing unit 13. Note that the overhead image display unit 15 may use the same monitor as the control image display unit 14, and in that case, the image is displayed in a format such as PIP (Picture-in-Picture) or PBP (Picture-by-Picture).

[0017] Fig. 2 is a diagram for explaining the optical characteristics of the optical unit 11 according to the first embodiment of the present invention. Fig. 2(A) is a diagram showing the image height y at each half angle of view on the light receiving surface of the imaging element of the optical unit 11 in the form of contour lines. Fig. 2(B) is a diagram showing the projection characteristics showing the relationship between the image height y and the half angle of view θ of the optical unit 11. In Fig. 2(B), the half angle of view (angle between the optical axis and the incident light) θ is shown on the horizontal axis, and the imaging height (image height) y on the light receiving surface (image plane) of the camera 10 is shown on the vertical axis.

[0018] The optical characteristics of the optical unit 11 of the camera 10 are configured such that the projection characteristic y(θ) changes between a region where the half angle of view θ is small (near the optical axis) and a region where the half angle of view θ is large (region away from the optical axis), as shown in the projection characteristic in Fig. 2(B). In other words, when the increase in image height y per unit half angle of view θ (i.e., the number of pixels per unit angle) is called the resolution, the resolution differs depending on the region.

[0019] This local resolution can be expressed as the differential value dy(θ) / dθ of the projection characteristic y(θ) at the half angle of view θ. In other words, the greater the gradient of the projection characteristic y(θ) in Fig. 2(B), the higher the resolution. Also, the greater the interval between the image heights y at each half angle of view on the contour lines in Fig. 2(A), the higher the resolution.

[0020] In this embodiment, when the half angle of view θ is equal to or larger than a predetermined half angle of view θa, the outer region formed on the light receiving surface of the sensor is called a high resolution region 20a, and when the half angle of view θ is smaller than the predetermined half angle of view θa, the central region formed on the light receiving surface of the sensor is called a low resolution region 20b.

[0021] In addition, in an optical system with such optical characteristics, the magnification in the radial direction with respect to the optical axis can be adjusted by adjusting the projection characteristic y(θ). This allows the aspect ratio in the radial direction and circumferential direction with respect to the optical axis to be controlled, so that, unlike a normal fisheye lens, it is possible to obtain an image with little distortion even in the peripheral area despite the wide angle.

[0022] FIG. 3 is a diagram illustrating the drone 200 and the imaging range of the camera according to the first embodiment of the present invention. FIG. 3 is a side view of the drone 200. The downward direction in FIG. 3 is the vertically downward direction. In this embodiment, the camera 10 is installed in a direction to capture an image in the vertically downward direction relative to the drone 200, but the imaging direction can be appropriately adjusted depending on the imaging purpose and the angle of view and optical characteristics of the optical unit 11. The imaging range 30a shows the entire angle of view of the camera 10. The imaging range 30b is the range captured by the high-resolution area 20a of the camera 10.

[0023] As shown in Fig. 3, by installing the camera 10 on the drone 200 facing vertically downward, it is possible to obtain a wide-angle image of a wide range of the ground surface from the drone 200 in the imaging range 30a, and an image captured in the horizontal direction relative to the drone 200 in the imaging range 30b. The wide-angle image of a wide range of the ground surface from the drone 200 in the imaging range 30a is transmitted to the overhead image display unit 15. The image captured in the horizontal direction relative to the drone 200 in the imaging range 30b is transmitted to the control image display unit 14. Based on the images captured in this way, both images to be displayed on the overhead image display unit 15 and the control image display unit 14 can be obtained from one camera 10.

[0024] Compared to a normal fisheye lens, there are two advantages to capturing a navigation image and an overhead image with an optical system having optical characteristics like those of this embodiment. The first advantage is that the navigation image can be captured in the high-resolution region 20a. As a result, a higher-resolution image can be obtained than with a normal fisheye lens, and since the aspect ratio can be maintained with the optical system of this embodiment, an image with less distortion can be obtained without being crushed in the radial direction in the peripheral region. The second advantage is that a wider range of overhead images can be obtained. When capturing an image of the ground surface from the sky using a normal fisheye lens, the center is greatly enlarged, while the ground surface (periphery) farther from the lens is strongly compressed on the imaging surface. In contrast, the optical system of this embodiment allocates more pixels to the peripheral region, so that an overhead image can be obtained without losing details in the peripheral region compared to a fisheye lens. As a result, an image of a wide range of the ground surface can be restored by performing a distortion correction process described later.

[0025] Fig. 4 is a diagram showing an image captured by the imaging unit 12. Fig. 4(A) shows, as an example, an intersection photographed from the sky by the drone 200. Fig. 4(B) shows another example of an image captured by the imaging unit 12. The processing unit 13 processes the image in Fig. 4(A) to generate an image to be output to the control image display unit 14 and an image to be output to the overhead image display unit 15. The imaging unit 12 is positioned so that a subject in a horizontal position relative to the drone 200 is reflected in the high-resolution area 20a.

[0026] Image area 40a enclosed by a dashed rectangle is an image area used for display on the control image display unit 14, and shows the front (traveling direction) of the drone 200. A part or all of image area 40a is included in the high resolution area 20a of the imaging unit 12. Image area 40b enclosed by a dashed circle is an image area used for display on the overhead image display unit 15, and shows a wide range of the ground surface including directly below the drone 200.

[0027] Considering the ease of maneuvering the drone 200, it is preferable that the horizon, i.e., a direction approximately perpendicular to the optical axis, is reflected on the maneuver image display unit 14. In order to include the horizon in the traveling direction of the drone 200 within the field of view and to make the most of the imaging area of ​​the imaging unit 12, it is preferable to arrange the imaging unit 12 so that the longitudinal direction of the imaging area is parallel to the traveling direction so that the image shown in FIG. 4(A) can be obtained. In this case, the lateral direction of the imaging area of ​​the imaging unit 12 intersects with the traveling direction of the drone 200. Alternatively, as shown in FIG. 4(B), the imaging unit 12 may be arranged with the center of the imaging unit 12 and the optical center of the optical unit 11 shifted from each other so that an image showing the front (traveling direction) more widely than the rear can be obtained.

[0028] Fig. 5 is a diagram showing images displayed on control image display unit 14 and overhead image display unit 15. Fig. 5(A) is a diagram showing an image displayed on control image display unit 14. Fig. 5(B) is a diagram showing an image displayed on overhead image display unit 15. Here, as an example, the image displayed when imaging unit 12 outputs the image shown in Fig. 4(A) is shown.

[0029] Processing unit 13, through operations described below, displays an image based on image area 40a (image in FIG. 5(A)) on control image display unit 14, and displays an image based on image area 40b (image in FIG. 5(B)) on overhead image display unit 15. This display allows the pilot to simultaneously obtain an image of the direction of travel for control and an overhead image from the sky from a single camera.

[0030] 6 is a block diagram showing processing blocks included in the processing unit 13. The processing unit 13 has an image processing unit 61, a clipping unit 62, and a distortion correction unit 63.

[0031] The image processing unit 61 executes image processing on the video signal received from the imaging unit 12. In this embodiment, the image processing unit 61 executes various correction processes such as WDR correction and gamma correction in addition to development processing. WDR is an abbreviation for Wide Dynamic Range.

[0032] The cropping unit 62 determines and crops out the image area 40a and the image area 40b (or an area including these) from the image processed by the image processing unit 61. The cropping unit 62 transmits a video signal including the image area 40a to the control image display unit 14 via the wireless video interface. Here, since the image area 40a includes the high resolution area 20a, a display with less distortion can be obtained even if it is output to the control image display unit 14 without distortion correction processing due to the optical characteristics of the optical unit 11. The cropping unit 62 also outputs a video signal including the image area 40b to the distortion correction unit 63. The image area 40a includes at least the high resolution area 20a. The image area 40b includes at least the low resolution area 20b.

[0033] The distortion correction unit 63 performs distortion correction processing on the video signal including the video area 40b received from the cropping unit 62. The distortion correction unit 63 corrects the video based on a correction coefficient according to the optical characteristics of the optical unit 11. The distortion correction unit 63 then transmits the processed video signal to the overhead image display unit 15 via a wireless video interface (not shown).

[0034] 7 is a flowchart showing processing according to the first embodiment of the present invention. In S70, the image processing unit 61 of the processing unit 13 performs development processing and image processing as necessary on the video signal received from the imaging unit 12. Next, in S71, the cropping unit 62 of the processing unit 13 performs processing to crop out areas corresponding to the video area 40a and the video area 40b. The cropping unit 62 of the processing unit 13 outputs an image including the cropped video area 40a to the control image display unit 14, and outputs an image including the cropped video area 40b to the distortion correction unit 63.

[0035] Next, in S72, control image display unit 14 displays an image including cut-out image area 40a. Next, in S73, distortion correction unit 63 performs distortion correction processing on the image including cut-out image area 40b. Finally, in S74, overhead image display unit 15 displays an image including image area 40b that has been subjected to distortion correction processing.

[0036] According to the image processing flowchart in Figure 7, a display image such as that shown in Figure 5 can be generated, allowing the pilot to simultaneously obtain images of the direction of travel for control and an overhead image from above from a single camera.

[0037] <Second embodiment> Some conventional drones have the ability to capture blur-free images even when the drone shakes or tilts, and this function is mainly performed by a gimbal that holds the camera. However, such a gimbal mechanism leads to an increase in the size and weight of the drone, which makes it difficult to operate.

[0038] Therefore, in the second embodiment of the present invention, an example is described in which a blur-free control image is displayed without a gimbal mechanism by dynamically controlling the cut-out area of ​​the image displayed on the above-mentioned control image display unit 14.

[0039] 8 is a block diagram showing a configuration according to a second embodiment of the present invention. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. The camera system 100 of the second embodiment differs from the first embodiment in that it has a tilt detection unit 81. In the second embodiment, the camera 10 may have the tilt detection unit 81. The tilt detection unit 81 may be provided separately from the camera 10 and installed in the drone 200.

[0040] The tilt detection unit 81 detects the tilt of the drone 200, and transmits tilt information, which is information on the detection result, to the cut-out unit 62. The tilt detection unit 81 detects the tilt of the drone 200 in the forward / backward and left / right directions from a horizontal state.

[0041] The cropping unit 62 determines a crop area to be displayed on the control image display unit 14 based on the tilt information received from the tilt detection unit 81. Specifically, the cropping unit 62 shifts the crop area so as to cancel the shift in the image area 40a caused by the tilt of the drone 200. For example, when the tilt information indicates that the front side of the drone 200 is tilted to lift up, the cropping unit 62 shifts the crop area downward. When the tilt information indicates that the right side of the drone 200 is tilted to lift up, the cropping unit 62 changes the crop area so as to rotate it clockwise. The other configurations are the same as those of the first embodiment.

[0042] Fig. 9 is a flowchart showing processing according to the second embodiment of the present invention. The same processes as those in the first embodiment of Fig. 7 are denoted by the same reference numerals, and detailed explanations will be omitted. In S90, the cropping unit 62 of the processing unit 13 acquires tilt information from the tilt detection unit 81. In S91, when cropping out the areas corresponding to the image area 40a and the image area 40b, the cropping unit 62 of the processing unit 13 determines and crops out the area corresponding to the image area 40a based on the tilt information acquired from the tilt detection unit 81.

[0043] According to the image processing flowchart in Figure 9, it is possible to display a blur-free control image without a gimbal mechanism.

[0044] (Other embodiments) The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. Also, the present invention can be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0045] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0046] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an imaging means that is installed on the moving body and captures an image of a first region corresponding to an angle of view equal to or larger than a predetermined angle of view and a second region corresponding to an angle of view smaller than the predetermined angle of view; A processing means for generating a first image and a second image based on the image captured by the imaging means; having The processing means generates the first image by cutting out a part of an image captured in the first region; The processing means generates the second image by performing a distortion correction process for correcting distortion of an image captured in an area including the second area. A camera system characterized by: (Configuration 2) The first region has a higher resolution, which is the number of pixels per unit angle of view, than the second region. 2. The camera system according to claim 1, (Configuration 3) a first display means for displaying the first image and a second display means for displaying the second image. 3. The camera system according to configuration 1 or 2. (Configuration 4) The moving object is a drone. 4. The camera system according to any one of configurations 1 to 3. (Configuration 5) A tilt detection means for detecting the tilt of the drone, The processing means determines a cut-out area when cutting out the first image based on information from the tilt detection means. 5. The camera system according to configuration 4. (Configuration 6) The imaging means is disposed so that a subject in a horizontal position relative to the drone is reflected in the first area. 6. The camera system according to configuration 4 or 5, (Configuration 7) The short side direction of the imaging area of ​​the imaging unit means intersects with the direction of travel of the drone. 7. The camera system according to any one of configurations 4 to 6, (Configuration 8) an optical means for forming an image of light on the imaging means; The imaging means is disposed so that the center of the imaging area is shifted from the optical center of the optical means. 8. The camera system according to any one of configurations 4 to 7, (Method 1) a processing step of generating a first image and a second image based on images captured by an imaging means that is installed on a moving body and captures a first area corresponding to an angle of view equal to or larger than a predetermined angle of view and a second area corresponding to an angle of view less than the predetermined angle of view, The processing step generates the first image by cutting out a part of an image captured in the first region; The processing step generates the second image by performing a distortion correction process for correcting distortion of an image captured in an area including the second area. A control method comprising: (Program 1) Computer, A processing means for generating a first image and a second image based on images captured by an imaging means that is installed on a moving body and captures a first area corresponding to an angle of view equal to or larger than a predetermined angle of view and a second area corresponding to an angle of view less than the predetermined angle of view. Functioning as a The processing means generates the first image by cutting out a part of an image captured in the first region; The processing means generates the second image by performing a distortion correction process for correcting distortion of an image captured in an area including the second area. A program characterized by: [Explanation of symbols]

[0047] 100 Camera System 10. Camera 13 Processing section 14. Control image display unit 15. Bird's-eye view image display section 11 Optics Department 12 Imaging unit 20a High Resolution Area 20b Low resolution area 200 Drone 30a Imaging range 30b Imaging range 40a Image area 40b Image area 61 Image processing section 62 Cut-out section 63 Distortion correction section 81 Tilt detection unit

Claims

1. an imaging means installed on the moving body for imaging a first area corresponding to an angle of view equal to or larger than a predetermined angle of view and a second area corresponding to an angle of view smaller than the predetermined angle of view; A processing means for generating a first image and a second image based on the image captured by the imaging means; having The processing means generates the first image by cutting out a part of an image captured in the first area, The processing means generates the second image by performing a distortion correction process for correcting distortion of an image captured in an area including the second area. A camera system characterized by:

2. The first region has a higher resolution, which is the number of pixels per unit angle of view, than the second region.

2. The camera system according to claim 1 .

3. a first display means for displaying the first image and a second display means for displaying the second image.

2. The camera system according to claim 1 .

4. The moving object is a drone.

2. The camera system according to claim 1 .

5. A tilt detection means for detecting the tilt of the drone, The processing means determines an area to be cut out when cutting out the first image based on information from the tilt detection means.

5. The camera system according to claim 4.

6. The imaging means is disposed so that a subject in a horizontal position relative to the drone is reflected in the first area.

5. The camera system according to claim 4.

7. The short side direction of the imaging area of ​​the imaging unit means intersects with the direction of travel of the drone.

5. The camera system according to claim 4.

8. an optical means for forming an image of light on the imaging means; The imaging means is disposed so that the center of the imaging area is shifted from the optical center of the optical means.

5. The camera system according to claim 4.

9. a processing step of generating a first image and a second image based on images captured by an imaging means that is installed on a moving body and captures a first area corresponding to an angle of view equal to or larger than a predetermined angle of view and a second area corresponding to an angle of view less than the predetermined angle of view, The processing step generates the first image by cutting out a part of an image captured in the first region; The processing step generates the second image by performing a distortion correction process for correcting distortion of an image captured in an area including the second area. A control method comprising:

10. Computer, A processing means for generating a first image and a second image based on images captured by an imaging means that is installed on a moving body and captures a first area corresponding to a field of view equal to or larger than a predetermined field of view and a second area corresponding to a field of view smaller than the predetermined field of view. Function as a The processing means generates the first image by cutting out a part of an image captured in the first area, The processing means generates the second image by performing a distortion correction process for correcting distortion of an image captured in an area including the second area. A program characterized by:

Images