Camera Layout For Aerial Imaging
By employing a multi-camera layout and computing unit stitching technology in the aerial imaging system, composite images of different modes are generated, solving the applicability and cost issues of existing systems and achieving efficient imaging and cost reduction in various application scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-19
- Publication Date
- 2026-03-24
AI Technical Summary
Existing aerial imaging systems are typically customized for a single specific scenario, are expensive, and have large hardware size and weight, making it difficult to adapt to the needs of multiple application scenarios.
The device employs a multi-camera layout, with each camera having a different observation direction, forming a combined continuous total field of view of at least 75 degrees in the horizontal direction of the device. Combined with the computing unit, the images are stitched together to generate synthetic images in different modes to adapt to various application scenarios.
It achieves efficient imaging in a variety of application scenarios, reduces system cost and hardware size, is suitable for small aircraft, supports multiple component layouts, reduces production costs, and can process image quality in real time or offline.
Smart Images
Figure CN121728367A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an aerial imaging device comprising a plurality of cameras and configured to be mounted on a vehicle to acquire aerial images of a target area. Background Technology
[0002] Aerial photography is used in a wide range of applications, such as cartography (especially in photogrammetry and land surveying), power line monitoring, infrastructure monitoring, and disaster recovery planning.
[0003] Specialized aerial imaging equipment carried by a vehicle can be used to acquire aerial photographs of a target area. The term "vehicle" refers to any type of aircraft, such as fixed-wing aircraft, helicopters, multi-rotor aircraft, and airships. These vehicles can be manned or unmanned (i.e., drones).
[0004] To cover a relatively large area, multiple aerial images are typically stitched together, which is standard practice. In aerial imaging, the area formed by stitching together images taken continuously along a flight path during flight is called a swath. In other words, a swath consists of multiple images taken along the flight path, which are then stitched together to form an image band. The flight path direction can be determined before or during imaging flight. Depending on the intended application, the swath must be continuous and have a uniform width. The swath width is defined as the dimension perpendicular to the flight path direction.
[0005] Image stitching can be performed in real time on the vehicle, for example, for immediate quality control. For instance, such quality control can be performed on the stitched, low-quality image first, while high-quality processing of the multiple images can be performed offline, such as on a server cluster, after image acquisition is complete and the aircraft returns to the ground.
[0006] The maximum width of the stripe is determined by the imaging device's field of view in the lateral direction (i.e., perpendicular to the flight path, also known as the direction of travel). The vehicle flies along a preset flight path to photograph the target area. Typically, this flight path contains multiple parallel tracks with alternating directions to achieve complete coverage of the target area, but other flight modes can also be used. The spacing between parallel tracks must ensure that adjacent stripes at least partially overlap; therefore, this spacing depends on the stripe width. Generally, to shorten flight time, a stripe width as large as possible is desirable. However, depending on the application scenario, the maximum usable stripe width and the corresponding maximum allowable track spacing will be affected by various factors, such as the ground sampling distance.
[0007] For example, the primary purpose of topographic mapping is to obtain a nadir view, such as generating orthophotos or regional land use maps. On the one hand, since topographic mapping may cover large areas, it is desirable to have stripes as wide as possible. On the other hand, nadir views are preferred over oblique views because oblique views may suffer from perspective distortion, which limits the usable field of view of the camera system. Imaging systems in the prior art typically consist of a single camera facing the nadir.
[0008] To give another example, in urban surveying and modeling, it may be necessary to acquire both nadir and oblique views. For instance, oblique views can capture building facades, which, when combined with the nadir view, can generate geospatial data (such as 3D models). Such imaging systems typically contain up to five cameras: one facing the nadir, and the other four facing forward, backward, left, and right respectively in four oblique directions, with their optical axes specifically at a 45-degree angle to the nadir. This arrangement of oblique cameras allows for capturing building facades from four directions. There may be uncovered areas between the nadir and oblique fields of view, but these areas are not a concern in this application scenario.
[0009] To achieve sufficient resolution and strip width, cameras with large image sensors are typically used. Furthermore, due to the requirements for specific camera layouts and optical components (such as lenses), customized equipment is often necessary for specific applications. Therefore, such aerial photography systems are expensive, only suitable for a single, specific scenario, and relatively large in size and weight.
[0010] Purpose of the invention
[0011] In view of the above, the purpose of this invention is to provide an improved aerial imaging system to cover a variety of application scenarios. Summary of the Invention
[0012] This invention relates to a system comprising an aerial imaging device configured to be mounted on a vehicle arranged to fly along a flight path at a certain altitude. The aerial imaging device is used to acquire aerial images of a target area and includes multiple cameras, each with a field of view. The cameras are arranged and configured to have different observation directions in the device's lateral direction, which corresponds to a cross-track direction perpendicular to the flight path.
[0013] The cameras among the plurality of cameras have overlapping fields of view, such that in the lateral direction of the device, these cameras together provide a combined continuous total field of view of at least 75 degrees (e.g., at least 90 degrees), which is greater than the individual field of view of each camera.
[0014] For example, multiple cameras can be visible light cameras, such as color cameras like RGB cameras.
[0015] For example, an aerial imaging device may have at least five cameras, or even at least seven cameras.
[0016] The system also includes a computing unit configured to stitch together images captured by multiple cameras to generate a rural-mode composite image, an urban-mode composite nadir image, and an urban-mode side-tilted image. The rural-mode composite image is generated by stitching together images from a subset of the multiple cameras, and its field of view in the device's lateral direction is only a portion of the total field of view. The urban-mode composite nadir image is also generated by stitching together images from a subset of the multiple cameras, and its field of view in the device's lateral direction is only a portion of the field of view of the rural-mode composite image. The urban-mode side-tilted image has a field of view in the device's lateral direction that is only a portion of the total field of view and has a more tilted observation direction than the urban-mode composite nadir image (e.g., towards the left or right of the vehicle / airborne imaging equipment).
[0017] The cameras used to generate the side-tilted images are arranged such that their optical axes are offset from the nadir direction in the device's lateral direction. In other words, these cameras are not facing the nadir. In the device's lateral direction, the angular difference between the observation directions (e.g., optical axes) of any two such cameras is at least 70 degrees. For example, each camera used to generate the side-tilted images of the city pattern is tilted to the left or right relative to the nadir direction and has a left-right observation direction with a deviation from the nadir direction between 30 and 60 degrees.
[0018] In addition to cameras used to generate side-tilted images, the aerial imaging device may also include cameras used to generate forward-tilted and / or backward-tilted fields of view, the optical axes of which are offset from the nadir direction in the flight path. For example, each camera used to generate forward-tilted and / or backward-tilted fields of view is tilted forward or backward relative to the nadir direction and has a forward or backward observation direction with a deviation from the nadir direction between 30 and 60 degrees.
[0019] It should be noted that the fields of view in the horizontal direction of the synthesized images in the rural mode are different from those in the synthesized nadir images in the urban mode.
[0020] The computing unit may be wholly or at least partially located on the aerial imaging equipment. Alternatively, the computing unit may be completely separated from the aerial imaging equipment; for example, the computing unit may be implemented as an offline server and / or a cloud server.
[0021] For example, aerial imaging equipment can be configured to stitch multiple images in real-time (or near real-time) for on-the-fly quality control. Such quality control may include generating low-quality (test) rural-pattern composite images, (test) urban-pattern composite nadir images, and / or (test) urban-pattern side-tilt images, e.g., lower quality compared to the final images generated by a more powerful computing unit separate from the aerial imaging equipment. Typically, high-quality image processing for generating the (final) rural-pattern composite images, (final) urban-pattern composite nadir images, and (final) urban-pattern side-tilt images is performed, for example, by a separate component of the computing unit separate from the aerial imaging equipment after the aircraft returns to the ground (i.e., after image acquisition is complete).
[0022] In a further embodiment, the computing unit is configured to generate a composite rural mode image based on images from one set of cameras (i.e., the rural mode group) from a plurality of cameras in rural mode. Additionally, in urban mode, a composite urban mode nadir image is generated based on images from another set of cameras (i.e., the urban mode nadir group). It should be noted that the urban mode nadir group differs from the rural mode group; that is, the cameras selected from the plurality of cameras to form the urban mode nadir group are different from those used to form the rural mode group. Furthermore, in urban mode, an urban mode side-tilt image can be generated based on images from yet another set of cameras (i.e., the urban mode tilt group). The urban mode tilt group may differ from both the urban mode nadir group and the rural mode group. Specifically, the side-tilt image can be generated based on one or more image segments from one or more images from one or more cameras provided by the urban mode tilt group. More specifically, in the lateral direction of the device, the field of view of the cameras in the urban mode tilt group deviates from the nadir to a greater extent than the field of view of the cameras in the urban mode nadir group.
[0023] For example, the cameras included in the rural mode group and the urban mode nadir group can be arranged symmetrically around the nadir direction, while the urban mode tilt group can include two cameras, one of which is tilted 45 degrees to the left relative to the nadir direction in the horizontal direction of the device, and the other camera is tilted 45 degrees to the right relative to the nadir direction in the horizontal direction of the device.
[0024] In a further embodiment, the rural mode group includes more cameras than the urban mode nadir group; for example, the cameras selected from multiple cameras in the rural mode group can provide a continuous combined total field of view in the lateral direction of the device. For instance, the rural mode group may include all of the multiple cameras.
[0025] In a further embodiment, the aerial imaging device can be configured such that the continuous combined total field of view covers a certain angle in the lateral direction of the device. Specifically, for a rural mode composite image, the field of view angle may correspond to at least 70 degrees (especially at least 100 degrees). For an urban mode composite nadir image, the field of view angle may correspond to less than 80 degrees (especially less than 60 degrees).
[0026] In a further embodiment, the aerial imaging device (e.g., as part of a plurality of cameras) may further include at least one camera for generating a forward-tilted field of view (e.g., comprising two or more cameras arranged such that their fields of view are laterally adjacent and partially overlap, e.g., individual fields of view complementarity form a larger continuous field of view in the lateral direction of the device). Additionally, the aerial imaging device may also include at least one camera for generating a backward-tilted field of view (e.g., comprising two or more cameras arranged such that their fields of view are laterally adjacent and partially overlap). For example, one or more cameras for generating the forward-tilted field of view are tilted forward at 45 degrees relative to the nadir direction. One or more cameras for generating the backward-tilted field of view are tilted backward at 45 degrees relative to the nadir direction.
[0027] Furthermore, the computing unit can be configured to provide forward-tilted and backward-tilted images for the city model. For example, the combined continuous total field of view angle of the city model nadir group in the device's lateral direction can be at least the same as the continuous total field of view angle of the forward-tilted or backward-tilted field of view in the device's lateral direction.
[0028] For example, images generated by the city-mode tilt group (combined with the city-mode nadir group) can be used to acquire 3D data. For instance, the city-mode tilt group can capture building facades, which can then be used to construct 3D models of urban scenes.
[0029] Generating orthophotos of cities typically requires the use of three-dimensional geometric data, which can be acquired through LiDAR and / or photogrammetry techniques. For instance, images from the city-mode nadir group and the city-mode tilt group can be used to construct a complete three-dimensional model, which, after perspective-free rendering, can generate a two-dimensional orthophoto.
[0030] In a further embodiment, the aerial imaging device may also include a near-infrared camera having a near-infrared field of view, and arranged such that the near-infrared field of view partially overlaps with the combined continuous total field of view of the plurality of cameras. In some embodiments, the aerial imaging device may include two near-infrared cameras arranged such that their near-infrared fields of view partially overlap. In a further embodiment, the overlapping region of the near-infrared fields of view may not coincide with the overlapping region of the fields of view of the plurality of cameras (cameras used to form the combined continuous total field of view).
[0031] The orientation of an image (and consequently, camera frames) is generally divided into two types: vertical orientation (portraitor orientation), where the image height is greater than its width; and horizontal orientation (landscape orientation), where the image height is less than its width. It should be noted that, in this application, width refers to the image's dimension in the horizontal direction of the device, and height refers to the image's dimension in the direction of the flight path.
[0032] In a further embodiment, one or more of the cameras may be in a horizontal mode, while another camera may be in a vertical mode, for example, the field of view of the horizontal mode camera is rotated 90 degrees compared to the field of view of the vertical mode camera. For instance, the cameras may employ rectangular sensors, each with two sides of different lengths. Specifically, the long side of the sensor in the horizontal mode is rotated 90 degrees compared to the long side of the sensor in the vertical mode.
[0033] In a further embodiment, at least a subset of the multiple cameras may be arranged in a straight line along the flight path. In some embodiments, the cameras in this subset may be triggered sequentially with a certain time delay, for example, these cameras may be triggered based on a function determined by the flight speed of the vehicle and / or its determined position.
[0034] In a further embodiment, each of the plurality of cameras may be configured to have the same focal length common to all of the plurality of cameras.
[0035] In a further embodiment, the focal length of the near-infrared camera may be different from the focal length of each of the multiple cameras. Specifically, in the case of multiple near-infrared cameras (see above), each near-infrared camera may be configured to have the same focal length common to all of the near-infrared cameras.
[0036] In a further embodiment, each of at least a subset of the multiple cameras may be equipped with a lens that produces barrel lens distortion.
[0037] In a further embodiment, the computing unit may be configured to generate at least one of near-infrared image stitching data (e.g., generating a rural-pattern composite image, an urban-pattern composite nadir image, and / or an urban-pattern side-tilt image). In some embodiments, the near-infrared image may be generated by the near-infrared camera described above.
[0038] In a further embodiment, the computing unit can be configured to plan one or more flight paths for the vehicle. In some embodiments, the track spacing for flight paths in rural patterns can be determined by the width of the composite image of the rural pattern, while the track spacing for flight paths in urban patterns can be determined by the width of the composite nadir image of the urban pattern and / or the width of the side-tilt image of the urban pattern. It should be noted that track spacing refers to the distance between adjacent tracks within a flight path. In some embodiments, adjacent tracks within a flight path can be parallel to each other.
[0039] The track spacing in the rural mode can be greater than that in the urban mode.
[0040] In a further embodiment, the computing unit may be configured to (e.g., continuously in real time) determine the roll angle of the aerial imaging device and, based on the roll angle, define a sub-field of view symmetrical about the nadir direction in the transverse direction of the combined continuous total field of view. It should be noted that this sub-field of view is smaller than the combined continuous total field of view provided by multiple cameras. In some embodiments, when generating a rural-pattern composite image or an urban-pattern composite nadir image, at least a portion of the combined continuous total field of view not included in this sub-field of view may be ignored.
[0041] In some implementations, the computing unit may be configured to stitch together multiple composite images of rural patterns, composite images of urban patterns with nadir, or composite images of urban patterns with side tilt to generate stripes of the target region.
[0042] It should be noted that the aircraft's heading and track direction may differ (e.g., due to crosswinds). However, in describing this device, it is assumed that the aircraft's actual heading and track direction are consistent.
[0043] In some embodiments, roll angle compensation can be performed during the operation of the device. Therefore, roll angle compensation can be performed individually for each image based on the measured roll angle of the vehicle when each image is captured. In other embodiments, roll angle compensation can be performed during post-processing: either by processing each image individually in a manner similar to that described above, or by simultaneously compensating for multiple images. Therefore, image data can also be transmitted to different computing devices (such as cloud computing services or server clusters) within the computing unit for processing; this transmission can occur during image acquisition or after the aerial imaging device returns to the ground. Since compensation during device operation does not require saving all image data, this method reduces the amount of image data that needs to be transmitted and saved compared to post-processing.
[0044] In aerial imaging, ground sampling distance refers to the distance between the centers of pixels in a ground digital image. The distance between adjacent pixels determines the image resolution: the greater the distance, the lower the resolution. Since ground sampling distance determines both strip width and imaging efficiency, a proper trade-off must be struck between these two factors. The ground sampling distance can be adjusted by selecting an appropriate flight altitude.
[0045] In addition to enabling the same device to adapt to multiple application scenarios, this invention also has other advantages: due to its more flexible component layout, this invention can design a more compact system than traditional systems, thus making it suitable for relatively small aircraft or unmanned aerial vehicles (UAVs). Furthermore, the price of small image sensors is much lower than that of large image sensors, thereby significantly reducing the overall system cost.
[0046] Furthermore, because this invention supports a variety of different component layouts, components can be selected from a wider range of products, allowing for the use of relatively inexpensive components (while components in conventional equipment are typically custom-made). Therefore, this invention can reduce the production cost of aerial imaging equipment.
[0047] Furthermore, the aerial imaging device of the present invention can be installed on a stabilization system to compensate for the motion of the aircraft.
[0048] The present invention also relates to a computer program comprising program code having computer-executable instructions, which, when executed by a computer, cause the computer to perform the following steps:
[0049] (i) Reading images provided by an aerial imaging device configured to acquire aerial imagery of a target area and comprising a plurality of cameras, each having a camera field of view, the cameras being arranged and configured to have different observation directions in the device's lateral direction (corresponding to a cross-track direction perpendicular to the flight direction of the aerial imaging device), the cameras having overlapping fields of view such that, in the device's lateral direction, these cameras collectively provide a combined continuous total field of view of at least 75 degrees (e.g., at least 90 degrees), which is greater than the individual field of view of each camera; and
[0050] (ii) The images are stitched together to generate: a rural-mode composite image, which is generated by stitching together images from a portion of multiple cameras and provides a field of view that is only a part of the total field of view; a urban-mode composite nadir image and an urban-mode side-tilted image, wherein the urban-mode composite nadir image is generated by stitching together images from a portion of multiple cameras and provides a field of view that is only a part of the field of view of the rural-mode composite image, and wherein the side-tilted image provides a field of view that is only a part of the total field of view and is tilted more in the observation direction (to the left or right) than the urban-mode composite nadir image.
[0051] Specifically, the computer program contains computer-executable instructions that enable the computing unit of the system to perform any computational steps related to the system. Attached Figure Description
[0052] The following will describe in more detail the specific embodiments of the present invention by way of example only, in conjunction with the accompanying drawings, wherein:
[0053] Figure 1 A schematic diagram shows an aerial imaging device mounted on a vehicle continuously imaging a target area below;
[0054] Figure 2 A schematic diagram is shown of an aerial imaging device mounted on a vehicle flying along the flight path.
[0055] Figure 3 A schematic diagram of multiple cameras with partially overlapping fields of view is shown;
[0056] Figure 4 A schematic diagram of the combined continuous total field of view formed by three cameras is shown;
[0057] Figure 5 A schematic diagram illustrating an exemplary combined continuous total field of view of multiple cameras is shown;
[0058] Figure 6 It shows Figure 5The illustration shows an exemplary combination of a continuous total field of view, additionally including a forward-tilting camera and a backward-tilting camera;
[0059] Figure 7 It shows Figure 6 An exemplary field of view, and a schematic diagram that additionally includes a near-infrared field of view;
[0060] Figure 8a , Figure 8b , Figure 8c The image shows a comparison between the actual ground field of view of the camera and the field of view virtually generated based on the flight path direction. The virtual field of view is generated by selecting specific cameras and is used to provide synthetic images in rural mode, synthetic nadir images in urban mode, and side tilt images in urban mode, respectively.
[0061] Figure 9 This diagram illustrates multiple images that require compensation for different roll angles of the vehicle.
[0062] Figure 10 An exemplary camera layout diagram of an aerial imaging device mounted on a vehicle is shown;
[0063] Figure 11 Another exemplary camera layout diagram of an aerial imaging device is shown;
[0064] Figure 12 Another exemplary camera layout diagram of an aerial imaging device is shown;
[0065] Figure 13 A schematic diagram illustrating the effect of a barrel lens on ground sampling distance (as it varies with the cross-track angle) is shown.
[0066] Figure Labels
[0067] 1. Aerial Imaging Equipment
[0068] 2. Vehicle
[0069] 3 Flight Path
[0070] 4. Flight path direction
[0071] 5. Track spacing
[0072] 6. Nadir direction
[0073] 10 strip width
[0074] 30-combination continuous total field of view
[0075] 31 Nadir Field
[0076] 32 Side-tilted field of view
[0077] 33 Forward tilt field of view
[0078] 34 Backward tilt field of view
[0079] 35 Near-infrared field of view for each near-infrared camera
[0080] Combined field of view of 46 infrared cameras
[0081] 47. Overlapping region of near-infrared field of view
[0082] Overlapping area of the overlapping fields of view of more than 48 cameras
[0083] 50 strips
[0084] 51 Images based on combined continuous total field of view
[0085] More than 52 images
[0086] 53 Lateral end
[0087] 54. Overlapping portions of consecutive images
[0088] 55. Overlapping portions of adjacent stripes
[0089] 100 Navel Cameras
[0090] 110 Side-tilted camera
[0091] 130 Forward-tilting camera
[0092] 140° rearward tilt camera
[0093] 150 Near Infrared Camera Detailed Implementation
[0094] Figure 1 The illustration shows a typical aerial survey of the terrain being flown over, using an aerial imaging device 1 mounted on a vehicle 2 (such as a fixed-wing aircraft).
[0095] For example, an aerial imaging camera can achieve a field of view of 45 degrees in the lateral direction of the device, providing a large combined continuous total field of view 30 in the lateral direction perpendicular to the flight direction. The lateral direction of the device refers to the direction perpendicular to the flight direction when the camera is mounted on the aircraft. During the measurement process, the terrain being flown over is continuously captured by the camera layout, with strips 50 of the ground being captured as the aircraft flies forward.
[0096] In the prior art, imaging sensors with large sensor surface sizes are typically used. These sensors are usually designed and customized specifically for aerial surveying and related applications (and are therefore typically expensive). For example, the sensor surface size of sensors used in the prior art is at least 60 mm in the lateral direction of the device. In contrast, the aerial imaging device of the present invention can be equipped with multiple more conventional (e.g., less expensive) sensors (e.g., sensors with smaller sensor surface sizes than those commonly used in aerial surveying), while still providing a lateral field of view comparable to or even larger than that of the prior art.
[0097] Figure 2 The illustration shows a scenario where a vehicle 2 carrying an aerial imaging device flies along a flight path 3 in a track direction 4. To achieve continuous coverage of the target area, adjacent strips 50 may partially overlap 55, therefore the track spacing 5 between adjacent tracks may differ from the strip width 10. Furthermore, the strip width depends on the width of the combined continuous total field of view 30 in the device's lateral direction (defined as the direction perpendicular to the track direction 4, i.e., the direction perpendicular to the aircraft's forward direction when the camera is mounted on the aircraft).
[0098] Flight path 3 can be planned before or during operation. Typically, flight paths are designed with alternating directions of adjacent tracks to minimize wasted flight time. In other words, such flight paths can be described as serpentine or meandering. However, those skilled in the art should understand that there are many possibilities for flight path design.
[0099] During flight along the flight path 4, the camera continuously captures images. The trigger frequency for capturing images may depend on the vehicle's flight speed relative to the ground, and the selection of the trigger frequency must ensure that there is partial overlap in the continuously captured images 54.
[0100] For example, imaging can be paused when a vehicle turns or undergoes other directional changes that may result in a large roll angle. However, in other situations, aerial imaging equipment can continue operating during turns.
[0101] Figure 3 An aerial imaging device 1 mounted on a vehicle 2 is shown. Cameras are arranged to partially overlap their fields of view to form a combined continuous total field of view 30. Depending on the operating mode, different synthetic images can be generated using images or image segments captured by multiple cameras. Specifically, images and / or image segments used to generate rural-mode synthetic images, urban-mode synthetic nadir images, and urban-mode side-tilt images are provided by corresponding camera groups, and the camera groups used to generate these three types of images are different from each other.
[0102] For example, a multi-camera setup may include seven cameras: five cameras whose fields of view are closest to the nadir view form a city-mode nadir group to generate a city-mode synthetic nadir image, the nadir field of view 31 of which is larger than the field of view of a single camera involved in imaging; two cameras each with their fields of view located at the far left and far right form a city-mode tilt group to generate city-mode synthetic side tilt images on the left and right sides (e.g., the side tilt field of view 32 of each synthetic side tilt image is larger than the field of view of a single camera in the city-mode tilt group). In rural mode, all cameras in the multi-camera setup are used to form a rural mode group to generate a rural mode synthetic image whose field of view substantially covers the full field of view provided by the multi-camera setup (typically, the field of view of the rural mode synthetic image is slightly smaller than the combined continuous total field of view 30 provided by the cameras, for example, as described below). Figure 9 The reserved space for roll angle correction mentioned above.
[0103] The combined continuous total field of view 30 can cover an angle of at least 100 degrees in the roll direction.
[0104] Figure 4 Multiple cameras are shown, their fields of view overlapping to form a combined continuous total field of view 30. It should be noted that the projected size of the field of view onto the virtual plane (also known as the footprint) varies with the angle of the camera's optical axis relative to the nadir and the distance of the camera from the virtual plane. For example, dashed lines represent areas of equal area within each camera's field of view. Thus, at the same distance, the footprint area captured by a camera facing an angled direction may be larger than that captured by a camera facing the nadir. It should also be noted that due to potential obstructions, obstacles, or holes, not all areas within the footprint may be clearly visible in the images captured by the cameras.
[0105] In other words, for example, the dotted lines define a plane of equal area for each camera's field of view. It is clear from this that, in this example, the effective footprint area of the tilting camera is larger than that of the nadir camera.
[0106] Figure 5An exemplary combined continuous total field of view 30 consisting of multiple camera units (with partially overlapping fields of view) is shown. For example, the city-mode nadir group comprises five cameras: three cameras with their fields of view closest to the nadir operate in a longitudinal mode, while one camera each with its field of view slightly to the left and right operates in a lateral mode; the city-mode tilt group comprises two cameras each with their fields of view located at the far left and far right, both operating in a lateral mode (therefore, the cameras located on the inner left and inner right sides of the city-mode tilt group belong to both the city-mode nadir group and the city-mode tilt group). In the lateral direction of the device, the ground fields of view of these cameras partially overlap. Due to the perspective effect of the cameras (i.e., the camera optical axes are tilted relative to the nadir direction), perspective distortion gradually increases towards both ends of the lateral direction.
[0107] Figure 6 It shows the relationship with Figure 5 The corresponding exemplary combined continuous total field of view, wherein the city-mode tilt group further includes cameras for generating a forward-tilted field of view 33 and a backward-tilted field of view 34. For example, in the flight path, two cameras are positioned forward and two cameras are positioned backward, all using a longitudinal mode, and their fields of view partially overlap. Similarly, perspective distortion increases with the angle to the nadir direction, thus increasing the ground sampling distance and reducing resolution.
[0108] Figure 7 It shows the relationship with Figure 6 The corresponding exemplary combined continuous field of view additionally includes the overlapping near-infrared fields of view 35 of two near-infrared cameras. For example, the near-infrared cameras are in a horizontal configuration. The near-infrared cameras are arranged such that the overlapping region 47 of their near-infrared fields of view does not coincide with the overlapping region 48 of the fields of view of multiple (e.g., color or RGB) cameras used to form the combined continuous field of view.
[0109] Figure 8a , Figure 8b , Figure 8c The illustration schematically shows a comparison between the actual ground field of view from multiple cameras and the field of view virtually generated based on the flight path direction 4 of vehicle 2. The virtual field of view is generated through specific camera selections and is used to provide composite images in rural mode, composite nadir images in urban mode, and urban side-tilt images (see also the above). Figure 5 (Description).
[0110] Figure 8a The individual (actual) fields of view provided by the multiple cameras included in the aerial imaging device of the present invention are depicted, with overlapping areas of the fields of view indicated by shading.
[0111] Figure 8bThe diagram depicts the nadir field of view 31 of a composite nadir image generated by the inner group (i.e., the camera whose field of view is closest to the nadir) of multiple cameras used in city mode. In the example shown, this city mode nadir group comprises five cameras whose fields of view are symmetrically arranged around the nadir direction. To the left and right of the nadir field of view 31, two side-tilted fields of view 32 provided by the composite side-tilted images are shown, respectively. The composite side-tilted images here are generated by two cameras each located at the leftmost and rightmost edges of the field of view; these cameras are referred to as the city mode tilt group. Therefore, the cameras located at the inner left and inner right edges of the city mode tilt group belong to both the city mode nadir group and the city mode tilt group. In other words, the side-tilted fields of view 32 are oriented further outward in the device's lateral direction than the city mode nadir field of view 31.
[0112] Figure 8c Possible groupings of multiple cameras used in a rural mode are depicted. In this example, the rural mode group includes all of the multiple cameras, i.e., the field of view of the rural mode composite image substantially corresponds to the combined continuous total field of view 30 provided by the multiple cameras (typically, the field of view of the rural mode composite image is only slightly smaller than the combined continuous total field of view 30 provided by the multiple cameras).
[0113] Figure 9 An example of multiple composite images generated by the aerial imaging device of the present invention is shown, which can be stitched together to form a strip.
[0114] For example, the width of each individual image generated based on the combined continuous total field of view can be individually compensated according to the roll angle of the vehicle. Therefore, a sub-field of view can be selected from each combined continuous total field of view (ignoring at least a portion of the combined continuous total field of view, such as the lateral end 53) such that the final generated image (and the final strip 50) is symmetrical with respect to the nadir direction.
[0115] As another example, the final stripe 50 (or at least a portion thereof) can be compensated for by ignoring the lateral ends 53 of individual images generated based on the combined continuous total field of view 51, based on the lateral offset of other combined continuous total field of view 51. In some embodiments, this compensation process can be calculated in post-processing. In other embodiments, it can be adjusted in real time during device operation.
[0116] Figure 10An exemplary camera layout for the aerial imaging device 1 is shown. The cameras are primarily arranged along the intended flight path 4 of the vehicle 2. Each camera can be triggered sequentially with a certain time delay to compensate for positional offsets along the flight path 4. In other words, the triggering timing of a camera must ensure that its triggering position coincides with the triggering position of the adjacent camera that was triggered first along the flight path. It should be noted that the actual camera layout may not be entirely determined by the corresponding camera group. However, in some embodiments, cameras within the same camera group may be arranged sequentially along the flight path.
[0117] It should be noted that, in addition to the examples shown, there are many other possible installation designs for the camera layout of the present invention. In some embodiments, external mounting can be used, for example, by using non-invasive clamps to mount the camera on an existing structure (such as a strut bracket or helicopter nose). In other embodiments, the aerial imaging equipment can be installed inside a vehicle, in which case the vehicle needs to be equipped with a transparent shell or opening (such as a hatch) to allow the equipment to operate.
[0118] Figure 11 Another exemplary camera layout of the aerial imaging device 1 is shown (relative to the expected flight path direction 4). Dashed lines indicate the assignment of each camera to multiple camera units.
[0119] For example, Figure 11 The illustrated camera layout includes a city-mode nadir group comprising five nadir cameras 100 arranged in a straight line along the flight path. These cameras have slightly different observation directions in the device's lateral direction, thus their ground fields of view overlap in the device's lateral direction. To improve image alignment and maximize the field of view in the flight path, these cameras can be triggered sequentially with a time delay to acquire images from the same axial position along the flight path. Furthermore, the layout also includes a city-mode tilt group comprising two side-tilt cameras 110, two forward-tilt cameras 130, and two backward-tilt cameras 140. The fields of view of the side-tilt cameras symmetrically overlap with the fields of view of the city-mode nadir group at the left and right ends in the device's lateral direction. The combined fields of view of the city-mode nadir group and the city-mode tilt group form a combined continuous total field of view for the camera layout. The fields of view of the forward-tilt and backward-tilt cameras are offset forward and backward, respectively, and the fields of view of every two cameras are laterally adjacent. Additionally, the layout also includes two near-infrared cameras 150. This exemplary layout can be used to form... Figure 7 The field of view is shown. In another example, all cameras can be arranged in a straight line or in some other layout to minimize lateral offset.
[0120] Figure 12Another exemplary camera layout for an aerial imaging device 1 mounted on a vehicle is shown from below. In this example, the aerial imaging device is configured to be mounted within a spherical housing. This relatively compact design makes the device smaller in size compared to other camera layouts.
[0121] Figure 13 The effect of a barrel lens on ground sampling distance (varying with cross-track angle) was qualitatively described. The ground sampling distance increases with the angle to the nadir. For example, with a distortion-free lens, the ground sampling distance may exhibit discontinuous abrupt changes with angle. Lenses that produce barrel distortion can make the change in ground sampling distance smoother. Furthermore, lenses that produce barrel distortion can also increase the field of view of the camera.
[0122] Although the present invention has been described above in conjunction with some specific embodiments, it should be understood that various modifications and combinations can be made to different features of the embodiments, and all such modifications are within the protection scope of the appended claims.
Claims
1. A system comprising an aerial imaging device (1) configured to be mounted on a vehicle (2) configured to fly along a flight path (4) and at a flight altitude, the aerial imaging device (1) configured to acquire aerial images of a target area and comprising a plurality of cameras, each having a camera field of view, the cameras being arranged and configured to have different observation directions in the lateral direction of the device, the lateral direction of the device corresponding to a cross-flight direction perpendicular to the flight path (4), Its features are, • The cameras among the plurality of cameras have overlapping fields of view, such that, in the lateral direction of the device, the cameras collectively provide a combined continuous total field of view (30) of at least 75 degrees, particularly at least 90 degrees, which is greater than the individual field of view of each of the cameras. • The system includes a computing unit configured to stitch together images from one of the plurality of cameras to generate a rural-pattern composite image, which is generated by stitching together images from some of the plurality of cameras and provides a field of view in the lateral direction of the device, which is only a portion of the total field of view (30), and The city-pattern composite nadir image and the city-pattern side-tilted image, wherein the city-pattern composite nadir image is generated based on image stitching from some of the multiple cameras and provides a field of view (31) in the lateral direction of the device, which is only a part of the field of view of the rural composite image, and wherein the city-pattern side-tilted image provides a field of view (32) in the lateral direction of the device, which is only a part of the total field of view and has a more tilted observation direction than the city-pattern composite nadir image.
2. The system according to claim 1, wherein, The computing unit is configured to provide a. In the rural mode, the synthesized rural mode image is generated from images from a group of cameras, i.e., the rural mode group, among the plurality of cameras, and b. In the urban mode, the urban mode synthesized nadir image is generated from images of another set of cameras among the plurality of cameras, namely the urban mode nadir group, wherein the urban mode nadir group is different from the rural mode group, and c. In the urban mode, the urban mode side-tilt image is generated from images of another group of cameras among the plurality of cameras, namely the urban mode tilt group, wherein the urban mode tilt group is different from both the urban mode nadir group and the rural mode group. In particular, the side-tilt image is generated based on one or more image segments of one or more images from one or more cameras provided by the urban mode tilt group. In particular, in the lateral direction of the device, the field of view orientation of the cameras in the urban mode tilt group deviates from the nadir (6) to a greater extent than the field of view orientation of the cameras in the urban mode nadir group deviates from the nadir. In particular, the plurality of cameras includes at least five cameras, and more particularly at least seven cameras.
3. The system according to claim 2, wherein, The rural mode group contains more cameras than the urban mode nadir group, and in particular, the rural mode group contains cameras selected from the plurality of cameras that provide the combined continuous total field of view in the lateral direction of the device.
4. The system according to any one of the preceding claims, wherein the combined continuous total field of view (30) covers an angle in the lateral direction of the device, wherein, a. For composite images in a rural pattern, the angle corresponds to at least 70 degrees, and more particularly at least 100 degrees; and b. For urban-patterned synthetic nadir images, the angle corresponds to less than 80 degrees, especially less than 60 degrees.
5. The system according to any one of the preceding claims, wherein, The aerial imaging device further includes, and in particular, the plurality of cameras. a. At least one camera configured to generate a forward tilted field of view (33), particularly comprising two or more cameras arranged such that their fields of view are laterally adjacent and partially overlap; as well as b. At least one camera configured to generate a backward tilted field of view (34), particularly comprising two or more cameras arranged such that their fields of view are laterally adjacent and partially overlap; The computing unit is further configured to provide forward tilt and backward tilt images for the city model, and in particular, the combined continuous total field of view (31) of the city model nadir group in the device lateral direction covers at least the same angle in the device lateral direction as the continuous total field of view of the forward tilt or backward tilt in the device lateral direction.
6. The system according to any one of the preceding claims, wherein, One or more of the plurality of cameras are in a horizontal mode, while another of the plurality of cameras is in a vertical mode, wherein the field of view of the horizontal mode camera is rotated by 90 degrees relative to the field of view of the vertical mode camera.
7. The system according to any one of the preceding claims, wherein, The aerial imaging device (1) further includes a near-infrared camera having a near-infrared field of view (35), the near-infrared camera being arranged such that the near-infrared field of view partially overlaps with the combined continuous total field of view (30) of the plurality of cameras, in particular wherein the aerial imaging device includes two near-infrared cameras arranged such that their near-infrared fields of view (35) partially overlap, and more particularly wherein the overlapping region (47) of the near-infrared fields of view (35) does not coincide with any overlapping region (48) of the overlapping camera fields of view of the plurality of cameras.
8. The system according to any one of the preceding claims, wherein, Each of the plurality of cameras is configured to have the same focal length common to all of the plurality of cameras.
9. The system according to claim 7, wherein, The near-infrared camera has a focal length different from that of each of the plurality of cameras, and in particular, where applicable, each of the near-infrared cameras is configured to have the same focal length common to all the near-infrared cameras.
10. The system according to any one of the preceding claims, wherein, At least a subset of the plurality of cameras are arranged in a straight line along the flight path, wherein the cameras in the subset are triggered sequentially with a certain time delay, and even more particularly wherein the cameras are triggered according to a function determined by the flight speed of the vehicle.
11. The system according to any one of the preceding claims, wherein, Each camera in at least a subset of the plurality of cameras contains a lens that produces barrel lens distortion.
12. The system according to any one of the preceding claims, wherein, The computing unit is configured to use near-infrared image stitching data, particularly near-infrared images generated by the near-infrared camera according to claim 7.
13. The system according to any one of the preceding claims, wherein, The computing unit is configured to plan one or more flight paths for the vehicle, wherein the track spacing (5) for the flight path (3) of the rural mode is determined by the width (10) of the composite image of the rural mode, while the track spacing for the flight path of the urban mode is determined by the width of the composite nadir image of the urban mode and / or the width of the side tilt image of the urban mode, wherein the track spacing refers to the distance between adjacent tracks in the flight path.
14. The system according to any one of the preceding claims, wherein, The computing unit is configured to determine, in particular, the roll angle of the aerial imaging device continuously, and based on the roll angle, define a sub-field of view that is symmetrical in the transverse direction of the device relative to the nadir direction (6) in the combined continuous total field of view. The sub-field of view is smaller than the combined continuous total field of view. In particular, when generating a rural pattern composite image or an urban pattern composite nadir image, at least a portion of the combined continuous total field of view that is not included in the sub-field of view is ignored (53).
15. A computer program comprising program code having computer-executable instructions, wherein when a computer executes the program, the computer-executable instructions cause the computer to perform the following steps: • Read images provided by an aerial imaging device (1), which is configured to acquire aerial images of a target area and includes multiple cameras, each with a field of view, the cameras being arranged and configured to have different observation directions in the lateral direction of the device, the lateral direction of the device corresponding to a cross-track direction perpendicular to the flight direction (4) of the aerial imaging device (1), wherein, The cameras among the plurality of cameras have overlapping fields of view, such that, in the lateral direction of the device, these cameras collectively provide a combined continuous total field of view of at least 75 degrees, particularly at least 90 degrees, which is greater than the individual field of view of each camera. • The images are stitched together to generate a rural-pattern composite image, which is generated by stitching together images from some of the plurality of cameras, and provides a field of view that is only a portion of the total field of view, and The city-pattern composite nadir image and the city-pattern side-tilted image, wherein the city-pattern composite nadir image is generated based on image stitching from some of the plurality of cameras and provides a field of view that is only a portion of the field of view of the city-pattern composite image, and wherein the side-tilted image provides a field of view that is only a portion of the total field of view and has a more tilted viewing direction than the city-pattern composite nadir image.