Flood danger multi-source data processing method, sensor load system and electronic equipment

Abstract

Embodiments of the present application disclose a flood disaster multi-source data processing method, a sensor load system and an electronic device. The method comprises: obtaining a thermal infrared orthographic image, a point cloud data set and a plurality of hyperspectral images of a flood disaster area, the plurality of hyperspectral images corresponding to a plurality of sets of pose information one-to-one, the pose information being used to represent the pose of a hyperspectral sensor when collecting images of the flood disaster area at a corresponding collection time; generating a first digital surface model of the flood disaster area based on the point cloud data set; respectively performing geometric correction on the plurality of hyperspectral images by using the first digital surface model and the plurality of sets of pose information to generate hyperspectral orthographic images; and performing data fusion on the thermal infrared orthographic image, the hyperspectral orthographic images and the first digital surface model to obtain a target image. This scheme can integrate spectral information, thermal radiation information and elevation information to obtain a target image containing multi-dimensional information.

Description

Technical Field

[0001] This invention relates to the field of dike inspection technology, and more specifically to a method for processing multi-source data on flood risks, a sensor payload system, electronic equipment, storage media, and computer program products. Background Technology

[0002] In recent years, drones, as a flexible and efficient remote sensing platform, have gradually become an important tool in dike patrol. Drones can quickly cover large areas of flood-prone areas, equipped with various types of sensors such as visible light sensors, thermal infrared sensors, and lidar sensors, providing real-time, high-precision data for flood risk monitoring. However, although existing drone platforms can carry multiple sensors for data acquisition, single optical devices (even combinations of thermal infrared and visible light sensors) still have significant limitations and cannot meet the comprehensive needs of flood risk monitoring. Visible light sensors can provide visual information about the surface morphology of flood-prone areas, while thermal infrared sensors can identify temperature anomalies in these areas, primarily used to detect surface seepage or structural damage. However, the combination of these two still cannot effectively capture potential risks within flood-prone areas, such as hidden dangers like groundwater activity, soil moisture changes, or seepage within the dike. Many potential risks in flood-prone areas are very small in the early stages and are often difficult to identify by changes in surface temperature or appearance. Therefore, existing sensor combinations and the way the collected data is processed lack sufficient depth and comprehensiveness when faced with the complex structure of flood-prone areas and the diverse risks. Summary of the Invention

[0003] This invention addresses the aforementioned problems. It provides a method for processing multi-source data on flood and waterlogging emergencies, a sensor payload system, electronic equipment, a storage medium, and a computer program product. This approach fuses hyperspectral image data, point cloud data, and thermal infrared image data, thereby integrating spectral information, thermal radiation information, and elevation information to obtain a target image containing multi-dimensional information.

[0004] According to one aspect of the present invention, a method for processing multi-source data on flood risks is provided. The method includes: acquiring thermal infrared orthophoto images, point cloud datasets, and multiple hyperspectral images of a flood risk area; each pixel of the thermal infrared orthophoto image having thermal radiation information representing the thermal radiation at the corresponding physical location; each set of point cloud data in the point cloud dataset including first location information representing the corresponding physical location; each pixel of each of the multiple hyperspectral images having spectral information corresponding to the physical location; and multiple hyperspectral images corresponding one-to-one with multiple sets of pose information, the pose information representing the pose of the hyperspectral sensor when acquiring images of the flood risk area at a corresponding acquisition time; generating flood data based on the point cloud dataset. The first digital surface model of the flood-prone area includes the first elevation information of each physical location of the surface features in the flood-prone area. Multiple hyperspectral images are geometrically corrected using the first digital surface model and multiple sets of pose information to generate a hyperspectral orthophoto image based on the geometrically corrected hyperspectral images. Each pixel in the hyperspectral orthophoto image has spectral information corresponding to its physical location. The thermal infrared orthophoto image, the hyperspectral orthophoto image, and the first digital surface model are fused to obtain a target image. Each pixel in the target image has elevation information, spectral information, and thermal radiation information corresponding to its physical location.

[0005] Optionally, acquiring thermal infrared orthophotos includes: acquiring multiple thermal infrared images collected by a thermal infrared sensor on a UAV targeting a flood-prone area; generating sparse three-dimensional point cloud data using multiple sets of exterior orientation parameters corresponding one-to-one with the multiple thermal infrared images and interior orientation parameters of the thermal infrared sensor, wherein each set of exterior orientation parameters is used to indicate the pose of the thermal infrared sensor when acquiring the corresponding thermal infrared image, and the interior orientation parameters include focal length, principal point position parameters, and distortion coefficients; generating high-density three-dimensional point cloud data based on the sparse three-dimensional point cloud data and multiple thermal infrared images using a multi-view stereo vision dense matching algorithm, and generating a second digital surface model based on the high-density three-dimensional point cloud data; mapping the pixel values ​​of the multiple thermal infrared images to the corresponding surface positions of the second digital surface model to obtain thermal infrared orthophotos, wherein thermal radiation information is represented by pixel values.

[0006] Optionally, before generating high-density three-dimensional point cloud data based on sparse three-dimensional point cloud data and multiple thermal infrared images, acquiring thermal infrared orthophotos of the flood-prone area further includes: determining the image positions of preset ground control points in multiple thermal infrared images, and optimizing the sparse three-dimensional point cloud data using the image positions of the ground control points and employing an overall adjustment method; wherein, the operation of generating high-density three-dimensional point cloud data based on sparse three-dimensional point cloud data and multiple thermal infrared images is performed on the optimized sparse three-dimensional point cloud data; and / or, generating sparse three-dimensional point cloud data using multiple sets of exterior orientation parameters corresponding one-to-one with multiple thermal infrared images and interior orientation parameters of thermal infrared sensors, including: extracting and matching feature points of multiple thermal infrared images using a motion reconstruction structure algorithm, and determining multiple sets of exterior orientation parameters and interior orientation parameters based on the matching results; Based on multiple sets of exterior and interior orientation parameters, a spatial topological relationship is established between multiple thermal infrared images to obtain sparse three-dimensional point cloud data; and / or, each surface position of the second digital surface model has second position information for representing the corresponding physical position, each thermal infrared image of the multiple thermal infrared images corresponds to a part of the surface region of the second digital surface model, and the pixel values ​​of the multiple thermal infrared images are mapped to the corresponding surface positions of the second digital surface model to obtain thermal infrared orthophotos, including: for each thermal infrared image of the multiple thermal infrared images, establishing a first collinearity equation based on the second position information of the second digital surface model, the exterior and interior orientation parameters corresponding to the thermal infrared image; determining the image position of at least a part of the surface region on the second digital surface model corresponding to the thermal infrared image in the thermal infrared image based on the first collinearity equation; for each surface position of the second digital surface model, mapping the pixel value of the target thermal infrared image located at the image position corresponding to the surface position to the surface position to obtain thermal infrared orthophotos, wherein the target thermal infrared image is the thermal infrared image corresponding to the surface region to which the surface position belongs.

[0007] Optionally, the plurality of thermal infrared images includes at least one pair of thermal infrared images, and the field of view of each pair of thermal infrared images partially overlaps. After mapping the pixel values ​​of the plurality of thermal infrared images to the corresponding surface positions of the second digital surface model to obtain thermal infrared orthophotos, obtaining thermal infrared orthophotos further includes: performing post-processing operations on the target image region of the thermal infrared orthophotos to obtain the final thermal infrared orthophotos. The post-processing operations include color balancing and mosaicking operations. The target image region includes the image region corresponding to the overlapping area of ​​the field of view of at least one pair of thermal infrared images.

[0008] Optionally, each surface position of the first digital surface model has third position information representing the corresponding physical position. The third position information includes first elevation information. Geometric correction is performed on multiple hyperspectral images using the first digital surface model and multiple sets of pose information to generate a hyperspectral orthophoto based on the geometrically corrected hyperspectral images. This includes: for each hyperspectral image of the multiple hyperspectral images, constructing a second collinearity equation based on the third position information of the first digital surface model, the pose information corresponding to the hyperspectral image, and the interior orientation parameters of the hyperspectral sensor; for each hyperspectral image of the multiple hyperspectral images, based on the graph of each pixel of the hyperspectral image... Based on the image position and the second collinearity equation, the surface position on the first digital surface model corresponding to each pixel is determined; based on the third position information of the surface position corresponding to each pixel of the hyperspectral image, the pixel value of each pixel of the hyperspectral image is mapped to a preset raster image corresponding to the first digital surface model, and a resampling algorithm is used to determine the pixel value of each grid of the preset raster image to obtain an orthophoto image corresponding to the hyperspectral image, wherein each image position of the preset raster image corresponds to at least a part of the surface position of the first digital surface model; multiple orthophoto images corresponding one-to-one with multiple hyperspectral images are stitched together to obtain a hyperspectral orthophoto image.

[0009] Optionally, each pixel of each of the multiple orthophoto images has fourth position information representing the corresponding physical location. The multiple orthophoto images, each corresponding to a different hyperspectral image, are stitched together to obtain a hyperspectral orthophoto image. This includes: aligning the multiple orthophoto images according to the fourth position information and determining the overlapping regions of the multiple orthophoto images; performing spectral consistency processing on the overlapping regions; determining the stitching lines of the multiple orthophoto images and stitching them together according to the stitching lines to obtain a stitched image; and feathering the image region containing the stitching lines in the stitched image to obtain a hyperspectral orthophoto image.

[0010] Optionally, acquiring multiple hyperspectral images includes: acquiring multiple hyperspectral datasets collected by a hyperspectral sensor for flood-prone areas, each hyperspectral dataset including multiple sets of hyperspectral data corresponding one-to-one with multiple sensing units of the hyperspectral sensor, and multiple hyperspectral datasets corresponding one-to-one with multiple sets of pose information; for each hyperspectral dataset, performing radiometric calibration on each set of hyperspectral data to obtain a radiance image of the hyperspectral dataset, each pixel of the radiance image having a radiance value; acquiring a reference radiance image, which is an image collected by the hyperspectral sensor for a calibration blanket in the flood-prone area; determining the reflectance of multiple radiance images corresponding one-to-one with the multiple hyperspectral datasets in at least one band based on the standard reflectance of the calibration blanket and the radiance value of the reference radiance image; generating multiple hyperspectral images based on the reflectance of each of the multiple radiance images, each pixel of each hyperspectral image also having reflectance information corresponding to the physical location, the reflectance information being used to indicate the reflectance of the corresponding physical location in at least one band.

[0011] Optionally, acquiring a point cloud dataset includes: acquiring multiple sets of raw point cloud data collected by a lidar sensor on a drone targeting a flood-prone area; filtering the multiple sets of raw point cloud data to obtain multiple sets of filtered point cloud data; merging the multiple sets of filtered point cloud data in a preset coordinate system, wherein each set of filtered point cloud data has a coordinate range in the preset coordinate system, and the coordinate range of each set of filtered point cloud data intersects with the coordinate range of at least one other set of filtered point cloud data; and performing a point density-weighted average operation on the filtered point cloud data located within the intersection to obtain a point cloud dataset.

[0012] Optionally, generating a first digital surface model of a flood-prone area based on a point cloud dataset includes: extracting third elevation information from the point cloud dataset, the third elevation information being used to represent the elevation of the corresponding physical location of the surface features in the flood-prone area; determining the grid resolution according to preset accuracy requirements; rasterizing the point cloud dataset based on the grid resolution; and determining the fourth elevation information of each raster cell based on the third elevation information using a preset interpolation algorithm, the fourth elevation information being used to represent the elevation of the physical location corresponding to the raster cell; and generating a first digital surface model based on the fourth elevation information of each raster cell.

[0013] Optionally, the thermal infrared orthophoto, hyperspectral orthophoto, and first digital surface model are fused to obtain a target image, including: fusing the hyperspectral orthophoto with the first digital surface model to obtain an intermediate fused image; and fusing the intermediate fused image with the thermal infrared orthophoto to obtain the target image.

[0014] Optionally, each surface location of the first digital surface model has third location information representing its corresponding physical location, including first elevation information. Each pixel of the hyperspectral orthophoto image also has fifth location information representing its corresponding physical location. The hyperspectral orthophoto image and the first digital surface model are fused to obtain an intermediate fused image, including: registering the hyperspectral orthophoto image and the first digital surface model based on the fifth location information of the hyperspectral orthophoto image and the third location information of the first digital surface model; and, based on the registration result of the hyperspectral orthophoto image and the first digital surface model, transferring the first elevation information of the first digital surface model... Embedding additional bands into a hyperspectral orthophoto image to obtain an intermediate fused image; and / or fusing the intermediate fused image with a thermal infrared orthophoto image to obtain a target image, including: determining key feature points of the intermediate fused image and the thermal infrared orthophoto image respectively; matching the key feature points of the intermediate fused image with the key feature points of the thermal infrared orthophoto image; registering the intermediate fused image and the thermal infrared orthophoto image using the feature point matching results; and fusing the thermal radiation information of the thermal infrared orthophoto image with the spectral information and first elevation information of the hyperspectral orthophoto image based on the registration results of the intermediate fused image and the thermal infrared orthophoto image to obtain the target image.

[0015] Optionally, the intermediate fused image and the thermal infrared orthophoto image are registered using the feature point matching results, including: constructing a second-order polynomial model using the feature point matching results, the second-order polynomial model being used to represent the image position transformation relationship of converting pixels of the thermal infrared orthophoto image to the intermediate fused image; performing a geometric transformation on the thermal infrared orthophoto image based on the second-order polynomial model; and fusing the thermal radiation information of the thermal infrared orthophoto image with the spectral information and first elevation information of the intermediate fused image according to the registration results of the intermediate fused image and the thermal infrared orthophoto image to obtain the target image, including: resampling the thermal infrared orthophoto image based on the geometric transformation results to obtain a new thermal infrared orthophoto image, wherein at least some pixels of the intermediate fused image correspond one-to-one with at least some pixels of the new thermal infrared orthophoto image; and fusing the spectral information and first elevation information of the intermediate fused image with the thermal radiation information of the new thermal infrared orthophoto image to obtain the target image.

[0016] Optionally, before fusing the intermediate fused image with the thermal infrared orthophoto to obtain the target image, the thermal infrared orthophoto, the hyperspectral orthophoto, and the first digital surface model are fused to obtain the intermediate fused image. This further includes: determining a mask region of the thermal infrared orthophoto based on the image region of the intermediate fused image; cropping the thermal infrared orthophoto using the mask region to obtain a new thermal infrared orthophoto, wherein the image region of the new thermal infrared orthophoto overlaps with the image region of the intermediate fused image; wherein the operation of fusing the intermediate fused image with the thermal infrared orthophoto is an operation performed on both the intermediate fused image and the new thermal infrared orthophoto.

[0017] Optionally, the raw data of thermal infrared orthophotos, point cloud datasets, and multiple hyperspectral images are data collected by the sensor module on the UAV for flood-prone areas. The UAV is equipped with a coupled satellite navigation module and an inertial measurement unit. Before performing geometric correction on the multiple hyperspectral images using the first digital surface model and multiple sets of pose information, the method further includes: acquiring base station observation data, which includes a first observation epoch sequence and multiple sets of satellite observation data corresponding one-to-one with multiple epochs in the first observation epoch sequence; acquiring UAV observation data, which includes a second observation epoch sequence and multiple sets of navigation state data corresponding one-to-one with multiple epochs in the second observation epoch sequence. The navigation state data includes satellite navigation data collected by the satellite navigation module and inertial measurement data measured by the inertial measurement unit. The satellite navigation data is used to indicate the position of the sensor module, and the inertial measurement unit is used to indicate the position of the sensor module. The data is used to indicate the attitude of the sensor module; epoch synchronization of satellite observation data and navigation status data is performed using a preset precise ephemeris and a preset clock difference file; a double-difference observation model is constructed based on the epoch-synchronized base station observation data and UAV observation data; an ambiguity fixing algorithm is used to determine the integer value of the carrier phase integer ambiguity in the double-difference observation model, so as to solve multiple sets of sensor positioning results corresponding one-to-one with multiple sets of satellite navigation data using the double-difference observation model with the determined carrier phase integer ambiguity; a tightly coupled Kalman filter algorithm is used to jointly estimate the multiple sets of sensor positioning results and inertial measurement data to obtain the trajectory data of the sensor module. The trajectory data is used to represent the pose of the sensor module at each acquisition time during the data acquisition process in the flood-prone area; a preset filtering and smoothing algorithm is used to process the trajectory data to obtain the target trajectory data, which includes multiple sets of pose information.

[0018] According to another aspect of the present invention, a sensor payload system for an unmanned aerial vehicle (UAV) is also provided. The sensor payload system includes a sensor module and a power distribution module. The sensor module includes a bracket, a lidar sensor, a hyperspectral sensor, and a thermal infrared sensor mounted on the bracket. The lidar sensor and the hyperspectral sensor are rigidly connected. The lidar sensor, the thermal infrared sensor, and the thermal infrared sensor are arranged along a preset direction. The scanning center of the lidar sensor, the optical center of the hyperspectral sensor, and the optical center of the thermal infrared sensor are aligned with each other to form the same observation baseline. At least a portion of the field of view of each of the lidar sensor, the hyperspectral sensor, and the thermal infrared sensor overlaps with each other. The power distribution module is connected to the sensor module and is used to supply power to the sensor module.

[0019] Optionally, the lidar sensor and hyperspectral camera are mounted on the first power supply circuit, and the thermal infrared sensor is mounted on the second power supply circuit. The power supply module is specifically used to supply power to the first power supply circuit and the second power supply circuit respectively.

[0020] Optionally, the sensor payload system also includes a control module, which is located inside the housing of the sensor module and is communicatively connected to the lidar sensor and the hyperspectral sensor, respectively. The control module, lidar sensor, and hyperspectral sensor are connected to the same power supply circuit. The control module is used to set the operating parameters of the lidar sensor and the hyperspectral sensor, send start commands and / or stop commands to the lidar sensor and the hyperspectral sensor, and receive data sent by the lidar sensor and the hyperspectral sensor.

[0021] Optionally, a first fan is provided on the inner side wall of the bracket. The airflow provided by the first fan can pass through the longitudinal air duct inside the bracket sequentially through the motherboard of the hyperspectral sensor and the motherboard of the control module, and be discharged through the exhaust port near the motherboard of the control module. A second fan is provided inside the thermal infrared sensor for heat dissipation of the thermal infrared sensor.

[0022] Optionally, the sensor payload system also includes a power cable. The power distribution module is electrically connected to the UAV platform via the power cable to obtain the power supply voltage provided by the UAV platform. It is also electrically connected to the lidar sensor, hyperspectral sensor, and thermal infrared sensor via the power cable to supply power to the lidar sensor, hyperspectral sensor, and thermal infrared sensor respectively.

[0023] Optionally, the sensor payload system further includes signal lines, with power lines and signal lines arranged in layers. The sensor payload system also includes a satellite navigation module; the thermal infrared sensor has a GPS module, which is connected to a GPS antenna on the UAV via a first signal line. The GPS module is used to calculate the timestamp information of the thermal infrared sensor during data acquisition based on the GPS signal received by the GPS antenna. The timestamp information of the GPS module is used to align with the trigger timestamp information generated by the control module. The trigger timestamp information is used to indicate the data acquisition time of the lidar sensor and the hyperspectral sensor. And / or, the sensor payload system also includes a trigger module, which is communicatively connected to the control module via a second signal line and also communicatively connected to the lidar sensor and the hyperspectral sensor via the second signal line respectively. The trigger module is used to respond to the trigger control command sent by the control module by sending a trigger signal. The data is sent to the lidar sensor and the hyperspectral sensor, and a trigger event is reported to the control module. The control module is also used to generate trigger timestamp information in response to the trigger event, based on the time reference provided by the satellite navigation module. The trigger timestamp information is used to indicate the data acquisition time of the lidar sensor and the hyperspectral sensor. And / or, the control module is communicatively connected to the lidar sensor and the hyperspectral sensor respectively via a third signal line. The control module is also used to acquire the data acquired by the lidar sensor and the data acquired by the hyperspectral sensor respectively via the third signal line. And / or, the sensor payload system also includes a navigation component, which includes a mutually coupled satellite navigation module and an inertial measurement unit. The navigation component is communicatively connected to the control module via a fourth signal line. The control module is also used to acquire satellite navigation data acquired by the satellite navigation module and inertial measurement data measured by the inertial measurement unit.

[0024] Optionally, the lidar sensor communicates with the control module via a network port; and / or, the hyperspectral sensor communicates with the control module via a PCIe interface; and / or, the sensor payload system further includes a navigation component, which includes a satellite navigation module and an inertial measurement unit coupled to each other, and the navigation component communicates with the control module via a network port and a serial port; and / or, the thermal infrared sensor communicates with the terminal device via Bluetooth.

[0025] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the above-described method for processing multi-source data on flood emergencies.

[0026] According to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, which are used to execute the above-described method for processing multi-source data on flood and waterlogging emergencies when the program instructions are executed.

[0027] According to another aspect of the present invention, a computer program product is also provided, including computer program instructions, which, when executed, are used to perform the multi-source data processing method for flood disasters as described above.

[0028] The aforementioned technical solution generates a first digital surface model using point cloud datasets of flood-prone areas. It then uses the first digital surface model and the pose information from the hyperspectral sensor during the acquisition of each hyperspectral image to perform geometric correction on each image. This allows for a more accurate establishment of the mapping relationship between each pixel in the hyperspectral image and the corresponding real geographical location in the flood-prone area. Consequently, the geometrically corrected hyperspectral images can be accurately stitched together to create a larger-scale hyperspectral orthophoto image. Furthermore, by fusing thermal infrared orthophotos, hyperspectral orthophotos, and the first digital surface model, the elevation, spectral, and thermal radiation information of the flood-prone area can be integrated into a single target image. This target image provides a solid data foundation for subsequent joint analysis of spectral features, thermal radiation features, and geometric features, as well as for cross-validation of flood-prone conditions.

[0029] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0030] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0031] Figure 1 A schematic flowchart of a method for processing multi-source flood disaster data according to an embodiment of the present invention is shown;

[0032] Figure 2 A schematic diagram of a thermal infrared orthophoto image according to an embodiment of the present invention is shown;

[0033] Figure 3 A comparative schematic diagram of a hyperspectral image and a corresponding orthophoto image according to an embodiment of the present invention is shown;

[0034] Figure 4 A schematic diagram of a hyperspectral orthophoto image according to an embodiment of the present invention is shown;

[0035] Figure 5A schematic diagram of a point cloud dataset according to an embodiment of the present invention is shown;

[0036] Figure 6 A schematic diagram of a first digital surface model according to an embodiment of the present invention is shown;

[0037] Figure 7 A schematic diagram of a target image according to an embodiment of the present invention is shown;

[0038] Figure 8 A schematic structural diagram of a sensor payload system according to an embodiment of the present invention is shown;

[0039] Figure 9 A physical schematic diagram of a sensor payload system according to an embodiment of the present invention is shown;

[0040] Figure 10 A schematic diagram of a sensor payload system mounted on a drone according to an embodiment of the present invention is shown.

[0041] Figure 11 A hardware architecture diagram of a sensor payload system according to an embodiment of the present invention is shown;

[0042] Figure 12 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0044] To at least partially solve the aforementioned technical problems, embodiments of the present invention provide a method for processing multi-source data on flood risks, a sensor payload system, electronic equipment, a storage medium, and a computer program product. This solution can fuse hyperspectral image data, point cloud data, and thermal infrared image data, thereby integrating spectral information, thermal radiation information, and elevation information to obtain a target image containing multi-dimensional information.

[0045] Please see Figure 1 The diagram shown is a schematic flowchart of a multi-source data processing method for flood risks according to an embodiment of the present invention. The method includes steps S110 to S140.

[0046] To facilitate the description and understanding of the following embodiments, a sensor module that can be mounted on a drone platform is first introduced here. The sensor module can integrate a hyperspectral sensor, a lidar sensor, and a thermal infrared sensor. The relative poses of each sensor are fixed, and at least part of the field of view of each sensor overlaps. The drone can have a control module, which can be, for example, a Linux host. The ground station equipment can communicate with the control module through a network port, and the control module can receive a mission parameter file sent by the ground station equipment. The mission parameter file can indicate the drone's flight parameters (e.g., flight path, flight speed, flight altitude, electronic fence, forward overlap rate, lateral overlap rate, etc.), and can also indicate the operating parameters of at least some of the sensors in the sensor module (e.g., trigger mode, data acquisition mode, etc.). In some embodiments, the control module can set the operating parameters of all sensors; in other embodiments, the operating parameters of some sensors are set by the control module, while the operating parameters of other sensors are controlled by application software (APP) on a remote terminal device (e.g., mobile phone, tablet computer, computer, etc.). For example, the operating parameters of lidar sensors and hyperspectral sensors can be set by the control module, while the operating parameters of thermal infrared sensors (such as frame rate, data output format, exposure mode, emissivity coefficient, etc.) can be set by the terminal device via Bluetooth wireless communication.

[0047] In step S110, thermal infrared orthophotos, point cloud datasets, and multiple hyperspectral images of the flood-prone area are acquired. Each pixel of the thermal infrared orthophoto image has thermal radiation information representing the thermal radiation situation of the corresponding physical location. Each set of point cloud data in the point cloud dataset includes first position information representing the corresponding physical location. Each pixel of each hyperspectral image has spectral information corresponding to the physical location. The multiple hyperspectral images correspond one-to-one with multiple sets of pose information. The pose information is used to represent the pose of the hyperspectral sensor when acquiring images of the flood-prone area at the corresponding acquisition time.

[0048] For example, a flood-prone area, also known as a dike inspection area, refers to an area where there is a risk of flooding or where flooding has already occurred. UAVs can be used to inspect dikes in flood-prone areas. In this embodiment of the invention, the raw data from the thermal infrared orthophotos, point cloud datasets, and multiple hyperspectral images can be data collected by the sensor modules of the aforementioned embodiments mounted on a UAV platform, and collected by the sensor modules when the UAV performs a flight mission targeting the flood-prone area. The mission execution parameters in the control module can specify the forward overlap rate and the lateral overlap rate. Based on the sensor performance (e.g., scanning frequency, field of view, and scanning mode), the forward overlap rate, and the lateral overlap rate, the UAV's flight speed, flight altitude, and the spacing between adjacent flight paths can be determined. In a specific embodiment, the forward overlap rate is specified to be ≥70% and the lateral overlap rate ≥80%. Those skilled in the art will understand the meaning of the forward overlap rate and the lateral overlap rate, which will not be elaborated here. In this embodiment of the invention, for any sensor of the sensor module, when the UAV is flying along the current flight direction, there is an overlapping area between the physical areas covered by the acquisition range of two adjacent data acquisitions, and there is an overlapping area between the physical areas covered by the data acquisition range of each of two adjacent flight paths.

[0049] For example, at least a portion of the raw thermal infrared images acquired by the thermal infrared sensor based on the sensor module can be used to obtain a thermal infrared orthophoto image. Specifically, the pose of the thermal infrared images in a unified world coordinate system can be calculated based on aerial triangulation techniques to construct a digital surface model (hereinafter referred to as the second digital surface model). The second digital surface model can be used to perform digital differential correction on each thermal infrared image, and the digitally corrected thermal infrared images can be stitched together to form a thermal infrared orthophoto image. For example, at least a portion of the raw point cloud data acquired by the lidar sensor module based on the sensor module can be used to obtain a point cloud dataset. In some embodiments, merging the raw point cloud data in the same world coordinate system can obtain a point cloud dataset; in other embodiments, filtering the raw point cloud data and then merging the filtered point cloud data in the same world coordinate system can obtain a point cloud dataset. For example, raw hyperspectral data acquired by the hyperspectral sensor based on the sensor module can be radiometrically calibrated to obtain a corresponding radiance image. In some embodiments, the radiance image can be directly used as a hyperspectral image. In some other embodiments, the reflectance corresponding to the radiance image can be calculated based on the radiance values ​​in the radiance image, thereby obtaining a hyperspectral image containing reflectance information. The spectral information may include radiance and / or reflectance in at least one band.

[0050] For example, for thermal infrared orthophotos, each pixel of the thermal infrared orthophoto can correspond to a physical location, and the thermal radiation information of each pixel can be the sum of thermal radiation energy of all wavelengths within a specific narrow band (e.g., 8~14μm) for the corresponding physical location. More specifically, the pixel value of each pixel in the thermal infrared orthophoto can represent the thermal radiation information of that pixel. For example, a navigation component can be installed on the UAV, which can be, for example, a GNSS / IMU integrated navigation module (consisting of a mutually coupled satellite navigation module and an inertial measurement unit). The coordinate system of the raw point cloud data measured by the lidar sensor is usually the sensor coordinate system of the lidar sensor itself. Using the navigation component (or using the navigation component and the real geographic information of preset ground control points), the raw point cloud data measured by the lidar sensor can be given real geographic information. Accordingly, each set of point cloud data in the point cloud dataset can be the coordinates of the corresponding physical location in the first world coordinate system (which is a geospatial world coordinate system with real geographic information), that is, the first location information. For example, the navigation component can measure the pose information of the UAV. Since the relative pose between the UAV and the onboard hyperspectral sensor is fixed and can be pre-calibrated, the pose information of the hyperspectral sensor can be determined based on the UAV's pose information. It is understood that during a UAV's flight mission targeting flood-prone areas, the hyperspectral sensor can acquire images of the flood-prone area at multiple acquisition times, and the navigation component can measure the pose information at each acquisition time. Multiple hyperspectral images correspond one-to-one with multiple acquisition times, and correspondingly, multiple hyperspectral images correspond one-to-one with multiple sets of pose information. Each set of pose information may include, for example, roll angle, pitch angle, and yaw angle to represent attitude, and longitude, latitude, and elevation to represent position, i.e., coordinates in a geodetic coordinate system. It is understood that the coordinate system used for the position information in the attitude information can also be other geospatial world coordinate systems besides the geodetic coordinate system. The embodiments of the present invention can set the coordinate system used by the navigation component according to actual needs. Since both pose information and point cloud datasets are determined based on the same navigation component, the coordinate system used for pose information and the coordinate system used for point cloud datasets can typically be the same or mutually convertible.

[0051] In step S120, a first digital surface model of the flood-prone area is generated based on the point cloud dataset. The first digital surface model includes the first elevation information of each physical location of the surface features in the flood-prone area.

[0052] For example, a digital surface model (DSM) is one of the core foundational data for surface topography analysis, flood simulation, and levee stability calculation. Elevation information of ground surfaces can be extracted from discrete 3D point cloud data collected from point cloud datasets, thereby generating a first digital surface model. The first digital surface model has a grid array, where each grid represents a surface area. For example, in a digital surface model with a grid resolution of 0.1m × 0.1m, each grid can represent a 0.1m × 0.1m square surface area. Each grid is enclosed by four adjacent grid points, and each grid point in the grid array has an elevation value corresponding to its physical location. The elevation of any point within each grid can typically be obtained by bilinear interpolation of the elevation values ​​of its four corner points. The first elevation information can at least include the elevation values ​​of each grid point of the first digital surface model. The first digital surface model can have a second world coordinate system, which can also be a geospatial world coordinate system with real geographic information. In one specific embodiment, the first digital surface model can be a GeoTIFF format DSM data file conforming to Geographic Information System standards, and each grid point of the first digital surface model can have coordinates in a second world coordinate system. In some embodiments, the coordinate system used by the point cloud dataset can be directly used as the coordinate system of the first digital surface model; in this case, the second world coordinate system is the first world coordinate system. In other embodiments, the coordinate system used by the point cloud dataset can be converted according to actual needs. For example, if the point cloud dataset uses the WGS84 coordinate system but the actual requirement is the GCJ02 coordinate system, then the coordinates of pre-calibrated ground control points in the WGS84 coordinate system and the GCJ02 coordinate system can be used to convert each group of point cloud data in the point cloud dataset from the WGS84 coordinate system to the GCJ02 coordinate system.

[0053] In step S130, the first digital surface model and multiple sets of pose information are used to perform geometric correction on multiple hyperspectral images respectively, so as to generate a hyperspectral orthophoto based on the multiple hyperspectral images after geometric correction. Each pixel of the hyperspectral orthophoto has spectral information corresponding to the physical location.

[0054] For example, factors causing geometric distortion in hyperspectral images include pose variations of the hyperspectral sensor, terrain undulations, and the curvature of the Earth. To minimize such geometric distortion, joint geometric correction (i.e., orthorectification) can be performed using the pose information corresponding to the hyperspectral image and a first digital surface model. Specifically, each set of pose information includes the position (X, Y, Z) of the hyperspectral sensor in physical space at the corresponding acquisition time and its attitude angle. The first digital surface model can provide terrain undulation constraints. Collinearity equations can be constructed using the pose information and the first digital surface model, thereby enabling geometric correction of the hyperspectral image. Each pixel in the geometrically corrected hyperspectral image (hereinafter referred to as the orthorectified image) has position information, which accurately represents the actual physical location of the corresponding pixel. It can be understood that the position information of the orthorectified image is typically the coordinates in the coordinate system adopted by the first digital surface model. After acquiring the orthorectified images corresponding to each hyperspectral image, the orthorectified images can be stitched together using the position information of the orthorectified images to obtain a hyperspectral orthorectified image. Optimization operations can also be optionally performed on the hyperspectral orthorectified image.

[0055] In step S140, the thermal infrared orthophoto image, the hyperspectral orthophoto image, and the first digital surface model are fused to obtain the target image. Each pixel of the target image has elevation information, spectral information, and thermal radiation information corresponding to its physical location.

[0056] For example, the thermal infrared orthophoto image has thermal radiation information, the hyperspectral orthophoto image has spectral information, and the first digital surface model has elevation information. Furthermore, each of the thermal infrared orthophoto image, hyperspectral orthophoto image, and first digital surface model has location information that can represent a physical location. Accordingly, by utilizing the location information of each of the thermal infrared orthophoto image, hyperspectral orthophoto image, and first digital surface model, data alignment and data fusion can be performed on the thermal infrared orthophoto image, hyperspectral orthophoto image, and first digital surface model. Thus, each pixel of the obtained target image can possess elevation information, spectral information, and thermal radiation information.

[0057] The aforementioned technical solution generates a first digital surface model using point cloud datasets of flood-prone areas. It then uses the first digital surface model and the pose information from the hyperspectral sensor during the acquisition of each hyperspectral image to perform geometric correction on each image. This allows for a more accurate establishment of the mapping relationship between each pixel in the hyperspectral image and the corresponding real geographical location in the flood-prone area. Consequently, the geometrically corrected hyperspectral images can be accurately stitched together to create a larger-scale hyperspectral orthophoto image. Furthermore, by fusing thermal infrared orthophotos, hyperspectral orthophotos, and the first digital surface model, the elevation, spectral, and thermal radiation information of the flood-prone area can be integrated into a single target image. This target image provides a solid data foundation for subsequent joint analysis of spectral features, thermal radiation features, and geometric features, as well as for cross-validation of flood-prone conditions.

[0058] Optionally, acquiring thermal infrared orthophotos includes: acquiring multiple thermal infrared images collected by a thermal infrared sensor on a UAV targeting a flood-prone area; generating sparse three-dimensional point cloud data using multiple sets of exterior orientation parameters corresponding one-to-one with the multiple thermal infrared images and interior orientation parameters of the thermal infrared sensor, wherein each set of exterior orientation parameters is used to indicate the pose of the thermal infrared sensor when acquiring the corresponding thermal infrared image, and the interior orientation parameters include focal length, principal point position parameters, and distortion coefficients; generating high-density three-dimensional point cloud data based on the sparse three-dimensional point cloud data and multiple thermal infrared images using a multi-view stereo vision dense matching algorithm, and generating a second digital surface model based on the high-density three-dimensional point cloud data; mapping the pixel values ​​of the multiple thermal infrared images to the corresponding surface positions of the second digital surface model to obtain thermal infrared orthophotos, wherein thermal radiation information is represented by pixel values.

[0059] For example, when a drone performs a flight mission targeting flood-prone areas, the thermal infrared sensor mounted on the drone can acquire images at preset time intervals, thereby obtaining multiple thermal infrared images. Each thermal infrared image can correspond to a set of exterior orientation parameters. This embodiment of the invention does not specifically limit the method of acquiring the exterior orientation parameters. In some embodiments, the thermal infrared sensor can be equipped with a GPS module, which can determine the timestamp information and approximate location information of each acquired thermal infrared image. By matching the timestamp information of the thermal infrared image with the time reference provided by the navigation component on the drone, the exterior orientation parameters of the thermal infrared sensor can be determined based on the pose information measured by the navigation component (GNSS / IMU integrated navigation module) on the drone. In other embodiments, the exterior orientation parameters of each thermal infrared image can be determined using computer vision and photogrammetry algorithms. It is understood that the exterior orientation parameters determined in this way are relative exterior orientation parameters. Furthermore, while using computer vision and photogrammetry algorithms to determine the relative exterior orientation parameters of each thermal infrared image, the interior orientation parameters of the thermal infrared sensor can also be self-calibrated simultaneously. Alternatively, the interior orientation parameters of the thermal infrared sensor can also be pre-calibrated. The exterior orientation parameters can include position parameters (e.g., three-dimensional coordinates) and attitude parameters (e.g., attitude angles). The interior orientation parameters can include focal length f, principal point position parameters (x0, y0), and distortion coefficient a.

[0060] For example, by extracting feature points from multiple thermal infrared images (e.g., through SIFT extraction or deep learning-based methods) and matching the extracted feature points, multiple sets of matching feature points can be obtained, each corresponding to the same physical location. Based on the image position of each feature point in the corresponding thermal infrared image, the exterior orientation parameters of each thermal infrared image, and the interior orientation parameters of the thermal infrared sensor, sparse three-dimensional point cloud data can be generated using aerial triangulation (AT) theory. The specific generation method will be described below and will not be elaborated here. In some embodiments, computer vision and photogrammetry algorithms used to calibrate the interior and exterior orientation parameters can realize feature point extraction and matching. In this case, the image position of the mutually matching feature points in the corresponding thermal infrared image can be determined during the calibration of the interior and exterior orientation parameters, without the need to perform separate feature extraction and matching operations. For example, the forward overlap rate and lateral overlap rate are usually set high for UAVs. Therefore, when UAVs perform flight missions in flood-prone areas, the thermal infrared sensor can acquire images of the same surface area from different poses. Under the geometric constraints provided by the interior and exterior orientation parameters corresponding to sparse 3D point cloud data, the multi-view stereo vision dense matching algorithm (MVS) can be used to perform pixel-level matching on multiple thermal infrared images, thereby generating high-density 3D point cloud data. Based on the high-density 3D point cloud data, a second digital surface model can be further generated. Similar to the first digital surface model, the second digital surface model can include elevation information of surface features such as buildings, bridges, and trees.

[0061] For example, similar to the first digital surface model, the second digital surface model can also have a grid array, where each grid can represent a surface region, and each grid point can have coordinates in a third-world coordinate system. If the exterior orientation parameter is a relative exterior orientation parameter, the third-world coordinate system can be a custom global coordinate system; if the exterior orientation parameter is an absolute exterior orientation parameter (i.e., an exterior orientation parameter with real geographic information), the third-world coordinate system can also be a geospatial world coordinate system with real geographic information. For example, but not limitingly, before generating sparse 3D point cloud data, the relative exterior orientation parameter can also be converted to an absolute exterior orientation parameter according to preset rules based on actual needs. Using the exterior orientation parameters of each thermal infrared image and the interior orientation parameters of the thermal infrared sensor, the coordinate range of the acquisition area corresponding to each thermal infrared image in the third-world coordinate system can be calculated. It can be understood that the coverage area of ​​each thermal infrared image can include a portion of the surface location of the second digital surface model.

[0062] In some embodiments, for each grid point of the second digital surface model, the grid point can be back-projected onto one or more corresponding thermal infrared images based on its coordinates in a third-world coordinate system to obtain the image position of the grid point in each corresponding thermal infrared image, i.e., the image coordinates in the image coordinate system of the thermal infrared image. The image coordinates corresponding to the grid point are usually not integers; the pixel value corresponding to the grid point can be determined using a resampling algorithm (e.g., bilinear interpolation, bicubic convolution interpolation, nearest neighbor interpolation, etc.). In this case, one grid point can represent a surface location. It should be noted that due to the high overlap of thermal infrared images, the surface location corresponding to the same surface location in the second digital surface model may appear in different thermal infrared images. In this case, a weighted average of the pixel values ​​of the corresponding image locations in each thermal infrared image can be performed. Alternatively, it is preferable to determine an optimal pixel value according to a preset filtering rule, thereby obtaining the final pixel value of the corresponding grid point. The filtering rule could be, for example, selecting the thermal infrared image with the best image quality. Thermal infrared orthophotos can be obtained based on the pixel values ​​corresponding to each grid point of the second digital surface model. It is understandable that this mapping method combines the inverse solution method (also known as the indirect method) of digital differentiation techniques.

[0063] In other embodiments, for each pixel in a thermal infrared image, the coordinates of the pixel in a third-world coordinate system can be determined based on its image position within the image and the elevation information of the second digital surface model. The pixel value of this pixel is then mapped to the corresponding surface position in the second digital surface model according to the third-world coordinate system. More specifically, for any pixel, an elevation Z1 can be assumed / traversed in the second digital surface model, and the corresponding ground coordinates (X, Y) can be calculated using collinearity equations. Then, an elevation Z2 (X, Y) can be determined in the second surface model. The consistency between Z1 and Z2 is compared, and this process is repeated until a Z2 that is consistent with or approximately consistent with Z1 is determined; this Z2 is then the elevation information corresponding to the pixel. In this case, a grid point or a location point within a grid can be considered a surface position. Thermal infrared orthophotos can be obtained based on the pixel values ​​corresponding to at least some surface positions in the second digital surface model. Similarly, pixels in different thermal infrared images may correspond to the same surface position. In this case, a weighted average or the optimal pixel value can be selected from the pixel values ​​corresponding to the same surface position to determine the final pixel value corresponding to that surface position. It is understandable that this mapping method combines the forward method (also known as the direct method) of digital differentiation techniques.

[0064] The above technical solution can reconstruct multiple thermal infrared images into thermal infrared orthophotos with accurate spatial location information and thermal radiation information, thereby providing a reliable thermal radiation reference layer for subsequent multi-source data fusion operations.

[0065] Optionally, before generating high-density three-dimensional point cloud data based on sparse three-dimensional point cloud data and the multiple thermal infrared images, acquiring thermal infrared orthophotos of the flood-prone area further includes: determining the image positions of preset ground control points in the multiple thermal infrared images, and using the image positions of the ground control points to optimize the sparse three-dimensional point cloud data using an overall adjustment method; wherein, the operation of generating high-density three-dimensional point cloud data based on sparse three-dimensional point cloud data and multiple thermal infrared images is performed on the optimized sparse three-dimensional point cloud data.

[0066] For example, the image coordinates (i.e., image positions) of pre-deployed ground control points (GCPs) can be measured in multiple thermal infrared images. Each GCP has known object coordinates (i.e., three-dimensional coordinates representing real geographic information). Using the image and object coordinates of the GCPs, an aerial triangulation model can be optimized using a global adjustment method. The aerial triangulation model includes the exterior orientation parameters calibrated in the aforementioned embodiment and sparse three-dimensional point cloud data. Under the geometric constraints provided by the exterior and interior orientation parameters corresponding to the optimized sparse three-dimensional point cloud data, a multi-view stereo vision dense matching algorithm (MVS) can be used to perform pixel-level matching on multiple thermal infrared images, thereby generating high-density three-dimensional point cloud data. This approach can effectively improve the absolute geometric accuracy of sparse three-dimensional point cloud data and effectively constrain and reduce the absolute positional deviation and relative positional deformation of the sparse three-dimensional point cloud data.

[0067] Optionally, sparse three-dimensional point cloud data is generated by using multiple sets of exterior orientation parameters corresponding one-to-one with multiple thermal infrared images and interior orientation parameters of thermal infrared sensors. This includes: extracting and matching feature points of multiple thermal infrared images using a motion recovery structure algorithm; determining multiple sets of exterior orientation parameters and interior orientation parameters based on the mutually matched feature points; and establishing spatial topological relationships between multiple thermal infrared images based on the multiple sets of exterior orientation parameters and interior orientation parameters to obtain sparse three-dimensional point cloud data.

[0068] For example, the structure-of-motion (SOG) algorithm is a computer vision algorithm that analyzes multiple 2D images of the same scene acquired from different viewpoints to simultaneously recover the motion trajectory of the camera (in this embodiment, a thermal infrared sensor) and the 3D structure of the scene. Specifically, the SOG algorithm includes a feature detection and matching algorithm. For each thermal infrared image, feature points can be extracted. The feature points of each thermal infrared image are matched, and the matched feature points are the pixels of the same location in physical space from different viewpoints. The relative pose between the thermal infrared images can be initially calculated using the matched feature points. Combined with the initial estimate of the interior orientation parameters of the thermal infrared sensor (which is a hypothetical value), bundle adjustment can be used to globally optimize the exterior orientation parameters and the interior orientation parameters of the thermal infrared sensor corresponding to each thermal infrared image to minimize the reprojection error, that is, to minimize the error between the theoretical position of the final sparse 3D point cloud data back-projected onto the image and the actual observation position. For example, after determining the exterior and interior orientation parameters, the image coordinates of the feature points can first be converted into direction vectors in the sensor coordinate system using the interior orientation parameters. Then, based on the principle of triangulation, projection rays originating from the camera's optical center are constructed using the exterior orientation parameters of the thermal infrared image. By performing intersection calculations on multiple projection rays corresponding to mutually matching feature points using a triangulation algorithm, the three-dimensional coordinates of the corresponding locations of the mutually matching feature points in the world coordinate system can be determined. These three-dimensional coordinates can constitute sparse three-dimensional point cloud data. Here is a brief introduction to the principle of triangulation: the same spatial point is photographed from at least two different locations, resulting in at least two projection rays (projection rays refer to rays pointing from the camera's optical center to the corresponding spatial point). The intersection point is the three-dimensional spatial position of that point. The interior orientation parameters can be used to convert the mutually matching feature points from the image coordinate system to the sensor coordinate system. The exterior orientation parameters can be used to project two or more rays originating from the optical center of the sensor coordinate system and pointing to the physical locations corresponding to the mutually matching feature points into the world coordinate system according to the collinearity condition equation. This approach can quickly and efficiently establish high-precision spatial topological relationships between thermal infrared images, and the relative positions of the points in the resulting sparse 3D point cloud data are relatively accurate.

[0069] Optionally, each surface position of the second digital surface model has second position information for representing the corresponding physical position, and each of the multiple thermal infrared images corresponds to a partial surface region of the second digital surface model. Mapping the pixel values ​​of the multiple thermal infrared images to the corresponding surface position of the second digital surface model to obtain a thermal infrared orthophoto image includes: for each of the multiple thermal infrared images, establishing a first collinearity equation based on the second position information of the second digital surface model, the exterior orientation parameter and the interior orientation parameter corresponding to the thermal infrared image; determining the image position of at least a portion of the surface position of the surface region on the second digital surface model corresponding to the thermal infrared image in the thermal infrared image based on the first collinearity equation; for each surface position of at least a portion of the surface position of the second digital surface model, mapping the pixel value of the target thermal infrared image located at the image position corresponding to the surface position to the surface position to obtain a thermal infrared orthophoto image, wherein the target thermal infrared image is the thermal infrared image corresponding to the surface region to which the surface position belongs.

[0070] For example, since the second digital surface model is generated based on high-density three-dimensional point cloud data reconstructed from multiple thermal infrared images, it can be understood that at least a portion of the physical region commonly covered by each thermal infrared image (i.e., the union of the acquisition areas of each thermal infrared image) is consistent with the physical region corresponding to the second digital surface model. The second position information can be coordinates in a third-world coordinate system. For example, the second position information of any surface position of the second digital surface model can be represented by (X, Y, Z), where Z is the second elevation information of the physical position corresponding to the surface position. The relevant description of the third-world coordinate system can be referred to the foregoing embodiments, and will not be repeated here. For any surface position of the second digital surface model and each thermal infrared image (i.e., the target thermal infrared image) corresponding to the surface position, the first collinearity equation between the surface position and the thermal infrared image can be expressed by the following formulas (1) and (2):

[0071] (1)

[0072] (2)

[0073] In formulas (1) and (2), r ijLet X0, Y0, and Z0 be the rotation matrix in the exterior orientation parameters corresponding to the thermal infrared image, respectively, and let f be the focal length of the thermal infrared sensor. Let x represent the abscissa of the image position in the thermal infrared image corresponding to the surface position, and y represent the ordinate of the image position in the thermal infrared image corresponding to the surface position. The pixel value mapping method can refer to the aforementioned embodiment and will not be repeated here. This scheme can quickly and efficiently establish the mapping relationship between the surface position of the second digital surface model and the image position of the thermal infrared image, and can effectively eliminate the projection difference caused by terrain undulations.

[0074] Optionally, the plurality of thermal infrared images includes at least one pair of thermal infrared images, and the field of view of each pair of thermal infrared images partially overlaps. After mapping the pixel values ​​of the plurality of thermal infrared images to the corresponding surface positions of the second digital surface model to obtain thermal infrared orthophotos, obtaining thermal infrared orthophotos further includes: performing post-processing operations on the target image region of the thermal infrared orthophotos to obtain the final thermal infrared orthophotos. The post-processing operations include color balancing and mosaicking operations. The target image region includes the image region corresponding to the overlapping area of ​​the field of view of at least one pair of thermal infrared images.

[0075] For example, when a drone performs flight missions targeting flood-prone areas, it typically needs to fly along multiple routes. The thermal infrared images collected by the thermal infrared sensor during the drone's flight along any of these routes can be considered a sequence of thermal infrared images. In each sequence, the field of view of every two consecutive thermal infrared images partially overlaps. For any image in any thermal infrared image sequence corresponding to any two adjacent flight routes, at least one thermal infrared image in another thermal infrared image sequence partially overlaps with the field of view of that image. Each pair of thermal infrared images with overlapping field of view is considered a pair, and each pair corresponds to an overlapping area of ​​field of view. The target image region can be the entire image region of a thermal infrared orthophoto, or it can be a portion of the image region. For ease of description, the image region corresponding to the overlapping area of ​​the field of view of each pair of thermal infrared images in the thermal infrared orthophoto is denoted as the stitched region. The pixel values ​​of each pixel within the stitched region are highly likely to be derived from the pixel values ​​of different thermal infrared images; therefore, the consistency of thermal radiation information within the stitched region is low. Furthermore, due to the different acquisition angles / times of various thermal infrared images, the consistency of thermal radiation information among different thermal infrared images may be low. The target image region may include only the stitched region, or optionally other image regions besides the stitched region. To improve the consistency of thermal radiation information in the target image region, color balancing and mosaicking operations can be performed on the thermal infrared orthophoto. Those skilled in the art will understand that if the target image region is the entire image region of the thermal infrared orthophoto, color balancing and mosaicking operations can employ histogram matching based on a preset reference image, adjustment methods based on a correction model established according to temperature measurements and pixel values ​​from ground control points, mean-variance standardization, etc. If the target image region is a stitched region of the thermal infrared orthophoto, color balancing and mosaicking operations can employ linear feathering, Poisson image editing, etc. Post-processing operations can yield a thermal infrared orthophoto with complete and uniform thermal radiation information. Please refer to [link to relevant documentation]. Figure 2 As shown, it is a schematic diagram of a thermal infrared orthophoto image according to an embodiment of the present invention. Figure 2 The thermal infrared orthophoto image shown is an image with accurate spatial location information and complete thermal radiation information.

[0076] Optionally, each surface position of the first digital surface model has third position information representing the corresponding physical position, the third position including first elevation information. Geometric correction is performed on multiple hyperspectral images using the first digital surface model and multiple sets of pose information to generate a hyperspectral orthophoto based on the geometrically corrected hyperspectral images. This includes: for each hyperspectral image of the multiple hyperspectral images, constructing a second collinearity equation based on the third position information of the first digital surface model, the pose information corresponding to the hyperspectral image, and the interior orientation parameters of the hyperspectral sensor; for each hyperspectral image of the multiple hyperspectral images, based on each pixel of the hyperspectral image... Based on the image position and the second collinearity equation, the surface position on the first digital surface model corresponding to each pixel is determined; based on the surface position corresponding to each pixel of the hyperspectral image, the pixel value of each pixel of the hyperspectral image is mapped to a preset raster image corresponding to the first digital surface model, and a resampling algorithm is used to determine the pixel value of each grid of the preset raster image to obtain an orthophoto image corresponding to the hyperspectral image, wherein each image position of the preset raster image corresponds to at least a part of the surface position of the first digital surface model; multiple orthophoto images corresponding one-to-one with multiple hyperspectral images are stitched together to obtain a hyperspectral orthophoto image.

[0077] For example, the third position information can be coordinates in a second world coordinate system. A description of the second world coordinate system can be found in the foregoing embodiments and will not be repeated here. For example, the UAV's navigation component can provide PPS (Pulse Per Second) signals and absolute time information as a unified time reference. The lidar sensor and hyperspectral sensor trigger data acquisition via a trigger signal. It can be understood that the time the sensor receives the trigger signal and the time the sensor acquires data can be considered synchronized. When the sensor receives the trigger signal, the time of data acquisition can be determined using the time reference provided by the navigation component. The pose information measured by the navigation component at that time corresponds to the hyperspectral data (raw data of the hyperspectral image) acquired at the corresponding time. For example, the second collinearity equation is similar to the first collinearity equation in the foregoing embodiments. The interior orientation parameters of the hyperspectral sensor include the focal length, and the pose information corresponding to the hyperspectral image is the exterior orientation parameter corresponding to the hyperspectral image, which can include a translation vector and a rotation matrix. The fourth position information (i.e., coordinates in the third-world coordinate system) of the corresponding physical location of each pixel in the hyperspectral image can be determined using the second collinearity equation. The forward and inverse methods of digital differential correction have been introduced in the preceding embodiments. Combining either of these methods can orthorectify the hyperspectral image to obtain the corresponding orthorectified image. For orthorectification of the hyperspectral image, the forward method is preferred for determining the fourth position information of each pixel. Specifically, based on the fourth position information, the image position of each pixel in the hyperspectral image mapped to a preset raster image can be determined. The preset raster image is established based on a first digital surface model, and its raster cells correspond one-to-one with at least a portion of the surface positions of the first digital surface model. Based on the image position of each pixel mapped to the preset raster image, the pixel value of each pixel in the hyperspectral image can be assigned to the corresponding raster in the preset raster image. Since non-integer multiple mappings may exist during the mapping process, a resampling algorithm can be used to interpolate the pixel value of each raster in the preset raster image to determine the final pixel value of each raster, thereby generating an orthorectified image corresponding to the hyperspectral image. Please see Figure 3 The diagram illustrates a comparison between a hyperspectral image and its corresponding orthorectified image according to an embodiment of the present invention. It can be understood that digital differential correction techniques can correct spatial distortions caused by variations in the exterior orientation parameters of the hyperspectral sensor, topographic relief, or the curvature of the Earth, thereby ensuring that each pixel of the obtained orthorectified image accurately corresponds to its actual location on the Earth's surface in the world coordinate system. In other words, orthorectification can correct original images with perspective distortion (such as the thermal infrared image and hyperspectral image in the embodiments of the present invention) into vertically projected images with a uniform scale and almost no perspective distortion.

[0078] For example, hyperspectral orthorectified images can be obtained by stitching together the orthorectified images. In some embodiments, image stitching can be performed through feature extraction and matching, as well as image transformation estimation; in other embodiments, stitching can be performed based on the positional information of each orthorectified image, a scheme which will be described in detail below. This embodiment of the invention utilizes the third positional information of the first digital surface model, the pose information of the hyperspectral sensor when acquiring the hyperspectral image, and the interior orientation parameters of the hyperspectral sensor to perform orthorectification on the hyperspectral image, thereby obtaining an orthorectified image whose pixel positional information matches its actual corresponding geographical location, thus obtaining a hyperspectral orthorectified image with accurate geospatial information.

[0079] Optionally, each pixel of each of the multiple orthophoto images has fourth position information representing the corresponding physical location. The multiple orthophoto images, each corresponding to a different hyperspectral image, are stitched together to obtain a hyperspectral orthophoto image. This includes: aligning the multiple orthophoto images according to the fourth position information and determining the overlapping regions of the multiple orthophoto images; performing spectral consistency processing on the overlapping regions; determining the stitching lines of the multiple orthophoto images and stitching them together according to the stitching lines to obtain a stitched image; and feathering the image region containing the stitching lines in the stitched image to obtain a hyperspectral orthophoto image.

[0080] For example, the fourth position information may be coordinates in the second world coordinate system described in the foregoing embodiments, i.e., coordinates in the world coordinate system used by the first digital surface model. After obtaining each orthorectified image, the fourth position information of the orthorectified image can be read, and the orthorectified images can be aligned using the fourth position information of each orthorectified image. After image alignment, the coverage area of ​​each orthorectified image and the overlapping area of ​​the coverage area (i.e., the field of view) can optionally be displayed in the map view on the display interface of the display device. It is understood that, similar to thermal infrared images, the field of view of each of two consecutively acquired hyperspectral images partially overlaps. For any hyperspectral image in any hyperspectral image sequence corresponding to each of two adjacent flight paths, there is at least one hyperspectral image in another hyperspectral image sequence that partially overlaps with the field of view of that hyperspectral image. Accordingly, the image areas of each orthorectified image may have overlapping areas. It is understood that the overlapping areas are usually located near the edges of the orthorectified images. Spectral consistency processing is performed on the overlapping areas.

[0081] Exemplary embodiments of the present invention do not specifically limit the method of spectral consistency processing. In some embodiments, for overlapping regions, one of the orthophoto images to which the overlapping region belongs can be used as a reference image, and the image region in the reference image corresponding to the overlapping region can be used as a reference block. Based on the histogram of the reference block in each band, the histogram distribution of the image regions corresponding to the overlapping region in other orthophoto images to which the overlapping region belongs (hereinafter referred to as the image to be processed) in the corresponding bands is adjusted to be consistent with the reference block. In other embodiments, pseudo-invariant feature points (e.g., pixels belonging to relatively stable ground features such as roads) can be automatically selected in the image regions corresponding to the overlapping regions of the reference image and the image to be processed. Linear regression analysis is performed based on the pixel values ​​of the pseudo-invariant feature points in each band to calculate the gain coefficient and bias coefficient, and then linear regression correction is performed on the image regions corresponding to the overlapping regions of the image to be processed. In some other embodiments, the image regions corresponding to the overlapping regions of the orthophoto image can be normalized by mean-variance. For mosaicking, the best seam search algorithm can be used to determine the seam path, and multi-scale feathering or distance-weighted fusion methods can be used on both sides of the seam to achieve a smooth transition.

[0082] For example, the geometric center line of the overlapping region can be used as the stitching line; alternatively, a dynamic programming algorithm can be used to search for a path with the smallest grayscale or spectral difference within the overlapping region as the stitching line; alternatively, the overlapping region can be divided into blocks, the optimal stitching line for each sub-block can be calculated separately, and then the stitching lines of each sub-block can be smoothly connected; alternatively, a trained deep learning model can be used to learn the features of the overlapping region to predict the optimal stitching line; alternatively, the stitching line can be determined based on the quality of the image region corresponding to the overlapping region in the orthophoto image. For example, in response to the user's stitching strategy setting operation, an image stitching strategy for the orthophoto image can be determined. The image stitching strategy may include, for example, a first stitching strategy that stitches according to the acquisition order of the corresponding hyperspectral images, a second stitching strategy that stitches according to the flight path order (i.e., the flight path order of the UAV), and a third stitching strategy that stitches according to the quality of the image region corresponding to the overlapping region. Specifically, for the third stitching strategy, the quality of each group of orthorectified images with overlapping regions can be evaluated. More specifically, the image quality of the region corresponding to the overlapping region (hereinafter referred to as the target region) in each orthorectified image of the group can be evaluated (regional quality can be evaluated, for example, through indicators such as sharpness). During the stitching process, the image region corresponding to the overlapping region in the stitched image can be the target region with the best image region quality among at least two corresponding orthorectified images. This stitching strategy can effectively reduce error accumulation and obtain hyperspectral orthorectified images with good overall effect even when the quality of orthorectified images is uneven. Based on the user-defined image stitching strategy, orthorectified images can be stitched together according to the stitching line. To ensure that there are no obvious seams at the stitching line position in the final hyperspectral orthorectified image, a transition zone can be set on both sides of the stitching line. Within the transition zone, the corresponding pixel values ​​of at least two orthorectified images can be weighted and averaged to obtain the pixel values ​​of the transition zone. This process is the feathering process. Please refer to [link to relevant documentation]. Figure 4 As shown, it is a schematic diagram of a hyperspectral orthophoto image according to an embodiment of the present invention. Figure 4 The hyperspectral orthophotos shown have a wide physical range, consistent radiation dynamic range, and no obvious seams.

[0083] The above technical solution, through spectral consistency processing, helps to eliminate illumination differences when a UAV flies along multiple routes in one go, automatically adjusts the radiation dynamic range of the orthophotos corresponding to each route, and facilitates a natural transition in the stitching result. By feathering the image area where the stitching line is located, it can further ensure a smooth transition of the seam, and finally obtain a hyperspectral orthophoto with a wider physical range, consistent radiation dynamic range, and no obvious seams, which can be directly used for flood risk area analysis and data fusion.

[0084] Optionally, acquiring multiple hyperspectral images includes: acquiring multiple hyperspectral datasets collected by a hyperspectral sensor for flood-prone areas, each hyperspectral dataset including multiple sets of hyperspectral data corresponding one-to-one with multiple sensing units of the hyperspectral sensor, and multiple hyperspectral datasets corresponding one-to-one with multiple sets of pose information; for each hyperspectral dataset, performing radiometric calibration on each set of hyperspectral data to obtain a radiance image of the hyperspectral dataset, each pixel of the radiance image having a radiance value; acquiring a reference radiance image, which is an image collected by the hyperspectral sensor for a calibration blanket in the flood-prone area; determining the reflectance of multiple radiance images corresponding one-to-one with the multiple hyperspectral datasets in at least one band based on the standard reflectance of the calibration blanket and the radiance value of the reference radiance image; generating multiple hyperspectral images based on the reflectance of each of the multiple radiance images, each pixel of each hyperspectral image also having reflectance information corresponding to the physical location, the reflectance information being used to indicate the reflectance of the corresponding physical location in at least one band.

[0085] For example, a hyperspectral dataset may include a header file (.hdr format) and a data file (.bin format). The header file may include the two-dimensional spatial dimensions of the data (image resolution), spectral parameters (including the number of bands, the center wavelength of each band, and the bandwidth), data type, data storage order, exposure time, gain coefficient, etc. The data file may include multiple sets of hyperspectral data, each set of hyperspectral data may include the energy values, i.e., DN values, received by the corresponding sensing unit in each band. In the raw hyperspectral data, the DN value represents the intensity of the electrical signal received by the hyperspectral sensor. Its value is affected by multiple factors such as the hyperspectral sensor response, solar incidence angle, atmospheric scattering, and the flight attitude of the UAV, and cannot directly reflect the physical characteristics of the ground objects. Therefore, it is necessary to perform radiometric calibration, reflectance calculation, and geometric orthorectification on the DN value to establish a physical and geometric mapping relationship from the digital signal to the true reflectance of the ground surface. The specific process of geometric orthorectification can be referred to the relevant description in the foregoing embodiments. The specific process of radiometric calibration and reflectance calculation will be described below.

[0086] In some embodiments, the header file may include radiometric calibration parameters (including gain and offset parameters). These parameters are used to radiometrically calibrate each set of hyperspectral data in each hyperspectral dataset, yielding the corresponding radiance value. In other embodiments, a mapping relationship between exposure time and radiance calibration files (including gain and offset parameters) can be pre-configured. The radiance calibration file corresponding to each hyperspectral dataset is determined based on the exposure time acquired, thereby enabling radiometric calibration of each set of hyperspectral data in that dataset. Those skilled in the art will understand that dark current data may optionally participate in radiometric calibration. Similarly, the radiometric calibration parameters in the header file may include dark current data, or a mapping relationship between exposure time and dark current data may be pre-configured. Dark current data is dark-field data acquired by the hyperspectral sensor under light-isolated conditions (typically by covering the lens cap), used to subtract the hyperspectral sensor's own thermal noise and electronic offset. It is understood that a radiance image corresponding to each hyperspectral dataset can be obtained based on the radiometrically calibrated hyperspectral dataset. It is also understood that each pixel of the radiance image corresponds one-to-one with each set of hyperspectral data in the hyperspectral dataset. This invention provides a linear calibration model for radiometric calibration. The linear calibration model for any set of hyperspectral data in any hyperspectral dataset is expressed by the following formula (3):

[0087] (3)

[0088] In formula (3), Gain λ The gain parameter represents band λ, DN represents the energy value received by the corresponding sensing unit in band λ within this set of hyperspectral data, and Offset represents the gain parameter. λ The offset, gain, and shift parameters for band λ can be predetermined in the laboratory using a standard radiation source (integrating sphere system). Using the gain and shift parameters for each band, radiance transformation can be performed on each group of hyperspectral data in any hyperspectral dataset band by band to obtain the corresponding radiance image for that hyperspectral dataset.

[0089] For example, a hyperspectral sensor can be used to collect hyperspectral data of a calibration blanket pre-placed in a flood-prone area to obtain a reference brightness image. Specifically, the calibration blanket has a known standard reflectance, such as 11%, 32%, and 56%. When collecting data from the calibration blanket, the exposure value of the hyperspectral sensor can be adjusted to adjust the maximum brightness signal of the hyperspectral sensor to a preset value (e.g., 80%~90% of the saturation exposure value). Using the radiance value of the calibration blanket area in the reference brightness image and the standard reflectance of the calibration blanket, a linear relationship between reflectance and radiance value can be established. Using this linear relationship, the reflectance of the radiance image in each band can be calculated. For any pixel in any radiance image, the formula for calculating the reflectance of that pixel in band λ is expressed by the following formula (4):

[0090] (4)

[0091] In formula (4), a λ b λ L is a linear coefficient determined using the numerical relationship between the radiance value of the calibration blanket region in band λ of the reference luminance image and the standard reflectance of the calibration blanket. λ ρ represents the radiance value of the pixel in band λ. λ This represents the reflectance of the pixel in band λ. For example, after determining the reflectance of each radiance image, the reflectance is appended as an independent band to the radiance image to obtain the corresponding hyperspectral image. Each pixel in the final hyperspectral image can have both radiance and reflectance values ​​in each band. Calculating the reflectance of each radiance image using the reference radiance image and the standard reflectance of the calibration blanket effectively eliminates the effects of variations in solar altitude angle, atmospheric radiation differences along different paths, and differences in observation angle. This helps ensure the comparability of hyperspectral images obtained from hyperspectral data collected at different flight paths and time periods.

[0092] The above technical solution can convert the raw digital quantization values ​​(DN values) recorded by hyperspectral sensors into surface reflectance information with real physical meaning through radiometric calibration and reflectance calibration, thereby obtaining ground feature spectra with comparability and physical authenticity.

[0093] Optionally, acquiring a point cloud dataset includes: acquiring multiple sets of raw point cloud data collected by a lidar sensor on a drone targeting a flood-prone area; filtering the multiple sets of raw point cloud data to obtain multiple sets of filtered point cloud data; merging the multiple sets of filtered point cloud data in a preset coordinate system, wherein each set of filtered point cloud data has a coordinate range in the preset coordinate system, and the coordinate range of each set of filtered point cloud data intersects with the coordinate range of at least one other set of filtered point cloud data; and performing a point density-weighted average operation on the filtered point cloud data located within the intersection to obtain a point cloud dataset.

[0094] For example, when a drone performs a flight mission targeting flood-prone areas, it can fly along multiple flight paths sequentially. While flying along each flight path, the lidar sensor on the drone can collect a set of raw point cloud data. It can be understood that the multiple sets of raw point cloud datasets correspond one-to-one with the multiple flight paths targeting the flood-prone areas. In a specific embodiment, the raw point cloud data collected by the lidar sensor is a multi-strip (i.e., multi-flight) LAS format point cloud. Each set of point cloud datasets has a corresponding fifth-world coordinate system, point density, and reflection intensity information. Exemplarily, and not limitingly, in response to a user's point cloud view preview command, the coverage area and overlapping areas of each set of raw point cloud data can be output. Exemplarily, for each set of raw point cloud data, a preset filtering algorithm can be used to filter the raw point cloud data. The preset filtering algorithm may include, for example, a statistical outlier removal (SOR) algorithm and a spatial filtering algorithm. In statistical filtering algorithms, the standard deviation of the distance from each point in the original point cloud data to its neighborhood average distance can be calculated, and isolated points with a deviation exceeding 2σ can be removed. In spatial filtering algorithms, downsampling operations can be performed on overly dense areas of the point cloud, and the point cloud resolution can be normalized to a preset accuracy range using the voxel grid method. This embodiment of the invention does not impose specific limitations on the preset filtering algorithm; any filtering algorithm that effectively removes noise and duplicate points from the original point cloud data is acceptable.

[0095] For example, during the sequential flight of a UAV along multiple routes, there may be overlapping flight paths, attitude drift, and relative pose calibration errors between the lidar sensor and the UAV platform. This results in different sets of original point cloud data being located in different fifth-world coordinate systems, and the same ground feature corresponding to different positions in different sets of original point cloud data. Furthermore, the same ground feature may also have angular differences in different sets of original point cloud data. Therefore, the original point cloud data can be registered before filtering, or the filtered point cloud data can be registered after filtering. Specifically, the Iterative Closest Point (ICP) algorithm can be used, with the goal of minimizing the Euclidean distance between point clouds, to solve for the rigid body transformation matrix and complete the point cloud data registration. Those skilled in the art will understand that the ICP algorithm is used to align multiple sets of point cloud data collected from different perspectives and at different times to the same coordinate system; the core idea is iterative optimization. The coordinate system of each set of point cloud data after data registration can be any fifth-world coordinate system used by any set of original point cloud data or filtered point cloud data. It is understood that, since unmanned vehicles are equipped with navigation components, the fifth world coordinate system can typically be a world coordinate system with real geospatial information. It is also understood that the preset coordinate system can be the fifth world coordinate system, or other world coordinate systems transformed from the fifth world coordinate system. The preset coordinate system is the first world coordinate system in the aforementioned embodiments. For example, for any set of filtered point cloud data, the coordinate range of this set of filtered point cloud data intersects with the coordinate range of at least one other set of filtered point cloud data. After merging the various sets of filtered point cloud data, a point density-weighted average operation can be performed on the point cloud data at the intersection, thereby obtaining the final point cloud dataset. In some embodiments, the data format of the final output point cloud dataset is LAS format or LAZ format. Please refer to... Figure 5 As shown, it is a schematic diagram of a point cloud dataset according to an embodiment of the present invention. Figure 5 The point cloud dataset is a view obtained by filtering and merging multiple raw point cloud data in LAS format, with different colors representing different heights.

[0096] The above technical solution can effectively remove noise and duplicate points from the original point cloud data by filtering it, which is beneficial to improve the data stability when generating the data elevation model. By merging the filtered point cloud data in the same coordinate system and performing a point density weighted average operation, the splicing seams of the merged point cloud data can be effectively avoided, thus obtaining an integrated, high-precision point cloud dataset.

[0097] Optionally, generating a first digital surface model of a flood-prone area based on a point cloud dataset includes: extracting third elevation information from the point cloud dataset, the third elevation information being used to represent the elevation of the corresponding physical location of the surface features in the flood-prone area; determining the grid resolution according to a preset accuracy requirement; performing rasterization processing on the point cloud dataset based on the grid resolution; and determining the fourth elevation information of each raster cell based on the third elevation information using a preset interpolation algorithm, the fourth elevation information being used to represent the elevation of the physical location corresponding to the raster cell; and generating a first digital surface model based on the fourth elevation information of each raster cell.

[0098] For example, each point in the point cloud dataset has coordinates (X, Y, Z) in a preset coordinate system, where Z can represent the elevation of the physical location corresponding to that point. For example, point cloud denoising algorithms and maximum value filtering algorithms can be used to remove non-surface points (i.e., noise points) from the point cloud dataset, retaining the Z-values ​​of surface points. For example, and not limitingly, in the maximum value filtering algorithm, the window size can be optionally dynamically adjusted based on the local height change rate / surface roughness of the feature to accommodate features of different sizes (such as buildings and vegetation canopies); alternatively, feature feature lines (ridgelines, building outlines) can be used as constraints, performing maximum value filtering only near the feature lines to preserve boundary clarity. The Z-values ​​of the ultimately retained surface points are the third elevation information extracted from the point cloud dataset. Specifically, the accuracy requirement can indicate the grid resolution; the higher the accuracy requirement, the smaller the size of a single raster cell (i.e., a single grid) indicated by the grid resolution. By rasterizing the Z-values ​​of surface points according to a grid resolution (e.g., 0.1m × 0.1m), an elevation raster with a regular topological structure can be obtained. At least some cells in the elevation raster possess third elevation information. Using a pre-defined interpolation algorithm (i.e., interpolation method), cells in the elevation raster that do not possess elevation information can be interpolated to obtain fourth elevation information for each cell. The fourth elevation information can be the third elevation information or the elevation information obtained through interpolation. A first digital surface model can be generated using the fourth elevation information of each cell. See also... Figure 6 The diagram shown is a schematic representation of a first digital surface model according to an embodiment of the present invention. In this embodiment, the first digital surface model is a DSM data file in GeoTIFF format conforming to Geographic Information System standards, which contains elevation information (Z value) and planar location information (X value and Y value). It can be understood that the fourth elevation information is the same as the first elevation information in the aforementioned embodiment.

[0099] The above technical solution can transform discrete point cloud data into a high-precision digital surface model with a regular rasterized elevation surface, thereby enabling a continuous representation of the topographic undulations in flood-prone areas and providing topographic constraints (such as projection difference correction for slopes or embankments) for the geometric orthorectification of hyperspectral images.

[0100] Optionally, the thermal infrared orthophoto, hyperspectral orthophoto, and first digital surface model are fused to obtain a target image, including: fusing the hyperspectral orthophoto with the first digital surface model to obtain an intermediate fused image; and fusing the intermediate fused image with the thermal infrared orthophoto to obtain the target image.

[0101] For example, a hyperspectral orthophoto image can be first fused with a first digital surface model. Each pixel in the resulting intermediate fused image contains elevation and spectral information corresponding to its physical location. The intermediate fused image is then fused with a thermal infrared orthophoto image, resulting in a target image where each pixel contains elevation, spectral, and thermal radiation information corresponding to its physical location. It should be noted that the spectral information may include only the radiance values ​​for each band, and optionally, reflectance for each band. This approach yields a target image containing spectral, topographic, and thermal radiation information. This target image can be used to train a deep learning model for anomaly identification and risk classification analysis of flood hazards, facilitating a comprehensive assessment of seepage, saturation zones, vegetation stress, and structural deformation.

[0102] Optionally, each surface location of the first digital surface model further includes third location information for representing the corresponding physical location. The third location information includes first elevation information. Each pixel of the hyperspectral orthophoto also has fourth location information for representing the corresponding physical location. The hyperspectral orthophoto and the first digital surface model are fused to obtain an intermediate fused image, including: registering the hyperspectral orthophoto and the digital surface model based on the fourth location information of the hyperspectral orthophoto and the third location information of the first digital surface model; and embedding the first elevation information of the first digital surface model into the hyperspectral orthophoto in the form of additional bands to obtain an intermediate fused image according to the registration result of the hyperspectral orthophoto and the first digital surface model.

[0103] For example, the third position information can be the coordinates in the second world coordinate system in the aforementioned embodiments. Since the hyperspectral orthophoto is obtained by stitching together orthophotos, the fifth position information of the hyperspectral orthophoto can use the fourth position information of the orthophoto, that is, it can be the coordinates in the second world coordinate system in the aforementioned embodiments. For example, if the world coordinate system used by the first digital surface model and the hyperspectral orthophoto is different, for example, after executing step S130, a coordinate system transformation operation is performed separately on the hyperspectral orthophoto based on user viewing needs or other processing needs, a projection transformation operation can be performed to align the hyperspectral orthophoto with the first digital surface model in terms of coordinate systems. Using the fifth position information of the hyperspectral orthophoto and the third position information of the first digital surface model, each pixel of the hyperspectral orthophoto can be registered with each grid of the first digital surface model, thereby determining the elevation information corresponding to each pixel of the hyperspectral orthophoto. For example, and not limitingly, after data registration, the user can optionally view the data registration result through a display device. For example, based on the registration result between the hyperspectral orthophoto and the first digital surface model, the first elevation information of the first digital surface model is embedded as an additional band into the hyperspectral orthophoto. Those skilled in the art will understand that a hyperspectral orthophoto is a data cube composed of multiple consecutive bands, each corresponding to a grayscale image. The intermediate fused image can be viewed as a comprehensive dataset containing spectral and elevation information, and optionally may also include third or fifth location information. In a specific embodiment, in response to a user's format selection operation, the intermediate fused image can be output in ENVI format and / or GeoTIFF format. This approach can quickly and efficiently obtain an intermediate fused image with complete spectral and spatial features.

[0104] Optionally, the intermediate fused image and the thermal infrared orthophoto image are fused to obtain the target image, including: determining the key feature points of the intermediate fused image and the thermal infrared orthophoto image respectively; matching the key feature points of the intermediate fused image with the key feature points of the thermal infrared orthophoto image; registering the intermediate fused image and the thermal infrared orthophoto image using the feature point matching results; and fusing the thermal radiation information of the thermal infrared orthophoto image with the spectral information and the first elevation information of the hyperspectral orthophoto image based on the registration results of the intermediate fused image and the thermal infrared orthophoto image to obtain the target image.

[0105] For example, a grayscale image in any band of the intermediate fused image (which is equivalent to a hyperspectral orthophoto with elevation information) can be used as a reference base map, and a thermal infrared orthophoto can be used as the image to be registered. Preferably, the grayscale image corresponding to the near-infrared band in the hyperspectral orthophoto can be used as the reference base map. For example, key feature points can be extracted on the reference base map first, and algorithms such as template matching, SIFT, and SURF can be used to search for and extract key feature points similar to those in the reference base map in the image to be registered. In some embodiments, automatic identification and matching of key feature points may result in insufficient number of key feature points in key regions (e.g., target monitoring areas, thermal anomaly areas) or key feature point matching deviations. In such cases, key feature points can be marked in the key regions of both the reference base map and the image to be registered in response to a user's marking operation, or the current key feature points can be adjusted in response to a user's adjustment operation. For example, referencing the key feature point corresponding to road corner A in the base image, the key feature point corresponding to road corner B in the image to be registered is automatically matched. In this case, the current key control point in the image to be registered can be adjusted to the key feature point corresponding to road corner A. Exemplarily, the feature point matching result can be used to register the intermediate fused image and the thermal infrared orthorectified image. Specifically, based on the image positions of the key feature points in the intermediate fused image and the thermal infrared orthorectified image, a spatial transformation model (e.g., affine transformation, polynomial transformation, etc.) can be calculated. This spatial transformation model can then be used to perform geometric correction and resampling on the thermal infrared orthorectified image to obtain a new thermal infrared orthorectified image. It can be understood that at least some pixels in the new thermal infrared orthorectified image correspond one-to-one with at least some pixels in the intermediate fused image. By registering the new thermal infrared orthorectified image with the intermediate fused image, the thermal radiation information of at least some pixels in the thermal infrared orthorectified image can be fused with the spectral and elevation information of the corresponding pixels in the intermediate fused image based on the registration result to obtain the target image. Please refer to [link to relevant documentation]. Figure 7 As shown, it is a schematic diagram of a target image according to an embodiment of the present invention. Figure 7 In the illustrated embodiment, the grayscale image corresponding to the RGB bands in the hyperspectral orthophoto is used as a reference base image. When displaying the target image, the target image can optionally be displayed in the manner of a thermal infrared orthophoto (e.g., Figure 7 The left half of the image area can be displayed, or optionally displayed in the manner of hyperspectral orthophoto (e.g., the left half of the image area). Figure 7 The right half of the image area is displayed. The specific display method can be selected by the user according to their actual needs. This method can quickly and efficiently obtain a target image integrating spectral, elevation, and thermal radiation information of various surface locations within the flood-prone area.

[0106] Optionally, the intermediate fused image and the thermal infrared orthophoto image are registered using the feature point matching results, including: constructing a second-order polynomial model using the feature point matching results, the second-order polynomial model being used to represent the image position transformation relationship of converting pixels of the thermal infrared orthophoto image to the intermediate fused image; performing a geometric transformation on the thermal infrared orthophoto image based on the second-order polynomial model; and fusing the thermal radiation information of the thermal infrared orthophoto image with the spectral information and first elevation information of the intermediate fused image according to the registration results of the intermediate fused image and the thermal infrared orthophoto image to obtain the target image, including: resampling the thermal infrared orthophoto image based on the geometric transformation results to obtain a new thermal infrared orthophoto image, wherein at least some pixels of the intermediate fused image correspond one-to-one with at least some pixels of the new thermal infrared orthophoto image; and fusing the spectral information and first elevation information of the intermediate fused image with the thermal radiation information of the new thermal infrared orthophoto image to obtain the target image.

[0107] For example, a second-order polynomial model can be fitted by matching pairs of key feature points. This model maps the image coordinates of each pixel in the thermal infrared orthophoto image to the image coordinates of the intermediate fused image. The second-order polynomial model also performs a geometric transformation on the image coordinates of each pixel in the thermal infrared orthophoto image, aligning the thermal infrared orthophoto image and the intermediate fused image in image space. For example, after the geometric transformation of the thermal infrared orthophoto image using the second-order polynomial model, the image coordinates corresponding to the pixels in the thermal infrared orthophoto image may be non-integer. It can be understood that a pixel array can be constructed based on the range of image coordinates of the pixels in the thermal infrared orthophoto image after the geometric transformation, where the image coordinates of each pixel in the pixel array are integers. For each pixel in the pixel array, the second-order polynomial model can determine the image coordinates of that pixel in the initial thermal infrared orthophoto image; similarly, these image coordinates are likely to be non-integer. Interpolation methods (such as nearest neighbor, bilinear interpolation, cubic convolution interpolation, etc.) can be used to calculate the pixel value of the image coordinates. The calculated pixel value is the pixel value of the corresponding pixel in the pixel array. The process of determining the pixel value of each pixel in the pixel array is the resampling process. After determining the pixel value of each pixel in the pixel array, a new thermal infrared orthophoto image can be generated based on the pixel array with determined pixel values. It can be understood that the coverage area of ​​the thermal infrared orthophoto image at least partially overlaps with the coverage area of ​​the intermediate fused image. The image position of any physical location within the partially overlapping coverage area is consistent with the image position in the intermediate fused image and the image position in the new thermal infrared orthophoto image. In other words, at least some pixels in the intermediate fused image correspond one-to-one with at least some pixels in the new thermal infrared orthophoto image. During fusion, according to the image coordinates of the intermediate fused image and the image coordinates of the new thermal infrared orthophoto image, the thermal radiation information of at least some pixels in the new thermal infrared orthophoto image can be used as a new information dimension and superimposed with the spectral information and elevation information of the corresponding pixels in the intermediate fused image to output the final target image. This approach can align thermal infrared orthophotos with intermediate fused images at sub-pixel level precision.

[0108] Optionally, before fusing the intermediate fused image with the thermal infrared orthophoto to obtain the target image, the thermal infrared orthophoto, the hyperspectral orthophoto, and the first digital surface model are fused to obtain the intermediate fused image. This further includes: determining a mask region of the thermal infrared orthophoto based on the image region of the intermediate fused image; cropping the thermal infrared orthophoto using the mask region to obtain a new thermal infrared orthophoto, wherein the image region of the new thermal infrared orthophoto overlaps with the image region of the intermediate fused image; wherein the operation of fusing the intermediate fused image with the thermal infrared orthophoto is an operation performed on both the intermediate fused image and the new thermal infrared orthophoto.

[0109] For example, to further ensure the consistency of data fusion, a mask region for the thermal infrared orthophoto image can be generated using the image regions of the intermediate fused image. This mask region can then be used to perform a mask cropping operation on the thermal infrared orthophoto image, resulting in a thermal infrared orthophoto image whose image regions overlap with those of the intermediate fused image. It is understood that the coverage area of ​​the cropped thermal infrared orthophoto image is consistent with the coverage area of ​​the intermediate fused image. This approach helps eliminate invalid boundary regions in the thermal infrared orthophoto image.

[0110] Optionally, the raw data of thermal infrared orthophotos, point cloud datasets, and multiple hyperspectral images are data collected by the sensor module on the UAV for flood-prone areas. The UAV is equipped with a coupled satellite navigation module and an inertial measurement unit. Before performing geometric correction on the multiple hyperspectral images using the first digital surface model and multiple sets of pose information, the method further includes: acquiring base station observation data, which includes a first observation epoch sequence and multiple sets of satellite observation data corresponding one-to-one with multiple epochs in the first observation epoch sequence; acquiring UAV observation data, which includes a second observation epoch sequence and multiple sets of navigation state data corresponding one-to-one with multiple epochs in the second observation epoch sequence. The navigation state data includes satellite navigation data collected by the satellite navigation module and inertial measurement data measured by the inertial measurement unit. The satellite navigation data is used to indicate the position of the sensor module, and the inertial measurement unit is used to indicate the position of the sensor module. The data is used to indicate the attitude of the sensor module; epoch synchronization is performed on the base station observation data and navigation status data using a preset precise ephemeris and a preset clock difference file; a double-difference observation model is constructed based on the epoch-synchronized base station observation data and UAV observation data; an ambiguity fixing algorithm is used to determine the integer value of the carrier phase integer ambiguity in the double-difference observation model, so as to solve multiple sets of sensor positioning results corresponding one-to-one with multiple sets of satellite navigation data using the double-difference observation model with the determined carrier phase integer ambiguity; a tightly coupled Kalman filter algorithm is used to jointly estimate the multiple sets of sensor positioning results and inertial measurement data to obtain the trajectory data of the sensor module. The trajectory data is used to represent the pose of the sensor module at each acquisition time during the data acquisition process in the flood-prone area; a preset filtering and smoothing algorithm is used to process the trajectory data to obtain the target trajectory data, which includes multiple sets of pose information.

[0111] For example, the descriptions of the sensor modules and navigation components (including the mutually coupled GNSS satellite navigation module and IMU) carried by the UAV can be found in the foregoing embodiments and will not be repeated here. Those skilled in the art will understand that GNSS base stations can be pre-deployed near flood-prone areas. Because airborne GNSS signals are easily affected by satellite geometry, signal obstruction, and multipath effects in dynamic environments, directly using airborne GNSS observation data can produce errors of tens of centimeters or even meters, failing to meet the accuracy requirements of orthorectification and multi-source data fusion. Therefore, this embodiment of the invention is based on the carrier phase differential positioning (PPK) method, jointly solving the observation data from the base station and GNSS, and combining the inertial measurement data output by the IMU for pose calculation. Specifically, the base station observation data can be a RINEX file. The first observation epoch sequence can include multiple observation epochs, each observation epoch in the first observation epoch sequence referring to a time point or time period during the GNSS base station measurement process when observing satellite signals at a specific moment. At each observation epoch, the GNSS base station can acquire a set of satellite observation data, which may include pseudorange, carrier phase, signal-to-noise ratio, etc. Similarly, the second observation epoch sequence may include multiple observation epochs, each corresponding to two sets of data: one set of data is satellite navigation data measured by the satellite navigation module, and the other set of data is inertial measurement data measured by the inertial measurement unit. In some embodiments, the measurement frequency of the inertial measurement unit is consistent with the measurement frequency of the satellite navigation module; in this case, the inertial measurement data is the raw data measured by the inertial measurement unit. In other embodiments, the measurement frequencies are inconsistent; in this case, measurement data synchronized with each observation epoch of the satellite navigation data can be obtained (using interpolation / filtering methods, etc.) based on the raw data measured by the inertial measurement unit as inertial measurement data. The satellite navigation data may include pseudorange, carrier phase, satellite visibility, etc. The approximate position of the UAV can be determined through the satellite navigation data, and correspondingly, the approximate position of the sensor module can be determined. Inertial measurement data can include acceleration and angular velocity. It can be understood that the attitude of the sensor module can be determined through inertial measurement data, and the attitude can be represented by roll angle, pitch angle and yaw angle.

[0112] For example, since the observation epochs of the base station and the UAV may be inconsistent, a preset precise ephemeris and clock difference file can be used, based on a preset timestamp alignment algorithm, to synchronize the observation epochs of satellite observation data and navigation status data. Each synchronized observation epoch can correspond to a set of satellite observation data and a set of navigation status data. Specifically, either the first observation epoch sequence or the second observation epoch sequence can be used as the reference epoch sequence. The precise ephemeris file (e.g., SP3 file) can provide the three-dimensional coordinates of the satellite at specific time intervals (e.g., 15 minutes), and the clock difference file can provide the deviation between the satellite time and standard GNSS time. Based on the precise ephemeris and clock difference file, the position and clock difference of each satellite at each precise moment can be calculated. Through the timestamp alignment algorithm and interpolation algorithms such as Lagrange interpolation and Chebyshev fitting, the satellite observation data of the base station and the navigation status data of the UAV can be unified onto the same time axis to achieve epoch synchronization of satellite observation data and navigation status data. A double-difference observation model can be constructed based on the synchronized base station observation data and UAV observation data. The double-difference observation model has a parameter called carrier phase integer ambiguity. An ambiguity fixing algorithm can determine the integer value of the carrier phase integer ambiguity, which can then be used to calculate the baseline vector of the UAV relative to the base station. By adding the baseline vector to the pre-determined three-dimensional coordinates of the base station, the three-dimensional coordinates of the UAV's GNSS antenna phase center can be obtained. Furthermore, based on the pre-determined relative pose between the UAV and the sensor module, the positioning result of the sensor module, i.e., the sensor positioning result, can be obtained.

[0113] For example, a tightly coupled Kalman filter algorithm can be used to jointly estimate sensor positioning results and inertial measurement data. It should be noted that when a UAV temporarily loses satellite signals due to low-altitude flight or obstructions, the UAV's pose can be estimated from the IMU's inertial measurement data in a short period. When the satellite signal recovers, the tightly coupled Kalman filter algorithm can be used to reverse-calculate the cumulative error generated by the IMU during the period of satellite signal loss, thereby correcting the inertial measurement data during this period. In other words, during the UAV's flight, the tightly coupled Kalman filter algorithm can fuse inertial measurement data and sensor positioning results into an optimal solution to obtain the trajectory data of the sensor module. The resulting trajectory data is continuous and smooth. For example, forward and backward filtering smoothing algorithms can be used to smooth the trajectory data, and the estimation results of forward and backward filtering can be fused. The state at each moment can be re-estimated using UAV observation data from all moments along the entire trajectory, thereby obtaining globally optimal smoothed trajectory data, i.e., target trajectory data. Target trajectory data includes multiple sets of pose information corresponding one-to-one with multiple acquisition moments. In a specific embodiment, the target trajectory data is an SPET file, which includes six core parameters for each acquisition moment: longitude, latitude, elevation, roll angle, pitch angle, and heading angle. The above technical solution, through epoch synchronization and ambiguity fixation, can eliminate satellite clock errors and most atmospheric delay errors through the constraints of the double-difference observation equation. By jointly estimating the sensor positioning results calculated based on GNSS and the inertial measurement data of the IMU, relatively accurate smoothed trajectory data can be obtained. Furthermore, the optimal solution for the trajectory data can be obtained through forward and backward filtering algorithms.

[0114] By way of example, and not limitation, the raw data format of base station observation data is typically a receiver-proprietary binary format, such as HCN format. In this case, the RINEX conversion tool can be used to convert the raw base station observation data into a RINEX file based on the RINEXI standard protocol. It is understood that HCN format base station observation data (i.e., satellite observation records) includes pseudorange, carrier phase, signal-to-noise ratio, and observation timestamps, and may also include metadata such as sampling interval and antenna type. In a specific embodiment, HCN format base station observation data can be reconstructed according to the RINEX 3.04 standard format. The reconstructed RINEX file can have standardized header information, which may include, for example, antenna type, altitude, base station coordinates, and sampling frequency. The reconstructed RINEX file can also include the aforementioned pseudorange, carrier phase, signal-to-noise ratio, and observation time. This format conversion facilitates the sharing of a unified time system and coordinate reference framework between base station observation data and UAV observation data, providing a more accurate data foundation for the aforementioned carrier phase difference calculation.

[0115] According to another aspect of the present invention, a sensor payload system for an unmanned aerial vehicle (UAV) is also provided, characterized in that the sensor payload system includes a sensor module and a power distribution module. The sensor module includes a bracket, a lidar sensor, a hyperspectral sensor, and a thermal infrared sensor mounted on the bracket. The lidar sensor and the hyperspectral sensor are rigidly connected. The lidar sensor, the thermal infrared sensor, and the thermal infrared sensor are arranged along a preset direction. The scanning center of the lidar sensor, the optical center of the hyperspectral sensor, and the optical center of the thermal infrared sensor are aligned with each other to form the same observation baseline. At least a portion of the field of view of each of the lidar sensor, the hyperspectral sensor, and the thermal infrared sensor overlaps with each other. The power distribution module is connected to the sensor module and is used to supply power to the sensor module.

[0116] For example, the sensor payload system is directly powered by the UAV platform, with a supply voltage range of, for example, 18-24VDC. The power distribution module may include multi-stage voltage regulation and filtering submodules. This module can acquire the supply voltage output from the UAV platform and power the sensor module connected thereto. For example, the sensor module's bracket may have a quick-release mounting mechanism, which can be used to mount the entire sensor payload system onto the UAV or detach it from the UAV. See also... Figure 8 As shown, it is a schematic structural diagram of a sensor load system according to an embodiment of the present invention. Figure 8 The quick-release bracket is a quick-release installation structure. Figure 8The gray area of ​​the sensor payload system represents the outer casing; the lens surfaces of the hyperspectral imager (i.e., the hyperspectral sensor) and the thermal infrared camera (i.e., the thermal infrared sensor) are not covered by the casing. In one specific embodiment, to balance structural rigidity and thermal conductivity, the bracket is made of anodized aerospace-grade aluminum alloy. Exemplarily, the lidar sensor, hyperspectral sensor, and thermal infrared sensor can all be mounted on the bracket, and the hyperspectral sensor can be rigidly connected to the lidar sensor. Please refer to [link to relevant documentation]. Figure 9 The diagram shown is a physical schematic of a sensor payload system according to an embodiment of the present invention. Please refer to [link / reference]. Figure 10 As shown, it is a schematic diagram of a sensor payload system mounted on a drone according to an embodiment of the present invention. Figure 10 The sensor payload system in the illustrated embodiment is... Figure 9 The sensor payload system is shown. When mounted on a UAV, the lidar sensor, hyperspectral sensor, and thermal infrared sensor can be arranged sequentially along the UAV's flight direction, with the lidar sensor in front, the hyperspectral sensor in the middle, and the thermal infrared sensor at the rear. The scanning centers of the lidar sensor, the optical centers of the hyperspectral sensor, and the thermal infrared sensor are aligned with each other, forming a common observation baseline that ensures that the fields of view of the three sensors overlap.

[0117] Optionally, the lidar sensor and hyperspectral camera are mounted on the first power supply circuit, and the thermal infrared sensor is mounted on the second power supply circuit. The power supply module is specifically used to supply power to the first power supply circuit and the second power supply circuit respectively.

[0118] For example, the first power supply circuit and the second power supply circuit are different power supply circuits. Specifically, the hyperspectral sensor and the lidar sensor can share the main power supply circuit (the main power supply circuit is the first power supply circuit), while the thermal infrared sensor can be powered by an independent power supply branch. This power supply method allows for staggered / time-sharing startup of the thermal infrared sensor, as well as the hyperspectral sensor and the lidar sensor, thereby reducing the transient impact of startup current on the sensor load system.

[0119] Optionally, the sensor payload system also includes a control module, which is located inside the housing of the sensor module and is communicatively connected to the lidar sensor and the hyperspectral sensor, respectively. The control module, lidar sensor, and hyperspectral sensor are connected to the same power supply circuit. The control module is used to set the operating parameters of the lidar sensor and the hyperspectral sensor, send start commands and / or stop commands to the lidar sensor and the hyperspectral sensor, and receive data sent by the lidar sensor and the hyperspectral sensor.

[0120] For example, the control module may be an embedded Linux computer host, the sensor module has a housing, and the control module may be located inside the housing of the sensor module. The control module may share the same power supply circuit with the lidar sensor and the hyperspectral sensor; more specifically, the control module, lidar sensor, and hyperspectral sensor share a main power supply circuit. For example, the control module may configure operating parameters for the lidar sensor and the hyperspectral sensor respectively, and the lidar sensor and hyperspectral sensor may acquire data according to their respective operating parameters. More specifically, the user may communicate with the control module via a network port, send the sensor's operating parameters to the control module, and monitor the sensor's operating status through the control module. The control module may also send start and / or stop commands to the lidar sensor and the hyperspectral sensor. The start command is used to set the sensor from a non-operating state to an operating state, and correspondingly, the stop command is used to set the sensor from an operating state to a non-operating state. When the sensor is in an operating state, it may acquire data in response to a trigger signal. The lidar sensor and the hyperspectral sensor may send the acquired data to the control module. For example, but not limitingly, the user may also remotely download data received by the control module via a network port.

[0121] The above technical solution can centrally control the start-up, sampling and data recording of the two types of sensors through the control module, thereby reducing user operation and improving the reliability and ease of use of the sensor system.

[0122] Optionally, a first fan is provided on the inner side wall of the bracket. The airflow provided by the first fan can pass through the longitudinal air duct inside the bracket sequentially through the motherboard of the hyperspectral sensor and the motherboard of the control module, and be discharged through the exhaust port near the motherboard of the control module. A second fan is provided inside the thermal infrared sensor for heat dissipation of the thermal infrared sensor.

[0123] Exemplarily, a longitudinal air duct is provided inside the bracket, and a first fan can be positioned near the end of the longitudinal air duct. The first fan can introduce external airflow and supply it to the interior of the bracket. More specifically, the airflow provided by the first fan can sequentially pass through the motherboard of the hyperspectral sensor and the motherboard of the control module. An exhaust port is provided on the side wall of the bracket near the motherboard of the control module, through which airflow can be discharged. Exemplarily, the thermal infrared sensor can control its own temperature through a built-in second fan. Exemplarily, and not limitingly, the thermal infrared sensor can automatically perform flat field correction (FFC) operations. This heat dissipation structure can provide stable airflow and reliable temperature management.

[0124] Optionally, the sensor payload system also includes a power cable. The power distribution module is electrically connected to the UAV platform via the power cable to obtain the power supply voltage provided by the UAV platform. It is also electrically connected to the lidar sensor, hyperspectral sensor, and thermal infrared sensor via the power cable to supply power to the lidar sensor, hyperspectral sensor, and thermal infrared sensor respectively.

[0125] For example, the power cord plug is preferably a quick-connect plug with anti-loosening features to ensure a secure connection, and the power cord cable is preferably an aviation-grade shielded cable to ensure signal integrity and interference immunity. The UAV platform can supply power to the power distribution module via the power cord, and the power distribution module can distribute the power supply voltage to the lidar sensor, hyperspectral sensor, and thermal infrared sensor via the power cord. Voltage distribution via the power distribution module helps ensure stable power supply voltage for each sensor and helps suppress electromagnetic interference. By powering the system from the UAV platform and having the power distribution module perform voltage regulation, filtering, and distribution, the use of additional batteries or power conversion modules can be avoided, thereby significantly reducing the weight and maintenance complexity of the sensor payload system.

[0126] Optionally, the sensor payload system also includes signal lines, with power lines and signal lines laid out in layers, and the sensor payload system also includes a satellite navigation module.

[0127] For example, power lines and signal lines are laid out in separate layers to avoid mutual interference and to simplify airborne wiring and facilitate inspection.

[0128] Optionally, the thermal infrared sensor has a GPS module, which is connected to the GPS antenna on the UAV via a first signal line. The GPS module is used to calculate the timestamp information of the thermal infrared sensor during data acquisition based on the GPS signal received by the GPS antenna. The timestamp information of the GPS module is used to align with the trigger timestamp information generated by the control module. The trigger timestamp information is used to indicate the data acquisition time of the lidar sensor and the hyperspectral sensor.

[0129] Those skilled in the art will understand that a satellite navigation module (GNSS module) can provide absolute time information (e.g., UTC time text) and PPS pulse signals, which together constitute a complete time reference. The control module can output a set of trigger timestamp information based on the time reference provided by the satellite navigation module at the time of data acquisition by the hyperspectral sensor and lidar sensor. This set of trigger timestamp information can then be bound to the data acquired by the hyperspectral sensor and lidar sensor based on the corresponding trigger signal. For example, the UAV can be equipped with a first GPS antenna and a second GPS antenna. The first GPS antenna can be connected to the GNSS module of the navigation component. The GNSS module of the navigation component can calculate the positioning result data, i.e., the satellite navigation data of the aforementioned embodiment, based on the GPS signal received by the first GPS antenna. The second GPS antenna can be connected to the GPS module of the thermal infrared sensor. Specifically, the thermal infrared sensor itself has a GPS module, which can be connected to the second GPS antenna. Each time the thermal infrared sensor acquires data, the GPS module can calculate the timestamp information of the thermal infrared sensor's data acquisition based on the GPS signal received by the second GPS antenna. In other words, the data acquired by the thermal infrared sensor each time can be bound to a set of timestamp information. The control module and / or the processor used to execute the flood disaster multi-source data processing method of this embodiment can both perform the operation of aligning the trigger timestamp information generated by the control module with the timestamp information of the GPS module. Specifically, the trigger timestamp information can be recorded in the UAV's track log, which may include the target trajectory data of the aforementioned embodiment. By aligning the timestamp information of the GPS module with the trigger timestamp information in the track log, the pose information of the thermal infrared sensor on the UAV / UAV at each data acquisition can be determined by interpolation calculation. Although this scheme does not use a PPS pulse signal to trigger the thermal infrared sensor for image acquisition, it can ensure that the deviation between the actual acquisition time of the thermal infrared sensor and the timestamp information is controlled within milliseconds, and the spatial positioning error of the corresponding pose information can be ignored.

[0130] Optionally, the sensor payload system further includes a trigger module. The trigger module is communicatively connected to the control module via a second signal line and is also communicatively connected to the lidar sensor and the hyperspectral sensor via the second signal line. The trigger module is used to respond to the trigger control command sent by the control module, send a trigger signal to the lidar sensor and the hyperspectral sensor, and report the trigger event to the control module. The control module is also used to respond to the trigger event and generate trigger timestamp information based on the time reference provided by the satellite navigation module. The trigger timestamp information is used to indicate the data acquisition time of the lidar sensor and the hyperspectral sensor.

[0131] For example, the control module can receive pose information output by the navigation component in real time, and determine whether the UAV has entered the target acquisition area based on the pose information and preset acquisition area or route planning information. When the control module determines that the UAV has entered the target acquisition area, it can send a trigger control command to the trigger module. Responding to the received trigger control command, the trigger module can output synchronous trigger signals to the lidar sensor and hyperspectral sensor through a multi-channel hardware trigger interface under the constraint of the PPS time base, thereby triggering the lidar sensor and hyperspectral sensor to synchronously acquire data under a unified time base. When the trigger module outputs a trigger signal, it can record the corresponding hardware count value or trigger event identifier, and report the trigger event to the control module through a second signal line. In response to the received trigger event, the control module can generate corresponding trigger timestamp information by combining the absolute time information provided by the satellite navigation module and the PPS time base. It can be understood that the trigger timestamp information indicates the time when the trigger module sends the trigger signal, that is, the data acquisition time of the lidar sensor and the hyperspectral sensor. This approach enables synchronous triggering and high-precision timestamp generation of multiple sensors in a unified time coordinate system, providing a reliable time basis for subsequent data fusion and high-precision positioning.

[0132] Optionally, the control module is connected to the lidar sensor and the hyperspectral sensor via a third signal line. The control module is also used to acquire data collected by the lidar sensor and the hyperspectral sensor via the third signal line.

[0133] For example, the control module can acquire data collected by the lidar sensor and the hyperspectral sensor through the third signal line, and can associate and store these data with the trigger timestamp information in the aforementioned embodiment.

[0134] Optionally, the sensor payload system also includes a navigation component, which includes a satellite navigation module and an inertial measurement unit coupled to each other. The navigation component is communicatively connected to the control module via a fourth signal line. The control module is also used to acquire satellite navigation data collected by the satellite navigation module and inertial measurement data measured by the inertial measurement unit.

[0135] Similarly, satellite navigation data and inertial measurement data can be associated and stored with the trigger timestamp information and data collected by sensors in the aforementioned embodiments, which can accurately associate the raw data of image data, pose information and time information.

[0136] Optionally, the lidar sensor communicates with the control module via a network port; and / or, the hyperspectral sensor communicates with the control module via a PCIe interface; and / or, the sensor payload system further includes a navigation component, which includes a satellite navigation module and an inertial measurement unit coupled to each other, and the navigation component communicates with the control module via a network port and a serial port; and / or, the thermal infrared sensor communicates with the terminal device via Bluetooth.

[0137] In one specific embodiment, the control module is a Linux host. The lidar sensor communicates with the Linux host via Ethernet, the hyperspectral sensor communicates with the Linux host via a high-speed PCIe interface, and the navigation component communicates with the Linux host via RS232 serial port and Ethernet. The thermal infrared sensor can be configured with a dedicated application that can be installed on a terminal device, and its operating parameters can be controlled via Bluetooth wireless communication. This approach helps ensure high throughput, low latency, and high reliability in communication.

[0138] Please see Figure 11 The diagram shown is a hardware architecture diagram of a sensor payload system according to an embodiment of the present invention. Figure 11 In the diagram, the dashed lines represent the outer casing of the sensor payload system, and the hardware within the dashed lines belongs to various types of hardware within the sensor payload system. The functions and connection methods of these various hardware components can be found in the descriptions of the foregoing embodiments, and will not be repeated here. For example, Figure 11 The network port 1 shown is used to connect to the data interface of the LiDAR sensor, receiving the raw point cloud data sent by the LiDAR sensor. Network port 2 is used to connect to an external computer, allowing access to a Linux host, which can then be used to view the operating status of each sensor and configure its parameters. It should be noted that the navigation component ( Figure 11 The high-precision GNSS / IMU module shown also communicates with the Linux computer host via the network port. Figure 11 The communication connection is not shown.

[0139] Please see Figure 12 As shown, it is a schematic block diagram of an electronic device 1200 according to an embodiment of the present invention. According to another aspect of the present invention, an electronic device is also provided, including: a processor 1210 and a memory 1220, wherein the memory 1220 stores computer program instructions, which are executed by the processor 1210 to perform the above-mentioned multi-source data processing method for flood disasters.

[0140] According to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the computer or processor performs the corresponding steps of the multi-source data processing method for flood disasters described in the embodiments of the present invention. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. A computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0141] According to another aspect of the present invention, a computer program product is also provided, including computer program instructions, which, when executed, are used to perform the multi-source data processing method for flood disasters as described above.

[0142] Those skilled in the art can understand the specific implementation and beneficial effects of the above-mentioned sensor payload system, electronic equipment, storage medium, and computer program products by reading the above-described detailed description of the multi-source data processing method for flood disasters. For the sake of brevity, they will not be described in detail here.

[0143] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0144] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0146] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0147] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0148] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or elements of any method or apparatus so disclosed may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0149] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0150] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0151] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0152] The above are merely specific embodiments or descriptions of the present invention, and the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for processing multi-source data on flood risks, characterized in that, The method includes: The system acquires thermal infrared orthophotos, point cloud datasets, and multiple hyperspectral images of the flood-prone area. Each pixel in the thermal infrared orthophoto image has thermal radiation information representing the thermal radiation at the corresponding physical location. Each set of point cloud data in the point cloud dataset includes first location information representing the corresponding physical location. Each pixel in each of the multiple hyperspectral images has spectral information corresponding to the physical location. The multiple hyperspectral images correspond one-to-one with multiple sets of pose information, and the pose information represents the pose of the hyperspectral sensor when acquiring images of the flood-prone area at the corresponding acquisition time. A first digital surface model of the flood-prone area is generated based on the point cloud dataset. The first digital surface model includes the first elevation information of each physical location of the surface features in the flood-prone area. The first digital surface model and the multiple sets of pose information are used to perform geometric correction on the multiple hyperspectral images respectively, so as to generate a hyperspectral orthophoto based on the geometrically corrected multiple hyperspectral images. Each pixel of the hyperspectral orthophoto has spectral information corresponding to the physical location. The thermal infrared orthophoto, the hyperspectral orthophoto, and the first digital surface model are fused to obtain a target image. Each pixel of the target image has elevation information, spectral information, and thermal radiation information corresponding to its physical location.

2. The method according to claim 1, characterized in that, Acquiring the thermal infrared orthophoto image includes: Multiple thermal infrared images of the flood-prone area were acquired by the thermal infrared sensor on the drone. Sparse three-dimensional point cloud data is generated by using multiple sets of exterior orientation parameters that correspond one-to-one with the multiple thermal infrared images and the interior orientation parameters of the thermal infrared sensor. Each set of exterior orientation parameters is used to indicate the pose of the thermal infrared sensor when acquiring the corresponding thermal infrared image. The interior orientation parameters include focal length, principal point position parameters, and distortion coefficient. Using a multi-view stereo vision dense matching algorithm, high-density three-dimensional point cloud data is generated based on the sparse three-dimensional point cloud data and the multiple thermal infrared images, and a second digital surface model is generated based on the high-density three-dimensional point cloud data. The pixel values ​​of the plurality of thermal infrared images are mapped to the corresponding surface positions of the second digital surface model to obtain the thermal infrared orthophoto image, wherein the thermal radiation information is represented by the pixel values.

3. The method according to claim 2, characterized in that, Before generating high-density three-dimensional point cloud data based on the sparse three-dimensional point cloud data and the multiple thermal infrared images, the acquisition of thermal infrared orthophotos of the flood-prone area further includes: The image positions of preset ground control points are determined in the multiple thermal infrared images. Using the image positions of the ground control points, the sparse three-dimensional point cloud data is optimized using the overall adjustment method. The operation of generating high-density three-dimensional point cloud data based on the sparse three-dimensional point cloud data and the multiple thermal infrared images is performed on the optimized sparse three-dimensional point cloud data. And / or, The process of generating sparse three-dimensional point cloud data by utilizing multiple sets of exterior orientation parameters corresponding one-to-one with the multiple thermal infrared images and the interior orientation parameters of the thermal infrared sensor includes: Feature points from the multiple thermal infrared images are extracted and matched using the structure-of-motion algorithm, and the multiple sets of exterior orientation parameters and interior orientation parameters are determined based on the matching results. Based on the multiple sets of external orientation parameters and the internal orientation parameters, a spatial topological relationship is established between the multiple thermal infrared images to obtain sparse three-dimensional point cloud data. And / or, Each surface location of the second digital surface model has second location information for representing the corresponding physical location. Each of the plurality of thermal infrared images corresponds to a partial surface region of the second digital surface model. The step of mapping the pixel values ​​of the plurality of thermal infrared images to the corresponding surface locations of the second digital surface model to obtain the thermal infrared orthophoto image includes: For each of the plurality of thermal infrared images, a first collinearity equation is established based on the second position information of the second digital surface model, the exterior orientation parameter and the interior orientation parameter corresponding to the thermal infrared image; Based on the first collinearity equation, at least a portion of the surface location of the surface region on the second digital surface model corresponding to the thermal infrared image is determined to be the image location of the thermal infrared image in the thermal infrared image. For each surface location of the second digital surface model, the pixel value of the target thermal infrared image located at the image location corresponding to that surface location is mapped to that surface location to obtain the thermal infrared orthophoto image, wherein the target thermal infrared image is the thermal infrared image corresponding to the surface region to which that surface location belongs.

4. The method according to claim 2, characterized in that, The plurality of thermal infrared images includes at least one pair of thermal infrared images, wherein the field of view of each pair of thermal infrared images partially overlaps. After mapping the pixel values ​​of the plurality of thermal infrared images to the corresponding surface positions of the second digital surface model to obtain the thermal infrared orthophoto, the acquisition of the thermal infrared orthophoto further includes: Post-processing operations are performed on the target image region of the thermal infrared orthophoto to obtain the final thermal infrared orthophoto. The post-processing operations include color balancing and mosaicking operations. The target image region includes the image region corresponding to the overlapping area of ​​the field of view of the at least one pair of thermal infrared images.

5. The method according to claim 1, characterized in that, Each surface position of the first digital surface model has third position information representing the corresponding physical position, the third position information including the first elevation information. The step of geometrically correcting the multiple hyperspectral images using the first digital surface model and the multiple sets of pose information to generate a hyperspectral orthophoto based on the geometrically corrected multiple hyperspectral images includes: For each of the plurality of hyperspectral images, a second collinearity equation is constructed based on the third position information of the first digital surface model, the pose information corresponding to the hyperspectral image, and the interior orientation parameters of the hyperspectral sensor. For each of the plurality of hyperspectral images, Based on the image position of each pixel in the hyperspectral image and the second collinearity equation, the surface position on the first digital surface model corresponding to each pixel is determined. Based on the surface position corresponding to each pixel of the hyperspectral image, the pixel value of each pixel of the hyperspectral image is mapped to a preset grid image corresponding to the first digital surface model, and a resampling algorithm is used to determine the pixel value of each grid of the preset grid image to obtain an orthophoto image corresponding to the hyperspectral image, wherein each image position of the preset grid image corresponds to at least a portion of the surface position of the first digital surface model. The hyperspectral orthophoto is obtained by stitching together multiple orthophotos that correspond one-to-one with the multiple hyperspectral images.

6. The method according to claim 5, characterized in that, Each pixel of the plurality of orthophotos has fourth location information representing the corresponding physical location. The step of stitching together the plurality of orthophotos, which correspond one-to-one with the plurality of hyperspectral images, to obtain the hyperspectral orthophoto image includes: Based on the fourth position information of the plurality of orthophoto images, the plurality of orthophoto images are aligned, and the overlapping areas of the plurality of orthophoto images are determined; Perform spectral consistency processing on the overlapping regions; Determine the stitching lines of the plurality of orthophotos, and stitch the plurality of orthophotos together according to the stitching lines to obtain a stitched image; The image region containing the stitching line in the stitched image is feathered to obtain the hyperspectral orthophoto.

7. A sensor payload system for an unmanned aerial vehicle (UAV), characterized in that, The sensor payload system includes a sensor module and a power distribution module. The sensor module includes a bracket, a lidar sensor, a hyperspectral sensor, and a thermal infrared sensor mounted on the bracket. The lidar sensor and the hyperspectral sensor are rigidly connected. The lidar sensor, the thermal infrared sensor, and the thermal infrared sensor are arranged along a preset direction. The scanning center of the lidar sensor, the optical center of the hyperspectral sensor, and the optical center of the thermal infrared sensor are aligned with each other to form the same observation baseline. At least a portion of the field of view of each of the lidar sensor, the hyperspectral sensor, and the thermal infrared sensor overlaps with each other. The power distribution module is connected to the sensor module and is used to supply power to the sensor module.

8. An electronic device, characterized in that, The method includes a processor and a memory, characterized in that the memory stores computer program instructions, which, when executed by the processor, are used to perform the multi-source data processing method for flood hazards as described in any one of claims 1-7.

9. A storage medium storing a computer program / instructions, characterized in that, The computer program / instructions are used to execute the multi-source data processing method for flood risks as described in any one of claims 1-7 when the program is running.

10. A computer program product, characterized in that, The method includes computer program instructions, characterized in that the computer program instructions are used to execute the multi-source data processing method for flood risks as described in any one of claims 1-7 when running.

Images