A method and system for low-altitude UAV waterlogging inspection based on deep learning

Abstract

The present invention belongs to the field of waterlogging monitoring, and discloses a low-altitude UAV waterlogging inspection method and system based on deep learning, the method comprising: obtaining remote sensing visible light images with high temporal and spatial resolution through a low-altitude UAV, and uploading the UAV data to the cloud using 5G communication technology; performing real-time identification and segmentation of waterlogging based on a deep learning automatic waterlogging recognition and analysis model of UAV waterlogging image segmentation, and realizing inspection of urban waterlogging information efficiently, accurately and quickly; combining UAV trajectory information and image edge feature splicing inspection area maps to segment waterlogged areas and draw and update the spatiotemporal distribution pattern of urban waterlogging, so as to provide technical support and basic data support for disaster reduction, prevention, emergency rescue and disaster assessment of urban waterlogging disasters.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of waterlogging monitoring technology, and in particular relates to a low-altitude unmanned aerial vehicle waterlogging inspection method and system based on deep learning. Background Art

[0002] At present, the monitoring methods of urban waterlogging include manual inspection, instrument measurement, and satellite remote sensing monitoring. Specifically, manual inspection: requires a large amount of manpower input, personnel safety is threatened to a certain extent during the inspection process, and human resources are consumed. Waterlogging instrument measurement: mostly fixed installation at historical flood-prone points, can only measure the waterlogging process within a limited space, so the spatial coverage is limited, and it is difficult to accurately capture the random occurrence of urban waterlogging information. The large-scale and high-density deployment of waterlogging instrument sensor networks is costly and difficult to maintain; satellite remote sensing observation: can observe urban flood information over a large area, but due to the serious obstruction of urban buildings and the complex and changeable terrain, the accuracy of waterlogging detection is insufficient. In addition, the satellite revisit cycle is long, and the real-time performance of urban waterlogging monitoring is poor.

[0003] With the rapid progress and development of UAV flight control technology, UAVs are playing an increasingly important role in agriculture, transportation, urban management, military and other fields. Urban waterlogging monitoring based on UAVs has the advantages of low cost, simple operation, and rapid data transmission, and has become a new growth point for urban waterlogging monitoring. The present invention takes low-altitude UAV images as the starting point, and adopts theoretical methods such as computer vision, deep learning, and probability theory to establish an automatic identification and regional segmentation method for waterlogging / ponding water; on this basis, the UAV trajectory information and image edge features are combined to splice the inspection area image to segment the waterlogged area and draw the spatiotemporal distribution of the urban waterlogging pattern, thereby realizing the perception and release of urban waterlogging information.

[0004] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:

[0005] (1) The manual inspection method requires a lot of manpower input, and the safety of personnel is threatened to a certain extent during the inspection process, which consumes a lot of human resources;

[0006] (2) The spatial coverage of waterlogging instrument measurement methods is limited, making it difficult to accurately capture random urban waterlogging information. They are also costly and difficult to maintain.

[0007] (3) Satellite remote sensing observation methods are not accurate enough in detecting waterlogging, and the real-time performance of waterlogging monitoring is poor.

[0008] The differences between this invention patent and current drone-related patents are as follows:

[0009] CN116776504A provides a method for arranging municipal rainwater and sewage pipes based on drone oblique photography technology. This patent determines the route of the drone by combining on-site exploration with electronic maps, and uses oblique photography data to generate a three-dimensional real-life model to further design the connection between the rainwater and sewage pipe network and other pipes and the location of the pump station, and proposes the optimal municipal rainwater and sewage pipe design. Its main contribution is to use drone oblique photography to generate a three-dimensional real-life model, which provides data support for the design and planning of municipal rainwater and sewage pipes.

[0010] CN111812268A provides a method and control device for atmospheric monitoring using drones. This patent obtains drone images and atmospheric data (including wind direction, wind speed, air quality, etc.) by drones cruising monitoring points and plans routes accordingly to determine unknown pollution sources. Its main contribution is to provide a route planning method and control device based on atmospheric data from monitoring points.

[0011] CN110927342A provides a method for tracing the source of atmospheric pollutants based on the beetle whisker search algorithm. This patent installs gas sensors on the left and right ends of the drone, reads the pollutant concentration at both ends of the drone in real time, determines the next position of the drone through the beetle whisker search algorithm, and continuously iterates to gradually approach the pollution source. Its main contribution is that it provides a pollution source location method (beetle whisker search algorithm) that does not rely on wind speed information, and the tracing is accurate and fast, with high search efficiency.

[0012] The invention patent provides a method for waterlogging inspection using a low-altitude drone based on deep learning. This patent controls the drone to conduct periodic inspections at low altitudes, captures orthophoto data, and processes and analyzes it through a high-precision, high-performance deep learning model to identify waterlogging and achieve accurate waterlogging area segmentation. Then, the inspection image is spliced ​​based on the drone information (including attitude, geographic location, lens focal length, etc.) and image edge features, the waterlogging area is segmented, and the spatiotemporal distribution pattern of waterlogging is drawn and published, providing high-spatiotemporal resolution waterlogging observation data for meteorological, hydrological and other related scientific research and business needs, which is of great significance for disaster warning, rescue and emergency relief, flood modeling, etc. Summary of the invention

[0013] In response to the problems existing in the prior art, the present invention provides a low-altitude UAV waterlogging inspection method and system based on deep learning.

[0014] The present invention is implemented as follows: a low-altitude UAV waterlogging inspection method based on deep learning, characterized in that the low-altitude UAV waterlogging inspection method based on deep learning specifically includes:

[0015] S1: Obtain remote sensing visible light images with high temporal and spatial resolution through low-altitude drones, and upload drone data to the cloud using 5G communication technology;

[0016] S2: A deep learning automatic waterlogging recognition and analysis model based on drone waterlogging image segmentation can be used to perform real-time waterlogging recognition and segmentation, and efficiently, accurately and quickly implement urban waterlogging information inspection;

[0017] S3: Combine the drone trajectory information and image edge features to stitch the inspection area map to segment the waterlogged area and draw and update the spatiotemporal distribution pattern of urban waterlogging.

[0018] Further, the S1 includes:

[0019] (1-1) Determine the latitude and longitude of the drone’s starting point ,The size of the area of ​​interest in vertical photography by drone Square meters, low altitude inspection height meters, S-shaped inspection route, every Meters to set the drone photography point;

[0020] (1-2) After checking that the drone equipment is normal, take off the drone vertically from the starting point to a low altitude inspection height rice;

[0021] (1-3) After the drone confirms that it has arrived at the photography point, it adjusts its posture to be stable and points the lens vertically downward to shoot 4K images, and then uploads the images and the drone's current coordinates via 5G communication technology. to the cloud;

[0022] (1-4) The drone accelerates along the inspection route until it reaches the next photography point and executes step (1-3);

[0023] (1-5) When the drone reaches the destination, it adjusts its direction, returns along the path and executes (1-2) to (1-4).

[0024] Further, the S2 includes:

[0025] (2-1) Separate the illumination component and reflectance component of waterlogging / flooding based on the Retinex theory:

[0026]

[0027] in, For drone images of waterlogging, is the reflectivity component, is the illumination component, The Retinex theory is used to model the complex changes of waterlogging / inland flooding images. Volume module separated from drone image The illumination component and reflectance component of the first image are obtained by , the illumination component of the first image , the reflectance component of the second image , the illumination component of the second image :

[0028]

[0029] (2-2) The light component and The reflectivity components are blended to reconstruct a new image , The light component and The reflectivity components are blended to reconstruct a new image , forming respectively , Images of the same scene with different lighting conditions are used to analyze the reflectivity of objects in the same scene with different lighting conditions.

[0030]

[0031] (2-3) Self-attention module Further extracting the image , Object edges, brightness, texture and other features are input into the parameter-sharing convolution module. Separate the image reflectance component , , and the lighting component , ;

[0032]

[0033] (2-4) The image component separation loss is defined as the pixel loss of the reflectance components extracted from the original image and the reconstructed image to constrain the model to learn the consistency of reflectance under different brightness conditions:

[0034]

[0035] (2-5) For each group of convolution modules Separating the results, we apply a retinex loss to ensure that the organization of reflectance and illumination is consistent with Retinex theory:

[0036]

[0037] (2-6) , Take the following as an example to illustrate the model structure. , Input segmentation parser, through module Extract separately , The high-level features are then input into the convolutional layer. Compute the attention mask to quantitatively evaluate the pixel richness of semantic information:

[0038]

[0039] in, , with different subscripts Represents the learnable weights of the convolution. represents the convolution operation, express Function, maps the pixel values ​​of the convolved image to (0,1) for normalization. is an attention mask that indicates the pixel-wise richness of semantic information in the lighting.

[0040] (2-7) Fuse with illumination and reflectance images and input into convolution module , and obtain the probability value of each pixel belonging to the waterlogging category. When the probability is greater than the set threshold of 0.5, it is judged as the waterlogged part to achieve accurate segmentation of the waterlogged area and obtain the final waterlogging segmentation image In the image, the white area is the waterlogged area and the black area is the background.

[0041]

[0042] (2-8) Add loss function Constraint Model:

[0043]

[0044] in represents the cross entropy loss function, Human-annotated images representing waterlogging images.

[0045] Further, the S3 includes:

[0046] (3-1) Map the pixel coordinate system to the camera coordinate system based on the intrinsic and extrinsic parameters transmitted by the camera, so that the image is mapped to the actual image. First, the center of the image in the pixel coordinate system is Multiply by the length of the pixel ,Width , mapped to the origin of the image coordinate system ,as follows:

[0047]

[0048] Then the other coordinates in the pixel coordinate system according to Scaling and translation, mapping to image coordinate system ,as follows:

[0049]

[0050] Finally, by scaling, Mapping to the camera coordinate system ,as follows:

[0051]

[0052] in, Indicates the distance between the camera lens and the ground. is the focal length of the camera;

[0053] (3-2) Assigning the coordinate information of the drone when taking the photo to the image as the coarse-grained position information of the image;

[0054] (3-3) Select the images to be stitched based on the drone inspection time-series trajectory information The scale-invariant feature transformation algorithm is used to extract image feature points.

[0055]

[0056] in is a scale-invariant feature transformation algorithm. is a set of feature points in the two matrix regions, and each feature point is represented by a high-dimensional vector.

[0057] (3-4) Retrieve the best matching feature points using the k-nearest neighbor algorithm.

[0058] for The feature points in ,exist Find its nearest neighbor and next nearest neighbor And calculate the Euclidean distance:

[0059]

[0060] in Represents the Euclidean distance between two feature points. Represents the dimension of feature points, Represents the highest dimension of feature points.

[0061] The ratio of the nearest neighbor distance to the next nearest neighbor distance is used to determine whether the match of the nearest neighbor feature point is valid. We set the ratio threshold to 0.2. If it is less than 0.2, the nearest neighbor feature point of the match is considered valid. Apply the above operation to all feature points to obtain the feature point matching set , including feature points and matching points corresponding to the feature points.

[0062]

[0063] (3-5) In The random sampling consensus algorithm is used to calculate the affine matrix and transform the image Then stitch the two images together to get .

[0064]

[0065] in represents the random sampling consensus algorithm, Indicates image stitching. For the overlapping pixels of two images, the new pixel points are calculated using the weighted average method of pixel values.

[0066] (3-6) Repeat steps (3-3) to (3-5) to stitch all inspection area sub-maps to obtain a complete inspection area map.

[0067] (3-7) Crop the complete inspection area map into several images of regions of interest;

[0068] (3-8) Feed the cropped image into the network to obtain several waterlogging segmentation images, which are then stitched together into a complete waterlogging area map within the inspection area;

[0069] (3-9) Calculate the area of ​​the waterlogged area and determine whether it is waterlogged by counting the number of white pixels in the waterlogged area map:

[0070]

[0071] in is the total number of white pixels, is the total number of image pixels, is the actual area of ​​the image region, when Greater than That is, it is identified as a waterlogged area and marked;

[0072] (3-10) Publish waterlogging area maps (including waterlogging location, area, and other information) to provide technical support and basic data for urban waterlogging disaster reduction, prevention, emergency rescue, and disaster assessment.

[0073] Another object of the present invention is to provide a low-altitude UAV waterlogging inspection system based on deep learning, which specifically includes:

[0074] The data acquisition module acquires remote sensing visible light images with high temporal and spatial resolution through low-altitude drones;

[0075] The waterlogging identification and analysis module performs real-time waterlogging detection, segmentation, and positioning based on a deep learning model, enabling efficient, accurate, and rapid inspection of urban waterlogging information.

[0076] The information release module combines the drone trajectory information and image edge features to splice the inspection area map to segment the waterlogged area and draw and update the spatiotemporal distribution pattern of urban waterlogging.

[0077] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0078] First, the technical solution of the present invention solves the problems of low efficiency, insufficient accuracy, and untimely emergency response in urban waterlogging monitoring in the existing technology through a low-altitude drone waterlogging inspection method based on deep learning. Traditional waterlogging monitoring methods rely on ground measurements, which are slow and have limited coverage, making it difficult to obtain comprehensive waterlogging data in a timely manner, especially in emergency response after a disaster. This limitation is even more obvious.

[0079] The present invention realizes large-scale, fast and efficient urban waterlogging monitoring by using low-altitude drones combined with 5G communication technology. The image processing model based on deep learning can automatically identify and segment waterlogging / ponding areas, generate waterlogging distribution maps, and update waterlogging dynamic information in real time. The system can automatically collect and transmit high-resolution images taken by drones, use deep learning algorithms for intelligent analysis, and combine the spatiotemporal trajectory information of drones and the edge features of images to splice inspection area maps, segment waterlogging areas, and draw accurate spatiotemporal distribution maps of waterlogging. This automated monitoring and analysis method significantly improves the accuracy and timeliness of waterlogging detection.

[0080] Through the present invention, the management department can more quickly grasp the urban waterlogging situation, so as to timely dispatch drainage measures, carry out disaster prevention and emergency treatment, and significantly reduce the losses caused by waterlogging disasters. Compared with the existing technology, the present invention has made significant technical progress in efficiency, accuracy, real-time performance and coverage, and has important industrial application value.

[0081] Second, the expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0082] The present invention can provide high temporal and spatial resolution waterlogging observation data for meteorological, hydrological and other related scientific research and business needs, which is of great significance for disaster warning, rescue and emergency relief, flood modeling, etc. The equipment required by the patented technical solution is easy to operate and economical, the process is simple and clear, it has strong portability and replicability, and has the potential for large-scale promotion and use.

[0083] The technical solution of the present invention fills the technical gap in the industry at home and abroad: At present, domestic and foreign waterlogging monitoring mainly adopts three methods: manual inspection, sensor monitoring, and satellite remote sensing. There is still little development and application of low-altitude inspection and waterlogging detection by drones. The patent of the present invention first provides a drone inspection method including inspection routes and shooting methods to obtain complete, high-temporal and spatial resolution images of waterlogged areas; secondly, it provides a low-altitude drone image waterlogging segmentation model based on deep learning with high segmentation accuracy, high efficiency, and strong generalization ability, which provides data support for the identification of waterlogging. Finally, a drone inspection image splicing method based on scale-invariant feature transformation, k-nearest neighbor matching, and random sampling consensus algorithm is provided to segment the waterlogged area and draw and update the spatiotemporal distribution pattern of urban waterlogging.

[0084] The technical solution of the present invention solves the technical problems that people have been eager to solve but have never succeeded in solving: the manual inspection method has low personnel safety, consumes a lot of human resources, has a narrow coverage area, and has a limited observation angle; the water accumulation instrument has limited spatial coverage, high cost, and difficult maintenance; satellite remote sensing waterlogging monitoring has low accuracy, low temporal resolution, and high cost; the patent of the present invention innovatively selects a highly maneuverable drone as the observation equipment. The drone is equipped with a high-resolution lens and uses a fast low-altitude inspection photography method to obtain timely and high-precision waterlogging area information. This method has low equipment cost and achieves both large-scale coverage of waterlogging monitoring and continuity of monitoring, solving the above problems and providing valuable data for the positioning and evolution of waterlogged areas.

[0085] The technical solution of the present invention overcomes technical prejudice: the present invention innovatively uses drones to collect waterlogged area information, realizes waterlogging identification and segmentation based on the waterlogging segmentation model, and broadens the application scenarios of drone low-altitude inspection. The drone low-altitude inspection method breaks the traditional waterlogging detection's reliance on large-area water monitoring instruments and satellite remote sensing. In addition, the present invention applies the waterlogging segmentation model to waterlogging images taken by low-altitude drones for the first time, breaking the technical prejudice that the waterlogging segmentation model can only be applied to waterlogging images with clear waterlogging features. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is a flow chart of a method for waterlogging inspection using a low-altitude UAV based on deep learning provided by an embodiment of the present invention;

[0087] Figure 2 This is a schematic diagram of obtaining orthophoto images by drone inspection provided by an embodiment of the present invention;

[0088] Figure 3 The inspection area provided by the embodiment of the present invention is Figure 1 ;

[0089] Figure 4The inspection area provided by the embodiment of the present invention is Figure 2 ;

[0090] Figure 5 This is a rainy day inspection map of a drone provided by an embodiment of the present invention;

[0091] Figure 6 is a model structure diagram provided by an embodiment of the present invention;

[0092] Figure 7 is a deep learning model training graph provided by an embodiment of the present invention;

[0093] Figure 8 It is a module diagram of a low-altitude UAV waterlogging inspection system based on deep learning provided by an embodiment of the present invention;

[0094] Fig. 9 The waterlogged area provided by the embodiment of the present invention Figure 1 ;

[0095] Fig.10 The waterlogged area provided by the embodiment of the present invention Figure 2 . DETAILED DESCRIPTION

[0096] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0097] like Figure 1 As shown, an embodiment of the present invention provides a low-altitude UAV waterlogging inspection method based on deep learning, and the method specifically includes:

[0098] S1: Obtain remote sensing visible light images with high temporal and spatial resolution through low-altitude drones, and upload drone data to the cloud using 5G communication technology;

[0099] S2: A deep learning automatic waterlogging recognition and analysis model based on drone waterlogging image segmentation can be used to perform real-time waterlogging recognition and segmentation, and efficiently, accurately and quickly implement urban waterlogging information inspection;

[0100] S3: Combine the drone trajectory information and image edge features to stitch the inspection area map to segment the waterlogged area and draw and update the spatiotemporal distribution pattern of urban waterlogging.

[0101] 1. UAV data acquisition and transmission

[0102] One of the core of this method is to obtain remote sensing visible light images with high temporal and spatial resolution based on low-altitude drones. Low-altitude drones can fly flexibly at a relatively low altitude and capture high-precision image data of urban waterlogged areas. By combining with 5G communication technology, the image data collected by drones during inspections can be uploaded to the cloud in real time. This high-bandwidth, low-latency 5G communication technology ensures the real-time transmission of large-scale image data, providing timely and accurate basic data support for subsequent deep learning processing.

[0103] 2. Waterlogging Identification Using Deep Learning Models

[0104] Remote sensing images uploaded to the cloud are processed by deep learning models. The drone waterlogging image segmentation model used is a deep learning architecture designed specifically for identifying urban waterlogging. Through a multi-layer convolutional neural network, the model can detect, segment, and locate waterlogged areas in images in real time. The model can not only automatically detect waterlogged areas, but also accurately locate the location of each area and conduct in-depth analysis. This efficient and accurate waterlogging identification capability has greatly improved the automation of inspections, reduced manual participation, and increased data processing speed.

[0105] 3. Mapping the spatiotemporal distribution of waterlogging information

[0106] While analyzing the waterlogging information in the image, the system combines the trajectory information of the drone to draw and update the spatiotemporal distribution pattern of urban waterlogging in real time. The GPS system carried by the drone can record the flight trajectory in real time, and combine this location information with the waterlogging detection results to generate a detailed waterlogging map. Through the constantly updated spatiotemporal distribution pattern of waterlogging, the dynamic changes of waterlogging can be intuitively displayed, helping relevant departments to carry out waterlogging control and emergency response in a timely manner.

[0107] 4. Comprehensive processing capabilities of cloud systems

[0108] The entire system relies on the powerful computing power of the cloud processing platform, which can not only accommodate large-scale image data transmission, but also use cloud computing resources to process, store and analyze these data in real time. The generation of deep learning models and urban waterlogging distribution maps are completed in the cloud, ensuring the lightweight and efficient inspection capabilities of drones. Through the comprehensive processing capabilities of the cloud, city managers can retrieve the latest waterlogging information at any time to conduct disaster assessment, risk management and disaster relief decisions.

[0109] Said S1 comprises:

[0110] (1-1) Determine the latitude and longitude of the drone’s starting point ,The size of the area of ​​interest in vertical photography by drone Square meters, low altitude inspection height meters, S-shaped inspection route, every Meters to set the drone photography point;

[0111] (1-2) After checking that the drone equipment is normal, take off the drone vertically from the starting point to a low altitude inspection height rice;

[0112] (1-3) After the drone confirms that it has arrived at the photography point, it adjusts its posture to be stable and points the lens vertically downward to shoot 4K images, and then uploads the images and the drone's current coordinates via 5G communication technology. to the cloud;

[0113] (1-4) The drone accelerates along the inspection route until it reaches the next photography point and executes step (1-3);

[0114] (1-5) When the drone reaches the destination, it adjusts its direction, returns along the path and executes (1-2) to (1-4).

[0115] Inspection area such as Figure 3 , Figure 4 The drone inspection on a rainy day is shown in the figure below. Figure 5 shown.

[0116] like Figure 6 As shown, the S2 comprises:

[0117] (2-1) Separate the illumination component and reflectance component of waterlogging / flooding based on the Retinex theory:

[0118]

[0119] in, For drone images of waterlogging, is the reflectivity component, is the illumination component, The Retinex theory is used to model the complex changes of waterlogging / inland flooding images. Volume module separated from drone image The illumination component and reflectance component of the first image are obtained by , the illumination component of the first image , the reflectance component of the second image , the illumination component of the second image :

[0120]

[0121] (2-2) The light component and The reflectivity components are blended to reconstruct a new image , The light component and The reflectivity components are blended to reconstruct a new image , forming respectively , Images of the same scene with different lighting conditions are used to analyze the reflectivity of objects in the same scene with different lighting conditions.

[0122]

[0123] (2-3) Self-attention module Further extracting the image , Object edges, brightness, texture and other features are input into the parameter-sharing convolution module. Separate the image reflectance component , , and the lighting component , ;

[0124]

[0125] (2-4) The image component separation loss is defined as the pixel loss of the reflectance components extracted from the original image and the reconstructed image to constrain the model to learn the consistency of reflectance under different brightness conditions:

[0126]

[0127] (2-5) For each group of convolution modules Separating the results, we apply a retinex loss to ensure that the organization of reflectance and illumination is consistent with Retinex theory:

[0128]

[0129] (2-6) , Take the following as an example to illustrate the model structure. , Input segmentation parser, through module Extract separately , The high-level features are then input into the convolutional layer. Compute the attention mask to quantitatively evaluate the pixel richness of semantic information:

[0130]

[0131] in, , with different subscripts Represents the learnable weights of the convolution. represents the convolution operation, express Function, maps the pixel values ​​of the convolved image to (0,1) for normalization. is an attention mask that indicates the pixel-wise richness of semantic information in the lighting.

[0132] (2-7) Fuse with illumination and reflectance images and input into convolution module , and obtain the probability value of each pixel belonging to the waterlogging category. When the probability is greater than the set threshold of 0.5, it is judged as the waterlogged part to achieve accurate segmentation of the waterlogged area and obtain the final waterlogging segmentation image In the image, the white area is the waterlogged area and the black area is the background.

[0133]

[0134] (2-8) Add loss function Constraint Model:

[0135]

[0136] in represents the cross entropy loss function, Human-annotated images representing waterlogging images.

[0137] Model training Figure 7 shown.

[0138] The S3 includes:

[0139] (3-1) Map the pixel coordinate system to the camera coordinate system based on the intrinsic and extrinsic parameters transmitted by the camera, so that the image is mapped to the actual image. First, the center of the image in the pixel coordinate system is Multiply by the length of the pixel ,Width , mapped to the origin of the image coordinate system ,as follows:

[0140]

[0141] Then the other coordinates in the pixel coordinate system according to Scaling and translation, mapping to image coordinate system ,as follows:

[0142]

[0143] Finally, by scaling, Mapping to the camera coordinate system ,as follows:

[0144]

[0145] in, Indicates the distance between the camera lens and the ground. is the camera focal length;

[0146] (3-2) Assigning the coordinate information of the drone when taking the photo to the image as the coarse-grained position information of the image;

[0147] (3-3) Select the images to be stitched based on the drone inspection time-series trajectory information The scale-invariant feature transformation algorithm is used to extract image feature points.

[0148]

[0149] in is a scale-invariant feature transformation algorithm. is a set of feature points in the two matrix regions, and each feature point is represented by a high-dimensional vector.

[0150] (3-4) Retrieve the best matching feature points using the k-nearest neighbor algorithm.

[0151] for The feature points in ,exist Find its nearest neighbor and next nearest neighbor And calculate the Euclidean distance:

[0152]

[0153] in Represents the Euclidean distance between two feature points. Represents the dimension of feature points, Represents the highest dimension of feature points.

[0154] The ratio of the nearest neighbor distance to the next nearest neighbor distance is used to determine whether the match of the nearest neighbor feature point is valid. We set the ratio threshold to 0.2. If it is less than 0.2, the nearest neighbor feature point of the match is considered valid. Apply the above operation to all feature points to obtain the feature point matching set , including feature points and matching points corresponding to the feature points.

[0155]

[0156] (3-5) In The random sampling consensus algorithm is used to calculate the affine matrix and transform the image After stitching the two images together, we get .

[0157]

[0158] in represents the random sampling consensus algorithm, Indicates image stitching. For the overlapping pixels of two images, the new pixel points are calculated using the weighted average method of pixel values.

[0159] (3-6) Repeat steps (3-3) to (3-5) to stitch all inspection area sub-maps to obtain a complete inspection area map.

[0160] (3-7) Crop the complete inspection area map into several images of regions of interest;

[0161] (3-8) Feed the cropped image into the network to obtain several waterlogging segmentation images, which are then stitched together into a complete waterlogging area map within the inspection area;

[0162] (3-9) Calculate the area of ​​the waterlogged area and determine whether it is waterlogged by counting the number of white pixels in the waterlogged area map:

[0163]

[0164] in is the total number of white pixels, is the total number of image pixels, is the actual area of ​​the image region, when Greater than That is, it is identified as a waterlogged area and marked;

[0165] (3-10) Publish waterlogging area maps (including waterlogging location, area, and other information) to provide technical support and basic data for urban waterlogging disaster reduction, prevention, emergency rescue, and disaster assessment.

[0166] like Fig. 9 As shown, an embodiment of the present invention provides a low-altitude UAV waterlogging inspection system based on deep learning, which specifically includes:

[0167] The data acquisition module acquires remote sensing visible light images with high temporal and spatial resolution through low-altitude drones;

[0168] The waterlogging identification and analysis module performs real-time waterlogging detection, segmentation, and positioning based on a deep learning model, enabling efficient, accurate, and rapid inspection of urban waterlogging information.

[0169] The information release module, combined with the drone trajectory information, draws and updates the spatiotemporal distribution pattern of urban waterlogging.

[0170] 1. Application in urban flood control management system:

[0171] In urban flood control management, the low-altitude drone waterlogging inspection system based on deep learning can monitor urban waterlogging in real time. The drone flies regularly in areas prone to waterlogging through automatic inspection functions and uploads images to the cloud in real time. Combined with the deep learning model to process and analyze the images, the system can quickly identify the specific location and area of ​​waterlogging, generate a waterlogging distribution map, and publish the results to the city's flood control department. This provides timely disaster information to the flood control command center, allowing flood control personnel to quickly take measures to reduce waterlogging risks and losses and improve emergency response efficiency.

[0172] 2. Application in traffic monitoring and emergency dispatch:

[0173] In urban traffic management, waterlogging can cause traffic jams and trapped vehicles. The system can be deployed in traffic monitoring systems, using drones to inspect waterlogging on roads and important transportation hubs. The system transmits road waterlogging data to the traffic control center in real time through 5G technology. Combined with automatic recognition and analysis models, it can accurately locate the area and extent of waterlogging and predict its impact on traffic. Traffic management departments can use the system to dispatch road diversion, vehicle diversion and drainage rescue in advance to ensure smooth traffic and reduce traffic interruptions and congestion caused by waterlogging.

[0174] Deep learning training diagram of waterlogging segmentation model Figure 7 As shown in the figure. The model has excellent segmentation effect on waterlogged areas, with an IoU of 84.45%, an F1 score of 91.57%, a precision of 93.09%, and a recall of 90.11%. The high IoU score indicates that the predicted area is highly consistent with the true area, and the high F1 score indicates that the model has achieved a good balance between precision and recall, and the classifier is of high quality. The high accuracy indicates that the prediction results of the model are very reliable overall.

[0175] Inspection area diagram Figure 3 , Figure 4 And the corresponding waterlogged area map is as follows Fig. 9 , Fig.10 Comparing the inspection area map and the waterlogged area map, the waterlogged area is basically completely and finely segmented by the model, especially the waterlogged area in the shadow of the object is also accurately identified. The model effectively reduces the impact of light on waterlogging identification and segmentation.

[0176] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0177] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A low-altitude UAV waterlogging inspection method based on deep learning, characterized in that: The method specifically includes: S1: Obtain remote sensing visible light images with high temporal and spatial resolution through low-altitude drones, and upload drone data to the cloud using 5G communication technology; S2: A deep learning automatic waterlogging recognition and analysis model based on UAV waterlogging image segmentation for real-time waterlogging recognition and segmentation; S3: Combine the drone trajectory information and image edge features to stitch the inspection area map to segment the waterlogged area and draw and update the spatiotemporal distribution pattern of urban waterlogging; The S2 comprises: (2-1) Separate the illumination component and reflectance component of waterlogging / flooding based on the Retinex theory: ； in, For drone images of waterlogging, is the reflectivity component, is the illumination component, The Retinex theory is used to model the complex changes of waterlogging / inland flooding images. Volume module separated from drone image The illumination component and reflectance component of the first image are obtained by , the illumination component of the first image , the reflectance component of the second image , the illumination component of the second image : ； (2-2) The light component and The reflectivity components are blended to reconstruct a new image , The light component and The reflectivity components are blended to reconstruct a new image , forming respectively , Images of the same scene under different lighting conditions are used to analyze the reflectivity of objects under different lighting conditions in the same scene; ； (2-3) Self-attention module Further extracting the image , Object edge, brightness, and texture features are input into the parameter-sharing convolution module Separate the image reflectance component , , and the lighting component , ; ； (2-4) The image component separation loss is defined as the pixel loss of the reflectance components extracted from the original image and the reconstructed image to constrain the model to learn the consistency of reflectance under different brightness conditions: ； (2-5) For each group of convolution modules Separating the results, we apply a retinex loss to ensure that the organization of reflectance and illumination is consistent with Retinex theory: ； (2-6) , Take the following as an example to illustrate the model structure. , Input segmentation parser, through module Extract separately , The high-level features are then input into the convolutional layer. Compute the attention mask to quantitatively evaluate the pixel richness of semantic information: ； in, , with different subscripts represents the learnable weights of the convolution; represents the convolution operation, express Function, maps the pixel values ​​of the convolved image to (0,1) for normalization, is the attention mask, which indicates the pixel-wise richness of semantic information in the illumination; (2-7) Fuse with illumination and reflectance images and input into convolution module , and obtain the probability value of each pixel belonging to the waterlogging category. When the probability is greater than the set threshold of 0.5, it is judged as the waterlogged part to achieve accurate segmentation of the waterlogged area and obtain the final waterlogging segmentation image ; The white in the image is the waterlogged area and the black is the background; ； (2-8) Add loss function Constraint Model: ； in represents the cross entropy loss function, Human-annotated images representing waterlogging images; The S3 includes: (3-1) Map the pixel coordinate system to the camera coordinate system according to the intrinsic and extrinsic parameters transmitted by the camera, so that the image is mapped to the actual phase; first, the center of the image in the pixel coordinate system is Multiply by the length of the pixel ,Width , mapped to the origin of the image coordinate system ,as follows: ； Then the other coordinates in the pixel coordinate system according to Scaling and translation, mapping to image coordinate system ,as follows: ； Finally, by scaling, Mapping to the camera coordinate system ,as follows: ； in, Indicates the distance between the camera lens and the ground. is the camera focal length; (3-2) Assigning the coordinate information of the drone when taking the photo to the image as the coarse-grained position information of the image; (3-3) Select the images to be stitched based on the drone inspection time-series trajectory information Apply scale-invariant feature transformation algorithm to extract image feature points; ； in is a scale-invariant feature transformation algorithm. is a set of feature points in the two matrix regions, each of which is represented by a high-dimensional vector; (3-4) Retrieve the best matching feature points through the k-nearest neighbor algorithm; for The feature points in ,exist Find its nearest neighbor and next nearest neighbor And calculate the Euclidean distance: ； in Represents the Euclidean distance between two feature points. Represents the dimension of feature points, Represents the highest dimension of feature points; The ratio of the nearest neighbor distance to the next nearest neighbor distance is used to determine whether the match of the nearest neighbor feature point is valid. We set the ratio threshold to 0.

2. If it is less than 0.2, the nearest neighbor feature point of the match is considered valid; apply the above operation to all feature points to obtain the feature point matching set , including feature points and matching points corresponding to the feature points; ； (3-5) In The random sampling consensus algorithm is used to calculate the affine matrix and transform the image After stitching the two images together, we get ; ； in represents the random sampling consensus algorithm, Indicates image stitching. For the overlapping pixels of two images, the new pixel is calculated by weighted average of pixel values. (3-6) Repeat steps (3-3) to (3-5) to stitch all inspection area sub-maps to obtain a complete inspection area map; (3-7) Crop the complete inspection area map into several images of regions of interest; (3-8) Feed the cropped image into the network to obtain several waterlogging segmentation images, which are then stitched together into a complete waterlogging area map within the inspection area; (3-9) Calculate the area of ​​the waterlogged area and determine whether it is waterlogged by counting the number of white pixels in the waterlogged area map: ； in is the total number of white pixels, is the total number of image pixels, is the actual area of ​​the image region, when Greater than That is, it is identified as a waterlogged area and marked; (3-10) Publish maps of urban waterlogging areas to provide technical support and basic data for urban waterlogging disaster reduction, prevention, emergency rescue, and disaster assessment.

2. The method for waterlogging inspection using a low-altitude drone based on deep learning as claimed in claim 1, characterized in that: Said S1 comprises: (1-1) Determine the latitude and longitude of the drone’s starting point ,The size of the area of ​​interest in vertical photography by drone Square meters, low altitude inspection height meters, S-shaped inspection route, every Meters to set the drone photography point; (1-2) After checking that the drone equipment is normal, take off the drone vertically from the starting point to a low altitude inspection height rice; (1-3) After the drone confirms that it has arrived at the photography point, it adjusts its posture to be stable and points the lens vertically downward to shoot 4K images, and then uploads the images and the drone's current coordinates via 5G communication technology. to the cloud; (1-4) The drone accelerates along the inspection route until it reaches the next photography point and executes step (1-3); (1-5) When the drone reaches the destination, it adjusts its direction, returns along the path and executes (1-2) to (1-4).

3. A low-altitude UAV waterlogging inspection system based on deep learning, used to implement the low-altitude UAV waterlogging inspection method based on deep learning as described in any one of claims 1 to 2, characterized in that: The system specifically includes: Low-altitude drone module, used to obtain remote sensing visible light images with high temporal and spatial resolution, and upload the data to the cloud via 5G communication technology; The image processing module is used to process the image data transmitted by the drone and detect, segment and locate the waterlogging in the image based on the deep learning model; Data transmission module, used to transmit the image data collected by the drone to the cloud data processing center in real time through 5G communication technology; The waterlogging analysis module is used to generate a spatiotemporal distribution pattern map of urban waterlogging by analyzing drone trajectory information and image data; The display module is used to display the waterlogging area map, including the waterlogging location, area and other relevant information, to provide a basis for disaster warning and emergency response.

4. The deep learning-based low-altitude UAV waterlogging inspection system according to claim 3 is characterized in that: The low-altitude UAV module includes: The UAV equipment control module is used to determine the starting latitude and longitude of the UAV, the size of the vertical photography area of ​​interest, the low-altitude inspection height and the S-shaped inspection route; The image acquisition module is used to shoot 4K images when the drone arrives at each photography point and upload them to the cloud through 5G communication technology.

5. The deep learning-based low-altitude UAV waterlogging inspection system according to claim 3 is characterized in that: The image processing module is based on the Retinex theory and uses a convolutional neural network to separate the illumination component and the reflectivity component of the image collected by the drone, and to perform real-time segmentation and prediction of the waterlogged area.

6. The deep learning-based low-altitude UAV waterlogging inspection system according to claim 3 is characterized in that: The waterlogging analysis module assigns image location information by combining internal and external parameters, splices the UAV's time-series trajectory and image features, segments the waterlogged area through a deep learning model, and finally generates a spatiotemporal distribution pattern map of the waterlogged area.

Images