Optical flow velocity field measurement method and system based on UAV thermal infrared image sequence

Abstract

The present invention relates to an optical flow velocity field measurement method and system based on a thermal infrared image sequence of an unmanned aerial vehicle, including: using geometric calibration to solve the internal orientation elements of a non-cooled thermal infrared camera; performing contrast enhancement on the collected thermal infrared image data set; calculating the trajectory of the thermal infrared image sequence by laying out ground control points; using the temperature change of the water body as a heat source tracer to track the image sequence features; filtering out erroneous image tracking results through image filtering and performing actual velocity calculation. The present invention realizes high-precision velocity field measurement by tracking the texture formed by the mixture of cold and hot water, and the method can be used for small streams as well as large rivers and lakes.

Description

Technical Field

[0001] The invention relates to an optical flow velocity field measurement method and system based on a thermal infrared image sequence of an unmanned aerial vehicle, and relates to the technical field of thermal infrared image velocity measurement. Background Art

[0002] Fluid velocity field measurement is crucial for studying hydrological issues such as fluid flow or pollutant propagation time. Traditional river velocity measurement methods (such as propeller velocimeters and acoustic equipment) can accurately measure fluid velocity information, but each deployment can only obtain single-point velocity information, which is time-consuming and labor-intensive, and it is difficult to quickly build a large-scale velocity field measurement. In addition, the velocity measuring instrument must be placed underwater, which may hinder or change the original velocity and direction of the river. Therefore, these traditional devices have the disadvantages of low measurement efficiency, long time consumption, high labor costs, and can only be measured in a safe environment.

[0003] The solution to replace the traditional velocity measuring instrument in the prior art is the method based on image feature tracking, because the method based on image feature matching does not need to directly contact the river and can provide continuous velocity measurement information. In addition, compared with the single-point measurement results, the measurement method based on image matching can obtain a dense velocity field. Therefore, the velocity measurement method based on image feature tracking is widely used in ground close-up velocity measurement and drone remote sensing velocity measurement. The ground close-up velocity measurement method requires the camera to be set up on the river bank, so it can only measure the velocity of a small stream or a local area in a large river, and cannot take into account the measurement of a large river. Drone remote sensing velocity measurement does not require the installation of a camera on the ground. The drone's airborne characteristics can achieve flexible large-scale observations and are not affected by extreme conditions such as floods. However, it is difficult for the drone image sequence to remain unchanged in a fixed aerial position. In order to measure the flow velocity of the fluid, the motion trajectory of the drone image sequence must first be determined, and then the movement velocity of the fluid can be calculated.

[0004] Fluid velocity measurement methods based on image feature tracking can be roughly divided into two categories: particle image velocimetry (PIV) and particle tracking velocimetry (PTV). PIV is an Eulerian velocimetry method that takes each image sub-block as the basic unit and measures the velocity of the fluid in the field of view by matching the texture features of the image sub-blocks; PTV belongs to the Lagrangian velocimetry method that measures the fluid velocity by tracking the changes in the trajectory of the detection particles within the field of view. Therefore, the PIV method can provide grid tracking results based on image sub-blocks, while the PTV method can obtain the motion trajectories of all detection particles. However, whether it is PIV or PTV, methods based on image feature tracking require tracers to achieve flow velocity measurement. Tracers are usually achieved by artificially placing floating objects. The most widely used tracers in existing research are dyes of different colors and floating particles. The disadvantage of dye tracers is that they have a potential pollution effect on the surrounding environment. The main problem with floating particles is that they are affected by viscosity and friction. Floating particles are easy to clump, which not only affects the natural flow of the river, but also the shape change of floating particles will cause the image matching accuracy to decrease. In addition, traditional image feature tracking methods are mainly designed for optical images, which are easily affected by uneven lighting or changes in lighting conditions, resulting in reduced speed measurement accuracy and inability to achieve nighttime observation. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a method, system, device and storage medium for measuring dense flow field of unmanned aerial vehicle thermal infrared image sequence, which can achieve high-precision flow field measurement by tracking cold and hot water mixed texture.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides an optical flow velocity field measurement method based on a UAV thermal infrared image sequence, which comprises:

[0008] The internal orientation elements of uncooled thermal infrared cameras are solved by geometric calibration;

[0009] Perform contrast enhancement on the collected UAV thermal infrared image dataset;

[0010] By laying out ground control points, the trajectory of the UAV thermal infrared image sequence is calculated;

[0011] Using the temperature change of water bodies as a heat source tracer to track the characteristics of image sequences;

[0012] Image filtering is used to filter out erroneous image tracking results and calculate the actual flow velocity.

[0013] Furthermore, the gradient histogram equalization method is used to enhance the contrast of the collected UAV thermal infrared image dataset.

[0014] Furthermore, by laying out ground control points, the trajectory of the UAV thermal infrared image sequence is calculated, including:

[0015] Deployment of ground control points;

[0016] Segment the acquired image sequence, extract the image objects using multi-scale segmentation method, and realize object-level extraction of thermal infrared images;

[0017] Based on the image segmentation results, the image control point image coordinates are extracted to achieve automatic registration of the UAV thermal infrared image sequence;

[0018] Based on the camera's internal orientation elements and the image coordinates of the control points, spatial resection is used to calculate the exterior orientation elements of each frame of image to determine the position and posture of the UAV thermal infrared image sequence in three-dimensional space.

[0019] Furthermore, based on the image segmentation results, the image control point image coordinates are extracted to achieve automatic registration of the UAV thermal infrared image sequence, including:

[0020] Through the object-oriented control point feature extraction method, the position of the ground control point is automatically detected on each frame of the UAV thermal infrared image, and the position and attitude changes of the camera are determined by comparing the position changes of the control points on the UAV thermal infrared image sequence;

[0021] Based on the image segmentation results, the edge index EI feature is used to extract the control point area from the candidate image objects:

[0022]

[0023] Among them, b m represents the boundary perimeter of object m, l m The minimum circumscribed square side length of object m, the lower the EI value, the more likely the object is a control point area. The automatic extraction of the control point area can be achieved by simply selecting the set object with the smallest EI feature value from all candidate objects;

[0024] The Shi-Tomasi algorithm is used to search for the strongest corner feature pixel by pixel in the control point area as the initial position, and the position of the control point is optimized using an iterative search method based on gradient descent. The image coordinates of the ground control point are extracted frame by frame from the UAV thermal infrared image sequence.

[0025] Using matching point pairs and the least squares method, the affine transformation between adjacent frames is calculated to determine the pixel shift caused by the displacement and attitude change of the drone:

[0026]

[0027] Among them, (x r ,y r ) represents the position of the ground control point on the first frame of the reference image in the thermal infrared image sequence, (x s ,y s ) represents the position of the ground control point on the subsequent images in the thermal infrared image sequence; (h xx ,h xy ,h xy ,h yy ,h x ,h y ) describes the affine changes of subsequent images relative to the reference image, and describes the position and attitude changes of the UAV from the perspective of two-dimensional images.

[0028] Furthermore, the image sequence feature tracking is determined, including:

[0029] Extract local hot spots in the cold and hot water mixed texture as feature points to be tracked, and perform feature tracking based on the image texture obtained by mixing cold and hot water;

[0030] Construct an image pyramid to achieve feature tracking based on local hot spots: First, use the LK algorithm to perform sub-pixel feature tracking on the image with the lowest resolution, then transfer the first-level image tracking result to the next-level higher-resolution image as the tracking initial value, and repeat the process until the original image with the highest resolution is obtained;

[0031] Perform LK algorithm feature tracking on the image to complete image matching.

[0032] Furthermore, the image filtering includes:

[0033] Based on LK forward and backward estimation offset filtering, obviously wrong tracking results are removed. Specifically, the LK algorithm is used to forward track the image feature point a at time t to obtain the position of the image feature point b at time t+1. Similarly, according to the position of the feature point b in the image at time t+1, the position of the feature point c on the image at time t is obtained. The offset between the two is calculated as the LK forward and backward estimation offset d. offset , when the estimated forward and backward offset of a feature point is d offset If it is greater than the set threshold, the feature tracking result is filtered out;

[0034] Based on the spatiotemporal filtering of flow velocity and direction, outliers in image matching are eliminated.

[0035] Furthermore, the actual flow velocity solution includes:

[0036] Fit the water surface position to a plane;

[0037] By using the collinearity condition equation and the exterior orientation elements of the UAV thermal infrared image sequence, the three-dimensional ground point coordinates of the first and last points of the image feature tracking results in the real scene are calculated by solving the intersection of single image positioning and the horizontal plane. The feature tracking results on the UAV thermal infrared image sequence are converted into ground speed measurements of the real scene.

[0038] In a second aspect, the optical flow velocity field measurement system based on the thermal infrared image sequence of an unmanned aerial vehicle provided by the present invention is characterized in that the system comprises:

[0039] A geometric calibration unit is configured to solve the internal orientation elements of the uncooled thermal infrared camera using geometric calibration;

[0040] An image enhancement unit configured to perform contrast enhancement on a collected thermal infrared image data set of a UAV;

[0041] A trajectory calculation unit is configured to calculate the trajectory of the thermal infrared image sequence of the UAV by laying out ground control points;

[0042] A feature tracking unit configured to determine a feature tracking of an image sequence;

[0043] The filtering and flow rate calculation unit is configured to filter out erroneous image tracking results through image filtering and perform actual flow rate calculation.

[0044] In a third aspect, the present invention provides an electronic device, comprising at least a processor and a memory, wherein a computer program is stored in the memory, and when the processor runs the computer program, the method described above is executed.

[0045] In a fourth aspect, the present invention provides a computer storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions can be executed by a processor to implement the described method.

[0046] The present invention adopts the above technical solution, which has the following advantages:

[0047] 1. Considering that thermal infrared images are not affected by changes in lighting conditions, the present invention uses the temperature change of the water body as a heat source tracer and realizes high-precision flow field measurement by tracking the texture formed by the mixture of cold and hot water. Therefore, the present invention is not limited to daytime, and can also be measured at night;

[0048] 2. The present invention uses a geometric calibration plate to solve the internal orientation elements of the uncooled thermal infrared camera and uses gradient histogram equalization to enhance the contrast of the thermal infrared image. Therefore, the UAV fluid velocity measurement image sequence has accurate internal orientation elements and high contrast, which lays a solid foundation for realizing automatic fluid velocity measurement based on high-precision feature tracking;

[0049] 3. The present invention uses a small UAV as a carrier, firstly realizes the automatic registration of the UAV thermal infrared image sequence by arranging ground control points and automatically identifying image control points; then uses the spatial resection method to realize the resolution of the exterior orientation elements of each frame of the image, and then obtains the motion trajectory of the thermal infrared image sequence, eliminating the influence of the UAV movement on the fluid velocity measurement accuracy;

[0050] 4. The present invention uses the local hot spots in the cold and hot water mixing area as tracking points, and uses the pyramid-based optical flow Lucas-Kanade (LK) algorithm to effectively compensate for the distortion and rotation transformation of the cold and hot water mixing texture during the flow process, and obtains feature tracking results with sub-pixel accuracy;

[0051] 5. The present invention firstly introduces forward and backward estimation offsets to roughly remove obviously erroneous LK matching results, and then uses spatiotemporal filtering based on flow velocity and flow direction to further eliminate outliers in image matching, thereby obtaining a high-precision dense flow velocity field. Finally, single-image positioning is used to convert feature tracking results on the image into actual flow velocity values ​​in real three-dimensional space.

[0052] In summary, the present invention realizes high-precision velocity field measurement by tracking the texture formed by the mixture of cold and hot water. This method can be used for small streams as well as large rivers and lakes. Specifically, in the field of velocity measurement in small streams, a traceable texture can be generated by adding additional hot water; in the field of velocity measurement in large rivers and lakes, the difference between the surface temperature of the water body and the temperature of the deep water body at night can be used as a signal source. This signal source is mainly caused by the large difference between the air temperature and the water surface temperature at night. The traceable texture is formed through the heat transfer of the mixture of cold and hot water, which can be widely used in fluid velocity field measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Throughout the accompanying drawings, the same reference numerals are used to represent the same components. In the accompanying drawings:

[0054] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0055] Figure 2 The geometric calibration plate of the embodiment of the present invention, wherein (a) is a visible light image of the geometric calibration plate, and (b) is a thermal infrared image of the geometric calibration plate;

[0056] Figure 3 A schematic diagram of the LK algorithm backward estimation offset according to an embodiment of the present invention;

[0057] Figure 4FIG. 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0059] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0060] Although the terms first, second, third, etc. can be used in the text to describe multiple elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms can only be used to distinguish an element, component, region, layer or section from another region, layer or section. Unless the context clearly indicates, terms such as "first", "second" and other numerical terms do not imply order or sequence when used in the text. Therefore, the first element, component, region, layer or section discussed below can be referred to as the second element, component, region, layer or section without departing from the teaching of the example embodiments.

[0061] For ease of description, spatially relative terms may be used herein to describe the relationship of one element or feature relative to another element or feature as shown in the figures, such as "inside", "outside", "inner side", "outer side", "below", "above", etc. Such spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation depicted in the figures.

[0062] The optical flow velocity field measurement method, system, equipment and medium based on the thermal infrared image sequence of unmanned aerial vehicles provided by the present invention include: using geometric calibration to solve the internal orientation elements of the uncooled thermal infrared camera; contrast enhancement of the collected thermal infrared image data set of the unmanned aerial vehicle; calculating the trajectory of the thermal infrared image sequence of the unmanned aerial vehicle by laying out ground control points; using the temperature change of the water body as a heat source tracer to track the image sequence characteristics; filtering out erroneous image tracking results through image filtering and performing actual flow velocity solution. The present invention realizes high-precision flow field measurement by tracking the mixed texture of cold and hot water.

[0063] Embodiment 1: Figure 1 As shown, the optical flow velocity field measurement method based on the thermal infrared image sequence of the unmanned aerial vehicle provided in this embodiment includes the following contents:

[0064] S1. Preprocessing of uncooled thermal infrared sensors

[0065] Due to the lack of protection from cooling sources, the uncooled thermal infrared sensor is affected by complex lens distortion, which reduces the geometric positioning accuracy; on the other hand, it is affected by the vignetting effect, and the image contrast is low. Therefore, this embodiment first uses geometric calibration to solve the internal orientation elements of the thermal infrared sensor, so as to convert the two-dimensional image feature tracking results (pixels / second) into the flow velocity value (meters / second) in the real three-dimensional world; then, the gradient histogram equalization is used to eliminate the vignetting effect of the thermal infrared image of the drone, and improve the image contrast and feature tracking accuracy. The specific process is as follows:

[0066] S1.1 Geometric calibration

[0067] This embodiment allows the drone to hover above the water body to be measured, which belongs to the close-range photogrammetry environment. Before measuring the water flow rate, the present invention uses the difference in emissivity of different materials to construct a geometric calibration field for the thermal infrared camera, and uses the self-calibration bundle method adjustment to achieve geometric calibration of the uncooled thermal infrared camera.

[0068] Specifically, since thermal infrared images have low spatial resolution, fuzzy edge features, and low image point detection accuracy, while visible light cameras have high spatial resolution, strong contrast, and high image point detection accuracy, in order to improve the stability and accuracy of thermal infrared image geometric calibration, this embodiment proposes a joint geometric calibration method for thermal infrared cameras and visible light cameras. To this end, this embodiment designs a three-dimensional calibration field suitable for joint calibration of visible light cameras and thermal infrared cameras, such as Figure 2As shown. In order to obtain high-contrast visible light calibration images and thermal infrared calibration images at the same time, the three-dimensional calibration field is mainly composed of black velvet foil as the background, and the coded visible light image target and the uncoded thermal infrared image target are used as the calibration source. The coded visible light image target is composed of black and white square paper materials, and the uncoded thermal infrared image target is constructed by gray circular silver foil, as shown. Figure 2 (a) shown.

[0069] In order to ensure that the thermal infrared image has a good contrast, it is usually necessary to place the three-dimensional geometric calibration field under outdoor conditions. The black velvet foil appears bright white on the thermal infrared image due to its strong radiation absorption characteristics, and the round silver foil appears dark due to its mirror reflection characteristics, thus forming a thermal infrared calibration image with high contrast; at the same time, the background of the three-dimensional calibration board appears black, and the visible light image target is composed of coded black and white square cardboard, thereby forming a visible light calibration image with high contrast, such as Figure 2 (b) is shown. The coded target and the calibration strip of known length have clear contrast in the visible light image, and can achieve sub-pixel-level image point detection and millimeter-level precision three-dimensional ground point measurement through software such as Aicon DPA. By constructing the above-mentioned high-precision visible light image geometric model, high-precision three-dimensional measurement of thermal infrared image targets (circular silver foil) in the object coordinate system can be achieved. At the same time, in the thermal infrared image, the coordinates of the center image point of the circular silver foil are solved by ellipse fitting. In theory, the measurement standard deviation of the center image point coordinates can reach 1 / 40 of the pixel.

[0070] When the high-precision object image coordinates are known, this embodiment uses the image point coordinate measurement values, the ground point three-dimensional coordinate values ​​and the initial values ​​of the camera internal orientation elements (provided by the camera supplier) as the input values ​​of the bundle adjustment, and uses the Brownian model shown in formula (1) as the imaging equation to achieve high-precision solution of the thermal infrared camera internal orientation elements (principal distance, principal point, distortion parameters) through bundle adjustment.

[0071] The object-image equation is mainly described by the Brownian model. Assuming that the object point P(X, Y, Z) corresponds to an image point p(u, v) in the image, the object-image relationship is expressed as the collinearity condition equation shown in formula (1).

[0072]

[0073] Among them, (X S ,Y S ,Z S ) T represents the projection center of the image, r r,c The rotation matrix R of the image can be expressed by the Euler angle These parameters represent the exterior orientation elements of the image. r represents the radial distance of the image, c represents the principal distance, (x0, y0) T Indicates the principal point, (K1, K2, K3, P1, P2, B1, B2) T represents the distortion parameters of the camera, which are the internal orientation elements that need to be solved in geometric calibration. Specifically, the distortion caused by the lens is mainly composed of radial distortion and tangential distortion, which can be described and corrected by a polynomial model. From formula (1), it can be seen that the ground point P (X, Y, Z) in the object coordinate system can be transformed by the collinear condition equation, distortion correction, image principal point translation, focal length solution and coordinate axis skewness correction to obtain the image point coordinate p (u, v) of the target in the image plane coordinate system.

[0074] Based on the given initial values, the iterative optimization of the self-calibration bundle adjustment is used to solve the internal orientation elements of the thermal infrared camera. In the iterative adjustment process, the significance test is introduced, and the 3-sigma rule is used to delete unimportant camera parameters, thereby avoiding the problem of non-convergence or convergence to local minimum values ​​caused by correlation between parameters.

[0075] S1.2 Gradient histogram equalization to improve contrast

[0076] In the process of acquiring the UAV thermal infrared image dataset, the UAV is hovered just above the water surface, and the thermal infrared camera is mounted on the UAV and observes straight down to capture the UAV thermal infrared image sequence of the fluid.

[0077] Since image contrast enhancement can significantly improve the contrast of the tracer and the accuracy of feature matching, image enhancement is widely used as a preprocessing method to improve the matching accuracy of tracking features. This embodiment uses a gradient histogram equalization method to remove the vignetting effect caused by differences in sensor manufacturing processes and improve the contrast of the image. Specifically, the gradient histogram is similar to the commonly used grayscale histogram, except that the ordinate of the histogram is converted into the accumulation of the absolute values ​​of the gradients of all pixels at the same grayscale level. Since the gradient change rule can describe the texture of the image, compared with histogram equalization, gradient histogram equalization can retain the image texture details and reduce the local over-enhancement of the image. After gradient histogram equalization, the vignetting effect in the original thermal infrared image can be effectively suppressed, significantly improving the image contrast, which helps to improve the matching accuracy of the tracking feature.

[0078] S2. Calculate the trajectory of thermal infrared image sequence

[0079] Since it is difficult for a hovering drone to remain motionless in three-dimensional space, it is necessary to use image matching methods to align the thermal infrared image sequence to the same reference image to eliminate the influence of the drone's own drift on the accuracy of water velocity measurement.

[0080] This embodiment is implemented by deploying ground control points. First, four ground control points are deployed on the river bank next to the water body. The control area is a square cardboard with an iron nail in the center as the control point. The present invention automatically detects the position of the fixed control point on the thermal infrared image sequence, and can determine the movement trajectory of the thermal infrared image sequence, including:

[0081] S2.1 Image segmentation to achieve object-level extraction of thermal infrared images

[0082] This embodiment uses a multi-scale segmentation method to extract image objects. This method can maximize the grayscale homogeneity within the image object while retaining the grayscale heterogeneity between different objects, thereby achieving object-level extraction of thermal infrared images.

[0083] S2.2 Image control point detection

[0084] Ground Control Points (GCP) are mainly used to align thermal infrared image sequences, determine the motion trajectory of drone image sequences, and convert the flow velocity measurement results (pixels / second) on the image to the velocity measurement results (meters / second) in the three-dimensional real world. The traditional ground control point extraction method mainly adopts the method of manually selecting control points. If the traditional method of manually obtaining control points is adopted, it is necessary to manually mark hundreds or thousands of images, which is undoubtedly time-consuming and laborious. Therefore, this embodiment proposes an object-oriented control point feature extraction method, which can automatically detect the position of square ground control points (GCP) on each frame of drone image. Considering that the position of the ground control point remains unchanged, the position and posture changes of the camera can be determined by comparing the position changes of the control points on the image sequence.

[0085] Based on the image segmentation result, considering that the shape of the control point area is approximately square, this embodiment uses the edge index (EI) feature to extract the control point area from the candidate image object, as shown in formula (2).

[0086]

[0087] Among them, b m represents the boundary perimeter of object m, l m Represents the minimum side length of the circumscribed square of object m.

[0088] The edge index feature (EI) represents the ratio of the perimeter of the object boundary to the perimeter of its smallest circumscribed square. The EI feature describes the smoothness of the image object and the square similarity. Specifically, the more similar the image object is to the square, the lower its EI value is, and the more likely the object is a control point area. Therefore, it is only necessary to select the four objects with the smallest EI feature values ​​from all candidate objects to achieve automatic extraction of control point areas.

[0089] Ground control points (GCPs) appear as feature corner points on the image, and they are all located at the geometric center of the control point area. Therefore, this embodiment uses the Shi-Tomasi algorithm to search for the strongest corner point feature pixel by pixel in the control point area as the initial position. Then, the position of the control point is optimized using an iterative search method based on gradient descent, and the positioning accuracy of the control point is improved to the sub-pixel level. The above method is used to extract the image coordinates of the four ground control points frame by frame in the UAV thermal infrared image sequence.

[0090] Considering that the drone is hovering statically above the water body, there is no huge rotation and displacement. Therefore, the control points with close distances on adjacent frame images are the corresponding matching point pairs. Using the matching point pairs and the least squares method, the affine transformation between adjacent frame images can be calculated through formula (3), and the pixel displacement caused by the displacement and posture change of the drone can be determined, laying the foundation for solving the water flow velocity. In this embodiment, the first frame image is used as the reference image, and the subsequent images are registered to the reference image to eliminate the influence of the drone displacement.

[0091]

[0092] Among them, (x r ,y r ) represents the position of the ground control point GCP on the first frame of the reference image in the thermal infrared image sequence, (x s ,y s ) represents the position of the ground control point GCP on the subsequent images in the thermal infrared image sequence. (h xx ,h xy ,h yx ,h yy ,h x ,h y ) describes the affine changes of subsequent images relative to the reference image, and describes the position and attitude changes of the UAV from the perspective of two-dimensional images.

[0093] S2.3 Resection solves the exterior orientation elements to determine the position and posture of the thermal infrared image sequence in three-dimensional space

[0094] In order to determine the position and posture of the image sequence in the real three-dimensional world, this embodiment uses a spatial resection method to determine the exterior orientation elements of each frame of image.

[0095] Specifically, the three-dimensional coordinates of the ground control point GCP are firstly measured accurately using a total station. Then, under the condition that the internal orientation elements of the camera and the image coordinates of the control points are known, the exterior orientation elements of each frame of the image are calculated using spatial resection to determine the position and posture of the thermal infrared image sequence in three-dimensional space.

[0096] S3. Determine the image sequence feature tracking

[0097] Different from the traditional visible light image solid tracers (such as debris and wood chips) floating on the water surface, this implementation uses the image texture obtained by mixing cold and hot water for feature tracking. Specifically, the local hot spots (i.e. the grayscale maximum value in the local area) in the cold and hot water mixed texture are first used as tracking seed points; then an image pyramid is constructed to facilitate small displacement feature tracking on the sequence image; finally, the LK algorithm based on the image pyramid is used to achieve sub-pixel-level high-precision feature tracking of adjacent frame images, including:

[0098] S3.1 Obtain local hotspots and complete local hotspot detection

[0099] This embodiment uses local hot spots in the cold and hot water mixing area as tracking features because local hot spots have high gradients and rich image texture structures, which can minimize the mismatching phenomenon caused by the loss of image texture. In actual operation, the cold and hot water mixing area full of tracking features is first extracted by manually drawing a rectangular frame, and then, inside the cold and hot water mixing area, local hot spots are extracted as feature points to be tracked. Specifically, the local hot spot pixels need to meet the following conditions: the grayscale value of the pixel is the local maximum value of the neighborhood area of ​​10 rows and 10 columns. As time goes by, the image texture formed by the mixing of cold and hot water will dissipate quickly, and the reduced image contrast will cause the image matching accuracy to drop significantly. Therefore, it is necessary to add some new local hot spot features to detect every few frames of images. The present invention updates the hot spot features every 10 frames of images, taking this as an example, but not limited to this.

[0100] S3.2 builds an image pyramid to achieve high-precision feature tracking based on local hotspots.

[0101] Since the feature tracking method based on optical flow is more accurate for estimating small-scale displacements, it is difficult to handle long-distance and large-scale image feature displacements. In order to overcome this problem, this embodiment uses an image pyramid to achieve high-precision feature tracking based on local hotspots.

[0102] The image pyramid construction process mainly includes low-pass filtering and image downsampling. Low-pass filtering is mainly used to smooth the original image and complete image noise reduction and other functions; downsampling is mainly used to achieve multi-resolution expression of the same field of view. By repeatedly using low-pass filtering and downsampling, the image description of the same field of view can be achieved from rough to detailed, and a multi-layer image pyramid can be constructed.

[0103] The LK method based on the image pyramid is to reduce the large-scale displacement under the original resolution image into a small-scale displacement under the low-resolution image, so as to meet the constraints of high-precision feature tracking. Image feature tracking follows a coarse-to-fine calculation method. First, the LK algorithm is used to perform sub-pixel feature tracking on the lowest resolution image, and then the first-level image tracking result is passed to the next-level higher resolution image as the tracking initial value. The above algorithm process is repeated until the highest resolution original image is obtained.

[0104] S3.3 performs LK algorithm feature tracking on the image to complete image matching

[0105] The LK method completes image matching by finding the pixel with the minimum grayscale square difference within the pixel block. The calculation formula is shown in formula (4).

[0106]

[0107] Where I represents the previous frame reference image, J represents the adjacent next frame search image, k represents the side length of the square search window, A represents the affine transformation matrix, d represents the velocity measurement value at pixel x, and ε represents the grayscale square residual of the matching image block. When the image matching residual ε reaches the minimum value, the reference image I and the search image J complete the image matching. The LK algorithm describes the fine affine transformation between the sequence images through six parameters, where d xx and d yy Used to describe the scale transformation between sequence images, d xy and d yx It is used to describe the rotation and shear transformation between the sequence images. The vector d represents the displacement of the tracking feature between the sequence images and is used to calculate the flow velocity of the tracking feature.

[0108] S4. Remove erroneous tracking results based on image filtering and calculate the actual flow velocity.

[0109] The velocity field construction method based on image feature matching will inevitably produce erroneous matching results during the calculation process. Therefore, first of all, it is necessary to use image filtering methods to remove abnormal values ​​in image matching and improve the calculation accuracy of fluid velocity; then, it is necessary to convert the image velocity measurement results (pixels / second) into real three-dimensional world measurement results (meters / second), including:

[0110] S4.1 LK forward and backward estimation offset filtering

[0111] This embodiment introduces the LK forward and backward estimation offset d offset Roughly remove obviously wrong tracking results. Figure 3 is the estimated offset d of LK forward and backward offset The calculation method is to use the LK algorithm to track the image feature point a at time t forward, and get the position of the image feature point b at time t+1. Similarly, the position of the feature point c on the image at time t can be tracked backward according to the position of the feature point b in the image at time t+1. Theoretically, the positions of feature points a and c should coincide, but due to errors in optical flow calculation, the feature matching results obtained have a certain offset. This offset is the LK forward and backward estimated offset d offset When the estimated forward and backward offset of a feature point is d offset When the value is greater than the threshold, for example, when the threshold is set to 0.2 (this is taken as an example, but not limited to this), the feature tracking result is filtered out.

[0112] S4.2 Spatiotemporal filtering based on flow velocity and flow direction

[0113] Considering that rivers generally have a main flow direction, when the angle between the direction of a feature tracking result and the main flow direction of the water body is greater than a threshold (such as 120 degrees, for example, not limited to this), the feature tracking result is considered to be a mismatch result, and the gross error is filtered. The specific threshold should be determined according to the actual situation of the fluid.

[0114] Since the flow velocity measurement process of this embodiment is short, it can be considered that the flow direction and flow velocity at each position in the fluid remain unchanged during the flow velocity measurement process. Therefore, a spatiotemporal filtering method based on mean filtering is used to achieve noise removal, and the upper and lower speed thresholds are shown in formula (5):

[0115]

[0116] in, represents the average flow velocity, σ uRepresents the velocity variance, and parameter n represents the limit range of statistical filtering (such as: 1.5, which is taken as an example and is not limited to this). The mean filter is applied in both the spatial domain and the temporal domain: in the spatial domain filtering, for each pixel position, this embodiment uses the neighborhood tracking results around the given pixel point to filter out outliers (i.e., abnormal velocity calculation results that are too large, too small, or reverse); in the temporal domain filtering, this embodiment uses the average velocity value of multiple frames of images of the same pixel as a reference value, and uses formula (5) to remove the velocity calculation results of abnormal frames. Spatial domain outliers are manifested as excessive differences in velocity calculation results at adjacent spatial positions, which are mainly caused by incorrect feature matching; and temporal domain outliers are manifested as velocity solution results of all feature points on the entire frame image that are too large or too small, mainly because the recording time of the frame image is inconsistent with the actual time.

[0117] S4.3 Actual flow velocity calculation based on single image positioning

[0118] After image filtering, the velocity field solution result is still based on the image measurement value (pixel / second). In order to convert it into the velocity measurement value (m / second) of the real scene, the present invention first assumes that the water surface is a plane, determines the precise position of the water surface, and then uses the single image positioning method to solve the three-dimensional ground point coordinates of the first and last points of the image feature tracking result in the real scene, including:

[0119] S4.3.1 Determine the water surface position

[0120] Assuming that the water surface is a plane, firstly, several waterline points located at the boundary between land and water are manually selected from five frames of images that are far away on the trajectory of the UAV. Then, under the condition that the exterior orientation elements of the thermal infrared image sequence are known, the three-dimensional coordinates of these waterline points in the real scene are determined by the spatial forward intersection method. Finally, the three-dimensional coordinates of these points are used to fit a plane as the water surface.

[0121] S4.3.2 Calculation of actual flow rate

[0122] Under the assumption that the water surface is flat, the collinearity condition equation and the exterior orientation elements of the thermal infrared image sequence are used to solve the intersection of single image positioning and the horizontal plane to calculate the three-dimensional ground point coordinates of the first and last points of the image feature tracking results in the real scene. The feature tracking results (pixels / second) on the image sequence are converted into ground velocity measurements (m / second) of the real scene.

[0123] Embodiment 2: The above-mentioned embodiment 1 provides a method for measuring the optical flow velocity field based on a thermal infrared image sequence of a drone. Correspondingly, this embodiment provides a system for measuring the optical flow velocity field based on a thermal infrared image sequence of a drone. The system provided in this embodiment can implement the method for measuring the optical flow velocity field based on the thermal infrared image sequence of a drone in embodiment 1, and the system can be implemented by software, hardware, or a combination of software and hardware. For the convenience of description, the present embodiment is described separately by dividing the functions into various units. Of course, the functions of each unit can be implemented in the same or more software and / or hardware during implementation. For example, the system may include integrated or separate functional modules or functional units to perform the corresponding steps in each method of embodiment 1. Since the system of this embodiment is basically similar to the method embodiment, the description process of this embodiment is relatively simple, and the relevant parts can refer to the partial description of embodiment 1. The embodiment of the system for measuring the optical flow velocity field based on the thermal infrared image sequence of a drone provided by the present invention is only schematic.

[0124] The optical flow velocity field measurement system based on the thermal infrared image sequence of the unmanned aerial vehicle provided in this embodiment includes:

[0125] A geometric calibration unit is configured to solve the internal orientation elements of the uncooled thermal infrared camera using geometric calibration;

[0126] An image enhancement unit configured to perform contrast enhancement on a collected thermal infrared image data set of a UAV;

[0127] A trajectory calculation unit is configured to calculate the trajectory of the thermal infrared image sequence of the UAV by laying out ground control points;

[0128] a feature tracking unit configured to track image sequence features using temperature changes of the water body as a heat source tracer;

[0129] The filtering and flow rate calculation unit is configured to filter out erroneous image tracking results through image filtering and perform actual flow rate calculation.

[0130] Embodiment 3: This embodiment provides an electronic device corresponding to the method for measuring optical flow velocity field based on a drone thermal infrared image sequence provided in Embodiment 1. The electronic device may be an electronic device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of Embodiment 1.

[0131] like Figure 4As shown, the electronic device includes a processor, a memory, a communication interface and a bus, and the processor, the memory and the communication interface are connected through the bus to complete mutual communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Component (EISA) bus, etc. The memory stores a computer program that can be run on the processor, and when the processor runs the computer program, it executes the method for measuring the optical flow velocity field based on the thermal infrared image sequence of a drone provided in the first embodiment. Those skilled in the art can understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computing device to which the scheme of the present application is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] In some implementations, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), optical disk and other media that can store program codes.

[0133] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0134] Embodiment 4: The method for measuring the optical flow velocity field based on the thermal infrared image sequence of an unmanned aerial vehicle in the embodiment 1 can be specifically implemented as a computer program product. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for executing the method for measuring the optical flow velocity field based on the thermal infrared image sequence of an unmanned aerial vehicle described in the embodiment 1 are loaded.

[0135] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0136] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some implementations", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for measuring optical flow velocity field based on UAV thermal infrared image sequence, characterized in that include: The internal orientation elements of uncooled thermal infrared cameras are solved by geometric calibration; Perform contrast enhancement on the collected UAV thermal infrared image dataset; By laying out ground control points, the trajectory of the UAV thermal infrared image sequence is calculated, including: Deployment of ground control points; Segment the acquired image sequence, extract the image objects using multi-scale segmentation method, and realize object-level extraction of thermal infrared images; Based on the image segmentation results, the image control point image coordinates are extracted to achieve automatic registration of the UAV thermal infrared image sequence, including: Through the object-oriented control point feature extraction method, the position of the ground control point is automatically detected on each frame of the UAV thermal infrared image, and the position and attitude changes of the camera are determined by comparing the position changes of the control points on the UAV thermal infrared image sequence; Based on the image segmentation results, the edge index EI feature is used to extract the control point area from the candidate image objects: in, Representation Object m The perimeter of the boundary, Representation Object m The smaller the minimum circumscribed square side length is, the lower the EI value is, and the more likely the object is a control point area. The automatic extraction of the control point area can be achieved by simply selecting the set object with the smallest EI feature value from all candidate objects. The Shi-Tomasi algorithm is used to search for the strongest corner feature pixel by pixel in the control point area as the initial position, and the position of the control point is optimized using an iterative search method based on gradient descent. The image coordinates of the ground control point are extracted frame by frame from the UAV thermal infrared image sequence. Using matching point pairs and the least squares method, the affine transformation between adjacent frames is calculated to determine the pixel shift caused by the displacement and attitude of the drone: in, Indicates the position of the ground control point on the first frame of the reference image in the thermal infrared image sequence. Indicates the position of the ground control point on subsequent images in the thermal infrared image sequence; It describes the affine changes of subsequent images relative to the reference image, and describes the position and attitude changes of the UAV from the perspective of the two-dimensional image; Based on the camera's internal orientation elements and the image coordinates of the control points, the spatial resection is used to calculate the exterior orientation elements of each frame of the image to determine the position and posture of the UAV thermal infrared image sequence in three-dimensional space; The temperature change of the water body is used as a heat source tracer to track the image sequence features. The local hot spots in the cold and hot water mixed texture are extracted as the feature points to be tracked, and feature tracking is performed based on the image texture obtained by mixing cold and hot water. Image filtering is used to filter out erroneous image tracking results and calculate the actual flow velocity.

2. The optical flow velocity field measurement method based on the thermal infrared image sequence of an unmanned aerial vehicle according to claim 1 is characterized in that: The gradient histogram equalization method is used to enhance the contrast of the collected UAV thermal infrared image dataset.

3. The optical flow velocity field measurement method based on the UAV thermal infrared image sequence according to claim 1 is characterized in that: Determine image sequence feature tracking, including: Construct an image pyramid to achieve feature tracking based on local hot spots: First, use the LK algorithm to perform sub-pixel feature tracking on the image with the lowest resolution, then transfer the first-level image tracking result to the next-level higher-resolution image as the tracking initial value, and repeat the process until the original image with the highest resolution is obtained; Perform LK algorithm feature tracking on the image to complete image matching.

4. The optical flow velocity field measurement method based on the UAV thermal infrared image sequence according to claim 1 is characterized in that: Image filtering, including: Based on the LK forward and backward estimation offset filtering, the obviously wrong tracking results are removed. Specifically, the LK algorithm is used to track the image feature point a at time t forward to obtain the position of the image feature point b at time t+1. Similarly, according to the position of the feature point b in the image at time t+1, the position of the feature point c on the image at time t is obtained by reverse tracking. The offset between the two is calculated as the LK forward and backward estimation offset. , when the estimated forward and backward offset of a feature point If it is greater than the set threshold, the feature tracking result is filtered out; Based on the spatiotemporal filtering of flow velocity and direction, outliers in image matching are eliminated.

5. The optical flow velocity field measurement method based on the thermal infrared image sequence of an unmanned aerial vehicle according to claim 1 is characterized in that: Actual flow rate calculation, including: Fit the water surface position to a plane; By using the collinearity condition equation and the exterior orientation elements of the UAV thermal infrared image sequence, the three-dimensional ground point coordinates of the first and last points of the image feature tracking results in the real scene are calculated by solving the intersection of single image positioning and the horizontal plane. The feature tracking results on the UAV thermal infrared image sequence are converted into ground speed measurements of the real scene.

6. A system for implementing the optical flow velocity field measurement method based on a UAV thermal infrared image sequence as described in any one of claims 1 to 5, characterized in that: The system includes: A geometric calibration unit is configured to solve the internal orientation elements of the uncooled thermal infrared camera using geometric calibration; An image enhancement unit configured to perform contrast enhancement on a collected thermal infrared image data set of a UAV; A trajectory calculation unit is configured to calculate the trajectory of the thermal infrared image sequence of the UAV by laying out ground control points; a feature tracking unit configured to track image sequence features using temperature changes of the water body as a heat source tracer; The filtering and flow rate calculation unit is configured to filter out erroneous image tracking results through image filtering and perform actual flow rate calculation.

7. An electronic device, comprising at least a processor and a memory, wherein a computer program is stored in the memory, wherein: When the processor runs the computer program, the method according to any one of claims 1 to 4 is implemented.

8. A computer storage medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1 to 4.

Images