Unmanned aerial vehicle multi-modal image processing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an unmanned aerial vehicle multi-modal image processing method and device, equipment and a storage medium, and the method comprises the steps: carrying out the geometric correction and radiation correction of multi-modal image data, obtained by different sensors, of an unmanned aerial vehicle; respectively carrying out feature extraction on the preprocessed multi-modal images to obtain features extracted from different modal images; matching the features extracted from the different modal images through a feature matching algorithm; the matched features are fused, and a fused feature vector is generated; target identification and classification processing are carried out on the different modal images, fusion decision making is carried out on identification results of the different modal images, and target object category and attribute information is obtained; and based on the fused image data, the target object is segmented by using an image segmentation algorithm based on region growth to obtain the segmented target object, so that the accuracy and richness of unmanned aerial vehicle image data processing can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle data processing, and in particular to a method, device, equipment and storage medium for unmanned aerial vehicle multimodal image processing. Background Art

[0002] In recent years, sensor unmanned aerial vehicles (UAVs) have been widely used in the civilian field.

[0003] With the widespread application of drone technology, the image data obtained by it has important value in many fields such as agricultural monitoring, environmental exploration, urban planning, etc. However, single-modal image data often has limitations. For example, visible light images are not effective when the lighting is complex or the target features are not obvious. Although infrared images can detect temperature differences, the spatial resolution may be low. Therefore, integrating multi-modal image data for processing can improve the accuracy and richness of information, but the current processing methods still need to be improved in terms of efficiency and accuracy of image processing, feature extraction and analysis. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, device, equipment and storage medium for multimodal image processing of unmanned aerial vehicles, which can improve the accuracy and richness of unmanned aerial vehicle image data processing.

[0005] To achieve the above objectives, the present application provides a method for processing multimodal images of a drone, the method comprising:

[0006] Performing geometric correction and radiation correction on multimodal image data of the drone acquired by different sensors to obtain a preprocessed multimodal image, wherein the multimodal image data includes visible light image data, infrared image data, and lidar data;

[0007] Extracting features from the preprocessed multimodal images respectively to obtain features extracted from images of different modalities; matching the features extracted from images of different modalities by a feature matching algorithm;

[0008] Fuse the matched features to generate a fused feature vector;

[0009] Respectively perform target recognition and classification processing on the different modal images, fuse the recognition results of the different modal images to obtain the target object category and attribute information;

[0010] Based on the fused image data, the target object is segmented using an image segmentation algorithm based on region growing to obtain the segmented target object.

[0011] Optionally, the geometric correction of the multimodal image data of the drone acquired by different sensors includes:

[0012] For the laser radar data, according to the attitude angle and installation position parameters of the drone, the point cloud data is mapped from the sensor coordinate system to a unified three-dimensional space coordinate system through a coordinate transformation matrix;

[0013] The visible light image data and the infrared image data are corrected according to a lens distortion model and correction parameters.

[0014] Optionally, performing radiation correction on the multimodal image data of the drone acquired by different sensors includes:

[0015] For the visible light image data, adjusting the image pixel values ​​according to the estimated value of the light intensity at the time of shooting to compensate for the light non-uniformity;

[0016] For the infrared image data, the pre-calibrated black body radiation curve and the current ambient temperature setting are used to calibrate the temperature value in the image.

[0017] Optionally, the extracting features of the preprocessed multimodal images respectively to obtain features extracted from images of different modalities includes:

[0018] For the visible light image data, extracting texture and color features using a convolutional neural network;

[0019] For the infrared image data, extract temperature distribution features;

[0020] For the laser radar data, extract the three-dimensional contour and distance features of the object;

[0021] The matching of the features extracted from the images of different modalities by a feature matching algorithm includes:

[0022] A scale-invariant feature transformation algorithm is used to find feature points of corresponding objects in the different modal images, and associations between feature vectors are established based on the feature points.

[0023] Optionally, the performing target recognition and classification processing on the different modality images respectively includes:

[0024] Identifying the target object category in the visible light image data using a target detection algorithm;

[0025] determining thermal properties of a target object based on temperature differences in the infrared image data;

[0026] The spatial form of the target object is identified by point cloud clustering analysis in the laser radar data.

[0027] Optionally, the recognition results of the different modal images are fused and decided to obtain the category and attribute information of the target object, including:

[0028] Assume that the probability that the target object is identified as a vehicle in the visible light image data is P1, the probability that the target object has the thermal characteristics of a vehicle in the infrared image data is P2, and the probability that the target object has the geometric characteristics of a vehicle in the lidar data is P3. At the same time, assume that there are m possible target categories, and the probabilities of other categories of objects are P4, P5, ..., P m ;

[0029] According to Bayes' theorem, the comprehensive probability P that the target object is a vehicle is calculated as follows:

[0030]

[0031] Where P i1 ,P i2 ,P i3 are the probabilities of the i-th type of object in the visible light image data, the infrared image data and the lidar data, respectively.

[0032] Optionally, segmenting the target object based on the fused image data using an image segmentation algorithm based on region growing includes:

[0033] Select a seed point (x s ,y s ), set the similarity threshold T sim For its neighborhood point (x, y), calculate the similarity measure S of the multimodal feature. The multimodal feature includes the color feature C from the visible light image data, the temperature feature from the infrared image data, and the distance feature D. The similarity measure S is expressed as:

[0034] S=w C d(C(x,y),C(x s ,y s ))+w T T(x,y)-T(x s ,y s )+w D D(x,y)-D(x s ,y s )

[0035] Among them, d(C(x,y),C(x s ,y s )) is the distance metric of the color feature, w C ,w T ,w D is the weighting coefficient;

[0036] When S<T sim When , the neighborhood point (x, y) is classified as the same area as the seed point; the above process is repeated until no new point meets the conditions, and the segmentation of the target object is completed.

[0037] On the other hand, the present application also provides a multimodal image processing device for an unmanned aerial vehicle, comprising:

[0038] A correction module is used to perform geometric correction and radiation correction on multimodal image data of the drone acquired by different sensors to obtain a pre-processed multimodal image, wherein the multimodal image data includes visible light image data, infrared image data and lidar data;

[0039] A feature extraction module is used to extract features from the preprocessed multi-modal images respectively to obtain features extracted from images of different modalities; and to match the features extracted from images of different modalities by a feature matching algorithm;

[0040] The feature fusion module is used to fuse the matched features and generate a fused feature vector;

[0041] The recognition module is used to perform target recognition and classification processing on the different modal images respectively, and to fuse the recognition results of the different modal images to obtain the target object category and attribute information.

[0042] On the other hand, the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the first aspect and any possible implementation method thereof.

[0043] On the other hand, the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes each step of the method described in the first aspect.

[0044] The present application provides a method, device, equipment and storage medium for multimodal image processing of unmanned aerial vehicles. The method performs geometric correction and radiation correction on multimodal image data of unmanned aerial vehicles acquired by different sensors to obtain preprocessed multimodal images, wherein the multimodal image data includes visible light image data, infrared image data and lidar data; performs feature extraction on the preprocessed multimodal images to obtain features extracted from images of different modalities; matches the features extracted from the images of different modalities through a feature matching algorithm; fuses the matched features to generate a fused feature vector; performs target recognition and classification processing on the images of different modalities respectively, performs fusion decision on the recognition results of the images of different modalities, and obtains the category and attribute information of the target object; based on the fused image data, uses an image segmentation algorithm based on region growing to segment the target object to obtain the segmented target object, thereby improving the accuracy and richness of the image data processing of unmanned aerial vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] in:

[0047] Figure 1 A schematic diagram of a flow chart of a multimodal image processing method for a drone provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the structure of a multi-modal image processing device for a drone provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0051] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0054] See also Figure 1 , is a flow chart of a multi-modal image processing method for a drone provided in an embodiment of the present application, such as Figure 1 As shown, the method includes:

[0055] 101. Perform geometric correction and radiation correction on multimodal image data of the drone acquired by different sensors to obtain a preprocessed multimodal image. The multimodal image data includes visible light image data, infrared image data, and lidar data.

[0056] The execution subject of the method in the embodiment of the present application is a multimodal image processing device for unmanned aerial vehicles, and in actual applications, it can be executed on a terminal device.

[0057] The multimodal image mentioned in the embodiments of the present application refers to the image data that combines images from different sensors or different modalities (such as visible light, infrared, radar, etc.). This technology can provide more comprehensive information than single-modal images and is widely used in feature extraction, image fusion, target recognition and other fields.

[0058] Specifically, the drone can be equipped with a variety of image sensors, including but not limited to visible light cameras, infrared thermal imagers, laser radars, etc. During the flight, these sensors are simultaneously activated to collect image data, and each collected image data is labeled with a timestamp and spatial coordinates for subsequent accurate matching and fusion.

[0059] After collecting the multimodal image data, image preprocessing may be performed first.

[0060] In an optional implementation, the geometric correction of the multimodal image data of the drone acquired by different sensors includes:

[0061] For the above-mentioned lidar data, according to the attitude angle and installation position parameters of the drone, the point cloud data is mapped from the sensor coordinate system to a unified three-dimensional space coordinate system through a coordinate transformation matrix;

[0062] The visible light image data and the infrared image data are corrected according to the lens distortion model and correction parameters.

[0063] The first aspect is geometric correction:

[0064] Aiming at the possible geometric deformation of images acquired by different sensors, the known UAV attitude information and sensor parameters are used to perform geometric correction on each modality image.

[0065] For example, for lidar point cloud data, the point cloud can be converted from the sensor coordinate system to a unified three-dimensional space coordinate system (world coordinate system) through a coordinate transformation matrix based on the attitude angle (pitch, yaw, roll) and installation position parameters of the drone. For visible light images and infrared images, the lens distortion model and correction parameters can be used to dedistort the image to ensure the accuracy of the geometric shape of the object in the image.

[0066] Specific:

[0067] (1) Assume that the attitude angle information of the drone is the pitch angle θ, the yaw angle Roll angle ω, the installation position coordinates of the laser radar in the drone body coordinate system are (x0, y0, z0). For any point P (x, y, z) in the point cloud data collected by the laser radar (in the laser radar coordinate system), it is converted to the world coordinate system (X ω ,Y ω ,Z ω )’s coordinate transformation formula is as follows:

[0068] First, convert point P from the laser radar coordinate system to the drone body coordinate system, taking into account the installation position offset. The formula is:

[0069]

[0070] Then, the rotation coordinate transformation is performed according to the attitude angle of the drone. The rotation matrix R around the x-axis (roll angle ω) x , the rotation matrix R around the y-axis (pitch angle θ) y and about the z-axis (yaw angle )’s rotation matrix R z They are:

[0071]

[0072] The coordinate transformation from the drone body coordinate system to the world coordinate system is:

[0073]

[0074] (2) Visible light and infrared image geometric correction (lens distortion correction)

[0075] For lens distortion correction, the radial distortion model can be:

[0076] x d =x(1+k1r 2 +k2r 4 )

[0077] y d =y(1+k1r 2 +k2r 4 )

[0078] Where (x, y) is the coordinate in the ideal undistorted image, (x d ,y d ) are the coordinates in the distorted image, k1 and k2 are radial distortion coefficients. By knowing the distortion coefficients and the coordinates in the image, the corrected coordinates can be calculated.

[0079] In an optional implementation, the radiometric correction is performed on the multimodal image data of the drone acquired by different sensors, including:

[0080] For the above visible light image data, the image pixel values ​​are adjusted according to the estimated value of the light intensity at the time of shooting to compensate for the unevenness of the light;

[0081] For the above infrared image data, the pre-calibrated black body radiation curve and the current ambient temperature setting are used to calibrate the temperature value in the image.

[0082] The second aspect is radiation correction:

[0083] Since different sensors have different response characteristics to ambient lighting and other conditions, radiation correction is performed on the collected images. For visible light images, the influence of uneven lighting is compensated based on the lighting model and the lighting conditions at the time of shooting; for infrared images, their temperature values ​​are calibrated according to the blackbody radiation calibration curve to ensure the accuracy of temperature measurement.

[0084] For visible light images, the image pixel values ​​can be adjusted to compensate for the uneven illumination based on the estimated illumination intensity at the time of shooting (which can be obtained through the illumination sensor on the drone or estimated based on information such as the sun angle). For infrared images, the temperature value in the image can be calibrated based on the pre-calibrated blackbody radiation curve and the current ambient temperature setting.

[0085] Specific:

[0086] (1) Visible light image radiation correction (compensation for uneven illumination)

[0087] Assume that the estimated light intensity at the time of shooting is I0, and the original pixel value of a pixel (i, j) in the image is I ij , the illumination model is assumed to be a linear model, and the corrected pixel value I i ' j The calculation is as follows:

[0088]

[0089] Among them I ref It is the standard pixel value under reference light intensity, which can be obtained by calibration under known lighting conditions.

[0090] (2) Infrared image radiation correction (temperature calibration)

[0091] For infrared images, according to the blackbody radiation law, the radiation intensity M(λ,T) of an object is related to the temperature T and the wavelength λ, and the formula is:

[0092]

[0093] Where h is Planck's constant, c is the speed of light, and k is the Boltzmann constant. Given the calibration parameters of the infrared thermal imager and the current ambient temperature T env , for the temperature measurement value T corresponding to a pixel in the image ij , calibrated by comparing with the blackbody radiation calibration curve. The specific calculation is based on the calibration formula of the thermal imager, and the general form is:

[0094] T calibrated =f(T ij ,T env ,calibration|parameters)

[0095] Where f is the calibration function provided by the thermal imager manufacturer.

[0096] 102. Perform feature extraction on the preprocessed multimodal images respectively to obtain features extracted from images of different modalities; and match the features extracted from the images of different modalities by a feature matching algorithm.

[0097] The multimodal image fusion in the embodiment of the present application mainly includes two aspects: feature-level fusion and decision-level fusion. First, feature-level fusion is introduced.

[0098] In an optional implementation, the above-mentioned performing feature extraction on the preprocessed multimodal images respectively to obtain features extracted from images of different modalities includes:

[0099] For the above visible light image data, a convolutional neural network is used to extract texture and color features;

[0100] For the above infrared image data, extract temperature distribution features;

[0101] For the above-mentioned lidar data, extract the three-dimensional contour and distance features of the object;

[0102] The features extracted from the above-mentioned different modal images are matched by the feature matching algorithm, including:

[0103] A scale-invariant feature transformation algorithm is used to find the feature points of corresponding objects in the above-mentioned different modal images, and the association between feature vectors is established based on the above-mentioned feature points.

[0104] Specifically, firstly, feature extraction is performed on each modality image after preprocessing. Optionally, for visible light images, convolutional neural network (CNN) is used to extract features such as texture and color; for infrared images, temperature distribution features are extracted; for lidar data, three-dimensional contours and distance features of objects are extracted.

[0105] Then, the features extracted from the images of different modalities are matched by a feature matching algorithm. The Scale Invariant Feature Transform (SIFT) algorithm can be used to find the feature points of the corresponding objects in the images of different modalities, and the association between the feature vectors is established based on these feature points.

[0106] Finally, the matched features are fused using weighted averaging, principal component analysis (PCA) and other methods to generate a fused feature vector, which contains both the appearance information of the object and its thermal information and three-dimensional structure information.

[0107] Specific:

[0108] (1) Feature extraction calculation

[0109] Visible light image feature extraction (CNN)

[0110] Taking the commonly used VGG-16 network as an example, the input visible light image size is 224*224*3. In the convolution layer, for the first layer of convolution, the convolution kernel size is 3*3, the step size is 1, and the padding is 1. Assuming that the number of input channels is 3 and the number of convolution kernels is 64, the size of the convolution layer output feature map is:

[0111]

[0112] The output feature map dimension is 224*224*64. After multiple layers of convolution, pooling and other operations, the final feature vector is used to represent the texture, color and other features of the visible light image.

[0113] (2) Infrared image feature extraction (LBP+GLCM)

[0114] For infrared images, the local binary pattern (LBP) is calculated as follows:

[0115] For the pixel point (i, j) in the image, take a circular neighborhood with a radius of R and a number of neighborhood points P as the center. Compare the neighborhood pixel value with the center pixel value, record it as 1 if it is greater than the center value, and record it as 0 if it is less than the center value, and get a P-bit binary number, which is converted into a decimal number as the LBP value of the point. For example, when R = 1 and P = 8, calculate the difference between the neighborhood pixel value and the center pixel (i, j), and get a binary number such as 01101001, which is converted to decimal as 105, which is the LBP value of the point.

[0116] The gray-level co-occurrence matrix (GLCM) mentioned in the embodiments of the present application describes the texture by calculating the gray-level joint distribution of pixel pairs at a certain distance and angle in the image. For example, the GLCM of pixel pairs with a horizontal distance of d is calculated, and the number of pixel pairs with gray-level values ​​of g1 and g2 is counted. Common texture feature parameters such as contrast, energy, entropy, etc. can be calculated based on GLCM. By combining LBP and GLCM features, the temperature distribution-related texture features of the infrared image are obtained.

[0117] (3) LiDAR point cloud data feature extraction (normal vector estimation, curvature calculation, etc.)

[0118] For the normal vector estimation of LiDAR point cloud data, the commonly used method is based on local plane fitting. i point, select its k nearest neighbor points {P i1, P i2, …, P ik}, fit a plane by the least squares method, the normal vector of the plane is point P i The estimated normal vector of .

[0119] There are many methods for calculating the curvature of a point in point cloud data. One method based on principal component analysis (PCA) is as follows: PCA analysis is performed on the k nearest neighbors of the point, and the curvature is calculated based on the eigenvalues ​​of the principal components. Assuming that the eigenvalues ​​of the three principal components are λ1≥λ2≥λ3, the curvature C of the point can be approximately calculated as:

[0120]

[0121] The feature matching in the embodiments of the present application may adopt the SIFT algorithm.

[0122] Specifically, we first construct a scale space on different modal images. We detect extreme points in the Gaussian difference scale space, and accurately determine the position and scale of the extreme points by fitting a three-dimensional quadratic function. For each extreme point, we calculate its direction histogram and generate a feature descriptor.

[0123] Suppose a SIFT feature descriptor of a visible light image is D vis =(d vis1 ,d vis2 ,…,d vis128 ), a SIFT feature descriptor of infrared image is D ir =(d ir1 ,d ir2 …,d ir128 ). Calculate the Euclidean distance between the two:

[0124]

[0125] By setting the threshold T, when d>T, the two feature points are considered to match.

[0126] 103. Fuse the matched features to generate a fused feature vector.

[0127] The feature fusion in the embodiment of the present application may adopt weighted averaging. Specifically, assuming that a pair of feature points of the visible light image and the infrared image match, the feature vector of the visible light image is F vis =(f vis1 ,f vis2 ,…,f visn ), the infrared image feature vector is F ir =(f ir1 ,f ir2 ,…,f irn ). The fused feature vector

[0128] F fusin =(f fusion1 ,f fusin2 ,…,f fusionn ) is calculated as follows:

[0129] f fusioni =w vis f visi +w ir f iri

[0130] where w vis and w iris a weighting coefficient, which can be set according to the importance of the feature or prior knowledge, and w vis +w ir =1.

[0131] 104. Perform target recognition and classification processing on the above-mentioned different modal images respectively, and make a fusion decision on the recognition results of the above-mentioned different modal images to obtain the target object category and attribute information.

[0132] For the decision-level fusion mentioned above, each modality image is first subjected to target recognition and classification processing. For example, in visible light images, the target detection algorithm is used to identify the possible target object category; in infrared images, the thermal properties of the target object are determined based on the temperature difference; in lidar data, the spatial form of the target object is identified through point cloud clustering analysis.

[0133] Then, the recognition results obtained from each modality can be integrated and decided. Methods such as Bayesian reasoning and DS evidence theory can be used to comprehensively consider the confidence of different modalities to obtain the final target object category and attribute information.

[0134] Specific:

[0135] Assume that the probability of a target being identified as a vehicle in a visible light image is P1, the probability of a target having a vehicle thermal signature in an infrared image is P2, and the probability of a target having a vehicle geometric signature in a lidar is P3. At the same time, assume that there are m possible target categories, and the probabilities of other categories of objects are P4, P5, ..., P m .

[0136] According to Bayes' theorem, the comprehensive probability P that the target object is a vehicle is calculated as follows:

[0137]

[0138] Where P i1 ,P i2 ,P i3 are the probabilities of the i-th object in visible light, infrared and lidar images, respectively.

[0139] Based on the fused image data, the target object is segmented using an image segmentation algorithm based on region growing to obtain a segmented target object.

[0140] The embodiments of the present application also involve image post-processing and analysis.

[0141] The region growing algorithm mentioned in the embodiment of the present application is a pixel-based image segmentation method, which starts from one or more seed points and merges adjacent pixels into the seed region according to a predefined growth criterion until a specific stop condition is met. The growth criterion usually includes the neighborhood of the pixel, the connectivity of the pixel, and the similarity judgment criterion,

[0142] Specifically, the target object can be segmented based on the fused image using a region growing-based image segmentation algorithm. Starting from the feature points or seed points in the fused image, the target object is gradually grown according to the similarity of the pixels (including multimodal features such as color, texture, temperature and distance) until the stop condition is met, and the target object is separated from the background. At the same time, the display effect of the target object in the fused image can be enhanced by contrast stretching and adaptive histogram equalization, and the contrast between the target object and the background can be improved.

[0143] In an optional implementation, the above step 105 includes:

[0144] Select a seed point (x s ,y s ), set the similarity threshold T sim For its neighborhood point (x, y), calculate the similarity measure S of the multimodal feature. The multimodal feature includes the color feature C from the visible light image data, the temperature feature from the infrared image data, and the distance feature D. The similarity measure S is expressed as:

[0145] S=w C d(C(x,y),C(x s ,y s ))+w T T(x,y)-T(x s ,y s )+w D D(x,y)-D(x s ,y s )

[0146] Among them, d(C(x,y),C(x s ,y s )) is the distance metric of the above color features, w C ,w T ,w D is the weighting coefficient;

[0147] When S<T sim , classify the neighborhood point (x, y) as the same area as the seed point; repeat the above process until no new point meets the conditions, and the segmentation of the target object is completed.

[0148] Specific:

[0149] 1. Target enhancement and segmentation calculation (image segmentation based on region growing)

[0150] Select a seed point (x s ,y s ), set the similarity threshold T sim For its neighborhood point (x, y), calculate the similarity measure S of the multimodal features. For example, the multimodal features include color feature C (from visible light image), temperature feature T (from infrared image) and distance feature D (the distance from the seed point in three-dimensional space, combined with lidar and geometric correction information), and the similarity measure can be a weighted distance:

[0151] S=w C d(C(x,y),C(x s ,y s ))+w T T(x,y)-T(x s ,y s )+w D D(x,y)-D(x s ,y s )

[0152] Among them, d(C(x,y),C(x s ,y s )) is the distance metric of color features (such as Euclidean distance), w C ,w T ,w D is the weighting coefficient. When S<T sim When , the neighborhood point (x, y) is classified as the same area as the seed point. This process is repeated until no new points meet the conditions, and the segmentation of the target object is completed. The above segmentation method improves the segmentation efficiency and accuracy.

[0153] 2. Parameter calculation and analysis:

[0154] For two-dimensional targets (such as objects in a plane image), its area, perimeter and other geometric parameters can be calculated. For three-dimensional targets (such as objects acquired by lidar), its volume, surface area and other parameters can be calculated based on point cloud data. Based on the temperature information in the infrared image, thermal parameters such as the average temperature and temperature gradient of the target object can be calculated. The movement speed and trajectory of the target object can also be calculated by combining the flight speed of the drone and the position change of the target object in the continuous frame images. These parameters will be used for subsequent analysis and decision-making, such as evaluating crop pests and diseases in agricultural monitoring and judging the diffusion trend of pollution sources in environmental monitoring.

[0155] Specific:

[0156] (1) Calculation of two-dimensional target parameters (area, perimeter)

[0157] For the segmented two-dimensional target object (such as an object in a plane image), assume that the boundary point coordinates of the target object are (x1, y1), (x2, y2), …, (x n ,y n ). The area A can be calculated using Green's formula:

[0158]

[0159] (where (x n+1 ,y n+1 )=(x1,y1))

[0160] The perimeter can be found by calculating the sum of the distances between adjacent boundary points:

[0161]

[0162] (2) Calculation of three-dimensional target parameters (volume, surface area)

[0163] For a three-dimensional target object (acquired by LiDAR), assuming that the point cloud data represents the target object, a simple volume calculation method is based on voxelization. The space containing the target object is divided into small voxels, and the number of voxels N occupied by the point cloud of the target object is counted. The volume of each voxel is v, and the volume of the target object V = N × v.

[0164] The surface area calculation is relatively complex and can be done through triangulation. Specifically, the surface of the target object can be approximated as a series of triangles, and the area of ​​each triangle is calculated and summed to get the surface area. For a triangle composed of three points P1 (x1, y1, z1), P2 (x2, y2, z2) and P3 (x3, y3, z3), its area S Δ It can be calculated based on the vector cross product:

[0165]

[0166] (3) Temperature parameter calculation (average temperature, temperature gradient)

[0167] For the target object in the infrared image, it is assumed that the temperature value of the pixel point in the target object is T ij (i=1,2,…,m,j=1,2,…n), then the average temperature T avg Calculated as:

[0168]

[0169] The temperature gradient calculation can be approximated by calculating the difference between the temperature values ​​of adjacent pixels. For example, in the horizontal direction, the temperature gradient Gx At the pixel point (i, j), it can be calculated as: G x =T ij -T i,j-1 (or use a more complex difference format).

[0170] (4) Motion speed calculation (combining UAV flight parameters and multi-frame images)

[0171] Assume that the flying speed of the drone is v drone (including horizontal and vertical velocity components), the center coordinates of the target object in two adjacent frames are (x1, y1, z1) and (x2, y2, z2), and the time interval between the two frames is Δt. The horizontal velocity v of the target object x and the vertical velocity v y (Assuming a simplified two-dimensional plane motion) can be calculated as:

[0172]

[0173] The target object's moving speed v is If three-dimensional motion is considered, more drone attitude and coordinate information needs to be combined for calculation.

[0174] The method in the embodiment of the present application performs geometric correction and radiation correction on the multimodal image data of the drone acquired by different sensors to obtain a preprocessed multimodal image, wherein the multimodal image data includes visible light image data, infrared image data and laser radar data; performs feature extraction on the preprocessed multimodal image to obtain features extracted from different modal images; matches the features extracted from the different modal images through a feature matching algorithm; fuses the matched features to generate a fused feature vector; performs target recognition and classification processing on the different modal images respectively, and fuses the recognition results of the different modal images to obtain the target object category and attribute information; based on the fused image data, uses an image segmentation algorithm based on region growth to segment the target object, and can integrate the multimodal images of the drone for processing to achieve segmentation of the target object and related information analysis. This method improves the accuracy and richness of the drone multimodal image data processing, and improves the efficiency and accuracy of multimodal image fusion, feature extraction and analysis. Based on the description of the aforementioned method embodiment, the embodiment of the present application also discloses a drone multimodal image processing device.

[0175] Figure 2 A schematic diagram of the structure of a multi-modal image processing device for a drone provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the drone multimodal image processing device 200 may include:

[0176] The correction module 210 is used to perform geometric correction and radiation correction on the multimodal image data of the drone acquired by different sensors to obtain a pre-processed multimodal image, wherein the multimodal image data includes visible light image data, infrared image data and lidar data;

[0177] The feature extraction module 220 is used to extract features from the preprocessed multi-modal images respectively to obtain features extracted from images of different modalities; and to match the features extracted from images of different modalities by a feature matching algorithm;

[0178] A feature fusion module 230 is used to fuse the matched features to generate a fused feature vector;

[0179] The recognition module 240 is used to perform target recognition and classification processing on the different modal images respectively, and to fuse the recognition results of the different modal images to obtain the target object category and attribute information.

[0180] It can be understood that the relevant contents of each module in the above device have been described in detail in the aforementioned method embodiment, and the specific contents can be referred to the method embodiment; that is, the multimodal image processing device for unmanned aerial vehicles provided in this application can perform the following steps: Figure 1 Any steps in the illustrated embodiment will not be described in detail here.

[0181] In one embodiment of the present application, an electronic device is also proposed. The electronic device may be a drone, which may include a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the following will be executed: Figure 1 Any step in the method embodiment shown. The electronic device may also include an input / output device and the like.

[0182] In one embodiment, a computer-readable storage medium is further provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes any step in the above method embodiment.

[0183] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0184] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multimodal image processing method for unmanned aerial vehicles, characterized in that: The method comprises: Performing geometric correction and radiation correction on multimodal image data of the drone acquired by different sensors to obtain a preprocessed multimodal image, wherein the multimodal image data includes visible light image data, infrared image data, and lidar data; Extracting features from the preprocessed multimodal images respectively to obtain features extracted from images of different modalities; matching the features extracted from images of different modalities by a feature matching algorithm; Fuse the matched features to generate a fused feature vector; Respectively perform target recognition and classification processing on the different modal images, fuse the recognition results of the different modal images to obtain the target object category and attribute information; Based on the fused image data, the target object is segmented using an image segmentation algorithm based on region growing to obtain the segmented target object.

2. The multimodal image processing method of unmanned aerial vehicle according to claim 1, characterized in that: The method of geometrically correcting the multimodal image data of the drone acquired by different sensors includes: For the laser radar data, according to the attitude angle and installation position parameters of the drone, the point cloud data is mapped from the sensor coordinate system to a unified three-dimensional space coordinate system through a coordinate transformation matrix; The visible light image data and the infrared image data are corrected according to a lens distortion model and correction parameters.

3. The multimodal image processing method of unmanned aerial vehicle according to claim 2, characterized in that: The method of performing radiation correction on the multimodal image data of the drone acquired by different sensors includes: For the visible light image data, adjusting the image pixel values ​​according to the estimated value of the light intensity at the time of shooting to compensate for the light non-uniformity; For the infrared image data, the pre-calibrated black body radiation curve and the current ambient temperature setting are used to calibrate the temperature value in the image.

4. The multimodal image processing method of unmanned aerial vehicle according to claim 1, characterized in that: The extracting features of the preprocessed multi-modal images respectively to obtain features extracted from images of different modalities includes: For the visible light image data, extracting texture and color features using a convolutional neural network; For the infrared image data, extract temperature distribution features; For the laser radar data, extract the three-dimensional contour and distance features of the object; The matching of the features extracted from the images of different modalities by a feature matching algorithm includes: A scale-invariant feature transformation algorithm is used to find feature points of corresponding objects in the different modal images, and associations between feature vectors are established based on the feature points.

5. The method for processing multimodal images of unmanned aerial vehicles according to claim 1, characterized in that: The performing target recognition and classification processing on the different modality images respectively includes: Identifying the target object category in the visible light image data using a target detection algorithm; determining thermal properties of a target object based on temperature differences in the infrared image data; The spatial form of the target object is identified by point cloud clustering analysis in the laser radar data.

6. The method for processing multimodal images of unmanned aerial vehicles according to claim 5, characterized in that: The recognition results of the different modal images are fused and decided to obtain the category and attribute information of the target object, including: Assume that the probability that the target object is identified as a vehicle in the visible light image data is P1, the probability that the target object has the thermal characteristics of a vehicle in the infrared image data is P2, and the probability that the target object has the geometric characteristics of a vehicle in the lidar data is P3. At the same time, assume that there are m possible target categories, and the probabilities of other categories of objects are P4, P5, ..., P m ; According to Bayes' theorem, the comprehensive probability P that the target object is a vehicle is calculated as follows: Where P i1 ,P i2 ,P i3 are the probabilities of the i-th type of object in the visible light image data, the infrared image data and the lidar data, respectively.

7. The method for processing multimodal images of unmanned aerial vehicles according to claim 6, characterized in that: The method further includes: segmenting the target object based on the fused image data using an image segmentation algorithm based on region growing, specifically including: Select a seed point (x s ,y s ), set the similarity threshold T sim , for its neighborhood point (x, y), calculate the similarity measure S of the multimodal feature; the multimodal feature includes the color feature C from the visible light image data, the temperature feature from the infrared image data, and the distance feature D, and the similarity measure S is expressed as: S=w C d(C(x,y),C(x s ,y s ))+w T |T(x,y)-T(x s ,y s )|+w D |D(x,y)-D(x s ,y s ) Among them, d(C(x,y),C(x s ,y s )) is the distance metric of the color feature, w C ,w T ,w D is the weighting coefficient; When S<T sim When , the neighborhood point (x, y) is classified as the same area as the seed point; the above process is repeated until no new point meets the conditions, and the segmentation of the target object is completed.

8. A multimodal image processing device for an unmanned aerial vehicle, characterized in that: include: A correction module is used to perform geometric correction and radiation correction on multimodal image data of the drone acquired by different sensors to obtain a pre-processed multimodal image, wherein the multimodal image data includes visible light image data, infrared image data and lidar data; A feature extraction module, used to extract features from the preprocessed multi-modal images respectively to obtain features extracted from images of different modalities; Matching the features extracted from the images of different modalities by a feature matching algorithm; The feature fusion module is used to fuse the matched features and generate a fused feature vector; The recognition module is used to perform target recognition and classification processing on the different modal images respectively, and to fuse the recognition results of the different modal images to obtain the target object category and attribute information.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images