Uncertainty modeling system for unmanned aerial vehicle aerial image sequence identification

Abstract

The invention relates to an uncertainty modeling system for unmanned aerial vehicle aerial image sequence recognition. The uncertainty modeling system comprises an image sequence input module, an unmanned aerial vehicle image analysis module, a time sequence-based uncertainty evaluation module and an output module, the image sequence input module comprises an unmanned aerial vehicle aerial image sequence primary identification model; in the unmanned aerial vehicle image analysis module, a Gaussian mixture model is used for estimating the probability of image recognition output; the time sequence-based uncertainty evaluation module comprises an initial estimation module and a time sequence optimization module; the initial estimation module is used for estimating initial variation condition probability distribution; the time sequence optimization module introduces image sequence correlation and estimates final variational conditional probability distribution; and the output module outputs the final uncertainty expression. According to the method, by considering the time sequence correlation of the image sequence, the identification accuracy of the aerial image sequence of the unmanned aerial vehicle and the reliability of uncertainty modeling are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and particularly relates to an uncertainty modeling system for identifying unmanned aerial vehicle (UAV) aerial image sequences. Background Art

[0002] With the continuous progress of technology, UAV aerial photography technology has become increasingly mature, and the application value of identifying UAV aerial image sequences has become more prominent in multiple fields. The image sequence of UAV aerial photography refers to a series of images obtained by continuously photographing the same area at a certain time interval. At present, the technology of identifying UAV aerial image sequences has been successfully applied in many fields. For example, in security monitoring, UAVs can quickly reach the monitoring area with their flexibility and mobility and photograph image sequences at a certain time interval. By identifying and analyzing these image sequences, the activities of personnel and vehicle flow in a specific area can be monitored in real time, and abnormal behaviors can be detected in time and alarms can be issued. In the field of geographical mapping, UAVs can photograph image sequences according to specific flight routes and time intervals in complex terrain areas with their flexible flight characteristics. By accurately identifying and analyzing these image sequences, high-precision topographic data can be quickly obtained, providing basic geographical information support for urban planning, road construction, water conservancy projects, etc.

[0003] In recent years, although machine learning has achieved remarkable success in identifying UAV aerial image sequences, there are still many challenges in developing sufficiently robust and reliable identification methods. In previous related studies, the identification model only outputs definite prediction results but does not provide the confidence interval of the predicted value, making it difficult to reflect the confidence level and quality level of the identification result. Therefore, it is necessary to perform uncertainty modeling on the image sequence identification model, measure and characterize the uncertainty of the model output, and clarify the confidence level of the identification result, which has important value for improving the stability and reliability of the identification. For this purpose, some distribution forms, such as Gaussian distribution, Dirichlet distribution, Softmax distribution, etc., are often used to model the output of the image sequence identification model and define uncertainty on these outputs. It should be noted that most of the existing related methods assume that the observed data are independent of each other, but in an image sequence, any image generally has a high degree of correlation with its previous and subsequent images. Therefore, the existing uncertainty estimation methods are difficult to accurately describe the information in the image sequence.

[0004] Therefore, the present invention establishes an uncertainty modeling system for identifying UAV aerial image sequences, which fully considers the temporal correlation to estimate the output uncertainty of the identification model. Summary of the Invention

[0005] The present invention aims to design an uncertainty modeling method for the recognition of UAV aerial photography image sequences, fully considering the temporal correlation to estimate the output uncertainty of the image sequence recognition model.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] An uncertainty modeling system for the recognition of UAV aerial photography image sequences, the system includes an image sequence input module, a UAV image analysis module, a temporal-based uncertainty evaluation module, and an output module;

[0008] The image sequence input module includes a primary recognition model for UAV aerial photography image sequences;

[0009] The UAV image analysis module integrates a Gaussian mixture model into the primary recognition model of the UAV aerial photography image sequence in the image sequence input module, and uses the Gaussian mixture model to analyze each image and estimate the probability of the image recognition output;

[0010] The temporal-based uncertainty evaluation module includes an initial estimation module and a temporal optimization module; the initial estimation module uses the variational Bayesian inference method to solve the Gaussian mixture model and estimate the initial variational conditional probability distribution; the temporal optimization module introduces the image sequence correlation and improves the variational Bayesian inference method, and estimates the final variational conditional probability distribution by optimizing the temporal correlation of the previous and subsequent images of each image;

[0011] The output module outputs the uncertainty expression of the UAV aerial photography image sequence recognition according to the final variational conditional probability distribution.

[0012] Further, the primary recognition model of the UAV aerial photography image sequence in the image sequence input module can be regarded as a structure composed of a feature extractor and a predictor in cascade, and a sequence of images containing pictures taken by the UAV is used as the input of the model.

[0013] Further, the construction of the UAV image analysis module includes:

[0014] Step SA1: Receive the image features processed by the primary recognition model from the image sequence input module;

[0015] Step SA2: Analyze the input image features using the Gaussian mixture model.

[0016] Further, the construction of the UAV image analysis module also includes:

[0017] Step SA3: After feature analysis, calculate the probability that the image belongs to each category at the current time node;

[0018] Step SA4: Judging the ground object category according to the probability and converting the probability value into an uncertainty index;

[0019] Step SA5: Outputting the calculated category probability and uncertainty index to the time-series based uncertainty evaluation module to provide inputs for subsequent time-series optimization and uncertainty modeling.

[0020] Furthermore, the UAV image analysis module estimates the probability output by image recognition, that is, calculates the probability density function:

[0021]

[0022] Wherein, represents the feature generated by the feature extractor h i , represents the vector form of the i-th image in the input image sequence, G represents the number of category recognition tasks, ω = [ω 1 , ω 2 , …, ω G T represents the non-negative mixing weights of G Gaussian mixture models, and ω G can be directly determined by the proportion of each type of sample contained in the training set. N(z i ; μ gt , Σ gt ) is a Gaussian distribution, and T is the number of components of each Gaussian mixture model; μ = {μ gt}, C ov = {COV gt} and G = {η gt} respectively represent the parameter sets of the mean, covariance, and mixing weights.

[0023] Furthermore, the operation process of the initial estimation module includes:

[0024] Step SB1: Defining θ = {μ, C ov} as the parameter set; D = {X, Y} as the data set, where represents the feature matrix of the image sequence, represents the label vector of the image sequence;

[0025] Step SB2: Defining as the latent variable matrix. Among them, if the i-th image feature in the UAV aerial image sequence is associated with the k-th mixing component, then f i [k] = 1, otherwise f i [k] = 0;

[0026] ​Step SB3: Use variational Bayesian inference to solve the mixture model (1) to maximize the conditional probability distribution of the data given the parameters:

[0027]

[0028] Furthermore, the operation process of the initial estimation module further includes:

[0029] Step SB4: Use the variational distribution Q to approximate the original distribution P. Among them, the variational distribution Q belongs to the set of variational distributions, and the difficult-to-directly-compute conditional probability distribution is approximated by minimizing the KL divergence;

[0030] Step SB5: According to the variational Bayesian inference theory, the log-likelihood of Q(D|θ) can be decomposed into:

[0031]

[0032] where ELBO represents the expected log-likelihood, and KL(Q(F)||P(F|D,θ)) represents the KL divergence from Q(F) to the prior distribution P(F|D,θ), expressed in the following form:

[0033]

[0034] Step SB6: According to the variational Bayesian inference theory, the two parameters F and θ can be optimized, and the initial optimal variational conditional probability distribution Q * (F) can be determined.

[0035] Furthermore, there is generally a correlation between the previous image and the subsequent image of any image in the UAV aerial image sequence. The temporal sequence optimization module introduces the image sequence correlation and takes the correlation into account when using the variational Bayesian inference method to solve Equation (2). The optimal variational distribution Q * (f) can be expressed as:

[0036] Q * (f i ) = Q(f i |f i-1 , f i+1 ) (6)

[0037] where f i represents the i-th hidden variable in the hidden variable matrix F; the above equation is equivalent to:

[0038]

[0039] According to the conditional probability conversion formula, Equation (7) can be further written as:

[0040]

[0041] Among them, k' and k" respectively represent the (i - 1)-th and (i + 1)-th images in the image sequence, that is, the mixing components of the maximum response degrees of the pre-image and the post-image of the i-th image.

[0042] Furthermore, when the temporal optimization module fuses the correlation of the image sequence, it is necessary to consider the information of the mixing components of the i-th image and the (i - 1)-th and (i + 1)-th images in the sequence, and the following definitions can be made:

[0043]

[0044] Among them, and represent the ratios of the k-th, k'-th, and k"-th mixing components. In addition, the following definitions are also available:

[0045]

[0046] Among them, and represent the ratios of the k'-th and k"-th mixing components.

[0047] Furthermore, the final variational conditional probability distribution of the temporal optimization module needs to substitute Equation (9) and Equation (10) into Equation (8) to obtain the expression form of, that is, the final variational distribution Q * (f i ) as the expression form of the uncertainty of the recognition output of the aerial image sequence.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] (1) An uncertainty modeling system for unmanned aerial vehicle (UAV) aerial image sequence recognition provided by the present invention establishes an uncertainty modeling system for UAV aerial image sequence recognition, and fully considers the temporal correlation to estimate the output uncertainty of the recognition model.

[0050] (2) An uncertainty modeling system for UAV aerial image sequence recognition provided by the present invention uses a Gaussian mixture model to represent the probability of features, and uses the temporal correlation in the image sequence to improve the variational Bayesian inference method to obtain the optimal variational conditional probability distribution, improving the adaptability of the model to the dynamic changes of the image sequence.

[0051] (3) An uncertainty modeling system for UAV aerial image sequence recognition provided by the present invention optimizes the variational Bayesian inference technology by considering the correlation between image sequences, obtains the final variational conditional probability distribution, and improves the classification accuracy. Description of the Drawings

[0052] Figure 1 This is the system structure diagram of the present invention;

[0053] Figure 2 This is the construction flow chart of the image analysis module for identifying the aerial photography image sequence of the drone in the present invention;

[0054] Figure 3 This is the operation flow chart of the initial estimation module for identifying the aerial photography image sequence of the drone in the present invention. Detailed implementation manners

[0055] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0056] This embodiment relates to an uncertainty modeling system for identifying an aerial photography image sequence of a drone, which uses a Gaussian mixture model to represent the probability of features, and uses the temporal correlation in the image sequence to improve the variational Bayesian inference method to obtain the optimal variational conditional probability distribution, so as to realize the uncertainty probability distribution of the identification result of the aerial photography image sequence of the drone.

[0057] An uncertainty modeling system for identifying an aerial photography image sequence of a drone, the system includes an image sequence input module, a drone image analysis module, a temporal-based uncertainty evaluation module, and an output module, as Figure 1 shown;

[0058] The image sequence input module includes a primary recognition model for the aerial photography image sequence of the drone;

[0059] The drone image analysis module integrates a Gaussian mixture model into the primary recognition model of the aerial photography image sequence in the image sequence input module, and uses the Gaussian mixture model to analyze each image and estimate the probability of the recognition output of the image;

[0060] The temporal-based uncertainty evaluation module includes an initial estimation module and a temporal optimization module; the initial estimation module uses the variational Bayesian inference method to solve the Gaussian mixture model and estimate the initial variational conditional probability distribution; the temporal optimization module introduces the image sequence correlation and improves the variational Bayesian inference method, and estimates the final variational conditional probability distribution by optimizing the temporal correlation of the previous and subsequent images of each image;

[0061] The output module outputs the uncertainty expression of the identification of the aerial photography image sequence of the drone according to the final variational conditional probability distribution.

[0062] Further, the primary recognition model of the UAV aerial photography image sequence in the image sequence input module can be regarded as a structure cascaded by a feature extractor and a predictor, and a sequence of images containing [number of images] captured by the UAV is used as the input of the model.

[0063] Further, the construction of the UAV image analysis module is as Figure 2 shown, including:

[0064] Step SA1: Receive the image features processed by the primary recognition model from the image sequence input module;

[0065] Step SA2: Analyze the input image features using the Gaussian mixture model.

[0066] Further, the construction of the UAV image analysis module also includes:

[0067] Step SA3: After feature analysis, calculate the probability that the image belongs to each category at the current time node;

[0068] Step SA4: Judge the ground object category based on the probability and convert the probability value into an uncertainty index;

[0069] Step SA5: Output the calculated category probability and uncertainty index to the time-series based uncertainty evaluation module to provide input for subsequent time-series optimization and uncertainty modeling.

[0070] Further, the UAV image analysis module estimates the probability of the image recognition output, that is, calculates the probability density function:

[0071]

[0072] where, represents the feature generated by the feature extractor for h i , represents the vector form of the i-th image in the input image sequence, G represents the number of category recognition tasks, ω = [ω 1 , ω 2 , …, ω G T represents the non-negative mixing weights of G Gaussian mixture models, and ω G The value of can be directly determined by the proportion of each type of sample contained in the training set. N(z i ; μ gt , Σ gt ) is the Gaussian distribution, T is the number of components of each Gaussian mixture model; μ = {μ gt}, C ov = {COV gt} and G = {ηgt} represent the parameter sets of the mean, covariance, and mixing weights respectively.

[0073] Furthermore, the operation process of the initial estimation module includes:

[0074] Step SB1: Define θ = {μ, C ov} as the parameter set; D = {X, Y} as the data set, where represents the feature matrix of the image sequence, represents the label vector of the image sequence;

[0075] Step SB2: Define as the latent variable matrix, where for the i-th image feature and the k-th mixture component in the UAV aerial image sequence, f i [k] = 1, otherwise f i [k] = 0;

[0076] Step SB3: Use variational Bayesian inference to solve the mixture model (1) to maximize the conditional probability distribution of the data given the parameters:

[0077]

[0078] Furthermore, the operation process of the initial estimation module also includes:

[0079] Step SB4: Use the variational distribution Q to approximate the original distribution P, where the variational distribution Q belongs to the set of variational distributions, and approximate the conditional probability distribution that is difficult to calculate directly by minimizing the KL divergence;

[0080] Step SB5: According to the variational Bayesian inference theory, the log-likelihood of Q(D|θ) can be decomposed as:

[0081]

[0082] where ELBO represents the expected log-likelihood, and KL(Q(F)||P(F|D,θ)) represents the KL divergence from Q(F) to the prior distribution P(F|D,θ), which is expressed in the following form:

[0083]

[0084] Step SB6: According to the variational Bayesian inference theory, the two parameters F and θ can be optimized, and the representation form of the initial optimal variational conditional probability distribution Q * (F) can be determined.

[0085] Furthermore, there is generally a correlation between the previous image and the subsequent image of any image in the UAV aerial image sequence. The temporal optimization module introduces the image sequence correlation and takes the correlation into account when using the variational Bayesian inference method to solve Equation (2) for the optimal variational distribution Q * (f) can be expressed as:

[0086] Q * (f i ) = Q(f i |f i-1 , f i+1 ) (6)

[0087] where f i represents the i-th latent variable in the latent variable matrix F; the above equation is equivalent to:

[0088]

[0089] According to the conditional probability conversion formula, Equation (7) can be further written as:

[0090]

[0091] where k′ and k″ respectively represent the i - 1-th and i + 1-th images in the image sequence, that is, the mixed components with the maximum response of the previous image and the subsequent image of the i-th image.

[0092] Furthermore, the temporal optimization module fuses the image sequence correlation and needs to consider the information of the mixed components of the i-th image and the i - 1-th and i + 1-th images in the sequence. The following definitions can be made:

[0093]

[0094] where and represent the ratios of the k-th, k′-th, and k″-th mixed components. Additionally, the following definitions are made:

[0095]

[0096] where and represent the ratios of the k′-th and k″-th mixed components.

[0097] Furthermore, the final variational conditional probability distribution of the temporal optimization module needs to substitute Equation (9) and Equation (10) into Equation (8) to obtain the expression form of P(f i [k] = 1), that is, the final variational distribution Q * (f i ) is used as the uncertainty expression form of the UAV aerial image sequence recognition output.

[0098] The present invention proposes to use a Gaussian mixture model to represent the probability of features, and utilize the temporal correlation in the image sequence to improve the variational Bayesian inference method to obtain the optimal variational conditional probability distribution, so as to realize the uncertainty probability distribution of the recognition results of UAV aerial image sequences.

[0099] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above embodiments, and also include technical solutions composed of any combination of the above technical features. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. An uncertainty modeling system for unmanned aerial image sequence recognition, characterized in that: The system includes an image sequence input module, a UAV image analysis module, a time sequence-based uncertainty assessment module, and an output module; The image sequence input module includes a primary recognition model for drone aerial image sequences; The drone image analysis module integrates a Gaussian mixture model into the primary recognition model of the drone aerial image sequence in the image sequence input module, and uses the Gaussian mixture model to analyze each image and estimate the probability of the image recognition output; The timing-based uncertainty assessment module includes an initial estimation module and a timing optimization module; The initial estimation module uses a variational Bayesian inference method to solve the Gaussian mixture model and estimate the initial variational conditional probability distribution; the timing optimization module introduces image sequence correlation, improves the variational Bayesian inference method, and estimates the final variational conditional probability distribution by optimizing the timing correlation between the preceding and succeeding images of each image; The output module outputs the uncertainty expression of the recognition of the UAV aerial image sequence according to the final variational conditional probability distribution.

2. The uncertainty modeling system for unmanned aerial image sequence recognition according to claim 1 is characterized in that: The primary recognition model in the image drone aerial image sequence input module can be regarded as a structure cascaded by a feature extractor and a predictor, and a sequence of images taken by the drone is used as the input of the model.

3. The uncertainty modeling system for unmanned aerial image sequence recognition according to claim 1, characterized in that: The construction of the drone image analysis module includes: Step SA1, receiving image features processed by the primary recognition model from the image sequence input module; Step SA2: Analyze the input image features using a Gaussian mixture model.

4. The uncertainty modeling system for unmanned aerial image sequence recognition according to claim 1, characterized in that: The construction of the drone image analysis module also includes: Step SA3: After feature analysis, calculate the probability that the image belongs to each category at the current time node; Step SA4: determine the type of the ground feature based on the probability, and convert the probability value into an uncertainty index; Step SA5: Output the calculated category probability and uncertainty index to the time series-based uncertainty assessment module to provide input for subsequent time series optimization and uncertainty modeling.

5. The uncertainty modeling system for identifying drone aerial image sequences according to claim 1, characterized in that: The drone image analysis module estimates the probability of image recognition output, that is, calculates the probability density function: in, Indicates h i Features generated by the feature extractor, represents the vector form of the i-th image in the input image sequence, G represents the number of category recognition tasks ω=[ω1,ω2,…,ω G ] T represents the non-negative mixing weights of G Gaussian mixture models, and ω G The value of can be directly determined by the proportion of samples of each category contained in the training set, N(z i ;μ gt ,Σ gt ) is a Gaussian distribution, T is the number of components in each Gaussian mixture model; μ = {μ gt }、C ov ={COV gt } and G={η gt } respectively represent the parameter sets of mean, covariance and mixing weight.

6. The uncertainty modeling system for unmanned aerial image sequence recognition according to claim 1, characterized in that: The operation process of the initial estimation module includes: Step SB1, define θ={μ,C ov } is the parameter set; D = {X, Y} is the data set, where Represents the feature matrix of the image sequence, A label vector representing the image sequence; Step SB2: Definition is a latent variable matrix, where the i-th image feature in the UAV aerial image sequence is associated with the k-th mixed component, then otherwise Step SB3: Use variational Bayesian inference to solve the mixed model (1) to maximize the conditional probability distribution of the data on the given parameters:

7. The uncertainty modeling system for unmanned aerial image sequence recognition according to claim 1, characterized in that: The operation process of the initial estimation module also includes: Step SB4: Use variational distribution Q to approximate the original distribution P, where variational distribution Q belongs to the variational distribution set, and approximates the conditional probability distribution that is difficult to directly calculate by minimizing KL divergence; Step SB5: According to variational Bayesian inference theory, the log-likelihood of Q(D|θ) can be decomposed into: Among them, ELBO represents the expected log likelihood, KL(Q(F)||P(F|D,θ)) represents the KL divergence from Q(F) to the prior distribution P(F|D,θ), which is expressed as follows: Step SB6: According to the variational Bayesian inference theory, the two parameters F and θ can be optimized and the initial optimal variational conditional probability distribution Q can be determined. * (F) representation.

8. The uncertainty modeling system for unmanned aerial image sequence recognition according to claim 1, characterized in that: There is generally a correlation between the previous and subsequent images of any image in the drone aerial image sequence. The temporal optimization module introduces the correlation of the image sequence. When the variational Bayesian inference method is used to solve equation (2), the consideration of the correlation is increased. The optimal variational distribution Q * (f) can be expressed as: Q * (f i )=Q(f i |f i-1 ,f i+1 )(6) Among them, f i represents the i-th latent variable in the latent variable matrix F; the above formula is equivalent to: According to the conditional probability conversion formula, formula (7) can be further written as: Among them, k ′ , k″ respectively represent the mixed components of the maximum response degrees of the i-1th and i+1th images in the image sequence, that is, the preceding image and the succeeding image of the i-th image.

9. The uncertainty modeling system for unmanned aerial image sequence recognition according to claim 1, characterized in that: The temporal optimization module fuses the correlation of the image sequence and needs to consider the information of the mixed components of the i-th image and the i-1th and i+1th images in the sequence, which can be defined as follows: in, and Indicates the kth and kth ′ The ratio of k″ mixed components is defined as follows: in, and Indicates the kth ′ and the ratio of k″ mixed components.

10. The uncertainty modeling system for unmanned aerial image sequence recognition according to claim 1, characterized in that: The final variational conditional probability distribution of the timing optimization module needs to be substituted into equation (8) by equation (9) and equation (10) to obtain P(f i [k] =1), that is, the final variational distribution Q * (f i ) is used as the uncertainty expression of the aerial image sequence recognition output.

Images