A target detection method for unmanned sanitation vehicles based on decoupled variational autoencoder network

Abstract

The present invention discloses a target detection method for an unmanned sanitation vehicle based on a decoupled variational autoencoder network. For multimodal target detection of unmanned sanitation vehicles, firstly, lateral connections are added to a feature pyramid network to balance the detection requirements of objects of different sizes; then, multi-scale features are input into a box suggestion module to generate a proposed box and a proposed feature, and a one-to-one interactive relationship is established between the ROI feature obtained by RoiAlign and the proposed feature, so as to optimize the ROI feature; then, a decoupled variational autoencoder is designed, and variational inference is adopted to model the global data distribution by maximizing the evidence lower bound of the log-likelihood of the true data distribution, and the data distribution of infrared and visible light images is decoupled to capture the distribution information related to the task from the data; finally, the features of the decoupled variational autoencoder are globally integrated and then input into a target detection network to perform multimodal target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and mainly to a target detection method based on a decoupled variational autoencoder network. Background Art

[0002] In unmanned sanitation vehicles, on-board target detection is a crucial task. It not only ensures the safety of vehicle driving, but also helps sanitation vehicles plan the optimal cleaning route, improve work efficiency, and reduce resource waste. At present, target detection methods based on visible light images have made significant progress, but these methods are mainly suitable for environments with good lighting conditions and low noise interference. However, in actual environments, unmanned sanitation vehicles often need to operate in complex scenarios such as weather changes, smoke interference, occlusion, and scale changes. This requires the target detection model to be able to cope with more complex situations to ensure the normal operation of sanitation vehicles.

[0003] Traditional vehicle-mounted target detection methods usually rely on a single visible light image, but the performance of visible light images is often unsatisfactory in low light, rainy and snowy weather, or strong light exposure. Therefore, combining the thermal radiation information of infrared images has become an effective solution. However, infrared images often cannot clearly show the detailed texture information of the target, which brings certain challenges to target recognition. The present invention aims to combine the detailed texture information reflecting the semantics of the target category in the visible light image with the saliency information revealing the target contour in the infrared image to improve the adaptability and accuracy of the model in complex environments.

[0004] In recent years, more and more researchers have tried to integrate the multimodal information of visible light images and infrared images through feature-level fusion. Although multimodal feature fusion methods have made significant progress, there are still two major challenges that limit the further improvement of model performance: task adaptability challenge and modality heterogeneity challenge. Multimodal target detection involves two main tasks: target localization and target recognition. Target localization emphasizes the recognition of spatial features of each target, while target recognition requires that targets of the same type exhibit consistent features. In addition, due to the differences in imaging mechanisms, there is modality heterogeneity between visible light images and infrared images. Summary of the invention

[0005] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, the present invention provides a target detection method based on a decoupled variational autoencoder network. First, in order to meet the challenge of task adaptability, the present invention decouples the class distribution and spatial distribution of the two modalities from the original data distribution through a decoupled variational autoencoder, and then specifically captures the task-related distribution information from the multimodal data. Then, in order to meet the challenge of modal heterogeneity, the present invention designs global information integration, and promotes cross-modal knowledge sharing by increasing the amount of shared information between modalities.

[0006] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0007] The present invention provides a target detection method based on a decoupled variational autoencoder network, comprising the following steps:

[0008] Step 1: Input the collected visible light image and infrared image into a shared backbone network, extract the feature sequence of the visible light image and the feature sequence of the infrared image respectively, and then input them into a feature pyramid network to fuse the features of different layers in the feature sequence to obtain multi-scale features of the visible light image and multi-scale features of the infrared image;

[0009] Step 2: The multi-scale features of the visible light image and the multi-scale features of the infrared image are input into the box proposal module to generate learnable proposal boxes A and proposal features F. p , use RoiAlign to process the proposal box A and multi-scale features to obtain the ROI feature x0, and propose the feature F p Establish a one-to-one interactive relationship with the ROI feature x0, optimize the ROI feature x0, and obtain the optimized ROI feature x;

[0010] Step 3: Use the decoupled variational autoencoder to decouple the Roi feature x into a class latent variable z related to the target recognition task c The spatial latent variable z related to the target positioning task s ;

[0011] Step 4: transform the class latent variables z of the two modes c Perform global information integration to obtain the fused class latent variables The spatial latent variables z of the two modes s Perform global information integration to obtain the fused spatial latent variables Class latent variables and spatial latent variables Input into the target detection network for target detection.

[0012] Furthermore, the specific steps of step 1 are as follows:

[0013] Step 1.1: Input the collected visible light image and infrared image into the shared backbone network to extract the feature sequence of the visible light image and the feature sequence of the infrared image respectively. The feature sequence is represented as {C1, C2, C3, C4, C5}, where C i represents the features of the output of the i-th layer, i=1, 2, 3, 4, 5; the shared backbone network is Resnet-50;

[0014] Step 1.2, the feature sequence of the visible light image and the feature sequence of the infrared image are input into the feature pyramid network respectively, and a horizontal connection is added to the feature pyramid network to fuse the features of different layers in the feature sequence to obtain the multi-scale features of the visible light image and the multi-scale features of the infrared image. The multi-scale feature P = {P1, P2, P3, P4} is expressed as:

[0015]

[0016] Among them, Conv(·) represents convolution and Upsample(·) represents upsampling.

[0017] Furthermore, in step 2, RoiAlign is used to process the proposal box A and the multi-scale features to obtain the ROI feature x0, which is expressed as:

[0018] x0=RoiAlign(P,A)

[0019] The proposed feature F p A one-to-one interaction relationship is established between the ROI feature x0 and the ROI feature x0 is optimized to capture the spatial attributes of the target. The optimized ROI feature x is expressed as:

[0020] x=IIH(x0,F p )

[0021] Among them, IIH(·) is the instance interaction head; in the instance interaction head, each ROI feature is enhanced by the proposal feature;

[0022] x={x V ,x I},in Represents the ROI features of the visible light image, Represents the ROI features of the infrared image.

[0023] Furthermore, in step 3, a decoupled variational autoencoder is used to decouple the Roi feature x into a class latent variable z related to the target recognition task. c The spatial latent variable z related to the target positioning task s Specifically, by adopting the variational inference method, the global data distribution is modeled by maximizing the evidence lower bound of the decoupled variational autoencoder, and then the class latent variable z is obtained c and spatial latent variables z s :

[0024] The evidence lower bound logp(x) of the decoupled variational autoencoder is expressed as:

[0025]

[0026] In the above formula, the first term is the reconstruction error, which is calculated based on the class posterior distribution and the spatial posterior distribution. q (x|z c ,z s ) to find the expectation, that is, the weighted average, where p q is a decoder parameterized by q that reconstructs x, and the second term aims to transform the class posterior distribution and class prior distribution p(z c ), the third term aims to transform the spatial posterior distribution q φ (z s |x) and the spatial prior distribution p(z s ) matches;

[0027] In order to optimize the lower bound of evidence, we need to maximize the expectation minimize and D KL (q φ (z s |x)||p(z s )).

[0028] Furthermore, the loss of the decoupled variational autoencoder is expressed as:

[0029]

[0030] Among them, L rec represents the reconstruction loss, represents the class posterior distribution and class prior distribution p(z c ), Denotes the spatial posterior distribution q φ (z s |x) and the spatial prior distribution p(z s ) is the KL divergence loss between .

[0031] Furthermore, the maximization expectation That is, to minimize the given class hidden variable z c and spatial latent variables z c Reconstruct ROI features under the condition of minimizing the difference between the reconstructed ROI features and the input ROI features; by minimizing the reconstruction loss L rec accomplish:

[0032]

[0033] Where MSE(·) represents mean square error, Represents the reconstructed ROI features.

[0034] Furthermore, we minimize the class posterior distribution and class prior distribution p(zc ) Here are the steps:

[0035] First, the class encoder is used to predict the mean matrix and standard deviation matrix of the class posterior distribution:

[0036]

[0037] in, and are the class mean vector and class standard deviation vector of x respectively, K is the vector dimension, is the class distribution encoder, are encoder parameters.

[0038] Then, ε is sampled from the Gaussian distribution Ν(0,I) and the Dirichlet distribution is approximated by the following formula to obtain the class latent variable z c :

[0039]

[0040] Among them, Σ c =diag(σ c ), indicating that it contains σ c Diagonal matrix with main diagonal elements.

[0041] Next, to optimize the class posterior distribution of the class latent variable and class prior distribution p(z c ), following the Laplace approximation, rewrite the standard Dirichlet prior distribution Dir(α) as 1 , the covariance matrix is ​​a Gaussian distribution with Σ1.

[0042] Finally, after approximating the standard Dirichlet distribution to a Gaussian distribution, the posterior distribution of the class and class prior distribution p(z c ) is calculated, and the calculated class posterior distribution is and class prior distribution p(z c ) as the KL divergence loss

[0043] Furthermore, we minimize the spatial posterior distribution q φ (z s |x) and the spatial prior distribution p(z s ) KL (q φ (z s |x)||p(z s )), the steps are as follows:

[0044] First, the spatial encoder is used to predict the mean matrix and standard deviation matrix of the spatial posterior distribution:

[0045]

[0046] in, is the spatial encoder, φ is the encoder parameter, and are the spatial mean vector and spatial standard deviation vector of x respectively.

[0047] Then, ε is sampled from the Gaussian distribution Ν(0,I) and reparameterized to obtain the spatial latent variable z s :

[0048] z s =μ s +σ s (ε)

[0049] The spatial KL divergence loss between the spatial posterior distribution and the spatial prior distribution is expressed as:

[0050]

[0051] Here, KL(·) represents the KL divergence.

[0052] Furthermore, in step 4, the class latent variables z of the two modes are c Perform global information integration to obtain the fused class latent variables The specific steps are as follows:

[0053] For the two-modal class latent variables represents the class latent variable of visible light, represents the class latent variable of infrared;

[0054] From the joint distribution and marginal distribution Positive sample pairs and negative sample pairs are sampled, and the discriminator is used to distinguish the positive sample pairs from the negative sample pairs. In order to allow the discriminator to accurately distinguish these two types of sample pairs, the joint distribution is maximized. and marginal distribution The distribution difference of and The information integration obtains the fused class latent variable

[0055] In the same way, the spatial latent variables z of the two modes are s Perform global information integration to obtain the fused spatial latent variables

[0056] For the two-modal spatial latent variables represents the spatial latent variable of visible light, Represents the spatial latent variable of infrared; respectively, from the joint distribution and marginal distribution Positive sample pairs and negative sample pairs are sampled, and the discriminator is used to distinguish the positive sample pairs from the negative sample pairs. In order to allow the discriminator to accurately distinguish these two sample pairs, the joint distribution is maximized. and marginal distribution The distribution difference of and The information integration obtains the fused class latent variable

[0057] The optimization loss of the discriminator is expressed as:

[0058]

[0059] Where σ(·) is the sigmoid activation function, D(·) represents the discriminator, (α, β) is a positive sample pair sampled from the joint distribution p(α, β), and (α, β) represents or is a pair of negative samples sampled from the marginal distribution p(α)p(β).

[0060] Furthermore, in step 4, the class latent variable and spatial latent variables Input into the target detection network for target detection, expressed as:

[0061]

[0062]

[0063] in, is the target recognition head, It is the target positioning head. It is used to predict the category and location of the target based on multimodal fusion features. and Represent category prediction and position prediction respectively.

[0064] Beneficial Effects

[0065] The present invention provides a target detection method based on a decoupled variational autoencoder network. In order to solve the challenge of task adaptability, the present invention adopts a decoupled variational autoencoder method to capture task-related distribution information in multimodal data in a targeted manner by decoupling the class distribution and spatial distribution of the two modalities from the original data. The decoupled variational autoencoder can effectively separate different factors in the data, so that the model can better understand the relationship between different modalities and extract useful information for target detection tasks. At the same time, in order to cope with the challenge of modal heterogeneity, the present invention introduces global information integration. This technology aims to increase the amount of shared information between modalities, thereby promoting the sharing and learning of cross-modal knowledge. By establishing a closer connection between modalities, the differences between different modalities can be effectively compensated, and the adaptability and robustness of the multimodal target detection system for complex scenes can be improved, especially in the target monitoring of unmanned sanitation vehicles, which has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a principle block diagram of the method of the present invention. DETAILED DESCRIPTION

[0067] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] The present invention provides an unmanned sanitation vehicle target detection method based on a decoupled variational autoencoder network. The specific principle is as follows: Figure 1 shown.

[0069] Step 1: Input the collected visible light image and infrared image into the shared backbone network to extract the feature sequence of the visible light image and the feature sequence of the infrared image respectively. The feature sequence is represented as {C1, C2, C3, C4, C5}, where C i Represents the characteristics of the output of the i-th layer. The shared backbone network has 5 layers in total. The shared backbone network of the present invention adopts Resnet-50.

[0070] The feature sequences of visible light images and infrared images are respectively input into the feature pyramid network, and lateral connections are added to the feature pyramid network to fuse features of different layers. Deep and shallow features are combined by convolution and upsampling to balance the detection requirements of objects of different sizes, and multi-scale features of visible light images and infrared images are obtained. The multi-scale features P = {P1, P2, P3, P4} are expressed as;

[0071]

[0072] Among them, Conv(·) represents convolution and Upsample(·) represents upsampling.

[0073] Step 2: The multi-scale features of the visible light image and the multi-scale features of the infrared image are input into the box proposal module, which generates a learnable proposal box A and proposal feature F. p , use RoiAlign to process the proposal box and multi-scale features to obtain ROI features, and propose features F p A one-to-one interaction relationship is established between the ROI features to further optimize the original ROI features. The specific steps include:

[0074] Specifically, first, N learnable proposal boxes are generated through the box proposal module and the proposed feature F p , input the proposal box and multi-scale features, and use RoiAlign to get the ROI features:

[0075] x0=RoiAlign(P,A)

[0076] Get the original ROI feature x0 of the area corresponding to the proposal box.

[0077] Then, by proposing feature F p A one-to-one interaction relationship is established between the ROI feature x0 and the original ROI feature x0 to further optimize the original ROI feature x0 to capture the spatial properties of the target. The optimized ROI feature x is:

[0078] x=IIH(x0,F p )

[0079] Among them, IIH(·) is the instance interaction head. In the instance interaction head, each ROI feature is enhanced by the proposal feature.

[0080] x={x V ,x I},in Represents the ROI features of the visible light image, Represents the ROI features of the infrared image.

[0081] Step 3: Use the decoupled variational autoencoder VAE to decouple the Roi feature x into a class latent variable z related to the target recognition task c The spatial latent variable z related to the target positioning task s ; Specifically: adopt the variational inference method, model the global data distribution by maximizing the evidence lower bound of the log-likelihood of the true data distribution, and design a decoupled variational autoencoder to capture task-related distribution information from the data;

[0082] Then, the prior distribution is customized for the target recognition task and the target localization task, and then the class distribution and spatial distribution that meet the task characteristics are modeled. The specific process of this part is introduced below:

[0083] Step 3.1: Decouple the evidence lower bound of the variational autoencoder, that is, solve the evidence lower bound that maximizes the log-likelihood of the true data distribution.

[0084] In the present invention, the decoupled variational autoencoder decouples the Roi feature x into a class latent variable z related to the target recognition task c The spatial latent variable z related to the target positioning task s Therefore, the evidence lower bound logp(x) of the decoupled variational autoencoder can be expressed as:

[0085]

[0086] In the above formula, the evidence lower bound ELBO includes three terms. The first term is the reconstruction error, which is calculated based on the class posterior distribution and the spatial posterior distribution of p. q (x|z c ,z s ) to find the expectation, that is, the weighted average, where p q is a decoder parameterized by q that reconstructs x, and the second term aims to transform the class posterior distribution and class prior distribution p(z c ), the third term aims to transform the spatial posterior distribution q φ (z s |x) and the spatial prior distribution p(z s ) matches.

[0087] Step 3.2, the evidence lower bound formula of the decoupled variational autoencoder optimization.

[0088] In order to optimize the lower bound of evidence, the present invention needs to maximize the expectation That is, to minimize the given class hidden variable z c and spatial latent variables z c Reconstruct ROI features under the condition of and minimize the difference between the reconstructed ROI features and the input ROI features. This goal can be achieved by minimizing the following reconstruction loss:

[0089]

[0090] Where MSE(·) represents mean square error, Represents the reconstructed ROI features.

[0091] Step 3.3, Loss. In order to optimize the lower bound of evidence, the present invention needs to minimize the class posterior distribution and class prior distribution p(z c ) is the KL divergence between .

[0092] First, the class encoder is used to predict the mean matrix and standard deviation matrix of the class posterior distribution:

[0093]

[0094] in, and are the class mean vector and class standard deviation vector of x respectively, K is the vector dimension, is the class distribution encoder, are encoder parameters.

[0095] Then, ε is sampled from the Gaussian distribution Ν(0,I) and the Dirichlet distribution is approximated by the following formula to obtain the class latent variable z c :

[0096]

[0097] Among them, Σ c =diag(σ c ), indicating that it contains σ c Diagonal matrix with main diagonal elements.

[0098] Next, to optimize the class posterior distribution of the class latent variable and class prior distribution p(z c ), following the Laplace approximation, rewrite the standard Dirichlet prior distribution Dir(α) as 1 , the covariance matrix is ​​a Gaussian distribution with Σ1.

[0099] Finally, after approximating the standard Dirichlet distribution to a Gaussian distribution, the present invention can classify the posterior distribution and class prior distribution p(z c ) is calculated, and the calculated class posterior distribution is and class prior distribution p(z c ) as the KL divergence loss

[0100] Step 3.4: D KL (q φ (z s |x)||p(z s )) loss. In order to optimize the lower bound of evidence, the present invention needs to minimize the spatial posterior distribution q φ (z s |x) and the spatial prior distribution p(z s ) is the KL divergence between .

[0101] The present invention first uses the spatial encoder to predict the mean matrix and standard deviation matrix of the spatial posterior distribution:

[0102]

[0103] in, is the spatial encoder, φ is the encoder parameter, and are the spatial mean vector and spatial standard deviation vector of x respectively.

[0104] Then, ε is sampled from the Gaussian distribution Ν(0,I) and reparameterized to obtain the spatial latent variable z s :

[0105] z s =μ s +σ s (ε)

[0106] in, represents the spatial latent variable of visible light, Represents the spatial latent variable of infrared.

[0107] The spatial KL divergence loss between the spatial posterior distribution and the spatial prior distribution can be expressed as:

[0108]

[0109] Here, KL(·) represents the KL divergence.

[0110] Step 3.5: Decouple the variational autoencoder loss.

[0111] The loss of the decoupled variational autoencoder can be expressed as:

[0112]

[0113] Among them, L rec represents the reconstruction loss, represents the class posterior distribution and class prior distribution p(z c ), Denotes the spatial posterior distribution q φ (z s |x) and the spatial prior distribution p(z s ) is the KL divergence between .

[0114] Step 4: Globally integrate the features of the decoupled variational autoencoder and then input them into the target detection network for target detection.

[0115] By integrating the category information and spatial information of the infrared and visible light modalities respectively, their category distribution and spatial distribution are made more consistent. Specifically, first, class latent variables and spatial latent variables are sampled from the class distribution and spatial distribution of the two modalities, and positive sample pairs and negative sample pairs are constructed respectively, where the class latent variables encoding the same target information in the two modalities are regarded as class positive sample pairs, and the class latent variables encoding different target information are regarded as class negative sample pairs; the spatial latent variables encoding the same spatial information in the two modalities are regarded as spatial positive sample pairs, and the spatial latent variables encoding different spatial information are regarded as spatial negative sample pairs. Then, the class positive and negative sample pairs and the spatial positive and negative sample pairs are input into the global information analysis module respectively, and the global information of the class and space are integrated respectively, and the class distribution integration loss and the spatial distribution integration loss are obtained.

[0116] Specifically, for variables α and β, in this specific implementation, variables α and β are visible light hidden variables and infrared hidden variables, or visible light space hidden variables and infrared space hidden variables. The global information analysis module samples positive sample pairs and negative sample pairs from the joint distribution p(α, β) and the marginal distribution p(α)p(β), respectively, and uses the discriminator to distinguish between positive sample pairs and negative sample pairs. In order for the discriminator to accurately distinguish these two types of sample pairs, the model needs to maximize the distribution difference between the joint distribution p(α, β) and the marginal distribution p(α)p(β). In this way, the information integration of variables α and β can be achieved. The optimization loss of the discriminator can be expressed as:

[0117]

[0118] Among them, σ(·) is the sigmoid activation function, D(·) represents the discriminator, and (α, β) represents or (α, β) is a positive sample pair sampled from the joint distribution p(α, β), is a pair of negative samples sampled from the marginal distribution p(α)p(β). It provides a basis for cross-modal feature fusion by maximizing the amount of shared information between modalities.

[0119] Realize the class latent variable z related to the target recognition task c The spatial latent variable z related to the target positioning task s The two modes in the image are fused by class distribution and space distribution to obtain the class features related to the target recognition task. Space related to the target positioning task Finally, the features and Input into the target detection network for target detection, obtain the target class prediction and spatial position prediction, and complete the target recognition task and target positioning task:

[0120]

[0121]

[0122] in, is the target recognition head, It is the target positioning head. It is used to predict the category and location of the target based on multimodal fusion features. and Represent category prediction and position prediction respectively.

[0123] The effect of the present invention is further described below in conjunction with a simulation experiment of a specific vehicle-mounted target detection data set:

[0124] Simulation experiment conditions:

[0125] The dataset used is M 3 FD is a dataset widely used for target detection of unmanned sanitation vehicles, which contains 4200 pairs of registered high-resolution infrared and visible light images with a resolution of 1024 × 768. The M3FD dataset contains a total of 34,407 labeled images of target detection objects, covering 6 target categories including people, cars, buses, motorcycles, trucks and traffic lights.

[0126] In the experimental setting, Resnet-50 pre-trained on Imagenet is used as the backbone network. To ensure that the model is fully trained, the training epoch is set to 100, the batch size is 2, the learning rate is set to 0.00025, and AdamW is selected as the optimizer. In all experiments, the mAP@.5 indicator is used to evaluate the detection performance of the present invention and various comparison algorithms, as shown in Table 1.

[0127] Table 1: The existing technology and the present invention in M 3 Comparison results on FD

[0128] category YOLOv5-V YOLOv5-I CAPTM TarDAL The present invention People 80.86 82.83 84.77 83.20 81.72 Car 87.17 87.04 87.91 88.44 88.03 Bus 84.51 84.18 86.35 87.76 89.95 Motorcycle 79.10 71.42 72.83 74.28 77.72 Lamp 68.71 71.02 71.59 71.34 73.44 Truck 67.38 73.61 68.80 77.48 78.87 mAP@.5 77.95 78.35 78.71 80.42 81.64

[0129] It can be seen from Table 1 that the present invention has achieved the best performance among all the methods. Compared with single-modal detection methods such as YOLOv5-I and YOLOv5-V, the multimodal detection method shows superior performance. This is because, in low-light and smoke-blocked complex environments, visible light images are difficult to clearly reflect the target location information and their semantic information is not class discriminative. Although infrared images can present target location information relatively clearly, single thermal radiation information is difficult to fully reflect the category of the target. The multimodal method can combine the advantages of infrared and visible light modalities to effectively capture the target spatial location information and semantic information in various scenarios. Compared with CAPTM, it ignores the feature preference differences between target recognition and target positioning tasks, and it is difficult to handle conflicts between tasks. In addition, CAPTM directly establishes feature interactions between modalities during feature fusion, ignoring the impact of modal heterogeneity on feature fusion, resulting in limited recognition of fused features. Compared with the pixel fusion method TarDAL, pixel-level information fusion may bring redundancy to target detection tasks, and a large amount of information irrelevant to the target detection task may be mixed in the fusion process, thereby interfering with the detection model. The present invention does not adopt the traditional deterministic feature fusion, but integrates multi-modal distribution information through distribution fusion, which has stronger adaptability to noise interference.

[0130] In summary, the present invention reveals the challenges of task adaptability and modality heterogeneity in infrared and visible light multimodal detection, and customizes a new unmanned sanitation vehicle target detection framework to address the above challenges. Experimental results on the dataset show that the present invention achieves advanced performance in the unmanned sanitation vehicle target detection task.

Claims

1. A target detection method based on a decoupled variational autoencoder network, characterized in that: The steps include: Step 1: Input the visible light image and the infrared image into the shared backbone network, extract the feature sequence of the visible light image and the feature sequence of the infrared image respectively, and then input them into the feature pyramid network to fuse the features of different layers in the feature sequence to obtain the multi-scale features of the visible light image and the multi-scale features of the infrared image; Step 2: The multi-scale features of the visible light image and the multi-scale features of the infrared image are input into the box proposal module to generate the proposed box A and the proposed feature F. p , use RoiAlign to process the proposal box A and multi-scale features to obtain the ROI feature x0, and propose the feature F p Establish a one-to-one interactive relationship with the ROI feature x0, optimize the ROI feature x0, and obtain the optimized ROI feature x; Step 3: Use the decoupled variational autoencoder to decouple the Roi feature x into a class latent variable z related to the target recognition task c The spatial latent variable z related to the target positioning task s , including the following steps: Step 3.1, by adopting the variational inference method, we can model the global data distribution by maximizing the evidence lower bound of the decoupled variational autoencoder, and then obtain the class latent variable z c and spatial latent variables z s : The evidence lower bound logp(x) of the decoupled variational autoencoder is expressed as: In the above formula, the first term is the reconstruction error, which is calculated based on the class posterior distribution and the spatial posterior distribution. θ (x|z c ,z s ) to find the expectation, that is, the weighted average, where p θ is a decoder parameterized by θ that reconstructs x, and the second term aims to transform the class posterior distribution and class prior distribution p(z c ), the third term aims to transform the spatial posterior distribution q φ (z s |x) and the spatial prior distribution p(z s ) matches; By maximizing expectations minimize and D KL (q φ (z s |x)||p(z s )) to achieve the optimization of the evidence lower bound; Step 3.2, maximize expectation That is, to minimize the given class hidden variable z c and spatial latent variables z c Reconstruct ROI features under the condition of minimizing the difference between the reconstructed ROI features and the input ROI features; by minimizing the reconstruction loss L rec accomplish: Where MSE(·) represents mean square error, Represents the reconstructed ROI features; Step 3.3, minimize the class posterior distribution and class prior distribution p(z c ) Here are the steps: First, the class encoder is used to predict the mean matrix and standard deviation matrix of the class posterior distribution: in, and are the class mean vector and class standard deviation vector of x respectively, K is the vector dimension, is the class distribution encoder, is the encoder parameter; Then, ε is sampled from the Gaussian distribution Ν(0,I) and the Dirichlet distribution is approximated by the following formula to obtain the class latent variable z c : Among them, Σ c =diag(σ c ), indicating that it contains σ c Diagonal matrix with main diagonal elements; Next, to optimize the class posterior distribution of the class latent variable and class prior distribution p(z c ), following the Laplace approximation, rewrite the standard Dirichlet prior distribution Dir(α) as 1 , the covariance matrix is ​​a Gaussian distribution of Σ1; Finally, after approximating the standard Dirichlet distribution to a Gaussian distribution, the posterior distribution of the class and class prior distribution p(z c ) is calculated, and the calculated class posterior distribution is and class prior distribution p(z c ) as the KL divergence loss Step 3.4, minimize the spatial posterior distribution q φ (z s |x) and the spatial prior distribution p(z s ) KL (q φ (z s |x)||p(z s )), the steps are as follows: First, the spatial encoder is used to predict the mean matrix and standard deviation matrix of the spatial posterior distribution: in, is the spatial encoder, φ is the encoder parameter, and are the spatial mean vector and spatial standard deviation vector of x respectively; Then, ε is sampled from the Gaussian distribution Ν(0,I) and reparameterized to obtain the spatial latent variable z s : z s =μ s +s s (e) The spatial KL divergence loss between the spatial posterior distribution and the spatial prior distribution is expressed as: Where KL(·) represents KL divergence; Step 3.5, the loss of the decoupled variational autoencoder is expressed as: Among them, L rec represents the reconstruction loss, represents the class posterior distribution and class prior distribution p(z c ), Denotes the spatial posterior distribution q φ (z s |x) and the spatial prior distribution p(z s ) KL divergence loss between ; Step 4: transform the class latent variables z of the two modes c Perform global information integration to obtain the fused class latent variables The spatial latent variables z of the two modes s Perform global information integration to obtain the fused spatial latent variables Class latent variables and spatial latent variables Input into the target detection network for target detection.

2. According to the target detection method based on the decoupled variational autoencoder network of claim 1, it is characterized in that: Step 1 The specific steps are as follows: Step 1.1: Input the collected visible light image and infrared image into the shared backbone network to extract the feature sequence of the visible light image and the feature sequence of the infrared image respectively. The feature sequence is represented as {C1, C2, C3, C4, C5}, where C i Represents the features of the output of the i-th layer, i = 1, 2, 3, 4, 5; Step 1.2, the feature sequence of the visible light image and the feature sequence of the infrared image are input into the feature pyramid network respectively, and a horizontal connection is added to the feature pyramid network to fuse the features of different layers in the feature sequence to obtain the multi-scale features of the visible light image and the multi-scale features of the infrared image. The multi-scale feature P = {P1, P2, P3, P4} is expressed as: Among them, Conv(·) represents convolution and Upsample(·) represents upsampling.

3. According to the target detection method based on the decoupled variational autoencoder network of claim 1, it is characterized in that: In step 2, RoiAlign is used to process the proposal box A and multi-scale features to obtain the ROI feature x0, which is expressed as: x0=RoiAlign(P,A) P represents multi-scale features; The proposed feature F p A one-to-one interaction relationship is established between the ROI feature x0, and the ROI feature x0 is optimized to obtain the optimized ROI feature x, which is expressed as: x=IIH(x0,F p ) Among them, IIH(·) is the instance interaction head; x={x V ,x I },in Represents the ROI features of the visible light image, Represents the ROI features of the infrared image.

4. According to the target detection method based on the decoupled variational autoencoder network of claim 1, it is characterized in that: In step 4, the class latent variables z of the two modes are c Perform global information integration to obtain the fused class latent variables The specific steps are as follows: For the two-modal class latent variables represents the class latent variable of visible light, represents the class latent variable of infrared; From the joint distribution and marginal distribution Sampling positive sample pairs and negative sample pairs, and using the discriminator to distinguish between positive sample pairs and negative sample pairs; In order to allow the discriminator to accurately distinguish these two types of sample pairs, maximize the joint distribution and marginal distribution The distribution difference of and The information integration obtains the fused class latent variable In the same way, the spatial latent variables z of the two modes are s Perform global information integration to obtain the fused spatial latent variables The optimization loss of the discriminator is expressed as: Where σ(·) is the sigmoid activation function, D(·) represents the discriminator, (α, β) is a positive sample pair sampled from the joint distribution p(α, β), and (α, β) represents or is a pair of negative samples sampled from the marginal distribution p(α)p(β).

5. According to the target detection method based on the decoupled variational autoencoder network of claim 4, it is characterized in that: In step 4, the class latent variable and spatial latent variables Input into the target detection network for target detection, expressed as: Among them, TCΩ c (·) is the target recognition head, TSΩ s (·) is the target localization head; it is used to predict the category and location of the target based on multimodal fusion features. and Represent category prediction and position prediction respectively.

Images