Target detection and pose estimation method based on data cross-modal transfer learning

Abstract

The application relates to the technical field of computer vision, and discloses a target detection and pose estimation method based on data cross-modal transfer learning, a domain classification and network model gradient negative optimization module is designed, and an optimal field transfer method, that is, an adversarial domain classifier transfer learning unit, is summarized; the application improves a pose angle task head, so that the network module performs pose angle estimation on a target in a detection frame on the basis of sharing a main feature extraction network, so that the systematic functions of target detection and motion pose estimation are completed on the basis of saving computing overhead, meanwhile, the pose estimation module can filter more unnecessary background features on the basis of the detection module, the accuracy of the module is improved, and the pose angle estimation task is converted into a classification and regression task, so that the network effect is improved through multi-task mode training.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method for target detection and pose estimation based on cross-modal transfer learning of data. Background Technology

[0002] Object detection and pose estimation are common tasks in computer vision. Mainstream object detection methods include one-stage methods like YOLO and SSD, and two-stage algorithms such as Faster R-CNN. The main metrics for object detection models are detection accuracy and forward feed rate; each type of algorithm has its advantages and disadvantages. One-stage algorithms do not require region proposals; they generate the object's class probability and location coordinates through regression, directly obtaining the final detection result in a single detection, thus offering faster detection speed, but lower accuracy compared to two-stage algorithms. Two-stage algorithms first generate region proposals (pre-selected boxes that may contain the object to be detected) using a region generation module, and then classify the object within each pre-selected box using a convolutional neural network. Deep learning-based pose estimation algorithms are often used to estimate hand and head poses. Common methods include template-based algorithms, which primarily rely on template-based operations on the object, locating key points in the template, and then calculating its pose angles, including pitch, roll, and yaw angles. The main drawback of this type of method is the need to predefine templates, requiring different keypoints to be designed for estimation depending on the problem. Another common type of pose angle estimation algorithm does not require keypoints, such as the FSA-Net algorithm, which directly estimates the angle based on regression and feature aggregation methods. The object detection and pose angle estimation methods mentioned above are often trained and tested in the same data domain, resulting in a large number of training samples. However, in real-world scenarios, real data and data annotations may be difficult to obtain, making it difficult to train the model with limited training samples.

[0003] For the YOLO method, the input image is first divided into N×N grids. If the center point of a detection box falls on one of these grids, that grid is responsible for predicting the detection box, including its category, center point coordinates, and dimensions. Anchor boxes are introduced, and each grid is pre-set with anchor boxes through clustering. The actual detection box estimation is based on the deviation from these anchor boxes. This simplifies the problem and makes the network easier to learn. For feature extraction, outputs at different resolutions are concatenated, enabling the network to learn well for targets at different scales. However, this method generates many redundant detection boxes, and non-maximum suppression is used to filter them, reducing network efficiency. For the FSA-Net method, a soft stagewise regression mode is implemented, and fine-grained structure mapping is proposed for feature aggregation. This method only requires one image for pose estimation. In terms of network structure design, this method first extracts features through multiple branches, and then aggregates them spatially through an aggregation module to reduce the number of feature channels. Finally, the regression problem is transformed into a classification problem based on the SSR-Net method, and the angle is predicted using the following soft stage-wise regression formula.

[0004]

[0005] Where K is the number of stages, p is the probability distribution at stage k, and U is a vector composed of angular groups at stage k. To reduce generalization error, an offset vector and a scaling vector are introduced to adjust the center of each bin and the width of the bins at stage k, respectively. This method can be used for any regression problem; however, it completes estimation through multiple stages in series, requiring features from the feature extraction network to undergo feature aggregation and other processing, resulting in significant forward computation overhead. Furthermore, its network structure is difficult to integrate with object detection tasks to form a coherent system. For the CDAN method, multilinear conditional adjustment and entropy conditional adjustment are introduced. The former improves the classifier's recognition rate by capturing the cross-variance between feature representations and classifier predictions, while the latter ensures the classifier's portability by controlling the uncertainty of predictions. However, this method requires significant differences in variance across different feature channels when processing features. When dealing with data samples where features are evenly distributed across different channels, it can lead to significant errors in feature selection. Therefore, existing technologies for object detection and pose estimation have many domain-specific design considerations in their model structures, making it difficult to achieve a concise and efficient model structure when designing a vision system that simultaneously incorporates object detection and pose estimation. Furthermore, many pose estimation algorithms require bounding boxes as input, and deep learning-based vision tasks have certain requirements regarding the amount of training data and the consistency between training and testing data, which limits their application in real-world scenarios. This is because sufficient data and data annotations are often difficult to obtain in real-world situations. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a target detection and pose estimation method based on cross-modal transfer learning. By summarizing and designing a domain classification and network model gradient negative optimization module, a vision system is completed, which can realize the functions of target detection and target pose angle estimation.

[0007] To achieve the above objectives, this invention provides a target detection and pose estimation method based on cross-modal transfer learning, comprising the following steps:

[0008] S1. Collect real-world scene data and annotate the data, including bounding box coordinates (x, y, w, h) and pose angles (yaw, pitch, row), and output real-world data images with corresponding annotations;

[0009] S2. Perform simulation in simulation software based on the real data image to obtain simulation image and annotation, including the detection box coordinates (x, y, w, h) and attitude angles (yaw, pitch, row), and output simulation data image and corresponding annotation;

[0010] S3. Input the simulated data images and their annotations into the shared feature extraction network and object detection module for training, and calculate the loss using the input annotations;

[0011] S4. The detection box information output by the target detection module and the features output by the shared feature extraction network are cropped and then input into the pose angle estimation module for training. The loss is calculated with the annotation input in step S3.

[0012] S5. Network optimization: The losses calculated in steps S3 and S4 are summed and then backpropagation is used to optimize the network until the error on the test dataset no longer decreases.

[0013] S6. Randomly mix simulation and real data to generate domain labels simultaneously, and output the mixed image, domain labels, target detection box annotations, and attitude angle annotations.

[0014] S7. Use the transfer learning module to perform adversarial training on the shared feature extraction network to complete the domain transfer using the data generated in step S6.

[0015] S8. Forward calculation: Input test or actual data and output attitude angles after passing through the shared feature extraction network, target detection module and attitude angle estimation module.

[0016] Preferably, the shared feature extraction network is a multi-scale fusion feature extraction network that uses multiple convolutional and pooling units in series to extract features from the input image. The first convolutional module contains a 3×3×3 convolutional layer with a stride of 1 and padding of 1, a batch normalization layer, and a LeakyReLU layer. Subsequently, it passes through 1, 2, 8, 8, and 4 convolutional unit groups, respectively. Each unit group contains a 3×3×n convolutional layer with a stride of 2, where n is the dimension of the input features and padding is 1. The features are downsampled, and each unit is connected by residuals to maintain good learning ability even at greater depths. Each convolutional unit group consists of a 1×1×n convolution with a stride of 1 (no padding), a batch normalization layer, and a LeakyReLU layer, followed by a 3×3×n convolution with a stride of 1 (1 padding), a batch normalization layer, and a LeakyReLU layer. The image input is x∈X, where X represents the image input space. Let the feature extraction network be f=G. f (x;θ f This transforms the input image x into a D-dimensional feature vector.

[0017] Preferably, the input to the target detection module is the feature output by the shared feature extraction network, which consists of three features at different scales. Multi-scale detection is used to detect targets of different sizes. In the loss calculation, an ignore parameter is defined, indicating that if the maximum intersection-union ratio (IU) of a predicted bounding box with all ground truth values ​​is less than the ignore parameter, then the predicted bounding box is a negative sample. If the center point of a ground truth value falls within a region, that region is responsible for detecting the object, and the predicted bounding box with the maximum IU with that object is considered a positive sample. In a single grid, there are B target bounding boxes, each consisting of a five-dimensional prediction parameter, including the center point coordinates (x, y), width and height (w, h), and confidence score s. i Confidence score s i The following can be calculated:

[0018]

[0019] Where Object represents the target object, and Pr(Object) represents the probability that an object exists within the bounding box of the current mesh target. The Intersection over Union (IoU) ratio represents the ratio of the predicted bounding box value to the ground truth value, demonstrating the accuracy of the target bounding box location predicted by the current model.

[0020] Given the target bounding box prediction value box pred With truth box truth ,but It can be represented as:

[0021]

[0022] The class probability of an object, Pr(C) i |O) represents the posterior probability that the target belongs to a certain type of object i given that the target exists in the bounding box. Assuming there are K types of objects in the object detection task, each grid predicts the object of type i, C. i The conditional probability is Pr(C) i |O),i=1,2,…,K,

[0023] The final training loss function calculation consists of three parts: ① the error caused by x, y, w, and h, which is the loss caused by the detection box; ② the error caused by the confidence score; and ③ the error caused by the classification, which is the loss caused by the classification category. The calculation formula is shown below:

[0024]

[0025]

[0026]

[0027] loss = lbox + lobj + lcls

[0028] For the object detection module, the input image is divided into S×S grids, and each grid is responsible for detecting target objects whose center point falls within it.

[0029] During testing, the confidence that an object exists within the target bounding box can be expressed as, through non-maximum suppression.

[0030] Retain the required detection boxes:

[0031]

[0032] Preferably, the attitude angle estimation module includes a feature decoupling module and a cross-class center loss module.

[0033] Preferably, the feature decoupling module is implemented using three channel attention blocks.

[0034] Preferably, the center loss portion of the cross-class center loss module at each angular branch is defined as follows:

[0035]

[0036] Where z(i) is the deep feature of the i-th embedding, c(yi) is the embedding center of the yi-th class, which is updated during training, m is the mini-batch sample size, and j represents each angle branch.

[0037] The above part reduces the distance between latent variables with the same discrete angle truth value according to different angle categories, thereby ensuring intra-class consistency. Latent variables of different angle categories should be distributed in decoupled subspaces, but this is not reflected in the central loss mentioned above. To mitigate this deficiency, the decoupling part of the cross-class central loss is further defined as follows:

[0038]

[0039] Where j, j′, j″∈{yaw;pitch;roll}, j≠j′≠j″, and This represents the cross-angle category correlation distance. Adding 1 to the denominator prevents the calculation result from overflowing.

[0040] The proposed CCC loss comprises the above two parts and can be written in the following form:

[0041]

[0042] Here, α is a hyperparameter used to weigh the two parts of the loss.

[0043] Preferably, step S7 specifically includes the following steps:

[0044] S71. Transfer learning label generation employs a domain-adversarial transfer learning algorithm. For most visual tasks, the problem can be transformed into having the network learn a posterior probability P(C,B|I), where I represents image features, B represents detection boxes, and C represents object categories. For an input image, it can be viewed as the joint probability distribution P(C,B,I) of B, C, and I. According to a simple Bayesian equation, the following formula can be obtained:

[0045] P(C,B,I)=P(C,B|I)P(I)

[0046] In most problems of transfer learning, there is a fundamental assumption that the feature distributions in the target and source domains are the same. This is because the overall data distributions of simulated and real infrared data are very similar, which is reflected in the formula that P(C,B|I) is the same for both domains, only P(I) differs. Therefore, for a detection network, its detection head should be identical, meaning it should have consistent performance in both the target and source domains. Thus, to ensure the entire network achieves the same detection performance in the target domain as in the source domain, it is necessary to minimize P(I) between the source and target domains. This involves transferring the feature extraction backbone to reduce the difference in P(I). A domain classifier is introduced to achieve this. The overall network structure consists of the domain classifier and the target task head as two parallel sub-branches connected to the feature extraction backbone network. Following this, the domain classifier is a binary classifier, classifying the labels as source domain / target domain. The goal of the target task head is to minimize the target task loss function, but it's clear that the target task head will overfit the source domain dataset (i.e., simulation data). The goal of the domain classifier is to minimize the binary classification error, that is, to distinguish the two domains as much as possible. The main task of the feature extraction network is to extract a feature shared by the target task head and the domain classifier. This feature has two objectives: minimizing the target task loss function (target task head) and maximizing the binary classification error (adversarial domain classifier). The second objective is achieved using gradient retrieval. Essentially, when the gradient from the domain classifier is backpropagated to the feature extraction backbone network, its gradient is multiplied by a negative coefficient to make the feature extraction backbone network optimize in the direction of the adversarial domain classifier, so that the source domain data distribution and the target domain data distribution are... The joint distribution space is respectively and Therefore, the function implemented by this classifier includes making the input samples {x1,x2,…,x} from the source domain and the target domain equal to the input samples {x1,x2,…,x}. N} performs domain classification, characterized by defining d iLet be the domain label of the i-th sample, where d i ∈{0,1}, if d i =0, then Conversely, if d i =1, then First, the input image passes through a feature extraction backbone network, which is then denoted as f = G. f (x;θ f The input image x is transformed into a D-dimensional feature vector f∈R. D During the learning phase, the goal is to minimize the label prediction loss of the labeled portion of the training set (i.e., the source domain dataset portion). Therefore, the parameters of the feature extraction network and the target task head are optimized to minimize the loss of the source domain samples. This ensures the discriminative power of feature f and the overall good predictive performance of the feature extractor and label predictor. The combination of the feature extractor and label predictor exhibits good predictive performance in the source domain. Simultaneously, it is desirable that feature f possesses domain invariance; that is, the goal of the network optimization during training is to make the distribution S(f) = {G}. f (x;θ f )|x~S(x)} and T(f)={G f (x;θ f If we consider the relationship between distributions S(f) and T(f), under the covariate transfer hypothesis, this would make the label prediction accuracy of the target domain the same as that of the source domain (Shimodaira, 2000). However, considering that f is high-dimensional and the distribution itself changes continuously with learning, measuring the dissimilarity between distributions S(f) and T(f) is relative. One way to estimate the dissimilarity is to look at the loss of the domain classifier, provided that the parameters of the domain classifier have been trained to distinguish the two feature distributions in the best way. Therefore, during training, in order to obtain domain-invariant features, we seek the parameters θ of the feature map that maximize the loss of the domain classifier. f (By making the two feature distributions as similar as possible), while seeking the parameter θ that minimizes the loss of the domain classifier. d Furthermore, we also seek to minimize the loss of the target task head; therefore, the task can be specified as:

[0047]

[0048] Among them, L y Let L be the loss function for the target task. d Let the loss function be the domain classifier. and We need to calculate the loss for the i-th sample. Therefore, based on the above formula, the parameters we seek are:

[0049]

[0050]

[0051] The forward computation and backpropagation of error to minimize the target task loss are consistent with the traditional deep learning process. However, to maximize the domain classification loss, we introduce a gradient backpropagation module. The gradient obtained by backpropagation in the domain classifier is multiplied by a negative coefficient -λ before entering the feature extraction network to optimize the feature backbone network. That is, the network parameter update method is as follows:

[0052]

[0053]

[0054]

[0055] Based on this, in addition to the annotation of the target task itself, the data annotation also needs to be annotated for the data domain. Since there are two data domains, namely real infrared data and simulated data, 0 is used to represent the real data domain and 1 is used to represent the simulated data domain. Therefore, for each input image, its data format is as follows: [input image (width, height, c), target task annotation (taking target detection as an example, (x, y, w, h), domain label (0 or 1))];

[0056] S72. Conduct network training.

[0057] Preferably, step S72 specifically includes the following steps:

[0058] S721. During the initial training, the adversarial learning module is not enabled. Only the feature extraction network and the target task head are enabled. The input data is only simulation data, so that the network can perform well on the target task on the simulation dataset. At the same time, a portion of the simulation data is randomly divided into test datasets. The model is trained until the performance on the test dataset no longer improves.

[0059] S722, Domain Adversarial Training: In terms of data input, each batch of data input during the training process is changed to simulated data and real data. The input labels include target task labels and domain labels. Then, training is carried out on the model obtained in step S721. The training enables the feature extraction network, target task head and domain classifier, and sets a small learning rate (e.g., 1e-4 for target detection task) for training, and trains for 5 to 10 rounds of iteration.

[0060] S723, Fineetue training: Fineetue training is performed on a real infrared training dataset without enabling the domain classifier module.

[0061] Compared with the prior art, the beneficial effects of the present invention are:

[0062] 1. This invention summarizes and designs a domain classification and network model gradient negative optimization module, and summarizes an optimal domain transfer method, namely the adversarial domain classifier transfer learning unit.

[0063] 2. This invention completes a vision system capable of object detection and object pose angle estimation. A shared feature extraction network (backbone) is designed for both vision tasks. This network, through residual connections and multi-scale feature fusion, exhibits good representation capabilities for features required at different scales and for different tasks. By improving the pose angle task head, this network module, based on the shared backbone feature extraction network, estimates the pose angle of targets within the detection box. This achieves systematic object detection and motion pose estimation while saving computational overhead. Furthermore, the pose estimation module, building upon the detection module, filters out more unnecessary background features, improving the module's accuracy. The pose angle estimation task is then transformed into classification and regression tasks, trained in a multi-task mode to enhance network performance. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the overall network structure provided by the present invention;

[0065] Figure 2 This is a schematic diagram of the overall structure of the feature extraction network provided by the present invention;

[0066] Figure 3 This is a schematic diagram of the overall network structure of the attitude angle estimation module provided by the present invention;

[0067] Figure 4 A hybrid loss flowchart is constructed for this invention;

[0068] Figure 5 This is a schematic diagram of the overall structure of the transfer learning network provided by the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] This invention primarily focuses on target detection and pose angle estimation. The computational flow during training and forward pass is as follows: First, the input image passes through a shared feature extraction network to obtain image features D. Then, these image features D are input into the target detection head to calculate the detection bounding box. Subsequently, based on the coordinates of the detection bounding box, the image features D output by the shared feature extraction network are cropped to obtain the cropped features d. Features d are then input into the pose angle estimation module to calculate the pose angle output. The overall network structure is as follows: Figure 1 As stated above.

[0071] The target detection and pose estimation method based on cross-modal transfer learning proposed in this invention includes the following steps:

[0072] S1. Collect real-world scene data and annotate the data, including bounding box coordinates (x, y, w, h) and pose angles (yaw, pitch, row), and output real-world data images with corresponding annotations;

[0073] S2. Perform simulation in simulation software based on the real data image to obtain simulation image and annotation, including the detection box coordinates (x, y, w, h) and attitude angles (yaw, pitch, row), and output simulation data image and corresponding annotation;

[0074] S3. Input the simulated data images and their annotations into the shared feature extraction network and object detection module for training, and calculate the loss using the input annotations;

[0075] S4. The detection box information output by the target detection module and the features output by the shared feature extraction network are cropped and then input into the pose angle estimation module for training. The loss is calculated with the annotation input in step S3.

[0076] S5. Network optimization: The losses calculated in steps S3 and S4 are summed and then backpropagation is used to optimize the network until the error on the test dataset no longer decreases.

[0077] S6. Randomly mix simulation and real data to generate domain labels simultaneously, and output the mixed image, domain labels, target detection box annotations, and attitude angle annotations.

[0078] S7. Use the transfer learning module to perform adversarial training on the shared feature extraction network to complete the domain transfer using the data generated in step S6.

[0079] S8. Forward calculation: Input test or actual data and output attitude angles after passing through the shared feature extraction network, target detection module and attitude angle estimation module.

[0080] In real-world scenarios, obtaining authentic data and data annotations can be difficult, making model training challenging with limited training samples. Therefore, data with similar distributions, such as simulation data, is often used in conjunction with real data for training. However, such data still differs in distribution from real-world test data. Therefore, this invention employs transfer learning for feature domain alignment. In the following description, the simulation dataset is used as the source domain dataset, and the real dataset as the target domain dataset. Transfer learning mainly includes: ① Transfer learning based on data domain samples: This involves adjusting the annotations of the source domain dataset and using source domain data as an aid to adjust the weights of the labels in the target domain dataset, co-training to obtain the target model. This method requires a certain amount of data in the target domain. ② Transfer learning based on input data features: This involves adjusting the parameters of the feature extraction network to obtain aligned features, reducing the differences between the source and target domains and lowering errors in visual tasks such as classification and regression. ③ Transfer learning based on model parameters: This involves discovering shared parameters or prior relationships between the source and target domains to complete the transfer. ④ Transfer learning based on correlation: This method establishes relevant knowledge between the source and target domains and completes sample transfer through knowledge mapping.

[0081] The feature extraction network consists of feature extraction units. This invention implements a multi-scale fusion feature extraction network, which uses multiple convolutional and pooling units in series to extract features from the input image. The first convolutional module contains a 3×3×3 convolutional layer with a stride of 1, padding of 1, a batch normalization layer, and a LeakyReLU. This is followed by 1, 2, 8, 8, and 4 convolutional unit groups, each group containing a 3×3×n convolutional layer with a stride of 2, where n is the input feature dimension, and padding of 1, to downsample the features. Residual connections are used between units, allowing the network to maintain good learning ability even at greater depths. Each convolutional unit group consists of a 1×1×n convolution with a stride of 1, no padding, a batch normalization layer, and a LeakyReLU. This is followed by a 3×3×n convolution with a stride of 1, padding of 1, a batch normalization layer, and a LeakyReLU. The image input is x∈X, where X represents the image input space. Let the feature extraction network be f=G. f (x;θ f This can transform the input image x into a D-dimensional feature vector. A schematic diagram of the overall structure of the feature extraction network is shown below. Figure 2 As shown.

[0082] like Figure 1As shown, this set of D-dimensional feature vectors needs to be input into both the target detection module and the pose angle estimation module. In this set of feature vectors, the semantic information required by one visual task may be useless noise for the other task, which will affect the training of the network. Therefore, this invention also introduces a channel self-attention module to enable this set of features to have good adaptability to both visual tasks. This module mainly includes three steps: (1) global pooling to perform feature aggregation; (2) specific feature activation, using a structure of fully connected layer 1 --> ReLU --> fully connected layer 2 --> Sigmoid. In order to reduce complexity and achieve better generalization, the number of channels in fully connected layer 1 is reduced compared to the original number of channels; the connection from fully connected layer 1 to fully connected layer 2 is increased in dimension to restore the original number of channels; (3) scaling operation, applying the weights obtained after the activation of specific features to the original feature map.

[0083] Both the object detection module and the pose angle estimation module take the features output by the shared feature extraction network as input, as shown in the overall system framework diagram. These two modules are two branches of the shared feature extraction network. The object detection module takes three features at different scales as input, borrowing from the feature pyramid, and uses multiple scales to detect targets of different sizes. In the loss calculation, this invention defines an ignored parameter, indicating that if the maximum intersection-union ratio (IU) of a predicted bounding box with all ground truth values ​​is less than the ignored parameter, then the predicted bounding box is a negative sample; if the center point of the ground truth value falls within a region, that region is responsible for detecting the object, and the predicted bounding box with the maximum IU with that object is taken as a positive sample. A single grid contains B target bounding boxes, each consisting of a five-dimensional prediction parameter, including the center point coordinates (x, y), width and height (w, h), and confidence score s. i .

[0084] Confidence score s i The following can be calculated:

[0085]

[0086] Here, Object represents the target object, and Pr(Object) represents the probability that an object exists in the current grid target bounding box. The Intersection over Union (IoU) ratio represents the ratio between the predicted bounding box value and the true value, demonstrating the accuracy of the target bounding box location predicted by the current model.

[0087] Given the target bounding box prediction value box pred With truth box truth ,but It can be represented as:

[0088]

[0089] The class probability of an object, Pr(C) i |O) represents the posterior probability that the target belongs to a certain type of object i, given that the target exists in the bounding box. Assuming there are K types of objects in the object detection task, each grid cell predicts the object C of type i. i The conditional probability is P r(C i |O),i=1,2,…,K.

[0090] The final training loss function calculation consists of three parts: ① the error caused by x, y, w, and h, which is the loss from the detection boxes; ② the error caused by the confidence score; and ③ the error caused by the classification, which is the loss from the classification category. The calculation formula is shown below:

[0091]

[0092]

[0093]

[0094] loss = lbox + lobj + lcls

[0095] For the object detection module, the input image is divided into S×S grids, and each grid is responsible for detecting the target object whose center point falls within it.

[0096] During testing, the confidence that an object exists within the target bounding box can be expressed as the required detection box being preserved through non-maximum suppression:

[0097]

[0098] The above process describes the object detection system, which crops the original image based on the detection boxes obtained from object detection. Due to the translation invariance of convolution and the design of the shared feature extraction network in this invention, the positions of the detection boxes in the original image can be cropped in the feature map using the same scaling and translation transformations. For example, if the width of the original image is (w1, h1), and the center point coordinates and dimensions of the detection boxes are (x, y, w2, h2), the feature map obtained after passing through the feature extraction backbone network is (w3, h3, c). Cropping can then be performed on the feature map, with the center point coordinates and dimensions of the cropped box being... The dimension remains C.

[0099] The pose angle estimation module includes a feature decoupling module and a cross-class center loss module. A schematic diagram of the overall network structure of the pose angle estimation module is shown below. Figure 3 As shown.

[0100] Feature Decoupling Module: This module takes the feature map output by the feature extraction network as input to decouple the latent variable space at different angles, as shown in the figure above. This invention employs a three-channel attention block to implement the feature decoupling module. Researchers generally believe that different layers in CNNs encode information at different levels. Pose information is typically encoded in high-level features of the network, while low-level features often encode more detailed image information, such as edge and texture features. By adding a feature decoupling module to the output feature map of the backbone convolutional neural network, the feature decoupling network proposed in this chapter can mitigate the adverse effects of background clutter and semantic uncertainty in low-level image features. In the feature decoupling module, a parameterized channel attention mechanism is used to adaptively readjust the channel-wise response value for each angle branch. Specifically, a bottleneck layer containing two fully connected layers and a non-linear activation function is used. Since the correlation between channels is implicitly encoded into the learned filters, the feature decoupling module selects channel features containing more information by performing angle-dependent feature readjustment, thereby explicitly capturing more discriminative features for each angle while suppressing less effective features. The module parameters are updated using angle-dependent loss.

[0101] Pose angle estimation can be viewed as a natural regression problem. Previous work has shown that joint supervision of classification and regression can improve model performance. Therefore, this invention also follows this approach by using two losses to construct a mixed loss, such as... Figure 4 As shown.

[0102] The angle-dependent Mix Loss takes decoupled features as input, including both classification and regression terms. These decoupled features are then fed into a fully connected layer, followed by a Softmax layer to obtain the probability prediction for each cell. Since there is an obvious semantic relationship between the sequential discrete labels in the attitude angle estimation problem, the classification labels in this method are a one-dimensional Gaussian distribution with a mean of the ground truth class and a small variance. The classification loss is obtained by calculating the Kullback-Leibler (KL) divergence between the label distribution and the prediction distribution. Next, the angle value is predicted by calculating the expectation of the small cell output, and the mean squared error (MSE) loss is used as the regression loss. The final Mix Loss for each angle branch is shown below:

[0103]

[0104] Where qj is the output classification probability, yj is the true angle, and G(·) represents the result of applying a one-dimensional Gaussian filter to the original one-hot code. is the final angle prediction, b is the index of the small grid, j∈{yaw;pitch;roll} represents each angle branch, and λ is a hyperparameter used to balance the classification loss and the regression mean squared error loss.

[0105] Cross-category center loss: Further, a cross-category center (CCC) loss is proposed to simultaneously achieve intra-class compactness and inter-class separability of the latent variable subspace. Similar to Wen et al., the center loss part of each angle branch proposed in this invention is defined as follows:

[0106]

[0107] Where z(i) is the deep feature of the i-th embedding, c(yi) is the embedding center of the yi-th class, which is updated during training, m is the sample size of the mini-batch, and j represents each angle branch.

[0108] The above section reduces the distance between latent variables with the same discrete angle truth value based on different angle categories, thus ensuring intra-class consistency. Latent variables from different angle categories should be distributed in decoupled subspaces, but this is not reflected in the aforementioned center loss. To mitigate this deficiency, the decoupling part of the cross-class center loss is further defined as follows:

[0109]

[0110] Where j, j′, j″∈{yaw;pitch;roll}, j≠j′≠j″, and This represents the cross-angle category related distance. Adding 1 to the denominator is to prevent the calculation result from overflowing.

[0111] The proposed CCC loss comprises the above two parts and can be written in the following form:

[0112]

[0113] Here, α is a hyperparameter used to weigh the two parts of the loss.

[0114] Network Structure and Optimization: Considering both feature extraction capability and model parameter size, this invention constructs the backbone network based on the Inverted Residual Block proposed by Sandler et al. The input, output, and internal structure of the Inverted Residual Block are shown in Table 1. Compared to the original residual block, the input feature map channels are first increased in dimensionality and then decreased in dimensionality, and no activation function is added after the last convolution. Specifically, the backbone neural network of this invention, besides the initial convolutional layer containing 32 filters and the final convolutional layer containing 640 filters, has a total of 9 Inverted Residual Blocks, the design details of which are shown in Table 2.

[0115] Finally, the loss for each angle branch can be written in the following form:

[0116]

[0117] Table 1 Input / output and internal structure of the inverse residual block

[0118]

[0119] Table 2. Backbone Neural Network Structure

[0120] 112×112×32 1 16 1 1 112×112×16 6 24 2 2 56×56×24 6 32 2 2 28×28×32 6 96 2 2 14×14×96 6 240 2 2

[0121] Where y∈{yaw,pitch,roll} represents each angle branch. The embedding center is updated using the following formula:

[0122]

[0123] Where jk represents the k-th center of the j-th angle branch, and δ(condition) = 1 if the condition is met, otherwise δ(condition) = 0. The following algorithm details the training process of the proposed method.

[0124]

[0125] In real-world scenarios, data acquisition for many training tasks is challenging, resulting in limited real-world data and making it difficult to train deep learning networks. In such cases, simulation data or other available data from similar environments can be used to aid training, and simulation data is relatively abundant. However, the two data domains may have some distributional discrepancies. Therefore, if the model is trained on simulation data and then tested directly on real data, its performance will significantly degrade. In this invention, a transfer learning algorithm will be introduced and combined with Finetune to fully utilize both simulation and real-world data, enabling the model to perform well in real-world scenarios.

[0126] Data preparation. Based on real-world data and background, conduct simulations in simulation software or find relevant similar datasets. Generate different annotations based on the different downstream subtasks. The annotation format needs to be consistent with the annotations in the real data. For example, the annotation of a detection bounding box is (x, y, w, h), representing the center coordinates and width and height of the target point, respectively.

[0127] Transfer learning label generation. The invention employs a domain-adversarial transfer learning algorithm. For most visual tasks, the problem can be transformed into having the network learn a posterior probability P(C,B|I), where I represents image features, B represents detection boxes, and C represents object categories. For an input image, we can consider it as the joint probability distribution P(C,B,I) of B, C, and I. Then, according to a simple Bayesian equation, we can obtain the following formula:

[0128] P(C,B,I)=P(C,B|I)P(I)

[0129] In most problems of transfer learning, there is a fundamental assumption that the feature distributions in the target and source domains are the same. This assumption holds true in this invention because the overall data distributions of simulated data and real infrared data are very similar. This is reflected in the formula: P(C,B|I) is the same for both domains, only P(I) differs. Therefore, the detection head of the detection network should be the same, meaning it should have consistent performance in both the target and source domains. Thus, to ensure the entire network achieves the same detection performance in the target domain as in the source domain, it is necessary to minimize P(I) between the source and target domains, i.e., to transfer the feature extraction backbone to reduce the difference in P(I). A domain classifier is introduced to achieve this. A schematic diagram of the overall transfer learning network structure is shown below. Figure 5 As shown:

[0130] The domain classifier and the target task head are connected as two parallel sub-branches after the feature extraction backbone. The domain classifier is a binary classifier, classifying the data into the source domain and the target domain. The target task head aims to minimize the target task loss function, but it's clear that the target task head will overfit the source domain dataset (i.e., the simulation data). The domain classifier aims to minimize the binary classification error, that is, to distinguish the two domains as much as possible. The main task of the feature extraction network is to extract a feature shared by the target task head and the domain classifier. This feature has two objectives: minimizing the target task loss function (target task head) and maximizing the binary classification error (adversarial domain classifier). The second objective is achieved using gradient backpropagation. Essentially, when the gradient from the domain classifier is backpropagated to the feature extraction backbone, its gradient is multiplied by a negative coefficient, causing the feature extraction backbone to optimize in the direction of the adversarial domain classifier.

[0131] In this step, the source domain data distribution and the target domain data distribution are set to... The joint distribution space is respectively and Therefore, the function implemented by this classifier includes making the input samples {x1,x2,…,x} from the source domain and the target domain equal to the input samples {x1,x2,…,x}. N} performs domain classification, characterized by defining d i Let be the domain label of the i-th sample, where d i ∈{0,1}. If d i =0, then Conversely, if d i =1, then First, the input image passes through a feature extraction backbone network, which is then denoted as f = G. f (x;θ f The input image x is transformed into a D-dimensional feature vector f∈R. D During the learning phase, the goal is to minimize the label prediction loss of the labeled portion of the training set (i.e., the source domain dataset portion). Therefore, the parameters of the feature extraction network and the target task head are optimized to minimize the loss of the source domain samples. This ensures the discriminative power of feature f and the overall good predictive performance of the feature extractor and label predictor. The combination of the feature extractor and label predictor achieves good predictive performance in the source domain. Simultaneously, it is desirable for feature f to possess domain invariance, meaning that the network optimizes during training to ensure that the distribution S(f) = {G}. f (x;θ f )|x~S(x)} and T(f)={G f (x;θ f The discrepancies between distributions S(f) and T(f) are similar. Under the covariate transfer assumption, this would make the label prediction accuracy of the target domain the same as that of the source domain (Shimodaira, 2000). However, considering that f is high-dimensional and the distribution itself changes continuously with learning, measuring the dissimilarity between distributions S(f) and T(f) is relative. One way to estimate the dissimilarity is to look at the loss of the domain classifier, provided that the parameters of the domain classifier have been trained to distinguish the two feature distributions in the best way. Therefore, during training, in order to obtain domain-invariant features, we seek the parameters θ of the feature map that maximize the loss of the domain classifier. f (By making the two feature distributions as similar as possible), while seeking the parameter θ that minimizes the loss of the domain classifier. d Furthermore, we seek to minimize the loss of the target task head. Therefore, the task can be specified as:

[0132]

[0133] Among them, L yLet L be the loss function for the target task. d Let the loss function be the domain classifier. and Let's calculate the loss for the i-th sample. Therefore, based on the above formula, the parameter we are looking for is...

[0134]

[0135]

[0136] The forward computation and error backpropagation for minimizing the target task loss are consistent with the traditional deep learning process. However, to maximize the domain classification loss, we introduce a gradient backpropagation module. The gradient obtained through normal backpropagation in the domain classifier is multiplied by a negative coefficient -λ before entering the feature extraction network to optimize the feature backbone network. The network parameter update method is as follows:

[0137]

[0138]

[0139]

[0140] Based on this, in addition to the annotation of the target task itself, the data annotation of this invention also needs to annotate the data domain. Since there are two data domains, namely real infrared data and simulated data, 0 is used to represent the real data domain and 1 is used to represent the simulated data domain. Therefore, for each input image, its data format is as follows: [input image (width, height, c), target task annotation (taking target detection as an example, (x, y, w, h), domain label (0 or 1))].

[0141] Network training. Network training mainly consists of the following steps:

[0142] ① During the initial training, the adversarial learning module is not enabled; only the feature extraction network and the target task head are used. The input data is only simulation data, allowing the network to perform well on the target task on the simulation dataset. At the same time, a portion of the simulation data is randomly divided into test datasets, and the model is trained until its performance on the test dataset no longer improves.

[0143] ② Domain Adversarial Training. For data input, each batch of input data during training is changed to simulated and real data, with input labels including the target task label and the domain label. Then, training is performed based on the model obtained in step ①, using a feature extraction network, a target task head, and a domain classifier, with a relatively small learning rate (e.g., 1e-4 for object detection). 5-10 iterations of training are sufficient to achieve good results. During training, it is necessary to monitor the target task's metrics on the real test set to avoid overfitting.

[0144] ③Finetue Training. Without enabling the domain classifier module, fineetue training is performed on the real infrared training dataset, further improving the network's performance on the real dataset.

[0145] The forward pass of the network does not require the participation of a domain classifier in the computation; the forward pass result is obtained only through the feature extraction network and the target task head.

[0146] The main effects of this invention are reflected in two aspects: ① By leveraging transfer learning, the model accuracy on the target dataset is significantly improved when using a small amount of real data; ② By utilizing a shared feature extraction network and an end-to-end overall architecture, the number of model parameters and computation time for target detection and pose angle estimation are significantly reduced.

[0147] Using the separate object detection network YOLOv3 and pose estimation network FDN, the forward computation time on a GTX 1080 is 213ms. Using the network structure in this paper, the forward computation time is reduced to 97ms on the same image size.

[0148] The table below illustrates the transfer learning method presented in this paper and compares it with direct testing and Finetune on real datasets. To simulate the source and target domains, the source dataset is the Cityscape dataset, and the target dataset is the Foggy-Cityscape dataset.

[0149]

[0150] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A target detection and pose estimation method based on cross-modal transfer learning, characterized in that, Includes the following steps: S1. Collect real-world scene data and label the data, including the coordinates of the detection box. and attitude angle Output real data images, along with corresponding annotations; S2. Perform simulation in simulation software based on the real data image to obtain a simulated image and annotations, including the coordinates of the detection box. and attitude angle Output simulation data images, along with corresponding annotations; S3. Input the simulated data images and their annotations into the shared feature extraction network and object detection module for training, and calculate the loss using the input annotations; S4. The detection box information output by the target detection module and the features output by the shared feature extraction network are cropped and then input into the pose angle estimation module for training. The loss is calculated with the annotation input in step S3. S5. Network optimization: The losses calculated in steps S3 and S4 are summed and then backpropagated to optimize the network until the error on the test dataset no longer decreases. S6. Randomly mix simulation and real data to generate domain labels simultaneously, and output the mixed image, domain labels, target detection box annotations, and attitude angle annotations. S7. Use the transfer learning module to perform adversarial training on the shared feature extraction network to complete the domain transfer using the data generated in step S6. S8. Forward calculation: Input test or actual data and output attitude angles after passing through the shared feature extraction network, target detection module and attitude angle estimation module.

2. The target detection and pose estimation method based on cross-modal transfer learning according to claim 1, characterized in that, The shared feature extraction network is a multi-scale fusion feature extraction network that uses multiple convolutional and pooling units in series to extract features from the input image. The first convolutional module contains a 3×3×3 convolutional layer with a stride of 1 and padding of 1, a batch normalization layer, and a LeakyReLU. This is followed by 1, 2, 8, 8, and 4 convolutional unit groups, each group containing a 3×3×n convolutional layer with a stride of 2, where n is the input feature dimension and padding of 1, to downsample the features. Residual connections are used between units to maintain good learning ability even at greater depths. Each convolutional unit group consists of a 1×1×n convolution with a stride of 1 (no padding), a batch normalization layer, and a LeakyReLU, followed by a 3×3×n convolution with a stride of 1, padding of 1, a batch normalization layer, and a LeakyReLU. The image input is... Let the feature extraction network be , input image Transform it into a D-dimensional feature vector.

3. The target detection and pose estimation method based on cross-modal transfer learning according to claim 2, characterized in that, The target detection module takes as input features from the shared feature extraction network, which output features at three different scales. This multi-scale approach detects targets of different sizes. In the loss calculation, an ignore parameter is defined, indicating that if the maximum intersection-union ratio (GUC) of a predicted bounding box with all ground truth values ​​is less than the ignore parameter, then the predicted bounding box is a negative sample; if the detected bounding box coordinates... The center point of the detection box falls within a region, and that region is responsible for detecting the coordinates of the detection box. For the corresponding target object, the predicted bounding box with the highest intersection-union ratio (IUU) with that object is taken as a positive sample. This means that within a single grid cell, there exists... There are 1 target bounding box, and each target bounding box consists of a five-dimensional prediction parameter, including the coordinates of the center point of the bounding box. Width and height With confidence score Confidence score The following can be calculated: ； in, Represents the target object. This indicates the probability that an object exists within the bounding box of the current mesh target. The intersection-over-union ratio (IoU) represents the predicted bounding box value compared to the true value, demonstrating the accuracy of the target bounding box location predicted by the current model. Given the target bounding box prediction value with truth value ,but It can be represented as: ； Object category probability This represents the posterior probability that a target belongs to a certain object class given the presence of a target within the bounding box. Assume the object detection task has a total of... For each type of object, the prediction for the first type is as follows: Object The conditional probability is , The final training loss function calculation consists of three parts: The error caused by this part is the loss due to the detection frame. Error introduced by confidence level The error caused by the category, also known as the loss due to the classification category, is calculated using the following formula: ； ； ； ； For the object detection module, the input image is divided into... There are 12 grids, each responsible for detecting target objects whose center point falls within it. During testing, the confidence that an object exists within the target bounding box can be expressed as the required detection box being preserved through non-maximum suppression: 。 4. The target detection and pose estimation method based on cross-modal transfer learning according to claim 3, characterized in that, The attitude angle estimation module includes a feature decoupling module and a cross-class center loss module.

5. The target detection and pose estimation method based on cross-modal transfer learning according to claim 4, characterized in that, The feature decoupling module is implemented using three channel attention blocks.

6. The target detection and pose estimation method based on cross-modal transfer learning according to claim 4, characterized in that, The cross-class center loss module has the center loss component for each angle branch defined as follows: ； in, It is the deep feature of the i-th embedding. It is the yth is The embedding centers of each class are updated during training, where m is the mini-batch size and ja represents each angular branch. The above part reduces the distance between latent variables with the same discrete angle truth value based on different angle branches, thus ensuring intra-class consistency. The latent variables of different angle branches should be distributed in decoupled subspaces, but this is not reflected in the central loss mentioned above. To mitigate this deficiency, the decoupling part of the cross-class central loss is further defined as follows: ； in , , ,and This represents the cross-angle branch correlation distance. Adding 1 to the denominator prevents overflow of the calculation result. The cross-class center loss includes both of the above parts and can be written in the following form: ； in It is a hyperparameter used to balance the two parts of the loss.

Images