Image text feature fusion domain adaptive target detection method for intelligent driving scene

Abstract

The invention discloses an image text feature fusion domain adaptive target detection method for an intelligent driving scene, and the method designs a deep neural network which can learn domain irrelevant features. Wherein the image text feature fusion module is used for enabling features to have domain independence, and the feature-level style migration module is used for converting feature styles of a target domain into feature styles of a source domain; the method comprises the steps of firstly pre-training a model, designing a text prompt phrase and extracting semantic features, then generating a pseudo tag by using a teacher model added with an image text feature fusion module, then training a student model additionally added with a feature-level style migration module, and finally updating the teacher model through an index moving average method as a final result. According to the method provided by the invention, the adaptive capacity of the automatic driving target detection model on the target domain data can be effectively improved, and meanwhile, the labeling cost of the data set is remarkably reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and pattern recognition, and specifically relates to an image-text feature fusion domain adaptation target detection method for intelligent driving scenarios. Background Art

[0002] As one of the most basic computer vision technologies, object detection aims to enable computers to locate specific objects in images and accurately identify and distinguish the categories of different objects. In recent years, the rapid development of deep learning has greatly promoted the advancement of object detection technology. Thanks to the powerful feature extraction capabilities of deep convolutional neural networks, many object detection models with excellent performance have been born. At present, due to the advantages of on-board cameras such as capturing richer information and lower cost, vision-based object detection technology plays a vital role in autonomous driving.

[0003] Supervised learning and same-domain object detection technology is now very mature and has achieved good detection results on existing datasets. However, unsupervised cross-domain object detection technology still faces great challenges. Existing object detection methods usually assume that the training data and test data are from the same domain. However, in actual autonomous driving scenarios, due to factors such as weather, geographical environment, and equipment, objects such as pedestrians, vehicles, and traffic signs in the test data often have very different appearances from the training data, which leads to a model that is trained on one dataset (source domain dataset) and has good performance, but often cannot perform well on another dataset (target domain dataset).

[0004] The most direct way to solve the above problem is to label all the target domain datasets and use them to retrain the target detection model, but the dataset labeling process will be very expensive in terms of manpower and time cost, and it is even impossible to complete when facing actual large-scale datasets. Therefore, how to perform unsupervised training of the model based on unlabeled target domain datasets is of great research value. Summary of the invention

[0005] The purpose of the present invention is to provide an image-text feature fusion domain adaptation target detection method for intelligent driving scenarios.

[0006] The technical solution to achieve the purpose of the present invention is: a method for image-text feature fusion domain adaptation target detection for intelligent driving scenarios, comprising the following steps:

[0007] Step (1): Copy the model pre-trained on the labeled source domain dataset into two identical copies, one as the teacher model and the other as the student model;

[0008] Step (2): Design a text prompt phrase based on the target of interest to be detected and the content of the dataset, and use a text encoder to extract its semantic features;

[0009] Step (3): In the teacher model and student model obtained in step (1), an image-text feature fusion module is added to the shallow layer of the backbone network, and a feature-level style transfer module and a domain discriminator are added to the deep layer of the backbone network of the student model;

[0010] Step (4): Input the text semantic features obtained in step (2) and the simply enhanced target domain data into the teacher model, and use the obtained detection results as pseudo labels of the target domain data;

[0011] Step (5): Input the text semantic features obtained in step (2) and the complex enhanced source domain and target domain data into the student model, and perform model training under the supervision of the source domain real annotations and the target domain pseudo labels obtained in step (4);

[0012] Step (6): After the student model completes a round of training, the parameters of the teacher model are updated using the exponential moving average method based on the parameters of the student model;

[0013] Step (7): Repeat the above steps (3)-(6) until the maximum number of training rounds, and the final teacher model is used as the result model.

[0014] Furthermore, the text prompt phrase designed in step (2) includes descriptions of all the interesting targets that need to be detected, other related targets that may appear in the dataset, and dataset scenes. The description needs to accurately indicate the category name of the target and the scene name of the dataset.

[0015] Furthermore, the step (3) is specifically as follows:

[0016] Step (3-1): Add an image-text feature fusion module to the shallow layer of the backbone network of the teacher model and the student model to make the features domain-independent. The formula for image-text feature fusion is:

[0017]

[0018] In the above formula, ψ s and ψ t Respectively represent the fused source domain and target domain features, x s and x t denote the image data samples of the source domain and the target domain respectively, t is the text prompt phrase, and W q , W k and W v is the projection matrix, is the image feature extractor, is the text encoder, d k It is the unified dimension of image features and text semantic features in the module;

[0019] Step (3-2): Add a feature-level style transfer module to the deep layer of the backbone network of the student model to transform the feature style of the target domain into the feature style of the source domain, and use the domain discriminator for adversarial learning to promote the backbone network to learn domain-independent features. The formula for feature-level style transfer is:

[0020]

[0021] In the above formula, f(·) is the shallow part of the backbone network of the student model, which is used to extract shallow features of the image. μ(·) and σ(·) are the channel mean and standard deviation of the feature, respectively, which are calculated independently in the spatial dimension of each channel of each feature, specifically expressed as μ nc (·) and σ nc (·), the calculation formula is:

[0022]

[0023] In the above formula, x represents the image feature, x∈R N×C×H×W , N is the number of features in the current batch, C is the number of feature channels, H is the feature height, W is the feature width, x nchw is the value at a specific position in the image feature, and ε is a very small number used to avoid division by zero.

[0024] Furthermore, in step (3), VGG16 is used as the backbone network, which includes five layers: vgg0, vgg1, vgg2, vgg3 and vgg4, each of which includes a 3×3 convolution layer, a ReLU activation function and a maximum pooling layer. An image-text feature fusion module is added after the vgg1 layer, and a feature-level style transfer module is added to the vgg4 layer.

[0025] Furthermore, the simple enhancement in step (4) includes image size change and random flipping.

[0026] Furthermore, the complex enhancement in step (5) includes image brightness change, contrast change, saturation change, grayscale change, Gaussian blur processing and random cropping.

[0027] Furthermore, in step (6), the exponential moving average method is used to update the parameters of the teacher model, so that the teacher model gradually adapts to the data of the target domain. The formula of the exponential moving average is:

[0028] θ t ←αθ t +(1-α)θ s

[0029] In the above formula, θ s and θ t They represent the network parameters of the student model and the teacher model respectively, and α∈[0,1) is the momentum coefficient.

[0030] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned image-text feature fusion domain adaptation target detection method for intelligent driving scenarios is implemented.

[0031] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned image-text feature fusion domain adaptation target detection method for intelligent driving scenarios.

[0032] A computer program product includes a computer program, which, when executed by a processor, implements the above-mentioned image-text feature fusion domain adaptation target detection method for intelligent driving scenarios.

[0033] Compared with the prior art, the present invention has the following significant advantages:

[0034] (1) By using the semantic features of text prompt phrases and fusing two different but related modalities, image and text, the features are made more domain-independent, reducing the difference between the features of source domain data and target domain data, thereby improving the adaptability of the object detection model on target domain data.

[0035] (2) There are significant differences between samples in different datasets in terms of lighting, color, geographical environment, shooting angle and resolution. By introducing a feature-level style transfer module and a domain discriminator into the backbone network, the impact of these style differences can be effectively reduced, thereby further improving the adaptability of the object detection model to the target domain data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of the unsupervised domain adaptation target detection method for autonomous driving scenarios of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0038] The present invention proposes a domain-adaptive target detection method for image-text feature fusion for intelligent driving scenarios, which adopts the mean teacher model framework and includes a feature fusion method based on images and texts and a domain difference reduction method based on feature-level style transfer. It mainly includes model pre-training, designing text prompt phrases, adding image-text feature fusion modules and feature-level style transfer modules and domain discriminators, generating pseudo labels through teacher models, training student models using real annotations and pseudo labels, and updating teacher models using exponential moving averages.

[0039] Some exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0040] According to the example of the present invention, a domain adaptation target detection method based on image and text feature fusion for intelligent driving scenarios is proposed to improve the adaptability of the target detection model on the target domain data. Figure 1 As shown, the implementation of this method generally includes the following 7 steps:

[0041] In step A1, the model pre-trained on the labeled source domain dataset is copied into two identical copies, one as the teacher model and the other as the student model.

[0042] In step A2, a text prompt phrase is designed according to the target of interest to be detected and the content of the dataset, and its semantic features are extracted using a text encoder.

[0043] In step A3, in the teacher model and the student model obtained in step A1, an image-text feature fusion module is added to the shallow layer of the backbone network, and a feature-level style transfer module and a domain discriminator are added to the deep layer of the backbone network of the student model.

[0044] In step A4, the text semantic features obtained in step A2 and the simply enhanced target domain data are input into the teacher model, and the obtained detection results are used as pseudo labels of the target domain data.

[0045] In step A5, the text semantic features obtained in step A2 and the complex enhanced source and target domain data are input into the student model, and the model is trained under the supervision of the real annotations of the source domain and the pseudo labels of the target domain obtained in step A4.

[0046] Step A6: After the student model completes a round of training, the parameters of the teacher model are updated using the exponential moving average method based on the parameters of the student model.

[0047] Step A7, repeat the above steps A3-A6 until the maximum training round, and the teacher model finally obtained is used as the result model.

[0048] In the above method, the specific steps of step A1 are: inputting the source domain dataset and the corresponding real annotations into the target detection model for pre-training, and after the training is completed, copying all the network parameters of the model to form two completely identical models, one of which is used as a teacher model and the other as a student model.

[0049] In the above method, the specific steps of step A2 are: according to the target of interest to be detected, other related targets that may appear in the data set and the data set scenes, design a text prompt phrase that can accurately indicate the category of the target and the scene of the data set, input it into the text encoder to generate semantic features and save it.

[0050] In the above method, the specific steps of step A3 are:

[0051] 1) Add an image-text feature fusion module to the shallow layer of the backbone network of the teacher model and the student model to make the features domain-independent. The formula for image-text feature fusion is:

[0052]

[0053] In the above formula, ψ s and ψ t Respectively represent the fused source domain and target domain features, x s and x t denote the image data samples of the source domain and the target domain respectively, t is the text prompt phrase, and W q , W k and W v is the projection matrix, is the image feature extractor, is the text encoder, d k It is the unified dimension of image features and text semantic features in the module.

[0054] 2) Add a feature-level style transfer module to the deep layer of the backbone network of the student model to transform the feature style of the target domain into the feature style of the source domain, and use the domain discriminator for adversarial learning to promote the backbone network to learn domain-independent features. The formula for feature-level style transfer is:

[0055]

[0056] In the above formula, f(·) is the shallow part of the backbone network of the student model, which is used to extract shallow features of the image. μ(·) and σ(·) are the channel mean and standard deviation of the feature, respectively, which are calculated independently in the spatial dimension of each channel of each feature, specifically expressed as μ nc (·) and σ nc (·), the calculation formula is:

[0057]

[0058] In the above formula, x represents the image feature, x∈R N×C×H×W , N is the number of features in the current batch, C is the number of feature channels, H is the feature height, W is the feature width, x nchw is the value at a specific position in the image feature, and ε is a very small number used to avoid division by zero.

[0059] In the above method, the specific steps of step A4 are: after the data of the target domain undergoes simple enhancement operations such as image size change and random flipping, it is input into the teacher model, wherein the image text feature fusion module of the backbone network simultaneously receives the image features of the target domain data extracted by the shallow network and the text semantic features saved in step A2 as input. The final detection result of the teacher model is saved as the pseudo label of the target domain data in the current training round.

[0060] In the above method, the specific steps of step A5 are: after the data of the source domain and the target domain undergo complex enhancement operations such as image brightness change, contrast change, saturation change, grayscale change, Gaussian blur processing and random cropping, they are respectively input into the student model, wherein the image text feature fusion module of the backbone network simultaneously receives the image features of the source domain (target domain) data extracted by the shallow network and the text semantic features saved in step A2 as input, and the feature-level style transfer module converts the target domain feature style in the current training round into the corresponding source domain feature style. The student model uses the real annotations of the source domain and the pseudo labels of the target domain saved in step A4 to calculate the classification and bounding box regression loss functions, and performs back propagation and parameter update through the stochastic gradient descent method.

[0061] In the above method, the specific steps of step A6 are: using the exponential moving average method to update the parameters of the teacher model, so that the teacher model gradually adapts to the data of the target domain. The formula of the exponential moving average is:

[0062] θ t ←αθ t +(1-α)θ s

[0063] In the above formula, θ s and θ t They represent the network parameters of the student model and the teacher model respectively, and α∈[0,1) is the momentum coefficient.

[0064] In summary, the present invention proposes a domain-adaptive target detection method for image-text feature fusion for intelligent driving scenarios. In view of the problems of cross-domain performance degradation and high data set annotation cost in autonomous driving target detection models due to factors such as weather, regional environment and equipment, the present invention designs an image-text feature fusion module to make the features domain-independent, and reduces the impact of style differences through a feature-level style transfer module. Specifically, it includes: pre-training the model, designing text prompt phrases and extracting semantic features, generating pseudo labels using a teacher model with an image-text feature fusion module added, training a student model with an additional feature-level style transfer module added, and updating the teacher model through an exponential moving average method as the final result. Using the method proposed in the present invention, the adaptability of the target detection model on the target domain data can be effectively improved, the degradation of model performance caused by domain differences can be reduced, and the annotation cost of the data set can be significantly reduced.

Claims

1. A domain-adaptive target detection method based on image-text feature fusion for intelligent driving scenarios, characterized in that: The steps include: Step 1: Copy the model pre-trained on the annotated source domain dataset into two identical copies, one as the teacher model and the other as the student model; Step 2: Design a text prompt phrase based on the target of interest to be detected and the content of the dataset, and use a text encoder to extract its semantic features; Step 3: In the teacher model and student model obtained in step 1, add an image-text feature fusion module to the shallow layer of the backbone network, and add a feature-level style transfer module and a domain discriminator to the deep layer of the backbone network of the student model; Step 4: Input the text semantic features obtained in step 2 and the simply enhanced target domain data into the teacher model, and the obtained detection results are used as pseudo labels for the target domain data; Step 5: Input the text semantic features obtained in step 2 and the complex enhanced source and target domain data into the student model, and perform model training under the supervision of the source domain real annotations and the target domain pseudo labels obtained in step 4; Step 6: After the student model completes a round of training, the parameters of the teacher model are updated using the exponential moving average method based on the parameters of the student model; Step 7: Repeat the above steps 3-6 until the maximum number of training rounds, and the final teacher model is used as the result model.

2. The method according to claim 1, characterized in that The text prompt phrase designed in step 2 includes descriptions of all interesting targets that need to be detected, other related targets that may appear in the dataset, and dataset scenes. The description needs to accurately indicate the category name of the target and the scene name of the dataset.

3. The method according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3-1: Add an image-text feature fusion module to the shallow layer of the backbone network of the teacher model and the student model to make the features domain-independent. The formula for image-text feature fusion is: In the above formula, ψ s and ψ t Respectively represent the fused source domain and target domain features, x s and x t denote the image data samples of the source domain and the target domain respectively, t is the text prompt phrase, and W q , W k and W v is the projection matrix, is the image feature extractor, is the text encoder, d k It is the unified dimension of image features and text semantic features in the module; Step 3-2: Add a feature-level style transfer module to the deep layer of the backbone network of the student model to transform the feature style of the target domain into the feature style of the source domain, and use the domain discriminator for adversarial learning to promote the backbone network to learn domain-independent features. The formula for feature-level style transfer is: In the above formula, f(·) is the shallow part of the backbone network of the student model, which is used to extract shallow features of the image. μ(·) and σ(·) are the channel mean and standard deviation of the feature, respectively, which are calculated independently in the spatial dimension of each channel of each feature, specifically expressed as μ nc (·) and σ nc (·), the calculation formula is: In the above formula, x represents the image feature, x∈R N×C×H×W , N is the number of features in the current batch, C is the number of feature channels, H is the feature height, W is the feature width, x nchw is the value at a specific position in the image feature, and ε is a very small number used to avoid division by zero.

4. The method according to claim 1, characterized in that: In the step 3, VGG16 is used as the backbone network, which includes five layers: vgg0, vgg1, vgg2, vgg3 and vgg4. Each layer includes a 3×3 convolution layer, a ReLU activation function and a maximum pooling layer; an image-text feature fusion module is added after the vgg1 layer, and a feature-level style transfer module is added to the vgg4 layer.

5. The method according to claim 1, characterized in that The simple enhancement in step 4 includes image size change and random flipping.

6. The method according to claim 1, characterized in that The complex enhancement in step 5 includes image brightness change, contrast change, saturation change, grayscale change, Gaussian blur processing and random cropping.

7. The method according to claim 1, characterized in that In step 6, the exponential moving average method is used to update the parameters of the teacher model, so that the teacher model gradually adapts to the data of the target domain. The formula of the exponential moving average is: i t ←th t +(1-α)θ s In the above formula, θ s and θ t They represent the network parameters of the student model and the teacher model respectively, and α∈[0,1) is the momentum coefficient.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images