Zero sample cross-domain target detection method and system based on feature remapping

Abstract

The invention provides a zero-sample cross-domain target detection method and system based on feature remapping, and relates to the technical field of image processing, and the method comprises the steps: obtaining a to-be-detected target domain image; inputting a to-be-detected target domain image into the final domain adaptive target detection model to obtain a detection result of the to-be-detected target domain image; wherein the final domain adaptive target detection model is obtained by training a zero-sample cross-domain target detection model based on feature remapping by adopting training data, and the training data comprises a source domain data set containing image samples and labels and a target domain language description set. According to the method, the specific technical features, which are crucial to the detection task, of the target domain are obtained through the target domain language description set, the feature learning problem under the zero target domain sample in the domain adaptive target detection task is solved through the zero sample cross-domain target detection model based on feature remapping, and the target detection performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a zero-sample cross-domain target detection method and system based on feature remapping. Background Art

[0002] Object detection technology aims to locate and classify objects simultaneously. This technology is widely used in various scenarios such as security monitoring and autonomous driving. However, due to the inherent complexity and diversity of real-world data, there is a domain shift problem between actual data and training data, which can lead to performance degradation of the model when it is actually deployed.

[0003] In order to solve this problem, existing technologies have explored multiple ways to mitigate the impact of domain shift. One way is to use unsupervised domain adaptation technology to reduce the distribution difference between the source domain and the target domain. However, unsupervised domain adaptation technology requires access to data in the source domain and the target domain during the adaptation process. In real applications, it is not realistic to access data in the target domain and cannot be implemented in practice. Another way is to use domain generalization technology, which does not require access to data in the target domain and only trains on source domain data. However, training based only on source domain data focuses on mining feature representations that are as general as possible to achieve feature sharing between different domains, ignoring subtle features that are unique to the target domain and are critical to the detection task, such as changes in lighting conditions, background complexity, or external appearance, which may not be fully covered in the source domain. As a result, when the model faces new domain data, it does not fully express the features of specific changes, and has problems such as poor targeting and weak adaptability to target domain features, which affects the accuracy and robustness of detection. Summary of the invention

[0004] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a zero-sample cross-domain target detection method based on feature remapping, comprising:

[0005] Obtain the target domain image to be detected;

[0006] Input the target domain image to be detected into the final domain adaptation target detection model to obtain the detection result of the target domain image to be detected;

[0007] The final domain-adapted object detection model is obtained by training a zero-shot cross-domain object detection model based on feature remapping using training data, and the training data includes: a source domain dataset containing image samples and labels and a target domain language description set.

[0008] Preferably, the zero-shot cross-domain object detection model based on feature remapping includes a student model and a teacher model;

[0009] The training process of the final domain adaptation object detection model includes:

[0010] Get training data;

[0011] The basic image features of the image samples are obtained through the student model, and the detection loss of the student model is calculated based on the basic image features;

[0012] The basic image feature encoding of the basic image features is obtained through the student model, and the standard image feature encoding of the image sample is obtained through the teacher model. The standard image feature encoding and the basic image feature encoding are compared for similarity to obtain the knowledge distillation loss;

[0013] The text feature encoding of the target domain language description set is obtained through the teacher model, and the standard image feature encoding and the basic image feature encoding are compared with the text feature encoding for similarity to obtain the deviation correction loss;

[0014] The student model is trained according to the detection loss, knowledge distillation loss, and bias correction loss to obtain the final domain-adapted object detection model.

[0015] Preferably, the student model includes: a student image feature extraction network, a feature remapping network, and a student object detection head network;

[0016] The basic image features of the image samples are obtained through the student model, and the detection loss of the student model is calculated based on the basic image features, including:

[0017] Input the image sample into the student image feature extraction network to obtain the initial basic image features of the image sample;

[0018] Input the initial basic image features into the feature remapping network to obtain the basic image features after feature remapping;

[0019] The basic image features are input into the student object detection head network for object detection, and the obtained detection results are compared with the labels in the source domain dataset to obtain the detection loss.

[0020] Preferably, the obtained detection results include classification detection results and regression detection results; the labels in the source domain data set include category labels and position labels;

[0021] Compare the obtained detection results with the labels in the source domain dataset to obtain the detection loss, including:

[0022] Compare the classification test results with the category labels to obtain the classification loss;

[0023] Compare the regression detection results with the position labels to obtain the regression loss;

[0024] The detection loss is determined as the sum of the classification loss and the regression loss.

[0025] Preferably, the teacher model includes a teacher image feature extraction network, a feature remapping network, and a teacher image feature encoding network;

[0026] The standard image feature encoding of image samples is obtained through the teacher model, including:

[0027] Input the image sample into the teacher image feature extraction network to obtain the initial standard image features of the image sample;

[0028] Input the initial standard image features into the feature remapping network to obtain the standard image features after feature remapping;

[0029] The standard image features are input into the teacher image feature encoding network to obtain the standard image feature encoding of the image sample.

[0030] Preferably, the student model includes a scale alignment network and a teacher image feature encoding network;

[0031] The basic image feature encoding of basic image features obtained through the student model includes:

[0032] Inputting the basic image features into the scale alignment network to obtain the basic image features with the same size as the standard image features, and determining the basic image features with the same size as the standard image features as the scale basic image features;

[0033] The scaled basic image features are input into the teacher image feature encoding network to obtain the basic image feature encoding.

[0034] Preferably, the teacher model further comprises: a teacher text encoding network; the target domain language description set comprises a target domain description set and an object description set;

[0035] The text feature encoding of the target domain language description set is obtained through the teacher model, and the standard image feature encoding and the basic image feature encoding are compared with the text feature encoding for similarity to obtain the deviation correction loss, including:

[0036] Inputting the target domain description set into the teacher text encoding network to obtain the domain text feature encoding; inputting the object description set into the teacher text encoding network to obtain the class text feature encoding;

[0037] Compare the similarity between the standard image feature encoding and the domain text feature encoding to obtain the domain deviation correction loss of the teacher model; compare the similarity between the standard image feature encoding and the class text feature encoding to obtain the class deviation correction loss of the teacher model;

[0038] The basic image feature encoding is compared with the domain text feature encoding for similarity to obtain the student model domain deviation correction loss; the basic image feature encoding is compared with the class text feature encoding for similarity to obtain the student model class deviation correction loss;

[0039] The sum of the domain bias correction loss of the teacher model and the domain bias correction loss of the student model is determined as the domain bias correction loss; the sum of the class bias correction loss of the teacher model and the class bias correction loss of the student model is determined as the class bias correction loss; the sum of the domain bias correction loss and the class bias correction loss is determined as the bias correction loss; the formula for the domain bias correction loss of the teacher model or the domain bias correction loss of the student model is as follows:

[0040]

[0041] in, represents the teacher domain bias correction loss or the student domain bias correction loss, represents the number of prompt templates in the target domain description set, x feat is the standard image feature or basic image feature, E attn is the teacher image feature encoding network, E attn (x feat ) is the standard image feature coding or basic image feature coding, E text For the teacher text encoding network, r i is the value of the i-th element in the target domain description set, E text (r i ) is the domain text feature encoding, g is the similarity measurement function;

[0042] The formula for the teacher class bias correction loss or the student class bias correction loss is as follows:

[0043]

[0044] in, is the teacher bias correction loss or the student bias correction loss, is the number of prompt templates in the object description set, x feat is the standard image feature or basic image feature, E attn is the teacher image feature encoding network, E attn (x feat ) is the standard image feature coding or basic image feature coding, E text is the teacher text encoding network, g is the similarity measurement function, r(c j ) is the jth element in the object description set, and its category is c, E text (r(c j )) is the text-like feature encoding.

[0045] Preferably, the process of obtaining the target domain description set includes:

[0046] Establish a common prompt template that describes the source and target domains;

[0047] The specific description of the target domain relative to the source domain is combined with the common prompt template to obtain a target domain description set.

[0048] Preferably, the feature remapping network comprises:

[0049] Construct the first branch of extracting and transforming local features by connecting multiple small-scale convolution kernels in series;

[0050] A second branch is constructed that connects pooling layers and multi-layer perceptrons in series to adjust the global mapping across channel dimensions;

[0051] The first branch is fused with the second branch to obtain a feature remapping network.

[0052] Preferably, the target domain image to be detected is input into the final domain adaptation target detection model to obtain the detection result of the target domain image to be detected, including:

[0053] Input the target domain image to be detected into the student image feature extraction network to obtain the basic features of the target domain image of the target domain image to be detected;

[0054] Input the basic features of the target domain image into the student target detection head network to perform target detection, and obtain the detection result of the target domain image to be detected;

[0055] The detection result includes at least one of the target frame pixel position, category, and confidence level.

[0056] Based on the same inventive concept, the present invention also provides a zero-sample cross-domain target detection system based on feature remapping, including: a target domain image acquisition module, a detection result acquisition module;

[0057] A target domain image acquisition module is used to acquire a target domain image to be detected;

[0058] A detection result acquisition module is used to input the target domain image to be detected into the final domain adaptation target detection model to obtain the detection result of the target domain image to be detected;

[0059] Among them, the final domain adaptation target detection model is obtained by training the zero-shot cross-domain target detection model based on feature remapping using training data, and the training data includes: a source domain dataset containing image samples and labels and a target domain language description set.

[0060] Preferably, the system further comprises: a zero-sample cross-domain target detection model construction module based on feature remapping,

[0061] The zero-shot cross-domain object detection model based on feature remapping includes a student model and a teacher model;

[0062] The zero-shot cross-domain object detection model building modules based on feature remapping include:

[0063] A training data acquisition submodule is used to acquire training data;

[0064] A detection loss acquisition submodule is used to obtain basic image features of image samples through a student model, and calculate the detection loss of the student model based on the basic image features;

[0065] The knowledge distillation loss acquisition submodule is used to obtain the basic image feature encoding of the basic image features through the student model, obtain the standard image feature encoding of the image sample through the teacher model, compare the similarity between the standard image feature encoding and the basic image feature encoding, and obtain the knowledge distillation loss;

[0066] The deviation correction loss acquisition submodule is used to obtain the text feature encoding of the target domain language description set through the teacher model, and compare the similarity between the standard image feature encoding and the basic image feature encoding and the text feature encoding to obtain the deviation correction loss;

[0067] The final domain-adapted target detection model acquisition submodule is used to train the student model according to the detection loss, knowledge distillation loss, and bias correction loss to obtain the final domain-adapted target detection model.

[0068] Preferably, the student model includes: a student image feature extraction network, a feature remapping network, and a student object detection head network;

[0069] The detection loss acquisition submodule includes:

[0070] An initial basic image feature acquisition unit, used for inputting an image sample into a student image feature extraction network to obtain an initial basic image feature of the image sample;

[0071] A basic image feature acquisition unit, used for inputting the initial basic image features into a feature remapping network to obtain the basic image features after feature remapping;

[0072] The detection loss acquisition unit is used to input basic image features into the student target detection head network for target detection, compare the obtained detection results with the labels in the source domain dataset, and obtain the detection loss.

[0073] Preferably, the detection results obtained in the detection loss acquisition unit include classification detection results and regression detection results; the labels in the source domain data set include category labels and position labels;

[0074] The detection loss acquisition unit is specifically used to: compare the classification detection result with the category label to obtain the classification loss;

[0075] Compare the regression detection results with the position labels to obtain the regression loss;

[0076] The detection loss is determined as the sum of the classification loss and the regression loss.

[0077] Preferably, the teacher model includes a teacher image feature extraction network, a feature remapping network, and a teacher image feature encoding network;

[0078] The knowledge distillation loss acquisition submodule is used to:

[0079] Input the image sample into the teacher image feature extraction network to obtain the initial standard image features of the image sample;

[0080] Input the initial standard image features into the feature remapping network to obtain the standard image features after feature remapping;

[0081] The standard image features are input into the teacher image feature encoding network to obtain the standard image feature encoding of the image sample.

[0082] Preferably, the student model includes a scale alignment network and a teacher image feature encoding network;

[0083] The knowledge distillation loss submodule is also used for:

[0084] Inputting the basic image features into the scale alignment network to obtain the basic image features with the same size as the standard image features, and determining the basic image features with the same size as the standard image features as the scale basic image features;

[0085] The scaled basic image features are input into the teacher image feature encoding network to obtain the basic image feature encoding.

[0086] Preferably, the teacher model further comprises: a teacher text encoding network; the target domain language description set comprises a target domain description set and an object description set;

[0087] The bias correction loss acquisition submodule includes:

[0088] The quasi-text feature coding acquisition unit is used to input the target domain description set into the teacher text coding network to obtain the domain text feature coding; input the object description set into the teacher text coding network to obtain the quasi-text feature coding;

[0089] The teacher model bias correction loss acquisition unit is used to compare the similarity between the standard image feature encoding and the domain text feature encoding to obtain the teacher model domain bias correction loss; compare the similarity between the standard image feature encoding and the class text feature encoding to obtain the teacher model class bias correction loss;

[0090] The student model bias correction loss acquisition unit is used to compare the similarity between the basic image feature encoding and the domain text feature encoding to obtain the student model domain bias correction loss; compare the similarity between the basic image feature encoding and the class text feature encoding to obtain the student model class bias correction loss;

[0091] The bias correction loss acquisition unit is used to determine the sum of the domain bias correction loss of the teacher model and the domain bias correction loss of the student model as the domain bias correction loss; determine the sum of the class bias correction loss of the teacher model and the class bias correction loss of the student model as the class bias correction loss; and determine the sum of the domain bias correction loss and the class bias correction loss as the bias correction loss;

[0092] The formula for the teacher model domain bias correction loss or the student model domain bias correction loss is as follows:

[0093]

[0094] in, represents the teacher domain bias correction loss or the student domain bias correction loss, represents the number of prompt templates in the target domain description set, x feat is the standard image feature or basic image feature, E attn is the teacher image feature encoding network, E attn (x feat ) is the standard image feature coding or basic image feature coding, E text For the teacher text encoding network, r i is the value of the i-th element in the target domain description set, E text (r i ) is the domain text feature encoding, g is the similarity measurement function;

[0095] The formula for the teacher class bias correction loss or the student class bias correction loss is as follows:

[0096]

[0097] in, is the teacher bias correction loss or the student bias correction loss, is the number of prompt templates in the object description set, x feat is the standard image feature or basic image feature, E attn is the teacher image feature encoding network, E attn (x feat ) is the standard image feature coding or basic image feature coding, E text is the teacher text encoding network, g is the similarity measurement function, r(c j ) is the jth element in the object description set, and its category is c, E text(r(c j )) is the text-like feature encoding.

[0098] Preferably, the system further comprises: a training data acquisition module;

[0099] The training data acquisition module includes a target domain description set acquisition submodule, which is specifically used for:

[0100] Establish a common prompt template that describes the source and target domains;

[0101] The specific description of the target domain relative to the source domain is combined with the common prompt template to obtain a target domain description set.

[0102] Preferably, the system further comprises a feature remapping network acquisition module, which is used to:

[0103] Construct the first branch of extracting and transforming local features by connecting multiple small-scale convolution kernels in series;

[0104] A second branch is constructed that connects pooling layers and multi-layer perceptrons in series to adjust the global mapping across channel dimensions;

[0105] The first branch is fused with the second branch to obtain a feature remapping network.

[0106] Preferably, the detection result acquisition unit is specifically used for:

[0107] Input the target domain image to be detected into the student image feature extraction network to obtain the basic features of the target domain image of the target domain image to be detected;

[0108] Input the basic features of the target domain image into the student target detection head network to perform target detection, and obtain the detection result of the target domain image to be detected;

[0109] The detection result includes at least one of the target frame pixel position, category, and confidence level.

[0110] Based on the same inventive concept, the present invention also provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0111] The memory is used to store one or more programs;

[0112] When the one or more programs are executed by the at least one processor, a zero-sample cross-domain target detection method based on feature remapping as described above is implemented.

[0113] Based on the same inventive concept, the present invention also provides a readable storage medium on which an execution program is stored. When the execution program is executed, a zero-sample cross-domain target detection method based on feature remapping as described above is implemented.

[0114] Compared with the closest prior art, the present invention has the following beneficial effects:

[0115] The present invention provides a zero-sample cross-domain target detection method based on feature remapping, comprising: obtaining a target domain image to be detected; inputting the target domain image to be detected into a final domain adaptation target detection model to obtain a detection result of the target domain image to be detected; wherein the final domain adaptation target detection model is obtained by training a zero-sample cross-domain target detection model based on feature remapping using training data, and the training data includes: a source domain data set containing image samples and labels and a target domain language description set. The present invention obtains technical features that are unique to the target domain and are crucial to the detection task through the target domain language description set, and solves the feature learning problem under zero target domain samples in the domain adaptation target detection task through the zero-sample cross-domain target detection model based on feature remapping, thereby improving the target detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] Figure 1 A flowchart of a zero-sample cross-domain target detection method based on feature remapping is provided for the present invention;

[0117] Figure 2 A schematic diagram of a sample of the source domain data set provided by the present invention;

[0118] Figure 3 A schematic diagram of a specific implementation process of the zero-sample cross-domain target detection method based on feature remapping provided by the present invention;

[0119] Figure 4 A schematic diagram of the structure of the feature remapping network provided by the present invention;

[0120] Figure 5 This is an example diagram of the maximum pooling provided by the present invention;

[0121] Figure 6 An example diagram of average pooling provided by the present invention;

[0122] Figure 7 An example diagram of a scale alignment network provided by the present invention;

[0123] Figure 8 A schematic diagram of a detection process of a target domain image to be detected provided by the present invention;

[0124] Fig. 9 A structural diagram of a zero-sample cross-domain target detection system based on feature remapping provided by the present invention;

[0125] Fig.10 A schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0126] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0127] In view of the problems existing in the existing technology, it is necessary to establish a model that can achieve better adaptability in the target domain without the need for target domain data during training. Some works have achieved remarkable results on zero-shot vision tasks, including object classification, object segmentation, and object detection, using large cross-modal vision-language models (VLMs). Large cross-modal models are trained on a large number of image-text pairs and are good at aligning language descriptions with visual features in various contexts. Although most of the parameters are frozen, excellent results can be achieved in various vision tasks by simply adding additional modules or fine-tuning. However, directly using the pre-trained weights of large cross-modal models for real-time target detection tasks, such as object detection, consumes huge computing resources and is not very practical in resource-constrained environments. Studies have shown that by extracting knowledge from large cross-modal models, a smaller model can be trained to classify objects in previously unseen target domains. However, object detection poses a more challenging problem, requiring the identification of richer semantic information from local features for accurate localization and classification. Using the semantic knowledge of a large cross-modal model to guide a small-scale model can solve the problem of detecting objects in target domain images under zero-shot conditions.

[0128] Embodiment 1:

[0129] The present invention provides a zero-sample cross-domain target detection method based on feature remapping, specifically, Figure 1 A flowchart of a zero-sample cross-domain target detection method based on feature remapping provided by an embodiment of the present invention includes the following steps as shown in the figure:

[0130] S1: Obtain the target domain image to be detected;

[0131] S2: Input the target domain image to be detected into the final domain adaptation target detection model to obtain the detection result of the target domain image to be detected;

[0132] The final domain-adapted object detection model is obtained by training a zero-shot cross-domain object detection model based on feature remapping using training data, and the training data includes: a source domain dataset containing image samples and labels and a target domain language description set.

[0133] The present invention obtains technical features that are unique to the target domain and crucial to the detection task through a target domain language description set, and solves the feature learning problem under zero target domain samples in the domain adaptation target detection task through a zero-sample cross-domain target detection model based on feature remapping, thereby improving the target detection performance.

[0134] The final domain-adapted object detection model in S2 is obtained by training the zero-shot cross-domain object detection model based on feature remapping through the training data.

[0135] The training data includes: source domain data set and target domain language description set.

[0136] Among them, the source domain dataset contains source domain image samples, category labels and location labels of the targets of interest in the source domain images; the target domain language description set includes the target domain description set and the object description set.

[0137] The image samples in the source domain dataset are visible light images in the common RGB (red-green-blue) color mode. The target of interest in the image is the target to be detected, and its category label is such as car, person, train, bicycle, etc. The position label is a rectangular box that can surround the target of interest, specifically the coordinates of the center point of the rectangular box surrounding the target of interest and the length and width of the rectangular box. Figure 2 As shown, it is a schematic diagram of a sample of the source domain dataset provided by an embodiment of the present invention. The whole picture is an image sample, and the category label of interest in the image is car. The category label of interest in the image is surrounded by rectangular boxes, and the coordinates of the center point of the rectangular box and the length and width of the rectangular box are recorded in the image information. The source domain dataset includes multiple image samples, and the category labels and position labels of the objects of interest in the images corresponding to the multiple image samples.

[0138] The target domain language description set includes the target domain description set and the object description set. Among them, the target domain description set contains image target description and target action description. The text format of these two descriptions follows a fixed template, and covers all targets to be detected by changing keywords, and highlights the difference between target domain images and source domain images. The target of the image target description follows the template "a photo of a{style}street scene", where {style} represents the domain style keyword. In order to enrich the prompts, WordNet (lexical database) is used to query synonyms of domain style. For example, in a foggy scene during the day, we use {foggy, misty, hazy} as style keywords. In addition, a template for describing target actions is set up: "driving on{style}roads", where {style} represents the domain style keyword.

[0139] Construct a target domain description set R d, each hint r∈R in the set d Both contain a specific key description for the target domain relative to the source domain, which is semantically similar but has different representations.

[0140] Specifically, a common prompt template that describes both the source domain and the target domain is established, and a set of semantically similar key terms is constructed based on WordNet. Combine the specific description of the target domain relative to the source domain with the common prompt template to generate the target domain description set R d : Where r represents the common prompt template.

[0141] For example, r(k) = "a photo taken on a{k}day", where k represents the key descriptive term of the domain, which indicates the day on which the photo was taken. k can be Daytime-Sunny (clear daytime), Night-Sunny (clear night), Dusk-Rainy (rainy dusk), etc.

[0142] At the same time, the object description set R is constructed through a simple and general template c ={r(c)}, where c represents the category of the object to be detected. For example, r(c) = "a photo of {c}".

[0143] The formula for constructing the category mean encoding is as follows:

[0144]

[0145] in, is the number of prompt templates in the object description set, E text is the teacher text encoding network, r(c j ) is the jth element in the object description set, and its category is c, E text (r(c j )) is the text-like feature encoding.

[0146] The zero-shot cross-domain object detection model based on feature remapping includes a student model and a teacher model.

[0147] The process of training the zero-shot cross-domain object detection model based on feature remapping through training data includes:

[0148] Obtain training data, which includes a source domain dataset containing image samples and labels and a target domain language description set;

[0149] The basic image features of the image samples are obtained through the student model, and the detection loss of the student model is calculated based on the basic image features;

[0150] The basic image feature encoding of the basic image features is obtained through the student model, and the standard image feature encoding of the image sample is obtained through the teacher model. The standard image feature encoding and the basic image feature encoding are compared for similarity to obtain the knowledge distillation loss;

[0151] The text feature encoding of the target domain language description set is obtained through the teacher model, and the standard image feature encoding and the basic image feature encoding are compared with the text feature encoding for similarity to obtain the deviation correction loss;

[0152] The student model is trained according to the detection loss, knowledge distillation loss, and bias correction loss to obtain the final domain-adapted object detection model.

[0153] In a specific embodiment, the training process includes the following steps: initializing the parameters of a zero-sample cross-domain target detection model based on feature remapping; obtaining basic image features of image samples through a student model, and calculating the detection loss of the student model based on the basic image features; obtaining basic image feature encoding of basic image features through the student model, obtaining standard image feature encoding of image samples through a teacher model, comparing the similarity between the standard image feature encoding and the basic image feature encoding, and obtaining knowledge distillation loss; obtaining text feature encoding of a target domain language description set through a teacher model, comparing the similarity between the standard image feature encoding and the basic image feature encoding and the text feature encoding, respectively, and obtaining deviation correction loss; updating the parameters of the student model based on the results of the various loss functions obtained, detection loss, knowledge distillation loss, and deviation correction loss, and obtaining a final domain-adapted target detection model based on the updated parameters.

[0154] like Figure 3 , which is a schematic diagram of the specific implementation process of the zero-sample cross-domain target detection method based on feature remapping provided by the present invention.

[0155] Specifically, the process of obtaining the detection loss is as follows:

[0156] The student model includes a student image feature extraction network, a feature remapping network, and a student target detection head network; a source domain image is selected as an image sample, and the image sample is input into the student image feature extraction network to obtain the initial basic image features of the image sample; the initial basic image features of the image sample are input into the feature remapping network to obtain the basic image features after feature remapping; the basic image features are then input into the student target detection head network for target detection, and the obtained detection results are compared with the category labels and position labels in the source domain dataset to obtain the detection loss results. Among them, the obtained detection results include classification detection results and regression detection results, and the classification detection results are compared with the category labels to obtain the classification loss; the regression detection results are compared with the position labels to obtain the regression loss; the sum of the classification loss and the regression loss is determined as the detection loss.

[0157] The classification loss is used to optimize the model's ability to identify the target category. In the present invention, the classification loss can be a cross entropy loss. The regression loss is used to optimize the model's prediction accuracy for the target position. In the present invention, the regression loss can be an L1 loss. The detection loss can be express.

[0158] The student image feature extraction network is used to extract the features of the image in the source domain dataset and obtain the initial basic image features. The student target detection head network is used to detect the target based on the basic image features output by the feature remapping network. The student target detection head network specifically includes: region proposal network and target detection network; the region proposal network is used to generate candidate target regions based on the basic image features; the target detection network is used to further classify and regress the candidate target regions to achieve accurate target detection.

[0159] Specifically, the process of obtaining knowledge distillation loss is as follows:

[0160] The teacher model includes a teacher image feature extraction network, a feature remapping network and a teacher image feature encoding network; the standard image feature encoding of the image sample is obtained through the teacher model, including: using the same source domain image used in the process of detecting loss as the image sample for teacher model training, inputting the image sample into the teacher image feature extraction network to obtain the initial standard image feature of the image sample, inputting the initial standard image feature into the feature remapping network to obtain the standard image feature after feature remapping; and then inputting the standard image feature into the teacher image feature encoding network to obtain the standard image feature encoding of the image sample.

[0161] At the same time, it is also necessary to obtain the basic image feature encoding of the image sample. The student model includes a scale alignment network and a teacher image feature encoding network; the basic image features of the image sample obtained in the process of detecting the loss are used to input the basic image features into the scale alignment network to obtain the basic image features of the same size as the standard image features, and the basic image features of the same size as the standard image features are determined as scale basic image features; the scale basic image features are then input into the teacher image feature encoding network to obtain the basic image feature encoding of the image sample.

[0162] After obtaining the standard image feature coding and basic image feature coding of the image sample, the basic image feature coding of the image sample is compared with the standard image feature coding for similarity to obtain the knowledge distillation loss.

[0163] Among them, the teacher image feature extraction network is used to extract the features of images in the source domain dataset and obtain the initial standard image features.

[0164] The teacher image feature encoding network is used to convert the initial standard image features or initial basic image features output by the teacher image feature extraction network or the student image feature extraction network into a fixed-size standard image feature encoding or basic image feature encoding.

[0165] The feature remapping network in the student model and the feature remapping network in the teacher model both use a deep convolutional neural network structure, which can learn the complex mapping relationship between the source domain feature distribution and the target domain feature distribution, remap the source domain image feature space to the target domain image feature space, and optimize the network parameters through the back propagation algorithm. The feature remapping network in the student model has the same structure as the feature remapping network in the teacher model, but the parameters are not shared.

[0166] The construction process of the feature remapping network (FR) is as follows: first, a branch is constructed, which extracts and transforms local features by connecting multiple small-scale convolution kernels in series, and at the same time, another branch is constructed, which combines the pooling layer and the multi-layer perceptron to adjust the global mapping across channel dimensions. Then, the outputs of these two branches are fused to obtain the feature remapping network.

[0167] like Figure 4 The figure is a schematic diagram of the structure of the feature remapping network provided by the present invention. Conv3×3 represents a 3×3 convolution kernel, i.e., a small-scale convolution kernel; Ave Pooling represents average pooling, Max Pooling represents maximum pooling, and MLP (Multi-Layer perceptron) represents a multi-layer perceptron.

[0168] The purpose of pooling is to sample and reduce the dimension of the input features to speed up the calculation.

[0169] Among them, the maximum pooling is similar to the convolution process. For the values ​​in the neighborhood of a high-dimensional feature map, a lower-dimensional filter is used to scan with a fixed step size, and the maximum value in the scan result is selected and output to the next layer. Figure 5 As shown in FIG. 1 , an example diagram of the maximum pooling provided by the present invention is shown. The values ​​of the first block scanned are 1, 1, 5, and 6 in sequence, with the maximum value being 6; scan one by one in this manner, and select the maximum value of the scan result to output to the next layer.

[0170] Average pooling is to use a lower-dimensional filter to scan the values ​​in the neighborhood of a high-dimensional feature map with a fixed step size to obtain the average value of the scan results. The average value of the scan results is input to the next layer. Figure 6 The figure shows an example of average pooling provided by the present invention. The values ​​of the first block scanned are 1, 1, 5, and 6 in sequence, and the calculated average value is 3.25; scan one by one in this way, and select the average value of the scan result to output to the next layer.

[0171] The role of average pooling and maximum pooling is to reduce parameters and calculations while maintaining image features to prevent overfitting. That is, the pooling layer is a process of feature selection and information filtering, which improves computing performance by losing some information.

[0172] The initial standard image features or initial basic image features obtained by multiple small-scale convolution kernels are extracted and local features are converted; the results of average pooling and maximum pooling are input into a multi-layer perceptron, and the output results of the multi-layer perceptron are operated with the results of multiple small-scale convolution kernels to obtain standard image features or basic image features of the image samples.

[0173] On this basis, an additional scale alignment network is connected after the feature remapping of the student model to adjust the output basic image features to the same size as the standard image features. Figure 7 The figure shows a schematic diagram of the structure of the scale alignment network (SA) provided by the present invention. The basic image features after feature remapping are processed by the Conv1×1 small-scale convolution kernel, and then by adaptive pooling (Adaptive Pooling) to obtain basic image features with the same size as the standard image features. Specifically, the basic image features are input into the scale alignment network to obtain basic image features with the same size as the standard image features, and the basic image features with the same size as the standard image features are determined as scale basic image features; the scale basic image features are then input into the teacher image feature encoding network to obtain the basic image feature encoding of the image sample.

[0174] The basic image feature encoding of the image sample is compared with the standard image feature encoding for similarity to obtain the knowledge distillation loss. The feature encoding capability of the large cross-modal model is transferred to the student model, so that the student model can map the source domain image features to the target feature space, and then when the target domain image to be detected is input into the final domain adaptation target detection model, the student target detection head network can achieve accurate classification and positioning. The knowledge distillation loss can be used It represents L1 loss, that is, mean absolute error (MAE) loss.

[0175] The teacher model also includes: a teacher text encoding network; the teacher text encoding network is used to convert the text of the target domain language description set into a fixed-size teacher text feature encoding, and the teacher text feature encoding and the standard image feature or basic image feature encoding can measure the correspondence in a common feature space.

[0176] Specifically, the process of obtaining the bias correction loss is as follows:

[0177] The target domain language description set includes a target domain description set and an object description set; the target domain description set is input into the teacher text encoding network to obtain the domain text feature encoding for the target domain description set; at the same time, the object description set is input into the teacher text encoding network to obtain the quasi-text feature encoding for the object description set; further, the standard image feature encoding is compared with the domain text feature encoding for the target domain description set to obtain the teacher model domain deviation correction loss; the standard image feature encoding is compared with the quasi-text feature encoding for the object description set to obtain the teacher model class deviation correction loss; the basic image feature encoding is compared with the domain text feature encoding for the target domain description set to obtain the student model domain deviation correction loss; the basic image feature encoding is compared with the quasi-text feature encoding for the object description set to obtain the student model class deviation correction loss. Wherein, the similarity comparison can be calculated based on cosine similarity. The sum of the teacher model domain bias correction loss and the student model domain bias correction loss is determined as the domain bias correction loss, the sum of the teacher model class bias correction loss and the student model class bias correction loss is determined as the class bias correction loss, and the sum of the domain bias correction loss and the class bias correction loss is determined as the bias correction loss. The domain bias correction loss includes the teacher model domain bias correction loss and the student model domain bias correction loss, which can be used Indicates that the class deviation correction loss includes the teacher model class deviation correction loss and the student model class deviation correction loss, which can be used express.

[0178] The teacher model domain deviation correction loss or the student model domain deviation correction loss is used to optimize the mapping effect of the overall characteristics of the source domain image sample from the source domain to the target domain. The calculation formula is as follows:

[0179]

[0180] in, represents the teacher domain bias correction loss or the student domain bias correction loss, represents the number of prompt templates in the target domain description set, x feat is the standard image feature or basic image feature, E attn is the teacher image feature encoding network, E attn (x feat ) is the standard image feature coding or basic image feature coding, E text For the teacher text encoding network, r i is the value of the i-th element in the target domain description set, E text (r i ) is the domain text feature encoding, and g is the similarity measurement function.

[0181] The teacher model class deviation correction loss or the student model class deviation correction loss is used to optimize the mapping effect of the local features of the source domain image samples from the source domain to the target domain. The calculation formula is as follows:

[0182]

[0183] in, is the teacher bias correction loss or the student bias correction loss, is the number of prompt templates in the object description set, x feat is the standard image feature or basic image feature, E attn is the teacher image feature encoding network, E attn (x feat ) is the standard image feature coding or basic image feature coding, E text is the teacher text encoding network, g is the similarity measurement function, r(c j ) is the jth element in the object description set, and its category is c, E text (r(c j )) is the text-like feature encoding.

[0184] In some optional implementations, the teacher model class bias correction loss or the student model class bias correction loss is:

[0185]

[0186] in, is the teacher bias correction loss or the student bias correction loss, is the number of prompt templates in the object description set, x feat is the standard image feature or basic image feature, E attnis the teacher image feature encoding network, E attn (x feat ) is the standard image feature encoding or basic image feature encoding, z is the category mean encoding, E text is the teacher text encoding network, g is the similarity measurement function, r(c j ) is the jth element in the object description set, and its category is c, E text (r(c j )) is the text-like feature encoding.

[0187] Based on the results of the loss functions of detection loss, knowledge distillation loss, and bias correction loss, the student model is trained, including: updating the parameters of the student image feature extraction network, feature remapping network, and student target detection head network to obtain the final domain-adapted target detection model.

[0188] The overall losses are as follows:

[0189]

[0190] in, For the overall loss, represents the detection loss, which is the sum of the classification loss and the regression loss, represents the sum of the teacher domain bias correction loss and the student domain bias correction loss, is the sum of the teacher bias correction loss and the student bias correction loss, is the knowledge distillation loss.

[0191] In the present invention, the teacher image feature extraction network, the teacher image feature encoding network and the teacher text encoding network adopt a pre-trained model, whose parameters remain fixed during the training process and do not participate in gradient updates, so as to ensure the stability and generalization ability of the final domain adaptation target detection model.

[0192] The student image feature extraction network, feature remapping network and student object detection head network are trainable models, and their parameters are optimized during the training process according to the loss functions of detection loss, knowledge distillation loss and bias correction loss to achieve the best domain adaptation performance and obtain the final domain adapted object detection model.

[0193] In some specific implementations of the present invention, a domain adaptability evaluation module and a model optimization module may also be provided. The domain adaptability evaluation module is used to evaluate the adaptability between the source domain and the target domain; the model optimization module is used to improve the detection performance of the model in the target domain by adjusting the network structure, model parameters, loss function weights or training strategies according to the feedback from the domain adaptability evaluation module, thereby improving the accuracy and robustness of cross-domain target detection.

[0194] After the final domain adaptation target detection model is obtained by the above training method, the target domain image to be detected is input into the final domain adaptation target detection model to obtain the detection result of the target domain image to be detected. Figure 8 FIG. 1 is a schematic diagram of a detection process of a target domain image to be detected provided by the present invention. The specific detection process includes:

[0195] The target domain image to be detected is input into the student image feature extraction network to obtain the target domain image basic features of the target domain image to be detected; the target domain image basic features are then input into the student target detection head network for target detection to obtain the detection result of the target domain image to be detected. The detection result includes at least one of the target frame pixel position, category, and confidence.

[0196] During the detection process, the parameters in the student image feature extraction network and the student target detection head network have been trained. Among them, the student image feature extraction network is used to extract the features of the target domain image to be detected and obtain the basic features of the target domain image to be detected; the student target detection head network is used to detect the target according to the basic features of the target domain image to be detected output by the student image feature extraction network.

[0197] The zero-sample cross-domain target detection method based on feature remapping provided by the present invention is verified below. Berkeley Deep Drive 100K (Berkeley Deep Driving Video 100,000, BBD-100K) and Cityscapes (city scenery) data sets are selected to verify the effectiveness of the method provided by the present invention. Among them, data under three weather conditions are sorted out from these two data sets, namely, sunny daytime (Daytime-Sunny), clear night (Night-Sunny) and rainy dusk (Dusk-Rainy). We trained the model based on the image and label data of sunny daytime (Daytime-Sunny), and the language description of clear night (Night-Sunny) and rainy dusk (Dusk-Rainy), and verified it in these two scenes respectively. The experimental results are shown in Table 1, and the values ​​shown are the average detection accuracy (Mean AveragePrecision, mAP) of all categories of targets. The results show that the zero-sample cross-domain target detection method based on feature remapping provided by the present invention plays an important role in improving the performance of the model in these two zero-sample domain adaptation target detection scenarios.

[0198]

[0199] Table 1

[0200] As can be seen from Table 1, the basic method achieves 34.4% mAP in the clear night scene and 26.0% mAP in the rainy dusk. Next, by gradually introducing different components, we observe their impact on the performance. Initially, we only set The performance in the clear night and rainy dusk scenes is improved, reaching 35% mAP and 26.5% mAP respectively. The subsequent addition of the feature remapping network module further improves the performance of the model, reaching 36.2% mAP and 28.5% mAP in these two scenes respectively. This shows that the feature remapping module retains the feature extraction capability of the student image feature extraction network while transferring the high-level semantic feature style, allowing the model to rely on the student image feature extraction network and the student object detection head network to achieve better performance. In addition, the introduction of When the feature remapping module is combined with the feature remapping module, 37.8% mAP and 31.9% mAP are achieved in clear night and rainy dusk scenes, respectively. Finally, when all four components are present, the performance of the zero-shot object detection model based on feature remapping reaches the best level, reaching 38.2% mAP and 32.5% mAP in clear night and rainy dusk scenes, respectively.

[0201] It can be seen that the feature remapping module, domain bias correction loss, class bias correction loss and knowledge distillation loss of the zero-shot target detection model based on feature remapping can effectively improve the domain adaptation performance of the model in different scenarios.

[0202] Embodiment 2:

[0203] Based on the same inventive concept, the present invention also provides a zero-sample cross-domain target detection system 900 based on feature remapping, the system structure is as follows: Fig. 9 As shown, the system 900 includes: a target domain image acquisition module 901, a detection result acquisition module 902;

[0204] The target domain image acquisition module 901 is used to acquire the target domain image to be detected;

[0205] A detection result acquisition module 902 is used to input the target domain image to be detected into the final domain adaptation target detection model to obtain a detection result of the target domain image to be detected;

[0206] Among them, the final domain adaptation target detection model is obtained by training the zero-shot cross-domain target detection model based on feature remapping using training data, and the training data includes: a source domain dataset containing image samples and labels and a target domain language description set.

[0207] Preferably, the system further comprises: a zero-sample cross-domain object detection model construction module based on feature remapping;

[0208] The zero-shot cross-domain object detection model based on feature remapping includes a student model and a teacher model;

[0209] The zero-shot cross-domain object detection model building modules based on feature remapping include:

[0210] A training data acquisition submodule is used to acquire training data;

[0211] A detection loss acquisition submodule is used to obtain basic image features of image samples through a student model, and calculate the detection loss of the student model based on the basic image features;

[0212] The knowledge distillation loss acquisition submodule is used to obtain the basic image feature encoding of the basic image features through the student model, obtain the standard image feature encoding of the image sample through the teacher model, compare the similarity between the standard image feature encoding and the basic image feature encoding, and obtain the knowledge distillation loss;

[0213] The deviation correction loss acquisition submodule is used to obtain the text feature encoding of the target domain language description set through the teacher model, and compare the similarity between the standard image feature encoding and the basic image feature encoding and the text feature encoding to obtain the deviation correction loss;

[0214] The final domain-adapted target detection model acquisition submodule is used to train the student model according to the detection loss, knowledge distillation loss, and bias correction loss to obtain the final domain-adapted target detection model.

[0215] Preferably, the student model includes: a student image feature extraction network, a feature remapping network, and a student object detection head network;

[0216] The detection loss acquisition submodule includes:

[0217] An initial basic image feature acquisition unit, used for inputting an image sample into a student image feature extraction network to obtain an initial basic image feature of the image sample;

[0218] A basic image feature acquisition unit, used for inputting the initial basic image features into a feature remapping network to obtain the basic image features after feature remapping;

[0219] The detection loss acquisition unit is used to input basic image features into the student target detection head network for target detection, compare the obtained detection results with the labels in the source domain dataset, and obtain the detection loss.

[0220] Preferably, the detection results obtained in the detection loss acquisition unit include classification detection results and regression detection results; the labels in the source domain data set include category labels and position labels;

[0221] The detection loss acquisition unit is specifically used for:

[0222] Compare the classification test results with the category labels to obtain the classification loss;

[0223] Compare the regression detection results with the position labels to obtain the regression loss;

[0224] The detection loss is determined as the sum of the classification loss and the regression loss.

[0225] Preferably, the teacher model includes a teacher image feature extraction network, a feature remapping network, and a teacher image feature encoding network;

[0226] The knowledge distillation loss acquisition submodule is used to:

[0227] Input the image sample into the teacher image feature extraction network to obtain the initial standard image features of the image sample;

[0228] Input the initial standard image features into the feature remapping network to obtain the standard image features after feature remapping;

[0229] The standard image features are input into the teacher image feature encoding network to obtain the standard image feature encoding of the image sample.

[0230] Preferably, the student model includes a scale alignment network and a teacher image feature encoding network;

[0231] The knowledge distillation loss acquisition submodule is also used for:

[0232] Inputting the basic image features into the scale alignment network to obtain the basic image features with the same size as the standard image features, and determining the basic image features with the same size as the standard image features as the scale basic image features;

[0233] The scaled basic image features are input into the teacher image feature encoding network to obtain the basic image feature encoding.

[0234] Preferably, the teacher model further comprises: a teacher text encoding network; the target domain language description set comprises a target domain description set and an object description set;

[0235] The bias correction loss acquisition submodule includes:

[0236] The quasi-text feature coding acquisition unit is used to input the target domain description set into the teacher text coding network to obtain the domain text feature coding; input the object description set into the teacher text coding network to obtain the quasi-text feature coding;

[0237] The teacher model bias correction loss acquisition unit is used to compare the similarity between the standard image feature encoding and the domain text feature encoding to obtain the teacher model domain bias correction loss; compare the similarity between the standard image feature encoding and the class text feature encoding to obtain the teacher model class bias correction loss;

[0238] The student model bias correction loss acquisition unit is used to compare the similarity between the basic image feature encoding and the domain text feature encoding to obtain the student model domain bias correction loss; compare the similarity between the basic image feature encoding and the class text feature encoding to obtain the student model class bias correction loss;

[0239] The bias correction loss acquisition unit is used to determine the sum of the domain bias correction loss of the teacher model and the domain bias correction loss of the student model as the domain bias correction loss; determine the sum of the class bias correction loss of the teacher model and the class bias correction loss of the student model as the class bias correction loss; and determine the sum of the domain bias correction loss and the class bias correction loss as the bias correction loss;

[0240] The formula for the teacher model domain bias correction loss or the student model domain bias correction loss is as follows:

[0241]

[0242] in, represents the teacher domain bias correction loss or the student domain bias correction loss, represents the number of prompt templates in the target domain description set, x feat is the standard image feature or basic image feature, E attn is the teacher image feature encoding network, E attn (x feat ) is the standard image feature coding or basic image feature coding, E text For the teacher text encoding network, r i is the value of the i-th element in the target domain description set, E text (r i ) is the domain text feature encoding, g is the similarity measurement function;

[0243] The formula for the teacher bias correction loss or student bias correction loss is as follows:

[0244]

[0245] in, is the teacher bias correction loss or the student bias correction loss, is the number of prompt templates in the object description set, x feat is the standard image feature or basic image feature, E attn is the teacher image feature encoding network, E attn(x feat ) is the standard image feature coding or basic image feature coding, E text is the teacher text encoding network, g is the similarity measurement function, r(c j ) is the jth element in the object description set, and its category is c, E text (r(c j )) is the text-like feature encoding.

[0246] Preferably, the system further comprises: a training data acquisition module;

[0247] The training data acquisition module includes a target domain description set acquisition submodule, which is specifically used for:

[0248] Establish a common prompt template that describes the source and target domains;

[0249] The specific description of the target domain relative to the source domain is combined with the common prompt template to obtain a target domain description set.

[0250] Preferably, the system further comprises a feature remapping network acquisition module, which is used to:

[0251] Construct the first branch of extracting and transforming local features by connecting multiple small-scale convolution kernels in series;

[0252] A second branch is constructed that connects pooling layers and multi-layer perceptrons in series to adjust the global mapping across channel dimensions;

[0253] The first branch is fused with the second branch to obtain a feature remapping network.

[0254] Preferably, the detection result acquisition unit is specifically used for:

[0255] Input the target domain image to be detected into the student image feature extraction network to obtain the basic features of the target domain image of the target domain image to be detected;

[0256] Input the basic features of the target domain image into the student target detection head network to perform target detection, and obtain the detection result of the target domain image to be detected;

[0257] The detection result includes at least one of the target frame pixel position, category, and confidence level.

[0258] Embodiment 3:

[0259] Based on the same inventive concept, Fig.10As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip device, an intelligent mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, the processor, and the transceiver component are connected via a bus; the memory may be used to store an execution program, and an exemplary execution program may include instructions; the processor is used to execute the instructions stored in the memory. The memory may also be used to store data, which may be called and / or modified when the instructions are executed.

[0260] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a readable storage medium to implement the corresponding method flow or corresponding functions, so as to implement the steps of a zero-sample cross-domain target detection method based on feature remapping in the above embodiment.

[0261] Embodiment 4:

[0262] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It can be understood that the readable storage medium here can include both built-in storage media in electronic devices and, of course, extended storage media supported by electronic devices. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The processor can load and execute one or more instructions stored in the storage medium to implement the steps of a zero-sample cross-domain target detection method based on feature remapping in the above embodiment.

[0263] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0264] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0265] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0266] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0267] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims to be approved.

Claims

1. A zero-sample cross-domain target detection method based on feature remapping, characterized in that: include: Obtain the target domain image to be detected; Inputting the target domain image to be detected into the final domain adaptation target detection model to obtain a detection result of the target domain image to be detected; The final domain adaptation target detection model is obtained by training a zero-shot cross-domain target detection model based on feature remapping using training data, wherein the training data includes: a source domain data set including image samples and labels and a target domain language description set.

2. The method according to claim 1, characterized in that The zero-shot cross-domain target detection model based on feature remapping includes a student model and a teacher model; The training process of the final domain adaptation target detection model includes: Get training data; Obtaining basic image features of the image sample through the student model, and calculating the detection loss of the student model according to the basic image features; Obtaining a basic image feature encoding of the basic image feature through the student model, obtaining a standard image feature encoding of the image sample through the teacher model, and comparing the standard image feature encoding with the basic image feature encoding for similarity to obtain a knowledge distillation loss; The text feature encoding of the target domain language description set is obtained by the teacher model, and the standard image feature encoding and the basic image feature encoding are respectively compared with the text feature encoding for similarity to obtain a deviation correction loss; The student model is trained according to the detection loss, the knowledge distillation loss, and the bias correction loss to obtain a final domain-adapted object detection model.

3. The method according to claim 2, characterized in that The student model includes: a student image feature extraction network, a feature remapping network, and a student target detection head network; The step of obtaining the basic image features of the image sample by using the student model and calculating the detection loss of the student model according to the basic image features includes: Inputting the image sample into the student image feature extraction network to obtain the initial basic image features of the image sample; Inputting the initial basic image features into the feature remapping network to obtain basic image features after feature remapping; The basic image features are input into the student object detection head network for object detection, and the obtained detection results are compared with the labels in the source domain dataset to obtain the detection loss.

4. The method according to claim 3, characterized in that The obtained detection results include classification detection results and regression detection results; the labels in the source domain data set include category labels and position labels; The step of comparing the obtained detection result with the label in the source domain dataset to obtain the detection loss includes: Comparing the classification detection result with the category label to obtain the classification loss; Comparing the regression detection result with the position label to obtain a regression loss; The sum of the classification loss and the regression loss is determined as the detection loss.

5. The method according to claim 2, characterized in that: The teacher model includes a teacher image feature extraction network, a feature remapping network and a teacher image feature encoding network; Obtaining a standard image feature encoding of the image sample through the teacher model includes: Inputting the image sample into the teacher image feature extraction network to obtain the initial standard image features of the image sample; Inputting the initial standard image features into the feature remapping network to obtain the standard image features after feature remapping; The standard image features are input into the teacher image feature encoding network to obtain the standard image feature encoding of the image sample.

6. The method according to claim 5, characterized in that The student model includes a scale alignment network and a teacher image feature encoding network; Obtaining the basic image feature encoding of the basic image feature through the student model includes: Inputting the basic image features into the scale alignment network to obtain basic image features with the same size as the standard image features, and determining the basic image features with the same size as the standard image features as scale basic image features; The scaled basic image features are input into the teacher image feature encoding network to obtain basic image feature encoding.

7. The method according to claim 5, characterized in that The teacher model also includes: a teacher text encoding network; the target domain language description set includes a target domain description set and an object description set; The text feature encoding of the target domain language description set is obtained through the teacher model, and the standard image feature encoding and the basic image feature encoding are respectively compared with the text feature encoding for similarity to obtain the deviation correction loss, including: Input the target domain description set into the teacher text encoding network to obtain domain text feature encoding; input the object description set into the teacher text encoding network to obtain class text feature encoding; Compare the similarity between the standard image feature encoding and the domain text feature encoding to obtain the domain deviation correction loss of the teacher model; compare the similarity between the standard image feature encoding and the class text feature encoding to obtain the class deviation correction loss of the teacher model; Compare the similarity of the basic image feature coding with the domain text feature coding to obtain the domain deviation correction loss of the student model; compare the similarity of the basic image feature coding with the class text feature coding to obtain the class deviation correction loss of the student model; The sum of the domain bias correction loss of the teacher model and the domain bias correction loss of the student model is determined as the domain bias correction loss; the sum of the class bias correction loss of the teacher model and the class bias correction loss of the student model is determined as the class bias correction loss; the sum of the domain bias correction loss and the class bias correction loss is determined as the bias correction loss; The formula for the teacher model domain deviation correction loss or the student model domain deviation correction loss is as follows: in, represents the teacher domain bias correction loss or the student domain bias correction loss, represents the number of prompt templates in the target domain description set, x feat is the standard image feature or basic image feature, E attn is the teacher image feature encoding network, E attn (x feat ) is the standard image feature coding or basic image feature coding, E text For the teacher text encoding network, r i is the value of the i-th element in the target domain description set, E text (r i ) is the domain text feature encoding, g is the similarity measurement function; The formula for the teacher class bias correction loss or the student class bias correction loss is as follows: in, is the teacher bias correction loss or the student bias correction loss, is the number of prompt templates in the object description set, x feat is the standard image feature or basic image feature, E attn is the teacher image feature encoding network, E attn (x feat ) is the standard image feature coding or basic image feature coding, E text is the teacher text encoding network, g is the similarity measurement function, r(c j ) is the jth element in the object description set, and its category is c, E text (r(c j )) is the text-like feature encoding.

8. The method according to claim 3 or 5, characterized in that: The feature remapping network comprises: Construct the first branch of extracting and transforming local features by connecting multiple small-scale convolution kernels in series; A second branch is constructed that connects pooling layers and multi-layer perceptrons in series to adjust the global mapping across channel dimensions; The first branch is fused with the second branch to obtain a feature remapping network.

9. The method according to claim 3, characterized in that: The step of inputting the target domain image to be detected into the final domain adaptation target detection model to obtain a detection result of the target domain image to be detected includes: Inputting the target domain image to be detected into a student image feature extraction network to obtain the target domain image basic features of the target domain image to be detected; Inputting the basic features of the target domain image into the student target detection head network to perform target detection, and obtaining a detection result of the target domain image to be detected; The detection result includes at least one of the target frame pixel position, category, and confidence level.

10. A zero-sample cross-domain target detection system based on feature remapping, characterized in that: include: A target domain image acquisition module is used to acquire a target domain image to be detected; A detection result acquisition module, used for inputting the target domain image to be detected into the final domain adaptation target detection model to obtain a detection result of the target domain image to be detected; The final domain-adaptive target detection model is obtained by training a zero-sample cross-domain target detection model based on feature remapping using training data, and the training data includes: a source domain data set including image samples and labels and a target domain language description set.

Images