A cross-scene target detection method based on structural consistency learning

Abstract

The application discloses a cross-scene target detection method based on structural consistency learning. The method comprises the following steps: obtaining an initial source domain image and an initial target domain image, wherein the initial source domain image is a real label image, and the initial target domain image is a label-free image; determining a first attention map and a first domain discrimination loss value of the initial source domain image and a second attention map and a second domain discrimination loss value of the initial target domain image based on a first image-level feature of the initial source domain image and a second image-level feature of the initial target domain image; determining a third attention map and a third domain discrimination loss value of the initial source domain image and a fourth attention map and a fourth domain discrimination loss value of the initial target domain image based on a first instance-level feature of the initial source domain image and a second instance-level feature of the initial target domain image. The application solves the technical problem of the limitation of the cross-scene target detection method based on unsupervised domain self-adaptation in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a cross-scene object detection method based on structure consistency learning. BACKGROUND

[0002] Object detection is a core task in the field of image analysis. In recent years, object detection technology based on deep learning has made great progress. However, deep learning methods rely heavily on a large amount of high-quality labeled data, which is a challenge for new object detection scenarios. To address this issue, cross-scene object detection technology has attracted widespread attention, aiming to assist in completing the object detection task on the target domain through learning on the source domain. Specifically, researchers attempt to use unsupervised domain adaptation methods to reduce the data labeling cost in new scenarios and improve the generalization performance of the detection system.

[0003] The first unsupervised domain adaptation applied to the object detection task is DAFaster proposed by Y. Chen et al. in "Y. Chen, W. Li, C. Sakaridis, D. Dai, and L. Van Gool, "Domain Adaptive Faster R-Cnn for Object Detection in the Wild," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3339-3348." This method measures the distribution difference between domains based on H-divergence and aligns the image-level and instance-level feature distributions between domains, while also designing consistency regularization to learn the region proposal network. Since then, many related methods have emerged, such as the multi-granularity alignment network proposed by W. Zhou et al. in "W. Zhou, D. Du, L. Zhang, T. Luo, and Y. Wu, "Multi-Granularity Alignment Domain Adaptation for Object Detection," in Proc. IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9581-9590." This method considers that the transferability of each class of features is different, so it changes the output dimension of the domain discriminator, making it not only able to identify the domain to which the feature belongs, but also to distinguish the class of the feature, so as to gradually align the feature distribution of each class in the process of adversarial training. The dual instance-consistent network proposed by Jiao et al. in "Y. Jiao, H. Yao, and C. Xu, "Dual Instance-Consistent Network for Cross-Domain Object Detection," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 6, pp. 7338-7352, 2023." introduces a main network and an auxiliary network to generate feature spaces specific to the source domain and the target domain. In the two feature spaces, not only is the feature consistency considered, but also the consistency of classification and regression.

[0004] The above methods do not consider the class imbalance problem existing in the data in the process of aligning the feature distribution. This problem can cause the samples of a certain class with less data to be incorrectly aligned to the center of samples with more data, i.e., the model has poor discrimination ability for samples of a certain class with less data. SUMMARY

[0005] The embodiment of the present application provides a cross-scene target detection method based on structural consistency learning, so as to at least solve the technical problem of the limitation of the cross-scene target detection method based on unsupervised domain adaptation in the prior art.

[0006] According to an aspect of the embodiment of the present application, a cross-scene target detection method based on structural consistency learning is provided. The method can include: obtaining an initial source domain image and an initial target domain image, wherein the initial source domain image is a real-labeled image, and the initial target domain image is an unlabeled image; determining a first attention map and a first domain discrimination loss value of the initial source domain image and a second attention map and a second domain discrimination loss value of the initial target domain image based on a first image-level feature of the initial source domain image and a second image-level feature of the initial target domain image; determining a third attention map and a third domain discrimination loss value of the initial source domain image and a fourth attention map and a fourth domain discrimination loss value of the initial target domain image based on a first instance-level feature of the initial source domain image and a second instance-level feature of the initial target domain image; determining a first global prototype of each class sample in the initial source domain image in a current iteration and a second global prototype of each class sample in the initial target domain image in the current iteration based on the instance-level feature of the different class samples in the initial source domain image, the sample number of the different class samples, and the initial global prototype of the initial source domain image in the last iteration, and the instance-level feature of the different class samples in the initial target domain image, the sample number of the different class samples, and the initial global prototype of the initial target domain image in the last iteration; determining a first structure matrix of the initial source domain image and a second structure matrix of the initial target domain image based on the first global prototype and the second global prototype; inputting the image domain consistency features of the initial source domain image and the initial target domain image into a classification and regression detection head to obtain a first detection result of the initial source domain image and a second detection result of the initial target domain image; obtaining a target domain image to be detected to obtain a target detection result of the target domain image to be detected, wherein the target domain image to be detected is an unlabeled image.

[0007] Optionally, before obtaining the initial source domain image and the initial target domain image, the method further includes: obtaining an original source domain image and an original target domain image, wherein the original source domain image is a real-labeled image, and the original target domain image is an unlabeled image; performing scaling and normalization operations on the original source domain image and the original target domain image respectively to obtain the initial source domain image and the initial target domain image.

[0008] Optionally, based on the first image-level feature of the initial source domain image and the second image-level feature of the initial target domain image, determining the first attention map and the first domain discriminative loss value of the initial source domain image and the second attention map and the second domain discriminative loss value of the initial target domain image comprises: inputting the first image-level feature and the second image-level feature into a region proposal network to obtain a first prediction map result of the initial source domain image and a second prediction map result of the initial target domain image; processing the first prediction map result and the second prediction map result through a channel to obtain a first feature of the first prediction map result and a second feature of the second prediction map result, and filtering the first feature and the second feature using a preset threshold to generate the first attention map of the initial source domain image and the second attention map of the initial target domain image; inputting the first image-level feature and the second image-level feature into a domain discriminator for adversarial learning to obtain a fifth domain discriminative loss value of the initial source domain image and a sixth domain discriminative loss value of the initial target domain image; based on the first attention map and the fifth domain discriminative loss value, obtaining the first domain discriminative loss value of the initial source domain image; based on the second attention map and the sixth domain discriminative loss value, obtaining the second domain discriminative loss value of the initial target domain image.

[0009] Optionally, based on the first instance-level feature of the initial source domain image and the second instance-level feature of the initial target domain image, determining the third attention map and the third domain discriminative loss value of the initial source domain image and the fourth attention map and the fourth domain discriminative loss value of the initial target domain image comprises: inputting the first instance-level feature and the second instance-level feature into a classification head to obtain a first detection result of a first object in the initial source domain image and a second detection result of a second object in the initial target domain image, wherein the first detection result comprises a category of each first object and a first probability score corresponding to the category of each first object, and the second detection result comprises a category of each second object and a second probability score corresponding to the category of each second object; based on the first probability score, obtaining the third attention map of the initial source domain image; based on the second probability score, obtaining the fourth attention map of the initial target domain image; inputting the first instance-level feature and the second instance-level feature into a domain discriminator for adversarial learning to obtain a seventh domain discriminative loss value of the initial source domain image and an eighth domain discriminative loss value of the initial target domain image; based on the third attention map and the seventh domain discriminative loss value, obtaining the third domain discriminative loss value of the initial source domain image; based on the fourth attention map and the eighth domain discriminative loss value, obtaining the fourth domain discriminative loss value of the initial target domain image.

[0010] Optionally, based on the instance-level features of different category samples in the initial source domain image, the sample quantity of different category samples, and the initial global prototype of the initial source domain image in the last iteration, based on the instance-level features of different category samples in the initial target domain image, the sample quantity of different category samples, and the initial global prototype of the initial target domain image in the last iteration, the first global prototype of each category sample in the initial source domain image in the current iteration and the second global prototype of each category sample in the initial target domain image in the current iteration are determined, comprising: based on the instance-level features of different category samples in the initial source domain image and the sample quantity of different category samples, the first local prototype of each category sample in the initial source domain image in the current iteration is determined; based on the first local prototype and the initial global prototype of the initial source domain image in the last iteration, the first global prototype of each category sample in the initial source domain image is determined; based on the instance-level features of different category samples in the initial target domain image and the sample quantity of different category samples, the second local prototype of each category sample in the initial target domain image in the current iteration is determined; based on the second local prototype and the initial global prototype of the initial target domain image in the last iteration, the second global prototype of each category sample in the initial target domain image is determined.

[0011] Optionally, based on the instance-level features of different category samples in the initial source domain image and the sample quantity of different category samples, the expression of the first local prototype of each category sample in the initial source domain image is: wherein, represents the first local prototype of the c category sample in the initial source domain image, N c represents the sample quantity of the c category sample in the initial source domain image, represents the instance-level features of the c category sample in the initial source domain image. The expression of the second local prototype of each category sample in the image is: wherein, represents the second local prototype of the c category sample in the initial target domain image, N c represents the sample quantity of the c category sample in the initial target domain image, represents the instance-level features of the c category sample in the initial target domain image.

[0012] Optionally, based on the first global prototype and the second global prototype, the first structure matrix of the initial source domain image and the second structure matrix of the initial target domain image are determined, comprising: based on the first global prototype of each category sample in the initial source domain image, the first structure matrix of the initial source domain image is constructed; based on the second global prototype of each category sample in the initial target domain image, the second structure matrix of the initial target domain image is constructed.

[0013] The beneficial effects of the present application are:

[0014] The application provides a cross-scene target detection method based on structural consistency learning, which can learn more effective domain consistency features by aligning image-level features, instance-level features and structural features of a source domain and a target domain, and simultaneously introduces an attention mechanism in the image-level and instance-level feature alignment stage to minimize the interference of background information, realizes cross-domain target detection, solves the limitation of the prior art cross-scene target detection method based on unsupervised domain self-adaptation, and achieves the technical effect that in the unsupervised cross-scene target detection, the method combining the double-attention mechanism and the structural consistency learning between the source domain and the target domain can learn more effective source domain and target domain domain consistency features, and high-efficiency cross-scene target detection is realized. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application, and do not limit the application. In the drawings:

[0016] Figure 1 FIG. 1 is a flowchart of a cross-scene target detection method based on structural consistency learning according to an embodiment of the application;

[0017] Figure 2 FIG. 2 is a schematic diagram of data quantity distribution of target domain test data according to an embodiment of the application. DETAILED DESCRIPTION

[0018] In order to enable persons skilled in the art to better understand the application scheme, the technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings of the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by persons skilled in the art without creative labor should be within the scope of protection of the application.

[0019] It should be noted that the terms "first", "second", and the like in the specification and claims of the application and the above-described drawings are used to distinguish similar objects, and are used to describe a particular order or sequence. It should be understood that the terms thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0020] Embodiment 1

[0021] According to an embodiment of the present application, a cross-scene object detection method based on structural consistency learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0022] Figure 1 is a flowchart of a cross-scene object detection method based on structural consistency learning according to an embodiment of the present application, as shown in Figure 1 The method can comprise the following steps:

[0023] Step S101, obtaining an initial source domain image and an initial target domain image, wherein the initial source domain image is a real label image, and the initial target domain image is a label-free image.

[0024] In the technical solution provided in the above step S101 of the present application, the initial source domain image and the initial target domain image are obtained, wherein the initial source domain image is a real label image, and the initial target domain image is a label-free image.

[0025] Step S102, based on the first image level feature of the initial source domain image and the second image level feature of the initial target domain image, determining the first attention map and the first domain discrimination loss value of the initial source domain image and the second attention map and the second domain discrimination loss value of the initial target domain image.

[0026] In the technical solution provided in the above step S102 of the present application, according to the first image level feature of the initial source domain image, the first attention map and the first domain discrimination loss value of the initial source domain image are obtained; according to the second image level feature of the initial target domain image, the second attention map and the second domain discrimination loss value of the initial target domain image are obtained.

[0027] Step S103, based on the first instance level feature of the initial source domain image and the second instance level feature of the initial target domain image, determining the third attention map and the third domain discrimination loss value of the initial source domain image, and the fourth attention map and the fourth domain discrimination loss value of the initial target domain image.

[0028] In the technical solution provided in the above step S103 of the present application, according to the first instance level feature of the initial source domain image, the third attention map and the third domain discrimination loss value of the initial source domain image are obtained; according to the second instance level feature of the initial target domain image, the fourth attention map and the fourth domain discrimination loss value of the initial target domain image are obtained.

[0029] Step S104, based on the instance-level features of different class samples in the initial source domain image, the sample number of different class samples, and the initial global prototype of the initial source domain image in the last iteration, the instance-level features of different class samples in the initial target domain image, the sample number of different class samples, and the initial global prototype of the initial target domain image in the last iteration, determine the first global prototype of each class sample in the initial source domain image in the current iteration and the second global prototype of each class sample in the initial target domain image in the current iteration.

[0030] In the technical solution provided by the above step S104 of the application, the first global prototype of each class sample in the initial source domain image in the current iteration is obtained by calculating according to the instance-level features of different class samples in the initial source domain image, the sample number of different class samples, and the initial global prototype of the initial source domain image in the last iteration; and the second global prototype of each class sample in the initial target domain image in the current iteration is obtained by calculating according to the instance-level features of different class samples in the initial target domain image, the sample number of different class samples, and the initial global prototype of the initial target domain image in the last iteration.

[0031] Step S105, based on the first global prototype and the second global prototype, determine the first structure matrix of the initial source domain image and the second structure matrix of the initial target domain image.

[0032] In the technical solution provided by the above step S105 of the application, the first structure matrix of the initial source domain image is constructed according to the first global prototype, and the second structure matrix of the initial target domain image is constructed according to the second global prototype.

[0033] Step S106, input the image domain consistency features of the initial source domain image and the initial target domain into the classification and regression detection head to obtain the first detection result of the initial source domain image and the second detection result of the initial target domain image.

[0034] In the technical solution provided by the above step S106 of the application, the image domain consistency features of the initial source domain image and the initial target domain are input into the classification and regression detection head to obtain the first detection result of the initial source domain image and the second detection result of the initial target domain image.

[0035] Step S107, obtain the target domain image to be detected to obtain the target detection result of the target domain image to be detected, wherein the target domain image to be detected is a unlabeled image.

[0036] In the technical solution provided by the above step S107 of the application, the target domain image to be detected is obtained, and the target domain image to be detected is input into the model trained in the above steps S101-S106 to obtain the target detection result of the target domain image to be detected.

[0037] The above method of this embodiment is further described below.

[0038] As an optional embodiment, before the step S101, the method further includes: obtaining an original source domain image and an original target domain image, wherein the original source domain image is a real labeled image, and the original target domain image is an unlabeled image; and performing scaling and normalization operations on the original source domain image and the original target domain image respectively to obtain the initial source domain image and the initial target domain image.

[0039] In this embodiment, the original source domain image and the original target domain image are obtained, and the original source domain image and the original target domain image are subjected to uniform scaling and normalization operations to obtain the initial source domain image and the initial target domain image.

[0040] As an optional embodiment, in the step S102, the first attention map and the first domain discriminant loss value of the initial source domain image and the second attention map and the second domain discriminant loss value of the initial target domain image are determined based on the first image-level feature of the initial source domain image and the second image-level feature of the initial target domain image, including: inputting the first image-level feature and the second image-level feature into a region proposal network to obtain a first prediction map result of the initial source domain image and a second prediction map result of the initial target domain image; processing the first prediction map result and the second prediction map result through a channel to obtain a first feature of the first prediction map result and a second feature of the second prediction map result, and filtering the first feature and the second feature using a preset threshold to generate the first attention map of the initial source domain image and the second attention map of the initial target domain image; inputting the first image-level feature and the second image-level feature into a domain discriminator for adversarial learning to obtain a fifth domain discriminant loss value of the initial source domain image and a sixth domain discriminant loss value of the initial target domain image; obtaining the first domain discriminant loss value of the initial source domain image based on the first attention map and the fifth domain discriminant loss value; and obtaining the second domain discriminant loss value of the initial target domain image based on the second attention map and the sixth domain discriminant loss value.

[0041] In this embodiment, the first image-level feature is input into the region proposal network to obtain the first prediction map result of the initial source domain image, and the second image-level feature is input into the region proposal network to obtain the second prediction map result of the initial target domain image, wherein the first prediction result map and the second prediction result map can be represented by F RPN The first prediction map result and the second prediction map result are processed through a channel to obtain a first feature of the first prediction map result and a second feature of the second prediction map result, wherein the first feature and the second feature can be represented by M RPNThe first feature and the second feature are filtered using a preset threshold to generate a first attention map of the initial source domain image and a second attention map of the initial target domain image, where the first attention map and the second attention map can be obtained through A img The following expression is used: img The following expression can be used for calculation:

[0042]

[0043] Where, the indicator function I(·) is used to distinguish the foreground and background regions in M RPN , and the threshold t is specified as the average value of M RPN .

[0044] The first image-level feature is input into the domain discriminator for adversarial learning to obtain a fifth domain discrimination loss value of the initial source domain image; based on the first attention map and the fifth domain discrimination loss value, a first domain discrimination loss value of the initial source domain image is obtained; the second image-level feature is input into the domain discriminator for adversarial learning to obtain a sixth domain discrimination loss value of the initial target domain image; based on the second attention map and the sixth domain discrimination loss value, a second domain discrimination loss value of the initial target domain image is obtained, where the first domain discrimination loss value and the second domain discrimination loss value can be calculated through the following formula:

[0045]

[0046] Where, is the first attention map or the second attention map, is the fifth domain discrimination loss value or the sixth domain discrimination loss value, L Aimg is the first domain discrimination loss value or the second domain discrimination loss value, ( u,v) is the row and column of A img or the row and column of L img .

[0047] As an optional embodiment, in step S103, the third attention map and the third domain discriminant loss value of the initial source domain image and the fourth attention map and the fourth domain discriminant loss value of the initial target domain image are determined based on the first instance-level feature of the initial source domain image and the second instance-level feature of the initial target domain image, including: inputting the first instance-level feature and the second instance-level feature into a classification head to obtain a first detection result of each first object in the initial source domain image and a second detection result of each second object in the initial target domain image, wherein the first detection result includes the category of each first object and the first probability score corresponding to the category of each first object, and the second detection result includes the category of each second object and the second probability score corresponding to the category of each second object; based on the first probability score, the third attention map of the initial source domain image is obtained; based on the second probability score, the fourth attention map of the initial target domain image is obtained; the first instance-level feature and the second instance-level feature are input into a domain discriminator for adversarial learning to obtain a seventh domain discriminant loss value of the initial source domain image and an eighth domain discriminant loss value of the initial target domain image; based on the third attention map and the seventh domain discriminant loss value, the third domain discriminant loss value of the initial source domain image is obtained; based on the fourth attention map and the eighth domain discriminant loss value, the fourth domain discriminant loss value of the initial target domain image is obtained.

[0048] In this embodiment, the first instance-level feature is input into the classification head to obtain the first detection result of each first object in the initial source domain image; the second instance-level feature is input into the classification head to obtain the second detection result of each second object in the initial target domain image; based on the first probability score, the third attention map of the initial source domain image is obtained; based on the second probability score, the fourth attention map of the initial target domain image is obtained, wherein the calculation process of the third attention map or the fourth attention map is as follows:

[0049]

[0050] wherein A ins is the third attention map or the fourth attention map, p is the first probability score or the second probability score, k is a hyperparameter, and c is the category of each first object in the first detection result or the category of each second object in the second detection result.

[0051] The first instance-level feature and the second instance-level feature are input into the domain discriminator for adversarial learning to obtain the seventh domain discriminant loss value of the initial source domain image and the eighth domain discriminant loss value of the initial target domain image, wherein the seventh domain discriminant loss value or the eighth domain discriminant loss value is represented by , wherein i is the number of image frames in a batch.

[0052] According to the third attention map and the seventh domain discriminator loss value, a third domain discriminator loss value of the initial source domain image is obtained; according to the fourth attention map and the eighth domain discriminator loss value, a fourth domain discriminator loss value of the initial target domain image is obtained, and the calculation process of the third domain discriminator loss value or the fourth domain discriminator loss value is as follows:

[0053]

[0054] wherein, L Ains is the third domain discriminator loss value or the fourth domain discriminator loss value, is the fourth attention map or the fifth attention map.

[0055] As an optional embodiment, in step S104, the first global prototype of each class sample in the initial source domain image in the current iteration and the second global prototype of each class sample in the initial target domain image in the current iteration are determined based on the instance-level features of different class samples in the initial source domain image, the sample number of different class samples, the initial global prototype of the initial source domain image in the last iteration, the instance-level features of different class samples in the initial target domain image, the sample number of different class samples, and the initial global prototype of the initial target domain image in the last iteration, and the step S104 includes: determining the first local prototype of each class sample in the initial source domain image in the current iteration based on the instance-level features of different class samples in the initial source domain image and the sample number of different class samples; determining the first global prototype of each class sample in the initial source domain image based on the first local prototype and the initial global prototype of the initial source domain image in the last iteration; determining the second local prototype of each class sample in the initial target domain image in the current iteration based on the instance-level features of different class samples in the initial target domain image and the sample number of different class samples; and determining the second global prototype of each class sample in the initial target domain image based on the second local prototype and the initial global prototype of the initial target domain image in the last iteration.

[0056] In this embodiment, the first local prototype of each class sample in the initial source domain image in the current iteration is obtained based on the instance-level features of different class samples in the initial source domain image and the sample number of different class samples; the second local prototype of each class sample in the initial target domain image in the current iteration is obtained based on the instance-level features of different class samples in the initial target domain image and the sample number of different class samples; the first global prototype of each class sample in the initial source domain image is obtained based on the first local prototype and the initial global prototype of the initial source domain image in the last iteration, and the second global prototype of each class sample in the initial target domain image is obtained based on the second local prototype and the initial global prototype of the initial target domain image in the last iteration, and the calculation process of the first global prototype or the second global prototype is as follows:

[0057] P global (m)=αplocal (m) + (1 - a)P global (m - 1)

[0058] wherein, P global (m) is the first global prototype or the second global prototype of the mth iteration, P local (m) is the first local prototype or the second local prototype, a represents the cosine similarity of the first local prototype and the initial global prototype of the initial source domain image in the last iteration or the cosine similarity of the second local prototype and the initial global prototype of the initial target domain image in the last iteration, P global (m - 1) is the initial global prototype of the last iteration of the initial source domain image or the initial global prototype of the last iteration of the initial target domain image.

[0059] As an optional embodiment, based on the instance-level features of different category samples in the initial source domain image and the sample number of different category samples, the expression of the first local prototype of each category sample in the initial source domain image is determined as:

[0060]

[0061] wherein, represents the first local prototype of the c-th category sample in the initial source domain image, N c represents the sample number of the c-th category sample in the initial source domain image, represents the instance-level features of the c-th category sample in the initial source domain image.

[0062] As an optional embodiment, based on the instance-level features of different category samples in the initial target domain image and the sample number of different category samples, the expression of the second local prototype of each category sample in the initial target domain image is determined as:

[0063]

[0064] wherein, represents the second local prototype of the c-th category sample in the initial target domain image, N c represents the sample number of the c-th category sample in the initial target domain image, represents the instance-level features of the c-th category sample in the initial target domain image.

[0065] As an optional embodiment, in step S105, based on the first global prototype and the second global prototype, the first structure matrix of the initial source domain image and the second structure matrix of the initial target domain image are determined, including: based on the first global prototype of each category sample in the initial source domain image, constructing the first structure matrix of the initial source domain image; based on the second global prototype of each category sample in the initial target domain image, constructing the second structure matrix of the initial target domain image.

[0066] In this embodiment, a first structure matrix of the initial source domain image is constructed according to the first global prototype of each class sample in the initial source domain image, a second structure matrix of the initial target domain image is constructed according to the second global prototype of each class sample in the initial target domain image, and the design of the first structure matrix or the second structure matrix considers the distance between different class global prototypes and reflects the difference between classes. In order to effectively reduce the structural distribution difference between the source domain and the target domain, the embodiment of the application introduces the following loss function.

[0067]

[0068] Wherein, L SFC is the loss function between the first structure matrix and the second structure matrix, is the first structure matrix, re is the rth row, e element of re is the rth row, e element of

[0069] The effect of the embodiment of the application can be further illustrated by the following simulation experiment.

[0070] 1. Simulation conditions

[0071] The simulation is carried out on an NVIDIA GeFore RTX 4090 GPU using Visual Stdio Code software. The data used in the simulation is a public cross-weather dataset.

[0072] 2. Simulation content

[0073] First, the source domain training data and its real label and the target domain training data are used to train the model according to the steps in the specific embodiment; then the trained model is applied to the target detection task of the target domain test data, and the AP and MAP are calculated to evaluate the model. According to the embodiment of the application, the data amount distribution of the target domain test data is provided, Figure 2 is a schematic diagram of the data amount distribution of the target domain test data according to the embodiment of the application.

[0074] ​​To prove the effectiveness of the present application, MGADA, DICN and CRADA are selected as comparative algorithms, the MGADA algorithm is proposed in the literature "W. Zhou, D. Du, L. Zhang, T. Luo and Y. Wu, "Multi-Granularity Alignment Domain Adaptation for Object Detection," in Proc. 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9571-9580." The DICN algorithm is proposed in the literature "Y. Jiao, H. Yao and C. Xu, "Dual Instance-Consistent Network for Cross-Domain Object Detection," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 6, pp. 7338-7352, 2023." The CRADA algorithm is proposed in the literature "Y. Liu, J. Wang, W. Wang, Y. Hu, Y. Wang and Y. Xu, "CRADA: Cross Domain Object Detection With Cyclic Reconstruction and Decoupling Adaptation," IEEE Transactions on Multimedia, vol. 26, pp. 6250-6261, 2024." The comparison results are shown in Table 1.

[0075] Table 1 Comparison results of different algorithms

[0076] Method Person Rider Car Truck Bus Train Mbike Bicycle MAP DICN 47.3 57.4 64.0 22.7 45.6 29.6 38.6 47.4 44.1 MGADA 43.9 49.6 60.6 29.6 50.7 39.0 38.3 42.8 44.3 CRADA 47.0 48.3 64.1 28.7 52.3 38.6 35.1 41.5 44.5 The invention 44.9 51.8 61.9 32.4 48.8 41.2 36.8 46.7 45.6

[0077] From Table 1, it can be seen that the present application achieves the highest MAP compared to the previous cross-scene target detection method. More notably, although the detection effect of the present application on the Bus category is general, it significantly improves the detection accuracy on the Truck and Train categories. The present application introduces an attention mechanism at the image level and the instance level stage, so that the network pays more attention to the foreground features, thereby improving the overall detection performance. In addition, the present application can effectively balance the distribution distance between different categories through structure consistency learning, thereby enhancing the detection accuracy of the categories with less sample quantity.

[0078] In the embodiment of the present application, the initial source domain image and the initial target domain image are obtained, wherein the initial source domain image is a real labeled image, and the initial target domain image is an unlabeled image; based on the first image-level features of the initial source domain image and the second image-level features of the initial target domain image, the first attention map and the first domain discrimination loss value of the initial source domain image and the second attention map and the second domain discrimination loss value of the initial target domain image are determined; based on the first instance-level features of the initial source domain image and the second instance-level features of the initial target domain image, the third attention map and the third domain discrimination loss value of the initial source domain image and the fourth attention map and the fourth domain discrimination loss value of the initial target domain image are determined; based on the instance-level features of different category samples in the initial source domain image, the sample number of different category samples and the initial global prototype of the last iteration of the initial source domain image, the instance-level features of different category samples in the initial target domain image, the sample number of different category samples and the initial global prototype of the last iteration of the initial target domain image, the first global prototype of each category sample in the initial source domain image in the current iteration and the second global prototype of each category sample in the initial target domain image in the current iteration are determined; based on the first global prototype and the second global prototype, the first structure matrix of the initial source domain image and the second structure matrix of the initial target domain image are determined; the image domain consistency features of the initial source domain image and the initial target domain are input into the classification and regression detection head to obtain the first detection result of the initial source domain image and the second detection result of the initial target domain image; the target domain image to be detected is obtained to obtain the target detection result of the target domain image to be detected, wherein the target domain image to be detected is an unlabeled image, which solves the technical problem of the limitation of the existing unsupervised domain adaptive cross-scene target detection method, and achieves the technical effect that in the unsupervised cross-scene target detection, the method combining the double attention mechanism and the structure consistency learning between the source domain and the target domain can learn more effective domain consistency features of the source domain and the target domain, and high-efficiency cross-scene target detection is realized.

[0079] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0080] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0081] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of units can be a logical function division, and actual implementation can have another division mode, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.

[0082] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected to achieve the purpose of the embodiment according to actual needs.

[0083] In addition, each functional unit in each embodiment of the present application can be integrated in a first processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0084] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A cross-scene object detection method based on structural consistency learning, characterized by: include: Obtain an initial source domain image and an initial target domain image, wherein the initial source domain image is a real labeled image and the initial target domain image is an unlabeled image; Inputting the first image-level feature and the second image-level feature into a region proposal network to obtain a first prediction image result of an initial source domain image and a second prediction image result of an initial target domain image; The first prediction image result and the second prediction image result are processed through the channel to obtain the first feature of the first prediction image result and the second feature of the second prediction image result, and the first feature and the second feature are filtered using a preset threshold to generate the first attention map of the initial source domain image and the second attention map of the initial target domain image, wherein the first attention map and the second attention map can be obtained by A img Express it and get A img It can be calculated by the following formula: The indicator function I(·) is used to distinguish the foreground and background regions in , and the threshold t is specified as M RPN The average value of Inputting the first image-level feature and the second image-level feature into the domain discriminator for adversarial learning, obtaining a fifth domain discrimination loss value of the initial source domain image and a sixth domain discrimination loss value of the initial target domain image; Based on the first attention map and the fifth domain discrimination loss value, the first domain discrimination loss value of the initial source domain image is obtained; Based on the second attention map and the sixth domain discrimination loss value, the second domain discrimination loss value of the initial target domain image is obtained, where the first domain discrimination loss value and the second domain discrimination loss value can be calculated by the following formula: is the first attention map or the second attention map, is the fifth domain discrimination loss value or the sixth domain discrimination loss value, L Aimg is the first domain discrimination loss value or the second domain discrimination loss value, (u, v) is A img The rows and columns or L img rows and columns; Inputting the first instance-level feature and the second instance-level feature into a classification head, obtaining a first detection result of a first object in an initial source domain image and a second detection result of a second object in an initial target domain image, wherein the first detection result includes a category of each first object and a first probability score corresponding to the category of each first object, and the second detection result includes a category of each second object and a second probability score corresponding to the category of each second object; Based on the first probability score, a third attention map of the initial source domain image is obtained; Based on the second probability score, a fourth attention map of the initial target domain image is obtained, wherein the calculation process of the third attention map or the fourth attention map is: A ins is the third attention map or the fourth attention map, p is the first probability score or the second probability score, k is a hyperparameter, and c is the category of each first object of the first detection result or the category of each second object of the second detection result; Inputting the first instance-level feature and the second instance-level feature into the domain discriminator for adversarial learning, obtaining the seventh domain discrimination loss value of the initial source domain image and the eighth domain discrimination loss value of the initial target domain image; Based on the third attention map and the seventh domain discrimination loss value, the third domain discrimination loss value of the initial source domain image is obtained; Based on the fourth attention map and the eighth domain discrimination loss value, the fourth domain discrimination loss value of the initial target domain image is obtained, wherein the calculation process of the third domain discrimination loss value or the fourth domain discrimination loss value is: L Ains is the third domain discrimination loss value or the fourth domain discrimination loss value, is the fourth attention map or the fifth attention map; Determine, based on instance-level features of samples of different categories in the initial source domain image, the number of samples of different categories, and the initial global prototype of the previous iteration of the initial source domain image, instance-level features of samples of different categories in the initial target domain image, the number of samples of different categories, and the initial global prototype of the previous iteration of the initial target domain image, a first global prototype of the current iteration of each category of samples in the initial source domain image and a second global prototype of the current iteration of each category of samples in the initial target domain image; Based on the first global prototype of each category sample in the initial source domain image, construct a first structure matrix of the initial source domain image; Based on the second global prototype of each category sample in the initial target domain image, the second structure matrix of the initial target domain image is constructed, where the loss function is L SFC is the loss function between the first structure matrix and the second structure matrix, is the first structure matrix, The re in The rth row, the eth element, The re in The rth row, the eth element; Inputting the image domain consistency features of the initial source domain image and the initial target domain into the detection head of classification and regression to obtain a first detection result of the initial source domain image and a second detection result of the initial target domain image; A target domain image to be detected is obtained, and a target detection result of the target domain image to be detected is obtained, wherein the target domain image to be detected is an unlabeled image.

2. The method according to claim 1, characterized in that Before obtaining the initial source domain image and the initial target domain image, the method further includes: Obtain the original source domain image and the original target domain image, where the original source domain image is a real labeled image and the original target domain image is an unlabeled image; The original source domain image and the original target domain image are scaled and normalized respectively to obtain the initial source domain image and the initial target domain image.

3. The method according to claim 1, characterized in that Based on instance-level features of samples of different categories in the initial source domain image, the number of samples of different categories, and the initial global prototype of the previous iteration of the initial source domain image, instance-level features of samples of different categories in the initial target domain image, the number of samples of different categories, and the initial global prototype of the previous iteration of the initial target domain image, determining a first global prototype of the current iteration of each category sample in the initial source domain image and a second global prototype of the current iteration of each category sample in the initial target domain image, including: Determine the first local prototype of the current iteration of each category of samples in the initial source domain image based on the instance-level features of the samples of different categories and the number of samples of different categories in the initial source domain image; Determine a first global prototype of each category sample in the initial source domain image based on the first local prototype and the initial global prototype of the previous iteration of the initial source domain image; Determine the second local prototype of the current iteration of each category of samples in the initial target domain image based on the instance-level features of the samples of different categories and the number of samples of different categories in the initial target domain image; Based on the second local prototype and the initial global prototype of the previous iteration of the initial target domain image, a second global prototype of each category sample in the initial target domain image is determined.

4. The method according to claim 3, characterized in that Based on the instance-level features of samples of different categories in the initial source domain image and the number of samples of different categories, the expression for determining the first local prototype of each category sample in the initial source domain image is: in, represents the first local prototype of the c-type sample in the initial source domain image, N c represents the sample size of class c samples in the initial source domain image, represents the instance-level features of class c samples in the initial source domain image.

5. The method according to claim 3, characterized in that Based on the instance-level features of samples of different categories in the initial target domain image and the number of samples of different categories, the expression for determining the second local prototype of each category sample in the initial target domain image is: in, represents the second local prototype of the c-type sample in the initial target domain image, N c represents the sample size of class c samples in the initial target domain image, represents the instance-level features of class c samples in the initial target domain image.

6. A processor, characterized in that: The processor is configured to run a program, wherein the program executes the method according to any one of claims 1 to 5 when the program is run.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 5.

Images