Cross-scene target detection method based on structural consistency learning

By introducing structural consistency learning and dual attention mechanisms in cross-scene object detection, the problem of category imbalance is solved, and more effective domain consistency characteristics are learned, which improves the efficiency and accuracy of cross-scene object detection, especially in the sample The detection accuracy of the category with small quantities has been significantly improved.

CN119027808AActive Publication Date: 2024-11-26NORTHWESTERN POLYTECHNICAL UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411046200.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-26
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

The existing cross-scene object detection method based on unsupervised domain adaptation fails to effectively deal with the problem of category imbalance when aligning feature distribution, resulting in a certain type of sample with a small amount of data incorrectly aligning to the sample center with a large amount of data. Poor discrimination ability of a certain type of sample with small data volume.

Method used

Using a structural consistency learning method, by aligning the image-level features, instance-level features and structural features of the source and target domains, we introduce attention mechanisms to minimize interference from background information and learn more effective domain consistency. Features, and use a dual attention mechanism between the source domain and the target domain combined with structural consistency learning to achieve cross-domain object detection.

Benefits of technology

In unsupervised cross-scene object detection, through dual attention mechanism and structural consistency learning, more effective domain consistency characteristics of the source domain and target domain are learned, which improves the efficiency and accuracy of cross-scene object detection, especially The detection accuracy is significantly improved in the category of small sample size.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027808A_ABST
    Figure CN119027808A_ABST
Patent Text Reader

Abstract

The invention discloses a cross-scene target detection method based on structural consistency learning. The method comprises the steps that an initial source domain image and an initial target domain image are acquired, the initial source domain image is an image with a real label, and the initial target domain image is an image without a label; determining a first attention map and a first domain discrimination loss value of the initial source domain image and a second attention map and a second domain discrimination loss value of the initial target domain image based on the first image level feature of the initial source domain image and the second image level feature of the initial target domain image; and determining a third attention map and a third domain discrimination loss value of the initial source domain image and a fourth attention map and a fourth domain discrimination loss value of the initial target domain image based on the first instance-level feature of the initial source domain image and the second instance-level feature of the initial target domain image. The technical problem of limitation of a cross-scene target detection method based on unsupervised domain self-adaption in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a cross-scene target detection method based on structural consistency learning. Background Art

[0002] Object detection is a core task in the field of image analysis. In recent years, object detection technology based on deep learning has made great progress. However, deep learning methods rely heavily on massive amounts of high-quality labeled data, which is a challenge for new object detection scenarios. To address this issue, cross-scenario object detection technology has attracted widespread attention, aiming to assist in completing object detection tasks in the target domain by learning from the source domain. Specifically, researchers have tried to use unsupervised domain adaptation methods to reduce the cost of data annotation in new scenarios and improve the generalization performance of the detection system.

[0003] The first application of unsupervised domain adaptation to the object detection task was DA Faster proposed by Y. Chen et al. in "Y. Chen, W. Li, C. Sakaridis, D. Dai, and L. Van Gool,"Domain Adaptive Faster R-Cnn for Object Detection in the Wild," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3339-3348." This method measures the distribution difference between domains based on H divergence and aligns the image-level and instance-level feature distributions between domains. It also designs consistency regularization to learn the region proposal network. Since then, many related methods have emerged, such as the multi-granularity alignment network proposed by W. Zhou et al. in "W. Zhou, D. Du, L. Zhang, T. Luo, and Y. Wu,"Multi-Granularity Alignment Domain Adaptation for Object Detection," in Proc. IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9581-9590.". Considering that the transferability of each type of feature is different, this method changes the output dimension of the domain discriminator so that it can not only identify the domain to which the feature belongs but also distinguish the category of the feature, thereby gradually aligning the distribution of various features during adversarial training. The dual instance consistency network proposed by Jiao et al. in "Y.Jiao,H.Yao and C.Xu,"Dual Instance-Consistent Network for Cross-Domain Object Detection,"IEEE Transactions on Pattern Analysis and Machine Intelligence,vol.45,no.6,pp.7338-7352,2023." introduces a main network and an auxiliary network to generate feature spaces specific to the source domain and the target domain, focusing not only on feature consistency but also on the consistency of classification and regression in the two feature spaces.

[0004] The above method does not take into account the class imbalance problem in the data during the process of aligning feature distribution. This problem will cause a certain type of samples with a small amount of data to be incorrectly aligned to the center of samples with a large amount of data, that is, the model has poor discrimination ability for a certain type of samples with a small amount of data. Summary of the invention

[0005] An embodiment of the present invention provides a cross-scene target detection method based on structural consistency learning, so as to at least solve the technical problems of limitations of the cross-scene target detection method based on unsupervised domain adaptation in the prior art.

[0006] According to one aspect of an embodiment of the present invention, a cross-scene target detection method based on structural consistency learning is provided. The method may include: obtaining an initial source domain image and an initial target domain image, wherein the initial source domain image is an image with a real label, and the initial target domain image is an image without a label; based on the first image-level features of the initial source domain image and the second image-level features of the initial target domain image, determining the first attention map and the first domain discrimination loss value of the initial source domain image and the second attention map and the second domain discrimination loss value of the initial target domain image; based on the first instance-level features of the initial source domain image and the second instance-level features of the initial target domain image, determining the third attention map and the third domain discrimination loss value of the initial source domain image, and the fourth attention map and the fourth domain discrimination loss value of the initial target domain image; based on the instance-level features of samples of different categories in the initial source domain image, the number of samples of different categories, and the initial global discrimination loss value of the previous iteration of the initial source domain image The first global prototype of each category sample in the initial source domain image in the current iteration and the second global prototype of each category sample in the initial target domain image in the current iteration are determined based on the first global prototype and the second global prototype; based on the first global prototype and the second global prototype, the first structure matrix of the initial source domain image and the second structure matrix of the initial target domain image are determined; the image domain consistency features of the initial source domain image and the initial target domain are input into the detection head of classification and regression to obtain the first detection result of the initial source domain image and the second detection result of the initial target domain image; the target domain image to be detected is obtained to obtain the target detection result of the target domain image to be detected, wherein the target domain image to be detected is an unlabeled image.

[0007] Optionally, before obtaining the initial source domain image and the initial target domain image, the method further includes: obtaining an original source domain image and an original target domain image, wherein the original source domain image is an image with a real label and the original target domain image is an image without a label; and performing scaling and normalization operations on the original source domain image and the original target domain image, respectively, to obtain the initial source domain image and the initial target domain image.

[0008] Optionally, based on the first image-level features of the initial source domain image and the second image-level features of the initial target domain image, determining the first attention map and the first domain discrimination loss value of the initial source domain image and the second attention map and the second domain discrimination loss value of the initial target domain image, including: inputting the first image-level features and the second image-level features into the region proposal network to obtain the first prediction map result of the initial source domain image and the second prediction map result of the initial target domain image; processing the first prediction map result and the second prediction map result through the channel to obtain the first feature of the first prediction map result and the second feature of the second prediction map result, and filtering the first feature and the second feature using a preset threshold to generate the first attention map of the initial source domain image and the second attention map of the initial target domain image; inputting the first image-level features and the second image-level features into the domain discriminator for adversarial learning to obtain the fifth domain discrimination loss value of the initial source domain image and the sixth domain discrimination loss value of the initial target domain image; based on the first attention map and the fifth domain discrimination loss value, obtaining the first domain discrimination loss value of the initial source domain image; based on the second attention map and the sixth domain discrimination loss value, obtaining the second domain discrimination loss value of the initial target domain image.

[0009] Optionally, based on the first instance-level feature of the initial source domain image and the second instance-level feature of the initial target domain image, determining a third attention map and a third domain discrimination loss value of the initial source domain image, and a fourth attention map and a fourth domain discrimination loss value of the initial target domain image, including: inputting the first instance-level feature and the second instance-level feature into a classification head to obtain a first detection result of a first object in the initial source domain image and a second detection result of a second object in the initial target domain image, wherein the first detection result includes a category of each first object and a first probability score corresponding to the category of each first object, and the second detection result includes a category of each second object. A second probability score corresponding to the category of each second object is obtained; based on the first probability score, a third attention map of the initial source domain image is obtained; based on the second probability score, a fourth attention map of the initial target domain image is obtained; the first instance-level feature and the second instance-level feature are input into the domain discriminator for adversarial learning to obtain the seventh domain discrimination loss value of the initial source domain image and the eighth domain discrimination loss value of the initial target domain image; based on the third attention map and the seventh domain discrimination loss value, a third domain discrimination loss value of the initial source domain image is obtained; based on the fourth attention map and the eighth domain discrimination loss value, a fourth domain discrimination loss value of the initial target domain image is obtained.

[0010] Optionally, based on instance-level features of samples of different categories in the initial source domain image, the number of samples of different categories and the initial global prototype of the previous iteration of the initial source domain image, instance-level features of samples of different categories in the initial target domain image, the number of samples of different categories and the initial global prototype of the previous iteration of the initial target domain image, determining the first global prototype of the current iteration of each sample of each category in the initial source domain image and the second global prototype of the current iteration of each sample of each category in the initial target domain image, including: determining the first local prototype of the current iteration of each sample of each category in the initial source domain image based on the instance-level features of samples of different categories and the number of samples of different categories in the initial source domain image; determining the first global prototype of each sample of each category in the initial source domain image based on the first local prototype and the initial global prototype of the previous iteration of the initial source domain image; determining the second local prototype of each sample of each category in the initial target domain image based on the instance-level features of samples of different categories in the initial target domain image and the number of samples of different categories; determining the second global prototype of each sample of each category in the initial target domain image based on the second local prototype and the initial global prototype of the previous iteration of the initial target domain image.

[0011] Optionally, based on instance-level features of samples of different categories and the number of samples of different categories in the initial source domain image, an expression for determining the first local prototype of each category sample in the initial source domain image is: in, represents the first local prototype of the c-type sample in the initial source domain image, N c represents the sample size of class c samples in the initial source domain image, Represents the instance-level features of the c-class samples in the initial source domain image. The expression of the second local prototype of each category sample in the image is: in, represents the second local prototype of the c-type sample in the initial target domain image, N c represents the sample size of class c samples in the initial target domain image, represents the instance-level features of class c samples in the initial target domain image.

[0012] Optionally, based on the first global prototype and the second global prototype, determining a first structure matrix of the initial source domain image and a second structure matrix of the initial target domain image includes: constructing the first structure matrix of the initial source domain image based on the first global prototype of each category sample in the initial source domain image; and constructing the second structure matrix of the initial target domain image based on the second global prototype of each category sample in the initial target domain image.

[0013] Beneficial effects of the present invention:

[0014] The present invention proposes a cross-scene target detection method based on structural consistency learning. By aligning the image-level features, instance-level features and structural features of the source domain and the target domain, more effective domain consistency features can be learned. Attention mechanism is introduced at the same time in the image-level and instance-level feature alignment stages to minimize the interference of background information and realize cross-domain target detection. The technical problem of the limitations of the cross-scene target detection method based on unsupervised domain adaptation in the prior art is solved. In unsupervised cross-scene target detection, a method combining a dual attention mechanism with structural consistency learning between the source domain and the target domain is used to learn more effective domain consistency features of the source domain and the target domain, thereby achieving the technical effect of efficient cross-scene target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0016] Figure 1 is a flow chart of a cross-scene target detection method based on structural consistency learning according to an embodiment of the present invention;

[0017] Figure 2 Schematic diagram of data volume distribution of target domain test data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] Example 1

[0021] According to an embodiment of the present invention, a cross-scene target detection method based on structural consistency learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0022] Figure 1 is a flow chart of a cross-scene target detection method based on structural consistency learning according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0023] Step S101, obtaining an initial source domain image and an initial target domain image, wherein the initial source domain image is an image with a real label, and the initial target domain image is an image without a label.

[0024] In the technical solution provided in the above step S101 of the present invention, an initial source domain image and an initial target domain image are obtained, wherein the initial source domain image is an image with a real label, and the initial target domain image is an image without a real label.

[0025] Step S102, based on the first image-level features of the initial source domain image and the second image-level features of the initial target domain image, determine the first attention map and the first domain discrimination loss value of the initial source domain image and the second attention map and the second domain discrimination loss value of the initial target domain image.

[0026] In the technical solution provided in the above step S102 of the present invention, a first attention map and a first domain discrimination loss value of the initial source domain image are obtained according to the first image-level features of the initial source domain image; and a second attention map and a second domain discrimination loss value of the initial target domain image are obtained according to the second image-level features of the initial target domain image.

[0027] Step S103, based on the first instance-level features of the initial source domain image and the second instance-level features of the initial target domain image, determine the third attention map and the third domain discrimination loss value of the initial source domain image, and the fourth attention map and the fourth domain discrimination loss value of the initial target domain image.

[0028] In the technical solution provided in the above step S103 of the present invention, a third attention map and a third domain discrimination loss value of the initial source domain image are obtained according to the first instance-level features of the initial source domain image; and a fourth attention map and a fourth domain discrimination loss value of the initial target domain image are obtained according to the second instance-level features of the initial target domain image.

[0029] Step S104, based on the instance-level features of samples of different categories in the initial source domain image, the number of samples of different categories and the initial global prototype of the previous iteration of the initial source domain image, the instance-level features of samples of different categories in the initial target domain image, the number of samples of different categories and the initial global prototype of the previous iteration of the initial target domain image, determine the first global prototype of the current iteration of each category sample in the initial source domain image and the second global prototype of the current iteration of each category sample in the initial target domain image.

[0030] In the technical solution provided in the above step S104 of the present invention, a calculation is performed based on the instance-level features of samples of different categories in the initial source domain image, the number of samples of different categories, and the initial global prototype of the previous iteration of the initial source domain image to obtain the first global prototype of the current iteration of each category of samples in the initial source domain image; a calculation is performed based on the instance-level features of samples of different categories in the initial target domain image, the number of samples of different categories, and the initial global prototype of the previous iteration of the initial target domain image to obtain the second global prototype of the current iteration of each category of samples in the initial target domain image.

[0031] Step S105 : determining a first structure matrix of the initial source domain image and a second structure matrix of the initial target domain image based on the first global prototype and the second global prototype.

[0032] In the technical solution provided in the above step S105 of the present invention, a first structure matrix of the initial source domain image is constructed according to the first global prototype, and a second structure matrix of the initial target domain image is constructed according to the second global prototype.

[0033] Step S106: input the image domain consistency features of the initial source domain image and the initial target domain into the detection head of classification and regression to obtain a first detection result of the initial source domain image and a second detection result of the initial target domain image.

[0034] In the technical solution provided in the above step S106 of the present invention, the image domain consistency features of the initial source domain image and the initial target domain are input into the detection head of classification and regression to obtain a first detection result of the initial source domain image and a second detection result of the initial target domain image.

[0035] Step S107 , acquiring a target domain image to be detected, and obtaining a target detection result of the target domain image to be detected, wherein the target domain image to be detected is an unlabeled image.

[0036] In the technical solution provided in the above step S107 of the present invention, a target domain image to be detected is obtained, and the target domain image to be detected is input into the model trained in the above steps S101 to S106 to obtain a target detection result of the target domain image to be detected.

[0037] The above method of this embodiment is further introduced below.

[0038] As an optional implementation manner, in step S101, before obtaining the initial source domain image and the initial target domain image, the method further includes: obtaining the original source domain image and the original target domain image, wherein the original source domain image is an image with a real label, and the original target domain image is an image without a label; and performing scaling and normalization operations on the original source domain image and the original target domain image, respectively, to obtain the initial source domain image and the initial target domain image.

[0039] In this embodiment, an original source domain image and an original target domain image are obtained, and the original source domain image and the original target domain image are respectively subjected to unified scaling and normalization operations to obtain an initial source domain image and an initial target domain image.

[0040] As an optional implementation method, step S102, based on the first image-level features of the initial source domain image and the second image-level features of the initial target domain image, determines the first attention map and the first domain discrimination loss value of the initial source domain image and the second attention map and the second domain discrimination loss value of the initial target domain image, including: inputting the first image-level features and the second image-level features into the region proposal network to obtain the first prediction map result of the initial source domain image and the second prediction map result of the initial target domain image; processing the first prediction map result and the second prediction map result through the channel to obtain the first feature and the first prediction map result The first feature of the second prediction image result is obtained, and the first feature and the second feature are filtered using a preset threshold to generate a first attention map of the initial source domain image and a second attention map of the initial target domain image; the first image-level feature and the second image-level feature are input into the domain discriminator for adversarial learning to obtain the fifth domain discrimination loss value of the initial source domain image and the sixth domain discrimination loss value of the initial target domain image; based on the first attention map and the fifth domain discrimination loss value, the first domain discrimination loss value of the initial source domain image is obtained; based on the second attention map and the sixth domain discrimination loss value, the second domain discrimination loss value of the initial target domain image is obtained.

[0041] In this embodiment, the first image-level feature is input into the region proposal network to obtain a first prediction image result of the initial source domain image; the second image-level feature is input into the region proposal network to obtain a second prediction image result of the initial target domain image, wherein the first prediction image and the second prediction image can be used to obtain the first prediction image. RPN The first prediction graph result and the second prediction graph result are processed through the channel to obtain the first feature of the first prediction graph result and the second feature of the second prediction graph result, wherein the first feature and the second feature can be obtained by M RPNRepresented, and the first feature and the second feature are filtered using a preset threshold to generate a first attention map of the initial source domain image and a second attention map of the initial target domain image, wherein the first attention map and the second attention map can be obtained by A img Expressed as A img It can be calculated by the following formula:

[0042]

[0043] Among them, the indicator function To distinguish M RPN The foreground and background regions in the image are given by the threshold t, which is specified as M RPN The average value of .

[0044] The first image-level feature is input into the domain discriminator for adversarial learning to obtain the fifth domain discrimination loss value of the initial source domain image; based on the first attention map and the fifth domain discrimination loss value, the first domain discrimination loss value of the initial source domain image is obtained; the second image-level feature is input into the domain discriminator for adversarial learning to obtain the sixth domain discrimination loss value of the initial target domain image; based on the second attention map and the sixth domain discrimination loss value, the second domain discrimination loss value of the initial target domain image is obtained, wherein the first domain discrimination loss value and the second domain discrimination loss value can be calculated by the following formula:

[0045]

[0046] in, is the first attention map or the second attention map, is the fifth domain discrimination loss value or the sixth domain discrimination loss value, L Aimg is the first domain discrimination loss value or the second domain discrimination loss value, (u,v) is A img The rows and columns or L img rows and columns.

[0047] As an optional implementation method, step S103, based on the first instance-level feature of the initial source domain image and the second instance-level feature of the initial target domain image, determines the third attention map and the third domain discrimination loss value of the initial source domain image, the fourth attention map and the fourth domain discrimination loss value of the initial target domain image, including: inputting the first instance-level feature and the second instance-level feature into the classification head, obtaining a first detection result of the first object in the initial source domain image and a second detection result of the second object in the initial target domain image, wherein the first detection result includes the category of each first object and the first probability score corresponding to the category of each first object, and the second detection result includes The category of each second object and the second probability score corresponding to the category of each second object; based on the first probability score, a third attention map of the initial source domain image is obtained; based on the second probability score, a fourth attention map of the initial target domain image is obtained; the first instance-level feature and the second instance-level feature are input into the domain discriminator for adversarial learning to obtain the seventh domain discrimination loss value of the initial source domain image and the eighth domain discrimination loss value of the initial target domain image; based on the third attention map and the seventh domain discrimination loss value, the third domain discrimination loss value of the initial source domain image is obtained; based on the fourth attention map and the eighth domain discrimination loss value, the fourth domain discrimination loss value of the initial target domain image is obtained.

[0048] In this embodiment, the first instance-level feature is input to the classification head to obtain a first detection result of the first object in the initial source domain image; the second instance-level feature is input to the classification head to obtain a second detection result of the second object in the initial target domain image, and a third attention map of the initial source domain image is obtained according to the first probability score; and a fourth attention map of the initial target domain image is obtained according to the second probability score, wherein the calculation process of the third attention map or the fourth attention map is as follows:

[0049]

[0050] Among them, A ins is the third attention map or the fourth attention map, p is the first probability score or the second probability score, k is a hyperparameter, and c is the category of each first object of the first detection result or the category of each second object of the second detection result.

[0051] The first instance-level feature and the second instance-level feature are input into the domain discriminator for adversarial learning to obtain the seventh domain discrimination loss value of the initial source domain image and the eighth domain discrimination loss value of the initial target domain image, wherein the seventh domain discrimination loss value or the eighth domain discrimination loss value is used It is represented by, where i is the number of image frames in a batch;

[0052] According to the third attention map and the seventh domain discrimination loss value, the third domain discrimination loss value of the initial source domain image is obtained; according to the fourth attention map and the eighth domain discrimination loss value, the fourth domain discrimination loss value of the initial target domain image is obtained. The calculation process of the third domain discrimination loss value or the fourth domain discrimination loss value is as follows:

[0053]

[0054] Among them, L Ains is the third domain discrimination loss value or the fourth domain discrimination loss value, It is the fourth attention map or the fifth attention map.

[0055] As an optional implementation manner, step S104, based on instance-level features of samples of different categories in the initial source domain image, the number of samples of different categories and the initial global prototype of the previous iteration of the initial source domain image, instance-level features of samples of different categories in the initial target domain image, the number of samples of different categories and the initial global prototype of the previous iteration of the initial target domain image, determines the first global prototype of the current iteration of each category sample in the initial source domain image and the second global prototype of the current iteration of each category sample in the initial target domain image, including: based on the instance-level features of samples of different categories in the initial source domain image and the number of samples of different categories The number of samples of the same category is used to determine the first local prototype of the current iteration of each category sample in the initial source domain image; based on the first local prototype and the initial global prototype of the previous iteration of the initial source domain image, the first global prototype of each category sample in the initial source domain image is determined; based on the instance-level features of samples of different categories and the number of samples of different categories in the initial target domain image, the second local prototype of the current iteration of each category sample in the initial target domain image is determined; based on the second local prototype and the initial global prototype of the previous iteration of the initial target domain image, the second global prototype of each category sample in the initial target domain image is determined.

[0056] In this embodiment, according to the instance-level features of samples of different categories and the number of samples of different categories in the initial source domain image, the first local prototype of the current iteration of each category sample in the initial source domain image is obtained; according to the instance-level features of samples of different categories and the number of samples of different categories in the initial target domain image, the second local prototype of the current iteration of each category sample in the initial target domain image is obtained; according to the first local prototype and the initial global prototype of the previous iteration of the initial source domain image, the first global prototype of each category sample in the initial source domain image is obtained, and according to the second local prototype and the initial global prototype of the previous iteration of the initial target domain image, the second global prototype of each category sample in the initial target domain image is obtained, wherein the calculation process of the first global prototype or the second global prototype is as follows:

[0057] P global (m) = αplocal (m)+(1-α)P global (m-1) (7)

[0058] Among them, P global (m) is the first global prototype or the second global prototype of the mth iteration, p local (m) is the first local prototype or the second local prototype, α represents the cosine similarity between the first local prototype and the initial global prototype of the previous iteration of the initial source domain image or the cosine similarity between the second local prototype and the initial global prototype of the previous iteration of the initial target domain image, P global (m-1) is the initial global prototype of the previous iteration of the initial source domain image or the initial global prototype of the previous iteration of the initial target domain image.

[0059] As an optional embodiment, based on the instance-level features of samples of different categories and the number of samples of different categories in the initial source domain image, the expression for determining the first local prototype of each category sample in the initial source domain image is:

[0060]

[0061] in, represents the first local prototype of the c-type sample in the initial source domain image, N c represents the sample size of class c samples in the initial source domain image, represents the instance-level features of class c samples in the initial source domain image.

[0062] As an optional embodiment, based on the instance-level features of samples of different categories and the number of samples of different categories in the initial target domain image, the expression for determining the second local prototype of each category sample in the initial target domain image is:

[0063]

[0064] in, represents the second local prototype of the c-type sample in the initial target domain image, N c represents the sample size of class c samples in the initial target domain image, represents the instance-level features of class c samples in the initial target domain image.

[0065] As an optional implementation method, step S105, based on the first global prototype and the second global prototype, determines the first structure matrix of the initial source domain image and the second structure matrix of the initial target domain image, including: based on the first global prototype of each category sample in the initial source domain image, constructing the first structure matrix of the initial source domain image; based on the second global prototype of each category sample in the initial target domain image, constructing the second structure matrix of the initial target domain image.

[0066] In this embodiment, a first structural matrix of the initial source domain image is constructed according to the first global prototype of each category sample in the initial source domain image, and a second structural matrix of the initial target domain image is constructed according to the second global prototype of each category sample in the initial target domain image. The design of the first structural matrix or the second structural matrix takes into account the distance between the global prototypes of different categories and reflects the differences between the categories. In order to effectively reduce the structural distribution differences between the source domain and the target domain, the embodiment of the present invention introduces the following loss function.

[0067]

[0068] Among them, L SFC is the loss function between the first structure matrix and the second structure matrix, is the first structure matrix, The re in The rth row, the eth element of The re in The r-th row, e-th element of .

[0069] The effects of the embodiments of the present invention can be further illustrated by the following simulation experiments.

[0070] 1. Simulation conditions

[0071] The present invention is a simulation performed on NVIDIA GeForce RTX 4090 GPU using Visual Studio Code software. The data used in the simulation is a public cross-weather data set.

[0072] 2. Simulation content

[0073] First, the model is trained using the training data of the source domain and its true labels and the training data of the target domain according to the steps in the specific implementation method; then the trained model is applied to the target detection task of the target domain test data, and AP and MAP are calculated to evaluate the model. According to an embodiment of the present invention, the data volume distribution of the target domain test data is provided. Figure 2 Schematic diagram of data volume distribution of target domain test data according to an embodiment of the present invention.

[0074] In order to prove the effectiveness of the present invention, MGADA, DICN and CRADA are selected as comparison algorithms. The MGADA algorithm is proposed in the document "W.Zhou, D.Du, L.Zhang, T.Luo and Y.Wu,"Multi-Granularity AlignmentDomain Adaptation for Object Detection," in Proc.2022IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp.9571-9580."; the DICN algorithm is proposed in the document "Y.Jiao, H.Yao and C.Xu,"Dual Instance-Consistent Network for Cross-DomainObject Detection," IEEE Transactions on Pattern Analysis and MachineIntelligence, vol.45, no.6, pp.7338-7352, 2023."; the CRADA algorithm is proposed in the document "Y.Liu, J.Wang, W.Wang, Y.Hu, Y.Wang and Y.Xu,"CRADA:Cross Domain Object Detection WithCyclic Reconstruction and Decoupling Adaptation,"IEEE Transactions on Multimedia,vol.26,pp.6250-6261,2024." The comparison results are shown in Table 1.

[0075] Table 1 Comparison results of different algorithms

[0076] method Person Rider Car Truck Bus Train Mbike Bicycle MAP DICN 47.3 57.4 64.0 22.7 45.6 29.6 38.6 47.4 44.1 MGADA 43.9 49.6 60.6 29.6 50.7 39.0 38.3 42.8 44.3 CRADA 47.0 48.3 64.1 28.7 52.3 38.6 35.1 41.5 44.5 The present invention 44.9 51.8 61.9 32.4 48.8 41.2 36.8 46.7 45.6

[0077] As can be seen from Table 1, the present invention achieves the highest MAP compared to previous cross-scenario target detection methods. More noteworthy is that although the detection effect of the present invention on the Bus category is average, it significantly improves the detection accuracy on the Truck and Train categories. The present invention introduces an attention mechanism at the image level and instance level to make the network pay more attention to foreground features, thereby improving the overall detection performance. In addition, the present invention can effectively balance the distribution distances between different categories through structural consistency learning, thereby enhancing the detection accuracy of categories with fewer samples.

[0078] In an embodiment of the present invention, an initial source domain image and an initial target domain image are obtained, wherein the initial source domain image is an image with a real label, and the initial target domain image is an unlabeled image; based on the first image-level feature of the initial source domain image and the second image-level feature of the initial target domain image, a first attention map and a first domain discrimination loss value of the initial source domain image and a second attention map and a second domain discrimination loss value of the initial target domain image are determined; based on the first instance-level feature of the initial source domain image and the second instance-level feature of the initial target domain image, a third attention map and a third domain discrimination loss value of the initial source domain image, and a fourth attention map and a fourth domain discrimination loss value of the initial target domain image are determined; based on the instance-level features of samples of different categories in the initial source domain image, the number of samples of different categories and the initial global prototype of the previous iteration of the initial source domain image, the instance-level features of samples of different categories in the initial target domain image, the number of samples of different categories and the initial global prototype of the previous iteration of the initial target domain image, an initial source domain map is determined; The first global prototype of the current iteration of each category sample in the image and the second global prototype of the current iteration of each category sample in the initial target domain image; based on the first global prototype and the second global prototype, determine the first structure matrix of the initial source domain image and the second structure matrix of the initial target domain image; input the image domain consistency features of the initial source domain image and the initial target domain into the detection head of classification and regression to obtain the first detection result of the initial source domain image and the second detection result of the initial target domain image; obtain the target domain image to be detected, and obtain the target detection result of the target domain image to be detected, wherein the target domain image to be detected is an unlabeled image, which solves the technical problem of the limitations of the cross-scene target detection method based on unsupervised domain adaptation in the prior art, and achieves that in unsupervised cross-scene target detection, by using a method combining a dual attention mechanism with structural consistency learning between the source domain and the target domain, more effective domain consistency features of the source domain and the target domain can be learned, thereby achieving the technical effect of efficient cross-scene target detection.

[0079] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0080] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0083] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0084] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A cross-scene target detection method based on structural consistency learning, characterized in that: include: Obtain an initial source domain image and an initial target domain image, wherein the initial source domain image is an image with a real label, and the initial target domain image is an image without a label; Based on the first image-level feature of the initial source domain image and the second image-level feature of the initial target domain image, determining a first attention map and a first domain discrimination loss value of the initial source domain image and a second attention map and a second domain discrimination loss value of the initial target domain image; Based on the first instance-level feature of the initial source domain image and the second instance-level feature of the initial target domain image, determining a third attention map and a third domain discrimination loss value of the initial source domain image, and a fourth attention map and a fourth domain discrimination loss value of the initial target domain image; Based on instance-level features of samples of different categories in the initial source domain image, the number of samples of different categories, and the initial global prototype of the previous iteration of the initial source domain image, instance-level features of samples of different categories in the initial target domain image, the number of samples of different categories, and the initial global prototype of the previous iteration of the initial target domain image, determine the first global prototype of the current iteration of each category sample in the initial source domain image and the second global prototype of the current iteration of each category sample in the initial target domain image; Based on the first global prototype and the second global prototype, determining a first structure matrix of the initial source domain image and a second structure matrix of the initial target domain image; Inputting the image domain consistency features of the initial source domain image and the initial target domain into the detection head of classification and regression to obtain a first detection result of the initial source domain image and a second detection result of the initial target domain image; A target domain image to be detected is obtained to obtain a target detection result of the target domain image to be detected, wherein the target domain image to be detected is an unlabeled image.

2. The method according to claim 1, characterized in that Before acquiring the initial source domain image and the initial target domain image, the method further includes: Obtain an original source domain image and an original target domain image, wherein the original source domain image is an image with a real label, and the original target domain image is an image without a label; The original source domain image and the original target domain image are scaled and normalized respectively to obtain the initial source domain image and the initial target domain image.

3. The method according to claim 1, characterized in that Based on the first image-level feature of the initial source domain image and the second image-level feature of the initial target domain image, determining a first attention map and a first domain discrimination loss value of the initial source domain image and a second attention map and a second domain discrimination loss value of the initial target domain image, including: Inputting the first image-level feature and the second image-level feature into a region proposal network to obtain a first prediction image result of an initial source domain image and a second prediction image result of an initial target domain image; Processing the first prediction image result and the second prediction image result through the channel to obtain a first feature of the first prediction image result and a second feature of the second prediction image result, and filtering the first feature and the second feature using a preset threshold to generate a first attention map of the initial source domain image and a second attention map of the initial target domain image; Inputting the first image-level feature and the second image-level feature into a domain discriminator for adversarial learning, and obtaining a fifth domain discriminant loss value of the initial source domain image and a sixth domain discriminant loss value of the initial target domain image; Based on the first attention map and the fifth domain discrimination loss value, the first domain discrimination loss value of the initial source domain image is obtained; Based on the second attention map and the sixth domain discrimination loss value, the second domain discrimination loss value of the initial target domain image is obtained.

4. The method according to claim 1, characterized in that: Based on the first instance-level feature of the initial source domain image and the second instance-level feature of the initial target domain image, determining a third attention map and a third domain discrimination loss value of the initial source domain image, and a fourth attention map and a fourth domain discrimination loss value of the initial target domain image, including: Inputting the first instance-level feature and the second instance-level feature into a classification head, obtaining a first detection result of a first object in an initial source domain image and a second detection result of a second object in an initial target domain image, wherein the first detection result includes a category of each first object and a first probability score corresponding to the category of each first object, and the second detection result includes a category of each second object and a second probability score corresponding to the category of each second object; Based on the first probability score, a third attention map of the initial source domain image is obtained; Based on the second probability score, obtaining a fourth attention map of the initial target domain image; Inputting the first instance-level feature and the second instance-level feature into a domain discriminator for adversarial learning, and obtaining a seventh domain discrimination loss value of an initial source domain image and an eighth domain discrimination loss value of an initial target domain image; Based on the third attention map and the seventh domain discrimination loss value, the third domain discrimination loss value of the initial source domain image is obtained; Based on the fourth attention map and the eighth domain discrimination loss value, the fourth domain discrimination loss value of the initial target domain image is obtained.

5. The method according to claim 1, characterized in that Based on instance-level features of samples of different categories in the initial source domain image, the number of samples of different categories, and the initial global prototype of the previous iteration of the initial source domain image, instance-level features of samples of different categories in the initial target domain image, the number of samples of different categories, and the initial global prototype of the previous iteration of the initial target domain image, determine the first global prototype of the current iteration of each category sample in the initial source domain image and the second global prototype of the current iteration of each category sample in the initial target domain image, including: Determine the first local prototype of the current iteration of each category sample in the initial source domain image based on the instance-level features of the samples of different categories and the number of samples of different categories in the initial source domain image; Determine a first global prototype of each category sample in the initial source domain image based on the first local prototype and the initial global prototype of the previous iteration of the initial source domain image; Determine the second local prototype of the current iteration of each category sample in the initial target domain image based on the instance-level features of the samples of different categories and the number of samples of different categories in the initial target domain image; Based on the second local prototype and the initial global prototype of the previous iteration of the initial target domain image, a second global prototype of each category sample in the initial target domain image is determined.

6. The method according to claim 5, characterized in that Based on the instance-level features of samples of different categories in the initial source domain image and the number of samples of different categories, the expression for determining the first local prototype of each category sample in the initial source domain image is: in, represents the first local prototype of the c-type sample in the initial source domain image, N c represents the sample size of class c samples in the initial source domain image, represents the instance-level features of class c samples in the initial source domain image.

7. The method according to claim 5, characterized in that Based on the instance-level features of samples of different categories in the initial target domain image and the number of samples of different categories, the expression for determining the second local prototype of each category sample in the initial target domain image is: in, represents the second local prototype of the c-type sample in the initial target domain image, N c represents the sample size of class c samples in the initial target domain image, represents the instance-level features of class c samples in the initial target domain image.

8. The method according to claim 1, characterized in that Based on the first global prototype and the second global prototype, determining a first structure matrix of the initial source domain image and a second structure matrix of the initial target domain image includes: Based on the first global prototype of each category sample in the initial source domain image, construct a first structure matrix of the initial source domain image; Based on the second global prototype of each category sample in the initial target domain image, a second structure matrix of the initial target domain image is constructed.

9. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method of any one of claims 1 to 8 when it is run.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods in claims 1 to 8.

Citation Information

Patent Citations

  • Unsupervised domain adaptive method for beneficial feature alignment under class condition

    CN113807371A

  • Training method based on image-instance alignment network and cross-domain target detection method

    CN114693983A

  • Image detection model training method and device and image detection method

    CN115331097A

  • Cross-domain target detection method based on comparative learning

    CN116309466A