Remote sensing cross-domain object counting method and device based on style migration

Abstract

The invention provides a remote sensing cross-domain object counting method and a remote sensing cross-domain object counting device based on style migration. The remote sensing cross-domain object counting method comprises the following steps of: aligning the size of a target category object in a source domain data set to the size of a target category object in a target domain data set on the basis of an identification result of a semantic visual perception model; dividing each image in the target domain data set and the size-aligned source domain data set into a plurality of local images, and screening the local images divided by each target domain data set from the local images divided by the size-aligned source domain data set to obtain an image with the highest semantic and distribution consistency with the local images to form a local image pair; and carrying out fine tuning on the counting model on an image pair data set formed by the plurality of groups of local image pairs to obtain a counting model of a target domain, and determining the number of target category objects in the to-be-identified image through a technical model of the target domain. According to the method, more stable and more robust cross-domain adaptation is realized while the precision of the cross-domain object counting model is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing cross-domain object counting method and device based on style migration. Background Art

[0002] In recent years, with the increasing application of remote sensing technology in fields such as traffic management, environmental monitoring, and urban planning, accurately counting objects in remote sensing imagery has become a critical task. Thanks to the rapid development of artificial intelligence, deep learning-based object counting methods have achieved remarkable results.

[0003] However, deep neural networks usually assume that training data and test data satisfy the independent and identically distributed assumption. However, in actual applications, this assumption is often violated due to differences in data collection methods and environments, resulting in a significant decrease in model performance when applied across domains.

[0004] In other words, the model trained using labeled data from the source domain will perform poorly on the test data in the target domain due to distribution differences, affecting the current cross-domain object counting accuracy. Summary of the Invention

[0005] The present invention provides a remote sensing cross-domain object counting method and device based on style transfer, which is used to solve the defect of poor cross-domain performance of neural network models in the existing technology and realize a remote sensing cross-domain object counting method with higher recognition and counting accuracy in the target domain.

[0006] The present invention provides a remote sensing cross-domain object counting method based on style transfer, comprising: Based on the recognition results of the semantic visual perception model, the salient regions of each image in the target domain dataset and the source domain dataset are determined. Based on the average size of the target category objects in the salient regions, the size of the target category objects in the source domain dataset is aligned with the size of the target category objects in the target domain dataset. Each image in the target domain dataset and the size-aligned source domain dataset is divided into multiple local images, and each local image divided in the target domain dataset is screened from the local images divided in the size-aligned source domain dataset to obtain the image with the highest semantic and distribution consistency to form a local image pair; Fine-tune the counting model on an image pair dataset consisting of multiple groups of local image pairs to make the model more stylistically adaptable to the target domain, obtain a counting model for the target domain, and determine the number of target category objects in the image to be identified using the technical model of the target domain. According to a remote sensing cross-domain object counting method based on style transfer provided by the present invention, before the step of aligning the size of the target category objects in the source domain dataset with the size of the target category objects in the target domain dataset based on the average size of the target category objects in the salient region, the method further includes: In the salient areas of each image, the open set object perception model is used to identify the initial candidate boxes; At least use non-maximum suppression to screen the initial candidate boxes to obtain confident candidate boxes; The average size of the target category objects in the confidence candidate box of each image is used as the average size of the target category objects in the salient area.

[0007] According to a remote sensing cross-domain object counting method based on style transfer provided by the present invention, the step of screening the local images divided by each target domain dataset from the local images divided by the source domain dataset after size alignment to obtain the image with the highest semantic and distribution consistency to form a local image pair specifically includes: Based on the semantic features of each local image, the local images divided by each target domain dataset are first screened among the local images divided by the source domain dataset after size alignment, and the preset number of images closest to them on the manifold are obtained as their corresponding source domain candidate local image group; For each local image divided by the target domain dataset, a second screening is performed in its source domain candidate local image group to obtain the image with the highest semantic and distribution consistency, which together constitutes a local image pair.

[0008] According to a remote sensing cross-domain object counting method based on style transfer provided by the present invention, the step of performing a second screening of each local image divided by the target domain dataset in its source domain candidate local image group to obtain the image with the highest semantic and distribution consistency specifically includes: Obtaining a first label of a target category object in each image of the source domain candidate local image group; Obtaining a second label of a target category object in each local image divided by the target domain dataset; Calculating a structural similarity index between each first label and the second label of the source domain candidate local image group; The first label corresponding to the structural similarity index having the largest calculated value is determined and used as the image with the highest semantic and distribution consistency.

[0009] According to a remote sensing cross-domain object counting method based on style transfer provided by the present invention, before the step of obtaining the first label of the target category object in each local image divided by the target domain dataset, the method further includes: A first baseline model is trained on a source domain dataset to obtain a first label; Inputting the unlabeled images in the target domain dataset into the first baseline model to obtain a density map output by the baseline model as an initial pseudo label for the unlabeled images in the target domain dataset; The first baseline model is fine-tuned using the initial pseudo-label as supervision information to obtain a second baseline model, where the second baseline model is used to obtain the second label.

[0010] According to a remote sensing cross-domain object counting method based on style transfer provided by the present invention, the step of fine-tuning the counting model on an image pair dataset consisting of multiple groups of local image pairs specifically includes: Adaptive instance normalization is used to fine-tune the counting model on an image pair dataset consisting of multiple local image pairs. During the fine-tuning process, a loss function is constructed based on the independent structural similarity loss function and the mean squared error of the local region. The local region is a candidate frame of a preset size that is pre-determined for the target category object, and the area of the candidate frame is smaller than the area of each image in the image pair dataset.

[0011] The present invention also provides a remote sensing cross-domain object counting device based on style transfer, comprising: An alignment module is used to determine the salient regions of each image in the target domain dataset and the source domain dataset based on the recognition results of the semantic visual perception model, and align the size of the target category objects in the source domain dataset with the size of the target category objects in the target domain dataset based on the average size of the target category objects in the salient regions; A screening module is used to divide each image in the target domain dataset and the size-aligned source domain dataset into multiple local images, and screen each local image divided in the target domain dataset from the local images divided in the size-aligned source domain dataset to obtain the image with the highest semantic and distribution consistency to form a local image pair; The recognition module is used to fine-tune the counting model on an image pair dataset consisting of multiple groups of local image pairs to make the model more stylistically adapted to the target domain, obtain the counting model of the target domain, and determine the number of target category objects in the image to be recognized through the technical model of the target domain.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the remote sensing cross-domain object counting method based on style transfer as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the remote sensing cross-domain object counting method based on style transfer as described above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described remote sensing cross-domain object counting methods based on style transfer.

[0015] The remote sensing cross-domain object counting method and device based on style transfer provided by the present invention realizes text-guided scale alignment through a semantic visual perception model, effectively eliminating the scale differences between the source domain and the target domain, and making the subsequent alignment and training process more stable and controllable, thereby significantly improving the applicability and accuracy of the cross-domain counting task; through semantic and distribution screening to construct an image pair dataset, the semantic and distribution differences between the source domain and the target domain are further eliminated. On this basis, through style alignment, the style features of the target domain are fused with the content features of the source domain, which not only can smoothly eliminate the differences between the source domain and the target domain in style aspects such as appearance and background, but also retains the content information that is crucial to the object counting task, thereby achieving more stable and robust cross-domain adaptation while ensuring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is one of the flow charts of the remote sensing cross-domain object counting method based on style transfer provided by the present invention; Figure 2 It is a schematic diagram of a process for determining the average size of target category objects in the remote sensing cross-domain object counting method based on style transfer provided by the present invention; Figure 3 This is a flow chart of obtaining a second label in the remote sensing cross-domain object counting method based on style transfer provided by the present invention; Figure 4 This is the second flow chart of the remote sensing cross-domain object counting method based on style transfer provided by the present invention; Figure 5 Schematic diagram of the structure of the remote sensing cross-domain object counting device based on style transfer provided by the present invention; Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0019] First, let’s introduce the following contents: To mitigate the impact of distribution discrepancies, some work has attempted to retrain on labeled data in the target domain. However, this faces high labeling costs and a cumbersome retraining process in the remote sensing field. Furthermore, remote sensing scenarios are often subject to limitations such as privacy protection and widespread geographical distribution, making it often impractical to obtain labeled data in the target domain. Therefore, unsupervised domain adaptation has become a more practical research direction. Given labeled data in the source domain and only unlabeled data in the target domain, the adaptability of the model can be improved by aligning or self-training the cross-domain feature distributions.

[0020] Early domain adaptation methods focused on transferring the style of the entire domain, often using adversarial learning to align the feature distributions of different domains. However, such methods often neglected the detailed modeling of the distribution of local regions in the image, making it difficult to preserve fine-grained features.

[0021] To address this issue, some research has attempted to mine more detailed feature distribution information between the source and target domains based on distribution alignment and combined with self-training strategies. However, in remote sensing counting tasks, large-scale variations exist across datasets, and self-training often leads to noise accumulation. These methods still have limitations in addressing these challenges.

[0022] To further improve cross-domain adaptation performance, current research is leveraging the generalization capabilities of large, pre-trained models. These models, trained on large amounts of data, possess strong zero-shot reasoning capabilities and can provide relatively stable visual semantic features or text-guided similarity maps for the target domain. While these basic models can provide some cross-domain guidance, they are not specifically designed for counting tasks in remote sensing imagery and often only provide relatively coarse-grained auxiliary information, failing to directly address the more nuanced domain differences and noise interference inherent in large-scale remote sensing counting. Therefore, leveraging the universal feature representation capabilities of large models while balancing the scale diversity of remote sensing imagery with self-trained noise suppression remains an urgent challenge.

[0023] To this end, the following Figure 1 This paper introduces a remote sensing cross-domain object counting method based on style transfer. Figure 1 Shown, including: Step 101: determining the salient regions of each image in the target domain dataset and the source domain dataset based on the recognition results of the semantic visual perception model, and aligning the size of the target category objects in the source domain dataset with the size of the target category objects in the target domain dataset based on the average size of the target category objects in the salient regions; The source domain dataset contains multiple labeled images from the source domain, while the target domain dataset can contain multiple unlabeled images from the target domain, or a large number of unlabeled images from the target domain and a small number of labeled images. The target category is the category of the object that the object counting model is used to identify.

[0024] Therefore, in order to make the cross-domain performance of the object counting model trained based on the source domain data more accurately applied to the counting and recognition of remote sensing images in the target domain, it is necessary to reconstruct the dataset and fine-tune the object counting model.

[0025] For the dataset used for model fine-tuning, one of the requirements is to reduce the impact of the object scale in the source domain dataset and the target domain dataset on the model performance.

[0026] It is understandable that if the scale of the target category objects in the source domain dataset is different from that in the target domain dataset, it will have a negative impact on the recognition accuracy of the object counting model. To this end, it is necessary to scale the image size in the source domain dataset as close as possible to the image size in the target domain dataset, so that the target category objects are at a similar scale as much as possible in the images of the two datasets, thereby achieving image size alignment.

[0027] On this basis, we first need to determine the size of the target category objects in the source domain dataset and the target domain dataset.

[0028] In this implementation, to obtain the accurate size of the target class objects in each image of the two datasets, a semantic visual perception model is first used to extract the salient regions of each image. Salient regions are areas of interest or importance in an image. In this implementation, they are understood to be areas where the target class objects are distributed.

[0029] Optionally, the semantic visual perception model may be a CLIP model, specifically a CLIP-Surgery model loaded with CLIP-RS parameter weights, which can extract salient areas that are most likely to contain objects, thereby avoiding false detection in complex backgrounds.

[0030] The semantic vision model generates text prompts based on the category label corresponding to the category of the target category object, extracts a similarity map from each image of the source domain dataset and the target domain dataset, and uses the Otsu method to perform binarization processing to generate a binary mask corresponding to the text prompt. After screening the binary mask, the circumscribed rectangular box of the largest connected area is extracted as the determined salient area.

[0031] Determining the average size of target category objects based on the salient regions extracted from each image can effectively improve the accuracy of the determined object size. In this embodiment, after determining the average size of target category objects in each image, the mean of the target category object sizes for all images in the source and target datasets is calculated, respectively, as the average object size of the source and target datasets.

[0032] A scale factor is calculated based on the average object size of the source domain dataset and the average object size of the target domain dataset. Based on the scale factor, the size of the target category objects in the source domain dataset is aligned with the size of the target category objects in the target domain dataset, thereby roughly aligning the significant scale differences between images.

[0033] Step 102: Divide each image in the target domain dataset and the size-aligned source domain dataset into multiple local images, and screen each local image divided in the target domain dataset from the local images divided in the size-aligned source domain dataset to obtain the image with the highest semantic and distribution consistency to form a local image pair; On the other hand, after achieving image scale alignment of the source domain dataset and eliminating scale differences, similarities are found between local images. These similarities can be used to discover more transferable cross-domain information and promote the transfer of knowledge from known domains to unknown domains.

[0034] In order to effectively utilize local similarity, each image in the target domain dataset and the size-aligned source domain dataset is further divided in the same scale space using a sliding window to obtain multiple source domain local images and multiple target domain local images.

[0035] Optionally, the sliding window is preferably a 128×128 square window, and the step size of the sliding window is half of the sliding window size.

[0036] In this embodiment, local images of different domains are aligned by exploring the similarity between local images.

[0037] Optionally, the semantic features and distribution of each local image are extracted, and for each local image divided by the target domain dataset, the image with the highest semantic and distribution consistency is screened out from the local images divided by the source domain dataset after scale alignment, and together with the local image, a local image pair is formed.

[0038] A local image pair consists of a target domain local image and a source domain local image with the highest semantic and distributional consistency. It is understood that in the local image pairs obtained in this way, the source domain local image has scale, semantic, and distributional consistency with the target domain local image.

[0039] Step 103: fine-tune the counting model on the image pair dataset consisting of multiple groups of local image pairs to make the model better adapt to the target domain in terms of style, obtain the counting model of the target domain, and determine the number of target category objects in the image to be identified through the technical model of the target domain.

[0040] By using the above method, multiple groups of local image pairs can be screened to form an image pair dataset, which is used as a dataset for fine-tuning the object calculation model.

[0041] Specifically, we believe that local images in the target domain can provide the model with accurate style information, such as weather, lighting, and texture details; while local images in the source domain are believed to have more accurate ground truth labels but differ stylistically from the target domain. Based on these complementary properties, we fine-tune the object counting model trained in the source domain by combining reliable content information from the source domain with accurate style information from the target domain, making it more stylistically adaptable to the target domain.

[0042] The entire fine-tuning process is implemented on an image pair dataset, so that the object counting model is fine-tuned through the dataset aligned in scale, semantics, and distribution during the fine-tuning process to obtain a counting model for the target domain, achieving better performance in the application of the target domain and effectively improving the accuracy of the model's cross-domain counting.

[0043] On this basis, the target domain image that needs to be counted is input into the trained target domain counting model, and the number of target category objects output by the target domain counting model can be obtained.

[0044] The present invention achieves text-guided scale alignment through a semantic visual perception model, effectively eliminating the scale differences between the source and target domains, and making the subsequent alignment and training processes more stable and controllable, thereby significantly improving the applicability and accuracy of cross-domain counting tasks; by constructing an image pair dataset through semantic and distribution screening, the semantic and distribution differences between the source and target domains are further eliminated. On this basis, through style alignment, the style features of the target domain are fused with the content features of the source domain, which not only smoothly eliminates the differences between the source and target domains in style aspects such as appearance and background, but also retains content information that is crucial to the target counting task, thereby achieving more stable and robust cross-domain adaptation while ensuring accuracy.

[0045] In the remote sensing cross-domain object counting method based on style transfer of the present invention, before the step of aligning the size of the target category objects in the source domain dataset with the size of the target category objects in the target domain dataset based on the average size of the target category objects in the salient region, the method further includes: In the salient areas of each image, the open set object perception model is used to identify the initial candidate boxes; In order to improve the accuracy of the determined average size of the target category objects and to better achieve scale alignment, in this embodiment, an open set object perception model is further used to identify the salient areas of each image.

[0046] Optionally, the open set object perception model may be an OWLv2 model (an open source zero-shot text-conditional object detection model).

[0047] In layman's terms, the open-set object perception model is also a target recognition model. Compared with the closed-set recognition model, the open-set object perception model has better recognition performance. However, directly using the open-set object perception model to recognize remote sensing images in the source and target domain datasets will introduce excessive noise, resulting in reduced recognition performance.

[0048] Therefore, in this embodiment, the open set object perception model is selected to identify the target category objects in the pre-extracted salient areas, thereby effectively suppressing noise and improving the recognition accuracy of the open set object perception model.

[0049] Specifically, the open-set object perception model combines the category output of the target category object with the text prompt generated by the category label, selects potential target category objects in the salient area, and obtains the initial candidate box corresponding to the object.

[0050] At least use non-maximum suppression to screen the initial candidate boxes to obtain confident candidate boxes; Since the initial perception of the open-set object perception model is inaccurate, overlapping initial candidate frames will be generated. Therefore, it is necessary to screen the initial candidate frames and select the ones with higher confidence.

[0051] Optionally, non-maximum suppression is applied to remove low-confidence initial candidate boxes.

[0052] Optionally, initial candidate boxes with a containment relationship are filtered. Specifically, if an initial candidate box has a significantly larger area or contains more than a certain number of small initial candidate boxes, and its intersection-over-union ratio is greater than a certain threshold, it is removed.

[0053] The average size of the target category objects in the confidence candidate box of each image is used as the average size of the target category objects in the salient area.

[0054] It is understandable that several confident candidate boxes can be screened for each image in the target domain dataset and the source domain dataset.

[0055] Therefore, the average area value of several confidence candidate boxes in each image is calculated to represent the average size of the identified target category objects, and it is used as the average size of the target category objects in the salient area and as the basis for calculating the scale factor to achieve scale alignment from the source domain to the target domain.

[0056] In a specific embodiment, taking the target class object as a vehicle as an example, a complete average size calculation process is as follows: Figure 2 As shown in FIG, an image containing a vehicle is first input into the semantic visual perception model to obtain its output similarity map. After the similarity map is binarized, the bounding rectangle of the largest connected area after screening is extracted as the salient area.

[0057] The extracted salient regions are further input into the open set object perception model, and the initial candidate boxes outputted are subjected to non-maximum suppression and containment filtering in turn to obtain the confident candidate boxes. Based on the confident candidate boxes, the average size can be determined. f scale .

[0058] In the remote sensing cross-domain object counting method based on style transfer of the present invention, the step of screening the local images divided for each target domain dataset from the local images divided for the source domain dataset after size alignment to obtain the image with the highest semantic and distribution consistency to form a local image pair specifically includes: Based on the semantic features of each local image, the local images divided by each target domain dataset are first screened among the local images divided by the source domain dataset after size alignment, and the preset number of images closest to them on the manifold are obtained as their corresponding source domain candidate local image group; In this embodiment, the step of determining the local image pair is implemented in two screening steps, wherein the first screening is a preliminary screening, which is used to filter out a group of images that are similar to the target domain local image on the manifold based on semantics from the source domain local image as candidates; the second screening is to further find the image with the highest semantic and distribution consistency with the target domain local image from the group of candidate images, so as to construct the local image group.

[0059] Optionally, a label-free self-distillation model is used to extract semantic features for each local image. Preferably, the DINOv2 model is used, which learns high-quality visual representations rich in semantic information and demonstrates excellent generalization across a wide range of open scenes. It also encodes the global structure of local images, enabling the model to distinguish objects from complex backgrounds.

[0060] Optionally, a diffusion retrieval model is used to extract the closest source domain candidate local image group on the manifold for the local images divided into the target domain dataset.

[0061] The construction method of the diffusion retrieval model is as follows: Step 1: Graph construction and normalization: Consider the source features as a set. For each feature, find its k nearest neighbors. If two features are nearest neighbors, set the edge weight to feature cosine similarity. Otherwise, set the edge weight to 0. This will yield the adjacency matrix A of the graph. The diagonal elements of the diagonal matrix D are the node degrees. Normalize the matrix A to obtain the normalized adjacency matrix S.

[0062] Step 2: Random walk and search: Perform random walk on the constructed and normalized graph, assuming the initial state is , then the update formula for each iteration can be written as ,in represents the probability of continuing random walk from the current state, It represents the probability of returning to the initial state. When the iteration converges, the analytical solution can be obtained ,in That is, the final score on all features, and the ranking is based on this score.

[0063] Among them, the source feature is the semantic feature extracted from the local image of the source domain. The semantic feature extracted from the local image of the target domain is used as the query input diffusion retrieval model to obtain the ranking of the semantic features extracted for each local image of the source domain. Finally, the local images of the source domain corresponding to the semantic features with the top preset number of rankings are taken to construct the source domain candidate local image group corresponding to the local image of the target domain.

[0064] The candidate local image group in the source domain constructed by coarse-grained retrieval is semantically closest to the local image in the target domain, and is therefore suitable for subsequent fine-grained distribution retrieval as a second screening.

[0065] For each local image divided by the target domain dataset, a second screening is performed in its source domain candidate local image group to obtain the image with the highest semantic and distribution consistency, which together constitutes a local image pair.

[0066] Each local image divided by the target domain dataset is the target domain local image. For each target domain local image, a second screening is performed in its corresponding source domain candidate local image group to obtain the image with the highest semantic and distribution consistency.

[0067] Specifically, it is believed that the images in the source domain candidate local image group already have a high semantic consistency with the target domain local image. Therefore, the second screening is mainly used to screen the distribution consistency. The source domain local image with the highest distribution consistency is taken as the image with the highest semantic and distribution consistency with the target domain local image, and together they constitute a pair of local image pairs.

[0068] The present invention performs an initial screening through robust semantic features, and then performs a secondary screening based on distribution consistency, and finally obtains local image pairs consisting of local images in the source domain that have the highest semantic and distribution consistency with the local images in the target domain. This achieves more accurate style transfer based on the local image pairs with consistent semantics and distribution, and obtains an object counting model with better cross-domain counting performance.

[0069] In the remote sensing cross-domain object counting method based on style transfer of the present invention, the step of performing a second screening of each local image divided by the target domain dataset in its source domain candidate local image group to obtain the image with the highest semantic and distribution consistency specifically includes: Determine a first label for each image in the source domain candidate local image group; Since the source domain dataset is a labeled dataset, when extracting the first label of the target category object, the source domain dataset can be directly used to train a baseline model for extracting the distribution of the target category objects in the source domain.

[0070] For each local image divided by the target domain dataset, each source domain local image in its corresponding source domain candidate local image group is input into the trained baseline model in turn to obtain the first label output by the baseline model.

[0071] Obtaining a second label of a target category object in each local image divided by the target domain dataset; It is understandable that since the data in the target domain dataset is an unlabeled dataset, it is difficult to directly train a model that can extract the distribution of target category objects in the local image of the target domain through the unlabeled target domain dataset.

[0072] On this basis, if the target domain dataset contains a small amount of labeled data, the baseline model trained on the source domain dataset can be directly fine-tuned based on transfer learning, and the second label of the target category object can be obtained through the fine-tuned baseline model.

[0073] If the target domain dataset only contains unlabeled data, the baseline model trained with the source domain dataset can also be used to obtain the second label of the local image of the target domain, where the second label represents the pseudo label of the local image of the target domain.

[0074] Calculating a structural similarity index between each first label and the second label of the source domain candidate local image group; The first label corresponding to the structural similarity index having the largest calculated value is determined and used as the image with the highest semantic and distribution consistency.

[0075] For each target domain local image, the structural similarity index between its second label and each first label of the corresponding source domain candidate local image group is calculated respectively, and the source domain local image corresponding to the structural similarity index with the largest calculated value is regarded as the image with the highest semantic and distribution consistency with the target domain local image, and a pair of local image groups is constructed.

[0076] In the remote sensing cross-domain object counting method based on style transfer of the present invention, before the step of obtaining the first label of the target category object in each local image divided by the target domain dataset, the method further includes: A first baseline model is trained on a source domain dataset to obtain a first label; In this embodiment, in order to obtain a more accurate second label, a first baseline model is first trained on the source domain dataset. On the one hand, the first baseline model can be directly used to obtain the first label. On the other hand, the first baseline model is also used to preliminarily perceive the distribution of target category objects in the image in the target domain dataset.

[0077] Inputting the unlabeled images in the target domain dataset into the first baseline model to obtain a density map output by the baseline model as an initial pseudo label for the unlabeled images in the target domain dataset; The first baseline model is fine-tuned using the initial pseudo-label as supervision information to obtain a second baseline model, where the second baseline model is used to obtain the second label.

[0078] Specifically, the trained first baseline model is used for the unlabeled data of the target domain dataset to output the corresponding density maps. Although these density maps are noisy, they are closer to the target domain object distribution and are used as initial pseudo-labels.

[0079] The obtained initial pseudo-labels are regarded as supervisory signals and used to train a new target domain model, namely the second baseline model, in this way to enhance the adaptability of the model in the target domain.

[0080] The trained second baseline model is used to obtain a more refined second label, making the second label closer to the actual target domain distribution. The complete process is as follows Figure 3 shown.

[0081] This paper generates pseudo-labels by combining a sliding window strategy with a self-training mechanism. However, direct reliance on pseudo-labels may be subject to noise interference. To this end, a first screening process extracts more discriminative semantic features, filtering out pseudo-labels that are inconsistent with the source domain semantics or are clearly unreliable. A second screening process prioritizes high-quality pseudo-labels during alignment, significantly reducing the impact of noise interference on target domain alignment and model training, and enhancing control over pseudo-label quality.

[0082] In the remote sensing cross-domain object counting method based on style transfer of the present invention, the step of fine-tuning the counting model on the image pair dataset consisting of multiple groups of local image pairs specifically includes: Adaptive instance normalization is used to fine-tune the counting model on an image pair dataset consisting of multiple local image pairs. During the fine-tuning process, a loss function is constructed based on the independent structural similarity loss function and the mean squared error of the local region. The local region is a candidate frame of a preset size that is pre-determined for the target category object, and the area of the candidate frame is smaller than the area of each image in the image pair dataset.

[0083] Adaptive Instance Normalization (AdaIN) is used to fine-tune the counting model to transfer the style features of the local image in the target domain to the content features of the local image in the source domain. Finally, the density map is obtained through the density prediction head for counting.

[0084] AdaIN can transform images into different styles while keeping the content unchanged. Its formula is: ; Where x is the content feature, y is the style feature, and represents the mean and standard deviation in spatial position respectively. and represent the mean and standard deviation in spatial location respectively.

[0085] In this task, style features are extracted from local images in the target domain, and content features are extracted from local images in the source domain. Then, adaptive instance normalization (AdaIN) is used to fuse the style features and content features to achieve the migration of the target domain style to the source domain content, thereby generating local features that are both consistent with the target domain in style and rich in accurate content information.

[0086] During fine-tuning, the independent structural similarity loss functions I-SSIM and MSE are used as loss functions. At the same time, to further improve the recognition accuracy of the fine-tuned model, in this implementation, a fixed preset size candidate box is pre-selected in the image for each target category object. When calculating the loss between the predicted image and the real image, the loss is calculated between the candidate box area of the predicted image and the candidate box area of the real image, rather than the loss of the entire image: ; Where, is the total loss, and are the predicted result and the actual result of the nth target respectively. The complete flow chart is shown in Figure 4.

[0087] After eliminating scale and semantic differences, this paper further utilizes a style alignment module based on adaptive instance normalization (AdaIN) to fuse the style features of the target domain with the content features of the source domain. This not only smoothly eliminates differences in style between the source and target domains in terms of appearance, background, and other aspects, but also preserves the content information that is crucial for object counting tasks, thereby achieving more stable and robust cross-domain adaptation while maintaining accuracy.

[0088] The remote sensing cross-domain object counting device based on style transfer provided by the present invention is described below. The remote sensing cross-domain object counting device based on style transfer described below and the remote sensing cross-domain object counting method based on style transfer described above can refer to each other.

[0089] like Figure 5 As shown, the remote sensing cross-domain object counting device based on style transfer provided by the present invention includes an alignment module 501, a screening module 502 and a recognition module 503; an alignment module 501 for determining a salient region of each image in the target domain dataset and the source domain dataset based on the recognition results of the semantic visual perception model, and aligning the size of the target category objects in the source domain dataset with the size of the target category objects in the target domain dataset based on the average size of the target category objects in the salient region; The source domain dataset contains multiple labeled images from the source domain, while the target domain dataset can contain multiple unlabeled images from the target domain, or a large number of unlabeled images from the target domain and a small number of labeled images. The target category is the category of the object that the object counting model is used to identify.

[0090] Therefore, in order to make the cross-domain performance of the object counting model trained based on the source domain data more accurately applied to the counting and recognition of remote sensing images in the target domain, it is necessary to reconstruct the dataset and fine-tune the object counting model.

[0091] For the dataset used for model fine-tuning, one of the requirements is to reduce the impact of the object scale in the source domain dataset and the target domain dataset on the model performance.

[0092] It is understandable that if the scale of the target category objects in the source domain dataset is different from that in the target domain dataset, it will have a negative impact on the recognition accuracy of the object counting model. To this end, it is necessary to scale the image size in the source domain dataset as close as possible to the image size in the target domain dataset, so that the target category objects are at a similar scale as much as possible in the images of the two datasets, thereby achieving image size alignment.

[0093] On this basis, we first need to determine the size of the target category objects in the source domain dataset and the target domain dataset.

[0094] In this implementation, to obtain the accurate size of the target class objects in each image of the two datasets, a semantic visual perception model is first used to extract the salient regions of each image. Salient regions are areas of interest or importance in an image. In this implementation, they are understood to be areas where the target class objects are distributed.

[0095] Optionally, the semantic visual perception model may be a CLIP model, specifically a CLIP-Surgery model loaded with CLIP-RS parameter weights, which can extract salient areas that are most likely to contain objects, thereby avoiding false detection in complex backgrounds.

[0096] The semantic vision model generates text prompts based on the category label corresponding to the category of the target category object, extracts a similarity map from each image of the source domain dataset and the target domain dataset, and uses the Otsu method to perform binarization processing to generate a binary mask corresponding to the text prompt. After screening the binary mask, the circumscribed rectangular box of the largest connected area is extracted as the determined salient area.

[0097] Determining the average size of target category objects based on the salient regions extracted from each image can effectively improve the accuracy of the determined object size. In this embodiment, after determining the average size of target category objects in each image, the mean of the target category object sizes for all images in the source and target datasets is calculated, respectively, as the average object size of the source and target datasets.

[0098] A scale factor is calculated based on the average object size of the source domain dataset and the average object size of the target domain dataset. Based on the scale factor, the size of the target category objects in the source domain dataset is aligned with the size of the target category objects in the target domain dataset, thereby roughly aligning the significant scale differences between images.

[0099] A screening module 502 is configured to divide each image in the target domain dataset and the size-aligned source domain dataset into multiple partial images, and screen each partial image divided in the target domain dataset from the partial images divided in the size-aligned source domain dataset to obtain the image with the highest semantic and distribution consistency to form a partial image pair; On the other hand, after achieving image scale alignment of the source domain dataset and eliminating scale differences, similarities are found between local images. These similarities can be used to discover more transferable cross-domain information and promote the transfer of knowledge from known domains to unknown domains.

[0100] In order to effectively utilize local similarity, each image in the target domain dataset and the size-aligned source domain dataset is further divided into multiple source domain local images and multiple target domain local images using a sliding window in the same scale space.

[0101] Optionally, the sliding window is preferably a 128×128 square window, and the step size of the sliding window is half of the sliding window size.

[0102] In this embodiment, local images of different domains are aligned by exploring the similarity between local images.

[0103] Optionally, the semantic features and distribution of each local image are extracted, and for each local image divided by the target domain dataset, the image with the highest semantic and distribution consistency is screened out from the local images divided by the source domain dataset after scale alignment, and together with the local image, a local image pair is formed.

[0104] A local image pair consists of a target domain local image and a source domain local image with the highest semantic and distributional consistency. It is understood that in the local image pairs obtained in this way, the source domain local image has scale, semantic, and distributional consistency with the target domain local image.

[0105] The recognition module 503 is used to fine-tune the counting model on the image pair dataset composed of multiple groups of local image pairs so that the model can better adapt to the target domain in terms of style, obtain the counting model of the target domain, and determine the number of target category objects in the image to be identified through the technical model of the target domain.

[0106] By using the above method, multiple groups of local image pairs can be screened to form an image pair dataset, which is used as a dataset for fine-tuning the object calculation model.

[0107] Specifically, we believe that local images in the target domain can provide the model with accurate style information, such as weather, lighting, and texture details; while local images in the source domain are believed to have more accurate ground truth labels but differ stylistically from the target domain. Based on these complementary properties, we fine-tune the object counting model trained in the source domain by combining reliable content information from the source domain with accurate style information from the target domain, making it more stylistically adaptable to the target domain.

[0108] The entire fine-tuning process is implemented on the image pair dataset, so that the object counting model is fine-tuned on the subsequent dataset through scale, semantics and distribution during the fine-tuning process to obtain the counting model of the target domain, achieving better performance in the application of the target domain and effectively improving the accuracy of the model's cross-domain counting.

[0109] On this basis, the target domain image that needs to be counted is input into the trained target domain counting model, and the number of target category objects output by the target domain counting model can be obtained.

[0110] The present invention achieves text-guided scale alignment through a semantic visual perception model, effectively eliminating the scale differences between the source and target domains, and making the subsequent alignment and training processes more stable and controllable, thereby significantly improving the applicability and accuracy of cross-domain counting tasks; by constructing an image pair dataset through semantic and distribution screening, the semantic and distribution differences between the source and target domains are further eliminated. On this basis, through style alignment, the style features of the target domain are fused with the content features of the source domain, which not only smoothly eliminates the differences between the source and target domains in style aspects such as appearance and background, but also retains content information that is crucial to the target counting task, thereby achieving more stable and robust cross-domain adaptation while ensuring accuracy.

[0111] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6As shown, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute a remote sensing cross-domain object counting method based on style transfer, which includes: determining the salient area of each image in the target domain dataset and the source domain dataset based on the recognition results of the semantic visual perception model, and aligning the size of the target category objects in the source domain dataset with the size of the target category objects in the target domain dataset based on the average size of the target category objects in the salient area; dividing each image in the target domain dataset and the source domain dataset after size alignment into multiple local images, and screening each local image divided by the target domain dataset from the local images divided by the source domain dataset after size alignment to obtain images with the highest semantic and distribution consistency to form local image pairs; fine-tuning the counting model on the image pair dataset composed of multiple groups of local image pairs to make the model better adapt to the target domain in terms of style, obtaining a counting model of the target domain, and determining the number of target category objects in the image to be identified through the technical model of the target domain.

[0112] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0113] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the remote sensing cross-domain object counting method based on style transfer provided by the above methods. The method includes: determining the salient area of each image in the target domain dataset and the source domain dataset based on the recognition results of the semantic visual perception model, and aligning the size of the target category objects in the source domain dataset towards the size of the target category objects in the target domain dataset based on the average size of the target category objects in the salient area; dividing each image in the target domain dataset and the size-aligned source domain dataset into multiple local images, and screening each local image divided by the target domain dataset from the local images divided by the size-aligned source domain dataset to obtain the image with the highest semantic and distribution consistency to form a local image pair; fine-tuning the counting model on the image pair dataset composed of multiple groups of local image pairs to make the model better adapt to the target domain in terms of style, obtaining a counting model of the target domain, and determining the number of target category objects in the image to be identified through the technical model of the target domain.

[0114] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the remote sensing cross-domain object counting method based on style transfer provided by the above methods, the method comprising: determining the salient area of each image in the target domain dataset and the source domain dataset based on the recognition results of the semantic visual perception model, and aligning the size of the target category objects in the source domain dataset toward the size of the target category objects in the target domain dataset based on the average size of the target category objects in the salient area; dividing each image in the target domain dataset and the size-aligned source domain dataset into multiple local images, and screening each local image divided in the target domain dataset from the local images divided in the size-aligned source domain dataset to obtain a local image pair with the image having the highest semantic and distribution consistency; fine-tuning the counting model on the image pair dataset composed of multiple groups of local image pairs to make the model better adapt to the target domain in terms of style, obtaining a counting model of the target domain, and determining the number of target category objects in the image to be identified through the technical model of the target domain.

[0115] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0116] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A remote sensing cross-domain object counting method based on style transfer, characterized by: include: Based on the recognition results of the semantic visual perception model, the salient regions of each image in the target domain dataset and the source domain dataset are determined. Based on the average size of the target category objects in the salient regions, the size of the target category objects in the source domain dataset is aligned with the size of the target category objects in the target domain dataset. Each image in the target domain dataset and the size-aligned source domain dataset is divided into multiple local images, and each local image divided in the target domain dataset is screened from the local images divided in the size-aligned source domain dataset to obtain the image with the highest semantic and distribution consistency to form a local image pair; The counting model is fine-tuned on an image pair dataset consisting of multiple groups of local image pairs to make the model better adapt to the target domain in terms of style, thereby obtaining a counting model of the target domain, and determining the number of target category objects in the image to be identified through the technical model of the target domain.

2. The remote sensing cross-domain object counting method based on style transfer according to claim 1 is characterized in that: Before the step of aligning the size of the target category objects in the source domain dataset with the size of the target category objects in the target domain dataset based on the average size of the target category objects in the salient region, the method further includes: In the salient areas of each image, the open set object perception model is used to identify the initial candidate boxes; At least use non-maximum suppression to screen the initial candidate boxes to obtain confident candidate boxes; The average size of the target category objects in the confidence candidate box of each image is used as the average size of the target category objects in the salient area.

3. The remote sensing cross-domain object counting method based on style transfer according to claim 1 is characterized in that: The step of screening the local images divided for each target domain dataset from the local images divided for the source domain dataset after size alignment to obtain the image with the highest semantic and distribution consistency to form a local image pair specifically includes: Based on the semantic features of each local image, the local images divided by each target domain dataset are first screened among the local images divided by the source domain dataset after size alignment, and the preset number of images closest to them on the manifold are obtained as their corresponding source domain candidate local image group; For each local image divided by the target domain dataset, a second screening is performed in its source domain candidate local image group to obtain the image with the highest semantic and distribution consistency, which together constitutes a local image pair.

4. The remote sensing cross-domain object counting method based on style transfer according to claim 3 is characterized in that: The step of performing a second screening of each local image divided from the target domain dataset in its source domain candidate local image group to obtain an image with the highest semantic and distribution consistency specifically includes: Obtaining a first label of a target category object in each image of the source domain candidate local image group; Obtaining a second label of a target category object in each local image divided by the target domain dataset; Calculating a structural similarity index between each first label and the second label of the source domain candidate local image group; The first label corresponding to the structural similarity index having the largest calculated value is determined and used as the image with the highest semantic and distribution consistency.

5. The remote sensing cross-domain object counting method based on style transfer according to claim 4 is characterized in that: Before the step of obtaining the first label of the target category object in each local image divided by the target domain dataset, the method further includes: A first baseline model is trained on a source domain dataset to obtain a first label; Inputting the unlabeled images in the target domain dataset into the first baseline model to obtain a density map output by the baseline model as an initial pseudo label for the unlabeled images in the target domain dataset; The first baseline model is fine-tuned using the initial pseudo-label as supervision information to obtain a second baseline model, where the second baseline model is used to obtain the second label.

6. The remote sensing cross-domain object counting method based on style transfer according to any one of claims 1 to 5, characterized in that: The step of fine-tuning the counting model on the image pair dataset consisting of multiple groups of local image pairs specifically includes: Adaptive instance normalization is used to fine-tune the counting model on an image pair dataset consisting of multiple local image pairs. During the fine-tuning process, a loss function is constructed based on the independent structural similarity loss function and the mean squared error of the local region. The local region is a candidate frame of a preset size that is pre-determined for the target category object, and the area of the candidate frame is smaller than the area of each image in the image pair dataset.

7. A remote sensing cross-domain object counting device based on style transfer, characterized in that: include: An alignment module is used to determine the salient regions of each image in the target domain dataset and the source domain dataset based on the recognition results of the semantic visual perception model, and align the size of the target category objects in the source domain dataset with the size of the target category objects in the target domain dataset based on the average size of the target category objects in the salient regions; A screening module is used to divide each image in the target domain dataset and the size-aligned source domain dataset into multiple local images, and screen each local image divided in the target domain dataset from the local images divided in the size-aligned source domain dataset to obtain the image with the highest semantic and distribution consistency to form a local image pair; The recognition module is used to fine-tune the counting model on an image pair dataset consisting of multiple groups of local image pairs to make the model more stylistically adapted to the target domain, obtain the counting model of the target domain, and determine the number of target category objects in the image to be recognized through the technical model of the target domain.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the remote sensing cross-domain object counting method based on style transfer is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the remote sensing cross-domain object counting method based on style transfer is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the remote sensing cross-domain object counting method based on style transfer is implemented.

Images