Shadow detection method and device based on attention mechanism and federated learning

Abstract

The present application relates to the technical field of computer vision, more particularly, to a shadow detection method and device based on attention mechanism and federated learning. The present application adopts attention feature fusion, generates an attention map according to the information in the deep feature maps of the encoding end and the decoding end, can judge whether the information in the encoding end skip connection feature map is conducive to the goal of shadow detection, suppresses irrelevant features, enhances effective features, avoids important information loss, generates more accurate decoding end features, and further strengthens the recognition ability of the artifacts in the image. The present application performs federated learning after each round of training, indirectly introduces the corresponding feature information extracted by the models of other data sets through the method of averaging the model parameters of different data sets in the same round of training, effectively improves the test effect of the model on other data sets, enhances the generalization ability of the model, and also improves the test effect of the model itself to a certain extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly, to a shadow detection method and device based on attention mechanism and federated learning. BACKGROUND

[0002] A problem often occurring in shadow detection is the artifact problem, that is, a dark non-shadow region is wrongly judged as a shadow region, or a light shadow region is wrongly judged as a non-shadow region. This problem is caused by information loss due to convolution between layers on one hand, and the fact that the decoding end cannot effectively use the rich context features extracted by the encoding end on the other hand, resulting in misjudgment of the generated image.

[0003] Existing traditional shadow detection methods are mainly based on the invariance of illumination prior theory to establish related physical illumination models, and some use clustering algorithms to divide the pixels in the image into different small regions, and classify the image pixels into shadow regions or non-shadow regions according to color, edge, texture and other features. However, in the traditional method, the region-based method often needs to specially design artificial features, which are usually not discriminative, and simple features are not suitable for complex images; and the light conditions of the scene are usually complex and changeable, so the physical model based on the illumination prior will become unstable, thereby affecting the results of shadow detection.

[0004] Existing deep shadow detection methods are data-driven methods with strong feature extraction capability, and perform much better than traditional shadow detection methods. Some common deep methods such as deep convolutional networks, UNet networks, and generative adversarial networks can be used to extract rich deep and shallow features for robust shadow detection because they contain a large number of learnable parameters in the model. The training target can be specified by a self-defined loss function to make the network output results continuously close to the true shadow label, and the deep method achieves good shadow detection effect on a single image. However, in the UNet type encoder-decoder network, there is no good decoding end to effectively use the rich feature information extracted by the encoding end, which often leads to loss of important information and affects the final shadow detection effect. In addition, due to the strong learning ability of neural networks, the network is often only applicable to the image of the trained data set, and has poor generalization on other images with large differences. SUMMARY

[0005] Therefore, it is necessary to provide a shadow detection method and device based on attention mechanism and federated learning to solve the technical problems that the decoding end of the existing UNet type encoder-decoder network cannot effectively use the rich feature information extracted by the encoding end, resulting in loss of important information and affecting the shadow detection effect, and the use of neural networks in shadow detection leads to poor model generalization.

[0006] The application adopts the following technical solutions:

[0007] In a first aspect, the application discloses a shadow detection method based on an attention mechanism and federated learning, which is applied to a UNet type codec network.

[0008] The shadow detection method comprises the following steps:

[0009] Step one, determine a corresponding number of network models A, data loaders B, and network optimizers C according to the number N of current shadow detection tasks;

[0010] Wherein, the architectures of the N network models A are the same; the nth network model A n , the nth data loader B n , and the nth network optimizer C n correspond one by one to form a group, which only serves the corresponding nth shadow detection task dataset; the nth shadow detection task dataset includes a training set and a test set; n = 1, 2, 3…, N;

[0011] Step two, add the training set of the nth shadow detection task to the nth network model A n using the nth data loader B n , and simultaneously train N groups of network models A for multiple rounds;

[0012] Wherein, each round of training includes:

[0013] Extracting features of different scales at the encoding end, extracting features containing rich context information at the encoding end, and fusing the two features to obtain features extracted at the encoding end;

[0014] Processing the features extracted at the encoding end and the features output by the deep decoding end through an attention feature fusion network to obtain the features output by the shallow decoding end;

[0015] Calculate the total loss of the current round q of the network model, and optimize the network model parameters using the corresponding network optimizer to obtain the network model obtained in the current round q; The total loss includes the final output loss and the side output loss;

[0016] Wherein, q = 1, 2, 3…, Q, and Q is the total number of training rounds;

[0017] Step three, after training each round, perform federated learning on the network model obtained in the current round q, average the network model parameters, and obtain a network model with better generalization performance;

[0018] The network model with better generalization performance obtained in the current round q The network model as the next round q+1 training;

[0019] After Q rounds of training, a shadow detection task obtains Q-1 network models with better generalization performance The network model obtained in the current round q

[0020] Step four, for any shadow detection task, use the corresponding Q network models obtained in the current round Test the test set of the shadow detection task for performance testing, and save the best network model according to the test index for subsequent shadow detection.

[0021] The shadow detection method based on attention mechanism and federated learning realizes the method or process according to the embodiments of the present disclosure.

[0022] In a second aspect, the present disclosure discloses a shadow detection device based on attention mechanism and federated learning, which uses the shadow detection method based on attention mechanism and federated learning disclosed in the first aspect.

[0023] The shadow detection device based on attention mechanism and federated learning comprises a feature extraction module, an attention feature fusion module, a loss parameter adjustment module, a federated learning module, and a model testing module.

[0024] The feature extraction module is arranged at the encoding end of the UNet type encoder-decoder network. The feature extraction module is used to extract features of different scales, extract features containing rich context information, and fuse the two features to obtain the features extracted at the encoding end. The attention feature fusion module is arranged at the decoding end of the UNet type encoder-decoder network. The attention mechanism effective feature fusion module is used to process the features extracted at the encoding end and the features output by the deep decoding end to obtain the features output by the shallow decoding end. The loss parameter adjustment module is used to calculate the total loss of the network model in the current round q, and use the corresponding network optimizer to optimize the network model parameters to obtain the network model obtained in the current round q The federated learning module is used to perform federated learning on the network model obtained in the current round q after each round of training, average the network model parameters, and obtain a network model with better generalization performance. The model testing module is used for any shadow detection task, using the corresponding Q network models obtained in the current round Test the test set of the shadow detection task for performance testing, and save the best network model according to the test index for subsequent shadow detection.

[0025] The cold chain storage and transportation environment temperature and humidity control device based on information fusion compensation implements the method or process according to the embodiments of this disclosure.

[0026] Thirdly, the present invention discloses a readable storage medium storing computer program instructions, which are read and executed by a processor to perform the shadow detection method based on attention mechanism and federated learning disclosed in the first aspect.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. This invention employs attention feature fusion, generating an attention map based on information from the deep feature maps of the encoder and decoder. This allows for the assessment of whether the information in the skip connection feature map of the encoder is beneficial for shadow detection, suppressing irrelevant features, enhancing effective features, avoiding the loss of important information, generating more accurate decoder features, thereby strengthening the ability to identify artifacts in images (including but not limited to light shadows and dark non-shadow areas), and significantly improving the overall shadow detection capability of the model.

[0029] 2. This invention performs federated learning after each round of training. By averaging the model parameters of different datasets in the same round of training, it indirectly introduces the corresponding feature information extracted by models of other datasets, effectively improving the model's test performance on other datasets, enhancing the model's generalization ability, and also improving the model's own test performance to a certain extent. Attached Figure Description

[0030] Figure 1 This is a flowchart of the shadow detection method based on attention mechanism and federated learning of the present invention;

[0031] Figure 2 for Figure 1 A schematic diagram illustrating the principle of each round of training;

[0032] Figure 3 for Figure 2 The principle diagram for fusing feature one and feature two to obtain feature three;

[0033] Figure 4 for Figure 2 The principle diagram of fusing feature three and deep feature four to obtain shallow feature four;

[0034] Figure 5 for Figure 1 Flowchart of Federated Learning;

[0035] Figure 6 This is a comparison chart showing the results of shadow detection using existing shadow detection methods and the shadow detection method of the present invention. Detailed Implementation

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0037] It should be noted that when a component is referred to as being "mounted on" another component, it can be directly on the other component or there can be a middle component. When a component is referred to as being "disposed on" another component, it can be directly disposed on the other component or there can be a middle component. When a component is referred to as being "fixed on" another component, it can be directly fixed on the other component or there can be a middle component.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing specific embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0039] Embodiment 1

[0040] Please refer to Figure 1 , Figure 1 The flowchart of the shadow detection method based on attention mechanism and federated learning of the present application.

[0041] As Figure 1 shown, the shadow detection method based on attention mechanism and federated learning is applied in a UNet type codec network. Specifically, the shadow detection method comprises the following steps:

[0042] Step one, according to the number N of the current shadow detection task, determine the corresponding number of network model A, data loader B, network optimizer C;

[0043] Wherein, the architecture of the N network models A is the same; the nth network model A n , the nth data loader B n , the nth network optimizer C n one by one correspond to form a group, only serve the corresponding nth shadow detection task data set; the nth shadow detection task data set includes training set and test set; n = 1, 2, 3…, N.

[0044] Step one is preparation, which mainly determines the number of network models needed according to the number N of current shadow detection tasks. The data loader and network optimizer are used with the network model.

[0045] This way can ensure that the data sets of all shadow detection tasks can participate in training at the same time without interfering with each other, and facilitate the subsequent federated learning to average the model parameters.

[0046] Step two, use the nth data loader B n to join the nth network model A n , and the N groups of network models A perform multiple rounds of training at the same time.

[0047] Referring to Figure 2 , Figure 2 is a schematic diagram of each round of training. Each round of training includes:

[0048] 1. Extracting feature one of different scales at the encoding end, extracting feature two containing rich context information at the encoding end, and obtaining feature three extracted at the encoding end by fusing the two features.

[0049] In step two, ResNext101 network is used to extract feature one, and multi-scale convolution network is used to extract feature two. Of course, VGG16, VGG19, ResNet, etc. pre-training network can also be used to replace feature one, and the training speed may be faster, but these networks are not as ingenious as ResNext101 in design, and the extracted features are not as rich as ResNext101, so the effect will be slightly reduced.

[0050] As shown in Figure 2 , the ResNext101 network includes M-1 layers, and the multi-scale convolution network includes M layers, M>1. As an example in Figure 2 , M=5, the ResNext101 network is divided into the 1st layer, the 2nd layer, the 3rd layer, the 4th layer, and the 5th layer from top to bottom (i.e. from shallow to deep). The multi-scale convolution network is divided into the 1st layer, the 2nd layer, the 3rd layer, the 4th layer, and the 5th layer from top to bottom (i.e. from shallow to deep).

[0051] Specifically, the mth feature one output by the mth layer of the ResNext101 network is fused with the mth feature two output by the mth layer of the multi-scale convolution network to obtain the mth feature three extracted at the encoding end.

[0052] The mth feature three is input into the m+1th layer of the multi-scale convolution network to obtain the m+1th feature two, and the m+1th feature one is fused to obtain the m+1th feature three, which is a down-sampling process; m=1, 2, 3…, M.

[0053] More specifically, refer to Figure 3, ResNext101 network extracts feature maps of different scales as feature one; the multi-scale convolutional network uses different convolution kernels to extract information of different scales, and after splicing the information of different scales and performing convolution, down-sampling and other operations, multi-scale fusion information is obtained, which is feature two containing rich context information; feature one and feature two are spliced and fused to obtain feature three extracted by the encoding end.

[0054] The above process can also be represented by a formula:

[0055] Cout = Conv(Cat(Pool(Conv(r, Cat(c1, c2, …, ck, …, cn-1, cn)))))). k ));

[0056] Wherein, Cout represents feature three extracted by the encoding end, r represents feature one extracted by ResNext101 network, c1, c2, c k respectively represent feature two obtained by using the first size, the second size, the kth size of the convolution kernel of the multi-scale convolution, Cat() represents the channel dimension splicing of the feature map, Pool() represents the pooling operation, and Conv() represents the convolution operation.

[0057] The number and size of the convolution kernel are selected according to the needs. After the inventor conducts experiments, it is found that the number of the convolution kernel is 3, and the size distribution is 1*1 pixel point, 3*3 pixel rectangle and 5*5 pixel rectangle, which is the best. That is, k = 3, the first size is 1*1 pixel point, the second size is 3*3 pixel rectangle, and the third size is 5*5 pixel rectangle.

[0058] 2, the feature three extracted by the encoding end and the feature four output by the deep decoding end are processed through the attention feature fusion network to obtain the feature four output by the shallow decoding end;

[0059] The attention feature fusion network also includes M layers, and the data sources include two parts: one part is the feature three extracted by the encoding end transmitted by the encoding end jump connection, and the other part is the information obtained by the deep decoding end (that is, the feature four).

[0060] Also refer to Figure 2 The attention feature fusion network is divided into the 5th layer, the 4th layer, the 3rd layer, the 2nd layer and the 1st layer from bottom to top (that is, from deep to shallow).

[0061] Specifically, the Mth feature three convolutional transformation obtains the Mth feature four output by the Mth layer of the attention feature fusion network; the ith feature three and the ith+1 feature four output by the ith+1 layer of the multi-scale convolutional network (i.e., transmitted through the skip connection) are fused through the ith layer of the attention feature fusion network to obtain the ith decoding end output feature four, which is an upsampling process; i=1, 2, 3…, M-1.

[0062] More specifically, referring to Figure 4 The size and the number of channels of the ith+1 feature four are converted to the same size as the ith feature three. This is because the feature four is smaller in size and has more channels than the feature three, so the size and the number of channels of the feature four are changed to the same size as the encoding end through the upsampling process.

[0063] The converted ith+1 feature four is spliced with the ith feature three to obtain the ith spliced feature map; and the ith spliced feature map is subjected to convolution transformation and Sigmoid activation to obtain the ith attention map. The attention map is composed of floating-point numbers between 0 and 1, 0 indicating that the feature information at the current position is inhibited, and 1 indicating that the feature information at the current position is enhanced.

[0064] The ith attention map is point multiplied with the ith feature three to obtain the ith point multiplied feature map. In this way, the effective features are enhanced and the irrelevant features are inhibited, and the obtained feature map is more effective.

[0065] The ith point multiplied feature map is spliced with the converted ith+1 feature four, and convolution transformation is performed to obtain the ith decoding end output feature four. In this way, the previously obtained decoding end shadow information can be further retained, and important information loss can be avoided.

[0066] The above process can also be represented by the following formulas:

[0067] Map=Sigmoid(Conv(Cat(S,D′)));

[0068] Dout=Conv(Cat(Map*S,D));

[0069] The former is the formula for obtaining the attention map, and the latter is the formula for obtaining the decoding end output feature four. Among them, Map is the attention map, Sigmoid() is the Sigmoid activation function, S represents the feature transmitted by the encoding end skip connection, D' represents the feature after the decoding end deep feature is upsampled, and D represents the decoding end deep feature.

[0070] 3, calculate the total loss of the current round q of the network model, and use the corresponding network optimizer to optimize the network model parameters to obtain the network model obtained in the current round q The total loss includes the final output loss and the side output loss.

[0071] wherein q = 1, 2, 3, …, Q, and Q is the total training round.

[0072] In this embodiment, the formula for calculating the total loss is designed as:

[0073]

[0074] wherein Total is the total loss, BCEloss is the final output loss, loss h is the side output loss of the hth layer of the attention feature fusion network, h = 2, 3, …, M.

[0075] BCEloss = ylog(p(x)) + (1-y)log(1-p(x));

[0076] The above BCEloss is designed to constrain the generation result of the network to be closer to the label image, and then the corresponding optimizer of the network is used to optimize the parameters. The label image refers to the structural feature map manually labeled in the shadow detection task, which is the correct result and contains the correct label.

[0077] In order to ensure that the decoding feature obtained by the method is correct, a side supervision loss loss h is added to constrain each layer of decoding feature. Specifically, two progressive convolution layers are added to each layer of decoding feature to change the number of decoding feature channels to a single channel, and then an intermediate result image is obtained through a Sigmoid activation function, and the label image is used together with the total loss to constrain the correct feature.

[0078] In short, the attention mechanism feature fusion is to enhance the effective information and suppress the irrelevant information, and the cross-entropy loss with weight is used to constrain the side output result of each layer of the decoding end, which can better respond to images with less shadow. It should be noted that the deeper the feature map, the smaller the size, so the corresponding weight is also lower.

[0079] wherein loss h is represented as: loss h = λylog(p(x h )) + (1-λ)(1-y)log(1-p(x h ));

[0080] y represents the label of the correct result of shadow detection, and x represents the final output result of shadow detection; x his the side output prediction result of the h-th layer of the attention feature fusion network; p() represents the probability of the prediction being correct in the corresponding prediction result; λ is the weight coefficient of the shaded part, and 1-λ is the weight coefficient of the unshaded part.

[0081] In this embodiment, after conducting experiments, the inventors found that M=5 and λ=0.8 yielded the best results.

[0082] Step 3: After each training epoch, analyze the network model obtained in the current epoch (q). Federated learning is used to average the network model parameters, resulting in a network model with better generalization performance.

[0083] The network model with better generalization performance obtained in the current round q As the network model for the next round of training q+1;

[0084] After Q rounds of training, a shadow detection task yields Q-1 network models with better generalization performance. Q network models acquired in the current round

[0085] For details, please refer to Figure 5 ,for Figure 1 Flowchart of federated learning. The network model acquired in the current iteration q. Methods for conducting federated learning include:

[0086] For any model A among the N network models A in the current round q n It obtains the corresponding network model acquired in the current round at the end of the current round of training q.

[0087] Reuse The parameters of A are averaged with the parameters of the other N-1 network models at this time to obtain A. n The corresponding network model with better generalization performance This will serve as the network model for the next round of training (q+1).

[0088] More specifically, the mean is calculated using the formula: m1' = (m1 + m2 + ... + mN) / N. Where m1 is the mean value obtained by A1 after completing the q-th training iteration. The parameters, m2, ..., mN, are other N-1 models obtained in A1. The parameter corresponding to the time, m1' is The parameters.

[0089] This is because the shadow detection tasks are different, and the datasets are also different. Therefore, although each training round is conducted simultaneously, the completion time of each round for the N models is different. It should be noted that if A2 completes the q-th training round, its completion time is later than A1. In this case, when averaging m2', A1 corresponds to m1', because A1 has already completed federated learning and its parameters are no longer updated. This prevents multiple models from having completely identical parameters after completing a single training round.

[0090] Step 4: For any shadow detection task, use the Q network models acquired in the current rounds. The performance of the shadow detection task was tested on the test set, and the best network model was saved based on the test metrics for subsequent shadow detection.

[0091] Specifically, the test metric is BER:

[0092]

[0093] Where TP represents the number of correctly identified shadow pixels, TN represents the number of correctly identified non-shadow pixels, and N represents the number of correctly identified non-shadow pixels. p N represents the total number of shadow pixels. n This represents the number of all non-shaded pixels.

[0094] Select The model with the smallest BER is selected as the best model. In this way, N shadow detection tasks will yield N corresponding best models, and the generalization ability of these N best models will be improved.

[0095] Example 2

[0096] This embodiment 2 provides an example verification of the shadow detection method based on attention mechanism and federated learning disclosed in embodiment 1.

[0097] See Figure 6 This is a comparison chart showing the results of shadow detection using existing shadow detection methods and the shadow detection method of the present invention.

[0098] Figure 6This section presents the qualitative comparison results of the experiment. The first column shows the input RGB shadow image of the network model (the original image is in color). The second column shows the manually annotated shadow mask image based on the input shadow image, where white areas represent shadowed regions and black areas represent unshaded regions. The third column shows the shadow region prediction results obtained from the input image using our method (i.e., Ours). The fourth column (ECA stands for Robust Shadow Detection by Exploring Effective Shadow Contexts), the fifth column (MTMT stands for A Multi-Task Mean Teacher for Semi-Supervised Shadow Detection), and the sixth column (DSD stands for Distraction-Aware Shadow Detection) show the prediction results from other methods.

[0099] Depend on Figure 6 As can be seen, our method far surpasses other methods in prediction accuracy for most images, especially in the first and fourth rows of images, where our method is superior to other models in correctly identifying artifact regions; other images also demonstrate that our method is better at detecting some details.

[0100] Additionally, please refer to Table 1 for the quantitative experimental results:

[0101] Table 1 Quantitative Experiment Results

[0102]

[0103] The table above shows the results of a standard qualitative experiment, which includes four datasets: SBU, ISTD, CUHK, and UCF. The SBU, ISTD, and CUHK datasets each have their own training and test sets, while the UCF dataset only has a test set. Its metrics are obtained by testing the best model trained on the SBU dataset.

[0104] Specifically, 1. SBU (Large-scale training of shadow detectors with noisily-annotated shadow examples, containing 4085 training images and 638 test images); 2. ISTD (Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal, containing 1330 training images and 540 test images); 3. UCF (Learning to recognize shadows in monochromatic natural images, containing 110 test images); 4. CUHK (Revisiting Shadow Detection: A New Benchmark Dataset for Complex World, containing 7350 training images and 2100 test images).

[0105] It can be seen that the best model obtained by this method outperforms the current best method ECA in both testing its own test set and testing the UCF dataset using SBU, proving that the accuracy of the best model obtained by this method is currently the best.

[0106] See Table 2 for cross-validation BER metrics:

[0107]

[0108] The table above shows the quantitative results of cross-validation. Cross-validation involves training the model on its own dataset and then testing it on other test sets that are different from its own dataset to verify the generalization performance of the current model.

[0109] The first three columns, "Train with SBU," indicate that the model was trained using the SBU dataset. The BER (Performance Ratio) results were then obtained by testing the SBU, ISTD, and UCF datasets respectively. The first row shows the results without federated learning, the second row shows the results of this method, the third row shows the test results of the ECA method, and the fourth row shows the test results of the STCGAN (Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal) method. The same applies to the three columns on the right for ISTD, and will not be elaborated further.

[0110] As can be seen from the data, this method already achieves good results even without federated learning. Adding federated learning significantly improves the test results, proving that federated learning can indeed enhance the model's generalization performance.

[0111] Example 3

[0112] This embodiment 3 provides a shadow detection device based on attention mechanism and federated learning, which uses the shadow detection method based on attention mechanism and federated learning disclosed in embodiment 1.

[0113] The shadow detection device based on attention mechanism and federated learning includes a feature extraction module, an attention feature fusion module, a loss parameter adjustment module, a federated learning module, and a model testing module.

[0114] A feature extraction module is set at the encoding end of the UNet-type codec network. This module extracts feature 1 at different scales, extracts feature 2 containing rich contextual information, and fuses the two features to obtain feature 3 extracted at the encoding end. An attention feature fusion module is set at the decoding end of the UNet-type codec network. This attention mechanism-effective feature fusion module processes feature 3 extracted at the encoding end with feature 4 output from the deep decoding end to obtain feature 4 output from the shallow decoding end. A loss parameter adjustment module calculates the total loss of the network model in the current round q and optimizes the network model parameters using the corresponding network optimizer to obtain the network model acquired in the current round q. The federated learning module is used to process the network model obtained in the current epoch q after each training epoch. Federated learning is performed to average the network model parameters, resulting in a network model with better generalization performance. The model testing module is used to test any shadow detection task using the network model obtained in the Q current rounds. The performance of the shadow detection task was tested on the test set, and the best network model was saved based on the test metrics for subsequent shadow detection.

[0115] This embodiment 3 also discloses a readable storage medium storing computer program instructions. When the computer program instructions are read and run by a processor, the shadow detection method based on attention mechanism and federated learning in embodiment 1 is executed.

[0116] When applying the method of Example 1, it can be applied in the form of software, such as by designing it as a program that can run independently on a computer-readable storage medium. The computer-readable storage medium can be a USB flash drive, designed as a USB security token, and the program can be designed to start the entire method through an external trigger.

[0117] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A shadow detection method based on attention mechanism and federated learning, applied in UNet-type encoder-decoder networks, characterized in that... The shadow detection method includes the following steps: Step 1: Determine the corresponding number of network models A, data loaders B, and network optimizers C based on the number N of current shadow detection tasks; Among them, N network models A have the same architecture; the nth network model A n The nth data loader B n The nth network optimizer C n Each set is formed in a one-to-one correspondence and serves only the corresponding nth shadow detection task; the dataset for the nth shadow detection task includes a training set and a test set; n = 1, 2, 3, ..., N; Step 2: Use the nth data loader B to load the training set for the nth shadow detection task. n Add to the nth network model A n In this process, N groups of network models A undergo multiple rounds of training simultaneously; Each round of training includes: Extract feature 1 at different scales from the encoding end, extract feature 2 containing rich contextual information from the encoding end, and fuse the two features to obtain feature 3 extracted from the encoding end; The feature three extracted from the encoding end and the feature four output from the deep decoding end are processed through an attention feature fusion network to obtain the feature four output from the shallow decoding end. Calculate the total loss of the network model in the current round q, and use the corresponding network optimizer to optimize the network model parameters to obtain the network model acquired in the current round q. The total loss includes the final output loss and the side output loss; Where q = 1, 2, 3, ..., Q, and Q is the total number of training rounds; Step 3: After each training epoch, analyze the network model obtained in the current epoch (q). Federated learning is used to average the network model parameters, resulting in a network model with better generalization performance. The network model with better generalization performance obtained in the current round q As the network model for the next round of training q+1; After Q rounds of training, a shadow detection task yields Q-1 network models with better generalization performance. Q network models acquired in the current round Step 4: For any shadow detection task, use the Q network models acquired in the current rounds. The performance of the shadow detection task was tested on the test set, and the best network model was saved based on the test metrics for subsequent shadow detection.

2. The shadow detection method based on attention mechanism and federated learning according to claim 1, characterized in that, In step two, the ResNext101 network is used to extract feature one, and a multi-scale convolutional network is used to extract feature two. The ResNext101 network consists of M-1 layers, the multi-scale convolutional network consists of M layers, and the attention feature fusion network consists of M layers; M>1. The m-th feature 1 output from the m-th layer of the ResNext101 network is fused with the m-th feature 2 output from the m-th layer of the multi-scale convolutional network to obtain the m-th feature 3 extracted from the encoding end; the m-th feature 3 is passed through the (m+1)-th layer of the multi-scale convolutional network to obtain the (m+1)-th feature 2, and then fused with the (m+1)-th feature 1 to obtain the (m+1)-th feature 3; m = 1, 2, 3, ..., M; The Mth feature is transformed by a convolutional transformation to obtain the Mth feature 4 output by the Mth layer of the attention feature fusion network; the i-th feature 3 and the i+1th feature 4 output by the (i+1)th layer of the multi-scale convolutional network are fused through the i-th layer of the attention feature fusion network to obtain the i-th decoder output feature 4; i = 1, 2, 3, ..., M-1; The first decoding end output feature four is the final output of the decoding end; the final output of the decoding end is the binarized shadow detection result.

3. The shadow detection method based on attention mechanism and federated learning according to claim 2, characterized in that, In step two, the formula for fusing feature one and feature two to obtain feature three is designed as follows: Cout=Conv(Cat(Pool(Conv(r,Cat(c1,c2,…,c k ))))); Where Cout represents feature three extracted from the encoder, r represents feature one extracted by the ResNext101 network, and c1, c2, c3, c4, c5, c6, c7, c8, c9 ... k The first, second, and k-th size convolution kernels represent the second feature maps obtained by multi-scale convolution, respectively. Cat() represents concatenating the channel dimensions of the feature maps, Pool() represents pooling, and Conv() represents convolution.

4. The shadow detection method based on attention mechanism and federated learning according to claim 3, characterized in that, k=3, the first type is a pixel with a size of 1*1, the second type is a pixel rectangle with a size of 3*3, and the third type is a pixel rectangle with a size of 5*5.

5. The shadow detection method based on attention mechanism and federated learning according to claim 2, characterized in that, In step two, the method for fusing feature three and the deep feature four to obtain the shallow feature four includes: Convert the size and number of channels of the (i+1)th feature four to be the same as the size of the ith feature three; The transformed (i+1)th feature 4 is concatenated with the ith feature 3 to obtain the ith concatenated feature map; The i-th stitched feature map is transformed by convolution and activated by Sigmoid to obtain the i-th attention map. Perform a dot product operation between the i-th attention map and the i-th feature 3 to obtain the i-th dot product feature map; The i-th dot-multiplied feature map is concatenated with the transformed (i+1)-th feature map, and then subjected to convolution to obtain the i-th decoder output feature map.

6. The shadow detection method based on attention mechanism and federated learning according to claim 3, characterized in that, In step two, the formula for calculating the total loss is designed as follows: Where Total represents the total loss, BCEloss represents the final output loss, and loss h The side output loss of the h-th layer of the attention feature fusion network is given by h = 2, 3, ..., M. BCEloss=ylog(p(x))+(1-y)log(1-p(x)); loss h =λylog(p(x h ))+(1-λ)(1-y)log(1-p(x h )); Where y represents the label of a correct shadow detection result, and x represents the final output result of shadow detection; x h is the side output prediction result of the h-th layer of the attention feature fusion network; p() represents the probability of the prediction being correct in the corresponding prediction result; λ is the weight coefficient of the shaded part, and 1-λ is the weight coefficient of the unshaded part.

7. The shadow detection method based on attention mechanism and federated learning according to claim 5, characterized in that, M = 5, λ = 0.

8.

8. The shadow detection method based on attention mechanism and federated learning according to claim 1, characterized in that, In step three, the network model obtained in the current round q Methods for conducting federated learning include: For any model A among the N network models A in the current round q n It obtains the corresponding network model acquired in the current round at the end of the current round of training q. Reuse The parameters of A are averaged with the parameters of the other N-1 network models at this time to obtain A. n The corresponding network model with better generalization performance This will serve as the network model for the next round of training (q+1).

9. The shadow detection method based on attention mechanism and federated learning according to claim 1, characterized in that, In step four, the test metric is BER: Where TP represents the number of correctly identified shadow pixels, TN represents the number of correctly identified non-shadow pixels, and N represents the number of correctly identified non-shadow pixels. p N represents the total number of shadow pixels. n The number of all non-shaded pixels; Select The model with the smallest BER is selected as the best model.

10. A shadow detection device based on attention mechanism and federated learning, characterized in that, The shadow detection method based on attention mechanism and federated learning as described in any one of claims 1-8 was used; The shadow detection device based on attention mechanism and federated learning includes: The feature extraction module is set at the encoding end of the UNet-type codec network; the feature extraction module is used to extract feature one at different scales, extract feature two containing rich contextual information, and fuse the two features to obtain feature three extracted at the encoding end; An attention feature fusion module is set at the decoding end of the UNet-type codec network; the attention mechanism effective feature fusion module is used to process the feature three extracted from the encoding end and the feature four output from the deep decoding end to obtain the feature four output from the shallow decoding end; The loss parameter adjustment module calculates the total loss of the network model in the current round q and optimizes the network model parameters using the corresponding network optimizer to obtain the network model acquired in the current round q. The federated learning module is used to process the network model obtained in the current epoch q after each training epoch. Federated learning is performed to average the network model parameters, resulting in a network model with better generalization performance; and The model testing module is used to test any shadow detection task using the Q network models acquired in the current rounds. The performance of the shadow detection task was tested on the test set, and the best network model was saved based on the test metrics for subsequent shadow detection.

Images