Construction scene inter-domain difference-oriented adaptation method and system during continuous test

Abstract

The invention relates to the technical field of computer vision, in particular to an adaptation method and system for continuous testing for differences between construction scene domains. According to the method, the similarity between Gram matrixes between adjacent domains is calculated, an elastic adjustment factor is set, the elastic adjustment factor is utilized, different weights are given to strong data enhancement and weak data enhancement, an elastic data enhancement strategy is provided, and an enhanced target domain image data set is input into a teacher model; updating the pseudo-tag by combining the elasticity regulation factor to obtain an elastic pseudo-tag; and inputting the target domain image data set into the student model to obtain a prediction result, constructing a global elastic symmetric cross entropy loss function based on the elastic adjustment factor, the cross entropy loss and the reverse cross entropy loss, updating student model parameters through the loss function, updating teacher model parameters, and finally obtaining a target model. According to the method, the construction scene monitoring model can adapt to complex domain changes when continuously learning test data, and the prediction result precision of the model in different environments is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for continuous testing and adaptation of inter-domain differences in construction scenarios. Background Art

[0002] In recent years, deep learning has demonstrated remarkable performance in tasks such as computer vision, natural language processing, and speech recognition. This success relies heavily on the distribution consistency of test data with training data. However, in test scenarios, the environment often changes, leading to shifts in the domain distribution. For example, with constant changes in weather and lighting, deployed models lack generalization capabilities, leading to performance degradation. For example, in water conservancy construction safety monitoring, the natural environment presents numerous uncertainties, such as variable weather, fluctuating temperatures, and varying light levels. Furthermore, the dynamic nature of construction activities, including the transitions between different construction phases, the frequent movement and operation of construction machinery, and the storage and transportation of construction materials, results in a constantly changing site environment. Finally, there are potential geological and hydrological risks, such as changes in slope stability, fluctuations in groundwater levels, and the rise and fall of river levels. Due to these constant environmental changes, existing construction monitoring models deployed in water conservancy construction sites struggle to adapt to the new environment, lack generalization capabilities, and experience performance degradation, making timely and accurate monitoring and early warning impossible.

[0003] In recent years, continuous test-time adaptation (CTTA) has received widespread attention. Its goal is to update the source pre-trained model to adapt to the new data domain by continuously learning from unlabeled test data. For privacy protection and practical considerations, during CTTA, the source domain data is no longer accessed, and each target domain data can only be accessed once. This means that the model loses the opportunity to learn from multiple target domain data, correct biases, and optimize parameters. The model needs to extract the characteristics and patterns of the current target domain data as comprehensively as possible in the only access, which places extremely high demands on the model's rapid learning and adaptability. Therefore, entropy minimization methods have been proposed. For example, TENT (Test-time Entropy Minimization) minimizes the test entropy and updates the trainable parameters in the batch normalization (Batchnorm) layer, allowing the model to better adapt to the test data. GTTA uses techniques such as blending and style transfer to artificially construct an intermediate domain, fusing target domain data of different types or characteristics accessed in the same session. This allows the model to learn the common features between different data. GTTA enables smooth transitions between different domains while minimizing entropy, allowing the model to extract data features as comprehensively as possible during its single access to the target domain data. However, entropy minimization methods rely heavily on the statistical properties of the data, assuming that the data distribution is relatively stable and statistically definable. Once the data distribution becomes complex and changes frequently, such as in real-time dynamic scenarios, the statistical properties of the data may change at any time. Entropy minimization methods then struggle to quickly and accurately capture data features, significantly reducing the model's adaptability. Therefore, algorithms based on the average teacher model have been introduced to the field of continuous test-time adaptation. For example, CoTTA constructs an average teacher model framework. During a single visit to the target domain data, it performs multiple random data augmentation transformations on the original test samples, enabling the model to adapt to data characteristics under different lighting conditions and color styles. By leveraging the knowledge learned from these data augmentation transformations, CoTTA enables the model to better adapt to interference factors in various real-world scenarios. CoTTA also uses weighted and augmented averaging strategies to assign weights to different prediction results, reducing the impact of errors caused by secondary or inaccurate predictions. It also combines random recovery with a certain probability to randomly restore some of the previous state parameters of the student model, fusing new and old knowledge and enabling the model to understand the target domain data from a more comprehensive perspective in a single visit. Building on CoTTA, PETAL further developed a data-driven parameter recovery technique that dynamically restores the model's parameters based on the characteristics and changes of the data, better aligning them with the original source configuration in the parameter space. This allows the model to retain its memory of the original data characteristics while adapting to new data, significantly enhancing the model's adaptive robustness.

[0004] However, traditional continuous testing adaptation methods usually directly use random data augmentation strategies to achieve model self-training, ignoring the existence of certain differences between domain changes. As a result, when the model continuously learns unlabeled test data, it is unable to accurately capture the unique characteristics of data in different scenarios. As a result, the adaptability and generalization ability of the construction scene monitoring model in different scenarios are severely limited, the accuracy of the prediction results is reduced, and timely and accurate monitoring and early warning are unable to be carried out. Summary of the Invention

[0005] To this end, the technical problem to be solved by the present invention is that the existing model ignores the differences between domain changes when continuously learning unlabeled test data, resulting in the construction scene monitoring model having low prediction accuracy in different environments, making it difficult to accurately detect and effectively warn of the construction scene.

[0006] To solve the above technical problems, the present invention provides a method for continuous testing and adaptation oriented to inter-domain differences in construction scenarios, comprising the following steps:

[0007] Obtain a construction scene image test set, where the construction scene image test set includes a target domain image dataset corresponding to T time steps in the construction area to be monitored;

[0008] The target domain image dataset corresponding to the t-th time step is used as the input of the feature extraction network to obtain the domain feature map of each target domain image at the t-th time step; where t = 1, 2…T;

[0009] Flatten the domain feature map of each target domain image at the t-th time step, and construct the Gram matrix of the t-th time step based on the flattened domain feature maps of all target domain images at the t-th time step;

[0010] Based on the similarity between the Gram matrices of the t-th time step and the t-1-th time step, the elastic adjustment factor of the t-th time step is set;

[0011] Based on the elastic adjustment factor of the t-th time step, strong data enhancement and weak data enhancement are performed on the target domain image dataset corresponding to the t-th time step respectively;

[0012] Input the target domain image dataset corresponding to the t-th time step into the student model and obtain the prediction result of the t-th time step;

[0013] The target domain image dataset corresponding to the t-th time step after strong data augmentation and weak data augmentation is input into the teacher model respectively, and the pseudo label output by the teacher model is updated according to the elastic adjustment factor of the t-th time step to obtain the elastic pseudo label of the t-th time step;

[0014] Based on the elastic adjustment factor at the t-th time step, the cross entropy loss and the reverse cross entropy loss between the elastic pseudo-label and the prediction result at the t-th time step, a global elastic symmetric cross entropy loss function at the t-th time step is constructed;

[0015] The parameters of the student model at the t-th time step are updated through the global elastic symmetric cross entropy loss function at the t-th time step. Based on the updated parameters of the student model at the t-th time step, the parameters of the teacher model at the t-th time step are updated. After completing the training of T time steps, the teacher model at the T-th time step with updated parameters is used as the target model.

[0016] Preferably, the prediction result of the t-th time step is passed through the Softmax activation function to obtain a prediction confidence list of the t-th time step;

[0017] In the prediction confidence list of the t-th time step, a category is randomly selected. If the difference between the average prediction confidence value of the category at the t-th time step and the average prediction confidence value of the category at the t-1-th time step is greater than the first set threshold, the elastic adjustment factor of the t-th time step is set by calculating the similarity between the Gram matrices of the t-th time step and the t-1-th time step;

[0018] If the difference between the average prediction confidence value of the category at the t-th time step and the average prediction confidence value of the category at the t-1-th time step is less than or equal to the first set threshold, the elastic adjustment factor of the t-th time step is directly set to 1.

[0019] Preferably, if the similarity between the Gram matrices of the t-th time step and the t-1-th time step is greater than or equal to the second set threshold, the elasticity adjustment factor of the t-th time step is set to 0.4;

[0020] If the similarity between the Gram matrices of the t-th time step and the t-1-th time step is less than the second set threshold, the elasticity adjustment factor of the t-th time step is set to 0.8.

[0021] Preferably, the elastic adjustment factor based on the t-th time step performs strong data enhancement and weak data enhancement on the target domain image dataset corresponding to the t-th time step, respectively, and the formula is:

[0022]

[0023] in, is the jth target domain image in the target domain image dataset corresponding to the tth time step after strong data enhancement, is the jth target domain image in the target domain image dataset corresponding to the tth time step after weak data enhancement, is the data enhancement operation, ω tis the elastic adjustment factor at the t-th time step, is the random factor of the i-th strong perturbation at the t-th time step, is the random factor of the i-th weak perturbation at the t-th time step, x j,t is the jth target domain image in the target domain image dataset corresponding to the tth time step, n is the number of strong / weak perturbations, and j is the index of the target domain image in the target domain image dataset corresponding to the tth time step.

[0024] Preferably, the elastic pseudo-label of the t-th time step is passed through the Softmax activation function to obtain a confidence list of the elastic pseudo-label of the t-th time step;

[0025] For each target domain image in the target domain image dataset corresponding to the t-th time step, if the elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the weak data enhancement is greater than the confidence threshold, then the elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the target domain image after weak data enhancement is used to correct the elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the strong data enhancement, and the elastic pseudo-label of the target domain image dataset after strong data enhancement at the t-th time step is obtained.

[0026] Preferably, the elastic pseudo label confidence value of the current category in the elastic pseudo label corresponding to the current target domain image after weak data enhancement is used to correct the elastic pseudo label confidence value of the current category in the elastic pseudo label corresponding to the current target domain image after strong data enhancement. The formula is:

[0027]

[0028] in, is the elastic pseudo-label confidence value of category c in the elastic pseudo-label corresponding to the current target domain image after strong data enhancement, is the elastic pseudo-label confidence value of the j-th target domain image in the target domain image dataset corresponding to the t-th time step in category c after weak data enhancement, is the elastic pseudo-label confidence value of the j-th target domain image in the target domain image dataset corresponding to the t-th time step in category c after weak data enhancement, σ[j,c] is the adaptive correction weight of the j-th target domain image in the target domain image dataset corresponding to the t-th time step in category c, and KL(.) is the KL divergence.

[0029] Preferably, the target domain image dataset corresponding to the t-th time step after strong data augmentation and weak data augmentation is input into the teacher model, and the pseudo label output by the teacher model is updated according to the elastic adjustment factor of the t-th time step to obtain the elastic pseudo label of the t-th time step. The formula is:

[0030]

[0031] in, is the jth target domain image x in the target domain image dataset corresponding to the tth time step j,t Elastic pseudo-label, M is the total number of target domain images in the target domain image dataset, ω t is the elastic adjustment factor at the t-th time step, is the pseudo label of the jth target domain image in the target domain image dataset corresponding to the tth time step after strong data enhancement, is the pseudo label of the j-th target domain image in the target domain image dataset corresponding to the t-th time step after weak data enhancement, and j is the index of the target domain image in the target domain image dataset corresponding to the t-th time step.

[0032] Preferably, the elastic adjustment factor at the t-th time step, the cross entropy loss between the elastic pseudo label at the t-th time step and the prediction result, and the reverse cross entropy loss are used to construct a global elastic symmetric cross entropy loss function at the t-th time step, and the formula is:

[0033]

[0034] in, is the global elastic symmetric cross entropy loss function at the t-th time step, C t is the number of categories of the target domain image dataset corresponding to the t-th time step, ω t is the elastic adjustment factor of the t-th time step, c is the category index, is the jth target domain image x in the target domain image dataset corresponding to the tth time step j,t Elastic pseudo-label value on category c, p c (x j,t ) is the jth target domain image x in the target domain image dataset corresponding to the tth time step j,t The predicted value for class c.

[0035] Preferably, the parameters of the teacher model at the t-th time step are updated by exponential moving average based on the updated parameters of the student model at the t-th time step.

[0036] The present invention also provides a continuous testing and adaptation system for inter-domain differences in construction scenarios, including:

[0037] A data acquisition model is used to acquire a construction scene image test set, wherein the construction scene image test set includes a target domain image dataset corresponding to T time steps in the construction area to be monitored;

[0038] The feature extraction module is used to take the target domain image dataset corresponding to the t-th time step as the input of the feature extraction network to obtain the domain feature map of each target domain image at the t-th time step;

[0039] t=1,2…T;

[0040] The Gram matrix construction module is used to flatten the domain feature map of each target domain image at the t-th time step, and construct the Gram matrix of the t-th time step based on the flattened domain feature maps of all target domain images at the t-th time step;

[0041] An elasticity adjustment factor setting module is used to set the elasticity adjustment factor of the t-th time step based on the similarity between the Gram matrix of the t-th time step and the t-1-th time step;

[0042] The data enhancement module is used to perform strong data enhancement and weak data enhancement on the target domain image dataset corresponding to the t-th time step based on the elastic adjustment factor of the t-th time step;

[0043] The prediction module is used to input the target domain image dataset corresponding to the t-th time step into the student model to obtain the prediction result of the t-th time step;

[0044] The elastic pseudo-label acquisition module is used to input the target domain image dataset corresponding to the t-th time step after strong data augmentation and weak data augmentation into the teacher model, and update the pseudo-label output by the teacher model according to the elastic adjustment factor of the t-th time step to obtain the elastic pseudo-label of the t-th time step;

[0045] A loss function construction module is used to construct a global elastic symmetric cross entropy loss function at the t-th time step based on the elastic adjustment factor at the t-th time step, the cross entropy loss between the elastic pseudo label and the prediction result at the t-th time step, and the reverse cross entropy loss;

[0046] The parameter update module is used to update the parameters of the student model at the t-th time step through the global elastic symmetric cross entropy loss function at the t-th time step, and update the parameters of the teacher model at the t-th time step based on the updated parameters of the student model at the t-th time step. After completing the training of T time steps, the teacher model at the t-th time step with updated parameters is used as the target model.

[0047] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0048] The present invention discloses a method and system for continuous testing and adaptation of domain differences in construction scenarios. The method introduces an elastic adjustment factor based on domain differences from the perspective of data preprocessing and model training optimization, and comprehensively considers the differences between domain changes in actual construction scenarios. By constructing a Gram matrix representing the style differences of inter-domain features, the feature differences between different domains are quantified. Based on the similarity between the Gram matrices of adjacent domains, an elastic adjustment factor is selected to control the strength of data enhancement, and strong and weak data enhancement are performed on the target domain image dataset. By considering inter-domain differences in data preprocessing, the model can more effectively extract features when faced with complex and changeable construction scene data, and can better adapt to the complex and changeable domain situation. The enhanced target domain image dataset is input into the teacher model, and the pseudo-labels output by the teacher model are updated in combination with the elastic adjustment factor to obtain elastic pseudo-labels. The introduction of the elastic adjustment factor enables the pseudo-labels to better reflect the differences between different domains, thereby providing more accurate supervision information for model training. Based on the elastic adjustment factor, the cross-entropy loss between the elastic pseudo-label and the predicted results, and the reverse cross-entropy loss, a global elastic symmetric cross-entropy loss function is constructed. The elastic adjustment factor plays a balancing role in the loss function, dynamically adjusting the weight according to the inter-domain differences, so that the model can achieve a dynamic balance between imitating the teacher model and focusing on its own predicted results. The student model is trained using the global elastic symmetric cross-entropy loss function, and the parameters of the teacher model are updated based on the updated parameters of the student model, ultimately obtaining the target model. Inter-domain differences are considered at the model optimization level, enhancing the model's understanding and adaptability to changes in different domains. This enables the construction scene monitoring model to fully adapt to the continuous changes in the weather, temperature, lighting, construction activities, and other environmental conditions in the construction scene, improving the prediction accuracy of the construction scene monitoring model and enabling timely and accurate monitoring and early warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0050] Figure 1 This is a flowchart of the steps of a method for continuous testing and adaptation for differences between construction scene domains according to the present invention.

[0051] Figure 2 This is an overall structural diagram of a continuous testing adaptation method for inter-domain differences in construction scenarios according to the present invention.

[0052] Figure 3 This is a schematic diagram of comparative experiments with different intensities of data enhancement. Figure 3 (a) is a schematic diagram showing the relationship between data enhancement of different strengths and model plasticity and stability. Figure 3(b) is a schematic diagram of the error rate comparison experiment using strong data enhancement, weak data enhancement, and elastic data enhancement. Figure 3 (c) in the figure is a schematic diagram of four methods of strong data enhancement. Figure 3 (d) in the figure is a schematic diagram of four methods of weak data enhancement.

[0053] Figure 4 This is a structural diagram of the self-correction strategy based on elastic pseudo-label confidence.

[0054] Figure 5 This is a schematic diagram of the T-SNE visualization comparison experiment for different image processing methods. Figure 5 (a) is a schematic diagram of the T-SNE visualization comparison experiment using the CoTTA method for image processing. Figure 5 (b) is a schematic diagram of the T-SNE visualization comparison experiment using the RMT method for image processing. Figure 5 (c) is a schematic diagram of the T-SNE visualization comparison experiment using the DSS method for image processing. Figure 5 (d) is a schematic diagram of a T-SNE visualization comparison experiment for image processing using a continuous test-time adaptation method for inter-domain differences in construction scenarios according to the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0056] Continual Test-Time Adaption aims to adapt the source pre-trained model to the continuously changing target domain without using any source data.

[0057] Currently, continuous test-time adaptation primarily relies on self-training methods. Based on the average teacher model framework, the predicted values ​​of data-augmented samples are used as pseudo-labels, and a consistency loss function is constructed to achieve model self-training. As a machine learning technique, continuous test-time adaptation allows the model to dynamically adjust its parameters based on the input test samples during the testing phase to adapt to changes in the test data distribution, thereby improving the model's performance on the test data. This is particularly applicable when the test data distribution changes over time, enabling the model to more flexibly respond to dynamic environments.

[0058] Given a model trained on the source domain Parameters θ of the source domain training model src In the source data (x S ,y S ) for training, x Srepresents the input data in the source domain, y S Represents the labeled data in the source domain and in multiple target datasets D1, D2, ..., D t Tested on, M t Denotes the target domain D t The number of samples is large, and the distribution of the target domain can change over time or recur (such as sunny, cloudy, rainy, foggy, and nighttime domains in autonomous driving scenarios).

[0059] Currently, continuous test-time adaptation updates the parameters of the student model by minimizing the cross entropy during testing. The formula is:

[0060]

[0061] where q θ,c is the teacher model prediction x t is the probability of category c, x t is the data input into the model at the tth time step, p θ,c is the student model prediction x t is the probability of category c. By constructing the average teacher model framework, the model can improve performance through self-optimization training without source data.

[0062] Reference Figure 1 As shown, Figure 1 The figure is a flow chart of a method for continuous testing and adaptation for inter-domain differences in construction scenarios according to the present invention.

[0063] This first embodiment provides a method for continuous testing adaptation targeting inter-domain differences in construction scenarios, including the following steps:

[0064] The present invention sets the target domain data D at time step t. t As input, the model Need to make predictions and output prediction results And adapt to the input of future domain accordingly. t The data distribution is constantly changing, and the model is evaluated based on online predictions.

[0065] like Figure 2 As shown, Figure 2 This is an overall structural diagram of the present invention's continuous testing adaptation method for inter-domain differences in construction scenarios.

[0066] Step S1: Acquire a construction scene image test set, wherein the construction scene image test set includes a target domain image dataset corresponding to T time steps in the construction area to be monitored;

[0067] The present invention constructs an average teacher model framework to realize self-training of the model. At time step t=1, the pre-trained model is used to initialize the student and teacher models, and the parameters of the pre-trained model are assigned to the corresponding parameters of the student model and the teacher model respectively.

[0068] For any target domain in the Continuous Target Domain Adaptation (CTTA) process The present invention takes into account the differences between domains. Specifically, there are differences in image color shift, background complexity, texture differences or lighting conditions between different target domains. These differences may cause simple random data enhancement to be unable to effectively simulate the true distribution of the target domain, thereby affecting the performance of the model. Therefore, the present invention believes that in the process of continuous testing and adaptation, it is necessary to detect and analyze the differences between adjacent domains in real time, deploy a controllable elastic data enhancement strategy, and achieve a balance between model plasticity and stability, thereby ensuring continuous adaptation and effective retention of knowledge. Therefore, the present invention represents the inter-domain feature style based on the Gram matrix, calculates the feature style differences between adjacent domains, and selects elastic adjustment factors based on the feature style differences.

[0069] Step S2: Using the target domain image dataset corresponding to the t-th time step as the input of the feature extraction network, and obtaining the domain feature map of each target domain image at the t-th time step; where t = 1, 2…T;

[0070] The target domain image dataset corresponding to the t-th time step is used as the input of the feature extraction network to obtain the domain feature map of each target domain image at the t-th time step. The formula is:

[0071] F j,t =φ(x j,t ),

[0072] Among them, x j,t is the jth target domain image in the target domain image dataset corresponding to the tth time step, φ(.) is the feature extraction network, F j,t is the domain feature map of the jth target domain image in the target domain image dataset corresponding to the tth time step.

[0073] In this embodiment, specifically, the feature extraction network is any one of a convolutional neural network (CNN), a residual neural network (ResNet), and a MobileNet network.

[0074] Step S3: Flatten the domain feature map of each target domain image at the t-th time step, and construct the Gram matrix of the t-th time step based on the flattened domain feature maps of all target domain images at the t-th time step;

[0075] In order to enable the Gram matrix to capture the global distribution of different channel features, the present invention flattens the domain feature map of each target domain image at the t-th time step. After flattening, its spatial dimension H j and W j The spatial information of the channel features is merged into a single dimension, and the spatial information of the channel features is ignored. Instead, we focus on the overall distribution of the spatial features, and obtain the flattened domain feature map of all target domain images at the t-th time step to describe the style information Gram matrix of the features.

[0076] Flatten the domain feature map of each target domain image at the t-th time step. The formula is:

[0077]

[0078] in, is the domain feature map after the jth target domain image in the target domain image dataset corresponding to the tth time step is flattened, Flatten(.) is the flattening operation, C j is the number of channels of the jth target domain image in the target domain image dataset corresponding to the tth time step, W j is the width of the jth target domain image in the target domain image dataset corresponding to the tth time step, H j is the height of the jth target domain image in the target domain image dataset corresponding to the tth time step.

[0079] The present invention constructs a Gram matrix to represent the feature style of the variation domain, and the Gram matrix quantifies the correlation between different features by calculating the inner product between feature maps.

[0080] Based on the flattened domain feature maps of all target domain images at the t-th time step, the Gram matrix of the t-th time step is constructed. The formula is:

[0081]

[0082] Among them, G t is the Gram matrix of the t-th time step, . T is the transposed, flattened domain feature map matrix of all target domain images at the t-th time step By M Combined, M is the total number of target domain images in the target domain image dataset, yes The transpose of .

[0083] The Gram matrix of the t-th time step is expanded as:

[0084]

[0085] Among them, G abThe element in the ath row and bth column of the Gram matrix at the tth time step reflects the correlation between channel a and channel b. is the element in the ath row and kth column of the domain feature map after the jth target domain image is flattened in the target domain image dataset corresponding to the tth time step, is the element in the ath row and kth column of the domain feature map after the jth target domain image in the target domain image dataset corresponding to the tth time step is flattened. Different channels represent different types of features, such as texture, edge, color, etc. If G ab The value is close to 0, indicating that their feature distributions in space are almost independent, which means that there is no obvious correlation between channels.

[0086] Step S4: Based on the similarity between the Gram matrices of the t-th time step and the t-1-th time step, set the elasticity adjustment factor of the t-th time step;

[0087] In this embodiment, specifically, the similarity between the Gram matrices based on the t-th time step and the t-1-th time step is any one of cosine similarity, Euclidean distance, and Pearson correlation coefficient.

[0088] In this embodiment, preferably, the elasticity adjustment factor of the t-th time step is set based on the cosine similarity between the Gram matrices of the t-th time step and the t-1-th time step, and the formula is:

[0089]

[0090] Among them, S sim (G t-1 ,G t ) is the cosine similarity between the Gram matrices at the t-th time step and the t-1-th time step, G t-1 is the Gram matrix of the t-1th time step, ε is the element index, G t,ε is the εth element in the Gram matrix at the tth time step, G t-1,ε is the εth element in the Gram matrix at the t-1th time step.

[0091] The Gram matrix contains the correlation information between image features in different channels. Its changes in consecutive time steps reflect the dynamic evolution of the feature distribution. Using cosine similarity to measure the Gram matrix of adjacent time steps can accurately capture the degree of change in the feature distribution in the time domain. When the cosine similarity is high, it indicates that the feature distribution of the data is relatively stable in consecutive time steps. At this time, a smaller elastic adjustment factor can be set to make the model more dependent on the previous processing method and parameters when processing the data in the tth time step to maintain processing stability; when the cosine similarity is low, it means that the feature distribution has changed significantly in the time domain. In this case, a larger elastic adjustment factor can be set to enable the model to respond quickly to such changes and flexibly adjust parameters and processing methods. This allows the model to achieve efficient and accurate adaptive adjustments while maintaining stability, significantly improving the model's performance in time domain adaptive tasks.

[0092] If the similarity S between the Gram matrix of the t-th time step and the t-1-th time step sim (G t-1 ,G t ) is greater than or equal to the second set threshold τ, then the elastic adjustment factor ω of the t-th time step is set t is 0.4;

[0093] If the similarity S between the Gram matrix of the t-th time step and the t-1-th time step sim (G t-1 ,G t ) is less than the second set threshold τ, then the elastic adjustment factor ω of the t-th time step is set to t is 0.8.

[0094] When t When it is 0.4, it is suitable for situations where the feature changes are small. It allows the model to be moderately optimized and adjusted while maintaining stability, making full use of the information and experience accumulated in the previous time steps to improve the learning efficiency and generalization ability of the model.

[0095] When t When it is 0.8, it is suitable for situations where features change greatly, giving the model stronger adaptability, enabling it to quickly get rid of previously inapplicable parameters and processing methods, and adapt to new feature environments more quickly. It helps the model capture dynamic changes in the data and improves the model's adaptability to complex and changing data, thereby maintaining good performance under different feature distributions and better adapting to complex and changing new domains.

[0096] Because domain changes are unpredictable in practical applications, this paper proposes a confidence-based domain change detection mechanism. Experiments have shown that prediction confidence fluctuates significantly with domain changes, a phenomenon that can be used to estimate whether the target domain has changed. To this end, this paper sets a first threshold and continuously calculates the difference between prediction confidences between adjacent domains.

[0097] In this embodiment, preferably, the prediction result of the t-th time step is passed through the Softmax activation function to obtain a prediction confidence list of the t-th time step;

[0098] In the prediction confidence list of the t-th time step, a category is randomly selected. If the difference between the average prediction confidence value of the category at the t-th time step and the average prediction confidence value of the category at the t-1-th time step is greater than the first set threshold, the elastic adjustment factor of the t-th time step is set by calculating the similarity between the Gram matrices of the t-th time step and the t-1-th time step;

[0099] If the difference between the average prediction confidence value of the category at the t-th time step and the average prediction confidence value of the category at the t-1-th time step is less than or equal to the first set threshold, the elastic adjustment factor of the t-th time step is directly set to 1.

[0100] The formula for calculating the difference between the average prediction confidence value of the category selected at the t-th time step and the average prediction confidence value of the category at the t-1-th time step is:

[0101] ΔConf(t)=Conf(t)-Conf(t-1),

[0102] Among them, ΔConf(t) is the difference between the average prediction confidence value of the category selected at the t-th time step and the average prediction confidence value of the category at the t-1-th time step, Conf(t) is the average prediction confidence value of the category selected at the t-th time step, and Conf(t-1) is the average prediction confidence value of the category selected at the t-1-th time step.

[0103] Gram matrix calculations typically involve operations like matrix multiplication, which can be computationally intensive on large datasets. Only when the difference in prediction confidence indicates a possible domain change is the similarity between Gram matrices calculated and a resilience adjustment factor set. This avoids the complex and time-consuming Gram matrix similarity calculations when the domains haven't significantly changed. This significantly saves computing resources and time, improves model efficiency, and avoids unnecessary calculations.

[0104] When the difference in average prediction confidence for the same category between adjacent time steps is small—that is, less than or equal to the first set threshold—the model is in a relatively stable prediction state, and the domain may not have changed significantly. In this case, simply setting the elastic adjustment factor to 1 maintains the consistency of the model training strategy and avoids training fluctuations caused by frequent adjustments to the elastic adjustment factor. This helps the model continuously optimize parameters in a stable environment, improving training stability and effectiveness. When the difference in average prediction confidence for the same category between adjacent time steps is greater than the first set threshold, it indicates that the domain may have changed. In this case, calculating the Gram matrix similarity to set the elastic adjustment factor allows for more precise adjustments to the model training strategy based on changes in the image feature distribution.

[0105] Step S5: Based on the elastic adjustment factor of the t-th time step, strong data enhancement and weak data enhancement are performed on the target domain image dataset corresponding to the t-th time step respectively;

[0106] In recent years, the fields of continuous test-time adaptation and unsupervised domain adaptation (UDA) have been widely used to improve model performance by using the predicted values ​​of the model after data augmentation as pseudo-labels during training to simulate the self-learning correction mechanism in the human learning process. However, giving the model random data augmentation samples will lead to a series of problems. Excessive data augmentation leads to stability problems, and too weak data augmentation makes it impossible to effectively simulate the true distribution of the target domain, resulting in insufficient generalization performance of the model. Therefore, based on the differences between domains, the present invention proposes a controllable elastic data augmentation strategy. Through the elastic adjustment factor, strong data augmentation and weak data augmentation are performed on the target domain image dataset corresponding to the tth time step respectively.

[0107] The elastic adjustment factor based on the t-th time step performs strong data enhancement and weak data enhancement on the target domain image dataset corresponding to the t-th time step, respectively. The formula is:

[0108]

[0109] in, is the jth target domain image in the target domain image dataset corresponding to the tth time step after strong data enhancement, is the jth target domain image in the target domain image dataset corresponding to the tth time step after weak data enhancement, is the data enhancement operation, ω t is the elastic adjustment factor at the t-th time step, is the random factor of the i-th strong perturbation at the t-th time step, is the random factor of the i-th weak perturbation at the t-th time step, x j,tis the jth target domain image in the target domain image dataset corresponding to the tth time step, n is the number of strong / weak perturbations, and j is the index of the target domain image in the target domain image dataset corresponding to the tth time step.

[0110] Traditional continuous test-time adaptation methods typically employ random data augmentation strategies to achieve model self-training. However, this approach has significant limitations and completely ignores the inherent variability between domain changes. The feature distribution of target domain data at different time steps is not static but rather dynamically changing. Domain changes between time steps may vary in degree, direction, and pattern.

[0111] The present invention has obvious advantages by introducing a strategy of strong data enhancement and weak data enhancement for the target domain image dataset by introducing an elastic adjustment factor based on inter-domain differences. It can flexibly adjust the strength and method of data enhancement according to different domain changes, so that the model can maintain stable performance in a stable domain environment during continuous testing and can quickly adjust in a drastically changing domain environment, thereby improving the generalization ability and robustness of the model, effectively overcoming the disadvantage of traditional methods that ignore domain change differences, and thus improving image processing accuracy.

[0112] like Figure 3 As shown, Figure 3 Schematic diagram of comparative experiments for data enhancement with different intensities. Figure 3 (a) is a schematic diagram of the relationship between data enhancement of different intensities and model plasticity and stability. Figure 3 (b) is a schematic diagram of the error rate comparison experiment using strong data enhancement, weak data enhancement, and elastic data enhancement. Figure 3 (c) in the figure is a schematic diagram of four methods of strong data enhancement. Figure 3 (d) in the figure is a schematic diagram of four methods of weak data enhancement.

[0113] Depend on Figure 3 From (a) in the figure, we can see that the elastic data enhancement proposed in the present invention (strong data enhancement and weak data enhancement of the target domain image dataset based on the elastic adjustment factor of inter-domain differences) achieves a balance between plasticity and stability.

[0114] Strong data augmentation aims to improve the model's generalization capabilities by significantly transforming the data, enabling it to cope with a variety of complex and changing situations. However, this approach, which excessively pursues generalization, has obvious drawbacks. When faced with certain specific inter-domain changes, especially when the inter-domain differences are not significant, such as a scene with little difference from heavy rain to light rain, excessive data augmentation can cause the model to forget previously learned knowledge in similar domains while learning new domain knowledge. This is because the drastic changes brought about by strong data augmentation obscure the similarities between the previous and next domains, making it difficult for the model to simultaneously consider new and old knowledge in a short period of time, resulting in the gradual overwriting of knowledge in the previous domain.

[0115] Weak data augmentation transforms the data to a smaller extent. While this ensures the model retains existing knowledge to a certain extent and maintains its stability, enabling it to more reliably apply previously learned knowledge to new data, due to the limited strength of data augmentation, the model is exposed to fewer new features and changes, resulting in relatively limited flexibility. In situations where rapid adaptation to new domain features is required, weak data augmentation may not prompt the model to adjust in a timely manner, resulting in poor performance in the new environment.

[0116] The elastic data augmentation method proposed in this paper, based on elastic adjustment factors, can dynamically adjust the augmentation strategy based on inter-domain differences. When inter-domain differences are small, weak data augmentation is used to moderately expand existing knowledge while ensuring model stability. When inter-domain differences are large, strong data augmentation is promptly switched to promote the model to actively learn new domain knowledge and improve its plasticity. This method effectively avoids the limitations of strong and weak data augmentation, achieves a balance between model plasticity and stability, and significantly improves the model's adaptability and performance in different domain environments.

[0117] Depend on Figure 3 As can be seen from (b) in the figure, when only strong data enhancement and weak data enhancement are used between different domains, the image processing error rate is greater than the elastic data enhancement method based on the elastic adjustment factor proposed in this invention.

[0118] Depend on Figure 3 As shown in (c) and (d), strong data enhancement methods include occlusion, noise addition, splicing, and blurring. Weak data enhancement methods include rotation, scaling, and changing saturation and color.

[0119] This paper proposes a flexible data enhancement strategy based on inter-domain differences. This strategy constructs a Gram matrix representing the inter-domain feature style, calculates the differences between adjacent domains, selects appropriate elastic adjustment factors to control the intensity of data enhancement, and considers inter-domain differences in data preprocessing, so that the model can better adapt to complex and changing domain situations.

[0120] Elastic data augmentation may produce a large number of strongly augmented samples. This can blur sample characteristics, making it difficult for the model to accurately predict the correct labels for the augmented samples. This can lead to poor pseudo-label quality and cause error accumulation. Furthermore, weak data augmentation, due to the smaller degree of change, makes the model more likely to predict the correct label, resulting in a relatively high degree of pseudo-label credibility and less error accumulation.

[0121] Step S6: Input the target domain image dataset corresponding to the t-th time step into the student model to obtain the prediction result of the t-th time step;

[0122] Step S7: Input the target domain image dataset corresponding to the t-th time step after strong data augmentation and weak data augmentation into the teacher model respectively, and update the pseudo label output by the teacher model according to the elastic adjustment factor of the t-th time step to obtain the elastic pseudo label of the t-th time step;

[0123] In this embodiment, preferably, the target domain image dataset corresponding to the t-th time step after strong data augmentation and weak data augmentation is input into the teacher model, and the pseudo label output by the teacher model is updated according to the elastic adjustment factor of the t-th time step to obtain the elastic pseudo label of the t-th time step. The formula is:

[0124]

[0125] in, is the jth target domain image x in the target domain image dataset corresponding to the tth time step j,t Elastic pseudo-label, M is the total number of target domain images in the target domain image dataset, ω t is the elastic adjustment factor at the t-th time step, is the pseudo label of the jth target domain image in the target domain image dataset corresponding to the tth time step after strong data enhancement, is the pseudo label of the j-th target domain image in the target domain image dataset corresponding to the t-th time step after weak data enhancement, and j is the index of the target domain image in the target domain image dataset corresponding to the t-th time step.

[0126] Pseudo-labels are generated through predictions from the teacher model and play a key role in CTTA learning. By passing elastic adjustment factors based on inter-domain differences to the pseudo-labels output by the teacher model, the model can dynamically adjust the pseudo-labels based on inter-domain differences, enabling the model to flexibly respond to continuous changes in data distribution and making these labels better reflect the model's understanding and adaptability to different domains.

[0127] like Figure 4 As shown, Figure 4The structure diagram of the self-correction strategy based on elastic pseudo-label confidence. Due to the error accumulation phenomenon of the model in knowledge transfer, the present invention proposes a self-correction strategy based on elastic pseudo-label confidence.

[0128] In this embodiment, preferably, the elastic pseudo-label of the t-th time step is passed through the Softmax activation function to obtain the elastic pseudo-label confidence list of the t-th time step;

[0129] For each target domain image in the target domain image dataset corresponding to the t-th time step, if the elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the weak data enhancement is greater than the confidence threshold, then the elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the target domain image after weak data enhancement is used to correct the elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the strong data enhancement, and the elastic pseudo-label of the target domain image dataset after strong data enhancement at the t-th time step is obtained.

[0130] In this embodiment, by setting a confidence threshold γ, for each target domain image in the target domain image dataset corresponding to the t-th time step, the category label whose elastic pseudo label confidence value is greater than the confidence threshold γ is selected. The formula is:

[0131]

[0132] Among them, x[j][c] represents whether the j-th target domain image in the target domain image dataset corresponding to the t-th time step satisfies the elastic pseudo label confidence greater than the confidence threshold in category c, and Π(.) is the gating function. is the jth target domain image x in the target domain image dataset corresponding to the tth time step j,t Elastic pseudo-label, γ is the confidence threshold, if If yes, then x[j][c] is 1, otherwise it is 0.

[0133] In this embodiment, preferably, the elastic pseudo label confidence value of the current category in the elastic pseudo label corresponding to the current target domain image after weak data enhancement is used to correct the elastic pseudo label confidence value of the current category in the elastic pseudo label corresponding to the current target domain image after strong data enhancement. The formula is:

[0134]

[0135]

[0136] in, is the elastic pseudo-label confidence value of category c in the elastic pseudo-label corresponding to the current target domain image after strong data enhancement, is the elastic pseudo-label confidence value of the j-th target domain image in the target domain image dataset corresponding to the t-th time step in category c after weak data enhancement, is the elastic pseudo-label confidence value of the j-th target domain image in the target domain image dataset corresponding to the t-th time step in category c after weak data enhancement, σ[j,c] is the adaptive correction weight of the j-th target domain image in the target domain image dataset corresponding to the t-th time step in category c, and KL(.) is the KL divergence.

[0137] If the KL divergence is small, it means that the prediction of the current target domain image after strong data augmentation is similar to the prediction after weak data augmentation, indicating high prediction accuracy and fewer strong data augmentation predictions that need to be corrected. Through the above operations, the pseudo labels output by the teacher model are corrected, reducing the error accumulation of the model during knowledge transfer.

[0138] Step S8: constructing a global elastic symmetric cross entropy loss function at the t-th time step based on the elastic adjustment factor at the t-th time step, the cross entropy loss between the elastic pseudo label and the prediction result at the t-th time step, and the reverse cross entropy loss;

[0139] In this embodiment, preferably, the elastic adjustment factor at the t-th time step, the cross entropy loss between the elastic pseudo label at the t-th time step and the prediction result, and the reverse cross entropy loss are used to construct a global elastic symmetric cross entropy loss function at the t-th time step, and the formula is:

[0140]

[0141] in, is the global elastic symmetric cross entropy loss function at the t-th time step, C t is the number of categories of the target domain image dataset corresponding to the tth time step, ω t is the elastic adjustment factor of the t-th time step, c is the category index, is the jth target domain image x in the target domain image dataset corresponding to the tth time step j,t Elastic pseudo-label value on category c, p c (x j,t ) is the jth target domain image x in the target domain image dataset corresponding to the tth time step j,t The predicted value for class c.

[0142] Increase ω t This will make the student model more focused on imitating the predicted distribution of the teacher model and reduce ω t This can make the student model pay more attention to its own prediction results. For example, if the difference between domains is large, increase ω tThis approach allows the student model to approximate the prediction distribution of the teacher model's strongly augmented samples, improving model generalization. By incorporating domain-specific elastic adjustment factors into data preprocessing and model optimization, we achieve global elastic knowledge transfer that adapts during continuous testing.

[0143] This paper proposes a global elastic symmetric cross-entropy loss function. This function applies elastic adjustment factors calculated based on inter-domain differences to the pseudo-labels output by the teacher model and the construction of the consistency loss function. By integrating adjustment factors based on inter-domain differences, the global elastic symmetric cross-entropy loss function enables the model to achieve a dynamic balance between mimicking the teacher model's predicted distribution and focusing on its own predictions. This design aims to optimize the generalization performance of the construction scenario monitoring model while ensuring its stability, thereby achieving a better balance between model training and updating. It considers inter-domain differences at the model optimization level and enhances the model's understanding and adaptability to changes between different domains.

[0144] Step S9: Update the parameters of the student model at the t-th time step through the global elastic symmetric cross entropy loss function of the t-th time step. Based on the updated parameters of the student model at the t-th time step, update the parameters of the teacher model at the t-th time step. After completing the training of T time steps, use the teacher model at the T-th time step after the updated parameters as the target model.

[0145] In this embodiment, specifically, based on the updated parameters of the student model at the t-th time step, the parameters of the teacher model at the t-th time step are updated by exponential moving average, and the formula is:

[0146]

[0147] in, are the parameters of the teacher model at the t+1th time step, is the parameter of the teacher model at the tth time step, δ t is the smoothing factor of the average movement of the t-th time step, min(.) is the minimum value, is the parameter of the student model at the tth time step, and e is a natural constant.

[0148] In the process of updating the parameters of the student model at the t-th time step through the global elastic symmetric cross entropy loss function at the t-th time step, the student model has a certain probability of randomly recovering and randomly selecting individual parameters to restore to the corresponding parameters of the student model at the previous t time steps.

[0149] The smoothing factor δ of the moving average t As the time step t increases, it gradually approaches 1. Therefore, in the early The weight is larger and tends to learn new knowledge. This dynamic adjustment can make the algorithm learn quickly in the early stage. As the time step t increases, δ t Gradually approaching 1, the model is more stable in the later training stage and is more inclined to maintain existing knowledge, reducing the occurrence of error accumulation during model training.

[0150] In this example, a series of experiments were conducted to verify the effectiveness of the proposed method. In the field of image processing, the proposed method was evaluated on three benchmark tasks: CIFAR-10-C, which expands on the CIFAR-10 dataset and contains 32×32 color images from 10 categories. This dataset introduces 15 different types of perturbations, each with five different severity levels. These perturbations are applied to CIFAR-10 test images, resulting in a dataset of 10,000 images. CIFAR-100-C, which expands on the CIFAR-100 dataset and covers 32×32 color images from 100 categories, also includes 15 different types of perturbations, each with five severity levels. These perturbations are applied to CIFAR-100 test images, resulting in a dataset of 10,000 images. ImageNet-C expands upon the ImageNet dataset, which contains over 14 million images covering over 20,000 categories. ImageNet-C also includes 15 different perturbation types, each with five severity levels, which are applied to the ImageNet test set.

[0151] All experiments in this paper were conducted using PyTorch and trained on a single NVIDIA RTX3090 graphics card. The experiments were set up strictly in accordance with CoTTA, meaning that source data was not reused. Standard pre-trained WideResNet, ResNeXt-29, and ResNet-50 models were used as source models for CIFAR10-C, CIFAR100-C, and ImageNet-C. The batch size was set to 200.

[0152] Table 1 shows experimental data comparing the error rates of image processing using different continuous testing and adaptation methods. These methods include TENT, Ada, Cotta, RMT, DSS, BeCoTTA, PALM, and a continuous testing and adaptation method proposed in this invention that addresses inter-domain differences in construction scenarios. The mean error rate for each method under various image interference or transformation types, such as Gau, Sho, Imp, and Def, is obtained.

[0153] Table 1

[0154]

[0155]

[0156] As shown in Table 1, the proposed continuous testing adaptation method for inter-domain differences in construction scenarios has a lower mean error rate than other methods under different types of image interference or transformations, such as Gau, Sho, Imp, and Def. This method can effectively improve the performance of the model in different domain environments and enhance the accuracy of model image processing.

[0157] As shown in Table 2, Table 2 compares experimental data under the condition of domain random input using different continuous test-time adaptation methods for image processing.

[0158]

[0159] Among them, Random 1, Random 2, Random 3, and Random 4 represent the test results of different randomization processing of images in the domain random input scenario. As shown in Table 2, the continuous test-time adaptation method for inter-domain differences in construction scenarios proposed in this invention is more effective than other methods when facing randomly input domain data, with the lowest error rate.

[0160] As shown in Table 3, Table 3 is the ablation experiment data of the continuous testing adaptation method for inter-domain differences in construction scenarios of the present invention.

[0161]

[0162]

[0163] Among them, GE-SCE is a global elastic symmetric cross entropy loss function, EDA is a controllable elastic data augmentation strategy, and PLSC is a self-correction strategy based on elastic pseudo-label confidence. The effectiveness and complementarity of GE-SCE, EDA and PLSC in improving image processing effects are verified.

[0164] like Figure 5 As shown, Figure 5 This is a schematic diagram of the T-SNE visualization comparison experiment for different image processing methods. Figure 5 (a) is a schematic diagram of the T-SNE visualization comparison experiment using the CoTTA method for image processing. Figure 5 (b) is a schematic diagram of the T-SNE visualization comparison experiment using the RMT method for image processing. Figure 5(c) is a schematic diagram of the T-SNE visualization comparison experiment using the DSS method for image processing. Figure 5 (d) is a schematic diagram of a T-SNE visualization comparison experiment for image processing using a continuous test-time adaptation method for inter-domain differences in construction scenarios according to the present invention.

[0165] Depend on Figure 5 It can be seen that compared with methods such as CoTTA, RMT and DSS, the continuous test-time adaptation method proposed in the present invention shows a tighter clustering of similar samples and a clearer separation of different categories of samples in the T-SNE visualization comparison experiment of image processing, effectively improving the separability of the feature space.

[0166] This second embodiment provides a continuous testing and adaptation system for inter-domain differences in construction scenarios, including:

[0167] A data acquisition model is used to acquire a construction scene image test set, wherein the construction scene image test set includes a target domain image dataset corresponding to T time steps in the construction area to be monitored;

[0168] The feature extraction module is used to take the target domain image dataset corresponding to the t-th time step as the input of the feature extraction network to obtain the domain feature map of each target domain image at the t-th time step;

[0169] t=1,2…T;

[0170] The Gram matrix construction module is used to flatten the domain feature map of each target domain image at the t-th time step, and construct the Gram matrix of the t-th time step based on the flattened domain feature maps of all target domain images at the t-th time step;

[0171] An elasticity adjustment factor setting module is used to set the elasticity adjustment factor of the t-th time step based on the similarity between the Gram matrix of the t-th time step and the t-1-th time step;

[0172] The data enhancement module is used to perform strong data enhancement and weak data enhancement on the target domain image dataset corresponding to the t-th time step based on the elastic adjustment factor of the t-th time step;

[0173] The prediction module is used to input the target domain image dataset corresponding to the t-th time step into the student model to obtain the prediction result of the t-th time step;

[0174] The elastic pseudo-label acquisition module is used to input the target domain image dataset corresponding to the t-th time step after strong data augmentation and weak data augmentation into the teacher model, and update the pseudo-label output by the teacher model according to the elastic adjustment factor of the t-th time step to obtain the elastic pseudo-label of the t-th time step;

[0175] A loss function construction module is used to construct a global elastic symmetric cross entropy loss function at the t-th time step based on the elastic adjustment factor at the t-th time step, the cross entropy loss between the elastic pseudo label and the prediction result at the t-th time step, and the reverse cross entropy loss;

[0176] The parameter update module is used to update the parameters of the student model at the t-th time step through the global elastic symmetric cross entropy loss function at the t-th time step, and update the parameters of the teacher model at the t-th time step based on the updated parameters of the student model at the t-th time step. After completing the training of T time steps, the teacher model at the t-th time step with updated parameters is used as the target model.

[0177] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0178] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0179] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0181] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A continuous testing and adaptation method for inter-domain differences in construction scenarios, characterized by: The following steps are involved: Obtain a construction scene image test set, where the construction scene image test set includes a target domain image dataset corresponding to T time steps in the construction area to be monitored; The target domain image dataset corresponding to the t-th time step is used as the input of the feature extraction network to obtain the domain feature map of each target domain image at the t-th time step; where t = 1, 2…T; Flatten the domain feature map of each target domain image at the t-th time step, and construct the Gram matrix of the t-th time step based on the flattened domain feature maps of all target domain images at the t-th time step; Based on the similarity between the Gram matrices of the t-th time step and the t-1-th time step, the elastic adjustment factor of the t-th time step is set; Based on the elastic adjustment factor of the t-th time step, strong data enhancement and weak data enhancement are performed on the target domain image dataset corresponding to the t-th time step respectively; Input the target domain image dataset corresponding to the t-th time step into the student model and obtain the prediction result of the t-th time step; The target domain image dataset corresponding to the t-th time step after strong data augmentation and weak data augmentation is input into the teacher model respectively, and the pseudo label output by the teacher model is updated according to the elastic adjustment factor of the t-th time step to obtain the elastic pseudo label of the t-th time step; Based on the elastic adjustment factor at the t-th time step, the cross entropy loss and the reverse cross entropy loss between the elastic pseudo-label and the prediction result at the t-th time step, a global elastic symmetric cross entropy loss function at the t-th time step is constructed; The parameters of the student model at the t-th time step are updated through the global elastic symmetric cross entropy loss function at the t-th time step. Based on the updated parameters of the student model at the t-th time step, the parameters of the teacher model at the t-th time step are updated. After completing the training of T time steps, the teacher model at the T-th time step with updated parameters is used as the target model.

2. The method for continuous testing and adaptation for inter-domain differences in construction scenarios according to claim 1 is characterized in that: The prediction result of the t-th time step is passed through the Softmax activation function to obtain the prediction confidence list of the t-th time step; In the prediction confidence list of the t-th time step, a category is randomly selected. If the difference between the average prediction confidence value of the category at the t-th time step and the average prediction confidence value of the category at the t-1-th time step is greater than the first set threshold, the elastic adjustment factor of the t-th time step is set by calculating the similarity between the Gram matrices of the t-th time step and the t-1-th time step; If the difference between the average prediction confidence value of the category at the t-th time step and the average prediction confidence value of the category at the t-1-th time step is less than or equal to the first set threshold, the elastic adjustment factor of the t-th time step is directly set to 1.

3. The method for continuous testing and adaptation for inter-domain differences in construction scenarios according to claim 1, characterized in that: If the similarity between the Gram matrices of the t-th time step and the t-1-th time step is greater than or equal to the second set threshold, the elasticity adjustment factor of the t-th time step is set to 0.4; If the similarity between the Gram matrices of the t-th time step and the t-1-th time step is less than the second set threshold, the elasticity adjustment factor of the t-th time step is set to 0.

8.

4. The method for continuous testing and adaptation for inter-domain differences in construction scenarios according to claim 1, characterized in that: The elastic adjustment factor based on the t-th time step performs strong data enhancement and weak data enhancement on the target domain image dataset corresponding to the t-th time step, respectively. The formula is: in, is the jth target domain image in the target domain image dataset corresponding to the tth time step after strong data enhancement, is the jth target domain image in the target domain image dataset corresponding to the tth time step after weak data enhancement, is the data enhancement operation, v t is the elastic adjustment factor at the t-th time step, is the random factor of the i-th strong perturbation at the t-th time step, is the random factor of the i-th weak perturbation at the t-th time step, x j,t is the jth target domain image in the target domain image dataset corresponding to the tth time step, n is the number of strong / weak perturbations, and j is the index of the target domain image in the target domain image dataset corresponding to the tth time step.

5. The method for continuous testing and adaptation for inter-domain differences in construction scenarios according to claim 1, characterized in that: Pass the elastic pseudo-label of the t-th time step through the Softmax activation function to obtain the elastic pseudo-label confidence list of the t-th time step; For each target domain image in the target domain image dataset corresponding to the t-th time step, if the elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the weak data enhancement is greater than the confidence threshold, then the elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the target domain image after weak data enhancement is used to correct the elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the strong data enhancement, and the elastic pseudo-label of the target domain image dataset after strong data enhancement at the t-th time step is obtained.

6. The method for continuous testing and adaptation for inter-domain differences in construction scenarios according to claim 5, characterized in that: The elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the weak data enhancement of the current target domain image is used to correct the elastic pseudo-label confidence value of the current category in the elastic pseudo-label corresponding to the strong data enhancement. The formula is: in, is the elastic pseudo-label confidence value of category c in the elastic pseudo-label corresponding to the current target domain image after strong data enhancement, is the elastic pseudo-label confidence value of the j-th target domain image in the target domain image dataset corresponding to the t-th time step in category c after weak data enhancement, is the elastic pseudo-label confidence value of the j-th target domain image in the target domain image dataset corresponding to the t-th time step in category c after weak data enhancement, σ[j,c] is the adaptive correction weight of the j-th target domain image in the target domain image dataset corresponding to the t-th time step in category c, and KL(.) is the KL divergence.

7. The method for continuous testing and adaptation for inter-domain differences in construction scenarios according to claim 1, characterized in that: The target domain image dataset corresponding to the t-th time step after strong data augmentation and weak data augmentation is input into the teacher model respectively, and the pseudo label output by the teacher model is updated according to the elastic adjustment factor of the t-th time step to obtain the elastic pseudo label of the t-th time step. The formula is: in, is the jth target domain image x in the target domain image dataset corresponding to the tth time step j,t Elastic pseudo-label, M is the total number of target domain images in the target domain image dataset, ω t is the elastic adjustment factor at the t-th time step, is the pseudo label of the jth target domain image in the target domain image dataset corresponding to the tth time step after strong data enhancement, is the pseudo label of the j-th target domain image in the target domain image dataset corresponding to the t-th time step after weak data enhancement, and j is the index of the target domain image in the target domain image dataset corresponding to the t-th time step.

8. The method for continuous testing and adaptation for inter-domain differences in construction scenarios according to claim 1, characterized in that: Based on the elastic adjustment factor at the t-th time step, the cross entropy loss and the reverse cross entropy loss between the elastic pseudo label and the prediction result at the t-th time step, a global elastic symmetric cross entropy loss function at the t-th time step is constructed, and the formula is: in, is the global elastic symmetric cross entropy loss function at the t-th time step, C t is the number of categories of the target domain image dataset corresponding to the t-th time step, ω t is the elastic adjustment factor of the t-th time step, c is the category index, is the jth target domain image x in the target domain image dataset corresponding to the tth time step j,t Elastic pseudo-label value on category c, p c (x j,t ) is the jth target domain image x in the target domain image dataset corresponding to the tth time step j,t The predicted value for class c.

9. The method for continuous testing and adaptation for inter-domain differences in construction scenarios according to claim 1, characterized in that: Based on the updated parameters of the student model at the t-th time step, the parameters of the teacher model at the t-th time step are updated by exponential moving average.

10. A continuous testing and adaptation system for inter-domain differences in construction scenarios, characterized by: include: A data acquisition model is used to acquire a construction scene image test set, wherein the construction scene image test set includes a target domain image dataset corresponding to T time steps in the construction area to be monitored; The feature extraction module is used to take the target domain image dataset corresponding to the t-th time step as the input of the feature extraction network to obtain the domain feature map of each target domain image at the t-th time step; t=1,2…T; The Gram matrix construction module is used to flatten the domain feature map of each target domain image at the t-th time step, and construct the Gram matrix of the t-th time step based on the flattened domain feature maps of all target domain images at the t-th time step; An elasticity adjustment factor setting module is used to set the elasticity adjustment factor of the t-th time step based on the similarity between the Gram matrix of the t-th time step and the t-1-th time step; The data enhancement module is used to perform strong data enhancement and weak data enhancement on the target domain image dataset corresponding to the t-th time step based on the elastic adjustment factor of the t-th time step; The prediction module is used to input the target domain image dataset corresponding to the t-th time step into the student model to obtain the prediction result of the t-th time step; The elastic pseudo-label acquisition module is used to input the target domain image dataset corresponding to the t-th time step after strong data augmentation and weak data augmentation into the teacher model, and update the pseudo-label output by the teacher model according to the elastic adjustment factor of the t-th time step to obtain the elastic pseudo-label of the t-th time step; A loss function construction module is used to construct a global elastic symmetric cross entropy loss function at the t-th time step based on the elastic adjustment factor at the t-th time step, the cross entropy loss between the elastic pseudo label and the prediction result at the t-th time step, and the reverse cross entropy loss; The parameter update module is used to update the parameters of the student model at the t-th time step through the global elastic symmetric cross entropy loss function at the t-th time step, and update the parameters of the teacher model at the t-th time step based on the updated parameters of the student model at the t-th time step. After completing the training of T time steps, the teacher model at the t-th time step with updated parameters is used as the target model.

Images