Normalized structure reconstruction and feature sharpness perception online adaptation method and related equipment

Abstract

The embodiment of the invention provides a normalized structure reconstruction and feature sharpness perception online adaptation method and related equipment, and belongs to the technical field of transfer learning. The method comprises the following steps: a normalization layer reconstruction stage: obtaining a batch normalization model; generating pseudo data matched with the source domain data distribution by using a data-free distillation technology; replacing a batch normalization layer in the batch normalization model with a grouping normalization layer or a layer normalization layer to form an initial student model; performing knowledge distillation on the initial student model by using the pseudo data to migrate knowledge of the batch normalization model to the reconstructed student model to obtain a normalization layer reconstruction model; an online adaptation stage: obtaining a test sample; screening the prediction confidence of the test sample based on the normalization layer reconstruction model to obtain a reliable sample subset; calculating feature sharpness loss based on the reliable sample subset; and with the purpose of minimizing the feature sharpness loss, updating parameters of the normalization layer reconstruction model to complete online adaptation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and transfer learning, and particularly relates to an online adaptation method of normalized structure reconstruction and feature sharpness perception and related equipment. BACKGROUND

[0002] Deep neural networks (DNNs) have achieved great success in many fields. Batch normalization (BN) as a key component of DNNs significantly accelerates the training process and improves the model generalization performance by reducing the internal covariate shift. BN uses the statistics (mean and variance) of the current batch for normalization during training, and records the running statistics; during testing (inference), the global running statistics obtained during the training phase are fixedly used. This method is based on a strong assumption that the training data and the test data are independent and identically distributed (i.i.d.).

[0003] However, in actual open scenarios (such as autonomous driving and intelligent monitoring), the test environment is complex and variable, and the test data and the training data often have distribution shift (Domain Shift). At this time, the fixed BN running statistics cannot accurately represent the distribution of the test data, which becomes a key bottleneck for the serious decline of the model performance. To address this problem, existing technologies propose test-time adaptation (TTA) or online adaptation methods, which aim to dynamically adjust the model parameters using unlabeled test data to adapt to the target domain.

[0004] Existing TTA methods mainly improve the model robustness by dynamically calibrating the BN statistics, for example: dynamically fusing the current batch statistics and the global statistics, or using instance-level and group-level statistics for normalization. However, these methods have inherent limitations: (1) They are heavily dependent on the quality of the current batch statistics. When there is class imbalance in the test batch, or multiple distribution shifts (such as light, weather, and scene changes) are mixed, the current batch statistics are not representative, leading to inaccurate normalization and performance degradation. (2) They cannot effectively cope with the inference scene of extremely small batch size (such as batch size = 1). In this scenario, the batch statistics lose statistical significance and even introduce noise, leading to unstable model output. (3) They are still essentially dependent on the BN structure, and only use statistical calibration to cope with distribution shift, without fundamentally improving the robustness of the model structure, and the adaptation ability is obviously limited.

[0005] Therefore, there is an urgent need for an online adaptation method that can fundamentally overcome the limitations of the BN layer to ensure the stability and reliability of the model in complex and dynamic environments. SUMMARY

[0006] The main purpose of the embodiments of the present application is to propose a normalization structure reconstruction and feature sharpness perception online adaptation method, electronic equipment, storage medium and program product, aiming at breaking through the bottleneck of insufficient adaptability and stability caused by the dependence of the existing technology on batch normalization structure and local statistics.

[0007] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application proposes a normalization structure reconstruction and feature sharpness perception online adaptation method, which comprises:

[0008] Normalization layer reconstruction stage: obtaining a pre-trained batch normalization model; using a data-free distillation technology, generating pseudo data matching the source domain data distribution; replacing the batch normalization layer in the pre-trained batch normalization model with a group normalization (GN) layer or a layer normalization (LN) layer to form an initial student model; using the pseudo data to perform knowledge distillation on the initial student model to migrate the knowledge of the batch normalization model to the reconstructed student model, obtaining a normalization layer reconstruction model;

[0009] Online adaptation stage: obtaining unlabeled test samples; filtering the prediction confidence of the test samples based on the normalization layer reconstruction model to obtain a reliable sample subset; based on the reliable sample subset, calculating a feature sharpness loss, which is used to represent the decision boundary sharpness of the normalization layer reconstruction model in the feature space; updating the parameters of the normalization layer reconstruction model to minimize the feature sharpness loss, so that the model learns robust feature representation away from the decision boundary in the feature space, thereby improving the robustness of the model in the online adaptation process.

[0010] In some embodiments, the use of a data-free distillation technology to generate pseudo data matching the source domain data distribution comprises:

[0011] Initializing noise data;

[0012] Inputting the noise data into the batch normalization model to calculate the batch statistics generated at each batch normalization layer during the forward propagation process;

[0013] Calculating the statistical matching loss between the batch statistics and the pre-stored running statistics in the batch normalization layer;

[0014] By optimizing the noise data to minimize the statistical matching loss, the optimized pseudo data is obtained, and the batch statistics of the pseudo data match the running statistics.

[0015] In some embodiments, the statistical matching loss is calculated by the following formula:

[0016]

[0017] wherein, l is a layer index, is the mean and variance of the current batch of noise data z at the l-th layer, is the mean and variance of the l-th layer obtained by training the model on the source domain; is the square of L2 norm; ∈ is a small constant.

[0018] In some embodiments, the batch normalization layer is replaced by a group normalization layer or a layer normalization layer, and the learnable parameters γ and β of the batch normalization layer are inherited.

[0019] In some embodiments, the knowledge distillation of the initial student model using the pseudo data comprises:

[0020] calculating the outputs of the batch normalization model and the student model on the same pseudo data;

[0021] updating the parameters of the student model to minimize the difference between the outputs of the batch normalization model and the student model;

[0022] the difference between the outputs is calculated by KL divergence to calculate the distillation loss, and / or by mean square error to calculate the feature layer alignment loss.

[0023] In some embodiments, the prediction confidence of the test samples is screened, comprising:

[0024] calculating the prediction entropy of each test sample;

[0025] samples with prediction entropy lower than a preset entropy threshold are screened as reliable samples to form the reliable sample subset.

[0026] In some embodiments, the feature sharpness loss is calculated, comprising:

[0027] calculating the average entropy loss on the reliable sample subset

[0028] calculating the average entropy loss the gradient with respect to the model feature representation φ;

[0029] constructing a normalized perturbation direction v according to the gradient;

[0030] perturbing the feature φ along the perturbation direction v to obtain a perturbed feature φ';

[0031] calculating the average entropy of the prediction based on the perturbed feature φ' as the feature sharpness loss

[0032] In some embodiments, the perturbation direction v is calculated by the following formula

[0033]

[0034] wherein, is the gradient of the feature representation φ, and |||2 is the L2 norm.

[0035] To achieve the above object, another aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0036] To achieve the above object, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the above method when executed by a processor.

[0037] To achieve the above object, another aspect of the embodiments of the present application provides a computer program product, which comprises a computer program, and the computer program implements the above method when executed by a processor

[0038] The embodiments of the present application at least have the following beneficial effects: the present application provides an online adaptation method of normalization structure reconstruction and feature sharpness perception, an electronic device, a storage medium and a program product, by using data-free distillation technology to generate representative pseudo data before online adaptation of the model, guiding the model to complete the structure conversion from batch normalization to grouped normalization or layer normalization, and building an internally robust adaptation foundation. The present application introduces a stability enhancement mechanism in the online adaptation process, which improves the stability of the model in complex and variable online adaptation scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of the online adaptation method of normalization structure reconstruction and feature sharpness perception provided by the embodiments of the present application;

[0040] Figure 2 is a flowchart of the stable online adaptation method of normalization structure reconstruction and feature sharpness perception provided by the embodiments of the present application;

[0041] Figure 3 is a hardware structure schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0042] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with embodiments of the present application. They are only examples of apparatuses and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0044] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0045] 1) Batch Normalization (abbreviated as BN) is a technique used in deep neural network training to improve the stability of model training and the convergence speed, which alleviates the gradient vanishing / explosion problem by normalizing the distribution of each layer input.

[0046] To cope with the distribution shift in the test phase, the existing technology mainly adjusts the use of batch normalization layer statistics to improve the robustness of the model, which can be summarized into two typical ideas: One method fuses the current batch statistics and the global statistics obtained in the pre-training phase to construct a dynamically weighted mixed statistics for normalization. The weight is self-adaptively adjusted according to the degree of change of the input data distribution. When the distribution changes significantly, it depends more on the current statistics, and when the distribution is stable, it retains more global statistics. Another method uses instance-level or group-level statistics modeling to independently calculate the normalization parameters for a single sample or a group of similar samples, and combines global statistics for calibration, which improves the stability of the batch normalization model to some extent.

[0047] The prior art improves the adaptability of the model to distribution shift to some extent by dynamically adjusting the statistical quantity usage of the normalization layer. However, these methods still have the following limitations when facing complex dynamic scenes in real open environments: 1) The existing methods rely on the statistical quantity estimation of the current input data. When the test data has batch class imbalance, mixed with multiple shifts in the batch, the local statistics are not representative, resulting in inaccurate normalization. For example, in the autonomous driving scene, the vehicle may suddenly enter the rain tunnel environment from the sunny urban road, and multiple factors such as light, weather, and road structure may change (i.e., mixed with multiple shifts in the batch). At this time, the mean and variance calculated based on the current batch cannot accurately reflect the overall distribution characteristics, resulting in inaccurate BN layer statistics, and the model performance decreases sharply; in addition, in the intelligent monitoring system, the camera-captured picture may contain only the background or a few categories for a long time (such as a runner in an empty corridor), causing severe class imbalance within the batch. At this time, the BN statistics are severely biased by the dominant class, and the model cannot correctly normalize the features of the minority class, resulting in degradation of the model's recognition ability for rare categories. 2) The existing methods cannot work stably in the scene of extremely small batch or single sample input. When the batch size is very small, the batch-based statistics lose statistical significance. For example, in the real-time face recognition system on the edge device, a single image or extremely small batch (such as batch size = 1) is often used for inference. At this time, the BN layer cannot effectively calculate the batch mean and variance, resulting in unstable normalization process, even introducing noise, which seriously affects the reliability of the model output. 3) The existing methods are still dependent on the batch normalization structure itself, and their essence is still to cope with distribution shift by calibrating the statistical quantity, rather than improving the robustness from the model structure level. When facing unknown and variable test environments, the adaptability is limited, and the stability is insufficient.

[0048] To solve the above problems, the present application aims to solve the limitations of existing online adaptation techniques in the face of complex dynamic scenarios such as class imbalance in batches, mixed multiple biases in batches, or extremely small batch input scenarios. These limitations include: dependence on local statistics leading to normalization misalignment, inability to work stably under small batch or single sample input, and dependence on the batch normalization (BN) structure itself limiting the robustness of the model. To overcome the above technical bottlenecks, the embodiments of the present application propose a stable online adaptation method based on data-free distillation and normalization structure reconstruction and related equipment, which fundamentally solves the problem of BN model normalization statistics misalignment in the adaptation process from the model structure level, and improves the stability of online adaptation through feature sharpness minimization, effectively dealing with challenging scenarios such as class imbalance in batches, mixed multiple biases, and extremely small batch input. The method includes two working modes: 1) normalization layer reconstruction mode without data distillation, which reconstructs the normalization layer from the model structure level to improve the intrinsic robustness of the model; 2) online adaptation mode based on feature sharpness minimization, which learns robust feature representation away from the decision boundary in the feature space to improve the robustness of the model during adaptation.

[0049] Specifically, the normalization layer reconstruction mode without data distillation uses data-free distillation technology to generate pseudo data from a pre-trained batch normalization (BN) model and guide the model to complete the structural conversion from the BN layer to the group normalization (GN) layer or the layer normalization (LN) layer. This method does not require access to real source domain data, but only relies on the internal statistics of the pre-trained BN model. By optimizing the noise data to match the running statistics of the pre-trained BN model, pseudo sample proxies similar to the source domain data are generated. After obtaining the pseudo data, the BN layer is replaced by the GN or LN layer, and the learnable parameters of the BN layer are inherited. Then, knowledge distillation is performed using pseudo data to transfer the knowledge of the teacher model (pre-trained BN model) to the student model (normalized layer reconstructed model) by minimizing the loss between the outputs of the teacher model and the student model, while preserving its decision boundary. This mode improves the intrinsic robustness of the model from the model structure level, laying the foundation for subsequent online adaptation.

[0050] An online adaptation method based on feature sharpness minimization, which aims to filter out reliable samples and make the model learn a more robust feature representation away from the decision boundary in the feature space, thereby improving its robustness to test data perturbations and the stability of online adaptation. This method first calculates the entropy value of the unlabeled test sample, and filters out the "reliable" sample subset by setting an entropy threshold. On the filtered reliable samples, the gradient of the entropy loss with respect to the feature representation is calculated, and the gradient is used to construct a perturbation direction. By finding the maximum step size in the perturbation direction, the sharpness of the decision boundary is quantified. Finally, the average entropy loss after perturbation is taken as the feature sharpness loss, and the model parameters are updated with this loss as the optimization target. By minimizing this loss, the model learns a robust feature representation away from the decision boundary in the feature space, thereby improving the robustness of the model during adaptation.

[0051] The present application innovatively proposes a data-free distillation normalization structure reconstruction, which improves the robustness of the model to unknown domain shifts from the model structure level, and further enhances the stability of online adaptation through feature sharpness minimization. This scheme not only solves the problem of limited adaptation capability of existing technologies in complex dynamic scenarios and small batch inputs, but also overcomes the bottleneck brought by traditional batch normalization structure, providing a new solution for stable deployment of models in real open environments.

[0052] The online adaptation method of normalization structure reconstruction and feature sharpness perception provided by the embodiments of the present application relates to the field of transfer learning. The online adaptation method of normalization structure reconstruction and feature sharpness perception provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a standalone physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; the software can be an application that implements an online adaptation method of normalization structure reconstruction and feature sharpness perception, and the like, but is not limited to the above forms.

[0053] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0054] Figure 1 is an optional flowchart of a normalization structure reconstruction and feature sharpness perception online adaptation method provided by an embodiment of the application, Figure 1 The method in the method can include but is not limited to including steps S1 to S2.

[0055] Step S1, normalization layer reconstruction stage: This stage is executed before model deployment, aiming to build an inherently robust base model.

[0056] Specifically, step S1 specifically includes steps S11-S14:

[0057] S11, obtaining a pre-trained batch normalization (BN) model as a teacher model.

[0058] S12, using a data-free distillation technology to generate pseudo data matching the source domain data distribution. Specifically, it includes: initializing noise data; inputting the noise data into the teacher model, calculating the batch statistics generated in each BN layer during the forward propagation process; calculating the statistical matching loss between the batch statistics and the pre-stored running statistics in the BN layer; obtaining the optimized pseudo data by optimizing the noise data to minimize the statistical matching loss.

[0059] S13, replacing the BN layer in the teacher model with a group normalization (GN) layer or a layer normalization (LN) layer that is not sensitive to batch size to form an initial student model, and inheriting the learnable parameters (scale and shift parameters γ and β) of the original BN layer.

[0060] S14, knowledge distillation is performed on the initial student model by using the pseudo data to transfer the knowledge of the teacher model to the reconstructed student model, to obtain a normalization layer reconstruction model. The knowledge distillation loss includes a KL divergence loss of output logits and a mean square error loss of intermediate layer features.

[0061] Step S2, online adaptation phase: this phase is performed in real time when facing the test data stream after model deployment.

[0062] Specifically, step S2 specifically includes the following steps S21-S24:

[0063] S21, obtaining unlabeled test samples.

[0064] S22, filtering the prediction confidence (such as entropy) of the test samples based on the normalization layer reconstruction model, filtering the samples with an entropy value lower than a preset threshold as reliable samples to form a reliable sample subset.

[0065] S23, calculating a feature sharpness loss based on the reliable sample subset. Specifically, it includes: calculating the average entropy loss on the reliable sample subset; calculating the gradient of the average entropy loss with respect to the model feature representation, and constructing a normalized perturbation direction based on the gradient; perturbing the feature along the perturbation direction; calculating the average entropy of the prediction corresponding to the perturbed feature, which is taken as the feature sharpness loss.

[0066] S24, updating the parameters of the normalization layer reconstruction model to complete online adaptation and improve the robustness of the model to the current test environment.

[0067] Next, the scheme of the embodiments of the present application will be described and explained in detail in combination with the drawings and specific application examples:

[0068] As shown in the drawings, Figure 2 The embodiments of the present application propose a normalization structure reconstruction and feature sharpness perception based stable online adaptation method without data distillation, which aims to break through the bottleneck of insufficient adaptability and stability caused by the dependence of the existing technology on batch normalization structure and local statistics. The method contains two working modes: 1) normalization layer reconstruction mode without data distillation, which reconstructs the normalization layer from the model structure level to improve the intrinsic robustness of the model; 2) online adaptation mode based on feature sharpness minimization, which learns robust feature representation away from the decision boundary in the feature space to improve the robustness of the model in the adaptation process.

[0069] (1) Normalization layer reconstruction mode without data distillation

[0070] This method proposes a technique to generate pseudo-data from pre-trained batch normalized (BN) models using data-free distillation and then distill grouped normalized (GN) / layer normalized (LN) models, aiming to improve the robustness of online adaptation at the model structure level. This method does not require access to real source domain data and operates solely on the internal statistics of the pre-trained BN model. The algorithm includes the following steps:

[0071] Step 1: Initialize noise and teacher model. Use the pre-trained Batch Normalization (BN) model as the teacher model f. teacher It contains multiple batch normalization (BN) layers, each with runtime statistics obtained from training the model on the source domain data. and (l represents the layer index); the initial noise z0~N(0,I) has a shape of (B,C,H,W), where B is the size of the synthesis batch. This step prepares the noise as the basis for pseudo-data generation and loads the BN statistic of the teacher model as a constraint prior.

[0072] Step 2: Optimize the pseudo-data to match the BN statistic. Specifically, pseudo-data optimization includes the following steps:

[0073] Step 2.1: For each BN layer l, calculate the batch mean of the noise z using the teacher model forward propagation. and batch variance Where B is the batch size, z l Activate the input of layer l.

[0074] Step 2.2: Define the statistical matching loss L stats This is used to minimize the difference between batch statistics and runtime statistics.

[0075]

[0076] in, ∈ is the square of the L2 norm, and ∈ is a small constant (e.g., 1e-5) to avoid numerical instability.

[0077] Step 2.3: Define the optimization objective as follows:

[0078]

[0079] The SGD optimizer is used to iterate over the noise z (considered a learnable parameter) to be optimized. This optimization ensures that the pseudo-data simulates the low-order moment distribution of the source domain in the feature space, i.e., μ. batch (z * )≈E source [x], This generates pseudo-sample proxies that are similar to the source domain.

[0080] Step 3, Replace the normalization layer and initialize the student model. Specifically, the normalization layer replacement includes the following procedures:

[0081] Step 3.1, For each BN layer, replace the student model with GN or LN and inherit the learnable parameters γ and β of BN. The original normalization formula of BN is:

[0082]

[0083] Step 3.2, If replaced by GN: divide the channels into G groups and independently normalize within each group:

[0084]

[0085] where, C g = C / G, initial γ and β are the same as BN.

[0086] Step 3.3, If replaced by LN: normalize the entire layer (channel and spatial dimensions):

[0087]

[0088] where, Initial γ and β are the same as BN.

[0089] Step 4, Knowledge distillation using pseudo data. Specifically, knowledge distillation includes the following procedures:

[0090] Step 4.1, For each pseudo data batch z * , calculate the logit output of the teacher and student: logit t = f teacher (z * ), logit s = f student (z * ).

[0091] Step 4.2, Define the distillation loss function as:

[0092]

[0093] where, is the KL divergence, which measures the difference between two probability distributions; τ is the temperature parameter, which is used to soften the softmax distribution.

[0094] Step 4.3, Add feature-level alignment loss where φ l is the activation feature of the l-th layer, is the mean square error.

[0095] Step 4.4, Total loss:

[0096]

[0097] Using the SGD optimizer based on The student model weights are updated while preserving the decision boundary of the teacher model. At the same time, the GN / LN weights of the student model are adjusted to adapt to pseudo-data, thereby achieving knowledge transfer without data.

[0098] Step 5: Repeat steps 1-4 to generate more pseudo-data for knowledge distillation until the student model performance is stable, at which point the method stops.

[0099] The grouped normalization (GN) or layered normalization (LN) models obtained after the above reconstruction no longer rely on batch statistics for normalization operations, but instead perform independent normalization based on channel or intra-group statistics within a single sample. Therefore, based on the reconstructed GN / LN models, independent normalization of each sample can be achieved, effectively solving the performance degradation problem caused by inaccurate batch statistics in traditional BN models in scenarios such as real-time inference of edge devices (e.g., batch size=1), highly imbalanced categories in surveillance videos (e.g., containing only background and rare pedestrians), and mixed scenarios with multiple offsets in batches due to synchronous changes in lighting, weather, and scene in autonomous driving.

[0100] (2) Online adaptation mode based on feature sharpness minimization

[0101] This paper proposes an online adaptation method based on feature sharpness minimization. Its goal is to select reliable samples and enable the model to learn more robust feature representations in the feature space that are far from the decision boundary, thereby improving its robustness to test data perturbations and the stability of online adaptation. This method effectively avoids potential model crashes during online adaptation in complex dynamic environments (such as the three scenarios mentioned above) by precisely controlling the samples used for adaptation and the optimization objective. The detailed steps of this method are as follows:

[0102] Step 1: Given a batch of unlabeled test samples This method first obtains the feature representation φ of each sample through the forward propagation of the model. i Then, the feature φ i The classification head input to the model In the process, the original output prediction is obtained. in Denotes the classifier, θ h Its parameters. Based on the prediction result, the entropy value E of each sample is calculated. i :

[0103]

[0104] where C is the total number of classes, is the predicted probability that sample i belongs to class c.

[0105] Step 2, To avoid the negative impact of noisy and uncertain samples on online adaptation, a reliable sample screening mechanism is introduced. A pre-defined entropy threshold E0 is set, and only those samples with entropy values lower than the threshold are kept. These samples are usually the ones that the model can make predictions with high confidence, and are considered as "reliable". The subset of screened samples is denoted as where All subsequent adaptation and optimization steps are only performed on the subset .

[0106] Step 3, On the subset of screened reliable samples , the average original entropy loss is calculated.

[0107]

[0108] Step 4, The feature gradient is calculated and a perturbation direction is constructed. For the subset of screened reliable samples , the entropy loss is calculated. The gradient with respect to the feature representation φ is This gradient vector indicates which direction in the feature space to move slightly, which will make the model's prediction uncertainty increase the fastest. A normalized perturbation direction v is constructed using this gradient:

[0109]

[0110] Step 5, To quantify the sharpness of the decision boundary, the method finds a maximum step size s in the perturbation direction v, such that after moving the feature φ to φ' = φ + s·v in the feature space, the model's predicted class remains unchanged. The step size s directly reflects the stability of the region.

[0111] Step 6, The maximum step size s and the perturbation direction v determined in step 5 are applied to the original feature φ of the reliable samples, obtaining the feature φ' = φ + s·v located on the edge of the decision boundary. Then, φ' is input into the classification head to obtain the perturbed prediction Finally, we calculate the average entropy loss after perturbation as the feature sharpness loss

[0112]

[0113] Step 7, The sharpness loss calculated in step 6 is used as the optimization target to update the model's parameters θ using the backpropagation mechanism. The formal representation is:

[0114]

[0115] wherein, θ represents the normalized layer learnable parameters of the model. By minimizing the loss, the model learns robust feature representation far away from the decision boundary in the feature space, thereby improving the robustness of the model in the adaptation process. The above steps are repeatedly performed until all test data is processed and the model stops online adaptation.

[0116] In summary, the present application aims to solve the performance degradation and insufficient stability of existing online adaptation techniques when facing complex dynamic scenarios such as class imbalance in batch, mixed multiple biases in batch, or extremely small batch input scenarios, due to the dependence on batch normalization structure and local statistics. To this end, the present application proposes a normalization structure reconstruction and feature sharpness perception stable online adaptation method without data distillation, electronic equipment, storage medium and program product, which contains two working modes: 1) Normalization layer reconstruction mode without data distillation: This mode uses data-free distillation technology to reconstruct the pre-trained batch normalization model into a group normalization or layer normalization model that is not sensitive to batch size from the model structure level. This fundamentally solves the limitations of existing methods that cannot work stably in batch class imbalance, mixed multiple biases in batch, or extremely small batch input scenarios. 2) Online adaptation mode based on feature sharpness minimization: This mode filters high-confidence samples and makes the model learn robust feature representation far away from the decision boundary in the feature space, effectively improving the robustness of the model in the adaptation process and improving the model's adaptation ability and stability in complex and variable environments.

[0117] In summary, the present application innovatively applies data-free distillation technology to normalization layer reconstruction, improving the intrinsic robustness of the model from the model structure level, and enhances the stability of the model in the online adaptation process through feature sharpness minimization. This method breaks through the dependence of existing batch normalization statistics calibration-based test-time adaptation techniques on batch normalization statistics, solves the adaptation bottleneck in small batch and complex dynamic scenarios, and enables the model to have stronger generalization ability and practicality in the open world. Compared with existing technologies, at least the following advantages and beneficial effects are included:

[0118] 1) Fundamental structural innovation: Through data-free distillation and normalization layer reconstruction, the model is converted from BN structure to GN or LN structure, completely eliminating the dependence on test batch statistics, and fundamentally solving the problem of batch normalization statistics misalignment in extremely small batch, batch class imbalance, mixed bias, etc. scenarios.

[0119] 2) Strong adaptability and high stability: An online adaptation mechanism based on feature sharpness minimization is introduced. By filtering reliable samples and allowing the model to learn robust feature representations far from the decision boundary in the feature space, the robustness of the model to distribution shifts and input perturbations is significantly enhanced, effectively avoiding the collapse problem that may exist when the model adapts online in a complex dynamic environment (such as the three scenarios described above).

[0120] 3) High practicality and security: The entire process does not require access to raw training data, only using the pre-trained model itself and unlabeled test data, reducing the risk of data privacy leakage and meeting the deployment needs of actual industrial applications.

[0121] 4) Performance significantly improved: This method provides an effective technical solution for the stable and reliable deployment of models in complex dynamic environments in the open world, with significant performance advantages and wide applicability.

[0122] The embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0123] It can be understood that the contents in the above method embodiments are applicable to the device embodiments. The device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0124] Please refer to Figure 3 , Figure 3 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0125] The processor 301 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0126] The memory 302 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 302 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 302 and are called and executed by the processor 301 to perform the above-mentioned methods of the embodiments of the present application.

[0127] The input / output interface 303 is configured to realize information input and output.

[0128] The communication interface 304 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0129] The bus 305 is configured to transmit information between various components (for example, the processor 301, the memory 302, the input / output interface 303, and the communication interface 304) of the device.

[0130] The processor 301, the memory 302, the input / output interface 303, and the communication interface 304 are connected to each other through the bus 305 to realize the communication connection between the device.

[0131] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned method.

[0132] It can be understood that the contents in the above-mentioned method embodiments are applicable to the present storage medium embodiments. The functions implemented by the present storage medium embodiments are the same as those of the above-mentioned method embodiments, and the beneficial effects achieved by the present storage medium embodiments are also the same as those achieved by the above-mentioned method embodiments.

[0133] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0134] The embodiment of the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the method described above.

[0135] It can be understood that the contents in the method embodiments described above are all applicable to the program product embodiments, the program product embodiments specifically implement the functions same as those of the method embodiments described above, and achieve the same beneficial effects as those of the method embodiments described above. The executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0136] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It can be understood by those skilled in the art that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0137] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps or different steps.

[0138] The device embodiments described above are merely schematic, and the units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0139] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0140] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but is used to connect like elements or to distinguish one claim from another. These terms can be used interchangeably when appropriate. Terms concerning the relative position of elements can be interpreted such that their use adheres to their normal meaning, but they can also be interpreted to mean the opposite according to specific claims.

[0141] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0142] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the above-described device embodiments are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0143] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the application.

[0144] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0145] If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0146] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. An online adaptive method for normalized structure reconstruction and feature sharpness perception, characterized in that, The method includes the following steps: Normalization layer reconstruction stage: Obtain a pre-trained batch normalization model; use data-free distillation technology to generate pseudo-data that matches the distribution of source domain data; replace the batch normalization layer in the pre-trained batch normalization model with a group normalization layer or a layer normalization layer to form an initial student model; use the pseudo-data to perform knowledge distillation on the initial student model to transfer the knowledge of the batch normalization model to the reconstructed student model, thus obtaining the normalization layer reconstruction model; Online adaptation phase: Obtain unlabeled test samples; filter the prediction confidence of the test samples based on the normalized layer reconstruction model to obtain a reliable sample subset; calculate the feature sharpness loss based on the reliable sample subset, the feature sharpness loss is used to characterize the decision boundary sharpness of the normalized layer reconstruction model in the feature space; update the parameters of the normalized layer reconstruction model with the goal of minimizing the feature sharpness loss.

2. The method according to claim 1, characterized in that, The method of generating pseudo-data that matches the distribution of source domain data using data-free distillation includes: Initialize noise data; The noise data is input into the batch normalization model to calculate the batch statistics generated at each batch normalization layer during the forward propagation process. Calculate the statistical matching loss between the batch statistics and the pre-stored runtime statistics in the batch normalization layer; By optimizing the noisy data to minimize the statistical matching loss, optimized pseudo-data is obtained, and the batch statistics of the pseudo-data are matched with the running statistics.

3. The method according to claim 2, characterized in that, The statistical matching loss Calculated using the following formula: Where l is the layer index, Let z be the mean and variance of the noise data z in the current batch at layer l. These are the mean and variance of the l-th layer obtained by training the model in the source domain. ∈ is the square of the L2 norm; ∈ is a small constant.

4. The method according to claim 1, characterized in that, The batch normalization layer is replaced with a group normalization layer or a layer normalization layer, and the learnable parameters γ and β of the batch normalization layer are inherited.

5. The method according to claim 1, characterized in that, The step of using the pseudo-data to perform knowledge distillation on the initial student model includes: Calculate the outputs of the batch normalization model and the student model for the same pseudo-data; The parameters of the student model are updated with the goal of minimizing the difference between the output of the batch normalization model and the student model. The difference between the outputs is calculated using KL divergence to determine distillation loss, and / or using mean square error to determine feature layer alignment loss.

6. The method according to claim 1, characterized in that, The process of filtering the prediction confidence of the test samples includes: Calculate the prediction entropy for each test sample; Samples whose predicted entropy is lower than a preset entropy threshold are selected as reliable samples, forming the reliable sample subset.

7. The method according to claim 1, characterized in that, The calculation of feature sharpness loss includes: Calculate the average entropy loss on the reliable sample subset. Calculate the average entropy loss The gradient of the model feature representation φ; A normalized perturbation direction v is constructed based on the gradient; The feature φ is perturbed along the perturbation direction v to obtain the perturbed feature φ. ′ ; Calculate the characteristic φ based on the perturbation ′ The obtained average entropy of the prediction is used as the feature sharpness loss.

8. The method according to claim 7, characterized in that, The disturbance direction v is calculated using the following formula. in, Let φ be the gradient of the feature, and ||||2 be the L2 norm.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images