Performing computer vision tasks

Abstract

The present disclosure relates to a computer-implemented method of performing a computer vision task. The computer-implemented method includes: receiving a corrupted image from a camera; estimating a damage type of the damaged image using a damage identification module; obtaining a normalization parameter associated with the estimated damage type; updating the computer vision model trained to perform the task by replacing normalization parameters of the computer vision model with the obtained normalization parameters; and performing the task using the updated computer vision model.

Description

Technical Field

[0001] This application generally relates to performing computer vision tasks. Background Technology

[0002] Developing reliable vision systems is a fundamental challenge in robotics (e.g., indoor service robots and outdoor autonomous robots) that ensures reliable navigation and guarantees high performance for safety-critical functions even in challenging environments such as severe weather conditions (e.g., fog, rain), poor lighting conditions (e.g., overexposure / underexposure), or sensor degradation (e.g., blurriness, noise). Current solutions for improving model robustness typically rely on general data augmentation techniques or expensive test-time adaptive methods. Furthermore, most approaches focus on utilizing synthetic data to solve a single vision task (typically image recognition).

[0003] The purpose of this invention is to improve upon existing technologies. Summary of the Invention

[0004] According to an embodiment, a method for controlling an electronic device to perform a computer vision task may include receiving a damaged image from a camera, identifying a damage type of the damaged image using a damage identification module, obtaining normalization parameters associated with the estimated damage type from a codebook, updating a computer vision model trained to perform the task by replacing the normalization parameters of a computer vision model with the obtained normalization parameters, and performing the computer vision task using the updated computer vision model.

[0005] The damage identification module may include a Fast Fourier Transform (FFT) model. Identifying the damage type of a damaged image using the damage identification model may include: extracting features from the damaged image; using the FFT model to retain only features that appear at frequencies above a frequency threshold; determining the probability that the image in the input is affected by the damage type associated with the retained features, where the damage type associated with the retained features is a damage type based on the distance distribution between the retained features and pre-trained prototype features of a known set of damage types; and identifying the damage type by selecting the damage type with the highest probability.

[0006] The distance distribution can be a Euclidean L2 distance distribution.

[0007] The damage identification module may include a machine learning model trained to estimate the damage type using a damaged image. Identifying the damage type of a damaged image using the damage identification module may involve feeding the damaged image into the machine learning model to estimate the damage type from the damaged image.

[0008] Machine learning models can be deep neural network models.

[0009] Computer vision models can be neural network models. Normalization parameters can include at least one of batch normalization (BatchNorm) parameters or layer normalization (LayerNorm) parameters.

[0010] The neural network model can be a convolutional neural network model.

[0011] Computer vision tasks can be selected from a list of computer vision tasks that include object detection, object recognition, and semantic segmentation.

[0012] The method further includes generating the codebook by: providing a pre-trained computer vision model and a training dataset, wherein, for each of a plurality of damage types, the training dataset includes a plurality of damaged images and corresponding labels associated with a computer vision task to which the model has been trained; for each damage type, retraining the pre-trained computer vision model using the plurality of damaged images and corresponding labels by updating only the normalization layer of the pre-trained computer vision model; for each damage type, extracting the normalization layer of the retrained computer vision model; and generating the codebook to associate each identifiable damage type with the corresponding normalization layer parameters.

[0013] The normalization layer may include at least one of a BatchNorm normalization layer and a LayerNorm normalization layer.

[0014] The method may include generating a codebook by providing a pre-trained computer vision model, a damaged image, and corresponding damage type labels estimated by a damage identification module; updating the normalization layer of the pre-trained computer vision model based on the damaged image and the corresponding damage type labels using a test-time adaptive algorithm; extracting updated normalization layers for each estimated damage type; and generating a codebook to associate each identifiable damage type with the corresponding normalization layer parameters.

[0015] According to an embodiment, an electronic device for performing a computer vision task may include a memory and at least one processor connected to the memory, wherein the at least one processor is configured to receive a damaged image from a camera, identify the damage type of the damaged image using a damage identification module, obtain normalization parameters associated with the estimated damage type from a codebook, update a computer vision model trained to perform the task by replacing the normalization parameters of the computer vision model with the obtained normalization parameters, and perform the computer vision task using the updated computer vision model.

[0016] The damage identification module may include a Fast Fourier Transform (FFT) model. At least one processor can extract features from the damaged image, using the FFT model to retain only features that appear at frequencies above a frequency threshold, determine the probability that the image in the input is affected by the damage type associated with the retained features, the damage type associated with the retained features being a damage type based on the distance distribution between the retained features and pre-trained prototype features of a known set of damage types, and identify the damage type by selecting the damage type with the highest probability.

[0017] The distance distribution can be a Euclidean L2 distance distribution.

[0018] The damage identification module may include a machine learning model trained to estimate the damage type using a damaged image. At least one processor may input the damaged image into the machine learning model to estimate the damage type from the damaged image. Attached Figure Description

[0019] Implementations of this technology will now be described by way of example only with reference to the accompanying drawings, wherein:

[0020] Figure 1 A robotic vacuum cleaner (RVC) operating in a room is shown according to at least one embodiment;

[0021] Figure 2 The illustration shows an embodiment of at least one of the following: Figure 1 A block diagram of a computer vision system running on RVC;

[0022] Figure 3 The flowcharts for training and inference are shown to determine the type of damage to the input image, which are formed... Figure 2 Part of a computer vision system;

[0023] Figure 4 The diagram shows the application to the source Figure 3 The flowchart shows the clustering algorithm associated with the feature extraction module.

[0024] Figure 5 It shows when the operation comes from Figure 3 Analysis diagram when using a flowchart;

[0025] Figure 6 The summary is shown from Figure 3 The flowchart shows a method for predicting the type of damage in labels and using it to perform computer vision tasks.

[0026] Figure 7 The flowcharts for training and inference are shown to determine the type of corruption in the input image and use it to perform computer vision tasks. Figure 2 Part of a computer vision system;

[0027] Figure 8 The summary is shown from Figure 7 The flowchart shows a method for predicting the type of damage in labels and using it to perform computer vision tasks.

[0028] Figure 9 A flowchart is shown illustrating a method for training a computer vision model using minimal training, based on at least one embodiment, using normalized statistics.

[0029] Figure 10 A summary is shown. Figure 9 Flowchart of the method;

[0030] Figure 11 A flowchart is shown of a method for summarizing and performing test time adaptation to set normalized statistics for a computer vision model according to at least one embodiment;

[0031] Figure 12 A summary is shown. Figure 11 Flowchart of the method;

[0032] Figure 13 It shows the use of Figure 1 The RVC method for performing computer vision tasks (i.e., object detection);

[0033] Figure 14 A method for performing computer vision tasks, namely object detection, using a device is illustrated;

[0034] Figure 15 A flowchart summarizing the methods for performing computer vision tasks is shown;

[0035] Figure 16 A flowchart summarizing the computer implementation of codebook generation is shown;

[0036] Figure 17 A flowchart summarizing the computer implementation of codebook generation is shown;

[0037] Figure 18 A flowchart summarizing the computer implementation methods for controlling robot movement is shown;

[0038] Figure 19 A smart oven 600 according to at least one embodiment is shown;

[0039] Figure 20 A smart washing machine 700 according to at least one embodiment is shown; and

[0040] Figure 21A smart refrigerator 800 according to at least one embodiment is shown. Detailed Implementation

[0041] refer to Figure 1 The RVC 10 operates in room 12. For navigation in room 12, the RVC 10 has at least one sensor 14, such as a camera. The RVC 10 also includes a computer vision system that performs at least one computer vision task, such as object detection, object recognition, semantic segmentation, etc. The RVC 10 also includes a trajectory planner and a control module, respectively configured to generate a trajectory based on the results of the computer vision task and control the drive system of the RVC 10 to move the RVC according to the trajectory.

[0042] RVC 10 includes at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform methods such as capturing (or receiving or acquiring) a damaged image using a camera, estimating (or identifying) the damage type of the damaged image using a damage identification module, acquiring normalized parameters related to the estimated damage type from a codebook, updating a computer vision model trained to perform a task by replacing the normalized parameters of the computer vision model with the acquired normalized parameters, performing the task using the updated computer vision model, determining a trajectory based on the results of the computer vision task, and controlling a drive system to move the robot according to the trajectory.

[0043] Storage devices can be non-transitory computer-readable media. When instructions are loaded onto a storage device, the medium can be temporary, i.e., a download signal.

[0044] exist Figure 1 In room 12, the computer vision task could be to detect and identify objects. For example, the computer vision task could be to identify objects such as chair 16 and table 18. Then, a trajectory planner could set trajectories to avoid collisions with those identified objects.

[0045] refer to Figure 2 The computer vision system 20 is represented as a flowchart showing its main components. The computer vision system 20 includes a damage recognition module 22, a module 24 for obtaining normalized statistics, and a computer vision model 26.

[0046] As can be understood from the following description, the computer vision system 20 includes the following steps: from camera 14 ( Figure 1The system receives a damaged image, estimates the damage type of the damaged image using a damage recognition model 22, obtains normalized parameters associated with the estimated damage type, updates a computer vision model 26 trained to perform the task by replacing the normalized parameters of the computer vision model with the obtained normalized parameters, and performs the task using the updated computer vision model 26. The step of obtaining the normalized parameters can be implemented using code.

[0047] There are two main embodiments involving the damage identification model. In one embodiment, the damage identification module includes a Fast Fourier Transform (FFT). In another embodiment, the damage identification module 22 includes a machine learning model trained to estimate the damage type using a damaged image.

[0048] There are also two main embodiments involving updating the computer vision model 26. In one embodiment, a codebook has been generated by training only the normalized layers of the computer vision model for each type of damage using minimal training. In another embodiment, a codebook has been generated using test-time adaptation, where unlabeled training data is available. The test-time adaptation updates the normalized statistics of the codebook. In either embodiment, at inference time, the codebook is used to identify the normalized statistics associated with the identified damage type.

[0049] refer to Figure 3 This can be understood as the overall pipeline of the method referred to as the FFT-based robust statistical selection method (or simply the FROST method). The FROST method can be associated with the FFT implementation described above. During training, the following are constructed: (i) a corruption-specific prototype using high-frequency FFT features and (ii) corruption-specific feature-normalized statistics. During testing, FFT features are extracted and inference is performed via prototype matching to select the most suitable statistics.

[0050] FROST employs a two-step approach: During training, FROST extracts high-frequency amplitudes from damaged images, aggregates them for images with the same damage, and constructs a set of feature prototypes for each damage. Then, it estimates damage-specific (Corr-S) or damage-specific (Corr-G) normalized layer parameters (normalized statistics) 40 starting from a pre-trained model 32. When the computer vision model is a neural network and can specifically be a convolutional neural network, the normalized layer parameters or normalization parameters can include at least one of batch normalization, BatchNorm, statistics and layer normalization, LayerNorm, and statistics. During testing, FROST identifies the types of damage present in the test images. and using codebook This type of corruption is mapped to the parameters of the normalization layer to minimize identification errors. These normalization parameters... It comes from a broken general model or a specific model, depending on the model's confidence level.

[0051] Background. Given a computer vision model (or model) F, 32, which uses a training set... To approximate samples Real value label In the current embodiment, the corrupted image is... Define, where It has width ,high and damage A clean RGB image. The goal is to improve the accuracy of object recognition for M objects on damaged images.

[0052] Damage. Previous work has shown that actual damage... This can be approximated by synthesizing damaged combinations. (Subset) Contrast, brightness, defocus blur, glass blur, motion blur, zoom blur, impulse noise, shot noise, and Gaussian noise were used to represent the nine most common types of realistic damage. The composite damage of a clean image was achieved through... Defined, so that for (e.g., k = contrast). This parameter It is an integer, whose definition depends on the damage intensity of the degradation level, where the lowest and the highest .

[0053] train. FFT feature extraction. For Each synthetic damage in the process is caused by the damage. Apply to all images To build a collection In this case, only the strongest damage (s=5) is used to obtain better separation between the characteristics of different damages. For each graph with damage k... We perform FFT on the input image A windowing operation is applied to preserve the first n high-frequency components of the amplitude spectrum, and flattening is used to extract FFT features. Specifically, based on experience, n is chosen to be n=15, and the size is adjusted to... Image computation Then, we averaged each set of features specific to damage k to obtain the damage prototype, 36. , where N is the size of the training set. This can be done during training by running an average, without needing to store all features in memory. We call this set... Analyze the feature sets of different types of damage. We noted that some results were well clustered (e.g., contrast, brightness, and defocus blur), while other results (e.g., blur type and noise distortion) were almost inseparable.

[0054] refer to Figure 4 To obtain better clustering scores, we adjust the FFT feature set. Calculate the k-means. This set was initially used with labels associated with damage (from...). Let's define the new label obtained through the k-means (L). Empirically setting the number of clusters for k-means to 5, we obtained cluster scores similar to the original labels. Specifically, if we group fuzzy and noisy impairments together, we obtain new labels. . and Through quantitative analysis and comparison, we obtained an adjusted random score of 89.1% (meaning the clusters are very similar). Therefore, we aggregated prototypes belonging to similar damage characteristics, thus obtaining a new damage set with 5 macro-damages. In addition, we obtained a new set of macro prototypes. It averages prototypes with very similar levels of damage. Note that in real-world scenarios, high-frequency visual content may interfere with frequency content associated with the damage. In these cases, multi-feature or multi-scaling FFT methods can be employed to extend the algorithm.

[0055] I retain S (and it is not) Because S = (, and = )]

[0056] Estimation of specific statistics for corruption. We define it by the set S of statistics estimated at all normalized layers (batch / layer normalization) in the identification model F. These layers are storage-friendly because they have only two parameters (scaling). and shift These two parameters have been shown to adapt differently to input images affected by different types of damage. Therefore, our aim is to use them to improve the accuracy of damaged image recognition. First, we add... Performing data augmentation on clean samples ( )of The model F is trained (the normalization layer is updated only with the S parameters). Image enhancement is selected based on a uniform distribution using the original damage function, for enhancement only of severely damaged images (total K=9), i.e., Through this training, we obtained corrupted generalized normalized statistics. Then, we (That is, damage k is damaged) , only use Train F on the normalization layer (updating the normalization layer only with the S parameters) to produce K distinct sets of normalized statistics specific to the damage. Grouped according to macro corruption We averaged the normalized statistics of indistinguishable damage to obtain A set, each macro corrupts a set.

[0057] infer. During testing, we use prototype features. As a codebook Use the key to select the best set .

[0058] Prototype matching. We match prototypes with unknown damage. Each test image Perform inference. First, we extract features. We retain the first n high-frequency components of the FFT amplitude spectrum. Then, we calculate the probability that the image is damaged by damage k, such that for each damage using L2 distance... , Note that the test image can also be undamaged; we will explain how to handle this situation in the next paragraph.

[0059] This can be represented as follows: the damage identification module includes a Fast Fourier Transform (FFT) model, and estimating the damage type of a damaged image using the damage identification module includes: extracting (or obtaining) features from the damaged image; using the FFT model to retain (or maintain) only features that appear at frequencies above a frequency threshold; determining the probability that the image in the input is affected by the damage type associated with the retained features, the damage type associated with the retained features being a distribution based on the distance between the retained features and pre-computed prototype features of a known set of damage types; and estimating the damage type by selecting the damage type with the highest probability. The distance distribution is a distribution of Euclidean L2 distance.

[0060] Statistical choice. We use probability scores to navigate through our codebook. Choose the most suitable set of normalized statistics and apply it to the model This is to enhance object recognition capabilities. First, we determine whether the damage is uncertain by applying a threshold operation to the two most likely sources of damage. We will... Defined as the most likely and second most likely estimated damage. If Therefore, we use the generalized normalized statistics of damage. Otherwise, we use damage-specific normalized statistics. In this case, T is chosen empirically (by comparing distance values). Note that clean images are typically mapped to... However, they have inherent noise and are sometimes used... This could be beneficial. We noted that in both standard pre-trained models and Corr-G models... In the original model, the damage shares the same normalization parameters. Conversely, each damage in the Corr-S model has its own set of normalization layer parameters, and the aggregation of FROST macro-damage provides a good approximation of it, which is more convenient for damage identification via FFT (see [link to Corr-S model]). Figure 5 ).

[0061] refer to Figure 6 During inference, the FFT method can be summarized as receiving unlabeled data (e.g., images) from the unlabeled memory 44 and extracting features 46 using the FFT as described above. Then, the damage identification module 22 identifies the damage type as described above. Normalized statistics of the identified damage type are retrieved from the damage-specific parameter storage device 48. The normalization layer of the pre-trained computer vision model 26 is replaced with the dedicated parameters from the storage device 48.

[0062] The type of damage is associated with the probability of being true. This probability is compared to a threshold probability as part of the uncertainty estimate 50. If the probability is below the threshold, a pre-trained computer vision model 26 can be used to perform a computer vision task, such as object recognition. If the probability is greater than or equal to the threshold, an updated computer vision model 52 is used for the computer vision task.

[0063] Figure 7 Another embodiment of a damage recognition model that has been trained to identify damage types from damaged images.

[0064] refer to Figure 7 A damage identification module (CIM)22, constructed using a deep neural network, is trained on damaged training images, and a set of damage-related prototype features is constructed by averaging the features z associated with each damage. Then, during inference, CIM 22 was frozen, and the codebook... Damage identified by CIM 22 is mapped to corresponding damage-specific BN parameters. These parameters are defined in the clean source image. Upstream using pre-trained downstream task models The parameters in (computer vision model 32) are initialized, and each identified defect is addressed separately. (Section 3.3) By adapting the TTA to the test image, a set of damaged features can be obtained. Finally, Insert A robust model 58 is generated, which achieves enhanced robustness on downstream tasks, particularly on the identified damage.

[0065] This method is based on the observation that the statistics of BN layers in any convolutional architecture differ significantly for images damaged according to different damage types, but are similar for images with the same damage type. Some previous work explored the adaptation of statistics for normalized layers used for TTA, maintaining a single set of normalized parameters for all damages to construct a universal normalized layer to adapt to any input damage. Instead, we construct a set of multiple normalized statistics estimated for each damage type. Damage-wise adaptive normalization (PAN) consists of three parts:

[0066] 1. A deep neural network module for damage type identification.

[0067] 2. A damage-by-damage adaptive method for making the statistics of the BN layer adapt to various damage types during inference.

[0068] 3. Codebook 56 is used to map the identified corruption types to the corresponding set of BN statistics.

[0069] Image corrupted: Set as a DNN model installed on a robot for visual scene understanding. The goal is to approximate the input image. truth labels This optimizes its learnable parameters. The set of parameters (e.g., the weights and biases of the model's network architecture). Among these parameters, we represent the set of parameters for its BN layer as... From the source domain probability distribution on Extract source (clean) dataset The samples. Then, we consider the target domain. The target (damaged) dataset of sampled distorted images. We distinguish between real (endogenous) and synthetic (exogenous) distortion as follows:

[0070] Endogenous distortion refers to the natural corruption of image statistics caused by inherent noise in the camera sensor, deformation of objects observed in the image, or divergence of object patterns. This is the most general case, where the target test data... It cannot be parameterized by any operator. We represent it as a corrupted set of images. They exhibit the same type of damage (e.g., dark images, where...) (Indicates the type of damage). The distribution of images in the damaged set differs from the distribution of the source images, i.e. .

[0071] Exogenous distortion is a synthetic approximation of true damage, provided by a function of clean images. It is assumed that damage exists given a set of clean images. operators In the case of passing The obtained synthetic damage represents an approximation of real damage, i.e. ,in, This indicates the type of corruption, and s represents the severity of the corruption. When the operator... Based on the corresponding damage type k and severity s, a nonlinear transformation is performed. When transforming the distribution, from Mid-sampling of each damaged set The image.

[0072] CIM 22 was designed using a convolutional encoder and a subsequent linear classifier.

[0073] The architecture of CIM. Extraction of specific damaged features is achieved through a DNN model. Completed, this DNN model is achieved through... A convolutional encoder that projects the input image x onto feature vectors. and linear layer of output damage recognition probability composition.

[0074] Training: CIM 22 performs a damage classification task to identify and approximate damage present in each input image. In the training set... Train the CIM model on the platform, where each It was damaged. (It can be endogenous or exogenous, in which case it is exogenous) A dataset of damaged images, labeled with damage tags to indicate their distortion. Use it to mark. It has a cardinality The set of possible damaged labels. The CIM model is trained end-to-end using a distance-based contrastive training method that utilizes the class-anchor clustering (CAC) loss defined by the following formula:

[0075] (1)

[0076] in It is an input image with its label y, and These are hyperparameters. The CAC loss aggregates two separate losses: i) a tuple loss that minimizes the distance between a training sample and its ground truth anchor class center. , and ii) anchor loss used to maximize distance to the center of other anchoring categories. Therefore, CAC loss Training data is encouraged to form tight, category-specific clusters, and category centers are anchored to fix the cluster center positions during training.

[0077] Inference: In Training CIM model Then, remove the last layer. and use feature extractor Extract q-dimensional features from damaged sample 1 Then, prototype features are computed from the training set. , where each prototype feature Is it related to being damaged? The feature vector corresponding to the damaged image, and It is damaged The number of samples affected. The calculated... Prototype passed Perform cascading to construct the prototype matrix ,in, Represents the transpose of a vector / matrix.

[0078] We employ a distance-based classifier. This classifier classifies features based on their relative distance to prototype features. Output ,in, This represents the Euclidean norm.

[0079] The output is normalized. ,in It is an element-wise product, and

[0080] (2)

[0081] This is used to match features with the closest prototype. Then, the model... The following formula can be used to predict the damage affecting the input:

[0082] (3)

[0083] In this way, CIM 22 includes a machine learning model (e.g., a deep neural network model) that is trained to estimate (or identify) the damage type using a damaged image. Estimating the damage type of a damaged image using CIM 22 involves feeding the damaged image into the machine learning model to estimate the damage type from the damaged image.

[0084] Batch normalization (BN) is a technique used to normalize layer inputs by re-centering and rescaling, thereby making the training of artificial neural networks faster and more stable. It is widely used in DNNs to mitigate the problem of internal covariate bias, where variations in the input distribution of each layer affect the network's learning. BN is applied to 4D inputs (mini-batches of 2D inputs with additional channel dimensions).

[0085] make Indicates the use of the model The obtained features are in small batches, and let This is the feature map in a mini-batch. The mean is used for each dimension in the channel-based mini-batch. and standard deviation (BN statistics), to normalize the features using the following formula,

[0086] (4)

[0087] That middle Let and be learnable affine parameter vectors of size D, and It is a small constant used for numerical stability.

[0088] Test-Time Adaptation (TTA) refers to adapting a DNN to the distribution shift, allowing it to access only data belonging to the target domain during the test period. Unlabeled test samples. The conventional approach to using Batch Normalization (BN) during test time is to... and Those are set to be estimated from the source data. Conversely, the TTA method estimates the BN statistics directly from the test batch to reduce the distribution bias at test time by:

[0089] (5)

[0090] This approach is simple and effective, and has therefore been adopted in many recent TTA studies. In our paper, we propose updating the BN statistics separately via TTA for each damage type, as described below.

[0091] Estimate the statistics of the test data. Let For the model The set of BN statistics. We will represent the set of BN statistics obtained after training the model on the source dataset as follows: We first initialize the source BN parameters. of We have several sets. Then, we update each set based on the types of damage present in the input image. Ideally, each set corresponds to a specific type of damage. They are associated, and each damage type is always correctly identified. Therefore, with that type Related BN statistics Only use those belonging to the test set Types of damage We update using corrupted images. This is used to define an ideal statistical reference set. However, the test images of the target damaged lack damage labels. And must be in accordance with CIM ( The BN parameter is calculated based on the estimated damage type.

[0092] When deployed on robotic equipment, our system includes (i) a CIM module for identifying the type of damage affecting unlabeled input test images, and (ii) training a model on clean training data. The resulting clean set of K BN statistics. Our PAN aims to identify the correct damage type using CIM, update the correct BN parameter set via TTA, and finally insert the updated BN parameter set into the network, thereby improving performance. The performance of downstream tasks.

[0093] Codebook mapping. Specifically, during inference, for each input test image... We use CIM through Estimate the type of damage. Then, we use the codebook with the following formula. For each estimated damage type Mapped to a set of BN statistics that are specifically corrupted. ,

[0094] (6)

[0095] Note that the BN statistic associated with each of the K damages is initialized to... And they will be allocated after being estimated by TTA. The more accurately CIM can identify damage (when...) When the time is right, BN statistics become more specialized for this type of damage and different from others.

[0096] refer to Figure 8 During inference, the deep network approach can be summarized as receiving unlabeled data (e.g., images) from an unlabeled storage device 44. The damage identification model 22 then identifies the damage type as described above. Normalized statistics of the identified damage type are retrieved from a damage-specific parameter storage device 48. The normalization layer of the pre-trained computer vision model 26 is replaced with the dedicated parameters from the storage device 48.

[0097] The damage type is associated with the probability of being true. This probability is compared to a threshold probability as part of the uncertainty estimate 50. If the probability of the most likely damage type is below the threshold, the damage type is marked as unseen. Unseen damage types are added to the damage type storage device 60. The damage type with the highest probability is considered the damage type, and the TTA can be used to update the normalization parameters in storage device 48. The computer vision model is then updated to a dedicated computer vision model 52. The dedicated computer vision model includes the normalization parameters from the storage device 48 associated with the damage types.

[0098] refer to Figure 9 and Figure 10 As described above, the method for training the normalization layer of a pre-trained computer vision model includes retrieving damage type labels and task outcome labels (e.g., object labels when the task is object detection) from a storage device 62 containing damaged images. The method includes providing a pre-trained computer vision model 26. Minimal training is performed on the pre-trained model 26 to train only the normalization layer 40 using backpropagation and an optimization algorithm. For training, task labels for one damage type are used only to optimize the normalization layer of the network. This is then done for all other damage types. The normalization layer for each damage type is then stored in a storage device 48. The normalization parameters in the storage device 48 form a codebook.

[0099] In this way, Figure 9 and Figure 10 The method shown can be summarized as generating (or obtaining) a codebook by: providing a pre-trained computer vision model and a training dataset, wherein for each of a variety of damage types, the training dataset includes multiple damaged images and corresponding labels associated with the computer vision task to which the model has been trained; for each damage type, retraining the pre-trained computer vision model using the multiple damaged images and corresponding labels by updating only the normalization layer of the pre-trained computer vision model; extracting the normalization layer of the retrained computer vision model for each damage type; and generating (or obtaining) a codebook to include the damage type and the corresponding normalization layer.

[0100] refer to Figure 11 and Figure 12 As described above, the method of setting normalized statistics for a computer vision model using TTA includes retrieving estimated damage labels from storage device 64. A pre-trained computer vision model 26 is provided. TTA (as described above) is performed only on the normalization layer 40 of the pre-trained model 26. Normalized statistics (or parameters) are stored in storage device 48.

[0101] Normalized statistics can include the rolling mean and variance. The rolling mean E[xt] can be updated using Equation 7:

[0102] (7)

[0103] Formula 8 can be used to update the variance Var[xt]:

[0104] (8)

[0105] Here, feature prototypes will also be added. Updates exist in both (even if obtained using different CIMs):

[0106] ,in, It is integrating images The average prototype characteristics of the damage type k obtained afterwards, Is Features obtained using CIM, and It is the average prototype characteristic of the damaged type k before the update.

[0107] In this way, Figure 11 and Figure 12 The method described herein can be summarized as follows: generating (or obtaining) a codebook by providing a pre-trained computer vision model, a damaged image, and corresponding damage type labels estimated by the damage recognition model; updating the normalization layer of the pre-trained computer vision model based on the damaged image and the corresponding damage type labels using a test-time adaptive algorithm; extracting (or obtaining) the updated normalization layer for each estimated damage type; and generating (or obtaining) a codebook to include the estimated damage type and the corresponding normalization layer.

[0108] refer to Figure 13 The method of operating RVC 10 can be summarized as including: acquiring 100 images 30, determining whether the images are corrupted 102, and if corrupted, performing computer vision tasks (such as object detection) 104.

[0109] If the image is corrupted, the corruption type is assessed to determine if it is known (106). If not, the corruption type is stored in storage device 60. If yes, the corruption type is used to select the corresponding normalization parameter (108). Next, it is determined (110) whether TTA is required. If yes, the normalization statistics are updated as described above. If no, object detection (104) is performed.

[0110] refer to Figure 14Another use case is summarized, this time a method for operating a user device (e.g., smartphone 70). The method for operating device 70 can be summarized as including acquiring a corrupted image 100. In this case, the image is known to be corrupted, therefore there is no step to determine whether the image is corrupted. The corruption type is used to select the normalization parameter corresponding to 108. Next, it is determined whether 110 requires a TTA. If yes, the normalization statistics are updated as described above. If no, object detection 104 is performed.

[0111] refer to Figure 15 The computer-implemented method for performing a computer vision task can be summarized as follows: receiving a damaged image from a camera 202; estimating the damage type of the damaged image 204 using a damage identification module; obtaining normalization parameters associated with the estimated damage type 206; updating the computer vision model trained to perform the task 208 by replacing the normalization parameters of the computer vision model with the obtained normalization parameters; and performing the task 210 using the updated computer vision model.

[0112] refer to Figure 16 The computer implementation of the codebook generation method can be summarized as follows: providing a 302 pre-trained computer vision model and a training dataset, wherein for each of a variety of damage types, the training dataset includes multiple damaged images and corresponding labels associated with the computer vision task performed by the model; for each damage type, retraining the 304 pre-trained computer vision model using the multiple damaged images and corresponding labels by updating only the normalization layer of the pre-trained computer vision model; extracting the 306 normalization layer of the retrained computer vision model for each damage type; and generating a 308 codebook to include the damage type and the corresponding normalization layer.

[0113] refer to Figure 17 The computer implementation of the codebook update method can be summarized as follows: 402 providing a pre-trained computer vision model, a damaged image, and corresponding damage type labels estimated by the damage recognition module; using a test-time adaptive algorithm, updating the normalization layer of the pre-trained computer vision model based on the damaged image and the corresponding damage type labels; extracting the updated normalization layer for each estimated damage type; and generating the codebook to include the estimated damage types and the corresponding normalization layers.

[0114] refer to Figure 18 The computer implementation method for controlling the robot's movement can be summarized as follows: capturing 502 damaged images using the robot's camera; performing 504 computer vision tasks using the captured damaged images; determining 506 a trajectory based on the results of the computer vision tasks; and moving 508 the robot according to the trajectory.

[0115] Other applications of the aforementioned method were also envisioned.

[0116] For example, refer to Figure 19 According to at least one embodiment, an oven 600 is provided. The oven may be a smart oven. The oven includes a processor and a storage device, similar to other embodiments, such as an RVC. The oven 600 also includes sensors, such as a camera, for example an RGB camera. The processor may implement instructions on the storage device to perform one or more computer-implemented methods, including recommended and optionally executed cooking programs. The cooking program may include a cooking duration, a cooking temperature at which the oven will cook the food, and whether the oven will use a fan or not. While the food is in the oven, the camera captures an image of the food. The image is a corrupted image. The method uses the captured image to perform a computer vision task. The computer vision task may be object recognition, for example, detecting what food is in the oven 600. The method determines the cooking program based on the results of the computer vision task, and may optionally run the cooking program by configuring the oven to cook the food according to the cooking program.

[0117] refer to Figure 20 According to at least one embodiment, a washing machine 700 is provided. The washing machine 700 may be an intelligent washing machine 700. The intelligent washing machine 700 includes a processor and a storage device, similar to other embodiments, such as an RVC. The washing machine 700 also includes sensors, such as a camera, for example an RGB camera. The processor may implement instructions on the storage device to perform one or more computer-implemented methods, including methods for recommending and optionally executing a washing program. The washing program may include water temperature, washing duration, spin cycle, etc. While clothes are in the washing machine, the camera captures images of the clothes. The images are defaced images. The method uses the captured images to perform a computer vision task. The computer vision task may be object recognition, such as identifying clothing material / type, clothing color, clothing soiling / cleanliness, etc. The method determines a washing program based on the results of the computer vision task, and may optionally run the washing program by configuring the washing machine 700 to wash clothes according to the washing program.

[0118] refer to Figure 21According to at least one embodiment, a refrigerator or refrigerator 800 is provided. The refrigerator 800 may be a smart refrigerator, including a processor and a storage device, similar to other embodiments, such as an RVC. The smart refrigerator 800 also includes sensors, such as a camera, for example an RGB camera. The processor may implement instructions on the storage device to execute one or more computer-implemented methods, including methods for recommending shopping lists, listing available items, or recipes that can be cooked with available items. When an item is added to the refrigerator, the method includes the camera capturing an image of the item. The image is a corrupted image. The method will use the captured image to perform a computer vision task. The computer vision task may be object recognition, for example, recognizing food or items entering and leaving the refrigerator. The method will determine a shopping list, listing available items or recipes that can be cooked with available items.

[0119] Those skilled in the art will understand that while the content considered the best mode and other modes of performing the technology where appropriate have been described above, the technology should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiments. Those skilled in the art will recognize that the technology has a wide range of applications and that extensive modifications can be made to the embodiments without departing from any inventive concept defined in the appended claims.

Claims

1. A method for controlling an electronic device for performing a computer vision task, the method comprising: Receive damaged images from the camera; The damage identification module is used to identify the type of damage to the damaged image; Obtain normalized parameters associated with the estimated damage type from the codebook; The computer vision model trained to perform the task is updated by replacing the normalized parameters of the computer vision model with the obtained normalized parameters. as well as The computer vision task is performed using an updated computer vision model.

2. The method according to claim 1, wherein, The damage identification module includes a Fast Fourier Transform (FFT) model, and The damage type used to identify the damaged image using a damage recognition model includes: Extract features from the damaged image; The FFT model is used to retain only features that occur at frequencies above a frequency threshold; Determine the probability that an image in the input is affected by a corruption type associated with the preserved features. The corruption type associated with the preserved features is a corruption type based on the distance distribution between the preserved features and pre-trained prototype features of a known set of corruption types; and Damage types are identified by selecting the damage type with the highest probability.

3. The method according to claim 2, wherein, The distance distribution is the Euclidean L2 distance distribution.

4. The method according to claim 1, wherein, The damage identification module includes a machine learning model trained to estimate the damage type using damaged images. The damage types identified by the damage recognition module include: The damaged image is input into the machine learning model to estimate the damage type from the damaged image.

5. The method according to claim 4, wherein, The machine learning model is a deep neural network model.

6. The method of claim 1, wherein the computer vision model is a neural network model, and in, The normalization parameter includes at least one of the BatchNorm parameter or the LayerNorm parameter.

7. The method according to claim 6, wherein, The neural network model is a convolutional neural network model.

8. The method of claim 1, wherein the computer vision task is selected from a list of computer vision tasks including object detection, object recognition, and semantic segmentation.

9. The method according to claim 1, further comprising: The codebook is generated in the following manner: Provide a pre-trained computer vision model and a training dataset, wherein, for each of a variety of damage types, the training dataset includes multiple damaged images and corresponding labels associated with the computer vision task performed by the model during training; For each damage type, the pre-trained computer vision model is retrained using the multiple damaged images and their corresponding labels by updating only the normalization layer of the pre-trained computer vision model. For each damage type, extract the normalization layer of the retrained computer vision model; and The codebook is generated to associate each identifiable corruption type with the corresponding normalization layer parameter.

10. The method according to claim 9, wherein, The normalization layer includes at least one of a BatchNorm layer and a LayerNorm layer.

11. The method according to claim 1, further comprising: The codebook is generated in the following manner: Provides a pre-trained computer vision model, a damaged image, and corresponding damage type labels estimated by the damage recognition module; The test-time adaptive algorithm is used to update the normalization layer of the pre-trained computer vision model based on the damaged image and the corresponding damage type label. Extract an updated normalization layer for each estimated damage type; as well as The codebook is generated to associate each identifiable corruption type with the corresponding normalization layer parameter.

12. An electronic device for performing computer vision tasks, comprising: Memory; and At least one processor connected to the memory; Wherein, the at least one processor is configured to: Receive damaged images from the camera; The damage identification module is used to identify the type of damage to the damaged image; Obtain normalized parameters associated with the estimated damage type from the codebook; The computer vision model trained to perform the task is updated by replacing the normalized parameters of the computer vision model with the obtained normalized parameters; and The computer vision task is performed using an updated computer vision model.

13. The electronic device according to claim 12, wherein, The damage identification module includes a Fast Fourier Transform (FFT) model, and Wherein, the at least one processor is configured to: Extract features from the damaged image; The FFT model is used to retain only features that occur at frequencies above a frequency threshold; Determine the probability that an image in the input is affected by a corruption type associated with the preserved features. The corruption type associated with the preserved features is a corruption type based on the distance distribution between the preserved features and pre-trained prototype features of a known set of corruption types; and Damage types are identified by selecting the damage type with the highest probability.

14. The electronic device according to claim 13, wherein, The distance distribution is the Euclidean L2 distance distribution.

15. The electronic device according to claim 12, wherein, The damage identification module includes a machine learning model trained to estimate the damage type using damaged images. Wherein, the at least one processor is configured to: The damaged image is input into the machine learning model to estimate the damage type from the damaged image.

Images