Out-of-distribution detection system and method based on channel attention and cascade early exit

Abstract

The present invention discloses an out-of-distribution detection system and method based on channel attention and cascade early exit. The system includes a neural network structure. By setting a TinyOOD detector on the output side of each convolutional layer of the neural network structure and introducing an NMD model based on a channel attention mechanism, namely CA-NMD, into the TinyOOD detector, the NMD vector can be used to represent the result of the squeezing step, which reduces the computational consumption. At the same time, the channel attention mechanism takes into account the importance of different channels and has better performance. Therefore, the lighter out-of-distribution detection system is more widely used. In the detection method, by connecting multiple TinyOOD detectors and transmitting CA-NMD vectors, the input of the logistic regression detection model can be refined. At the same time, by adding TinyOOD dynamic decision-making, the task can be uploaded to the cloud for execution without increasing the local device inference delay, which can greatly improve the inference accuracy.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to an out-of-distribution detection system and method based on channel attention and cascade early exit mechanism. Background Art

[0002] Tiny machine learning (TinyML) is a rapidly growing field encompassing research and development at the intersection of embedded systems and machine learning. TinyML aims to bring machine learning inference to microcontroller units (MCUs) that consume very little power (~1mW), are inexpensive (~$1), and are very small (~1cm³). Recently, TinyML has gained widespread adoption on ubiquitous embedded devices that process sensor data in real time.

[0003] However, in the complex and ever-changing real world, the input data received by a TinyML application may differ from the training data. These input samples are called out-of-scope (OOD) samples. These OOD samples can make TinyML applications unreliable and insecure. Therefore, the ability to detect OOD samples is crucial for the deployment of TinyML applications.

[0004] Currently, many methods have been developed for OOD detection. One of them is the likelihood ratio method for deep generative models, which can effectively detect data instances far from the training distribution. ODIN adds small perturbations to the input and smoothes the softmax score to separate the softmax score distributions of in-distribution and out-of-distribution inputs. However, these methods usually incur a lot of overhead, which is unacceptable for TinyML. To solve this problem, standard off-the-shelf models already contain enough data distribution information of the training dataset. Based on this observation, technicians proposed the neural mean difference (NMD) metric, which can use the model hidden layer information for reliable OOD detection. Although the NMD method reduces the resource consumption of OOD detection, it still relies on the output of the last layer and requires a complete forward reasoning for any given input. The processing process is complex and time-consuming. Summary of the Invention

[0005] In view of the above problems, the purpose of the present invention is to provide an out-of-distribution detection system and method based on channel attention and cascade early exit to achieve the purpose.

[0006] To achieve the above-mentioned objectives, the present invention provides an out-of-distribution detection system based on channel attention and cascade early exit, including a neural network structure, wherein the neural network structure includes multiple convolutional layers, and the output side of each convolutional layer is connected to a TinyOOD detector, and the multiple TinyOOD detectors are respectively connected to a cascade early exit branch, and the multiple TinyOOD detectors are connected sequentially.

[0007] Furthermore, the TinyOOD detector includes a CA-NMD model and a logistic regression prediction model.

[0008] Furthermore, the CA-NMD model applies the channel attention mechanism to the original NMD model.

[0009] Another aspect of this embodiment is to provide an out-of-distribution detection method for the above-mentioned out-of-distribution detection system, comprising the following steps:

[0010] S1: Sample input neural network structure;

[0011] S2: Sample features extracted by the convolutional layer are input into the TinyOOD detector;

[0012] S3: TinyOOD detector operates on the input sample through the CA-NMD model and obtains the CA-NMD vector;

[0013] S4: The logistic regression prediction model predicts whether the input sample type is an out-of-distribution sample through the CA-NMD vector and the CA-NMD vector previously passed to the current convolutional layer;

[0014] S5: TinyOOD detector performs TinyOOD dynamic decision making.

[0015] Furthermore, the method for calculating the CA-NMD vector in step S3 includes the following steps:

[0016] S31: Squeeze step, squeeze the global spatial information of the channel into the channel descriptor;

[0017] S32: Excitation step, fully capturing the channel dependency and defining the CA-NMD vector.

[0018] Furthermore, the squeezing step is used to squeeze the global spatial information into a channel descriptor, and the channel descriptor is represented as follows:

[0019]

[0020] Among them, l is the number of current convolution layers, C is the number of channels, z l is the channel descriptor vector.

[0021] Furthermore, the excitation step is to use a two-layer fully connected network to capture the dependencies between different channels under the same convolutional layer. The calculation formula is:

[0022] s l =σ(W2δ(W1z l ));

[0023] Among them, s l represents the channel attention weight in layer l, δ refers to the ReLU function, σ refers to the sigmoid function, and are the parameters of the two fully connected networks, and r is the reduction ratio;

[0024] Finally, the CA-NMD vector of layer l can be defined as:

[0025] CA-NMD l =s l z l .

[0026] Furthermore, in step S5, the TinyOOD detector detects whether the current sample is an out-of-distribution sample by:

[0027] Get the input of the logistic regression prediction model at the current layer Specifically:

[0028]

[0029] Among them, l is the number of current convolution layers;

[0030] The logistic regression prediction model includes the prediction score And threshold γ, the judgment result of TinyOOD detector on the current input sample is:

[0031]

[0032] Among them, l is the ordinal number of the TinyOOD detector corresponding to the current number of convolution layers. Then the current input sample is an out-of-distribution sample, otherwise it is an in-distribution sample.

[0033] Furthermore, the TinyOOD dynamic decision is based on the Q-Learning learning method. The TinyOOD dynamic decision is determined by the state space, action space and reward function, and produces the following results: continue execution, exit early and upload to the cloud;

[0034] The state space is the operating state of the model, which is represented by a one-dimensional vector and is discretized into multiple different states, represented as: S = {1, 2, 3, 4, ..., N};

[0035] The action space includes decisions of continuing execution, exiting early, and uploading to the cloud, which is represented by: A = {continue execution: 0, exit early: 1, upload to the cloud: 2}, where the continuing execution decision includes local execution and cloud execution;

[0036] The reward function is composed of parameters: OOD score (scores ood ), local model inference accuracy (scores local ), cloud model inference accuracy (scores cloud ) and the normalized local elapsed inference time (T local_used ), the remaining time of local reasoning (T local_unused ), cloud inference time (T cloud ), the reward function is specifically:

[0037]

[0038] Among them, α1, β1, α2, β2, α3 and β3 are hyperparameters used to weigh the sum of the weights of each parameter.

[0039] Furthermore, the TinyOOD dynamic decision-making method in step S5 includes:

[0040] S51: If the TinyOOD detector detects that the current sample is an out-of-distribution sample, it executes the early exit decision, otherwise goes to step S52;

[0041] S52: Execute the decision to continue executing the current sample;

[0042] S53: Repeat S2-S5 until it reaches the last TinyOOD detector. If the TinyOOD detector detects that the current sample is an out-of-distribution sample, it executes the early exit decision; otherwise, it executes the upload to the cloud decision and performs classification output.

[0043] The beneficial effects of the present invention are:

[0044] 1. By setting the TinyOOD detector at the output side of each convolutional layer and introducing an NMD model based on the channel attention mechanism, namely CA-NMD, into the TinyOOD detector, the NMD vector can be used to represent the result of the squeezing step, reducing computational consumption. At the same time, the channel attention mechanism takes into account the importance of different channels and has better performance. Therefore, this lighter out-of-distribution detection system can be deployed in shallow networks and has a wider range of applications.

[0045] 2. By connecting multiple TinyOOD detectors and passing the CA-NMD vector, the input of the logistic regression detection model can be refined, which can effectively improve the accuracy of OOD detection.

[0046] 3. By adding TinyOOD dynamic decision-making based on Q-Learning reinforcement learning, it is possible to upload tasks to the cloud for execution without increasing the inference latency of local devices, which can greatly improve the inference accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings illustrate exemplary embodiments of the present invention and together with the description serve to explain the principles of the present invention. These drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification.

[0048] Figure 1 This is a structural block diagram of the distributed detection system of this embodiment;

[0049] Figure 2 This is a schematic diagram of the TinyOOD detector structure in this embodiment;

[0050] Figure 3 This is a flow chart of the out-of-distribution detection method of this embodiment;

[0051] Figure 4 Schematic diagram of the calculation method of the CA-NMD vector in this embodiment;

[0052] Figure 5 This is a flow chart of the TinyOOD dynamic decision-making method in this embodiment;

[0053] Figure 6 This is a schematic diagram of the out-of-distribution detection algorithm of this embodiment;

[0054] Figure 7 Schematic diagram of the Q-Learning algorithm of this embodiment. DETAILED DESCRIPTION

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the relevant content and are not intended to limit the present invention. It should also be noted that, for ease of description, only portions relevant to the present invention are shown in the accompanying drawings.

[0056] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0057] Example:

[0058] refer to Figure 1 and Figure 2 An out-of-distribution detection system based on channel attention and cascade early exit mechanism includes a neural network structure, which can be any off-the-shelf neural network structure. The neural network structure includes multiple convolutional layers, and the output side of each convolutional layer is connected to a TinyOOD detector. The multiple TinyOOD detectors are respectively connected to a cascade early exit branch. The multiple TinyOOD detectors are connected sequentially. The TinyOOD detectors include a CA-NMD model and a logistic regression prediction model. The connected TinyOOD detectors can pass the CA-NMD vector from the previous layers to the current convolutional layer to refine the input of the logistic regression prediction model, which can improve the accuracy of OOD detection. Among them, the CA-NMD model applies the channel attention mechanism to the original NMD model. NMD is a neural mean difference metric that can use the model hidden layer information for reliable OOD detection. After the channel attention mechanism is applied to NMD, the importance of different channels can be considered, which has better performance and is conducive to deployment in shallow networks such as MCUs. At the same time, the cascade early exit branch allows the model to exit early when an OOD sample is detected.

[0059] Among them, the NMD model is used to compare the neural means of the input sample and the training sample through calculation. The specific calculation method is as follows:

[0060]

[0061] in, is the training data set, I={x1,...,x |I|} is a batch of input samples from an unknown distribution, is the output of channel c in layer l, d is the spatial size of the activation map, and (m,n) represents the spatial position.

[0062] This embodiment also discloses an out-of-distribution detection method, which is applied to the above-mentioned out-of-distribution detection system to achieve lightweight but accurate OOD detection.

[0063] refer to Figure 3 、 Figure 4 、 Figure 5 and Figure 6 , an out-of-distribution detection method based on channel attention and cascade early exit mechanism, comprising the following steps:

[0064] S1: Sample input neural network structure;

[0065] S2: Sample features extracted by the convolutional layer are input into the TinyOOD detector;

[0066] S3: TinyOOD detector operates on the input sample through the CA-NMD model and obtains the CA-NMD vector;

[0067] S4: The logistic regression prediction model predicts whether the input sample type is an out-of-distribution sample through the CA-NMD vector and the CA-NMD vector previously passed to the current convolutional layer;

[0068] S5: TinyOOD detector performs TinyOOD dynamic decision making.

[0069] refer to Figure 4 ,The channel attention mechanism consists of the squeeze channel attention mechanism and the ,stimulation channel attention mechanism. Specifically, the calculation method of the CA-NMD vector in step S3 includes the following steps:

[0070] S31: Squeeze step, squeeze the global spatial information of the channel into the channel descriptor;

[0071] S32: Excitation step, used to fully capture the channel dependency and define the CA-NMD vector.

[0072] In step S31, the squeezing step is used to squeeze the global spatial information into the channel descriptor. The result of the squeezing step is represented by the NMD vector, which can effectively reduce the computational cost of the squeezing step. Specifically, the global spatial information of all channels C in layer l is compressed into the channel descriptor vector z l , which is expressed as:

[0073]

[0074] Among them, l is the number of current convolution layers, C is the number of channels, z l is the channel descriptor vector.

[0075] In step S32, the motivation step is to use a two-layer fully connected network to capture the dependencies between different channels in the same convolutional layer. The calculation formula is:

[0076] s l =σ(W2δ(W1z l ));

[0077] Among them, s l represents the channel attention weight in layer l, δ refers to the ReLU function, σ refers to the sigmoid function, and are the parameters of the two fully connected networks, and r is the reduction ratio;

[0078] Finally, the CA-NMD vector of layer l can be defined as:

[0079] CA-NMD l=s l z l .

[0080] In step S4, the TinyOOD detector uses a logistic regression prediction model to detect whether the current sample is an out-of-distribution sample. The detection method is:

[0081] Get the input of the logistic regression prediction model at the current layer Specifically:

[0082]

[0083] Among them, l is the number of current convolution layers;

[0084] The logistic regression prediction model includes the prediction score And threshold γ, the judgment result of TinyOOD detector on the current input sample is Specifically:

[0085]

[0086] Among them, l is the ordinal number of the current convolution layer corresponding to the TinyOOD detector, if Then the current input sample is an out-of-distribution sample, otherwise it is an in-distribution sample. When the sample is an out-of-distribution sample, the detection process of the out-of-distribution detection system exits early, otherwise the current CA-NMD vector is passed to the next TinyOOD detector.

[0087] TinyOOD dynamic decision-making is based on the Q-Learning learning method. TinyOOD dynamic decision-making is determined by the state space, action space and reward function. Specifically:

[0088] The state space is the running state of the model, which is represented by a one-dimensional vector and is discretized into multiple different states. For example, the model of this embodiment includes a seven-level TinyOOD detector, and the state space is represented as: S = {1, 2, 3, 4, 5, 6, 7}, where S1 to S5 are the running states of each hidden layer of the local model, S6 is the early exit state, and S7 is the upload to the cloud state.

[0089] The action space is a set of execution decisions, including continue execution, exit early, and upload to the cloud. Action a is used to describe the decision of reasoning task execution, which is expressed as: A = {continue execution: 0, exit early: 1, upload to the cloud: 2}, where the continue execution decision also includes local execution and cloud execution. For different states S, the action space may be different. For example, for state S5, its action space is A(S5) = {exit early: 1, upload to the cloud: 2}, because the S5 state is at the bottom of the local model in this embodiment, and only the exit and upload to the cloud actions can be taken.

[0090] The reward function is composed of parameters: OOD score (scores ood ), local model inference accuracy (scores local ), cloud model inference accuracy (scores cloud ) and the normalized local elapsed inference time (T local_used ), the remaining time of local reasoning (T local_unused ), cloud inference time (T cloud ) is composed of. The reward function in this embodiment is an instant reward function. The goal of constructing the reward function is to achieve the minimum time consumption and the highest reasoning accuracy. Therefore, the reward value of the reward function needs to be negatively correlated with the time consumption value and positively correlated with the reasoning accuracy value. The reward function in this embodiment is specifically:

[0091]

[0092] Among them, α1, β1, α2, β2, α3 and β3 are hyperparameters used to weigh the sum of the weights of each parameter.

[0093] refer to Figure 5 Based on the above-mentioned content-based TinyOOD dynamic decision-making, the TinyOOD dynamic decision-making method in step S5 includes:

[0094] S51: If the TinyOOD detector detects that the current sample is an out-of-distribution sample, it executes the early exit decision, otherwise goes to step S52;

[0095] S52: Execute the decision to continue executing the current sample;

[0096] S53: Repeat steps S2-S5 until the last TinyOOD detector is reached. If the TinyOOD detector detects that the current sample is an out-of-distribution sample, it executes the early exit decision; otherwise, it executes the upload to the cloud decision and outputs the score.

[0097] Therefore, based on the reward function, for the local execution decision and cloud execution decision in the continue execution decision, the performance of the cloud server is much higher than that of the local device. Transferring local tasks to the cloud device for execution has higher processing efficiency. However, uploading tasks to the cloud on a large number of devices will cause network communication congestion, which may eventually lead to a sharp increase in the time taken for inference tasks. Therefore, the task time is weighed by the difference between the cloud execution time and the local execution time. Depending on different inference tasks, the task time also varies greatly. For example, assuming that the cloud execution time is less than the local execution time, in theory, cloud execution should be prioritized. However, we also consider whether the sample is an out-of-distribution sample, that is, if the local model believes that the current sample is likely to be an OOD sample, but has not reached the level of early exit, in this case, uploading the task to the cloud may increase system consumption. If the cloud time is extremely small at this time, cloud execution is chosen.

[0098] For the decision to exit early and upload to the cloud, tasks with high OOD scores will be terminated early, and the local inference time will be defined as additional time consumption as a penalty item. At the same time, the decision to upload to the cloud also means that the local inference time will be used as a penalty item.

[0099] refer to Figure 7 Q-Learning is an effective model-free reinforcement learning method. For this method, the current environment and state transition probabilities are not clear and even time-varying. Therefore, the Q-Learning algorithm attempts to enable the agent to automatically learn the best behavior in the specific context of each time step. The Q-Learning method is used to solve the TinyOOD dynamic decision-making problem. In particular, the Q-Learning method can estimate the optimal action value Q(s, a) of the state and the allowed action pairs at each time step. These values are stored or updated in the Q table.

[0100] It should be understood by those skilled in the art that the above embodiments are merely for the purpose of illustrating the present invention clearly, and are not intended to limit the scope of the present invention. For those skilled in the art, other changes or modifications may be made based on the above invention, and these changes or modifications are still within the scope of the present invention.

Claims

1. An out-of-distribution detection method based on channel attention and cascade early exit, characterized in that The out-of-distribution detection method based on channel attention and cascade early exit is applied to the out-of-distribution detection system. The out-of-distribution detection system includes a neural network structure, the neural network structure includes multiple convolutional layers, the output side of each convolutional layer is connected to a TinyOOD detector, the multiple TinyOOD detectors are respectively connected to a cascade early exit branch, and the multiple TinyOOD detectors are connected sequentially; The TinyOOD detector includes a CA-NMD model and a logistic regression prediction model; The CA-NMD model applies the channel attention mechanism to the original NMD model; The out-of-distribution detection method includes the following steps: S1: Sample input neural network structure; S2: Sample features extracted by the convolutional layer are input into the TinyOOD detector; S3: TinyOOD detector operates on the input sample through the CA-NMD model and obtains the CA-NMD vector; S4: The logistic regression prediction model predicts whether the input sample type is an out-of-distribution sample through the CA-NMD vector and the CA-NMD vector previously passed to the current convolutional layer; S5: TinyOOD detector performs TinyOOD dynamic decision making; The TinyOOD dynamic decision is based on the Q-Learning learning method. The TinyOOD dynamic decision is determined by the state space, action space and reward function, and produces results: continue execution, exit early and upload to the cloud.

2. The out-of-distribution detection method according to claim 1, characterized in that: The method for calculating the CA-NMD vector in step S3 includes the following steps: S31: Extrusion step, extruding the global spatial information of the channel into the channel descriptor; S32: Excitation step, fully capturing the channel dependency and defining the CA-NMD vector.

3. The out-of-distribution detection method according to claim 2, characterized in that: The squeezing step is used to squeeze the global spatial information into the channel descriptor, and the channel descriptor is represented as follows: Among them, l is the number of current convolution layers, C is the number of channels, z l is the channel descriptor vector.

4. The out-of-distribution detection method according to claim 3, characterized in that: The motivation step is to use a two-layer fully connected network to capture the dependencies between different channels in the same convolutional layer. The calculation formula is: s l =σ(W2δ(W1z l )); Among them, s l represents the channel attention weight in layer l, δ refers to the ReLU function, σ refers to the sigmoid function, and are the parameters of the two fully connected networks, and r is the reduction ratio; Finally, the CA-NMD vector of layer l is defined as: CA-NMD l =s l z l 。 5. The out-of-distribution detection method according to claim 1, characterized in that: In step S4, the TinyOOD detector detects whether the current sample is an out-of-distribution sample by: Get the input of the logistic regression prediction model at the current layer Specifically: Among them, l is the current number of convolutional layers; The logistic regression prediction model includes the prediction score And threshold γ, the judgment result of TinyOOD detector on the current input sample is: Among them, l is the ordinal number of the TinyOOD detector corresponding to the current number of convolution layers. Then the current input sample is an out-of-distribution sample, otherwise it is an in-distribution sample.

6. The out-of-distribution detection method according to claim 1, characterized in that: The state space is the operating state of the model, which is represented by a one-dimensional vector and is discretized into multiple different states, represented as: S = {1, 2, 3, 4, ..., N}; The action space includes decisions of continuing execution, exiting early, and uploading to the cloud, which is represented by: A = {continue execution: 0, exit early: 1, upload to the cloud: 2}, where the continuing execution decision includes local execution and cloud execution; The reward function consists of parameters: OOD scores scores ood , local model inference accuracy scores local , cloud model inference accuracy scores cloud and the normalized local elapsed inference time T local_used , the remaining time of local reasoning T local_unused , cloud inference time T cloud The reward function is specifically: Among them, α1, β1, α2, β2, α3 and β3 are hyperparameters used to weigh the sum of the weights of each parameter.

7. The out-of-distribution detection method according to claim 6, characterized in that: The decision-making method of the TinyOOD dynamic decision in step S5 includes: S51: If the TinyOOD detector detects that the current sample is an out-of-distribution sample, it executes the early exit decision, otherwise goes to step S52; S52: Execute the decision to continue executing the current sample; S53: Repeat S2-S5 until it reaches the last TinyOOD detector. If the TinyOOD detector detects that the current sample is an out-of-distribution sample, it executes the early exit decision; otherwise, it executes the upload to the cloud decision and performs classification output.

Images