distance between distributions belonging to intra-distribution measures for images

Abstract

The present invention relates to processing input data by a neural network. Some embodiments provide a method and apparatus for processing the input data by at least one layer in the neural network and obtaining a feature tensor from the input data. Then, a distribution of the obtained feature tensor is estimated. Other distributions are obtained. Such other distributions can be distributions of other input data or distributions obtained by combining a plurality of distributions obtained from respective input data. Then, a distance value representing a distance between the two distributions is calculated and a characteristic of the input data is determined based on the distance value. The characteristic can be related to a reliability of data or distribution out of a certain class detection or class determination.

Description

Technical Field

[0001] The embodiments of this application generally relate to the field of neural networks and processing performed through neural networks. Background Technology

[0002] Neural networks have recently been applied to various data classification tasks. With improvements in hardware architecture, some neural networks have been deployed for purposes such as image or video classification or recognition.

[0003] A neural network (NN) is a signal processing model that supports machine learning. This model mimics the human brain, consisting of multiple interconnected neurons. In neural network implementations, the signal at the connection between two neurons is a number, and the output of each neuron is calculated using a nonlinear function of its weighted input sum. These connections are called edges. Neurons and edges typically have weights that adjust as learning progresses. Weights increase or decrease the signal strength at the connection. The nonlinear function of the weighted sum is also called the "activation function" or "neuron's transfer function." In some simple implementations, the output can be binary, depending on whether the weighted sum is greater than a certain threshold, corresponding to a step function as a nonlinear activation function. In other implementations, other activation functions, such as sigmoid, can be used. Typically, neurons are clustered in layers. Different layers can perform different transformations on their inputs. Signals are transmitted from the first layer (input layer) to the last layer (output layer), possibly traversing multiple layers. Weights are learned through training, which can be performed through supervised or unsupervised learning. It should be noted that the above model is only a general model. For specific applications, neural networks can have different processing stages, which can correspond to convolutional neural network (CNN) layers and be suitable for the desired input, such as an image.

[0004] Feedforward neural networks are a type of artificial neural network in which connections between nodes do not form loops. These types of NNs are commonly used for data classification tasks. In deep neural networks (DNNs), the adjective "deep" in deep learning comes from the multiple layers of the network. Deep learning is a class of machine learning algorithms that use multiple layers to progressively extract high-level features from the raw input. For example, in image processing, lower layers might identify edges, while higher layers might identify human-relevant concepts such as numbers, letters, or faces.

[0005] During DNN training, a labeled dataset of (x, y) pairs is used, where x is the model's input (e.g., an image) and y is the label of that input category. Training a DNN is a process of searching for optimal parameters in a parameterized model f(θ,·), such that...

[0006]

[0007] Here, L is the loss function (loss). In the case of a classification task, f(x) is decomposed into g°h(x) (meaning f(x) is in the form g(h(x))), where h(x) is the feature vector (embedding) obtained for the input, and g(h(x)) is the similarity vector between the feature vector h(x) of each class y and the average feature vector c_y (centroid). The similarity is the value of the similarity function between h(x) and c_y.

[0008] Closed-set scenarios are tasks that require identifying the same category during both the training and production (inference) phases. A typical and commonly used loss function in closed scenarios is the cross-entropy (CE) loss function. This loss trains the model so that the distance between each input feature and the centroid of its class is less than the distance to any other centroid. For closed-set scenarios, both functions h and g are used during production.

[0009] CE (Complete Similarity Analysis) is not suitable for open-set tasks (i.e., tasks that need to identify new categories during production) because open-set tasks require not only well-separated known categories but also the preservation of positions for new categories in the feature (feature vector) space. In this case, open-set scenarios might only retain the function h. In other words, the function h that estimates similarity during training can be replaced by a modified or extended function that estimates the similarity between feature vectors not fixed to predefined categories. The function h can be omitted entirely, so the output during the inference phase can simply be feature vectors. The function h can also be simplified to distinguish only some key categories.

[0010] The acquired features can be used outside the model, depending on the context. For production, a different dissimilarity function can be applied than that used during training. Specifically, the classes used in training need not be used in production. Even a single example of a new class (e.g., x_2) is sufficient to create a new centroid. Accordingly, the similarity function is typically computed between h(x) and h(x_2), rather than between h(x) and some c_y. Therefore, two (or more) objects are directly compared.

[0011] Regardless of the use case, training process, or model architecture, DNNs share a common problem stemming from limited training datasets. This problem is that the model tends to predict similar feature vectors of images without considering semantics.

[0012] (1) Images have similar pixel RGB value distributions and / or have the same mid-level semantic features (which may be unrelated to the features that are actually used for identification);

[0013] (2) The images are far from the common distribution of the training dataset.

[0014] Several well-known examples confirm this problem. First is MNIST (Y. LeCun, C. Cortes, and Ch. J. C. Burges' "The MNIST Database of Handwritten Digits," available at http: / / yann.lecun.com / exdb / mnist / )—a classic closed-set task for digit recognition. At the beginning of the neural network era, the MNIST model achieved an accuracy of over 99.5%. Nevertheless, even the best solutions suffer from the same problem: because there are no examples with the peak value "6" in the training dataset, the model cannot predict the correct digit if the peak value 6 is input.

[0015] For example, low false-positive rate (Low FPR) face recognition (FaceRec) can be considered. This is an open-set task because everyone must be identified from among people captured by surveillance cameras. Open datasets suitable for training Face Rec models mainly consist of celebrity images collected from the internet. Only a few of these are older or younger celebrities. Therefore, Face Rec models typically do not handle young or older faces well. Summary of the Invention

[0016] This application provides apparatus and methods that can improve the performance of input data, similar to rare examples in the training dataset distribution.

[0017] The foregoing and other objectives are achieved through the subject matter claimed in the independent claims. Other implementations are apparent from the dependent claims, the specification, and the drawings.

[0018] According to one aspect, the present invention relates to a method for processing first input data using a neural network. The neural network is a feedforward neural network. The method includes: processing the first input data through at least one layer of the neural network to obtain a first feature tensor; estimating a first distribution of the obtained first feature tensor; obtaining a second distribution; determining a distance value representing the distance between the first distribution and the second distribution; and determining characteristics of the first input data based on the determined distance value.

[0019] The distance value representing the distance between the first distribution and the second distribution can be any value related to distance. For example, the distance value can be any dissimilarity or similarity measure. Distance is generally considered a measure of dissimilarity because an increase in distance indicates an increase in dissimilarity. However, distance can also be considered a similarity indicator, but with the opposite sign / proportion, indicating that the higher the distance, the lower the similarity. Therefore, the distance value here can typically be obtained, for example, by calculating a dissimilarity measure (or generally called an index) or a similarity measure (or generally called an index), since the results of both measures represent the distance between distributions.

[0020] This approach improves performance on input data, particularly rare examples in the training dataset distribution. Furthermore, the additional evaluation of the first tensor distribution provides flexibility for any pre-training, open-set, or closed-set task.

[0021] In one exemplary embodiment, estimating the first distribution includes: obtaining the number n of value intervals for the first feature tensor value; and for each of the n value intervals, determining the number of times the value belonging to the value interval appears in the first feature tensor value.

[0022] This can be considered as discretizing the tensor distribution into n intervals (ticks). This discretization provides scalable complexity. The number n can be chosen based on the application and considerations including complexity (the number of computations required) and accuracy. For example, n can be adjustable.

[0023] For example, obtaining n value intervals includes: determining the number n based on the dimension of the first feature tensor.

[0024] Adjusting n in this way allows for feasible accuracy in terms of the amount of available data and the meaningful size and distribution of the intervals to which that data is allocated.

[0025] Alternatively, obtaining the n value intervals may include: determining the length of the value interval based on the entropy of the first feature tensor value.

[0026] Adjusting the width of the interval can achieve an efficient distribution of the interval length relative to the corresponding tensor eigenvalues, for example, making the information content of each interval approximately the same.

[0027] According to one embodiment, in determining the characteristics of the first input data, the characteristic is at least one of the following: a category of a plurality of predetermined data categories and whether the first input data belongs to a category of the predetermined data categories.

[0028] Additional processing, including evaluating the distribution, can improve the first data processing, which can be open set classification, closed set classification, or out-of-distribution classification.

[0029] According to an exemplary implementation, obtaining the second distribution includes: processing the second input data through the at least one layer to obtain a second feature tensor; estimating the second distribution of the obtained second feature tensor; and determining the characteristics of the first input data includes: determining the similarity between the first input data and the second input data through a similarity metric, wherein the similarity metric is a function of the distance value.

[0030] This allows for open-set classification, which involves comparing input data with other input data and constructing possible new categories / clusters.

[0031] For example, the similarity measure is also a function of the feature tensor obtained by processing the first input data through all layers of the neural network and / or the feature tensor obtained by processing the second data through all layers of the neural network.

[0032] This embodiment can improve classification tasks. In other words, according to an embodiment of the invention, the characteristics of the neural network output can be supplemented by evaluating a distribution function or a certain tensor. The results can be combined, i.e., by applying weights or some conditions based on a threshold.

[0033] Alternatively or alternatively, the function is a monotonically increasing function of the distance value.

[0034] These properties enable consistent evaluation of neural network-based tasks based on distribution similarity.

[0035] Specifically, in one exemplary implementation, the function ψ is given by ψ(s,d)=p1(s)+w·p2(min(d,Θ)), where p i (x)=x ai , where ai is a positive integer, i is 1 or 2, w is a predetermined weight factor, and Θ is a predetermined maximum distance.

[0036] This exemplary function monotonically increases with distance d and is easy to compute.

[0037] For example, the second distribution is obtained by averaging multiple distributions determined from corresponding input data belonging to the same category.

[0038] This feature enables the comparison of distribution functions with out-of-distribution detections, which enable rare input data detection and / or corresponding actions, including estimating the reliability of neural network classifications.

[0039] According to one possible implementation, the characteristics of the first input data include: comparing the distance value with a threshold; and estimating the reliability of the first input data belonging to the same category based on the comparison result.

[0040] The classification reliability provides information that can correctly interpret the results of neural network processing and / or provides training quality.

[0041] For example, the distance value is calculated based on the Hellinger distance.

[0042] Hellinger distance is a metric particularly well-suited for comparing distributions.

[0043] Specifically, in one exemplary implementation, the distance value is calculated by approximating the Hellinger distance as the sum of squares projected onto a space reduced by applying Principal Component Analysis (PCA).

[0044] This approach can reduce the complexity of additional processing required to evaluate the distribution.

[0045] The method may further include: determining the at least one layer in the neural network as the layer that provides the maximum classification accuracy for the output feature tensor.

[0046] This feature provides the possibility of adapting the evaluated tensor to a specific task. In other words, this feature can find the layer of the desired feature that best distinguishes between classifications.

[0047] For example, the first input data is image data including c channels, where c is an integer equal to or greater than 1.

[0048] Channels can be color channels and / or depth channels and / or motion channels, or other channels. Processing more channels can improve processing accuracy because additional information is being processed.

[0049] In one embodiment, the steps of processing the first input data, estimating the first distribution of the obtained first feature tensor, and determining the distance value are performed separately for each channel c; the determination of the characteristics of the first input data is performed by aggregating the distance values ​​determined for each channel.

[0050] Firstly, processing individual channels separately allows for better extraction of their respective characteristics. Then, combining the contribution rates of these channels can improve the accuracy of the evaluation.

[0051] According to one embodiment, any of the above methods are used for face recognition.

[0052] Face recognition, especially open-set recognition, is a challenging task. The embodiments described above are well-suited for improving the reliability estimation of any face recognition method for rare inputs such as the elderly or children, which may not be frequently present in the training dataset.

[0053] According to one aspect, a signal processing apparatus is provided that processes first input data through a neural network. The neural network is a feedforward neural network, and the signal processing apparatus includes a processing circuit for: processing the first input data through at least one layer of the neural network to obtain a first feature tensor; estimating a first distribution of the obtained first feature tensor; obtaining a second distribution; determining a distance value representing the distance between the first distribution and the second distribution; and determining characteristics of the first input data based on the determined distance value.

[0054] According to one aspect, a signal processing apparatus is provided that processes first input data through a neural network. The neural network is a feedforward neural network, and the signal processing apparatus includes: a neural network for processing the first input data through at least one layer of the neural network to obtain a first feature tensor; a distribution estimator for estimating a first distribution of the obtained first feature tensor; an input for acquiring a second distribution; a distance determination unit for determining a distance value representing the distance between the first distribution and the second distribution; and a feature determination unit for determining features of the first input data based on the determined distance value.

[0055] According to one aspect, a computer-readable storage medium storing instructions is provided. When the instructions are executed, the instructions cause one or more processors to decode video data. The instructions cause the one or more processors to perform any of the methods described above. The storage medium may be a non-transitory medium.

[0056] According to one aspect, the present invention relates to a computer program comprising program code. When the program code is executed on a computer, the program code is used to perform any of the methods described above or any of the possible embodiments described below.

[0057] The following drawings and description illustrate one or more embodiments in detail. Other features, objectives, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0058] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings, in which:

[0059] Figure 1 It is a schematic diagram of the functional blocks and signal flow used to determine the discrete distribution of features;

[0060] Figure 2 This is a schematic diagram of the function blocks and signal flow used to determine the modified similarity index;

[0061] Figure 3A A flowchart of a method provided for one embodiment;

[0062] Figure 3B This is a block diagram of the functional structure of a device provided in one embodiment;

[0063] Figure 4 This is a schematic diagram of function blocks and signal flows used to distinguish between ordinary inputs and distributed external inputs;

[0064] Figure 5 This is a block diagram illustrating an example of a device;

[0065] Figure 6 These are illustrative representations of three different cumulative distribution functions;

[0066] Figure 7 This is a schematic diagram illustrating the weak separation of correct and incorrect dataset pairs;

[0067] Figure 8 This is a schematic diagram of the improved separation of correct and incorrect dataset pairs;

[0068] Figure 9 This is a schematic diagram illustrating a specific example of an image from a rare dataset that was detected as outside the distribution;

[0069] Figure 10A This is a schematic diagram of a neural network architecture used for closed-set image recognition;

[0070] Figure 10B This is a schematic diagram of the inference part of a neural network architecture used for open set image recognition;

[0071] Figure 10C This is a schematic diagram of the training portion of a neural network architecture used for open set image recognition.

[0072] Unless otherwise explicitly stated below, the same reference numerals refer to the same features or at least functionally equivalent features. Detailed Implementation

[0073] In the following description, reference is made to the accompanying drawings, which form part of this invention, and which illustrate by way of description specific aspects of embodiments of the invention or aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention may be used in other aspects and may include structural or logical variations not depicted in the drawings. Therefore, the following detailed description is not to be construed as limiting, and the scope of the invention is defined by the appended claims.

[0074] For example, it should be understood that the disclosure relating to the described method also applies to the corresponding device or system for performing the method, and vice versa. For instance, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units perform one or more of a plurality of steps respectively), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, if a specific apparatus is described according to one or more units (e.g., functional units), the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or multiple steps perform the function of one or more of a plurality of units respectively), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it should be understood that, unless otherwise expressly stated, features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0075] According to one embodiment, input data is processed by a neural network including at least one intermediate layer. Tensors of predetermined layers are analyzed against the input data to obtain a discrete distribution function of the tensor features. This distribution function can then be used as the basis for further processing.

[0076] For example, for open-set or closed-set classification, the distribution of the input data tensor can be compared with the distributions of other data obtained in a similar manner. In the case of closed-set classification, each category can be associated with a representative distribution of the tensor. Comparing the distribution of the input data tensor with the representative tensor distributions corresponding to the category produces an indicator (indicator) that conforms to that category.

[0077] Calculating the distribution and comparing the distribution, or accordingly acquiring the characteristics of the input data, can be considered as additional processing relative to the normal operation of a neural network. The results of this additional processing can be used in a variety of different ways.

[0078] As detailed below, the distributions obtained for two input data sets (their feature tensors) can be used to determine distance values, which can then be used to modify the results of the neural network's normal operation. For example, a neural network (NN) might output similarities to some predefined categories. By comparing the input data tensor distribution with the tensor distributions of representative categories, the category with the highest similarity can be determined. This category might differ from the category determined by the NN. Therefore, the classification can be modified based on the results of additional processing. Similarly, for open-set classification, the input data tensor distribution can be compared with the tensor distributions of other previously input data to obtain distance values. These distance values ​​can be used to modify the similarity of the NN's output feature vectors before comparing their similarity. This can improve the model's key metrics, such as accuracy. The resulting similarity function can improve test results on large, representative production test sets.

[0079] Additional processing can be used to or contribute to the formation of new categories, or to assign input data to previously formed categories. Additional processing can be used to estimate the accuracy of the NN results or to detect out-of-distribution input data, and so on. In other words, the tensor distribution does not necessarily replace the neural network output, which is actually used for classification. Besides the general classification capabilities of neural networks, tensor distribution comparisons are also advantageously used. For example, a weighted sum of the two results can be taken, and / or an indication obtained through tensor distribution comparisons can be used to represent the reliability of the classification, and so on.

[0080] In addition to or as a substitute for improved classification (especially for data types rare in the training set), the tensor distribution of the input data can be used to determine whether the input data deviates from a predetermined distribution (or the degree to which the input data deviates from a predetermined distribution) (e.g., the mean of the distribution of the training set data or previously produced input data). This approach is advantageous when it is necessary to estimate how far the input deviates from the common distribution of the training dataset (i.e., whether it is an out-of-the-distribution (OOD) example). This also aids in uncertainty estimation or confidence estimation.

[0081] One possible improvement to embodiments of the present invention addresses the problem of how to improve existing solutions to enhance their performance on rare examples in terms of training dataset distribution. The present invention specifies additional processing that can be integrated with any existing (feedforward) neural network to improve its performance. This additional processing is independent of the training phase. Therefore, the additional processing can be deployed in production on any device using the neural network.

[0082] It should be noted that this invention is not necessarily aimed at improving the model's primary speed metrics, such as accuracy, True-Positive Rate (TPR), and F1 score, in all scenarios. For example, OOD estimation is used to understand when a prediction (classification) can be trusted and / or the degree of trust. Out-of-distribution (or reliability) measurements can be viewed as a result of better distinguishing rare training set examples.

[0083] According to one embodiment, a method for processing first input data using a neural network is provided. For example, the neural network (NN) is a feedforward neural network. The neural network can be a convolutional NN that can be applied to the input data (i.e., image data), such as... Figure 1 and Figure 2 As shown. Figure 1 and Figure 2 A specific exemplary implementation of the above method is shown. Figure 1 In this context, the first input image 101 can represent the first input data.

[0084] The above methods are as follows Figure 3A As shown, step 310 involves processing the first input data through at least one layer in the neural network to obtain a first feature tensor. This step can be achieved through... Figure 1 Steps 110_1...110_K up to the final layer (a fully connected layer in this example) 120 are implemented, as detailed below. Specifically, processing the first input data through one or more layers in a neural network can be understood as extracting features from the first input data; for example, if the first data is image data, image features are extracted. The output of one or more NN layers can be called feature channels.

[0085] The above method also includes estimating the first distribution of the first feature tensor obtained in step 320. This step can be implemented through steps 140_1 to 145_n, 150, and 155. Then, step 330 is performed: obtaining the second distribution. An example of this step can be implemented through step 160, representing the same pipeline described in steps 110_1 to 155.

[0086] The method described above also includes step 340 (corresponding to steps 230_1 to 230_C, etc.): determining a distance value representing the distance between the first distribution and the second distribution. The distance value can be, for example, a similarity or dissimilarity metric, as these all represent distance values ​​(associated with distance values).

[0087] Finally, the method described above includes step 350: determining the characteristics of the first input data based on the determined distance value. An example of this step can be implemented via step 250. Determining the characteristics can be, for example, classification, such as determining a category among multiple closed or open categories. However, the invention is not limited to classification. The determination described above can include determining whether the first input data is out-of-distribution data or ordinary data, or determining the confidence level that the first input data belongs to a category detailed below.

[0088] Contrary to the previous method, refer to Figure 3A The described embodiment achieves the following effects. The goal of this embodiment is to improve the model's fundamental objectives (e.g., accuracy, True Positive Ratio (TPR), fixed False Positive Ratio (FPR), F1 score, etc.). Instead of filtering out suspicious data, it identifies it more effectively. This method requires no special training process but can use any training method. Therefore, it can be implemented in the production stage of any pre-trained model. Even OOD detection can be performed without any additional training. The parameters of the above method can be adjusted on the terminal device. Furthermore, it can be used for both closed-set and open-set scenarios, as shown below. The implementation of the above method, when running with the adjusted parameters, may only slightly increase computation time.

[0089] Figure 1 and Figure 2 This example illustrates a Convolutional Neural Network (CNN). A CNN takes a 3D tensor of size 101 as input. In this example, the 3D tensor is an image with width W, height H, and C components (channels). The C components in this example could be three color components: red (R), green (G), and blue (B). However, in general, components can belong to color spaces other than RGB, such as YUV, YCbCr, or other color spaces. Furthermore, components do not necessarily have to be color components; they can include different components, such as depth maps or other feature images, or combinations of color and feature maps. Components are often referred to as channels.

[0090] in other words, Figure 1 The dimensions of the tensor in the image are called height (H) (the height of the image in pixels), width (W) (the width of the image in pixels), and number of channels (C) (RGB images have 3 channels). The terms "sample" and "pixel" are used interchangeably here. In the case of images and videos, samples of data are sometimes called pixels, corresponding to the pixel matrix structure of typical sensors and display devices.

[0091] Neural networks apply transformations to tensors, typically reducing their width and height while increasing the number of channels. However, some layers can also increase width and height while reducing the number of channels, or flatten the tensor into a 1D vector, etc. This results in a 1×1×C' tensor. This tensor is a feature vector, with each of the C' channels representing a feature. In the next step, a fully connected (FC) layer can be applied to the output. It's worth noting that C can generally be equal to C', but C' can also be smaller or larger. Typically, in CNNs used for object recognition, C' is larger. The FC layer 120 is typically used to embed the complex features extracted from previous layers into a feature vector of the desired size. The output 125 of the FC layer 120 is a feature vector of size S. These vectors can also be used to determine new categories or the relevance of input data to existing categories, etc.

[0092] It's important to note that a neural network (NN) doesn't necessarily have to include any fully connected (FC) layers; it can directly output the feature vector (feature) of the last (e.g., convolutional) NN layer. It's also worth noting that in this paper, the feature tensor can be referred to as the feature map (with a certain height and width) corresponding to different channels. This tensor can be vectorized and is therefore called a feature vector. In other words, the term "feature (embedding)" in this paper is used to describe any grouping of data such as feature vectors and feature tensors.

[0093] The values ​​of intermediate tensors in the CNN inference process are often referred to as intermediate layer features because they are not used in the final features, but the final features are computed based on these features. Therefore, these tensors contain useful semantic information about the input, which can help identify target features after some additional processing.

[0094] The value of an intermediate tensor of size H'×W'×C (assuming C is not reduced to C' here) is used to evaluate the distribution functions of features from one or more intermediate layers, which are then used as input. The height H' and width W' are typically different from H and W. If a CNN is used, H' and W' are usually smaller than H and W, respectively, because CNNs typically apply subsampling at each step to extract features from different layers.

[0095] It's important to note that the term "distribution function" typically refers to the continuous probability distribution of a random variable. This distribution function is usually unknown for the first input data and therefore must be estimated. Estimation can be performed by calculating the frequency of certain values ​​(e.g., the rate of occurrence of values ​​belonging to certain intervals) (also called ticks). This distribution function is calculated at some fixed points, resulting in, for example, a discretized distribution of the intermediate semantics for each input. Note that a discretized distribution can be evaluated not only for a single input but also for the entire category or the entire dataset using the arithmetic mean of the distribution.

[0096] The obtained distributions can be compared using any distribution distance / bias / similarity metric. This allows for the identification and penalty of high similarity between objects with closely distributed intermediate layer features, preventing the aforementioned problems. In other words, the similarity function is modified to be more accurate for OOD examples, thereby improving the overall recognition rate.

[0097] exist Figure 1 In this context, the input processing section includes the input of image 101. Image 101 can be acquired from sources such as a camera, streaming media services, or any type of storage.

[0098] The above processing also includes applying the first 110_1 layer and other layers (including the Kth layer 110_K in the intermediate NN layers of the original model) to the first input data. Through processing, a series of tensors of shape H1×W1×C1……H′×W′×C……1×1×S are obtained, where S is the size of the normal (ordinary) feature 125, i.e., the result of the last layer 120. In this example, K is 1 or greater than 1, up to the total number of layers in the NN. It should be noted that this invention is applicable to neural networks comprising any number (one or more) layers (including the last layer, and possibly one or more intermediate layers).

[0099] The intermediate tensor of shape H′×W′×C after layer K (110_K) is divided into C tensors 130_1 to 130_C of shape H′×W′ along the channel axis. The calculation of values ​​less than in function blocks 140_1 to 140_n corresponding to tensor 130_1 is performed. The occurrence rate and proportion of the value. Each channel c is calculated independently. This is shown by function block 145_n, where the highest (nth) value is obtained for the C-th channel. Division 150 is used for normalization purposes. In this case, normalization is performed by normalizing the total number of occurrences calculated for each channel C. The total number of occurrences corresponds to the size of the tensor, which in this case is H′×W′.

[0100] In function block 155, the normalized discretized distribution is concatenated into a vector of shape n×1. The less than on channel c... The proportion of the value is expressed as This corresponds to the cumulative distribution function used to estimate the value of the Kth tensor in the c-th channel. The index i spans the discretized range of values; for example, i can be an integer equal to or greater than 1 and equal to or less than n. The cumulative distribution function is an increasing function with possible values ​​between 0 and 1. In general, the cumulative distribution function is 0 at negative infinity and 1 at positive infinity.

[0101] Each vector of shape n×1 represents a new distribution function:

[0102]

[0103] These are the discretized distributions of each channel c of the intermediate tensor. All vectors obtained for all corresponding C channels are concatenated into a tensor of shape n×C. Here, c is equal to or greater than 1 and equal to or greater than C.

[0104] In this example, individual channel distributions are estimated because each value in a channel is calculated using the same formula with different parts of the image (spatially) as input. Therefore, each discretized distribution represents the distribution of a specific semantic feature on the input.

[0105] exist Figure 1 In the diagram, output 1 (represented by a dashed circle marked with 1) is the result of processing the first input data through neural network layers 110_1……110_K……120. Output 2 (represented by a dashed circle marked with 2) is a discretized distribution of C channels including n value intervals (spaces).

[0106] In summary, the discretization distribution is calculated by obtaining the number n of value intervals for the first feature tensor value, and for each of the n value intervals, determining the frequency of occurrence of the value belonging to that interval in the first feature tensor value. In the exemplary implementation shown above, the discretization distribution is a cumulative distribution function (CDF), where the value intervals overlap in a specific way, i.e., a value can fall into several intervals. However, the invention is not limited to the distribution being a CDF. Instead, the distribution can be an estimate of a probability density function (PDF) with non-overlapping intervals.

[0107] The original feature f(x) (output 1) can be concatenated with a tensor (output 2) having a discretized distribution D(x) to form a new feature e(x) = [f(x), D(x)] corresponding to some input data.

[0108] Figure 2This involves the similarity calculation part. After obtaining the estimated distribution, the distribution can be further evaluated for various purposes, such as comparing it with other (estimated or ideal) distributions. Evaluation can be performed by determining (calculating) similarity function values, dissimilarity or deviation values, or typically by any distance metric, etc. For example, a distance metric can be determined between the first and second data points. For this purpose, in Figure 1 In the example, the second input data 102 is processed through processing pipeline 160 to obtain output 3 (represented by a dashed circle marked 3), which is the result of processing the second input data through neural network layers 110_1……110_K……120. Output 4 (represented by a dashed circle marked 4) is the discretized distribution of the Kth tensor of the second input data 102, which includes C channels with n value intervals. Discretization can correspond to... Figure 1 The processing is from 130 to 155.

[0109] In other words, obtaining the second distribution includes: processing the second input data through at least one layer to obtain a second feature tensor; and estimating the second distribution of the obtained second feature tensor. Then, determining the characteristics of the first input data may include: determining the similarity between the first input data and the second input data through a similarity metric (i.e., a function of the distance value), as described below. Figure 2 Further description.

[0110] However, the present invention is not limited to comparing the distributions of two input images. Instead, comparisons can be made using a centroid, such as the average discretized distribution of all examples of a certain category, and so on.

[0111] The exemplary similarity calculation section receives new features e(x1) and e(x2). Then, the new features e(x1) and e(x2) are split into the original features f(x1) (output 1) and f(x2) (output 3) and discretized distributions D(x1) (output 2) and D(x2) (output 4). Here, the term "feature" refers to a feature vector, i.e., a grouping, set, or permutation of feature data. In general, a discretized distribution also corresponds to a feature. Specifically, the discretized distribution can be considered as a feature vector of length n.

[0112] Then, outputs 1 and 3 are used to calculate the value s1 of the (first) similarity function between the original features f(x1) and f(x2) of 210. Outputs 2 and 4 are also used to determine new similarity values. Specifically, for each channel map, the tensor with a discretized distribution is split into C independent vectors 220_1 to 220_C of shape n×1. This splitting is illustrative and does not necessarily have to be performed. For example, refer to... Figure 1 The input processing described does not necessarily need to be cascaded, and therefore does not need to be split.

[0113] In the next step, calculate the distribution dissimilarity, distance, or deviation d1 between the distributions in the first channel plot of 230_1:

[0114]

[0115] Where H(·,·) is the distribution distance / bias. This is the discretized distribution function when c = 1. This is performed at each c = 1...C until the distribution distance / deviation d between the 230_C distributions is obtained for each channel plot in the Cth channel plot. C .

[0116] Then, the distribution distance / deviation can be aggregated or combined into a scalar d of 240. aggr Next, calculate the new (second) similarity function (s1, d) for 250. aggr The second similarity function ψ(s1,d) aggr ) can be designed for each variable s1, d aggr It is a monotonically increasing function.

[0117] It should be noted that similarity s1 is not necessarily calculated for other (second) input data 102. As mentioned above, comparisons can be performed using the average distribution of the training dataset, in which case the detection of OOD examples can be easily achieved. Specifically, OOD is an example whose distribution differs significantly from the average distribution of the dataset. The distance can be measured using the newly obtained (second) similarity.

[0118] In other words, according to one embodiment, the second distribution is obtained by averaging multiple distributions determined from corresponding input data belonging to the same category. For example, determining the characteristics of the first input data may include: comparing a distance value with a threshold; and estimating the reliability of the first input data belonging to the same category based on the comparison result.

[0119] In these embodiments, the categories described above can correspond to categories set in any pre-training process. The category set can be closed, meaning it includes multiple predefined and pre-trained categories that do not change over time. However, the invention is also applicable to open models where new categories can be defined during training or even production (processing).

[0120] It should be noted that, generally speaking, a category is not necessarily an identifiable category, such as a category labeled with a real name, like people, cars, trees, person A, person B, etc. Instead, a category can be a meta-category, such as a blurry image, a child's photo, or even a category whose meaning is not quite obvious.

[0121] Now according to Figure 4 The functional block scheme shown describes an example of OOD detection.

[0122] exist Figure 4 In the example, input image 401 corresponds to the first input data. In this example, input image 401 is an input image similar to image 101, having a height of H samples and a width of W samples, as well as 3 color channels (components). As mentioned above, the present invention is not limited to 3 color channels. Generally, the present invention is also applicable to one channel, such as a luminance channel. There can be 2, 3, or more color channels. Optionally or additionally, in addition to color channels, depth channels, motion flow channels, or any other feature channels can be used.

[0123] The input image 401 is fed into a convolutional neural network comprising layers 410_1…410_K…420. In other words, the operations of the CNN are consistently applied to the input, resulting in a series of tensors of shape H1×W1×C1…H′×W′×C. Figure 4 As shown, in this example, H1×W1×C1 corresponds to H / 2×W / 2×64. In this example, after processing through K layers (410_K in the Kth layer), the result of processing by the CNN is used to derive the discretized distribution. For this purpose, the intermediate tensor of shape H′×W′×C is split into C tensors of shape H′×W′ along the channel axis. This is consistent with the reference... Figure 1 The processing shown is similar. As mentioned above, multiple channels are not required. In an implementation with only one channel, no splitting is performed.

[0124] In function blocks 440_1 to 440_n and the other function blocks up to 445_n, the estimation of the distribution function is performed. Similar to the above, for each channel c, a value less than [a certain value] is independently determined. The proportion of values ​​less than c (i = 1 to n) on channel c is estimated, and the results of each channel are concatenated into a vector of shape n×1. The number (or the proportion under normalization) of values ​​less than c on channel c is the estimated distribution function, expressed as:

[0125] Each vector of shape nx1 represents a new distribution function (in this case, the cumulative distribution function):

[0126]

[0127] These It is the discretized distribution of each channel c of the intermediate tensor. All vectors are concatenated into a tensor of shape n×C.

[0128] Then, one or more distributions 450_1 to 450_C of the obtained first input data (in this example, input image 401) are compared with one or more average distributions 455_1 to 455_C of the training dataset 460_1 to 460_C, and the results are combined 470 into d. aggr (This can be a scalar). The combination (aggregation) can be, for example, the calculation of the deviation or dissimilarity between one or more distributions 450_1 to 450_C of the first image 401 and the average distribution 455_1 to 455_C. This deviation or dissimilarity can be any index representing the distance between the two distributions. The index can be calculated per channel, resulting in C results. These C results can then be combined by calculating the average or weighted average, or by calculating any norm such as the minimum or maximum value.

[0129] In function block 480, the characteristic value d aggr Compare with the threshold. If d aggr If the value is greater than the threshold, then the first input data (e.g., image 401) is OOD data 495 because the first input data (i.e., the distribution of the intermediate tensor) differs significantly from the common distributions of the intermediate tensors in the training dataset (455_1 to 455_C). If d aggr If the value is not greater than the threshold, then the first input data is ordinary data (not OOD data) 490. If the equation applies, the first data can be defined as OOD by a fixed convention. However, it can also be stipulated that the first data is not OOD.

[0130] It should be noted that using the above method, some training examples may also be labeled as OOD examples. This is correct because some training examples may be rare in terms of the intermediate layer features they contain (the properties of the Kth intermediate tensor).

[0131] The following section discusses the two embodiments described above. Figure 1 / Figure 2 , Figure 4 Other exemplary detailed parameters and implementations are provided below. Many parameters exist to improve the performance of a specific application, such as the index K of the intermediate tensor used to determine one or more distributions, the points (range of values) used for calculating the distribution function and their number, the distribution distance / bias index ψ (function), etc. The threshold for OOD detection 480 can be empirically selected based on the training set and the desired application.

[0132] In general, this invention provides a method that not only more accurately identifies outliers but also reduces computational complexity, thereby improving efficiency. For example, all discretized distribution comparisons can be simplified to a single binary operation. This can be achieved, for instance, if the distance values ​​are calculated based on Hellinger distances. Hellinger distances can be calculated between C (where C is an integer equal to or greater than 1) first distributions and their corresponding second distributions. In this case, the resulting value is a scalar corresponding to the characteristic value of the first input data. In other words, a tensor with discretized distributions can be reduced from n×C to 1×1 to be implemented using Hellinger distances.

[0133] The Hellinger distance H(P,Q) between two distributions P and Q is usually defined as:

[0134]

[0135] For discrete (or discretized) distributions, the Hellinger distance has the following form:

[0136]

[0137] Where t0=-∞, t n+1 =∞.

[0138] In d aggr equal In the case of the sum of squared distances, then

[0139]

[0140] in,

[0141]

[0142] t is the cumulative distribution function of the discretized distribution of the channel graph c of input k. i These are the points (ranges) used for evaluating the distribution function, t0 = -∞ (therefore) ), t n+1 =∞ (therefore) ).

[0143] Since Hellinger distance simplifies to a simple sum of differences, principal component analysis (PCA) can be applied to reduce the number of computations. In other words, applying PCA is equivalent to calculating the sum of differences projected onto a subspace rather than the initial space.

[0144] For example, d aggr The estimate simplifies to a sum of squares and can be rewritten as follows:

[0145]

[0146] in, It is an L2 norm.

[0147] Consider ν as Euclidean We take points in space and evaluate the values ​​of these points against a large amount of input to obtain sufficient statistics to fit PCA to these points. Once PCA is fitted, it can be used to obtain Euclidean space through the matrix-vector dot product. New points in (m << C·(n+1)):

[0148] μ=Mv τ ,

[0149] Here, M is a matrix of shape m×C(n+1), where the l2 norm of each row is equal to 1. This equation indicates that μ is the coordinate vector of the projection of point v onto the subspace, and the coordinates of the basis vectors of the subspace are defined by the rows in matrix M.

[0150] Therefore, if PCA is used to obtain the new d′ aggr Then it becomes This is equal to the squared l2 distance between the projections of v1 and v2 onto the subspace, whose basis vector coordinates are defined by the rows in matrix M.

[0151] Since m (the dimension of the subspace used for projection) can be any value, all points can be projected onto a line (a one-dimensional subspace). Therefore, all n·C values ​​defining the discretized distribution can be mapped to a single value.

[0152] In summary, the new space can be determined using PCA. Therefore, all v values ​​can be projected onto... On a straight line in space. Therefore, v values ​​can be characterized, for example, as a floating-point number (the coordinates of the projection points on the straight line). In other words, the spatial size can be reduced by PCA, making it possible to calculate distances in a smaller-dimensional space (e.g., between projections on a straight line).

[0153] An exemplary pipeline is described below:

[0154] 1. Select the layer (K) for statistics, and select the number of distributions n and the position of the interval.

[0155] 2. (Optional) If the system requires fast computation, aggregation in the form of Hellinger distance and sum of squares + PCA can be used to reduce the number of operations in the production process. The values ​​in intervals are estimated based on a large number of inputs from previous steps to fit the PCA, but this process is predetermined. The PCA is fitted using a fixed formula.

[0156] 3. The new similarity function depends on the second step. Its parameters are determined according to the formula for d_aggr.

[0157] Hellinger distance and PCA are methods to simplify calculations, but are merely exemplary and optional. Other distances can be used in this invention, and PCA is not required.

[0158] Therefore, the Hellinger distance can be approximated using a binary function (specifically, the l2 distance on a straight line), requiring only a slight increase in computational power compared to the original similarity function.

[0159] In summary, according to one exemplary implementation, the distance value is calculated by approximating the Hellinger distance as the sum of squared differences projected onto a space reduced by applying principal component analysis.

[0160] The specific tensor index K depends on the network architecture and the complexity of the intermediate layer features. The following describes how to obtain the distribution from any tensor.

[0161] Consider the value of the channel graph c of tensor l as a random variable. The goal is to estimate the distribution function of the values ​​on each channel plot c of tensor l. Since any number of correct inputs (inputs in the distribution) can be used, the distribution function can be approximated as follows, using normalization by calculating the proportions:

[0162]

[0163] In other words, to obtain an estimated distribution function independent of the number of values ​​used for estimation, normalization is performed using the number of values. Distribution functions have certain properties, and then satisfying those properties is crucial. For example, the cumulative distribution function returns values ​​between 0 and 1, while the sum of the areas of the probability density functions is 1.

[0164] To estimate a continuous distribution function from a discrete distribution function, the range of values ​​must be discretized. This can be done in various ways. For example, n points (defining the resolution of the distribution function) can be selected, such that these points are equidistant along the range of values. This range is the range of values ​​of the Kth intermediate tensor. Then, the values ​​falling between two adjacent points are calculated for all intervals, resulting in an estimated discrete distribution function representing the frequency of occurrence of values ​​falling into the corresponding intervals. However, this invention is not limited to selecting points equidistantly corresponding to intervals of the same size. While this method is effective and simple, selecting points based on the frequency of occurrence of values ​​has advantages.

[0165] For example, points (t1...t2) can be selected in (a relatively small number) of floating-point numbers. n Then apply the transformation. The points can be more densely packed near the median of the distribution (or near the mean, etc.). In other words, the distribution function of frequently occurring values ​​has a higher resolution, corresponding to smaller intervals and denser points within the defined interval.

[0166] For example, the selection of points can be described as follows:

[0167]

[0168]

[0169] Where std(.) represents the standard deviation from the mean, i.e., the random variable The expected value E. and This represents the standard deviation of the left and right portions of the distribution relative to the mean for the c-th channel.

[0170] The number of points (the resolution of the distribution function) can be determined as follows:

[0171]

[0172] Here, the layer height is the height H' of the tensor in the Kth layer of the CNN. The height of the penultimate layer is the height of the (K–1)th layer. For example, for unit 2 in layer 2 of the well-known ResNet described by He K. et al. in Deep Residual Learning for Image Recognition, 2016 / / IEEE Conference Proceedings on Computer Vision and Pattern Recognition, pp. 770–778, n′ can be

[0173] Furthermore, the point (also known as the interval or threshold) can be given by the following formula:

[0174]

[0175]

[0176] Here, P is a parameter that maximizes the random variable. The sum of the Shannon entropies. The number 8 is an example. In general, suitable values ​​can vary and can be determined experimentally beforehand (before the production stage) and then used as predefined values. Other values ​​besides 8 can be used, for example, in the range of 0.5 to 16. The Shannon entropy H(X) of a discrete random variable X is defined as:

[0177]

[0178] Wherein, P(x i ) is a value of x iThe probability of X. The base of the logarithmic function is 2, in which case the unit of entropy is bits. However, the base of the logarithm and the unit may be different.

[0179] Therefore, parameter P allows the interval locations to provide more information, since each point has the same proportion in the distribution.

[0180] Figure 6 Examples of small P (top left), large P (top right), and optimal P (bottom) are shown. The x-axis shows i intervals. The values ​​are shown on the y-axis.

[0181] Using the points determined in the previous steps, input the discrete distribution tensor T of X. dist The calculation is as follows:

[0182]

[0183] This tensor has C distributions. in

[0184]

[0185] Finally, the obtained distributions can be compared using any distribution distance or bias as described above.

[0186] In summary, obtaining n value intervals may include at least one of (i) determining the number n based on the dimension of the first feature tensor and (ii) determining the length of the value interval based on the entropy of the first feature tensor value.

[0187] It should be noted that entropy is only one possibility, which can provide the advantage of adapting the resolution of the distribution function to the frequency of occurrence of values. However, this invention is not limited to this method, and other statistical data besides entropy can generally be used, or the interval can be selected according to predefined criteria (e.g., equidistant intervals). The number n can be fixed or selected according to criteria other than the dimension (or one dimension) of the Kth tensor.

[0188] To select the tensor (i.e., the value of K) used to obtain the discrete distribution, an OOD example set can be evaluated. The OOD example set can include correct inputs (ODDs), which are rare in normal training datasets. The training set for ODD examples should advantageously have at least several hundred examples. Generating such data is outside the scope of this invention, but is known to those skilled in the art in the context of neural network training. Alternatively, an equally large dataset can be provided for examples within the distribution.

[0189] Using these two datasets, the optimal tensor can be determined as follows:

[0190] (a) Average Discretized Distribution Function (c∈{1,...,C l The tensor output by layer l is generated using examples from all distributions of each tensor l. As mentioned above, the tensor output by layer l is the l-th tensor.

[0191] (b) Calculation (c∈{1,...,C l Distribution distance / deviation between}) and the distribution of each input X (OOD input and in-distribution input) of each tensor l. Aggregate values ​​(e.g., squared Halinger distance sum).

[0192] (c) Next, consider that this problem is a binary classification task, where only one explanatory variable is available, namely In other words, according to the variables The value of determines whether the input belongs to the OOD category or the in-distribution category. Specifically, if If the value is greater than or equal to the threshold, it is used for calculation. The input data is classified as ODD, otherwise it is classified as in-distribution.

[0193] (d) Then, select tensor l (choose the Kth tensor), and based on a certain metric, such as accuracy, F1 score, etc., use the distribution distance / bias. to (c∈{1,...,C l The aggregated value of}) is used as input to achieve the optimal binary classification rate. For example, a metric such as FPR when TPR = 90% can be used.

[0194] The example above for finding K is not limiting. However, this method can find suitable tensors for analyzing specific applications / desired accuracy and / or complexity or other criteria.

[0195] In other words, the above method may further include: identifying at least one layer in the neural network as the layer that provides the maximum classification accuracy for the output feature tensor. It should be noted that in the above description, only a single layer (and its corresponding tensor) is selected and used. However, the invention is not limited to a single layer. Instead, two, three, or more layers in the neural network can be analyzed by discretizing them into a similar pipeline illustrated by a single layer. The results can then be used individually or aggregated.

[0196] To find new similarities, a set of difficult (meaning hard to distinguish) pairs can be used, i.e., input pairs from different categories with high similarity in their original features and input pairs from one category with low similarity in their original features. In addition to difficult pairs, a larger set of ordinary pairs can also be provided, which can be correctly identified using different confidence levels. Generating such data is beyond the scope of this method, but is within the capabilities of those skilled in the art.

[0197] Provide a monotonically increasing function ψ(s1,d) for each variable aggr This might be advantageous. Here, s1 is the original similarity between inputs, and d... aggr It is an aggregated value of distribution distance / bias. Therefore, inputs with greater initial similarity have greater new similarity. At the same time, it increases the similarity of inputs, as the distributions of intermediate layer features differ significantly from each other. This is equivalent to penalizing inputs with tightly distributed intermediate layer features.

[0198] Please note that, for example, in the case of cosine similarity, the initial similarity function is usually constrained, while the distribution distance / bias is usually unconstrained. This is because ψ(s1,min(d)) is expressed as a certain value θ. aggr Search for ψ(s1,d) in the function θ) aggr The reason why functions have advantages.

[0199] To make the new similarity function ψ fast and prevent overfitting, it can be expressed as ψ(s1,d) aggr )=p1(s1)+w·p2(min(d aggr Searching within functions of ,θ) is advantageous, where, a1 and a2 are (positive) integer parameters, and w and θ are predefined parameters. These parameters can be floating-point or integer parameters, depending on the specific implementation. The parameter Θ is the predefined maximum distance.

[0200] This problem can be considered as a binary classification task (a pair of classes belonging to one class and a pair of classes not belonging to another class), where only one explanatory variable is available, namely ψ(s1,d). aggr Then, based on a metric such as accuracy, F1 score, etc. (accuracy in this implementation), the optimal combination of parameters a1, a2, w, and θ is selected. This new similarity metric can then be used for specific applications where the parameters have been optimized, such as face recognition, general object recognition, or other specific object recognition, such as car recognition.

[0201] The above-described exemplary novel similarity function does not limit the invention. Instead, other functions can be used. Generally, any similarity measure can be used, which is a function of the distance between the distributions of the output tensors in a predetermined layer (K). The novel similarity measure can also be a function of the feature tensors obtained by processing the first input data through all layers of the neural network and / or the feature tensors obtained by processing the second data through all layers of the neural network. Specifically, the function can be the distance value d. aggr It is a monotonically increasing function.

[0202] It should be noted that, in Figure 1 , Figure 2 and Figure 4 In the example above, the first input data is image data comprising C channels, where C is an integer equal to or greater than 1. In other words, the invention can be applied to a single channel (e.g., brightness in the case of an image input). However, the invention can be applied to more channels, such as color channels or other channels. Furthermore, as shown in the example above, the layers for analyzing the tensor and other parameters can be selected based on the application and the type of data input to the neural network. It does not necessarily have to be image data. Audio or text data, or other types of data or feature data, can be used.

[0203] In the case of multiple channels, according to one embodiment (e.g.) Figure 1 , Figure 2 and Figure 4 As shown in the diagram, for each channel c (channels 1 to C), the steps of processing the first input data, estimating the first distribution of the obtained first feature tensor, and determining the distance value (e.g., a metric) are performed individually. The characteristics of the first input data are determined by aggregating the distance values ​​determined for each channel.

[0204] For example, when dealing with multiple instances of a single class, an average can be used. When combining distributions from multiple channels (including processing individual input instances), aggregations, such as sum of squares and Heiling distance, can be used. Because the Heiling distance applies the square root, the square can be particularly easy to determine. However, other forms of aggregation can also be used.

[0205] As described above, the present invention can be used for facial recognition.

[0206] This invention can be implemented by the method described above. However, embodiments and examples of this invention can also be implemented as various devices. For example, a signal processing device can be provided that processes first input data through a neural network. The signal processing device includes processing circuitry for: processing the first input data through at least one layer of the neural network to obtain a first feature tensor; estimating a first distribution of the obtained first feature tensor; obtaining a second distribution; determining a distance value representing the distance between the first distribution and the second distribution; and determining characteristics of the first input data based on the determined distance value.

[0207] Figure 5 This is a simplified block diagram of a device 500 provided in an exemplary embodiment. Device 500 can be used as... Figure 1 and Figure 2 Or Figure 3 or Figure 4 The signal processing device in the middle.

[0208] The processor 502 in device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or multiple devices, existing or to be developed in the future, capable of operating or processing information. While the disclosed implementation may be carried out using a single processor such as the processor 502 shown in the figure, using multiple processors can improve speed and efficiency.

[0209] In one implementation, the memory 504 in device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. The memory may be internal or external to the processing circuitry (including one or more processors) and may be implemented on the same chip or separately. Any other suitable type of storage device may be used as memory 504. Memory 504 may include code and data 506 accessed by processor 502 via bus 512. Memory 504 may also include an operating system 508 and an application program 510, which includes at least one program that causes processor 502 to perform the methods described herein. For example, application program 510 may include application 1 to application N, and also includes signal processing means for performing the methods described herein.

[0210] The device 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element capable of sensing touch input. The display 518 may be coupled to the processor 502 via a bus 512.

[0211] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. Furthermore, auxiliary memory 514 may be directly coupled to other components in device 500 or accessible via a network, and may include a single integrated unit (e.g., a memory card) or multiple units (e.g., multiple memory cards). Therefore, device 500 can be implemented in a variety of configurations.

[0212] References in this article Figures 1 to 4The embodiments and functions described herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions can be stored as one or more instructions or code in a computer-readable medium or transmitted via a communication medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium (e.g., a data storage medium), or any communication medium that facilitates the transmission of a computer program from one place to another according to a communication protocol, etc. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described herein. A computer program product may include a computer-readable medium.

[0213] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store required program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source via coaxial cable, fiber optic cable, twisted pair and DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair and DSL, or wireless technologies such as infrared, radio, and microwave are also included in the above definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. The disks and optical discs used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), and Blu-ray discs, wherein disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above items should also be included within the scope of computer-readable media.

[0214] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), one or more general-purpose microprocessors, one or more application-specific integrated circuits (ASICs), one or more field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the various functions described herein can be provided within dedicated hardware and / or software modules for encoding and decoding, or incorporated into combined codecs. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0215] The techniques of this invention can be implemented in a variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The various components, modules, or units described herein are intended to emphasize functional aspects of the apparatus for performing the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Rather, as described above, the various units can be combined with suitable software and / or firmware within a codec hardware unit, or provided as a collection of interoperable hardware units including one or more processors as described above.

[0216] An exemplary implementation provides a signal processing apparatus for processing first input data via a neural network. The signal processing apparatus includes functional modules, which can be implemented in any hardware and / or software modules as described above. The signal processing apparatus is as follows... Figure 3B As shown, a neural network 360 is included for processing the first input data through at least one layer of the neural network to obtain a first feature tensor. The neural network can be any type of neural network suitable for processing the first input data. For example, it can be a CNN for processing image data. The neural network can include an interface / output for providing the Kth tensor (the first feature tensor) to other functional modules, such as... Figure 1 , Figure 2 and Figure 4 Shown in detail.

[0217] Furthermore, the signal processing apparatus includes a distribution estimator 370 for estimating a first distribution of the obtained first feature tensor and an input 375 for acquiring a second distribution. The second distribution can be acquired similarly to the first distribution, i.e., by estimating the distribution from a tensor of the second image. Alternatively, the second distribution is a predetermined distribution acquired based on multiple (training) data, such as the centroids or average distribution of categories.

[0218] The signal processing apparatus further includes a distance determination unit 380 for determining a distance value representing the distance between the first distribution and the second distribution, and a characteristic determination unit 390 for determining characteristics of the first input data based on the determined distance value. The functions of these modules have been described in detail above.

[0219] The present invention, including methods and apparatus, employs a discretized (estimated) distribution of intermediate tensor values. This constitutes additional signal processing compared to the normal deployment of neural networks. Embodiments of the invention can be used only for inference (production phase), while learning (training) proceeds normally, i.e., according to any well-known training method. The invention can be used for any deep learning-based computer vision task using the same pipeline. The invention can be used for both open-set and closed-set recognition tasks.

[0220] Determining the characteristics of the first input data (e.g., an input image) can correspond to determining one of several predetermined data categories. This can be either open-set or closed-set classification.

[0221] Optionally or additionally, determining the characteristics of the first input data can correspond to determining whether the first input data belongs to a category within a predetermined data category. This allows for the differentiation between out-of-distribution and in-distribution situations of the first input data.

[0222] Other embodiments and exemplary models may provide additional effects and advantages. For example, as described above, in the case of Hellinger distance, the required operations can be reduced to a single binary operation. Optimal parameters can be selected (automatically) using a parameter selection algorithm. Therefore, optimal parameters can be adjusted on the terminal device. For example, an intermediate tensor K can be selected to best distinguish the desired features. The resolution of the discretization distribution can be selected based on the selected tensor.

[0223] Figures 7 to 9 This is a schematic diagram illustrating the effects of the above embodiments.

[0224] Figure 7The neural network constructed in the initial face recognition model is shown. The input layer takes a face image as input. The face image can be a "hard example," difficult to recognize. As mentioned above, a child's face or an elderly person's face may be difficult to recognize. Furthermore, some unusual faces, computer-generated faces, or animal instances may be difficult to recognize or distinguish from faces. This can be important in applications such as FaceID or other applications employing face recognition. The output of the input layer is passed through more (intermediate) layers (low, intermediate, and high) of the initial face recognition model. On the right, the distribution of estimated cosine similarity between the output (or high-level) feature pairs of the input data is shown, including correct pairs (should be similar) and incorrect pairs (should not be similar). As can be seen from the distribution plot of hard examples, the similarity distributions of correct and incorrect pairs overlap and invert, making it difficult to set a threshold for reliable differentiation. The similarity of correct pairs is on the left side of the plot, but in general, they should be on the right. The similarity of incorrect pairs should be on the left side. Error localization is due to the visualization being for hard examples. The method of the present invention shifts at least some correct pairs to the right.

[0225] Figure 8 The illustration schematically shows the use of one embodiment of the invention regarding input image comparison. In addition to direct evaluation (0), for example, as in combination with... Figure 7 Further additional processing is performed. Specifically, for the first image (dP) and the second image (dQ) in a pair, the distribution of features of the intermediate layer is calculated according to (1). Then, the distributions are compared according to (2), and a new similarity (distance between the two distributions) is determined as shown in (3), as a supplement to the similarity aggregation (and correction) calculated according to (0) to better distinguish between correct and incorrect pairs.

[0226] Figure 9 A schematic example of an embodiment for detecting out-of-distribution examples (input image) is shown. Steps (1) and (2) are combined with... Figure 8 The description is the same. The distribution is compared with the predetermined distribution to determine, according to (3), whether the tensor distribution of the input data is sufficiently similar to the predetermined distribution (in-distribution). Some examples of faces outside the distribution are shown in the upper right corner.

[0227] As described above, the present invention is applicable to any type of feedforward neural network and is suitable for tasks including closed-set and open-set architectures.

[0228] Exemplary architectures of neural networks for closed-set image recognition include Figure 10AAs shown. This architecture may be the same for both the training and inference phases. An input image 1010 with height H, width W, and 3 input channels is fed into the neural network. The neural network consists of N convolutional layers 1110_1 to 1110_N. In this example, the first convolutional layer outputs a feature map with height H / 2, width W / 2, and 64 channels. The last convolutional layer outputs a feature map with height 7, width 7, and S channels. An average pooling layer 1020 outputs a feature map with height 1, width 1, and S channels. The pooling operation calculates the average of 7×7 features for each j between 1 and S. In step 1030, the 1×1×S tensor is flattened into a one-dimensional feature vector of size S. The next layer is a fully connected (dense) layer 1040, denoted as FC. A feature vector of length S is input into this layer, which performs a dot product of this vector (as a row vector) with a matrix of size S×K to obtain a vector 1045 of size K. This vector is a vector containing K similarities for each of the corresponding K categories. Here, (unlike the description above) K corresponds to the number of categories in this closed-set model. Because it is a closed-set model, the number of categories K remains constant during both the training and inference (production) phases.

[0229] The above-described embodiment employing additional processing based on the distribution function of intermediate layers can be implemented after any of the convolutional layers 1110_1 to 1110_N. The additional processing then outputs additional similarities (or distances) for the K categories. The new similarities can then be computed as a function of both the similarity output from the neural network and the similarity obtained through the additional processing.

[0230] Figure 10B An exemplary neural network architecture for the open-set inference stage is shown. A similar architecture is applicable to image recognition in Face ID. The input image is 1010 and the convolutional N layers are 1110_1 to 1110_N, as referenced above. Figure 10A Similar to the previous one. In this architecture, there are no pooling layers. Instead, the output 7×7×S tensor of the last convolutional layer is flattened 1050 into a one-dimensional feature vector of size 49S. Then, a fully connected (FC) layer 1060 multiplies the 49S input vector with a matrix of size 49S×512. Here, 512 is a commonly used value. However, the invention is not limited to this value. Rather, other sizes may be more suitable for some applications. The output feature vector 1065 of size 512 (features) can then be stored for comparison with other such feature vectors (obtained from other input images) to evaluate similarity.

[0231] Similar to reference Figure 10AIn the described architecture, additional processing can be implemented after any of the convolutional layers 1110_1 to 1110_N. The additional processing then outputs additional similarities (or distances). The new similarity is then computed as a function comparing the similarity from the neural network output and the similarity obtained through the additional processing, based on the feature vectors.

[0232] Figure 10C An exemplary neural network architecture for the open-set training phase is shown. The input image 1010, convolutional N layers 1110_1 to 1110_N, flattening layer 1050, and first FC layer 1060 are referenced above. Figure 10B The description is similar. Additionally, a second FC layer can be used during the training phase, which multiplies a feature vector of size 512 with a matrix of size 512×K. Therefore, the result 1075 represents the similarity to the corresponding K classes trained during the training phase.

[0233] Additional processing can also be used for training purposes, for example, by using... Figure 10A and Figure 10C The architecture is shown. Specifically, the novel similarity ψ(s1,d) with K categories can be evaluated during training. aggr ), rather than similarity s1(1045 and 1075).

[0234] It should be noted that the NN architecture is merely an example. Generally, the dimension and number of channels of the feature map may differ. Furthermore, some architectures may not necessarily have a fully connected (FC) layer of 1040. The same applies to a flattened layer of 1030.

[0235] In summary, this invention relates to processing input data via a neural network. Some embodiments provide methods and apparatus that process the input data through at least one layer of the neural network and obtain a feature tensor from the input data. Then, the distribution of the obtained feature tensor is estimated. Other distributions are obtained. These other distributions may be distributions of other input data, or distributions obtained by combining multiple distributions obtained from corresponding multiple input data. Then, a distance value representing the distance between the two distributions is calculated, and characteristics of the input data are determined based on the distance value. These characteristics may relate to the reliability of data detection or category determination for a particular class of data or out-of-distribution data.

Claims

1. A method of processing first input data by a neural network, characterized in that, The neural network is a feed-forward neural network, the first input data comprises image data, the method comprises: processing (310) the first input data through at least one layer in the neural network to obtain a first feature tensor; estimating (320) a first distribution of the obtained first feature tensor; processing (110_1 to 120) second input data through the at least one layer to obtain a second feature tensor; estimating (130_1 to 150) a second distribution of the obtained second feature tensor; determining (340) a distance value representing a distance between the first distribution and the second distribution; determining (350) a property of the first input data according to the determined distance value; the property is at least one of the following: a category of a plurality of predetermined data categories and whether the first input data belongs (480) to a category of the predetermined data categories; wherein the estimating (320) a first distribution comprises: obtaining a number n of value intervals of first feature tensor values; for each of the n value intervals, determining a number of occurrences of values belonging to the value interval in the first feature tensor values; the obtaining n value intervals comprises at least one of (i) determining the number n according to a dimension of the first feature tensor and (ii) determining a value interval length according to an entropy of the first feature tensor values.

2. The method of claim 1, wherein: the determining (350) a property of the first input data comprises: determining (250) a similarity of the first input data to the second input data by a similarity measure, wherein the similarity measure is a function of the distance value.

3. The method of claim 2, wherein, the similarity measure is also a function of a feature tensor obtained by processing the first input data through all layers in the neural network and / or a feature tensor obtained by processing second data through all layers in the neural network.

4. The method of claim 2, wherein, the function is a monotonically increasing function of the distance value.

5. The method of claim 4, wherein, The function ψ is given by ψ(s, d) = p1(s) + w - p2(min(d, Θ)), where p i (x) = x ai , ai is a positive integer, i is 1 or 2, w is a predetermined weight factor, Θ is a predetermined maximum distance, s is an original similarity between inputs, d is an aggregated value of distribution distances between the first distribution and the second distribution.

6. The method according to any one of claims 1 to 5, characterized in that, the second distribution is obtained (240, 470) by averaging a plurality of distributions determined for a respective plurality of input data belonging to the same category.

7. The method of claim 6, wherein, the determining (350) a property of the first input data comprises: comparing (480) the distance value to a threshold value; estimating (490, 495) a reliability of the first input data belonging to the same category according to a comparison result.

8. The method according to any one of claims 1 to 5, characterized in that, the distance value is calculated according to a Hellinger distance.

9. The method of claim 8, wherein, the distance value is calculated by approximating the Hellinger distance as a difference sum of squares projected to a space reduced by applying principal component analysis.

10. The method according to any one of claims 1 to 5, characterized in that, the method further comprises: determining the at least one layer in the neural network as a layer providing a maximum classification accuracy for an output feature tensor.

11. The method according to any one of claims 1 to 5, characterized in that, the first input data is image data comprising c channels, where c is an integer equal to or greater than 1.

12. The method of claim 11, wherein: the steps of processing (310) the first input data, estimating a first distribution of the obtained first feature tensor and determining a distance value are performed separately for each channel c. The determining (340) of the property of the first input data is by aggregating distance values determined for each channel.

13. The method of claim 11, wherein, The method is for face recognition.

14. A computer program stored in a non-transitory medium, comprising instructions, characterized in that, The instructions, when executed on one or more processors, perform the method according to any one of claims 1 to 13.

15. A signal processing apparatus (500) for processing first input data by a neural network, characterized in that The neural network is a feed-forward neural network, the first input data comprises image data, and the signal processing apparatus comprises processing circuitry (502) configured to: process the first input data through at least one layer in the neural network to obtain a first feature tensor; estimate a first distribution of the obtained first feature tensor; process second input data through the at least one layer to obtain a second feature tensor; estimate a second distribution of the obtained second feature tensor; determine a distance value representing a distance between the first distribution and the second distribution; determine a property of the first input data from the determined distance value; the property is at least one of a category of a plurality of predetermined data categories and whether the first input data belongs to a category of the predetermined data categories; wherein the estimating (320) of the first distribution comprises: determining, for each of the n value intervals, a number of occurrences of values belonging to the value interval in the first feature tensor values; 16. A signal processing apparatus that processes first input data by a neural network, characterized by, the obtaining of the n value intervals comprises at least one of (i) determining the number n from a dimension of the first feature tensor and (ii) determining a value interval length from an entropy of the first feature tensor values. The neural network is a feed-forward neural network, the first input data comprises image data, and the signal processing apparatus comprises: a neural network (360) configured to process the first input data through at least one layer in the neural network to obtain a first feature tensor; a distribution estimator (370) configured to estimate a first distribution of the obtained first feature tensor; an input (375) configured to process second input data through the at least one layer to obtain a second feature tensor; estimate a second distribution of the obtained second feature tensor; a distance determination unit (380) configured to determine a distance value representing a distance between the first distribution and the second distribution; a property determination unit (390) configured to determine a property of the first input data from the determined distance value; the property is at least one of a category of a plurality of predetermined data categories and whether the first input data belongs to a category of the predetermined data categories; wherein the estimating (320) of the first distribution comprises: determining, for each of the n value intervals, a number of occurrences of values belonging to the value interval in the first feature tensor values; the obtaining of the n value intervals comprises at least one of (i) determining the number n from a dimension of the first feature tensor and (ii) determining a value interval length from an entropy of the first feature tensor values.

Images