METHOD FOR MACHINE LEARNING A MULTIMEDIA DATA PREDICTION MODEL AND METHOD FOR DETECTING ANOMALIES FROM SUCH A MODEL

Description

[0001] The invention relates to the field of machine learning methods and concerns a new method for learning a multimedia data prediction model and a method for detecting anomalies using such a multimedia data prediction model. The data considered are, for example, images, image sequences, videos, audio sequences, multispectral images, or more generally, data that can be multidimensional and multimodal.

[0002] Anomaly detection involves identifying data that is considered "abnormal" within a given application context. Data is considered "abnormal" when it is unusual, unpredictable, or unwanted. More generally, "abnormal" data can be defined as data that deviates significantly from "normal" data within a given application context.

[0003] The abnormal nature of data depends on the type of data, the intended application, and the context. For example, if the data consists of images of a component or product from an industrial manufacturing line, an anomaly corresponds to a visible defect on the component or product.

[0004] In the case where the data is video sequences, an anomaly corresponds, for example, to an unusual behavior of a pedestrian in a given area.

[0005] Anomaly detection is particularly useful in the field of video surveillance to identify unusual behavior of a person in a public or private place, in the field of autonomous driving to identify an unexpected obstacle on the road, and in the field of industrial control to identify a defect on a manufactured product.

[0006] In the field of machine learning, the task of detecting a category of data is performed by a detector optimized after learning or training, supervised by data representative of the categories to be detected.

[0007] Machine learning methods can therefore be used to develop an anomaly detection model in multimedia data.

[0008] By definition, abnormal data are inherently rare and diverse compared to normal data, which are abundant. Therefore, to avoid detection bias, it is advisable to train a detector on datasets that are balanced across the different categories.

[0009] For this reason, supervised machine learning methods are poorly suited to addressing the problem of anomaly detection because the task of annotating anomalous data is extremely expensive and requires covering a vast domain of heterogeneous anomalies depending on the context and the intended application. Furthermore, these approaches are inefficient given the inherent imbalance between normal and abnormal data classes.

[0010] A general problem that the invention aims to solve is therefore to develop an unsupervised machine learning method for detecting anomalies in multimedia data.

[0011] One-class classification machine learning methods are better suited to the problem of anomaly detection because they only use "normal" data as input and aim to predict or reconstruct this data. The trained model is thus based on the extraction of relevant features from the normal data and is subsequently used to infer the degree of abnormality of new data to be evaluated.

[0012] Unsupervised learning approaches for anomaly detection rely on training an artificial intelligence model (e.g., a deep neural network) to perform a pretext task on normal data only. In other words, the model is not directly trained to detect anomalies but is trained on another task, such as reconstructing normal data, and then indirectly uses the model to solve the anomaly detection task. During the inference phase, an anomaly score can be deduced from the model's inability to perform the task correctly.

[0013] For the trained model to effectively characterize the normality of the data and distinguish it from anomalies, the pretext tasks must satisfy two necessary conditions. They must be performed correctly by normal data and they must be performed poorly in the presence of anomalies. In other words, the chosen pretext tasks must lead to poor generalization of the model to anomalies.

[0014] Unsupervised learning anomaly detection methods can be grouped essentially into two categories: reconstruction methods and prediction methods.

[0015] Reconstruction-based approaches aim to train a model to reconstruct, as output, the normal training data received as input. A key assumption of these approaches is that the reconstruction model will not be able to properly generalize the reconstruction to anomalies; that is, the abnormal data will not be reconstructed correctly.

[0016] Unlike reconstruction-based methods, prediction-based approaches teach models to predict missing information, such as hidden parts of normal data, in order to better learn their characteristics.

[0017] Methods based on a reconstruction approach have the disadvantage of sometimes also correctly reconstructing abnormal data.

[0018] Reference [1] describes an anomaly detection method by reconstruction that involves training the data distribution using a multi-hypothesis autoencoder. Furthermore, the model is evaluated by a discriminator, which prevents the generator from producing implausible predictions. Autoencoders have the drawback of being able to reconstruct anomalous features due to their extrapolation capabilities. Thus, they induce a non-negligible failure rate since some anomalies will be detected as corresponding to normal behavior.

[0019] Predictive methods generally underestimate anomalies because they are trained solely on normal data, and the information to be predicted is absent from the input data. However, these methods are less well-suited to normal data because the missing information to be predicted is not present in the normal data, which can lead to prediction difficulties.

[0020] Most anomaly detection methods based on prediction rely on learning a single prediction, which has the drawback of failing to account for the diversity of normality. Indeed, a single prediction often cannot characterize the diversity of so-called normal behavior. For example, if we consider a simple scenario of a camera observing a vehicle moving along a road and approaching an intersection with three possibilities: turn right, go straight, or turn left, these three possible future states can all be described as "normal." In such a scenario, a model based on a single predictor will not be able to predict the different future states of the vehicle's trajectory with a single prediction. Instead, the generated prediction will correspond to an average of the three possible "normal" states.If normal data is not correctly predicted by such a model, then it will also be unable to detect anomalies by comparison.

[0021] One solution to this problem is to design a multi-prediction model to predict all the "normal" states of a data point.

[0022] Reference [2] describes a predictive anomaly detection method that proposes training a model to produce several different predictions for the same masked data. Learning multiple predictions allows for better coverage of the diversity of so-called normal behaviors in masked input data.

[0023] Reference [3] describes a method for detecting anomalies in a video by focusing on several tasks using an associated predictor.

[0024] The authors propose a stochastic method for predicting normal video data using a conditional variational autoencoder. The method's predictions are stochastic, meaning the samples may not be representative of the learned distribution; furthermore, the anomaly score does not precisely quantify membership in the normal data distribution.

[0025] Therefore, there is a need for a new anomaly detection method that overcomes the drawbacks of reconstruction or prediction approaches.

[0026] The proposed invention combines the advantages of reconstruction and prediction methods. It involves training a multi-prediction model that does not generalize well in the presence of anomalies, thereby improving its ability to discriminate between them. Furthermore, due to the use of multiple predictors, the proposed model adapts better to normal data than a single-prediction model. The predictions are deterministic, ensuring the repeatability of the anomaly scores provided by the system. The predictions are also diversified, allowing coverage of the entire distribution of normal data, with each predictor specializing in a particular pattern from among the set of patterns corresponding to a normal characteristic.

[0027] The invention relates to a computer-implemented method for training a multimedia data reconstruction model represented by at least one modality, the model being composed of a set of several different predictors for each modality of the data, the training method comprising the steps of, for each data point in a training dataset containing no anomalies, For each modality of the data: Mask at least part of the modality of the data, Train each predictor of the set associated with said modality to calculate a different prediction of the same masked data, each predictor being specialized in a possible prediction of the masked data among different alternatives considered normal in the context of said multimedia data, For each predictor, calculate a distance between the prediction and a reference data extracted from the training data, Select the predictor of the set corresponding to the smallest distance, Calculate a first cost function equal to the sum over all modalities, of the distances corresponding to each selected predictor, Update the parameters of the selected predictors for each modality so as to minimize the first cost function during the training of the predictors.

[0028] In one particular embodiment, the process further comprises, For each category of the data: Select a subset of the set of predictors that were not optimized in a previous iteration of the training, Calculate the sum of the distances between the respective predictions provided by the predictors of said subset and the reference data, Calculate a second cost function equal to the sum of said distances for all categories and modify the first cost function by adding to it the second cost function weighted by a weighting coefficient, Update the parameters of the predictor that provides the prediction closest to the reference data and the predictors of said subset for each category so as to minimize the first modified cost function.

[0029] According to a particular aspect of the invention: Multimedia data is represented, according to a basic modality, by a temporal sequence of successive images. The step of masking at least part of the basic modality includes the application of a predefined spatial mask to each image of the sequence so as to mask at least one area of ​​the image. The reference data corresponds to a successive image in the temporal sequence relative to the current image provided as input to the model. Each predictor is trained to predict said successive image from the current image masked by means of said mask.

[0030] According to a particular aspect of the invention: Multimedia data is further represented, according to an additional modality, by an optical flow sequence. The step of masking at least part of the additional modality includes masking the entire optical flow. The reference data is the optical flow. Each predictor is trained to predict the optical flow from the current image masked according to the basic modality.

[0031] According to a particular aspect of the invention: Multimedia data is further represented, according to an additional modality, by a sequence comprising, for each image, a set of object classes detected in the image. The step of masking at least part of the additional modality includes masking the entire sequence of object classes. The reference data is the sequence of object classes. Each predictor is trained to predict the sequence of object classes from the current image masked according to the basic modality.

[0032] According to a particular aspect of the invention: The multimedia data is further represented, according to an additional modality, by an audio sequence synchronized with the image sequence. The step of masking at least part of the additional modality includes the deletion of at least part of the audio sequence. The reference data is the audio sequence. Each predictor is trained to predict the audio sequence from the current image masked according to the basic modality.

[0033] According to a particular aspect of the invention: Multimedia data is represented, according to a basic modality, by a set of images. The step of masking at least part of the basic modality includes the application of a predefined spatial mask to each image so as to mask at least one area of ​​the image. The reference data corresponds to the current image provided as input to the model but unmasked. Each predictor is trained to predict said current image from the current image masked by means of said mask.

[0034] According to a particular aspect of the invention: Multimedia data is further represented, according to a second modality, by a set of multispectral images. The step of masking at least part of the second modality includes the suppression of images at at least one given wavelength. The reference data is the set of multispectral images. Each predictor is trained to predict a multispectral image from the current image masked according to the basic modality.

[0035] According to one particular aspect of the invention, the model comprises: a first projector neural network receiving as input a masked data and trained to transform the input into a latent representation, a recurrent neural network trained to produce a sequence of states, in a recurrent manner, the initial state being equal to the latent representation, several predictor neural networks each corresponding to a modality, each predictor network receiving as input the masked data and a state provided by the recurrent network, the number of states generated by the recurrent network being equal to the number of predictors for a modality.

[0036] The invention also relates to a computer-implemented method for detecting anomalies in a multimedia dataset having at least one modality, the method comprising the steps of: Execute, for said dataset, the machine learning model trained by means of the training method according to the invention, the model receiving as input the data masked by means of masks identical to those used for the training of said model and producing as output several predictions, Calculate the first cost function from the predictions generated by the model, Calculate an anomaly score from a distance between the first cost function and a representative value of the first cost function for a dataset containing no anomalies, Compare the anomaly score to a predetermined detection threshold and deduce the presence or absence of anomalies in the data.

[0037] The invention also relates to a computer program comprising code instructions for implementing the invention, as well as a support d'enregistrement computer readable on which the computer program according to the invention is stored.

[0038] Other features and advantages of the present invention will become more apparent from the following description in relation to the following attached drawings. [ Fig. 1 ] represents a diagram of a first example of the architecture of a multi-prediction machine learning model according to one embodiment of the invention, [ Fig. 2 ] represents a flowchart detailing the steps of a method for training an anomaly detection model according to an embodiment of the invention, [ Fig. 3 ] represents a diagram of a second example of the architecture of a multi-prediction model suitable for training using the method of figure 2 , [ Fig. 4 ] illustrates a backpropagation step of the gradient to update the model parameters of the figure 3 , [ Fig. 5 ] illustrates the operation of backpropagation of the gradient for a model taking into account three distinct modalities of the input data, [ Fig. 6 ] represents an example of a machine learning model for implementing the different networks in the example of the figure 3 . [ Fig. 7 [ ] represents a flowchart detailing the steps of an anomaly detection process based on the model trained according to the process of the figure 2 ,

[0039] There figure 1 schematically represents an example of a multi-prediction machine learning model according to one embodiment of the invention.

[0040] The model receives as input data masked according to a predefined mask M. It includes a first encoder network E capable of converting the masked input data into a latent representation and several predictor networks D(1)< , D(2)< ... D(n)< which are trained to determine each a possible prediction of the original data from the masked data.

[0041] More generally, the n predictions can be generated by n distinct predictor networks or by a single network capable of producing n distinct predictions.

[0042] The model of the figure 1 is trained using so-called "normal" training data, meaning data that does not contain any anomalies in the intended application context. For example, if the intended application involves detecting defects on parts during manufacturing, the training data will only contain images of parts without defects.

[0043] If the intended application involves detecting abnormal behavior in surveillance videos, the training data only includes images of normal behavior in the context of the monitored area.

[0044] Thus, the model of the figure 1 The model is trained to generate different predictions for the hidden parts of the input data, all of which are assumed to correspond to normal data. Once trained, the model can detect anomalies because if an anomaly is hidden, the model will not reconstruct the anomaly but instead predict a reconstruction corresponding to normal behavior. By comparing the predictions with the actual data, it is therefore possible to detect an anomaly when the predictions deviate from the data.

[0045] Using several different predictors allows for optimal training in the sense that the model will not simply learn to generate a prediction corresponding to an average of all possible (credible) reconstructions, but rather, each predictor will specialize in a type of possible reconstruction.

[0046] For example, in the case of an image of a car at an intersection with several possible routes, each predictor will specialize in predicting a trajectory for the car towards one of the routes. In this context, all these predictions correspond to a possible normal behavior for the car. Conversely, a car driving between two roads (on a sidewalk or, more generally, in an area not corresponding to a road) corresponds to abnormal behavior and therefore to an anomaly in this context.

[0047] In general, the invention applies to any type of multimedia data represented by at least one modality. For example, it applies to video sequences, still images, audio sequences, RGB or multispectral images, or a combination of these different media.

[0048] Subsequently, the invention is described for data corresponding to video sequences, but it is generalizable to the other types of data mentioned above.

[0049] There figure 2 diagram the steps involved in implementing a model training process using a flowchart. figure 1 according to one embodiment of the invention.

[0050] The training is performed using 201 training data that does not contain anomalies, so as to train the model to reconstruct so-called "normal" data, meaning data that does not contain any anomalies. Thus, the model is trained to reconstruct partially masked "normal" data.

[0051] In step 202, the training data is partially masked using a predefined mask M. The masking step 202 can take various forms. It consists of masking or altering at least part of at least one modality of the data. More precisely, if the input data is represented by several modalities, each modality can be masked totally or partially, with the caveat that at least one modality must be masked only partially in order to provide a minimum amount of input information to the model. Examples of different modalities will be explained later.

[0052] For example, in the case of an image sequence, the masking step 202 might consist of masking one or more areas of each image according to a predefined spatial mask M. The mask can be purely spatial or spatiotemporal, in which case it also depends on the image's time index in the sequence. The masking step might involve completely removing an area of ​​an image or applying noise, such as white noise, to certain areas of an image.

[0053] The masked data is then provided as input to the model to be trained. In step 203, the various predictions of the original data are calculated from the model parameters and the masked input data. Depending on the chosen embodiment, the predictions aim to predict the current masked frame or a future frame in the video sequence, for example, the frame following the masked frame in the sequence.

[0054] The model therefore provides n predictions Ŷ (1)< , ..., Ŷ ( n Then, in step 204, a first cost function or loss function LNN(Y) is calculated. The chosen cost function consists of selecting, from among the n predictions, the one that is closest to a reference Iref corresponding to the original data and calculating the error between this selected prediction and the reference. For example, if the model aims to predict an image It+1 at time t+1 from a masked image It at time t, then the reference is the original image It+1. Alternatively, if the model aims to directly predict the current image It, then the reference is the original image It.

[0055] The first cost function is thus given by the following relation: L NN Y = min k ∈ 1 n Y ^ k − Y ref

[0056] Y denotes a representation of the data according to a modality. For example, Ŷ ( k )< is a prediction of an image I t+1 and Y ref is the original image I t+1.

[0057] The cost function L NN ( Y ) can be calculated for one or more categories of the data as will be illustrated later.

[0058] This first cost function aims to encourage the diversity of predictions through the selection, at each iteration, of the prediction closest to the reference.

[0059] In step 206, a backpropagation algorithm based on gradient calculation is applied to update the model parameters and minimize the first cost function. At this stage, only the parameters of the selected predictor with index k used to calculate the cost function are optimized with the parameters of encoder E during backpropagation. Unselected predictors are not optimized. In other words, for each category, only the predictor closest to the reference data is optimized.

[0060] In one alternative implementation, a second cost function is calculated at step 205 via the following relationship: L NP Y = ∑ k ∈ U T Y ^ k − Y ref

[0061] UT is the set of predictors that were not selected, or only slightly selected, to calculate the first cost function during a previous iteration of the training. To determine the UT set, a selection threshold is set, for example, below which a predictor is considered to have been selected only slightly during an iteration.

[0062] More specifically, the training is performed in several iterations, with each iteration producing a training dataset called an epoch. After processing the previous epoch, the predictors that were never selected are identified to calculate the first cost function, and the second cost function is calculated for these predictors.

[0063] It is possible that the predictors selected to calculate the second cost function include the predictor selected to calculate the first cost function for a current epoch.

[0064] In step 206, the backpropagation algorithm is applied to minimize a combination of the two cost functions: L = LNN + λLNP, where λ is a weighting coefficient. Preferably, the parameter λ is a positive number strictly less than 1, for example, equal to 0.1, in order to give more weight to the first cost function, LNN. In this way, the predictor whose prediction is closest to the actual data is optimized (via the first cost function, LNN), while the predictors that did not participate sufficiently in the training at the previous epoch are optimized (via the additional term λLNP). Predictors that participated sufficiently at the previous epoch but are too far from the actual data are not optimized (indeed, for them, the calculated gradient is essentially zero with respect to both the cost functions LNN and LNP).

[0065] Backpropagation is performed so as to update the parameters of the predictor selected to calculate the first cost function and of the predictors selected to calculate the second cost function, as well as the parameters of the encoder E common to all predictors.

[0066] The optimization of the second cost function L NP aims to enable the optimization of all predictors, even those that are never or very rarely selected, in order to promote the diversity of predictions.

[0067] In one embodiment, the same machine learning model can be optimized separately to make different predictions relating to different modalities of the data.

[0068] For example, input data can be organized into a primary or basic modality and one or more additional modalities. In other words, input data can be represented by several modalities whose importance can be varied depending on the application.

[0069] Multimodal data refers to a set of information that combines several different modes or data sources. For example, in an audio / video sequence, sound and image can be considered two modalities of this data.

[0070] For example, in an application for detecting visual anomalies and object movement in a video, the input data is a partially masked image sequence. The basic modality here is the image sequence. In this scenario, an additional modality could be, for example, an optical flow sequence calculated on the image sequence, or a sequence of object classes present in each image (detected using an object detector applied to the image sequence). Optical flow is information that characterizes the movement of each pixel of an object between two successive images. It allows us to characterize the movement of objects over time in a video sequence. Objects detected in a video sequence can also be classified into object categories. Thus, each image can be accompanied by information on the classes of the objects present in that image.

[0071] In this application example, the machine learning model can also be trained to predict optical flow from masked images received as input, with the optical flow being completely masked, i.e., suppressed in the sense that it is not provided as input to the model. In this case, a second set of n predictors is optimized using the same cost function and the same learning procedure as that described in the figure 2 The only difference is that the predictions are predictions of the optical flow and are compared to a reference, which is the original optical flow.

[0072] Similarly, by replacing the optical flow with a sequence of object classes, a third set of n predictors can be optimized.

[0073] The use of cross-modal prediction tasks, for example optical flow prediction from a masked image, also makes it possible to detect anomalous correlations that might not be detected if the modalities were treated independently.

[0074] The machine learning model can thus be optimized globally by minimizing a cost function which is a combination of the different cost functions calculated for each modality; this combination is, for example, a sum.

[0075] Learning predictions according to different modalities: spatial, temporal, optical flow or other allows us to better characterize the heterogeneity of normal data and to discriminate them from abnormal data.

[0076] The principle of learning the multi-predictor model for multimodal data can be applied to other modalities.

[0077] For example, when the intended application is anomaly detection in a still image, the model input is unimodal and corresponds to an image. The predictions provided by the model are predictions of the current image based on the spatially masked image.

[0078] In the case of multispectral images, the model input is multimodal, with each modality corresponding to an image at a specific wavelength or wavelength range. In this scenario, some wavelengths are suppressed at the model input, and the predictors are trained to predict images at the suppressed wavelengths based on images at the other wavelengths. For example, the suppressed wavelengths might correspond to infrared.

[0079] In another application example, the data is multimedia and multimodal, comprising both a video sequence and an audio sequence. In this scenario, the audio sequence can be removed from the model's input, and the predictors are trained to predict the audio sequence from the video sequence.

[0080] Without departing from the scope of the invention, other multimodal data can be considered. A general objective of the invention is to train the model to reconstruct so-called normal data from multiple predictions for one or more modalities of the data.

[0081] When the data is multimodal, a basic modality can be defined which corresponds to the modality in which the data is provided as input to the model, partially masked, with the additional modalities being, for example, totally masked.

[0082] We now describe a particular embodiment of the invention which is based on video data and a particular learning model.

[0083] In this example illustrated at figure 3 The machine learning model includes a first projector network P that receives as input each image Y It of the video sequence masked by a spatial mask M. The projector network P is trained to convert its input into a latent representation in another projection space, allowing the information contained in the input to be characterized. In this example, the input data is represented by a single modality, which is the image sequence I.

[0084] The model also includes a recurrent neural network R which is trained to produce a sequence of states hi recurrently, the initial state h 0 being equal to the latent representation provided as output of the projector network P.

[0085] The succession of states hi for i varying from 1 to n allows the input data to be characterized according to n different representations.

[0086] Each output state hi of the recurrent network R is provided, with the masked image, to a predictor network f I to provide as output n different predictions of the next image Y It+1.

[0087] As will be described later, when multiple categories of the data are considered, a different predictor network is used for each category. For example, if m categories are considered, m predictor networks {f T} therefore generate m*n predictions, or n different predictions for each of the m categories.

[0088] The cost function is calculated in the manner described above by comparing the predictions generated by the predictor network to the actual image Y It+1 of the sequence.

[0089] There figure 4 illustrates an example of the application of the gradient backpropagation algorithm when the prediction Ŷ ( k *)< is selected as the closest to the real image. The parameter set of the predictor network f I corresponding to the predictor of rank k* is updated, as well as the parameters of the recurrent network and the projector network P.

[0090] In one alternative implementation of the example of the figure 3 , the data is multimodal and the video sequence is accompanied by an optical flow designated YF and / or an object class vector YC which indicates, for each image, the classes of the objects present in the image.

[0091] This variant is represented at the figure 5 which illustrates, in a diagram, the operation of the gradient backpropagation algorithm in the case where three modalities I (image sequence), F (optical flow sequence), C (object class vector sequence) of the data are exploited.

[0092] In this variant, the model is trained for the three modalities I, F, C so that n*3 different predictions are obtained by the three predictors corresponding to the three modalities of the data. Each predictor has its own specific set of parameters.

[0093] Thus, for each value in the data, the model provides n predictions, which may differ for each value. These n predictions can be generated by n distinct predictor networks or by a single n-branch network.

[0094] During the backpropagation phase of the gradient, the parameters of each predictor network fI, fF, fC are optimized from the sum of the respective cost functions calculated for each modality, i.e., the sum L = ∑ T = I , C , F L NN T + λ L NP T

[0095] The calculated gradients impact the parameters of each predictor network fI, fF, fC according to the respective variations of the cost functions calculated for each of the categories. Thus, the parameters of each predictor network are updated independently.

[0096] For example, the figure 5 , predictor number 2 is retained to calculate the first cost function for the first modality I, predictor number n is retained to calculate the first cost function for the second modality F and predictor number 2 is retained to calculate the first cost function for the third modality C.

[0097] Predictor number 1 is selected to calculate the second cost function for each of the modalities I,F,C.

[0098] This example is given for illustrative purposes only, it being understood that the predictors used may be different or identical for each of the categories and each of the cost functions. They are determined independently for each category according to the criteria relating to the calculation of the cost functions described above.

[0099] The calculated gradients are backpropagated in each of the three predictor networks f I , f F , f C to their input layer then the results are summed via a summer Σ before being propagated to the set consisting of the projector network P and the recurrent network R which is common to all modalities.

[0100] In other words, the entire model is affected by the sum of the three cost functions calculated for each category, but in practice, each respective part of the model (predictors fT, recurrent network R, and projector P) is affected differently. The overall cost function is calculated as the sum of the cost functions for each category, itself a weighted sum of two cost functions, LNN and LNP. This induces backpropagation gradients of varying amplitudes in the different predictors fT, depending on the role played by the parameters of these predictors in calculating the predictions. In other words, each predictor network fT is optimized only by the gradient generated by the cost function calculated for the corresponding category. The networks P and R ultimately receive the sum of the gradients from the three predictor networks.

[0101] One advantage of using a recurrent network R is that it allows increasing the number of predictors without increasing the size of the model in terms of the number of parameters, thus reducing the requirements on the computing power of the machine running the algorithm.

[0102] There figure 6 illustrates an example of a possible architecture for the projector network P, the recurrent network R and the three predictor networks f I ,f F and f C corresponding to the basic modality (the image sequence), and the two secondary modalities (optical flow F and object class C).

[0103] In this example, the same predictor network is used for the main modality and the optical flow, while a predictor network with a different architecture is used for the object class. This example is not exhaustive; it is possible to use the same predictor network for all modalities with different parameter sets for each modality, or different predictor networks for each modality, or a combination of both approaches.

[0104] For example, the figure 6 The P-projector network is composed of a series of four convolution layers conv, three layers of maxpooling max pool, four activation layers implementing a ReLU activation function and two fully connected FC convolutional layers arranged in the order shown on the figure 6 .

[0105] The recurrent network R consists of 7 fully connected convolutional layers (of identical dimensions) alternating with an activation layer implementing a ReLU activation function. The input is summed to the output via a residual connection.

[0106] The predictor network fI,fF consists of two stacked autoencoders. The two autoencoders are identical and each comprises a four-layer convolutional encoder (conv) and four activation layers (ReLU), followed by a decoder implementing the same layers in reverse order. The last activation layer of the decoder implements a sigmoid activation function.

[0107] The input is added to the output of the first autoencoder via a residual connection. The hk state provided by the recurrent network is concatenated to the outputs of each encoder.

[0108] Also, the last layer of each decoder can be adapted for each of the two modalities I and F, in particular the size of the convolution layer can vary according to the modality.

[0109] The predictor network f C includes an encoder consisting of four convolutional layers (conv) and three maxpooling layers. max pool, five activation layers implementing a ReLU activation function and two fully connected FC convolutional layers arranged in the order shown on the figure 6 .

[0110] The encoder output is concatenated to the hk state provided by the recurrent network and is provided as input to a decoder consisting of two fully connected convolution layers, a ReLU activation layer and an activation layer implementing the Softmax activation function.

[0111] The architectural examples given to the figure 6 are illustrative and not exhaustive.

[0112] Once the machine learning model has been trained to reconstruct "normal" data through several credible predictions, it can be used to detect anomalies in a new dataset that may include anomalies.

[0113] To this end, an anomaly detection process involving the implementation of the model described above is proposed to the figure 7 .

[0114] The process is applied to 701 data of the same nature and modalities as those used for training, with the difference that they may now contain anomalies.

[0115] At step 702, the data is masked using the same masking procedure as in step 202 of the training, and then the masked data is provided as input to the previously trained model.

[0116] At step 703, the model calculates several predictions in the same way as at step 203 of the training.

[0117] In step 704, only the first cost function is calculated. L NN ( Y ) using equation (1) by selecting the prediction closest to the reference data. The cost function is calculated taking into account one or more categories of the available data.

[0118] Finally, in step 705, an anomaly score is calculated which is based on the difference between the cost function L NN ( Yand a characteristic value of the average of this cost function calculated during training. Indeed, if the input data 701 contains no anomalies, the predictors will correctly reconstruct the hidden parts of the data, and the calculated cost function will be close to the average of this function calculated during training. Conversely, if the input data contains an anomaly in the hidden areas, the predictors will not reconstruct this anomaly, and the cost function calculated in step 704 will have a value significantly different from that calculated during training.

[0119] An example of an anomaly scoring formula is given in relation (3): S Y = ∑ T ∈ T AM w T L NN T Y T − μ T σ T

[0120] T AM = { I, F, C} is the set of modalities that the variable T traverses L NN T Y T is the cost function calculated for each modality.

[0121] µ T And σ T are respectively the mean and standard deviation of the same cost function calculated during training for training data that do not contain any anomalies.

[0122] w T is a weighting coefficient that allows different weights to be given to each modality.

[0123] The anomaly score formula can be replaced with any other metric that measures the difference between the cost function calculated in step 604 and a representative value of the cost function calculated on anomaly-free training data.

[0124] In step 706, the anomaly score is compared to a detection threshold to deduce the presence or absence of anomalies in the input data.

[0125] The invention can be implemented as a computer program comprising instructions for its execution. The computer program can be stored on a storage medium readable by a processor.

[0126] The reference to a computer program that, when executed, performs any of the functions described above, is not limited to an application program running on a single host computer. Rather, the terms computer program and software are used here in a general sense to refer to any type of computer code (e.g., application software, firmware, microcode, or any other form of computer instruction) that can be used to program one or more processors to implement aspects of the techniques described herein. Computing means or resources may, in particular, be distributed ( " Cloud computing"), possibly using peer-to-peer technologies. The software code can be executed on any suitable processor (e.g., a microprocessor) or processor core, or a set of processors, whether located in a single computing device or distributed across multiple computing devices (e.g., as potentially accessible within the device's environment). The executable code for each program enabling the programmable device to implement the processes according to the invention can be stored, for example, on the hard drive or in read-only memory. Generally, the program(s) can be loaded into one of the device's storage means before being executed.The central unit can command and direct the execution of instructions or portions of software code of the program(s) according to the invention, instructions which are stored in the hard drive or in read-only memory or in the other storage elements mentioned above.

[0127] The invention can be implemented on a computing device based, for example, on an embedded processor. The processor can be a generic processor, a specific processor, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The computing device can use one or more dedicated electronic circuits or a general-purpose circuit. The technique of the invention can be implemented on a reprogrammable computing machine (a processor or a microcontroller, for example) executing a program comprising a sequence of instructions, or on a dedicated computing machine (for example, an array of logic gates such as an FPGA or an ASIC, or any other hardware module).

[0128] The invention makes it possible to improve both reconstruction and prediction learning approaches by using a trained multi-prediction model to reconstruct masked normal data.

[0129] Furthermore, using multiple predictors trained to perform different pretext tasks for various modalities of multimodal data allows for a better capture of the diversity of the "normal" nature of the training data. Indeed, learning predictions according to different modalities—spatial, temporal, optical flow, or others—allows for a better characterization and discrimination of the different credible contents of normal data. Références

[0130] [1] Duc Tam Nguyen, Zhongyu Lou, Michael Klar, Thomas Brox. Anomaly Detection With Multiple-Hypotheses Predictions. In ICML, 2018 [2] Zhian Liu, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction. In ICCV, 2021 [3] Mariana-luliana Georgescu et al.: Anomaly Detection in Video via Self-Supervised and Multi-Task Learning,ARXIV.ORG, 2021

Claims

1. Computer-implemented method for training a model for reconstructing multimedia data represented by at least one modality (201), the model comprising a set of a plurality of different predictors for each datum modality, the training method comprising the steps of, for each datum of a training dataset containing no anomalies, - for each datum modality: • masking (202) at least part of the datum modality, • training each predictor of the set associated with said modality to compute (203) a different prediction of the same masked datum, each predictor being specialized in one possible prediction of the masked datum among various alternatives considered normal in the context of said multimedia data, • for each predictor, computing a distance between the prediction and a reference datum extracted from the training data, • selecting the predictor of the set corresponding to the smallest distance, - computing (204) a first cost function equal to the sum, over all the modalities, of the distances corresponding to each selected predictor, - updating (206) the parameters of the predictors selected for each modality so as to minimize the first cost function during training of the predictors.

2. Method for training an anomaly-detecting model according to claim 1, further comprising, - for each datum modality: • selecting a subset of the set of predictors that have not been optimized in an earlier iteration of the training, • computing the sum of the distances between the respective predictions provided by the predictors of said subset and the reference datum, - computing (205) a second cost function equal to the sum of said distances for all the modalities and modifying the first cost function by adding thereto the second cost function weighted by a weighting factor, - updating (206) the parameters of the predictor that provides the closest prediction to the reference datum and of the predictors of said subset for each modality so as to minimize the modified first cost function.

3. Method for training an anomaly-detecting model according to any one of the preceding claims, wherein: - the multimedia data (201) are represented, in a basic modality, by a temporal sequence of successive images, - the step of masking (202) at least part of the basic modality comprises applying a predefined spatial mask to each image of the sequence so as to mask at least one area of the image, - the reference datum corresponds to one successive image in the temporal sequence with respect to the current image delivered as input to the model, - each predictor is trained to predict said successive image from the current image masked by means of said mask.

4. Method for training an anomaly-detecting model according to claim 3, wherein: - the multimedia data (201) are further represented, in an additional modality, by an optical flow sequence, - the step of masking (202) at least part of the additional modality comprises masking the entirety of the optical flow, - the reference datum is the optical flow, - each predictor is trained to predict the optical flow from the masked current image in the basic modality.

5. Method for training an anomaly-detecting model according to any one of claims 3 or 4, wherein: - the multimedia data (201) are further represented, in an additional modality, by a sequence comprising, for each image, a set of classes of objects detected in the image, - the step of masking (202) at least part of the additional modality comprises masking the entirety of the sequence of classes of objects, - the reference datum is the sequence of classes of objects, - each predictor is trained to predict the sequence of classes of objects from the masked current image in the basic modality.

6. Method for training an anomaly-detecting model according to any one of claims 3 to 5: - the multimedia data (201) are further represented, in an additional modality, by an audio sequence synchronized with the sequence of images, - the step of masking (202) at least part of the additional modality comprises removing at least part of the audio sequence, - the reference datum is the audio sequence, - each predictor is trained to predict the audio sequence from the masked current image in the basic modality.

7. Method for training an anomaly-detecting model according to any one of claims 1 or 2, wherein: - the multimedia data (201) are represented, in a basic modality, by a set of images, - the step of masking (202) at least part of the basic modality comprises applying a predefined spatial mask to each image so as to mask at least one area of the image, - the reference datum corresponds to the current image provided as input to the model but not masked, - each predictor is trained to predict said current image from the current image masked by means of said mask.

8. Method for training an anomaly-detecting model according to claim 7, wherein: - the multimedia data (201) are further represented, in a second modality, by a set of multispectral images, - the step of masking (202) at least part of the second modality comprises removing images at at least one given wavelength, - the reference datum is the set of multispectral images, - each predictor is trained to predict one multispectral image from the masked current image in the basic modality.

9. Method for training an anomaly-detecting model according to any one of the preceding claims, wherein the model comprises: - a first projector neural network (P) receiving as input a masked datum and trained to convert the input into a latent representation (h0), - a recurrent neural network (R) trained to produce a succession of states (h1, ... hn) recurrently, the initial state (h0) being equal to the latent representation, - a plurality of predictor neural networks (fT) each corresponding to one modality, each predictor network receiving as input the masked datum and a state provided by the recurrent network, the number of states generated by the recurrent network being equal to the number of predictors for one modality.

10. Computer-implemented method for detecting anomalies in a multimedia dataset having at least one modality, the method comprising the steps of: - executing (703), for said dataset, the machine-learning model trained by means of the training method according to any one of the preceding claims, the model receiving as input the data (702) masked by means of masks identical to those used for the training of said model and producing as output a plurality of predictions, - computing (704) the first cost function from the predictions generated by the model, - computing (705) an anomaly score from a distance between the first cost function and a value representative of the first cost function for a dataset containing no anomalies, - comparing (706) the anomaly score with a predetermined detection threshold and deducing therefrom the presence or absence of anomalies in the data.

11. Computer program comprising code instructions for implementing methods according to any one of claims 1 to 10 when said program is executed on a computer.

12. Computer-readable recording medium on which the computer program according to claim 11 is recorded.